RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Data source
We introduce SIB-200 —a S imple I nclusive and B ig topic classification
dataset for over 200 languages and dialects. We leveraged the multi-way parallel Flores-200 dataset (NLLB-Team et al., 2022 ) for the creation of the dataset. Flores-200 corpus is an extension of Flores-101 (Goyal et al., 2022 ) —for 101 languages. In both datasets, the source sentences were collected in English and translated by professional translators to several languages. In total, the corpus contains 3,001 sentences divided into DEV (997 sentences), DEVTEST (1,012 sentences) and TEST (992 sentences) sets. However, the authors did not release the TEST set. Additionally, we added N’Ko—a West African language that was recently added to Flores-200 dataset (Doumbouya et al., 2023 ) . 4 4 4 https://oldi.org/

Flores-200 released additional information to provide meta-data information about the domains and topics of the articles covered in the dataset. The domains are based on WikiNews, WikiJunior, and WikiVoyage with a total of 842 articles while the topics are based on “crime”, “disasters”, “entertainment”, “geography”, “health”, “nature”, “politics”, “science”, “sports”, and “travel”. 5 5 5 We note that in the open-sourced dataset, there are more categories than the ten reported in the paper. However, a quick review of the dataset revealed that at the sentence level, the article can belong to more than one topic. Therefore, we decided to add our topic categorization at the sentence level. Performing annotation at the sentence level also gives us the additional advantage of having more samples to annotate (2,009 rather than 562 instances 6 6 6 Although 842 articles are in Flores-200, only 562 articles are open-sourced as part of DEV and DEVTEST sets. ).
Data annotation
We recruited four annotators who are native speakers of English to label 2,009 sentences obtained from the DEV and DEVTEST sets of Flores-200 7 7 7 All annotators are also authors of this paper. . We make use of an internal annotation tool for text classification. The annotation labelling scheme covers 15 categories, 10 are from the original Flores-200 categorization of articles (§ 2.1 ), and the others are “business”, “religion”, “technology”, “education”, and “uncategorized”. We assigned sentences that do not fit any of the defined categories, and sentences lacking sufficient context about their topic to “uncategorized”. An example of a sentence labelled as “uncategorized” is “In Berlin, police estimated 6,500 protestors” .
The annotators took about two weeks to complete the task, however on average it takes up to 60 seconds to annotate a sentence (approximately, 33 hours of annotation time).
Quality control
We report Fleiss Kappa score (Fleiss et al., 1971 ) to measure the agreement of annotation. The Fleiss Kappa score among the four annotators is 0.44 —which signifies a moderate level of agreement.
Choosing the final label per sentence
We assigned the final label to a sentence by majority voting. Specifically, we assign a label to a sentence if at least two annotators agree on the category, but we excluded the situation, where any two annotators conflicted with the other two annotators. For example, for the sentence “The major organ of the circulatory system is the heart, which pumps the blood.” , the first two annotators assigned “science” while the last two assigned “health”. In total, we assigned a single label to 1,695 sentences, but there were 314 sentences with conflicts in the annotation. We asked the lead annotator to adjudicate the sentences with conflicting annotations and assigned a single label to each sentence. We later combined the fixed conflicting annotations with the others to give us back a total of 2009 annotated sentences.
Final classification dataset
For the final dataset, we excluded sentences with the label of “uncategorized”, we only selected label categories with more than 80 sentences, this removed categories such as “business” (80 sentences), “disasters” (73 sentences), “crime” (72 sentences), “education” (52 sentences), and “religion” (46 sentences). We note that having too many categories with few sentences makes building text classification models a bit difficult leading to a lower performance. Also, we combined “science” (138 sentences) and “technology” (114 sentences) category into a single category of “science/technology”. Finally, we removed the “nature” category because there is a lot of conflict with “science” and “geography” categories. Our preliminary experiments show that adding “nature” significantly lowers the performance of our classifier. About half of the Flores-200 is part of the SIB-200 dataset (i.e. 1004 out of 2009 sentences).

Table 1 shows the number of sentences per label in each of the TRAIN , DEV , and TEST splits. We divided the sentences into the split using the 70%, 10%, 20% ratio. The dataset will be released under CC BY-SA 4.0 licence. While the SIB-200 dataset only includes seven labels, we are also releasing another version of the dataset that is more challenging with all the 14 labels (excluding “uncategorized”). We compared the performance of English dataset using both seven and 14 labels in Appendix C .
Experimental setup
Here, we describe different categorization of languages, text classification models developed for SIB-200 , and the experimental settings (i.e. full supervised setting and zero-shot transfer setting).
Languages and their categorizations
Table 2 and Table 6 shows the grouping of languages in the SIB-200 dataset. We categorized them based on the following characteristics:
(1) geographical regions, (2) language family, (3) coverage in multilingual PLMs, and (4) Joshi’s classification (Joshi et al., 2020 ) —a categorization based on their labelled/unlabelled resources on the web—making it easy to analyze results.
Categorization by geographical regions
Table 2 shows the grouping of languages into regions according to the United Nations Geoscheme 8 8 8 https://en.wikipedia.org/wiki/United_Nations_geoscheme . The regions are: Africa, Americas, Asia 1 or Western & Central Asia, Asia 2 or Southern Asia, Asia 3 or South-Eastern & Eastern Asia, Europe 1 or Northern/Western/Southern Europe, Europe 2 or Eastern Europe, and Oceania.
Categorization by language family
SIB-200 languages are grouped into 21 language families as shown in Table 6 , the largest groups are: Indo-European (79 languages), Atlantic-Congo (35 languages), Afro-Asiatic (21 languages), Austronesian (21 languages) and Turkic (11 languages).
Categorization by Joshi’s classification
Table 2 also shows the number of languages in each Joshi’s class—a measure of the unlabelled or labelled resources available for each language on the web (Joshi et al., 2020 ) . 128 languages can be categorized as low-resource since they fall between class “0” and “2”, 30 languages are mid-resource in class “3”, and the others are high-resource (only 39 languages). This also corresponds to the NLLB classification for machine translation resources available on the web, but with only two categories—150 low-resource languages (LRLs) and 54 high-resource languages (HRLs).
Categorization by availability in PLM
Lastly, we grouped languages and language families by their inclusion in the training of multilingual PLMs. XLM-R (Conneau et al., 2020 ) covered 90 out of the 205 languages in our dataset while GLOT-500 (ImaniGooghari et al., 2023 ) covered 177. This is a good indication of performance in general since languages that are included during pre-training often have better performance (Ponti et al., 2020 ; Pfeiffer et al., 2020 ; Adelani et al., 2022b ) .
Finally, we show the number of languages covered by region-specific PLMs such as AfriBERTa (Ogueji et al., 2021 ) , AfroXLMR (Alabi et al., 2022 ) , Serengeti (Adebara et al., 2023 ) , MuRIL (Khanuja et al., 2021 ) , and IndicBERTv2 (Doddapaneni et al., 2023 ) . The grouping is provided in Table 6 .
Text classification models
We trained a simple Multilayer Perceptron (MLP), fine-tuned multilingual PLMs and prompted large language models for text classification.
Multi-Layer Perceptron
For the input features, we make use of either n-gram features (n=1 up to 3 in our experiments) or XLM-R tokens obtained by first tokenizing the sentences using XLM-R tokenizer. We make use of the default setting on scikit-learn tool (Pedregosa et al., 2011 )
Masked Language Models (MLM)
Next, we fine-tune massively multilingual PLM such as XLM-R-base (270M parameters), XLM-R (550M) Glot-500 (395M), which are trained on several languages: XLM-R and Glot-500 were trained on 100 and 500 languages respectively. We also fine-tune region-specific PLM trained on multiple country-level or continent-level languages: AfriBERTa (126M), Serengeti (278M), AfroXLMR (550M), MuRIL (236M) and IndicBERTv2 (278M). We restrict region-level analysis to Africa and India because we only found these two regions with multilingual PLMs covering many languages.
MAFT with fewer data and synthetic data
We explore how to improve over regional PLMs using MAFT—adaptation of an existing multilingual PLM to multiple or new set of languages simultaneously, this was effective for adapting XLM-R to 20 languages spoken in Africa (Alabi et al., 2022 ) . To extend to more languages, we apply MAFT to 61 African languages with at least 10MB of monolingual data (AfroXLMR-61). To further extend to more languages with less than 10MB of data, we generate machine-translated data using NLLB for 34 African languages (including 18 in AfroXLMR-61). We refer to the resulting model after adaptation as AfroXLMR-76. We provide more details on the pre-training corpus in Appendix B .
Large Language Models
Lastly, we also report results by prompting two popular large language models: GPT-3.5-Turbo (gpt-3.5-turbo-0613) and GPT-4 (gpt-4-0613). Compared with smaller language models from MLM and MAFT, they feature strong instruction-following capabilities without task-specific fine-tuning.
Fully-supervised
In this setting, we trained on each language in SIB-200 and evaluated on the same language. We did this evaluation for 205 languages and compared the performance of different text classification models. The MLP models were trained for 300 iterations, and we used either word ngram tokens or XLM-R tokens . For the multilingual PLM, we fine-tune each language training data for 20 epochs, with a maximum sequence length of 164, batch size of 16, and learning rate of 1e^{-5} on a single Nvidia A10 GPU. Here, we assume access to labelled data in the target language.
Cross-lingual transfer
For this setting, we fine-tune XLM-R on a language in Joshi’s class 5 (we call it a “source” language), and evaluate on other languages. For this setting, we fine-tune XLM-R on a language in Joshi’s class 5 (we call it a “source” language), and evaluate on other languages. We trained in four languages with three different scripts i.e. English, French, Arabic and Chinese (Simplified). Here, we assume access to labelled data in a few high-resource languages.
Zero-shot prompt
We prompt GPT-3.5/4 for text classification for the 205 languages using an English template. We make use of a simple template from Sanh et al. ( 2022 ) : ‘Is this a piece of news regarding {{ “science, technology, travel, politics, sports, health, entertainment, or geography” }} ? {{ INPUT }} ’ . Here, we assume no access to labelled data in any language
Baseline results
In order to demonstrate the effectiveness of our data set for multilingual evaluation, we benchmark the performance across various models and group the results by categorizations ( Table 3 ). As XLM-R consistently outperforms Glot-500 across almost all language families, we use XLM-R as the baseline model in the cross-lingual transfer experiments. 9 9 9 The full results are in Appendix D Comparing English versus other languages, fine-tuning XLM-R on English achieved an accuracy of 92.1%, indicating that the task itself is not difficult if given a properly pre-trained MLM and \sim 700 training samples . However, when fine-tuning the same model in other languages, the performance drops vastly to an average accuracy of 75.9%. Similarly, in the cross-lingual transfer and zero-shot prompt scenarios, the performance further drops.
Performances across language families
The distribution of accuracy scores is imbalanced across language families. Atlantic-Congo, Nilotic, Mande, Aymaran and Quechuan languages have the lowest accuracy scores . Even under the fully supervised scenario, the best-performed model reaches <65% accuracy scores on these languages. There also tends to be a larger performance gap between fully-supervised and cross-lingual transfer scenarios, suggesting a poor semantic alignment Conneau and Lample ( 2019 ) for these languages. Surprisingly, Tupian is the only additional language family that has >10% drop from the fully supervised to cross-lingual transfer scenario. When moving further to the zero-shot prompt scenario, Basque shows the biggest performance drop (-36%), next come the above-mentioned languages. Interestingly, despite this large decrease, Basque scores exceptionally high ( \approx 90%) in the fully supervised and cross-lingual transfer scenarios.
Performances across Joshi’s classes and geographical regions
Figure 1 visualizes the performance of XLM-R 10 10 10 We omit other models and only show XLM-R as Table 3 has shown fine-tuning the XLM-R model performs the best across different regions and Joshi’s classes. We see a clear trend that languages with higher Joshi’s classes perform better. Specifically, all languages with Joshi’s class \geq 3 have accuracy scores of \approx 90%. For languages in the same Joshi’s class, African languages perform the worst, and European languages perform the best . On Joshi’s class 0, African languages are even at least 20% worse than languages from other continents. Notably, there is no language with Joshi’s class >3 in Africa and no American/Oceania
languages have Joshi’s class >1. African and Oceania languages are also the only exceptions where MLP outperforms XLM-R , implying a poorly learned representation of them. Future research should focus more on languages from these regions. Appendix E provides the evaluation across the eight sub-regions instead of four in Figure 1 .
Performances across models
In the fully supervised scenario, XLM-R performs the best on 16 out of the 22 language families. Among the remaining 6 language families, applying the simplest MLP classifier with n-gram input features outperforms more complex transformer-based MLMs (Glot-500 and XLM-R), suggesting they are not well adapted to these 6 language families. Glot-500, despite being pre-trained with many more languages, outperforms XLM-R only on Sino-Tibetan languages . Even on Sino-Tibetan languages, it fails to out-perform the simplest MLP baseline. Cross-lingual transfer results are similar when using different source languages. On most language families, the results are comparable to fully supervised ones. Zero-shot prompting leads to a big drop due to the lack of supervised samples. The performance is good only for a few language families such as Indo-European, Uralic, Japonic and Koreanic.
Factors affecting performance
In order to determine the critical factor in this multilingual classification task, we conducted in-depth case studies on the model architecture choices and language categorizations.
Effect of language coverage in pre-training
Figure 2 compares MLP, XLM-R and Glot-500 models based on language and script coverage in pre-training based on four groups: (1) language seen, script seen in XLM-R (2) language unseen, script seen in XLM-R (3) script unseen in XLM-R, language seen in Glot-500 (4) script unseen by both models. The results in each group are sorted by their performance on fine-tuned XLM-R model. Overall, XLM-R performs the best on all languages seen in its pre-training corpus without any exception .
Even for languages unseen in the pre-training corpus of XLM-R, it outperforms Glot-500 in most cases as long as the written scripts are seen. Glot-500 performs the best only for 3 out of all the 205 languages, implying their learned representations are far from sufficient. The reason could be that Glot-500 is pre-trained and evaluated on a religious corpus, which is quite different from the news domain in our task. In order to achieve a better generalization, we may have to mix text from various domains in the pre-training stage.
Effect of pre-training corpora size
Figure 3 shows the change of accuracy scores with increasing corpus included in the pre-training stage of XLM-R, where the corpus size is logarithmically scaled for better visualization. We can see that with as little as 0.1GB pre-training corpus, the XLM-R model can already achieve > 80% accuracy for almost all languages , which further verified that this task itself is not difficult.
Though the accuracy generally grows with increasing corpus size, and the model performance starts to saturate with >1 GB pre-training corpus.
Effect of script
To see how the choice of scripts affects the model performance, we choose eight languages that can be written in different scripts, and visualize the performance of XLM-R, MLP with n-gram features (MLP-ngram), and MLP with words from the XLM-R tokenizer (MLP-XLM-R) in Figure 4 . We can see that (1) The performance of MLP-XLMR usually correlates with that of XLM-R. This implies that under the XLM-R tokenizer, languages have their own preferred written scripts regardless of the effects from pre-training; (because this preference stays the same even with the simplest MLP classifier);
(2) The slope of XLM-R is often steeper than that of MLP-XLMR, implying the preferred script for a language also has better pre-trained representations; (3) The slope of MLP-n-gram is often less steep. This implies that n-gram features are more robust across different scripts compared with word features obtained from the XLM-R tokenizer; (4) The preferred script is often the more commonly used one for every language. 11 11 11 We define “preferred written scripts” as the writing system or script that individuals or communities predominantly choose or favor when expressing written language.
Fine-tune vs. Prompted
Out of all the 205 languages, GPT-4 outperforms GPT-3.5-turbo in 157 languages. Only on Buginese, Kabiyè, Mizo, Nuer and Ayacucho Quechua, GPT-3.5-Turbo outperforms GPT-4 for >10\% . However, zero-shot prompting consistently underperforms fine-tuned methods. It is hard to include extensive descriptions of the classification criteria in the prompt. Adding more examples to the prompt might improve the performance.
Cross-Lingual transfer vs Fully supervised
Here, we compare the performance between cross-lingual transfer and fully-supervised methods. We observe that all languages that are included in the pre-training corpus of XLM-R, the cross-lingual transfer performs similarly to fully supervised methods. The best source language for cross-lingual transfer is, surprisingly, French, rather than English, which has the largest amount of pre-training corpus, though the difference among various source languages is tiny.
This suggests languages included in the XLM-R pre-training corpus are pretty well aligned with all the four chosen high-resource languages .
The advantage of fully supervised methods over cross-lingual transfer becomes prominent mainly when the target language is not included in the pre-training corpus of XLM-R and its script is included. In this case, fully supervised methods can improve the performance by fine-tuning the model on the target languages, but cross-lingual transfer fails to capture the alignment with high-resource languages. Figure 5 provides comparison between cross-lingual transfer and fully-supervised methods, including GPT-4 evaluation.
Evaluation of region-specific PLMs
While our evaluation is primarily focused on multilingual PLMs trained on 100 languages or more, models pre-trained on a group of linguistically or geographically related languages often lead to better performance as observed for Indian languages ( Table 4 ) and African languages ( Table 5 ). IndicBERTv2 and MurilBERT achieved better overall performance over XLM-R (550M parameters) despite their smaller capacity (236M-278M parameters), especially for Indian languages they both support, and better for languages not covered by XLM-R. Similarly for African languages, AfroXLMR—an adaptation of XLM-R through multilingual adaptive fine-tuning (MAFT) (Alabi et al., 2022 ) to 17 African languages gave roughly +9 improvement in performance. AfriBERTa on the other hand slightly gave worse result than XLM-R despite seeing the same number of African languages during pre-training (although not the exact languages) because it was pre-trained on less amount of data (1GB). Despite the improvement of AfroXLMR, it performs terribly for Nilotic, Mande and many Atlantic Congo families which shows that including more African languages in pre-training could improve performance.
Performance of applying MAFT to more African languages
We evaluated on two MAFT models described in (§ 4.1 ). Our evaluation of AfroXLMR-76 shows that MAFT with synthetic data was effective in improving the accuracy over AfroXLMR in many languages in Africa, especially for Nilotic ( +7.9 ), Mande ( +4.5 ) and Atlantic-Congo ( +7.4 ) languages, similar to the findings of Urbizu et al. ( 2023 ) .
The performance improvement for AfroXLMR-61 was smaller on average ( +3.4 ). There are few cases where it leads to a slight drop in performance on more-resource languages due to curse-of-multilinguality (Conneau et al., 2020 ) . The newly developed PLMs are available on HuggingFace. 12 12 12 https://huggingface.co/Davlan Despite the improvement in performance, African languages whose script were not covered by the XLM-R tokenizer (like N’ko and Tamazight) did not improve. To address this issue,
we provide an extension of AfroXLMR-76 with vocabulary augumentation in Appendix F .
Related Work
There have been several efforts to curate multilingual evaluation datasets, including various downstream tasks such as part-of-speech tagging (Nivre et al., 2016 , 2020 ; Dione et al., 2023 ) , named entity recognition (Pan et al., 2017 ; Adelani et al., 2022b ; Mhaske et al., 2023 ) , entity linking Botha et al. ( 2020 ) , natural language inference (Conneau et al., 2018 ) , text classification FitzGerald et al. ( 2023 ); Ma et al. ( 2023 ) , machine translation (Goyal et al., 2022 ; NLLB-Team et al., 2022 ; Adelani et al., 2022a ) , and question answering Lewis et al. ( 2020 ); Shen et al. ( 2023 ); Doddapaneni et al. ( 2023 ); Bandarkar et al. ( 2023 ) . All these initiatives have played a pivotal role in advancing the field of cross-lingual and multilingual NLP. Our work, which focuses on the creation of an extensive multilingual text classification dataset covering 200 languages, builds upon a line of related works that have contributed to the expansion of the NLP community.

Specific to text classification, a few multilingual datasets are IndicNLP BBC news (Kunchukuttan et al., 2020 ) , KINNEWS & KIRNEWS (Niyongabo et al., 2020 ) , ANTC Alabi et al. ( 2022 ) , MasakhaNEWS (Adelani et al., 2023 ) , and Taxi1500 (Ma et al., 2023 ) . To the best of our knowledge, Taxi1500 is the most recent and largest of them all covering 1500 languages. However, this dataset is focused on the religious domain as the data comes from the Bible. Our work addresses a gap in multilingual text classification datasets by curating SIB-200 that covers a broader range of topics and domains.
Conclusion
In this paper, we created SIB-200—a large scale open-sourced benchmark dataset for topic classification in 200 languages and dialects to address the lack of evaluation datasets for natural language understanding especially for low-resource languages. We performed extensive evaluation across full-supervised setting, cross-lingual transfer setting and prompting of LLMs settings. Furthermore, we grouped the 200 languages in different categories based on language families, geographical regions, Joshi’s class and coverage in multilingual pre-trained language models to provide insights into which group of languages have poor performance on this simple and inclusive benchmark.