RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Data size
One of the limitations of our work is the size of the benchmark data which is 1,004. Having more instances would be better. However, we believe this is an important contribution for many languages that often do not have dataset (e.g. news articles or Wikipedia articles) that can be used for topic classification annotation.
Translationese effect
One of the main limitation of our work is that the labelled dataset created for other non-English languages are based on human translation and may suffer from translationese effect including a slight drop in performance.
Few PLMs evaluated
Another limitation is the choice of multilingual pre-trained language models, we note that XLM-R may not be the best multilingual encoder out there, there are other publicly available ones like InfoXLM (Chi et al., 2021 ) , mDeBERTa (He et al., 2023 ) and others, however due to the scale of the experiments, we limited our evaluation to three multilingual models (XLM-R-base, XLM-R, and Glot-500). We believe our result may still be consistent with newer PLMs since they often cover similar set of languages as XLM-R.
Acknowledgement
David Adelani acknowledges the support of DeepMind Academic Fellowship programme. Jesujoba Alabi was partially funded by the BMBF project SLIK under the Federal Ministry of Education and Research grant 01IS22015C. This work was supported in part by Oracle Cloud credits and related resources provided by Oracle. We thank Google for providing GCP credits to train the AfroXLMR-61 and AfroXLMR-76 models. Finally, we are grateful to OpenAI for providing API credits through their Researcher Access API programme to Masakhane for the evaluation of GPT-3.5 and GPT-4 large language models.
Categorization by geographical regions
Table 2 shows the grouping of languages into regions according to the United Nations Geoscheme 13 13 13 https://en.wikipedia.org/wiki/United_Nations_geoscheme . The regions are: Africa, Americas, Asia 1 or Western & Central Asia, Asia 2 or Southern Asia, Asia 3 or South-Eastern & Eastern Asia, Europe 1 or Northern/Western/Southern Europe, Europe 2 or Eastern Europe, and Oceania. Asia, Europe, and Africa regions have the largest number of languages with 82, 57, and 56 languages respectively. The Oceania and the Americas regions have the lowest number of languages with four and five respectively.
Categorization by language family
SIB-200 languages are grouped into 21 language families as shown in Table 6 , the largest groups are: Indo-European (79 languages), Atlantic-Congo (34 languages), Afro-Asiatic (21 languages), Austronesian (21 languages) and Turkic (11 languages).
Categorization by Joshi’s classification
Table 2 also shows the number of languages in each Joshi’s class—a measure of the unlabelled or labelled resources available for each language on the web (Joshi et al., 2020 ) . 128 languages can be categorized as low-resource since they fall between class “0” and “2”, 30 languages are mid-resource in class “3”, and the others are high-resource (only 39 languages). This also corresponds to the NLLB classification for machine translation resources available on the web, but with only two categories—150 low-resource languages and 54 high-resource languages.
Categorization by availability in PLM
Lastly, we grouped languages and language families by their inclusion in the training of multilingual PLMs. XLM-R (Conneau et al., 2020 ) covered 90 out of the 205 languages in our dataset while GLOT-500 (ImaniGooghari et al., 2023 ) covered 177. This is a good indication of performance in general since languages that are included during pre-training often have better performance (Ponti et al., 2020 ; Pfeiffer et al., 2020 ; Adelani et al., 2022b ) .
Finally, we show the number of languages covered by region-specific PLMs such as AfriBERTa (Ogueji et al., 2021 ) , AfroXLMR (Alabi et al., 2022 ) , MuRIL (Khanuja et al., 2021 ) , and IndicBERTv2 (Doddapaneni et al., 2023 ) . The grouping is provided in Table 6 .
Vocabulary augmentation
to address the non-coverage of some African scripts like Nkoo and Tfng , we perform vocabulary augumentation of the original XLM-R tokenizer. We follow these steps: (1) we train a tokenizer on a combined multilingual texts for N’ko, Tamasheq (Tifinagh) and Tamazight languages using sentencepiece, and vocabulary size of 30K. (2) We added the top-20K new vocabulary tokens to the XLM-R vocabulary. (3) We performed MAFT on XLMR. The resulting model is called AfroXLMR-76-script. As an additional experiment, we repeated the vocabulary augmentation and MAFT approach for XLM-R-base model, resulting into AfroXLMR-base-76-script.
Results categorized by script
Our results in Table 7 shows that vocabulary augumentation was effective for the languages that use Nkoo and Tfng with over +18 points improvement but the performance is still lower than using MLP for these languages. This analysis shows the importance of building PLMs with diverse scripts during the pre-training phase.
Overall African languages evaluation
Table 9 shows the overall results for the African languages.
Figure 1: Heatmap of the performance by Region in each Joshi’s class.; Heatmap of the performance by Region in each Joshi’s class.
Figure 2: Fully supervised Model Performance . We group languages by whether they and their scripts are seen in the pre-training corpus of XLM-R. Languages are ordered by the XLM-R performance in every group.; Fully supervised Model Performance . We group languages by whether they and their scripts are seen in the pre-training corpus of XLM-R. Languages are ordered by the XLM-R performance in every group.
Figure 3: Accuracy of the XLM-R model vs Pre-Training corpus size in the fully supervised scenario. Bigger pre-training corpus in a target language generally improves the model performance.; Accuracy of the XLM-R model vs Pre-Training corpus size in the fully supervised scenario. Bigger pre-training corpus in a target language generally improves the model performance.
Figure 4: Script performance differences when one language has two different scripts. XLM-R and MLPs show the same trend. Using ngram features are more robust to script changes than using the XLM-R tokenizer.; Script performance differences when one language has two different scripts. XLM-R and MLPs show the same trend. Using ngram features are more robust to script changes than using the XLM-R tokenizer.
Figure 5: Comparison of Various Scenarios . We group languages by whether they and their scripts are seen in the pre-training corpus of XLM-R. Languages are ordered by the XLM-R fully-supervised performance in every group.; Comparison of Various Scenarios . We group languages by whether they and their scripts are seen in the pre-training corpus of XLM-R. Languages are ordered by the XLM-R fully-supervised performance in every group.
Figure 6: Box plot comparison of MLP and XLM-R model across regions; Box plot comparison of MLP and XLM-R model across regions
SIB-200 dataset . We provide the data size of the annotated data by their SPLIT and category
Language families covered in SIB-200 dataset grouped by United Nations geoscheme regions , Joshi’s classes (Joshi et al., 2020 ) (None – for languages not found in Joshi’s dataset), and NLLB classification (NLLB-Team et al., 2022 ) of languages by the size of resources on the internet—High-resource language (HRL) or low-resource language (LRL).
Overall result of the performance of different text-classification models across different language families. We compared different settings: fully-supervised, cross-lingual transfer and zero-shot prompting of LLMs. Cross-lingual transfer is based on the XLM-R model as it is the best-performing PLM. Performances from 4 source languages: English, French, Chinese and Arabic are reported.
Indic-centric Evaluation on SIB-200 .
African-centric Evaluation on SIB-200 .
Languages covered in multilingual pre-trained language model and their language families. We excluded MuRIL because it was trained in similar languages as IndicBERTv2 except for Santali in the Austroasiatic family.
African-centric Evaluation based on script on SIB-200 . The number of languages per script are in brackets.
Overall result of the performance of different text classification models across different languages. We compared different settings: fully-supervised, cross-lingual transfer and zero-shot prompting of LLMs. We report cross-lingual transfer performances from 4 source languages: English, French, Chinese and Arabic.
Evaluation result on different African languages pre-trained language models