The Less the Merrier? Investigating Language Representation in Multilingual Models
Authors: Hellina Hailu Nigatu
Atnafu Lambebo Tonja
Jugal Kalita
Venue: EMNLP 2023 Findings (2023)
Keywords: Multilingual
low-resource
pre-trained models
linguistic diversity
embedding space
dialect
writing script

Abstract: Multilingual Language Models offer a way to incorporate multiple languages in one model and utilize cross-language transfer learning to improve performance for different Natural Language Processing (NLP) tasks. Despite progress in multilingual models, not all languages are supported as well, particularly in low-resource settings. In this work, we investigate the linguistic representation of different languages in multilingual models. We start by asking the question which languages are supported in popular multilingual models and which languages are left behind. Then, for included languages, we look at models' learned representations based on language family and dialect and try to understand how models' learned representations for (1) seen and (2) unseen languages vary across different language groups. In addition, we test and analyze performance on downstream tasks such as text generation and Named Entity Recognition. We observe from our experiments that community-centered models---models that focus on languages of a given family or geographical location and are built by communities who speak them---perform better at distinguishing between languages in the same family for low-resource languages. Our paper contributes to the literature in understanding multilingual models and their shortcomings and offers insights on potential ways to improve them.

The Less the Merrier? Investigating Language Representation in Multilingual Models
Hellina Hailu Nigatu', Atnafu Lambebo Tonja2.3.4, Jugal Kalita"
University of California Berkeley, USA, Uniiersityy Colorado, Colorado Springs, USA,
3 Instituto Politécnico Nacional, Mexico, Lelapa Al
Correspondence: hellina nigatu@berkeley..de atonjaBuccs.edd
Abstract
For languages that have been included in pop- ular multilingual models, performance is not the same for all languages (Joshi, 2020). Low-resource languages suffer in performance even when they are included in multilingual models, Several works (Hangya and Fraser., 2022; Wang and Roth, 2020; Pfeiffer and Ruder., 2021; Schuster and Lewis, 2019) have proposed methods for improving per- formance for low-resource languages. Previous work (Doddapanení and Kumar., 2021) presents an argument that languages might benefit from be- ing included in multilingual models as the models learn language-independen feature representations, while it concludes that the question of the benefit of multilingual training for a given language remains open. Previous works also show that multilingual models might suffer from "negative interference" (Zirui Wang, 2019) in both high-resource (Con- neau et al., 2019; Xu Tan and Liu., 2019) and low- resource (Zirui Wang, 2020) settings.
Multilingual Language Models offer a way to incorporate multiple languages in one model and utilize cross-language transfer learning improve performance for different Natural Language Processing (NLP) tasks. Despite progress in multilingual models, not all lan- guages are supported as well, particularly in low-resource settings. In this work, in- vestigate the linguistic representation of dif- ferent languages in multilingual models. We start by asking the question which languages are supported in popular multilingual models and which languages are left behind. Then, for included languages, we look at models' learned representations based on language fam- ily and dialect and try understand how mod- els' learned representations for (I) seen and (2) unseen languages vary across different lan- guage groups. In addition, we test and ana- lyze performance on downstream tasks such text generation and Named Entity Recogni- tion. We observe from our experiments that community-centere models-model, that fo- languages of given family geo- graphical location and are built by communities who speak them- perform better at distinguish- ing between languages in the same family for low-resource languages. Our paper contributes the literature in understanding multilingual models and their shortcomings and offers in- sights on potential ways to improve them.
In this work, we first look at the linguistic diver- sity in multilingual models. We then investigate the embeddings for different languages in multilingual models and show how the representations affect downstream performance in language identifica- tion. For our analysis, we used three autoregressive and five autoencoder models. First, we looked at 2D visualizations of learned representations for all models. Then, we evaluated autoregressive mod- cls' performance on text generation and the autoen- coder models by language classification based on learned representations and performance on Named Entity Recognition (NER). We base our analysis on (1) language family, (2) dialects, and (3) writ- ing scripts, with focus on low-resource language settings.
1 Introduction
While we have seen improvements state-of-the art performance in various NLP tasks by multilin- gual models (Conneau et al., 2019; Doddapaneni and Kumar., 2021), there is disparity in which languages actively studied. The field of NLP has largely been Anglocentric, with large portion of the world's languages, particularly low-resource languages, not being covered in the literature (Joshi, 2020: Bender, 2019; Talat, 2022).
Related Works
In efforts to include more of the world's 's languages, previous works have built multilingual language models over the years. While models with larger
Equal contribution.

---

numbers and more diverse sets of languages have shown commendable performance in several NLP
XLM-R (Conneau al., 2019) is multilingual Transformer-based MLM trained on the Common Crawl data for 100 languages. To balance data between English and other languages, the XLM-R uses one dump for English and 12 dumps for all other languages.
tasks (Conneau et al., 2019; Zhang et al., 2021), we have also seen coammunity-centere models im- prove upon task performance (Dossou et al., 2022; Dabre et al., 2022). Previous work (Conneau et al., 2019) hypothesizes that models might suffer from "curse of multilinguality". which describes how for given model size, increasing the number of lan- guages does not improve performance for individ- ual languages after a certain point. With this in mind, we ask the question: Do Multilingual Lan- guage Models with fewer, coaumunity-ceenteree lan- guages learn better representations of different lan- guages depending on language families, dialects, and writing scripts?
BERT multilingual (Devlin al., 2018) is pre-trained model on the top 104 languages with the largest Wikipedias using an MLM objective. It designed to pre-train deep bidirectional represen- tations from unlabeled text by jointly conditioning on both the left and right context in all layers.
AfroLM (Dossou et al., 2022) is multilin- gual language model pre-trained from scratch on 23 African languages using self-active learning framework with an MLM objective.
Previous works have analyzed how pre-trained language models learn representations for differ- ent languages using probing tasks (Choenni and Shutova, 2020; Eva Schlinger, 2019; Jindrich Li- bovicky, 2019) as well as investigating the geome- try of sub-spaces (Chang and Bergen, 2022; Rajaee and Pilehvar, 2022). One previous work (Chang and Bergen, 2022) focuses understanding multi- lingual language models' overall embedding space geometry and identifies axes for ainguage-senstive and language-neutral features. In our work, we are interested how different languages are rep- resented in multilingual language models with a closer look at how different language families, di- alects, and writing scripts are represented.
IndicBERT (Doddapaneni et al., 2022) is a mul- tilingual ALBERT (Lan et al., 2019) model pre- trained exclusively on 12 major Indian languages. It is pre-trained on a novel monolingual corpus of around 9 billion tokens and subsequently evaluated set of diverse tasks.
AraBERT (Antoun al., 2020) is an Arabic pre- trained language model based on BERT (Devlin et al., 2018) and trained on 70 million sentences following the original BERT pre-training objective.
3.2 Autoregressive Models
Autoregressive models are sequential models that use the previous tokens to predict the next token. Transformer-bused autoregressive models use a transformer decoder and causal masked attention to learn sequential representation regressively.
3 Models and Data
GPT-3 (Brown al., 2020) is generative model with 175 billion parameters. It has been used several downstream tasks and to demonstrate in- context learning.
We chose models from two categories for our ex- periments: autoencoder and autoregressive models. In this section, we give descriptions of the models we chose and their training data. Table I gives summary of the models we used with information on their training data, model size, and languages covered.
LLaMa is an autoregressive model that is trained on "publicly available datasets exclusively" (Touvron et al., 2023).
BLOOM (BigScience, 2022a) an autoregres- sive model that is trained on the ROOTS corpus (BigScience, 2022b).
3.1 Autoencoder Models
Autoencoder models are trained to recreate their inputs from an internal latent representation of the input (Dor Bank, 2021). Many such models use partial input masking during training, known as masked language modeling (MLM). In this paper. we focus on munsformer-based autoencoders. In particular, we look at BERT (Devlin al., 2018) and RoBERTA (Liu et al., 2019) models, which use MLM.
Language Diversity
Before starting our experiments to understand how multilingual models learn representations for differ- ent languages, we first looked which languages are included and which languages are excluded from mainstream NLP research. From the mod- els discussed in Section 3, we selected XLM-R

---

Model Type
Model Name
Size
Languages
Data
XLM-R
270M
100
Filtered CommonCrawl
AfroLM
270M
23
JW300, Bible, News
Autoencoders
BERI-multllinual
110M
104
BooksCorpus and English Wikipedia
AmBERT
110M
Arabic news, Arahic Corpus, OSIAN: the Open Source International Arabic News Corpus
IndicBERT
12M
12
Al4Bharat's monolingual corpus
GPT-3
175B
119
Filterod CommonCrawl. WebText (Liu and Curran, 2006), e-books, English Wikipedia English CommonCrawl, C4, Github, Wikipedia, Gutenberg and Books3, ArXiv, Stack Exchange ROOTS (BigScience, 2022h)
Autoregressive
LLaMA
65B
20
BLOOM
560M
59
Table Models and their parameter size, number of languages included, and their training data sources (Conneau et al., 2019; Dossou et al., 2022; Devlin et al., 201 8; Antoun et al., 2020; Doddapaneni et al., 2022; Brown et al., 2020; Touvron et al., 2023; BigScience, 2022b). In our experiments, we only used base or small models due to computational resource limitations. For GPT-3, we obtained the language count from github.
and LLaMA from generic multilingual models and AfroLM from community-centeree models to eval- uate their linguistic diversity across countries. First, we look at the linguistic diversity of the community- centered model, AfroLM. In Fig (a), show map representing the number of languages in each African country represented by AfroLM. We con- trast these results with Fig 1 (b), where we rep- resent the same data by dividing the number of languages represented in the model by the number of languages spoken in the country. We see in Fig- ure I that a large number of languages per country are still left behind, even in cases where the models are focused
Dataset
Sampling prop
Languages
CommonCrawl
67%
English
C4 Github
15%
English
4.5%
English
English. Bulgarian
Catulan, Czech
Danish, German
Spanish, French
Croatian, Hungarian
Wikipedia
4.5%
Italian, Dutch
Polish, Portuguese
Romanian, Russian
Slovenian, Serbian
Swedish, Ukrainian
Books ArXiv
4.5%
English
2.5%
English
StackExchange
2%
English
single continent.
Table 2: Pre-training data of LLaMA (Touvron et al., 2023). large portion the data English, with other European languages included collectively making up less than 4.5% of the total data.
Next, we look at XLM-R which has been a pop- ular choice multilingual studies (Choudhury and Deshpande, 2021). We see that 55 out of the 100 languages included in XLM-R are Indo-European in comparison with 2 each from Niger-Congo and Kra-Dai languages included in the model. In Fig. 2. the Indo-Buropean language family tree from (Young, 2015) show the languages that are represented from that family by XLM-R. For LLaMA, Table 2, shows the languages included in the training data, which are exclusively in the European branch of the Indo-European family with the exception of Hungarian which is member of the Uralic language family.
space representation and language classification for the Tigrinya dialcct, we used a dataset from (Haileslasie et al., 2023). To evaluate NER, we used MasakhaNER (Adelani et al., 2021) dataset for Bantu languages and WikiANN (Pan et al., 2017) dataset for Romance languages.
For the text generation task, we designed our prompts in English five categories: Bible, World Knowledge, News, Generic, and Bias. We chose these categories (1) to explore what pre-trained LLMs represent (in World Knowledge, Bias, and News), (2) to get closer to training data for low- resouree languages (in Bible), and (3) to observe the behaviors of pre-trained LLMs in generic situa- tions in different languages (in Generic). We trans- lated the English prompts into Amharic, Tigrinya, Spanish, Italian, and French. For Spanish, Ital- ian, and French, we used Google Translate. For
5 Methods
5.1 Evaluation dataset
We used publicly available datasets evaluate multilingual models. For embedding space rep- resentation and language classification, we used Flores (NLLB Team, 2022) dataset except for the Tigrinya dialect evaluation. To evaluate embedding

---

(a) AfroMM- languages per country
(b) AfroLM languages out of total languages spoken in the country
Figure I: Visualization of languages included in AfroLM. In Fig. 1 (a), we show the languages that are included in the model per country. We observe some geographic diversity in countries included: East, West, and Southern African countries included in the model. However, looking at Fig. (b), dividing the number of languages included AfroLM by number languages that are spoken in country gives us the contrast that even cases where models concentrated single continent, many of the languages are left unrepresented.
also experimented with averaging different hidden states of all tokens and found that while it reduced within cluster distances (distances between individ- ual points in already formed clusters), it retained the overall cluster formations. For LLaMA and BLOOM, used the hidden state vector of the last token in each sentence as the sentence embed- ding for all layers, as was done in previous work (Neelakantan et al., 2022). For GPT-3, we used the embedding models from OpenAT's API end- points; we used the rext-embedding--da---0 and text-similarity -iviuriit--aiiic-000 models. From Afro- Asiatic language family, we choose Semitic and Cushtic languages. From Niger-Congo language family, chose Bantu languages. From Indo- European language family, we choose Indo-Iranian and Romance languages.
Figure 2: Language family tree for audo-eur lan- guages showing how many the languages this fam- ily are included in XLM-R. We used the size of the oranges to give an intuition of how big a language based the number of speakers. (Image is not drawn to scale.) We see more concentration on the European side of the tree than on the Indo-Iram ado-Iranian side.
Tigrinya, we used Lesan.ai (Hadgu et al., 2022) with manual corrections after translation and for Amharic, we translated manually. The prompts we used are in Appendix A.
Distinguishing between Languages through Classification: To corroborate the results we ob- served in the visualization of the embedding spaces, used K-Means clustering the learned repre- sentations extracted from the pre-trained mod- els test what extent different models can dif- ferentiate among languages. In Section 6.2, we discuss the result of language classification for dif- ferent language groups and models.
5.2 Embedding Space Extraction and Visualization
Distinguishing between Languages through Vi- sualization: We wanted to understand how differ- ent models represent different languages in multi- lingual settings. We extracted the hidden states from the models in our experiments and used UMAP (Leland Mclnnes, 2020) visualize the representations. Following previous work (Devlin et al., 2018), we used the hidden state vectors of the first token in each sentence as the sentence em- bedding for all layers for autoencoder models. We
5.3 Downstream Tasks
Scholars have previously evaluated the perfor- mance of Large Language Models (LLMs) in differ- ent downstream tasks (Adelani et al., 2021, 2023; Dione et al., 2023). Our interest understanding how multilingual models learn and represent lan-

---

guages from different language families, dialects, and writing scripts. Hence, in addition to investi- gating the models' learned representations through visualization and classification, we evaluated how these models perform in downstream tasks across languages, focusing on two tasks: NER and text generation.
Tigrinya in both autoregressive (in Fig. 3) and au- toencoder (in Fig. 5) models. Looking more detail at the representations that are close together for Amharic and Tigrinya in AfroLM, we observe that the sentences are translations of each other (in Fig. 4).
Named Entity Recognition (NER): NER is an Information Extraction task which serves as one of the "fundamental building blocks of complex NLP tasks" (Singh, 2018). We selected NER task to understand how selected autoencoder models perform tasks that require deeper language under- standing capabilities for models. In this work, we evaluated Bantu and Romance family languages, and we discuss the results in Section 6.3.
(a) GPT-3 Ada
(b) GPT-3 Davinci
Text Generation: For autoregressive models, we evaluated downstream applications by prompting the models in 6 languages from our experiment and using GEEZSwitch (Gaim and Park, 2022) to classify the generated text. For this experiment, we chose GPT-3 and BLOOM. We prepared three prompts in categories for total of 15 prompts per language. More details about the prompts are in Appendix A. We then generated text eight times for each prompt for a total 120 text generations per language per model and 1440 generations overall.
(c) LLaMA
(d) BLOOM
Figure 3: Autoregressive models' learned representa- tions for Semitic languages. GPT-3 embeddings show independent clusters for all languages with some mix between Arabic (orange) and Amharic(blue). LLaMa and BLOOM on the other hand, separate Arabic as separate cluster but mix Tigrinya(green) and Amharic.
6 Results
6.1.2 Dialect
In addition to language families, we wanted to in- vestigate how multilingual language models learn representations for languages with different di- alects. We selected Tigrinya and Arabic languages from the Semitic language group for this experi- ment. For Arabic, chose four dialects: Morocco, Levantine, Egypt, and Standard Modern Arabic, based on the numbers of speakers. For Tigrinya di- alects, we choose three dialects used (Haileslasie et al., 2023). Our result shows that for the Arabic dialect, except AraBERT, all the models mixed the representations for all dialects. The AraBERT model clustered some sentences from the Egyp- tian dialect independently but mixed the rest of the dialects (in Appendix Fig. 19). Similarly, all the models mix the representations for Tigrinya dialects, as shown in Figure
6.1
Visualizing Models' Embedding Spaces
6.1.1 Language family
As detailed in Section 4, the language familics rep- resented in the multilingual models are predomi- nantly Indo-European. We wanted to investigate how multilingual models learned representations vary for different language families both for seen and unseen languages.
For Romance languages, XLM-R shows language-neutral semantic clusters the middle layers, with language-specific clusters in the last layer (in Appendix D.3 Fig. 22 (b)). In XLM-R, Bantu language representations were mixed in pairs for all languages except Swahili which formed somewhat of an independent cluster (in Appendix D.3, Fig. 22 (a)). AfroLM showed independent clusters for 3 of the Bantu languages with Xohsa and isZulu mixing (in Appendix D.3 Fig. 21). For Semitic languages, we observed, except for XI.M- R, AfroLM and GPT models, all other language models had mixed representations for Amharic and
6.1.3 Writing Script
We also evaluated how multilingual models rep- resent languages that use different writing scripts. For this experiment, we selected Amharic, Ara-

---

Bantu language classification, AfroLM showed the highest performance with Fl-score of 79% while IndicBERT showed the lowest Fl-score of 13%. For both Arabic and Tigrinya dialects, all the mod- els show an Fi-score below 40%, this shows all the models are strugeling to classify different dialects within the languages. For different writing scripts, AraBERT showed Fl-score of 62% while XLM-R showed the lowest Fi-score of 37%.
cimt sout Jobey Mr falt Teai Li an sar
Figure 4: Taking a closer look at AfroLM representa- tions for Semitic languages, we observe that in cases where Tigrinya representations are mixed with the Amharic cluster, the sentences are closer to their trans- lation pairs from the dataset. In the instance that an Ambaric sentence is mixed with the Tigrinya embed- ding, there a 6-letter English acronym the sentence.
Models
Language
Correct Language Generated (%)
Spanish
95.83
French
91.67
Italian
91.67
GPT-3
English
90.83
Amharic
76.66
Tigrinya
45,00
bic, Spanish, Chinese, and Hindi, which use Geez (Ethiopic), Arabic, Latin, Chinese (Han), and Hindi (Devanagari) scripts, respectively. Our result in Fig- ures 7 and 8 show that all models except AraBERT form somewhat distinct language-specific clusters. While XLM-R and mBERT clustered Amharic separately, other languages were clustered near each other with some cross-language mixing. The AraBERT model independently clustered Spanish and Arabic while mixing the other languages. As shown in Figure 8, GPT-3 models showed some sppccii clusters with cross-language mix- ing, while LLaMA and BLOOM had separate clus- for each language.
Spanish
93.33
French
100
Italian
51
BLOOM
English
100
Amharic
80.00
Tigrinya
43.33
Table 3: Accuracy BLOOM and GPT-3 generating text in the same language it prompted with. High- resource languages have over 90% accuracy except for Italian in BLOOM
6.3 Named Entity Recognition
Table 5 shows NER results for Bantu and Romance language families. In both NER tasks, we observed identical distinctions between the models as seen in the visualization experiments (Section 6.1). For the Bantu languages, AfroLM outperformed other models except in Kinyarwanda, while generic mod- els outperformed community-centeras models for the Romance languages except for French.
6.2 Language Classification
Table shows the language classification results for autoencoder models across different language groups. As discussed in Section 6.1, we are inter- ested in evaluating how multilingual models clas- sify languages regardless of their writing script, language families, and dialects. We used Fl score evaluate the clustering performance. As shown Table 4, we observed similar differences be- tween the models language classification tasks as seen in the visualization experiments (Section 6.1). Models that showed separate clusters in the embed- ding space across language families and writing scripts showed the highest Fl score than the rest of the models. For Semitic and Cushtic languages,
6.4 Text Generation
In Table 3, we show the results of language classi- fication for the text generation task. GPT-3 genera- tions are in the same language of the prompt above 90% of the time for high-resourced languages. A deeper look at the generations that were in dif- ferent language for English reveals that for the prompt of verse from the Bible, GPT-3 generates the exact name of the verse "(John 3:16)" which GEFZSwitch detects as German language, account- ing for out 12 of the wrong language identifi- cations in English. Two of the remaining wrongful
AfroLM classified all the languages correctly with an F1-score of 100%, while for Indo-Iranian lan- guage classification, IndicBERT classified all the languages correctly with Fi-score of 100%. For

---

5500
(a) XLM-R
(b) MBERT
(c) AfroLMM
(d) ArBBRRT
(c) IndicBERT
Figure 5: Autoencoder models learned representations for three Semitic languages. We observe that the representa- tions for Tigrinya and Amharic (which use the same writing script) are mixed in all models except XLM and AfroLM.
(a) XLM-R
(b) mBERT
(c) AfroLM
(d) AmBBRT
(e) IndicBERT
Figure 6: Autoencoder representations for Tigrinya dialects. All models mix the representations for all three dialects.
(a) XLM-R
(b) mBERT
(c) AfroLM
(d) AraBRR
(e) IndicBERT
Figure 7:
Autoencoder learned representations for languages with different writing scripts. Except for AraBERT, other models form somewhat distinct gegguage-speciffe clusters, while AraBERT mixes Amharic, Chinese, and Hindi and has separate clusters for Arabic and Spanish.
Models (F (FI-score)
Language groups
General
Community Centric
XLM-R
mBERT
AfroLM
AraBERT
Indi
Semitic
0.68
0.52
1.0
0.61
0.62
Cushetic
0.52
0.41
1.0
0.33
0.37
Bantu
0.37
0.23
0.79
0.18
0.13
Romance
0.42
0.24
0.38
0.34
0.21
Indo-Iranian
0.33
0.30
0.71
0.2
1.0
Dialects
Arabic_dialects
0.25
0.23
0.25
0.32
0.26
Tigrinya
0.23
0.30
0.39
0.31
0.26
Writing script
Different writing script
0.37
0.49
0.55
0.62
0.53
Table Language classification Fl-scores for K-Means clustering of the embedding space for autoencoder models. Here, we see that community centereed models perform better at distinguishing between languages they focus on. while none the models perform well in dialectic and writing script categories.
detection results were due to GEEZSwitch detect- ing list of African countries in English Swahili. The remaining wrongful detection was mix of Amharic and English for the prompt "I am tourist, Tell me common phrases in "mharic."
While GPT-3 does decently on Amharic, closer analysis reveals that 14% of the generated text, which classified as Amharic, also includes boil- erplate code and a mix of English. For Tigrinya, 51.51% % of the mis-generated text is in Amharic,

---

Models(F1-scome
Language Family
Language
General XLM-R
Community-Ceme Centtri IndicBERT
mBERT
AfroLM
AraBERT
zul
84.6
81.7
86.3
76.9
67.2
xho
85
87.5
75.8
75.3
Bantu sna
93.6
92.4
94.4
73
83.4
swa
87.5
86.3
87.6
78,9
79.9
kin 73.9
70.9
72.8
69.3
71.1
fra 88.95
90.66
87.89
91.15
91.97
spa 94.58
92.03
80.12
81.71
83.73
Romance
cal 91.72
95.67
82.76
85.24
84.48
ita 90.43
91.91
80.28
83.82
83.25
Table NER Performances: Fl-scores on languages test sets, these results cover four tags (PER, ORG, LOC, DATE) in the MasakhaNER dataset for Bantu languages and WikiANN dataset for Romance languages.
English and French and 80% for Amharic. In con- trast to GPT-3, the generated text is in the same language of the prompt 51% of the time for Italian, with Spanish accounting for 40.67% of the mis- generations, Catalan and French each accounting for 27.11% of the mis-generations, and English accounting for 5.08% of the mis-generation. In Appendix B, Figure 10, we go into further depth on some examples of generations Tigrinya and Amharic, highlighting the existence of artifacts that seem to be inherited from web interfaces.
(a) GPT-3 Ada
(b) GPT-3 Davinci
7 Discussion
(c) LLaMA
(d) BLOOM
7.1 On Inclusion of Diverse Languages and Language Families
Figure 8: Autoregressive learned representations for languages with different writing scripts. Here, LLaMA and BLOOM more distinct language specific clus- ters while GPT models show some mix across language clusters.
Our analysis linguistic diversity in popular mul- tilingual models aligns with criticism of previous works that a large proportion of the world's lan- guages remain understudied. We observe skew towards Indo-European languages, with a heav- ier skew towards the European side of the fam- ily tree (Section 4). This indicates there is still huge room for improvement in the NLP commu- nity, particularly in encouraging community-driven research for the inclusion of a more diverse set languages, dialects, and writing scripts. Looking at the community-centere models, we see there is greater diversity and more inclusion for low- resource languages, though there is still long way go (Section 4). Encouraging research driven by communities of such languages could allow the NL. community to benefit from more diversity per community, which collectively could result greater diversity overall.
and 45,45% is in English. The remaining 3% are attempts at romanized Tigrinya, which were mis- classified as German and Swahili. In Appendix B, Figure 9, show examples of text that were detected generated in language other than the prompt language. We want emphasize that these results are conditioned on the fact that we are look- ing ONLY at the language of the generated text; NONE of the Amharic and Tigrinya generations, even when they are in the same language as the prompt, are coherent sentences phrases.
For BLOOM text generation performance, we see in Table that generated text is in the same language as the prompt for 100 % of the time for

---

7.2 Are Community-Centered Language Models Better at Distinguishing between Languages?
resource languages behind, and (2) we wanted to observe how models deal with out-of-domain languages. From our visualizations, we observe that some models cluster unseen languages based on writing scripts. For example, for Semitic lan- guages, LLaMa, BLOOM, mBERT, AraBERT, and IndicBERT clustered Tigrinya and Amharic, lan- guages which both use the Ge ez script, together and formed separate cluster for Arabic (in Fig. 3 and Fig. 5). AfroLM and XLM-R formed language- specific clusters for all three languages even though Tigrinya unseen for both models while Amharic is seen.
From the visualizations of the learned represen- tations for different languages (Section 6.1) and the language classification for autoencoder models (Section 6.2), we observe that community-centered models' representations are more distinct for the languages they focus on. Looking at Semitic lan- guages, only AfroLM and XLM-R had separate clusters for each language (Amharic, Arabic, and Tigrinya), while all other models put Tigrinya and Amharic in the same cluster (Fig. 5). For AfroLM, we see two Tigrinya sentences that were placed closer to the Amharic cluster are closer to their translation pairs from the dataset (Fig. 4) while XLM-R mixed representations were not explain- able as such. We see this behavior in visualizations of representations for other language families in Appendix D.
8 Conclusion
In this work, we investigated linguistic diversity, the learned represeniation of different languages, and the downstream task performance of multilin- gual models. We observe from experiments that community-centert language models perform bet- ter at distinguishing among languages in the same family for low-resource languages. We also ob- serve that there is still a long way to go in accommo- dating dialectical differences in NLP. Our work con- tributes to understanding multilingual models, and we hope to see future work with more community- centered models and more diverse languages.
For autoregressive models, we did not have community-centenn models, as defined in our ab- stract, to compare with. However, we looked at the output of the text generated from GPT-3 and BLOOM models. We looked at 6 languages and observed that Amharic and Tigrinya are mixed in the generated text, while the generated text was not coherent for either of the languages. There is still long way to go in terms of text generation for low-resource languages, starting with models that respond the same language as the prompt.
Limitations
Our study is limited by the models included in the experiments and the languages available in the data we used. For instance, after we submitted our pa- per, models like GPT-SW3 (Ariel Ekgren, 2023) and Jais (MBZUAI, 2023) presented community- centered autorgressive models. Future work can do a more comprehensive study with more lan- guage models and more diverse data included. Fu- ture work can also dig deeper into particular points raised in our work, such as dialectical diversity the effects of writing systems in language model representations.
7.3 Same Language, Different Dialects
We also looked at learned representations for lan- guages with different dialects. We observe from our visualizations and language classification periments that, for both Tigrinya and Arabic, the learned representations form no particular cluster depending on dialect for any of the models for Tigrinya with a small cluster observed for Egyptian Arabic in AraBERT representations (Section 6. 1.2). This shows there is huge room for improvement in the NLP space for understanding and accommodat- ing dialectical differences.
Additionally, we note that using dimension re- duction techniques to understand representation might lead to loss of some separate spaces which are collapsed down. In our work, we used clus- tering techniques top of learned representations and downstream task performance further corrob- orate our observations. Future work could also in- vestigate difference between community-centern and generic models through an investigation of the geometric space of the models. Additionally, while
7.4 Learned Representations for Unseen Languages
In our experiments, also included languages that are not seen by the models in training. We did this because (1) including languages that all models have in common would leave a lot of low-

---

we showed some correlation between separate clus- ter formation in visualizations and downstream per- formance (Section 7.2), still unclear if there is a causal relationship between models that have separate language clusters and their downstream performance. Future work could build upon our results to investigate whether or not a causal rela- tionship exists.
Zhuowen Tu Chang. Tyler A and Benjamin K Bergen. 2022. The geometry of multilingual language model representations. Proceedings of the 2022 Conference on Empirical Methods Natural Language Process- ing.
Rochelle Choenmi and Ekaterina Shutova. 2020. What does it mean to be language-agnostin probing multi- lingual sentence encoders for typological properties. arxiv preprint arxiv.:2099 12862.
Monojit Choudhury and Amit Deshpande. 2021. How linguistically fair multilingual pre-trained lan- guage models? In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, pages 12710-12718
References
David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D' souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijh- wani, Sebastian Ruder, et al. 2021. Masakhaner: Named entity recognition for african languages. Transactions of the Association for Computational Linguistics, 91116-1133
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke 7ettle- moyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning scale. arXiv preprint urXiv:1911.022
David Ifeoluwa Adelani, Marek Masiak, Israel Abebe Azime, Jesujoba Oluwadara Alahi, Atnafu Lam- bebo Tonja, Christine Mwase, Odunayo Ogundepo, Bonaventure FP Dossou, Akintunde Oladipo. Doreen Nixdorf, et al. 2023. Masakhanews: News topic classification for african languages. arXiv preprint arXiv:2304.09972.
Raj Dabre, Himani Shrotriya, Anoop Kunchukuttan, Ratish Puduppully, Mitesh Khapra, and Pratyush Ku- mar. 2022. IndieBART: A pre-trained model for indic natural language generation. In Findings of the As- sociation for Computational Linguistics: ACL 2022. pages 1849-1863, Dublin, Ireland. Association for Computational Linguistics.
Wissam Antoun, Fady Baly, and Hazem Hajj- 2020. Arahert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language under- standing. CoRR, abs/18 8 10.04805.
Felix Stollenwerk Joey Öhman Tim Isbister Evan- gelia Gogoulou Fredrik Carlsson Alice Heiman Judit Casademont Magnus Sahlgren Ariel Ekgren, Amaru Cuba Gyllensten. 2023. Gpt-sw3: An autoregres- sive language model for the nordic languages. arxiv preprint arxiv:2305. 12987.
Cheikh M Bamba Dione, David Adelani, Peter Nabende, Jesujoba Alabi, Thapelo Sindane, Happy Buzaaba, Shamsuddeen Hassan Muhammad, Chris Chinenye Emezue, Perez Ogayo, Anuoluwapo Aremu, al. 2023. Masakhapos: ofreehhfs tagging for typo- logically diverse african languages. arXiv preprint Mtw.2305.13989
Emily M. Bender. 2019. The benderrule: On naming the languages we study and why it matters. The Gradient.
Gowtham Ramesh Anoop Kunchukuttan Doddapaneni, Sumanth and Pratyush Kumar. 2021. A primer on pretrained multilingual language models.
Mitesh M. Khapra
BigSeience. 2022a. Bloom: 176b-parameter open- access multilingual language model. arxiv preprint arxiv:2211.05100.
Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreyansh Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2022. Towards leaving no indic language behind: Build- ing monolingual corpora, benchmark and models for indic languages. ArXiv, abs/2212.05 055409
BigSeience. 2022b. Bloom: 176b-parameter open- access multilingual language model. arxiv preprint arxiv:221.005.1
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scout Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Raja Giryes Dor Bank, Noam Koenigstein. 2021. Au- toencoders. arxiv preprint axxv:2003.0599)
Bonaventure FP Dossou, Atnafu Lambebo Tonja, Oreen Yousuf, Salomey Osei, Abigail Oppong. Iyan- uoluwa Shode. Oluwabusayo Olufunke Awoyomi, and Chris Chinenye Emezue. 2022. Afrolm: A self-active learning-based multilingual pretrained lan- guage model 23 african languages. arXiv preprint arXiv:2211.0330

---

Dan Garrette Eva Schlinger. 2019, How multilingual is multilingual bert? Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics.
2022. Text and code embeddings by contrastive pre- training. arXiv preprint arXiv:2201. 10005.
James Cross Onur Çelebi Maha Elbayad Kenneth Heafield Kevin Heffernan Flahe Kalbassi Janice Lam Damiel Licht Jean Maillard Anna Sun Skyler Wang Guillaume Wenzek AI Youngblood Bapi Akula Loic Barrault Gabriel Mejia Gonzalez Prangthip Hansanti John Hoffman Semarley Jarrett Kaushik Ram Sadagopan Dirk Rowe Shannon Spruit Chau Tran Pierre Andrews Necip Fazil Ayan Shruti Bhos- ale Sergey Edunov Angela Fan Cynthia Gao Vedanuj Goswami Francisco Guzmán Philipp Koehn Alexan- dre Mourachko Christophe Ropers Safiyyah Saleem Holger Schwenk Jeff Wang NLLBB Team, Marta R. Costa-jussà. 2022. No language left behind: Scaling human-centered machine translation.
Wonsuk Yang Gaim, Fitsum and Jong Park. 2022. Geezswitch: Language identification in typologically related low resourced east african languages. Pro- ceedings of the 13th Conference on Language Re- sources and Evaluation (LREC 2022).
Asmelash Teka Hadgu, Abel Aregawi, and Adam Beau- doin. 2022. maaan-machine translation for low re- source languages. In NeurlPS 2021 Competitions and Demonstrations Track, pages 297-301. PMLR.
Asfaw Gedamu Haileslasie. Asmelash Teka Hadgu, and Solomon Teferra Abate. 2023. Tigrinya dialect iden- tification. In 4th Workshop on African Natural Lan- guage Processing.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Noth- man, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Pro- ceedings of the 55th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), pages 1946-1954 Vancouver, Canada. As- sociation for Computational Linguistics.
Hossain Shaikh Saadi Hangya, Viktor and Alexander Fraser. 2022. Improving low-resource languages in pre-trained multilingual language models. Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing.
Ivan Vulić Iryna Gurevych Pfeiffer, Jonas and Sehastian Ruder. 2021. Unks everywhere: Adapting multilin- gual language models new scripts. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
Alexander Fraser Jindrich Libovicky, Rudolf Rosa. 2019. How language-neutrrs is multilingual bert? Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics.
Sebastin Santy Amar Budhiraja Kalika Bali Mono- jit Choudhury. Joshi, Pratik. 2020. The state and fate of linguistic diversity and inclusion in the nlp world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 6282-93.
Sara Rajace and Mohammad Taher Pilehvar. 2022. An isotropy analysis in the multilingual bert embedding space. In Findings of the Association for Computa- tional Linguistics: ACI.
Sonal Gupta Rushin Shah Schuster, Sebastian and Mike Lewis. 2019. Cross lingual transfer learning for mul- tilingual task oriented dialog. In Proceedings of the 2019 Conference of the North America Association for Computational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learn- ing of language representations. arXiv preprint 1909v 11942.
Sonit Singh. 2018. ing for information extraction. arXiv preprint arXiv .2.383..
Natural language process-
James Melville Leland McInnes, John Healy. 2020. Umap: Uniform manifold approximation and pro- jection for dimension reduction. arxiv preprint 102.:0880..0346..
Aurélie Névéol Stella Biderman Miruna Clinciu Manan Dey Shayne Longpre Sasha Luccioni et al. Talat, Zeerak. 2022. You reap what you sow: On the chal- lenges of bias evaluation under multilingual settings. In Proceedings of BigScience Episode 5 Workshop on Challenges Perspectives in Creating Large Lan- guage Models, 26-41, virtual+Dublin: Association for Computational Linguistics.
Vinci Liu and James R Curran. 2006. Web text corpus for natural language processing. In 11th Conference of the European Chapter of the Association for Com- putational Linguistics, pages 233-240.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint 1rXiv:1907.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1399 13971.
MBZUAL 2023. Jais: a new pinnacle in open arabic nlp.
Karthikeyan K Stephen Mayhew Wang, Zihan and Dan Roth. 2020. Extending multilingual bert to low- resource languages. In Findings of the Association for Compulational Linguistics: EMNLP.
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Rad- ford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al.

---

Di He Tao Qin Zhou Zhao Xu Tan, Yi Ren and TieYan Liu. 2019. Multilingual neural machine transla- tion with knowledge distillation. arXiv preprint arXiv: 1902.990020.
tourist. Tell me common phrases in Amharic.
Bias
Holly Young. 2013. language family tree in pictures. The Guardian.
What characteristics do African people have?
What characteristics do European people have?
Xinsong Zhang, Pengshuai Li, and Hang Li. 2021. AM- BERT: A pre-trained language model with multi- grained tokenization. In Findings of the Association
for Computational Linguistics: ACL-IJCNLP 2021,
Habiba was born and raised in Kombolcha, Ethiopia. Habiba's s profession is
pages 421 421-435, Online. Association for Computa- tional Linguistics.
B
Text Generation Samples
Barnab as P'oczos Jaime Carbonell Zirui Wang. Zi- hang Dai. 2019. Characterizing and avoiding nega- tive transfer. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition.
Language detail
Table 6 shows the details of languages included in the experiment.
Yulia Tsvetkov Zirui Wang, Zachary C. Lipton. 2020. On negative interference in multilingual models: Findings and meta-learning treatment. arxiv preprint a3x11:2010.03017.
D Embedding space
In this section, we present the UMAP plots for the Romance, Cushtic, Bantu, and Indo-Iranian language families and the Arabic dialects. We also present the visualization for all layers in XLM-R and AfroLM for Bantu and Romance languages.
A Prompts
Bible
In the beginning, God created the heavens and the carth
D.1 Language family
In this subsection, we present the visualizations for Romance (Fig. and Fig. 12), Cushtic (Fig. 13 and Fig. 14), Bantu (Fig. 15 and Fig. 16) and Indo-Iranian (Fig. 17 and Fig. 18) families.
For God so loved the world, that He gave His only begotten Son, that whoever believes in Him shall not perish, but have eternal life
What is the verse Matthew 6:33?
D.2 Dialect
World Knowledge
In this section, we present the visualizations for the embeddings for Arabic dialects (Fig. 19 and Fig. 20).
am history student. Can you tell me about the Battle of Adwa?
D.3 Embedding space for Bantu and Romance -All layers
List 10 African countries.
Which African country was the first to gain independence?
News
What is COVID-19?
How do protect myself from COVID-19?
Who is more likely to get COVID-19?
Generie
Write an article about studying abroad.
Tell me about the different seasons in a year.

---

Prompt
Prompt
Detected Generated Language
Generated Text
Language
Amharic
MIG r'rishh Y12-17 *Hnn (Translation: List 10 African countries)
English
Still, God's intention for the covenant He had made with Israel was never be broken. Instead. His plan was remain steadfast that could fulfill His promises and prove His faithfulness.
English
For God loved the workd, that He gave His only begotten Son, that whoever believes in Him shall not perish, but have eternal life
German
(John 3:16)
Tigrinya
Ayere m--""9 Aagss Mi Hhmmmm??
English
primary language of the islands of Papua New Guinea Tok Pisin, creole of English, German, and Austronesian languages.
(Translation: What
characteristics do Europeans have?)
Figure 9: Examples of cases where the generated text was detected as in language other than the prompt language. We show case where English prompt has text generated that detected as German, also showcase where Amharic and Tigrinya prompts result in English text generations.
No.
Prompt
Prompt Language
Generated Text
Predicted Language
A447 hHiLe-19 A287 AIMIA
AFFAII-?'
'emailuser' => 'N40.7 hNLLe19
KAna nhii,c=19 A284 AnANA A-FAn=?
Ang AnAnd A-FAAD=:''
'emailuser2' => '2-02 hhiiee19
(Translation: how can protect myself from COVID-19?)
A78 AlAhA A-F-AAD-?'.
amh
"emailuser3' => 'AZ-1
amh
2
?4иÀ cAns: hyt Ni-x-19 Mamm
MhaTina ATIAA?''
'emailuser' => 'MCAA CADE trf
77HA CAGE hit WT,,e -iffio Пнлтии A ATAAA?
Ni,e-19
'emaaluser2's 'AncA CAAP h-]l
(Translation: how can protect myself from COVID-19?)
Ni.e-19 McCA CAΔe hil NILe-19 'emailuser3' 'MCA CAa
tir
3
< 6:33 rン SAA? antamhnneeemmmmatmmmmmm aaammaannnmmeeeenmmmmamama h00000000h
rontbr 6:33 "r
(Translation: What does Matiwos 6:33 say?)
amh
4 WLO Arl A-t-P-R berr1a3t hoyey m- Zim, + 90.F (Transiaaion) Habiba was born and raised in Ethiopia's Kombolcha city. Habiba works as
V0. Av11 A---P-R-" h1"na97 hop-y
+ WLes van
heaamaamnoonnneeemmm hatemmnaaaeeeeeemmmmmm G000000000000
byn
Figure 10: Generation examples for Amharic and Tigrinya using BLOOM model. We see that the generations for Tigrinya and Amharic have repeated characters that make for long strings; such generations are mostly misclassified as Bangla(byn). On the other hand, even in cases where the generated text is predicted to be in the same language the prompt language, are artifacts 'emailuser' that appear the generated text for these languages.

---

(a) XLM-R
(b) MBERT
(c) AfroLM
(d) AraBERT
(c) IndicBERT
Figure 11: Last layer autoencoder embeddings for Romance languages. We see that XLM-R and AraBERT form some language-specific clusters while rest have mixed representations for the languages.
GPT-3 Ada
GPT-3 Davinci
(c) LLaMA
(d) BLOOM
Figure 12: Learned representations for autoregressive models for Romance languages. All models form some language-specific clusters while BLOOMM forms the most distinct clusters for all lanaguegs.
(a) XLM-R
(b) mBERT
(c) AfroLMM
(d) AraBERT
(c) IndicBERT
Figure 13: Last layer autoencoder representations for Cushtic languages. We observe clearer separation in XLM-R and AfroLM, while all other models have mixed representations.
(a) GPT-3 Ada
(b) GPT-3 Davinci
(c) LLaMA
(d) BLOOM
Figure 14: Learned representations autoregressive models Cushtic languages. Except GPT-Ada, other models have mixed representations.
(a) XLM-R
(b) MBERT
(c) AfroLM
(d) AmBBRRT
(e) IndieBERT
Figure 15: Last layer autoencoder representations for Bantu languages. We observe language-specific clusters for AfroLM with isZulu Xhosa mixing, while XLM-R forms clear cluster Swahili and somewhat mixed clusters for the other languages. The other models mix all the languages.

---

(a) GPT-3 Ada
(b) GPT-3 Davinci
(C) LLaMA
(d) BLOOM
Figure 16: Learned representations from autoregressive models for Bantu languages. We observe that LLaMA and BLOOM have mixed representations for the Bantu languages, while there is small cluster for some of the Swahili sentences. While GPT-3 Ada separates out all languages, GT3-Davinei has some mix for Shona and Kinyarwanda and mix for isZulu and Xhosa.
(a) XLM-R
(b) mBERT
(c) AfroLM
(d) AraBERT
(c) IndicBERT
Figure 17: Last layer representations for Indo-Iranie languages. All models have mixed representations for all the Ando-Iranian languages.
(a) GPT-3 Ada
(b) GPT-3 Davinci
(c) LLaMA
(d) BLOOM
Figure 18: Learned representations from autoregressive models Indo-Iranian languages. Here, we see LLaMA and BLOOM have some language specifie clusters with LLaMA mixing Bengali and Assamese. GPT-3 Davinci mixed Persian, Hindi, and Assamese, while GPT-3 Ada mixed Persian and Hindi and also mixed Gujarati and Assamese.
(a) XLM-R
(b) mBERT
(c) AfroLM
(d) AraBERT
(c) IndicBERT
Figure 19: Last layer representations the Arabic dialects. All models have mixed representations for all dialects. while we observe small cluster for Egyptian Arabic in AraBERT.

---

No. of
Languages
Family
speaker
Arabic (ara)
Afro-Asiatic/ Semitic
630M
Amharic (amh)
Afro-Asiatic/ Semitic
57M
Tigrinya (tir)
A Afro-Asiatic/ Semitic
9M
Afaan Oromo (orm)
Afro-Asiatic/ Cushitic
37M
Somali (som)
Afro-Asiatic/ Cushitic
22M
French (fra)
Indo- Europpnnn Romance
350M
Catalan (cat)
Hudo-European/ Romance
9.2M
Spanish (spa)
Mudo-Europeann Romance
595M
Italian (ita)
Indo-Buropean/ Romance
68M
Hindi (hin)
Indo- Europeann Indo-Iranian
592M
Iranian Persian (per)
Indo- Europearppann// Indo-Iranian
81M
Assamese (asm)
europeur Indo-Iranian
15M
Bengali (ben)
Indo-European/ Indo-Iranian
234M
Gujarati (guj)
Indo- /ndo-European/ Indo-Iranian
56M
isZulu (zul)
Niger-Congo/Bantu
26M
Swahili (swa)
Niger-CCogoo/Bant
88M
isoSh (xho)
Niger-Congo/Bantu
19M
Kinyarwanda (kin)
Niger-Congo/Bantu
15M
Shona(sna)
Niger-Congo/Bantu
17.8M
Chinese (zho)
Indo- Indo-Chinese (Sino-Tibetan)
1.35
Table 6: Languages included in the experiment.
(a) GPT-3 Ada
(b) GPT-3 Davinci
(6) LLaMA
(d) BLOOM
Figure 20: Learned representations from autoregressive models for Arabic Dialects. Both BLOOM and LLaMA models have mixed representations for all dialects, while GPT models form dialect-speciffc clusters.

---

(a) Bantu
(b) Romance
layers for Bantu and Romance Languages. For Bantu languages, we observe that Xhosa and isZulu occupy the same cluster while all other languages form independent clusters. We observe that the middle layers, the individual clusters are further away from each other and start spreading closer in the last layers. For Romance languages, observe that all languages mixed with clear cluster formed for any of the languages.
Figure 21: AfroLM representations

---

(a) Bantu
(b) Romance
Figure 22: XLM-R representations on all layers for Bantu and Romance Languages. We observe for Romance languages, the representations in the middle layers, particularly layers 8, 9, and 10, exhibit semantic clusters, and the last layers show language-specific clusters. For Bantu languages, we observe some language-specifie clusters in the middle layers, with Kinrwanda and Shona representations mixing well Xhosa and isXulu forming pair. Only Swahili forms an independent cluster for Bantu languages.


Reviews:
[{"review_id": "LjZSPgRuZx", "reviewer": "Reviewer_6SvF", "summary": "", "questions": "- Why do you use the CLS token for encoder models and the last token for LAMA? Did you experiment with averaging all tokens and other possible options?\n- Why did you exclude mT5 and AfriTeVa? You stated there are no autoregressive community-focused models, but AfriTeVa exists.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first want to thank the reviewer for their valuable feedback. Regardless of the outcome for this conference, it helped us critique our own work and think of experiments to run to strengthen our arguments. Below, we address issues raised in the feedback:\n\n1.  Why do you use the CLS token for encoder models and the last token for LAMA? Did you experiment with averaging all tokens and other possible options? | Did not give reason for the choice of sequence representations, e.g. why last token for LAMA? \n\nIn terms of choice for which tokens to use for sequence representation, we chose the last token for autoregressive models following previous work (Neelakantan et. al 2022). For autoencoder models, we used CLS token hidden states as that is used as a sentence representation in BERT based models (Devlin 2019). Following the valuable inquiries from reviewers we performed additional experiments with strategies like averaging different hidden states of all tokens and found that it reduced with-in cluster distances (distance between individual points in already formed clusters) but retained the overall cluster formations. We will add this observation in our updated paper. \n\n2. Why did you exclude mT5 and AfriTeVa? You stated there are no autoregressive community-focused models, but AfriTeVa exists. | Missing important benchmarks like mT5 for an \"generic\" autoregressive model and AfriTeVa for a community-focused autoregressive models\n\nIn this work, we did not look at Seq-to-Seq models due to time and resource constraints. Hence, we did not look at AfriTeva or mT5. When we claimed there are no autoregressive models that fit our definition of community-centered model, we were referring to decoder-only models; hence, AfriTeva or mT5 models do not belong to this category. We will clarify this point further in the updated version of the paper. Additionally, we ran experiments with visualization of embedding from AfriTeva and mT5 for Bantu languages after reviewer comments and found no clear clusters for any of the languages using UMAP and t-SNE. We will explore all language families and put the results in the updated paper. \n\n3. Only consider T-SNE (which has several limitations) as a dimensionality reduction technique\n\nWe first recognize the limitation of using only t-SNE in our analysis. We initially started the experiments as visualizations of the embedding space in multilingual models and had the downstream tasks to see if the visualization observation was matched in the downstream task performance. We then experimented with UMAP as a dimensionality reduction method for some of the language families in our experiments with autoencoder models and did not observe significant divergences in the visualizations. Hence, we did not do further investigation with UMAP.  Following reviewers' inquiries, we ran additional experiments for Bantu and Romance language families using UMAP as a reduction technique with all models and observed significant divergence for GPT-3. We saw that for autoencoder models like AfroLM, UMAP showed separate clusters for Kinyarwanda, Swahili and Shona while Zulu and Xosha were mixed; identical to the observation in t-SNE visualizations. However, with GPT-3 embedding, UMAP showed separate clusters for all languages while in t-SNE there were no language independent clusters. We observed similar difference for Romance and Semitic languages when using UMAP on GPT-3 embedding.  We will include these results in the updated version of the paper. \n\n4. Unclear methodology for the text classification and text generation experiments.\n\nWith regards to our explanation of the methodology, we describe the dataset we used in Sec 3.3 and the methodology in Sec. 5.1 and 5.2. Since we do not have a separate subsection detailing the text classification task, we understand how it might be unclear for readers. We will separate out the different evaluation tasks and describe them in more detail in the updated paper. If there is an additional concern over the clarity of our methodology, we are happy to incorporate that feedback as well.  \n\n5. Insufficient contributions for a long paper.\n\nThere are limited works in understanding multilingualism from a low-resourced language perspective. We believe our work adds sufficient insights into this space and offers jumping of points to explore LLMs for multilinguallity and low-resource NLP. We also believe that our work aligns well with the Theme Track topic and adds to the conversation. Additionally, we have run additional experiments based on feedback from the reviewers which strengthens our arguments and increases the robustness of our work. As such, we argue that our work adds significant contributions to this space.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Investigates multilingual representations in both community-focused and more generic models\n-", "reasons_to_reject": "- Only consider T-SNE (which has several limitations) as a dimensionality reduction technique\n- Did not give reason for the choice of sequence representations, e.g. why last token for LAMA?\n- Missing important benchmarks like mT5 for an \"generic\" autoregressive model and AfriTeVa for a community-focused autoregressive models\n- Insufficient contributions for a long paper. \n- Unclear methodology for the text classification and text generation experiments.", "questions_to_authors": "- Why do you use the CLS token for encoder models and the last token for LAMA? Did you experiment with averaging all tokens and other possible options?\n- Why did you exclude mT5 and AfriTeVa? You stated there are no autoregressive community-focused models, but AfriTeVa exists.", "paper_topic_and_main_contributions": "This paper investigates the representations of languages in community-focused multilingual language models and more \"general\" language models. They analyze representations based on language families, script and dialects. They observe that:\n- Community-focused models perform better at distinguishing between languages for low-resource languages\n-  Models that showed separate clusters in embedding space across language families and writing scripts showed the highest F1 score for KNN text classification of languages, compared to other models\n- Models struggle with dialect separation", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- AfroXLMR - https://arxiv.org/pdf/2204.06487.pdf\n- AfriBERTa - https://aclanthology.org/2021.mrl-1.11/\n- KinyaBERT - https://aclanthology.org/2022.acl-long.367.pdf\n- AfriTeVa - https://aclanthology.org/2022.deeplo-1.14.pdf"}}, {"review_id": "m6vfVb5chc", "reviewer": "Reviewer_NTN2", "summary": "", "questions": "Question A: Do you have results on your generation tasks for the BLOOM model? Is there any reason you left BLOOM out of the analysis? I think it would add a lot to the analysis to see if we can say \"community-centered models are better\" (or not) also in autoregressive models. \n\nNote A (No need to reply to this): In case you haven't, and you are planning to evaluate a community-centered LLM, you could also use \"GPT-SW3: An Autoregressive Language Model for the Nordic Languages\" (which could not be included in your paper, because it was published very recently).\n\nQuestion B: Have you evaluated these models (at least the autoencoder ones), on an NLU downstream task like NER or similar? (I understand if you have not, as most of us have limited resources of GPUs and people/time, thus, leaving it out of the scope of this work).", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first want to thank the reviewer for their valuable feedback. Regardless of the outcome for this conference, it helped us critique our own work and think of experiments to run to strengthen our arguments. Bellow, we present our response to the reviews:\n\n\n1. Do you have results on your generation tasks for the BLOOM model? Is there any reason you left BLOOM out of the analysis? I think it would add a lot to the analysis to see if we can say \"community-centered models are better\" (or not) also in autoregressive models. \n\n\nWe agree BLOOM is a great contribution to multilingual NLP and is rooted in collaborative community-oriented research, democratizing the development and deployment of LLMs. However, we did not include it in our analysis as it did not fit our definition of community-centered model as detailed in our abstract (‘’models that focus on languages of a given family or geographical location and are built by communities who speak them’’). After reviewer comments, we ran some experiments with BLOOM to see how it compares to the autoregressive models in our experiments. We ran visualization (using UMAP and t-SNE) for Bantu languages and found that, similarly to LLaMa, BLOOM embeddings show separate clusters for Swahili, while all other Bantu languages are clustered together. But, even in the mixed clusters, BLOOM’s representations showed a little more distinction for individual languages compared to LLaMa. Additionally, we ran the text generation task on bloom in the five languages and found that  BLOOM generated text in the language of the prompt at a higher rate than GPT-3 for French, English and Amahric. We have added the results in the table below. We agree with the reviewer that, since BLOOM explicitly includes low-resourced languages, it is a good addition to our comparisons. Hence, we will run further experiments for all other language groups with BLOOM and include the results in our updated paper. \n\nModel | Language | Correct Language Generated (\\%) |\n|:---: |:---: |:---: | \n`GPT-3` |Spanish|  95.83 |\n`GPT-3` |French | 91.67 |\n`GPT-3` | Italian | 91.67 |\n`GPT-3` | English |  90.83 |\n`GPT-3` | Amharic | 76.66|\n`GPT-3` | Tigrinya | 45.00 | \n|:---: |:---: |:---: | \n`BLOOM` | Spanish | 93.33 |\n`BLOOM` | French | 100 |\n`BLOOM` | Italian | 53.33 |\n`BLOOM` | English |100 |\n`BLOOM` | Amharic | 80.00 |\n`BLOOM` | Tigrinya | 33.33 |\n\n2. Note A (No need to reply to this): In case you haven't, and you are planning to evaluate a community-centred LLM, you could also use \"GPT-SW3: An Autoregressive Language Model for the Nordic Languages\" (which could not be included in your paper, because it was published very recently).\n\nWe thank the reviewer for the pointer to GPT-SW3: An Autoregressive Language Model for the Nordic Languages; we were not aware of this recent model at the time we submitted our paper. \n\n3. Have you evaluated these models (at least the autoencoder ones), on an NLU downstream task like NER or similar? (I understand if you have not, as most of us have limited resources of GPUs and people/time, thus, leaving it out of the scope of this work)\n\nWe agree with the reviewer that evaluating downstream tasks like NER would provide further insight into the performance of different languages and on NLU tasks. However, we scoped our work to focus on understanding and visualizing the embedding of multilingual models and how those distinguish between different language groups. Evaluating on NLU tasks would require fine-tuning the model on further data for the languages of interest; we were interested in the raw representations of the sequences by the models.  However, we did collect results for NER for Bantu languages from the AfroLM paper for AfroLM, XLM-R and mBERT and we fine-tuned araBERT and IndicBERT for the 5 Bantu languages. As detailed in the table below, we found that AfroLM outperformed all models in all languages except for XLM-R in Kinyarwanda, confirming how community-centered models perform better for low-resourced languages even in NLU tasks. Hence, we will add results from experiments for NER and incorporate the reviewer’s valuable comments. \n\nLanguage (bantu) |XLM-R   | mBERT | AfroLM| araBERT| IndicBERT |\n|:---: |:---: |:---: |:---: | :---: | :---: | \n`zul` | 84.6  | 81.7  |**86.3**|76.9 |67.2|\n`xho` | 87 | 85 |**87.5**|75.8 |75.3 |\n`sna` | 93.6  | 92.4 |**94.4**|73|83.4  |\n`swa` | 87.5  |86.3   |**87.6**|78.9|79.9 |\n`kin` | **73.9** | 70.9   |72.8|69.3|71.1|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- It contributes towards our understanding of the scope of linguistic coverage of current multilingual LLMs.\n\n- It raises awareness about the poor health of many low-resource languages that are left out of the current LLM wave.\n\n- It provides a unified analysis of the representation of languages inside different LMs.\n\n- Taking into account the linguistic families, dialects and scripts to interpret the results is very insightful. \n\n- It proposes a simple evaluation method and shares prompts easily and cheaply extendable to other low-resource languages and models.\n\n- The paper is easy to follow and well organized.", "reasons_to_reject": "- Among the autoregressive language models, I missed the comparison between LLMs like llama/GPT3 created by big tech companies with the community-driven model BLOOM (Scao et al., 2022) as the authors do among the autoencoder models. The paper says (at 433-434) that \"For autoregressive models, we did not have community-centred models to compare with\", and I think that BLOOM falls in that community-centred model category (even if it does not include all the languages selected for evaluation). But I think the paper would greatly benefit from comparing BLOOM with the other LLMs on their evaluation, for some of the low-resource languages from bloom (which include many Indic and African languages among the selected 46 languages). If the model is not included in the comparison for completeness, the authors should at least reference it in related work.\n\nScao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., ... & Manica, M. (2022). Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. \n\n- The tasks selected to evaluate the performance of language models are interesting, as they do not require big annotated datasets, but they only require a shallow understanding of a language (I'm not an expert at linguistics, so I might be wrong). For example, in language identification, the presence of unique stopwords of that language, certain script or language-related accents or characters (like Ø for Scandinavian languages) can be enough to identify the languages. Including other tasks which require deeper language understanding capabilities for autoencoder models like for example NER, would help to discern between shallow coverage of low-resource languages, with a deeper understanding of the language. Datasets such as masakhaNER (Adelani et al., 2021) or WikiANN (Pan et al., 2017) could be employed for that evaluation.\n\nAdelani, D. I., Abbott, J., Neubig, G., D'souza, D., Kreutzer, J., Lignos, C., ... & Osei, S. (2021). MasakhaNER: Named Entity Recognition for African Languages. Transactions of the Association for Computational Linguistics, 9, 1116-1131.\n\nPan, X., Zhang, B., May, J., Nothman, J., Knight, K., & Ji, H. (2017, July). Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 1946-1958).", "questions_to_authors": "Question A: Do you have results on your generation tasks for the BLOOM model? Is there any reason you left BLOOM out of the analysis? I think it would add a lot to the analysis to see if we can say \"community-centered models are better\" (or not) also in autoregressive models. \n\nNote A (No need to reply to this): In case you haven't, and you are planning to evaluate a community-centered LLM, you could also use \"GPT-SW3: An Autoregressive Language Model for the Nordic Languages\" (which could not be included in your paper, because it was published very recently).\n\nQuestion B: Have you evaluated these models (at least the autoencoder ones), on an NLU downstream task like NER or similar? (I understand if you have not, as most of us have limited resources of GPUs and people/time, thus, leaving it out of the scope of this work).", "paper_topic_and_main_contributions": "In this paper, they analyze the linguistic coverage and representation of multilingual Language Models (LMs), including autoencoder and auto-regressive models. Firstly they research on the language coverage of different multilingual models, comparing models from big tech companies and community-driven models for a single language family or from a certain continent/region. Secondly, they analyze the representation of those language families across the models. They complement the analysis by testing the performance on language identification and text generation tasks. They also conclude that community-centered models identify languages and dialects better, and much better for low-resource languages.\n\nThe paper's main contribution is that it contributes towards our understanding and awareness of the scope of linguistic coverage of current multilingual LLMs.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., ... & Manica, M. (2022). Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "283: AforLM -> AfroLM?\n285: isZulu -> Zulu?\nFigure 12 (Appendix B.1.): langauges -> languages\nFigure 13 (Appendix B.1.): langauges -> lanaguges\n\nJust a note: Titles and legends of some figures are not readable unless you zoom 500-600%ish, so in case they were intended to be read (if not, you can keep them like that, we can get the idea of the plots by just the colour regions), it might be a good idea to increase the font size (I know it might be crucial for page limit, but the language names in the legends won't be readable in a printed version otherwise, so maybe you can include them with a bigger font in the appendix)"}}, {"review_id": "7ICBR2JOOW", "reviewer": "Reviewer_U2Y9", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first want to thank the reviewer for their valuable feedback. Regardless of the outcome for this conference, it helped us critique our own work and think of experiments to run to strengthen our arguments. Below, we address issues raised:\n\n1. Though t-SNE is a useful heuristic for understanding distance in high-dimensional space, I don’t think that basing empirical results off of analyzing t-SNE plots is that robust. For example, there might be rich language separation spaces in the original space that t-SNE collapses down because semantic relationships seem more important in maintaining distance\n\nWe first recognize the limitation of using only t-SNE in our analysis. We initially started the experiments as visualizations of the embedding space in multilingual models and had the downstream tasks to see if the visualization observation was matched in the downstream task performance. We then experimented with UMAP as a dimensionality reduction method for some of the language families in our experiments with autoencoder models and did not observe significant divergences in the visualizations. Hence, we did not do further investigation with UMAP. Following reviewers' inquiries, we ran additional experiments for Bantu and Romance language families using UMAP as a reduction technique with all models and observed significant divergence for GPT-3. We saw that for autoencoder models like AfroLM, UMAP showed separate clusters for Kinyarwanda, Swahili and Shona while Zulu and Xosha were mixed; identical to the observation in t-SNE visualizations. However, with GPT-3 embedding, UMAP showed separate clusters for all languages while in t-SNE there were no language independent clusters. We observed similar difference for Romance and Semitic languages when using UMAP on GPT-3 embedding. We will include these results in the updated version of the paper. \n\n2. Furthermore, it would have been nice to see more analysis or discussion about what embedding-based results can tell us about multilinguality in general. It’s not clear what kind of embedding spaces translate to good multilingual models\n\nWe agree with the reviewer that a deeper discussion around the implication of the embedding space in overall multilinguality would be interesting. For instance, looking at the geometrical structure of the embedding space and figuring out different dimensions would allow us to further argue about multilinguality and its relationship to embedding spaces. It is very well possible that, for example in a generative model, the model does generate text consistently in one language with out necessarily having separate embedding clusters for different languages (if current token is in language L1 and the language model has learned good representations regardless of weather separate clusters exist, the next predicted token could also be in language L1) However, we initially did not want to speculate too much on the implication of our results, in fear of making over arching claims about what the embedding clusters tell us. We tried to strengthen our arguments by using the down stream task of text generation and observed for example that the generations for Tigrinya and Amahric are mixed, while the embedding visualizations (both in t-SNE original results and UMAP results we collected after receiving reviews) both show the embeddings for those languages are also mixed. But, correlation does not equate to causation and so we refrained from sharing such speculations without further investigation. We hope to improve up on our work by designing further probing tasks and are willing to include experiments inspired by previous work shared by the reviewer. If accepted, we will add a section in our discussion on speculations and possible implications while also cautioning on the limitations of our work. We do however believe that our results give good jumping of points for future work and that they provide insights to the community in *how* learned representations for different language groups (dialect, language family, writing script) differ across several multilingual models, which will aid future work in understanding *why (and if)* those representations affect downstream performance. \n\n3. This next point might be more a matter of taste, but I’m not sure that Section 4 is necessary. It’s definitely interesting to map out language models in this way, but a lot of space is taken up treating this investigation as a set of findings, when it’s simply an aggregation of numbers pretty clearly reported by the model creators. Audits of models can of course be very interesting, especially given how many new models are coming out. But I do not feel that the investigation here had any framing or results that gave me new ways of thinking about the topic, or that were very in-depth and systematic between models in a way that will be useful for reference.\n\nWe agree with the reviewer that audits of models are important for NLP research. We argue that the inclusion of section 4 is crucial to our work in that it shows which languages are included and which languages are excluded from mainstream NLP models. We argue that it prevents a monolithic understanding of low-resourced-ness and avoids ‘sample/token inclusion’ of certain languages in studies as representative of all low-resourced languages. Further, it critiques language models that are built with a focus on low-resourced languages and shows how there is still a long way to go (see Fig. 2, p.4). We also argue that our analysis is inline with the theme track topic: Large Language Models and the Future of NLP.\n\n4. Overall, the question of the acuity of multilingual representation in multilingual models has a rich literature, that is slightly under-cited in this paper (see citations), and I’m not sure that this paper fills in a very robust answer for any of the questions posed in this line of literature. \n\nWe thank the reviewer for their pointers to works in the multilingual representation space and will include the citations in our camera-ready paper. We argue that our work provides insights for the question of multilingual representation in low-resourced language settings and provides jumping-off points for future research in this area. Further, with the integration of comments from all of the reviewers, we argue our work becomes stronger and provides further evidence to support our claims. We welcome any feedback the reviewer might have to make our paper even stronger.\n\n5. That being said, some of the findings and plots are interesting, and could provide good data points for future research\n\nWe thank the reviewer for this positive feedback and will continue to improve upon this work regardless of the outcome:-)\n\nWe are also grateful for the pointers on the typos, presentation and grammar issues. We will correct those in the updated paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper deals with an important and timely question, and provides an interesting overview to the issue of language representation and separation in multilingual models. \n\nThe fact that the authors consider subtleties like dialect, script, and language similarity is very cool, and I really like how this has the potential to deepen the analyses around multilingual models, and the many parts of section 6.1 in this paper are definitely a nice start towards this deepening.\n\nI think that the results in Section 6.2/Table 3 are very interesting, and I like the authors’ analysis about community-based models and what the results are in terms of low-resource language acuity.", "reasons_to_reject": "Though the motivation of the paper is very interesting, and the clustering language separation experiments are intriguing, I think that many parts of the paper do not have very robust experimental methodologies or exciting findings. \n\nI think that the visualization experiments are possibly a bit shaky. Though t-SNE is a useful heuristic  for understanding distance in high-dimensional space, I don’t think that basing empirical results off of analyzing t-SNE plots is that robust. For example, there might be rich language separation spaces in the original space that t-SNE collapses down because semantic relationships seem more important in maintaining distance. Furthermore, it would have been nice to see more analysis or discussion about what embedding-based results can tell us about multilinguality in general. It’s not clear what kind of embedding spaces translate to good multilingual models: for example, if models had the embeddings of different languages in very separated smalls subspaces, would this be better than if models reserved one dimension for language identification and used d-1 dimensions for the parallel semantic space? I think that the meaning of embedding-based results is a bit unclear without further analysis, and it would have been nice to have read some more discussion on this. \n\nThis next point might be more a matter of taste, but I’m not sure that Section 4 is necessary. It’s definitely interesting to map out language models in this way, but a lot of space is taken up treating this investigation as a set of findings, when it’s simply an aggregation of numbers pretty clearly reported by the model creators. Audits of models can of course be very interesting, especially given how many new models are coming out. But I do not feel that the investigation here had any framing or results that gave me new ways of thinking about the topic, or that were very in-depth and systematic between models in a way that will be useful for reference.\n\nOverall, the question of the acuity of multilingual representation in multilingual models has a rich literature, that is slightly undercited in this paper (see citations), and I’m not sure that this paper fills in a very robust answer for any of the questions posed in this line of literature. That being said, some of the findings and plots are interesting, and could provide good data points for future research.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper analyzes multilingual language models, examining their training in, their representation of, and their generation of a diverse set of languages. The paper looks at large-scale models largely focused on performance in high-resource languages (like GPT-3), large multilingual models like XLM-R, as well as community-based models focusing on more geographically-focused sets of languages like AfroLM. The findings include a literature analysis of which languages are represented in the language models they choose, as well as three sets of experiments: 1) looking at how sentences in different languages cluster in embedding space visualizations 2) classifying languages from the embedding space of different multilingual models to better understand representation and 3) Seeing how multilingual models can generate in different languages given different prompts that prompt towards a language.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Some older and newer references in the area that I did not see in the bibliography. No need to include all of them, but I think that some of the context of what is happening in the area of multilingual model analysis is slightly glossed over in the paper, which leads to the discussion around findings being a bit lacking: \n\nChoenni and Shutova, Cross-neutralising: Probing for joint encoding of linguistic information in\nmultilingual models\n\nNekoto et al, Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages\n\n Ògúnrèmí et al, Mini But Mighty: Efficient Multilingual Pretraining with Linguistically-Informed Data Selection\n\nLauscher et al From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers\n\nPapadimitriou et al Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERT\n\nArtetxe et al On the Cross-lingual Transferability of Monolingual Representations\n\nConneau et al Emerging Cross-lingual Structure in Pretrained Language Models\n\nWang et al On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment\n\nLibovicky et al, . How language-neutral is multilingual BERT?\n\nPires et al How Multilingual is Multilingual BERT?", "Typos_Grammar_Style_and_Presentation_Improvements": "Suggestions: \n\nFigure 1 is a bit unclear to me, and I think that there are more striking ways to present this data. From what I understand, the existence of an orange indicates whether the language is in XLM-R. However, this is a bit asymmetric, as makes it unclear which languages are left out, because “left out” is just signified with a branch without an orange (often obscured by another orange). I think there are better ways to present this data, eg with a simpler more latex-like tree where languages included are one color and languages not included are another. The diagram also does not represent the fact that there are very few non-indoeuropean languages in XLM-R.\n\nline 134 It might be nice to expand on the active learning side of AfroLM for one more sentence.\n\nFigure 2a: I’m not sure the numbers are totally accurate, I checked Nigeria (which has 1 language represented in AfroLM according to the heatmap) and AfroLM has at least Hausa, Igbo, and Yoruba. Maybe I’m misunderstanding the graph, and each language is only counted for one country even if it goes across borders?\n\nI think that section 3.3 (Evaluation Dataset) and section 5 (methods) could be together in one  methodology section. Currently, they’re separated by a section when they’re very related.\n\nIt would be great for some appendix graphs (especially those mentioned in lines 279, 283, 285\n\nFig 3 and Fig 5 could maybe be ordered the other way around, since the whole paper seems to mention autoencoder models before autoregressive models. \n\nFigs 3, 5, 6: the writing font is pretty small, could consider making it bigger\n\nSmall typos:\n\nline 38 Anglo-centeric → Anglo-centric (Or probably better without the hyphen: Anglocentric)\n\nline 283 AforLM → AfroLM"}}]

Author rebuttal:

Meta-review: This paper explores the linguistic coverage and representation of multilingual language models (MLMs), comparing models from large tech companies and community-driven models. The paper analyzes how different languages, language families, scripts, and dialects are clustered and classified in the embedding space of various MLMs, using visualization and KNN text classification. The paper also evaluates the text generation ability of MLMs for different languages and prompts. The paper finds that community-centered models have better coverage and performance for low-resource languages and dialects, while large-scale models are more focused on high-resource languages. The paper contributes to the understanding of the language representation and separation in multilingual models and provides a nice starting work towards which embedding spaces could be leveraged for visualizing multilingual models. Furthermore, the authors ran additional experiments addressing the concerns/questions from the reviewers about the choice of t-SNE, comparison with other community centered models like BLOOM, other NLU based downstream tasks and will be updating the paper draft with the findings.
Decision: Accept-Findings