Quantifying the Dialect Gap and its Correlates Across Languages
Authors: Anjali Kantharuban
Ivan Vulić
Anna Korhonen
Venue: EMNLP 2023 Findings (2023)
Keywords: dialect gap
machine translation
automatic speech recognition
performance correlation
multilingual
TL;DR: This paper cross-linguistically quantifies the dialect gap in MT and ASR, identifies correlates, and analyzes the impact of data features on dialect performance.

Abstract: Historically, researchers and consumers have noticed a decrease in quality when applying NLP tools to minority variants of languages (i.e. Puerto Rican Spanish or Swiss German), but studies exploring this have been limited to a select few languages. Additionally, past studies have mainly been conducted in a monolingual context, so cross-linguistic trends have not been identified and tied to external factors. In this work, we conduct a comprehensive evaluation of the most influential, state-of-the-art large language models (LLMs) across two high-use applications, machine translation and automatic speech recognition, to assess their functionality on the regional dialects of several high- and low-resource languages. Additionally, we analyze how the regional dialect gap is correlated with economic, social, and linguistic factors. The impact of training data, including related factors like dataset size and its construction procedure, is shown to be significant but not consistent across models or languages, meaning a one-size-fits-all approach cannot be taken in solving the dialect gap. This work will lay the foundation for furthering the field of dialectal NLP by laying out evident disparities and identifying possible pathways for addressing them through mindful data collection.

Quantifying the Dialect Gap and its Correlates Across Languages
Anjali Kantharuban, Ivan Vulić, and Anna Korhonen Language Technology Lab University of Cambridge {atk30, iv250, alk23 Jecam. Jecam.c..uk uc.uk
Abstract
counterparts (Trudgill, 1979). These effects are compounding; those who speak minority dialects are provided fewer opportunitics to advance so- cially and economically, resulting in self-fulfiliing cycle of oppression. Many people who speak a mi- nority dialect as first language find themselves modifying their use of language throughout their life to appear to belong to the group of standard di- alect speakers, much the detriment of the mainte- nance of dialectal diversity (Carlson and McHenry, 2006). In losing these dialects, we lose not only the form of expression itself but aspects of the unique culture and society it belongs to (Fishman, 2007).
Historically, researchers and consumers have noticed a decrease in quality when applying NLP tools to minority variants of languages (i.e. Puerto Rican Spanish Swiss German), but studies exploring this have been limited to select few languages. Additionally, past studies have mainly been conducted in monolingual context, cross- cross-linguistic trends have not been identified and tied to external factors. this work, conduct comprchensive evaluation of the most influential, state-of-the-art large language models (LLMs) across two high-use applications, machine translation and automatic speech recognition, to assess their functionality the regional dialects of several high- and low-resource languages. Additionally, we ana- lyze how the regional dialect gap is correlated with economic, social, and linguistic factors. The impact of training data, including related factors like dataset size and its construction procedure, is shown be significant but not consistent across models or languages, mean- ing one-size-fits fits--lll approach cannot be taken in solving the dialect gap. This work will lay the foundation for furthering the field dialec- tal NL.P by laying evident disparities and identifying possible pathways for addressing them through mindful data collection.
NLP has been moving in recent years to provide more methods of communication, both between people and with digital systems. In doing so, it has been bridging information- and access-based gaps for people in many historically marginalized communities (Bouillon al., 2021: Mariani et al., 2022; Zhang et al., 2022). However, is important to acknowledge that variation within languages is rarely addressed in mainstream tools. Modern sys- tems that do provide access to variants still focus on wealthy, standard dialects, such as British, Aus- tralian and American English, while disregarding commonly spoken minority dialects like African American English. Speakers of under-resourced di- alects, variants of both high- and low-resource lan- guages with little available training data, face lan- guage barriers when using many the tools taken for granted by speakers of well-resourced dialects. This reduced accessibility further entrenches exist- ing disparities by continuing the historical trend of disenfranchising speakers of minority dialects (Trudgill, 1979).
1 Introduction
Across the globe, humans speak over seven thou- sand unique languages (Eberhard et al., 2022). Many of these languages contain a plethora of in- ternal variation due to the environmental, cultural, and socioeconomic diversity inherent to large pop- ulations (Honkola et al., 2018). These dialects are categorised into two groups: standard and non- standard (Trudgill, 2004). Standard dialects are, by definition, supported by governmental and educa- tional institutions resulting more opportunities of all kinds for their speakers. On the other hand, speakers of non non-standard and minority dialects find themselves at a disadvantage compared to their
In this paper, examine the performance of large language models (LLMs) from two crucial multilingual tasks, machine translation and auto- matic speech recognition, across a diverse set of dialects and analyze the linguistic, socioeconomic, and computational factors that may contribute to

---

the dialect gap. This study determines that the largest indicator for better performance for under- resourced dialects linguistic proximity to well- resourced dialects, regardless of the size or wealth of the dialects' speaker base. This connection is predicted to be due to the lack of dialectal data in- cluded in training large language models, leading to dialects performing better worse on the basis of incidental similarity to the dialect used in training. Unfortunately, the size of the performance gup and the amount/makeup of data required to overcome is not predictable from external information about the language since it varies across task, model, and environment. As a result, further analysis will need to be done by researchers for individual systems to examine how the dialect gap can be closed for their work through a unique combination of higher- quality, larger, and more balanced datasets.
2022; Santiago et al., 2021). As such, existing evaluations of linguistic diversity in NLP are fun- damentally incomplete.
Dialectal Models The advent of large language models has made it possible to train models that perform well on even low-resource languages (Aha- roni et al., 2019; Conneau et al., 2020). The term LLM is not strictly defined, but in this study, we use it to refer to multilingual Transformer-basse systems pretrained on large amounts of scraped internet data and finetuned for specific tasks. In these systems, under-resourced languages have their training supplemented by this unannotated, scraped data and longull transfer (Dabre al., 2020). The performance gain seen by low-resource languages when using LLMs does not extend to under-resourced variants of languages.
Some LLMs provide allocational support for di- alects by treating them as separate languages but their performance is not necessarily comparable that of the standard form. As an example. Arabic speakers often write in their native dialects when communicating casually online, a phenomenon noted by both the linguistic and NLP research com- munities (Alshutayri, 2017: Abdul-Mageed et al., 2018). Still, attempts by social media to trans- late Arabic posts are far less successful than their attempts on French and English, despite many con- sumer translation systems offering support for ma- jor regional dialects of Arabic (Harrat et al., 2019). For dialects outside of those explicitly included in systems, this problem is only exacerbated by a lack of allocational support.
2 Dialect Diversity in Research
Studies in Linguistic Diversity A significant prob- lem in the study of linguistic diversity across NLP is the lack of attention paid to language variation. In the past few years, increased awareness has been drawn within the NLP community to the dispari- ties present in modern research. In particular, re- searchers have begun to notice the relative lack of papers that address languages spoken outside of Europe and East Asia, even in subfields like multi- lingual NLP (Blasi et al., 2022; Joshi et al., 2020; Ruder et al., 2022; Søgaard, 2022).
While these works offer insight into the disad- vantages faced by speakers of under-resourced lan- guages, they still are discussed under the assump- tion that if languages were appropriately attended to, all their speakers would gain equal access to NLP tools. Similarly, they present their compar- isons as if all speakers of well-resoureed languages, especially English, have superior access to tools, Unfortunately, this is not necessarily the case. Two- thirds of English's 's one-and-a-hall billion speak- ers are second-language (I.2) speakers (Eberhard et al., 2022). Many L2 speakers struggle with NLP systems due to their accent or their use of code-switched and mixed language. Even many first-language (LI) speakers, such speakers of African American Scottish English, do not see their native dialect supported by speech, dialogue, or translation systems and are forced mask their natural speech patterns, which is harmful to their mental health and sense of identity (Johnson et al.,
The Data Problem The same marginalised lan- guages that face lower performance at the hands of LLMs also face larger data problem across dialects. Most of the task-annotated data avail- able online for low-resource languages comes from religious texts. government documents, or multina- tional newspapers (Agié and Vulié, 2019; Skadinš et al., 2014; Chen et al., 2020). These sources often use a formal manner and avoid dialectal markers, especially when their target population is mostly diglossic and has already had to learn a more stan- dard dialect for survival in the modern world (AI- shutayri, 2017; Abdul-Magees et al., 2018). As result, the LLMs trained on this data are not built to function on minority dialects and have unclear performance capabilities. Before this problem can be solved, questions must be answered about the amount, quality, and type of data needed to over-

---

Task
Models
Languages
Metrics
Google NMT (Johnson et al., 2019)
Arabic (16). Finnish (2).
Machine Translation (MT)
Meta NIL.B (Costa-jussà el al., 2022)
Mandarin (2), German (3),
BLEC, SentenceBERT Similarity
Helsinkii OpusMIT
Malay (3), Purtuguese (2),
(Tiedemann and Thotingal, 2020)
Swahili (2)
Google USM (Zhang et al., 2023)
Arabic (8), Spanish (8), Bengali (3), Georgian (2), Tamil (5), Telugu (4), Togulog
Automatic Speech Recognition (ASR)
Google STT (Chiu al., 2018)
WER, CER
OpenAl Whisper (Radford al., 2022)
Meta XLS-R (Conneau et al,, 2020)
Table Tasks addressed in this study along with models, languages (with the number of dialects), and metries.
come the data problem. The survey done in this paper across languages provides insight into how well dialects perform 'as is' and identifies that lin- guistic and socioeconomic knowledge should be leveraged inform future decisions data collec- tion and usage.
parameter variant'.
Another popular MT model is Google's 's Neural Machine Translation (NMT), which available for use through Google Cloud API (Johnson et al., 2019). NMT is widespread consumer tool, to the point that Google has had to parse out bitext generated using when scraping internet data for training (Ni et al., 2022).
3 Tasks
We also evaluate the University of Helsinki's OpusMT, model based on MarianMT and trained on Wikimedia monolingual and multilingual text (Tiedemann and Thottingal, 2020). This model is an interesting comparison to NLLB and NMT because it is not an LLM and represents a different approach covering more languages at the cost of performance across the board. This model was constructed in an academic setting with a more transparent set of training data and significantly fewer parameters. All evaluations are conducted with English as either the target or source language due data constraints.
The two tasks evaluated in this paper are machine translation (MT) and automatic speech recognition (ASR). These tasks are some of the few with suffi- cient data for evaluation of dialects and both have a focus on increasing access to people, tools, and information by removing linguistic barriers (Jin al., 2021). They also safe tasks use starting point because they do not deal with per- sonal information or abusive language. The list of models, languages, and metrics used in the evalu- ation of cach task can be found in Table 1. More information about the datasets and languages used can be found in Appendix A. In total, there are six model versions evaluated for each task and 30 di- alects across 7 languages compared for MT and 33 dialects across languages compared for automatic speech recognition. Other than Tamil and Telugu for ASR, each language taken from a different language family in order to extract information that is independent of specific linguistic features.
Evaluation metrics are biased measure of out- put quality and fluency but are required empir- ically showcase the dialect gap. To reduce some of the negatives associated with each metric, we report two types of metrics that measure differ- ent aspects of the output. The first metric is a BLEU score, which is classic n n-gram evaluation technique for translation (Papineni et al., 2002). Secondly, representation- to determine the semantic similarity between two sentences since MT task with multiple possi- ble solutions. Most semantic similarity metrics are based on transformer embedding models, so we use a multilingual variant of SentenceBERT (Reimers and Gurevych, 2019). Full results for both metrics are reported in Appendix C.
metric is used
3.1 Machine Translation
Machine translation is already used in domains such as medicine, law, and information as method of increasing access to systems (Büttner et al., 2022; Vicira et al., 2021). A leader in the field of multilingual MT is Meta's No Language Left Behind (NLLB), model that claims "safe, high- quality results" for two hundred languages (Costa- jussà et al., 2022). The specific version of the model evaluated in this study is the distilled 600M
"huggingface wwwagiinnfacc..oofaaeooooo co/facebook/ n11b-20 20--2000ttteee--6000
Thuggingface hor.sentence-rraasffrreeee transformers/ paraphrase-me mul tilliiiiiii tiw.inngll--iiiM--ITT

---

3.2 Automatic Speech Recognition
4 Linguistic Analysis of Dialects
Automatic speech recognition (ASR) is a task that is important in bringing access to those who are unable or disinclined to communicate through text (Ahn and Lee, 2016; Doumbouya et al., 2021). As of late, representation learning and LLMs for end- to-end ASR have been becoming more common. Many models are trained on unsupervised audio data and then finetuned for specific tasks. This is the case for Meta's XLS-R, a model that is trained on thousands of hours of speech data across lan- guages (Conneau et al., 2020). We evaluate both on a multilingual variant² and monolingual variant of the 300M parameter base model finetuned single language time using the Common Voice dataset (Ardila et al., 2020).
There are many ways to identify and quantify the similarity between two variants of a language. Many have been explored in NLP for cross-lingual transfer using features from syntax, lexicon, and morphology (Philippy et al., 2023; Eronen et al., 2023; Lin et al., 2019; Ponti al., 2019). There have also been studies on dialects in computational linguistics, examining whether dialects are consis- tent across corpora and registers (Dunn, 2021). A similar method is used in this paper to examine lexical similarity, using Spearman's 's Rank Correla- tion Coefficient. This has been used previously to calculate corpus similarity and homogeneity (Kil- garriff and Rose, 1998). In Appendix Figure 3a, the similarity between each dialect and the best- performing variant of that language is shown, as well as the lexical similarities between scripted and conversational samples from each dialect of the Babel dataset.
Another model examined is OpenAT's Whisper, which is trained on a combination of existing ASR datasets and automatically generated transcripts scraped from the internet (Radford et al., 2022). The version of the model tested here is the medium variantS. Like XLS-R. the Common Voice dataset used to finetune this model by language for an additional evaluation (Ardila al., 2020).
Additionally, we examine the phonetic similarity of selected ASR datasets, specifically for Arabic and Spanish. Here, random samples were manually annotated for vowel positioning through formant analysis and plotted in the Appendix; see Figure 3b. Then, the average Euclidean distance across vow- els between each dialect and the standard form was taken to serve as a measure of phonetic similar- ity. More details on the exact methodology can be found in Appendix B.
Lastly, Google has released two ASR models: the monolingual Speech-To-Tie Toxxet (STT) and their newer multilingual Universal Speech Model (USM) (Chiu et al., 2018; Zhang et al., 2023). These mod- els were both evaluated through Google Cloud API because neither has been released for open-source use. STT in particular functions as good com- parison the LLMs evaluated here because it is an older, monolingual model. Overall, six models will be compared three "monolingual" models (including those finetuned for a specific language) and three multilingual models.
5 Dialect- Wise Performance Gaps
Examining the performance across dialects in Fig- ure I, some trends appear immediately. As men- tioned in Appendix A, the dialects evaluated were largely dictated by data availability. As a result, Arabic and Spanish are heavily represented while other lower resource (dialect-wise) languages see coverage of only two three dialects. This is some- thing that may be reflected as well in the training data for pre-trained models, resulting Arabic and Spanish both having relatively more even dialectal performance than the other surveyed languages. For MT, there are steeper performance gaps when translating into the dialect. This makes sense if input robustness is taken into account; in other words, models may be able to handle some level of dialect variation in their input but cannot know to output the non-dominant dialect. Additionally, models that perform better on the standard dialect show steeper drop offs in performance, something
While there has been discussion on whether word error rate (WER) and character error rate (CER) adequately predict performance, no better system has been used by the community large (Favre et al., 2013). There have been other op- tions, but these are primarily for downstream end- to-end tasks, such as speech translation, natural language understanding, and information retrieval (Kim et al., 2021; Roy, 2021). For this work, we will stick with the community standard and use WER, with CER scores reported in Appendix C.
wawzvee2-xler-mmlliinnnee co/voidful/
"https://huggingffa cot.aebbboo///////////ooc/ccc///
waw2vec2-x1s-r-300m
hhttps: http://eaa...e.ceeeeeeeeeeeeaeee aee/apenai/iiiiser--ddiii

---

Machine Translation
dialect EN
50
Finnish
Malay
Portggeee
Swahili
Machine Translation EN to dialect
190
680
Arabic
Fünnish
Mandarin
German
Malay
Portggeee
Swahil
Automatic Speech Recognition Multilingual
1.00
Word
200
de
Spanish
Aralic
Bengali
Geoglan
Tamil
Automatic Speech Recognition Finetuned
Word Enor
100
200
26
FRS PIT Spanish
0003
16
car
cav
14t Tamil
Arshic
Bengali
Grorgan
Google
OpeaMT
Google
OpanAl Whisper
Figure The performance of various dialects across Machine Translation and Automatic Speech Recognition. Within each language the same dataset used. Because Bengali, Georgian, and Tamil are heavily diglossic, the standard written form (spt) and the best performing spoken form (cnv) are compared rather than regional dialects.
very clearly exemplified across the Finnish dialects. This demonstrates the interesting point that higher performing models which have access to more parameters and data during training have greater inequalities their coverage.
in Figure 1. In order examine this phenomenon in an equivalent environment, we compare performance across MT using BLEU % which the percentage of best-performing di- alect's BLEU score achieved by the minority alect. Likewise, for ASR, the relative percentage loss of performance for each dialect compared to the standard dialect is used. Note that this means that in Figure 2, positive MT correlation and negative ASR correlation both mean there is positive correlation between the metric and per- formance.
The same trends are not apparent for ASR, where the worst performing model (OpenAl's 's Whisper) has the highest amount of variance across dialects. Interestingly, all three multilingual models seem to prefer the spoken dialect (env), likely due to the fact that they are mostly trained on unsupervised inter- net data from websites like Youtube. On the other hand, the finetuned models prefer the written di- alect (spt), which is understandable since most are finetuned using Common Voice, heavily scripted data source.
In choosing metrics for comparison, we aimed to cover the range of economic, social, and lin- guistic factors that capture the idea of proximity to power. As proxies for wealth, examine gross domestic product (GDP) for cumulative economic power and GDP per capita for individual economic power (The World Bank, 2023). Socially, we are interested in both population size and how well-
6 Correlations with Proximity to Power
Even within the same task and model, different dialects have different performance disparities,

---

Machine
Translation di + EN
Speech Recognition Multilingual
Metric
EN + di
Finetuned
Gross Domestic Product
0.11
0.16
-0.25
-0.13
Gross Domestic Product per Capita
0.16
0.31
0,44*
0.16
Population Size
0.04
-0.13
-0.30
-0.00
Human Development Index
-0.06
-0.03
0.23
0.09
Lexical Similarity
0.48*
0.69*
-0,70*
-0.57*
Phonetic Similarity
-0.49*
-0.63*
Table 2: Pearson correlation coefficients for each language metric. For MT, correlation calculated against percentage drop in BLEU performance while for ASR, correlation calculated against percentage increase in WER. As such, the correlations are reversed for WER. Correlations with < 0.05 are marked.
served the population is in education, healthcare. and standard of living, estimated via the Human De- velopment Index (HDI) (US Census Bureau, 2017:
Gross Domestic Product Per Capita GDP per capita is an important metric a proxy for esti- mating the wealth of individuals and we would expect those with access to wealth to be well-served even their population smaller. Surprisingly, it seems to have no impact at all on MT across models, so wealthier minority popula- tions are not better served than poorer ones despite having access to increased resources. In ASR, the result is even more unexpected with wealth corre- lating negatively with performance.
population
United Nations Development Programme, 2021). Lastly, for linguistic factors, we utilize the lexical similarity and phonctic similarity extracted from evaluation data and normalized to scale from -1 (lowest similarity) to 1 (highest similarity). Un- fortunately, economic and social metrics only reported the national level, there is no data for minority dialect groups within countries. As a result, certain dialects (e.g. Kven Finnish, Vernacular Malay) are not included.
Population Size Population size intuitively would correlate with better performance, but previous studies language diversity in NLP have shown that even languages with extremely high popula- tions are not well-served they are impacted by other factors like geographic distance from research institutions and low wealth (Blasi et al., 2022). Here, population size has little impact on MT per- formance, to the point that certain models show a negative correlation between the two. On the other hand, in ASR there is strong positive correlation across all models except for the finetuned version of W Whisper. This an unexpected result because Whisper originally showcases a matching positive correlation and is finetuned on the same Common Voice datasets as XLS-R R but demonstrates a com- plete trend reversal. This difference between MT and ASR may be result of the type of data used for training each and the sources it came from, but further analysis is needed confirm this.
In the past, population factors have been shown to loosely correlate with factors such as perfor- mance and appearance in NLP research (Blasi et al., 2022). Here, in Figure 2, we see that these correla- tions do not necessarily hold for dialects. In fact, these results are contradictory common expecta- tions and narratives, which assume that wealthier, larger, and more educated populations are better served across the board.
Gross Domestic Product GDP represents the over- all wealth of speaker population and their eco- nomic power in the world. As such, we would expect groups with high cumulative wealth to be well-served by technology. While GDP has a small impact, varies heavily by model and can't be used as a consistent predictor of performance. Cer- tain models show a relatively consistent positive correlation, such as OpusMT and USM/STT, but others show no correlation at all. Ohers showcase a correlation only in one set of models, such as NLLB which is uncorrelated when translating into English but positively correlated when translating into the dialect. On average, worse-performing models and environments show stronger correla- tion, with translation into the dialect being much more correlated than translation into English.
Human Development Index HDT is measure of how well population is served in other access- based metrics, such as education, healthcare, and standard of living. It would logically follow that a high HDI would then correlate with better perfor- mance, but this does not hold for MT. Instead, MT performance shows no correlation at with HDI.

---

NLLB dialect to EN
OpenAI Whisper Scripted
OpenAl Whisper Conversational
EN to dialect
60
90
40
WER
Ream
BLEU
BLEU
20
20
Training
0
2500
5000
Training
2500
5000
Vemaculer Malay
-200
-200
Helsinki Op
Helsinki OpusMT
Standardd Genman
Meta XSL-R
Meta XSL Conversational
dialect EN
EN dialect
Stexdard Malay
Scripted
60
60
100
100
40
40
8. WWIR
WER
EU
BL EU
20
20
-100
-100
16
Training Dataset Size
Training Dataset Size
-200
10
-00
bn
ka
(a) BLEU scores achieved over different finetuning dataset sizes for
(b) Change WER after finetuning on scripted Bengali. Georgian, and Tamil data.
MT on Low German and Vernacular Malay.
Figure 2: Impact of training dataset modifications on performance.
Surprisingly, HDI correlates negatively with ASR performance, so better-educated and healthy minor- ity dialect speakers have a harder time accessing ASR systems despite being otherwise well-served economically and socially.
However, these results clearly already show that phonetic similarity plays a large part in determining the performance dialects.
The results surrounding similarity suggest that the most useful method of addressing the dialect gap may lie in focusing on how reduce the lin- guistic distance between the language used at eval- uation versus training. In other words, this can be compared to domain shift problem rather than multilingual problem. A way to begin is by in- creasing the dialectal diversity of the training data to cover a larger variety of language patterns.
Lexical Similarity Lexical similarity, on the other hand, is very correlated with performance for both MT and ASR. Since dialectal data is not used for training regardless of population features, perfor- mance is likely mostly based on linguistic proxim- ity to the standard form. This result is also more robust than the other correlations mentioned here because every dialect of every language evaluated was included since the similarity score was not de- pendent on external data availability. Again, we also see in MT that the worse-performe porse-performing direc- tionality (EN- >> dialect) has a stronger correlation. This is expected in context since these models do provide allocational support these dialects, they are translating into the standard dialect re- gardless of user intent but they may be robust to some amount of the lexical variation the input.
7 The Impact of Datasets
7.1 Machine Translation & Dataset Size
For many languages, lower performance in MT is seen in parallel with smaller dialect gap. As an example, the Mandarin dialects perform compara- bly on NLLB and OpusMT but the disparity be- comes statistically significant under NMT, a model where Mandarin as a whole performs better. This trend suggests that the benefits of larger models and more training data not equally felt by all dialects due to disparities in the training pipeline more training data does not solve the dialect gap. makes worse. The question can then be raised: would training on more specifically dialectal data be sufficient to overcome these disparities?
Phonetic Similarity The importance of linguistic similarity extends to phonetic similarity for ASR, which is strongly positively correlated with per- formance. Again we see that finetuning on the smaller, scripted Common Voice datasets makes the corrllation stronger for XLS-R and Whisper, which suggests that models overfit to the dialects present in training data. It is important remember that phonetics is broad area of study in linguistics that encompasses many measures acoustic simi- larity, so other forms of analysis may capture even higher impact forms of variation between dialects.
To answer this question, two languages with enough dialectal data were chosen to finetune NLLB and OpusMT. Each model was trained with thirty different dataset sizes, on three different data subsets per size and three seeds per data subset to ensure that the results were statistically significant.

---

In Figure 2a, the training curves for each model and translation direction can be seen.
sational samples. This ties back into the impact of lexical variation discussed in Section 6 since both scripted and conversational samples were col- lected by speakers of the same dialect with similar accents. The low lexical similarity between these dialects amplifies the fact that ensuring the training dataset accurately and fully represents the lexical variations across language and its dialects is an important step in creating systems that perform well across dialects, domains, and registers.
As more data is added, the languages begin to perform better, but not all at the same rate. For example, Vernacular Malay sees relatively little improvement as data is added train NLLB, but OpusMT's initial training curve is steep. Therefore the same amount of data causes two different out- comes depending on the model architecture and the data it was previously trained on. In some cases, the improvements are marginal, while in others, even small amount of data is enough to com- pletely overcome the dialect gap. Likewise, the same inconsistencies can be seen between the two directionalities of the same model. Despite both versions of OpusMT being trained on the same Low German data, translation into English sees benefits while translation into Low German remains poor. This makes it clear that the amount of data that dissipates the dialect gap in one situation may not be enough for another model language.
Implications of the Dialect Gap
The existence of a dialect gap means that not all speakers are inherently well-served by a tool just because their language is supported. Past analy- ses examined inequities from the perspectives of multilingualism and therefore likely overestimated the number of speakers benefiting from the current system. As the field moves forward, it is important to step back and remember that languages are not static or monolithic.
Additionally, as we saw, the dialect gap not identical in severity or structure across every sys- tem. This implies that researchers cannot take a one-size- ffits-lll--ll approach towards solving the di- alect gap. This issue needs addressing in different ways depending the task and the existing state of the gap. A large component of dialect gaps is based on datasets both dataset size and datasct makeup. As the NLP community moves towards furthering research for medium- and low-resource languages, discussions must be had on both collecting suffi- cient amounts of dialectal data and capturing the natural variations of every language by ensuring that data is collected from diverse populations. Ap- preciating and accounting for variation not only makes our systems more robust but supports groups that face marginalization other ways.
7.2 Speech Recognition & Dataset Makeup
Besides finetuning on more dialectal data, another possible method of addressing the dialect gap is modifying the makeup of training or finetuning data. Across the board, the data used to finetune LLMs for speech recognition is heavily influential performance. This difference can be seen when comparing the performance of these ASR systems on conversational and scripted samples from the TARPA Babel dataset (Bengali, Georgian, Tagalog, Tamil, & Telugu). The models evaluated here are largely trained on unsupervised speech data from the internet, which mostly comes from unscripted conversational recordings. As result, the multi- lingual models perform slightly better on conversa- tional speech. To test the impact of data makeup, XLS-R and Whisper were finetuned for three lan- guages (Bengali, Georgian, & Tamil) on Common Voice, an entirely scripted dataset (Ardila et al., 2020). These languages are all spoken by a diglos- sic population that uses both a regional dialect and a more linguistically conservative standard writ- ten form. As result, the lexical distance between conversational and scripted samples is farther than might otherwise be expected. In Figure 2b, finetun- ing on scripted data almost exclusively benefits per- formance for scripted samples over conversational samples. In some cases, such as with Whisper, this comes at the detriment of performance on conver-
Conclusion
This work examined an important subspace in NLP by evaluating the (regional) dialect gap present in tasks with the highest likelihood of impacting speakers directly. Still, there are countless LLMs which have been rapidly gaining popularity in the past few years with the release of open-ended dia- logue and image models. Most tasks outside of MT and ASR do have the data necessary to analyze the impact of language variation but as more data is collected and annotated, this may change. As a direct continuation of the line of inquiry started

---

in this work, multi-dialectal analyses of the dialect gap across wider variety of tasks should be next.
guage was not evaluated. Spanish speech, for ex- ample, has been recorded in the OpenSLR, CALL- HOME, and Fisher datasets but CALLHOME was chosen alone to be used. On the other hand, a multitude of English accent and dialect datasets are available for speech, but because each was col- lected independently, they again could not be di- rectly compared and were therefore omitted. Lastly, some languages supported by models (Telugu and Tagalog) were not present in the Common Voice finetuning dataset used for the ASR experiments and were therefore omitted from a large part of the discussion surrounding dataset makeup.
For MT and ASR, the next steps are two-fold. Firstly, the datasets used for evaluation and fine- tuning in this work were primarily determined by availability, but using a broader and higher-quality set of samples may lead to the rise of other interest- ing trends. Additionally, to address the dialect gap identified here, there is clear path forward that in- volves collecting more dialcctal data and ensuring it is representative of the languages and dialects it aims to serve. This should be done in conjunc- tion with speakers of the language, linguists, and members of the NLP community to maximise util- ity while minimising the burden or harm the speaker population. Lastly, this analysis is hardly complete. As new LLMs come out, it is on the de- velopers of these tools and the researchers behind them to continuously produce evaluations around language diversity to ensure that the benefits these LLMs bring do not come at the cost of access for minority dialect speakers.
Computational Restraints Many of the models evaluated are large industry models, with hundreds of millions if not billions of parameters. Natu- rally, as an academic institution, we were limited in the computational power made available to train these models; certain models were so large that even with batch size of one they are incapable of running on the machines we have available. If we had greater computational power available, we would have run our evaluations on the largest ver- sion of each model to provide a picture of the most state-of-the-arr performance for each task and fine- tune these larger models longer. On the other hand, many minority dialect speakers do not have the eco- nomic resources to train or finetune super-massive models, the evaluation of more accessible mod- els is an appropriate reflection of what is available to these speakers. In the future, with access to greater resources, the evaluation of more systems and larger models, along with the evaluation on other user-facing tasks (Ruder et al., 2023), again through the optics of regional dialects, ould be a valuable extension of this work.
Limitations
Dataset Size & Quality Factors Dataset size is a very significant factor when evaluating models and drawing language-wide conclusions. While the languages seen in this work had enough data for evaluation, very few provided enough data for finetuning L.LMs and none provided enough to train a model from scratch. As a result, models were largely evaluated out of the box, which serves to identify performance gaps as they may appear in non-academic use cases but does not fully address solutions to this problem.
Likewise, dataset quality makes a massive im- pact on the result of training and evaluation. Be- cause the number of available datasets was already quite low, crowd-sourced datasets such as Tatoeba were used without additional filtering, which may result in increased noise due to improper annota- tions. For some datasets, such as the IARPA Babel speech dataset, there was filtering done but spon- taneous speech data in general is often paired with background noise and distortion, causing a further drop in performance.
Ethics Statement
Dialectal NLP rescarch is a burgeoning field with- out many precedents set for ethical research, but direction can be taken from the field of multilin- gual NL.P for how to work with the languages of minoritised groups ethically. In this paper, the issue of ethics was largely sidestepped through the use of anonymised, public, and voluntarily collected datasets and the evaluation of tasks with low likelihood of causing harm. Additionally, despite the importance of idiolects and moving beyond re- gional dialects, we purposefully did not work with dialects connected to identity features that may put people at risk, such as sexuality, gender, and reli-
Some languages have several datasets available, but because these datasets were not all collected with the same methodology (and therefore simi- lar errors and distortions), they were not directly comparable so only one dataset was used or the lan-

---

gion. Even as this paper supports the collection of larger and more representative datasets, these arguments do not apply in cases where would be against the wishes or best interests of the groups involved.
Appen Pty Ltd. 2006a. Gulf Arabic Conversational Telephone Speech LDC2006S43. Linguistic Data Consortium.
Appen Pty Ltd. 2006b. Gulf Arabic Conversational Telephone Speech, Transcripts LDC2006T15. Lin- guistic Data Consortium.
Acknowledgements
Appen Pty Ltd. 2006c. Iraqi Arabic Conversational
Telephone Speech LDC2006545. Linguistic Data Consortium.
This work has been in part supported by the UK Re- search and Innovation (UKRI) Frontier Research Grant EP/Y031 1500/ (the UK government's fund- ing guarantee for ERC Advanced Grants) awarded to Anna Korhonen at the University of Cambridge. The work of Ivan Vulić has been supported by a personal Royal Society University Research Fel- lowship 'Inclusive and Sustainable Language Tech- nology for a Truly Muirilingual World' (no 221137; 2022-). The work of Anjali Kantharuban has been supported by a Gates Cambridge Scholarship.
Appen Pty Ltd. 2006d. Iraqi Arabic Conversational Telephone Speech, Transcripts LDC2006T16. Lin- guistic Data Consortium.
Appen Pty Ltd. 2007a. Levantine Arabic Conversational Telephone Speech LDC2007501. Linguistic Data Consortium.
Appen Pty Ltd. 2007b. Levantine Arabic Conversa- tional Telephone Speech, Transcripts LDC2007TOL. Linguistic Data Consortium.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuhen Morais, Lindsay Saunders, Francis Tyers, and Gre- gor Weber. 2020. Common voice: A massively- multilingual speech corpus. In Proceedings of the 12th Language Resources and Evaluation Confer- ence, pages 4218 42222.
References
Muhammad Abdul-Mageed, Hassan Alhuzali, and Mo- hamed Elaraby. 2018. You tweet what you speak: A city-Ievel dataset of arabic dialects. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Damián Blasi, Antonios Anastasopoulos, and Gra- ham Neubig, 2022. Systematic inequalities in lan- guage technology performance across the world's languages. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5486-5505.
Željko Agié and Ivan Vulié. 2019. Jw300: A wide- coverage parallel corpus for low resouree languages. Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 3204- 3210.
Pierrette Bouillon, Johanna Gerlach, Jonathan Mutal, Nikos Tsourakis, and Hervé Spechbach. 2021. A speech-enabled fixed- exed-phraaee translator for healthcare accessibility. In Proceedings of the Ist Workshop on NLP for Positive Impact, pages 135-142.
Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. Volume (Long and Short Papers). pages 3884-3884.
Benjamin Büttner, Murat Firat, and Emilio Raiteri. 2022. Patents and knowledge diffusion: The impact of ma- chine translation. Research Policy, 11418)4
Tae Youn Ahn and Sangmin-Michelle Lee. 2016. User experience of mobile speaking application with automatic speech recognition for ell learning. British Journal of Educational Technology, 47(4):778-78)
Alexandra Canavan and George Zipperlen. 1996. CALLHOME Spanish Speech LDC96S35. Linguis- tic Data Consortium.
Holly K Carlson and Monica A McHenry. 2006. Effect of accent and dialect on employability. Journal of employment counseling, 43(2):70-88 83.
Alham Aji, Genta Indra Winata, Fajri Koto, Samuel Cahyawijaya, Ade Romadhony, Rahmad Mahendra, Kemal Kurniawan, David Moeljadi, Radityo Eko Pra- sojo, Timothy Baldwin, et al. 2022. One country, 700+ languages: NIp challenges for underrepresented languages and dialects indonesia. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Ahlem Chelghoum. 2017. Social network sites and ara- bic diglossia between threatening modern standard arabic and strengthening colloquial arabic. Inter- national Journal of Language and Linguistics, 5(3- 1):36.
pages 7249..
Peng-Jen Jen Chen, Ann Lee, Changhan Wang, Naman Goyal, Angela Fan, Mary Williamson, and Jiatao Gu. 2020. Facebook ai's wmn20 news translation task submission. In Proceedings of the Fifth Conference on Machine Translation, pages l 113-125.
AOO Alshutayri. 2017. Exploring twitter as source of an arabic dialect corpus. International Journal of Computational Linguistics (IJCL), 8(2):37-44.

---

Chung-Chemn Cheng Chiu, Tara Sainath, Yonghui Wu, Rohit Prabhavalkar, Patrick Nguyen, Zhifeng Chen, An- juli Kannan, Ron J Weiss, Kanishka Rao, Ekaterina Gomina, et al. 2018. State-of-the-art speech recog- nition with sequence-to-sequence models. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4774-4778. IEEE.
Terhi Honkola, Kalle Ruokolainen, Kaj JJ Syrjänen. Unni-Päivä Leino, Ip Tammi, Niklas Wahlberg, and Outi Vesakoski. 2018. Evolution within a language: environmental differences contribute to divergence of dialect groups. BMC evolutionary biology. 18:1-1 1-5.
Ben Hutchinson, Vinodkumar Prabhakaran, Emily Den- ton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020. Social biases in nip models as barriers for persons with disabilities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5491-5501.
Alexis Conneau, Alexei Baevski, Ronan Collobert, Ab- delrahman Mohamed, and Michael Auli. 2020. Un- supervised lingual representation learning for speech recognition. Preprint.
Zhijing Jin, Geeticka Chauhan, Brian Tse, Mrinmaya Sachan, and Rada Mihalcea. 2021. How good is nip? sober look at nip tasks through the lens of social impact. In Findings the Association for Computa- tional Linguistics: ACL-IJCNLP 2021, pages 3099- 3113.
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human- centered machine translation. Preprint.
Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan. 2020. A survey of multilingual neural machine trans- lation. ACM Computing Surveys (CSUR), 53(5) 38.
Darin G Johnson, Bradley D Mattan, Nelson Flores, Nina Lauharatamahirun, and Emily B Falk. 2022. Social-cognitive and affective antecedents of code switching and the consequences of linguistic racism for black people and people of color. Affective sci- ence, 3(1):5 13.
Oliver J Bear Don't t Walk, Harry Reyes Nieva, Sandra Soo-Jin Lee, Noémie Elhadad, et al. 2022. scoping review of ethics considerations in clinical natural language processing. JAMIA open, 5(2):00ac039.
Melvin Johnson, Orhan Firat, and Roce Aharoni. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3874-3884, Minneapolis, Minnesota.
Moussa Doumbouya, Lisa Einstein, and Chris Piech. 2021. Using radio archives for low-resource speech recognition: towards an intelligent virtual assistant for illiterate users. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 35, pages 14757-14765.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the nlp world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282-6293.
Jonathan Dunn. 2021. Representations of language varieties are reliable given corpus similarity measures. In Proceedings of the Eighth Workshop on NLP for Similar Languages. Varieties and Dialects, pages 28- 38.
Adam Kilgarriff and Tony Rose. 1998. Measures for corpus similarity and homogeneity. In Proceedings of the Third Conference on Empirical Methods for Natural Language Processing, pages 46-52.
David Eberhard, Gary F. Simons, and Charles D. Fennig. 2022. Ethnologue, 26 edition. SIL International.
Juuso Eronen, Michal Ptaszynski, and Fumito Masui. 2023. Zero-shot cross-lingual transfer language se- lection using linguistic similarity. Information Pro- cessing & Management, 6033).0
Suyoun Kim, Abhinay Arora, Due Le, Ching-Feng Yeh, Christian Fuegen, Ozlem Kalinli, and Michael L Seltzer. 2021. Semantic distance: A new metric for asr performance analysis towards spoken lan- guage understanding. In INTERSPEECH, pages 1977-1981.
Benoit Favre, Kyla Cheung, Siavash Kazemian, Adam Lee, Yang Liu, Cosmin Munteanu, Ani Nenkova, Dennis Ochei, Gerald Penn, Stephen Tratz, et al. 2013. Automatic human utility evaluation of asr systems: Does wer really predict performance? In INTERSPEECH, pages 3463-3467.
Allison Koenecke, Andrew Nam. Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R Rickford, Dan Jurafsky, and Sharad Goel. 2020. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, I 117(14)7664--669
Joshua Fishman. 2007. What do you lose when you lose your language? Stabilizing Indigenous Languages. page 71.
Salima Harrat, Karima Meftouh, and Kamel Smaili. 2019. Machine translation for arabic dialects (sur- vey). Information Processing & Management, 56(2):262-23.
John Lalor, Yi Yang, Kendall Smith, Nicole Fors- gren, and Ahmed Abbasi. 2022. Benchmarking inter- sectional biases in nlp. In Proceedings of the 2022 Conference of the North American Chapter of the

---

Association for Computational Linguistics: Human Language Technologies. pages 3598-3661 3609.
Nils Reimers and Iryna Gurevych. 2019, Sentence-berts Sentence embeddings using siamese bert-networke In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCN(P), pages 3982-3992.
Sam Levin. 2017. Lgbt groups denounce 'dangerous' ai that uses your face to guess sexuality. The Guardian.
Yu-Hsiang Lin, Chian-Yu Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junx- ian He, Zhisong Zhang, Xuezhe Ma, et al. 2019. Choosing transfer languages for cross-lingual learn- ing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3125-3135.
Parker Riley, Timothy Dozat, Jan A Botha, Xavier Gar- cia, Dan Garrette, Jason Riesa, Orhan Firat, and Noah Constant. 2022. Frmt: A benchmark for few-shot region-aware machine translation. Preprint.
Anna Rogers, Timothy Baldwin, and Kobi Leins. 2021. "just what do you think you's re doing, dave?'a a check- list for responsible data use in nip. In Findings of the Association for Computational Linguistics: EMNLP 2021. pages 4821-4833,
Ilaria Mariani, Maryam Karimi, Grazia Concilio, Giuseppe Rizzo, and Alberto Benincasa. 2022. Im- proving public services accessibility through natural language processing: Challenges, opportunities and obstacles. In Intelligent Systems and Applications: Proceedings of the 2022 Intelligent Systems Confer- ence (IntelliSys) Volume 3, pages 272-289. Springer.
Somnath Roy. 2021. Semantic wer: A unified metric for the evaluation of asr transcript for end usability. Preprint.
Sebastian Ruder, Jonathan H. Clark, Alexander Gutkin, Mihir Kale, Min Ma, Massimo Nicosia, Shruti Rijh- wani, Parker Riley, Jean Michel A. Sarr, Xinyi Wang. John Wieting. Nitish Gupta, Anna Katanova, Christo Kirov, Dana L. Dickinson, Brian Roark, Bidisha Samanta, Connie Tao, David Ifcoluwa Adelani. Vera Axelrod, Isaac Caswell, Colin Cherry, Dan Garrette, R. Reeve Ingle, Melvin Johnson, Dmitry Panteleev, and Partha Talukdar. 2023. XTREME-UP: A user- centric scarce-data benchmark for under-represented languages. CoRR, abs/2305. 119388 1938.
Paul Mozur. 2019. One month, 500,000 face scans: How china is using a.i. to profile minority. The New York Times.
Jingwei Ni. Zhijing Jin, Markus Freitag, Mrinmaya Sachan, and Bernhard Scholkopf. 2022. Original or translated? causal analysis of the impact of trans- lationese on machine translation performance. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 5303-5320, Seattle, United States.
Sebastian Ruder, Ivan Vulić. and Anders Søgaard. 2022. Square one bias in NI.P: Towards a multi- dimensional exploration of the research manifold. In Findings of the Association for Computational Linguistics: ACI. 2022, pages 2340-2355 Dublin, Ireland, Association for Computational Linguisties.
Lauren Olson, Emitza Guzman Ortega, and Florian Kun- neman. 2023, Along the margins: Marginalised com- munities' ethical concerns about social platforms. In 45th International Conference on Software Engineer- ing. IEBE/ACM.
Hassan Sajjad, Ahmed Abdelali, Nadir Durrani, and Fahim Dalvi. 2020. Arabench: Benchmarking di- alectal arabic-english machine translation. In Pro- ceedings of the 28th International Conference on Computational Linguistics, pages 5094-5107.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 31 1-318.
Rawn Santiago, Nehopia Nwokoma. and Jasmin Crentsil. 2021. Investigating the implications of code- switching and assimilating at work for african ameri- can professionals. The Journal of Business Diversity. 21(4):72-8) 81.
Fred Philippy, Siwen Guo, and Shohreh Haddadan. 2023. Identifying the correlation between language distance and cross- cross-lingual lingual transfer in a multilingual representation space. SIGTYP 2023, page 22.
Boaz Shmueli, Jan Fell, Soumya Ray, and Lun-Wei Ku. 2021. Beyond fair pay: Ethical implications of nlp crowdsoureing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3758-3769.
Edoardo Maria Ponti, Helen O' Horan, Yevgeni Berzak, Ivan Vulié, Roi Reichart, Thierry Poibeau, Ekate- rina Shutova, and Anna Korhonen. 2019. Modeling language variation and universals: A survey on ty- pological linguistics for natural language processing. Computational Linguistics, 45(3)-559-601.
Raivis Skadinš, Jörg Tiedemann, Roberts Rozis, and Daiga Deksne. 2014. Billions of parallel words for free: Building and using the eu bookshop corpus. In Proceedings of the Ninth International Conference on Language Resources and Evaluation 14). pages 1850-1855.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak su- pervision. Preprint.

---

Anders Søgaard. 2022. Should we ban english nip for a year? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5254-5260.
Lucas Nunes Vieira, Minako O' O'Hagan, and Carol O'Sullivan. 2021. Understanding the societal im- pacts of machine translation: critical review of the literature on medical and legal use cases. Informa- lion, Communication & Society, 24(1 1):1515-1532
Tony Sun, Andrew Gaut, Shirlyn Tang. Yuxin Huang, Mai EiSherief. Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-We Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review, In Proceedings of the 57th Annual Meeting of the Association for Compu- national Linguistics, pages 1630-1640.
Yilun Wang and Michal Kosinski. 2018. Deep neural networks are more accurate than humans at detecting sexual orientation from facial images. Journal of personality and social psychology, 11(2))246-2577
Alicia Beckford Wassink, Cady Gansen, and Isabel Bartholomew. 2022. Uneven success: automatic speech recognition and ethnicity-related dialects. Speech Communication, 140:50-70.
Hanna Suominen, Tuija Lehtikunnas, Barbro Back, He- lena Karsten, Tapio Salakoski, and Sanna Salanterä. 2007. Applying language technology to nursing doc- uments: pros and cons with focus ethics. Inter- national Journal of Medical Informatics, 76:S293- S301.
Barbara Wheatley. 1996. CALLHOME Spanish Tran- scripts LDC96T17. Linguistic Data Consortium.
Shiyue Zhang, Ben Frey, and Mohit Bansal. 2022. How can nlp help revitalize endangered languages? case study and roadmap for the cherokee language. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 1529 1541.
The World Bank. 2023. Economy & Growth. The World Bank Group.
Hale M Thompson, Brihat Sharma, Sameer Bhalla, Randy Boley, Connor McCluskey, Dmitriy Dligach, Matthew M Churpek, Niranjan S Karnik, and Ma- jid Afshar. 2021. Bias and fairness assessment of a natural language processing opioid misuse classifier: detection and mitigation of electronic health record data disadvantages across racial subgroups. Journal of the American Medical Informatics Association: JAMIA, 28(11):233))
Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, et al. 2023. Google usm: Scaling automatic speech recognition beyond 100 languages. Preprint.
A Languages
Jörg Tiedemann. 2020. The tatoeba translation challenge- realistic data sets for low resource and multilingual mt. In Proceedings of the Fifth Confer- ence on Machine Translation, pages 117--1882.
In total, there are seven languages evaluated in this study per task and over thirty dialects. The de- tails of these datasets are discussed in Section A, but for easy reference, table listing the datasets, the languages included, and any dialects is pro- vided in Table 3. Additionally, any abbreviations used in figures throughout the paper are included, with language abbreviations pulled from ISO 639-1 and dialect abbreviations either based on the ISO 31 66 country codes or created based on the regional names provided in the dataset.
Jörg Tiedemann and Santhosh Thottingal. 2020. Opus- mt mt-building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation. European Association for Machine Translation.
Suzanne Tolmeijer. Naim Zierau, Andreas Janson, Jalil Sebastian Wahdatchagh. Jan Marco Marco Leimeister, and Abraham Bernstein. 2021. Female by default?-exploring the effect of voice assistant gender and pitch on trait and trust attribution. In Extended abstracts of the 2021 CHI conference on human factors in computing systems, pages 1-7.
AraBench Arabench a dataset collected by the Qatar Research Computing Institute to encourage research into machine translation for Arabic di- alects (Sajjad et al., 2020). The dataset includes parallel text data grouped by region, nation, and city from religious, media, and speech sources. Across the board for Arabic, Modern Standard Ara- bic (MSA) outperforms the vast majority of dialec- tal forms, even though social media's 's rise has led to dialectal forms of Arabic being used in written com- munication more often. In fact, in personal chat, dialectal forms of Arabic are represented more than MSA (Chelghoum, 2017).
Peter Trudgill. 1979. Standard and non-standard di- alects of english in the united kingdom: Problems and policies. International journal of the socialogy of language, 21:9-24.
Peter Trudgill. 2004. Dialects. Psychology Press.
United Nations Development Programme. 2021. Hu- man development index.
US Census Bureau. 2017. International data base.

---

Dataset
Language
ISO 639-1
Dialects (Abrv)
Modern Standard (msa), Iraqi (iq). Omani (om), Qatari (qu). Saudi (sa), Yemeni
AraBench
Arabic
ar
(ye), Jordanian (jo). Palestinian (ps), Lebanese (lb). Syrian (sy), Algerian (dz).
Moroccan (ma). Libyan (ly), Tunisian (tn). Egyptian (eg). Sudanese (sd)
Region-Aware MT
Mandarin
zh
Mainland (en), Taiwanese (Iw)
Portuguese
pt
Brazilian (br), European (pt)
Finnish
Standard (f)), Kven (kv)
German
de
Standard (de), Low (lo), Swiss (SW)
Tatoeba
Malay
ms
Standard (ms), Vernacular (vn), Moluccan (mo)
Swahili
sw
Coustal (cn). Congolese (eg)
Conversational Telephone Speech
Iraqi (iq). Omani (om). Saudi (sa). Emirati (ac), Jordanian (jo). Lebanese (lb).
Arabic
Palestinian (ps). Syrian (sy)
Argentinian (ar). Chilean (el), Colombian (co), European (es). Mexican
CALLHOME
Spanish cs
(mx). Peruvian (pe). Puerto Rican (pr). Venezuelan (ve)
Bengali
bn
Kamrupa (kp), Radha (rd). Varendra (vd)
Georgian
ka
Easter (en). Western (Wn)
Babel
Tagalkog
Central (en), Northern (nn). Southern (sn)
Tamil
ta
Central (en), Madurai (md). Northem (nn), Souther (sn), Western (wn)
Telugu
Central (cn). Northern (nn), Southern (sn). Western
Table 3: The datasets, languages, dialects and abbreviations used throughout this paper.
The dataset used for Mandarin and Portuguese MT is Google's Region- Aware Machine Translation Dataset, a small bench- marking dataset for few-shot translation for these two high-resource languages (Riley et al., 2022). The dataset consists of Wikipedia articles that exist in both English and the target dialects.
Google Region-Aware MT
antine (Jordamian, Lebanese, Palestinian & Syrian) dialects of Arabic.
The dataset used to evaluate Span- ish ASR is the CALLHOME telephone speech cor- pus, which encompasses several primarily Latin American Spanish datasets (Canavan and Zip- perlen, 1996; Wheatley, 1996). In this work, we specifically focus on eight dialects: Argentinian. Chilean, Columbian, European, Mexican, Peruvian, Puerto Rican, and Venezuelan Spanish. Spanish is an interesting case of two "standard" forms, Mexi- can and European Spanish.
CALLHOME
Tatoeba The Tatoeba datasets are set of trans- lation datasets crowdsourced by the Tatoeba or- ganisation (Tiedemann, 2020). Languages have varying amounts of data, ranging from over a mil- lion sentences in English to fewer than ten for lan- guages such as Sindhi. The majority of parallel text available in these datasets is for low-resource languages paired with English, so most translation systems trained on this data use English either the source target language. While Tatoeba is a project focused on language diversity, there efforts within it include some major regional dialects. Dialects of languages such as German (Standard, Low, Swiss), Finnish (Standard & Kven), Malay (Standard, Venacular, & Moluccan), and Swahili (Coastal & Congolese) are available and included in our analysis.
TARPA Babel The TARPA Babel dataset? was large set of speech recognition datasets collected for medium- and low-resource languages to make ASR effective on a broader set of languages. While there are many languages included, most are not currently supported by large ASR systems, so five have been selected for these experiments: Bengali (Kamrupa, Radha, & Varendra), Georgian (Eastern & Western), Tagalog (Central, Northern, & South- ern), Tamil (Central, Madurai. Northern, Southern. & Western), and Telugu (Central, Eastern, North- ern, & Southern). Each of these languages is writ- ten in a different script, increasing the difficulty. It is important to note that the Babel project's 's goal was not to represent all dialects of these languages, so certain significant dialects (e.g. Sri Lankan and Singaporean Tamil for the Tamil dataset) are omit- ted due the focus single major region. The Babel dataset divides each language into two five regional dialects, with half the samples being
Conversational Telephone Speech The dataset used to evaluate Arabic for ASR is the Conver- sational Telephone Speech (CTS) dataset, a spon- taneous spoken language dataset with transcrip- tions available through the Linguistic Data Consor- tium (Appen Pty Ltd, 2006c,d, 2007a,b, 2006a,b). This of datasets encompasses the Gulf (Emirati, Omani Saudi), Mesopotamian (Traqi), and Lev-
hhttss//tatocaa.or//

---

scripted speech and the other half being sponta- neous utterances. While we are primarily interested in spontaneous speech for this work, all of these languages have a distinct written form which is con- sidered a separate dialect. As result, the scripted audios are kept but separated from the spontaneous samples to examine how much lexical variation (between the scripted and spontancous samples of each dialect) impacts performance in comparison phonetic variation (across dialects).
is extracted in order reduce discrepancy due variations in distortion or recording environments as much as possible, Ten samples per dialect, from unique speakers (when possible) are then annotated with the first (f1) and second (f2) formant, which provides information on the height and backness of the vowel. While roundedness can be gleaned from further formants, this adds another level of complexity it was not considered in this study. The height of the vowel was then set as fl and the backness was set as f2 1--1, f1. The average distance between the dialects' mean vowel positions and those of the pest-performing dialect was taken to represent phonetic similarity.
B Linguistic Analysis
Lexical Similarity The lexical similarity be- tween dialects is calculated by splitting the full dataset for each dialect and randomly sampling it into halves one hundred times. The top hundred most frequently occurring words are then ranked by frequency to generate a vocabulary list. For each dialect pair of interest, the position of each word in the ranking is compared using the following for- mula:
C Full Evaluation Results
Machine Translation Here, we include the full results achieved across dialects for transparency. The BLEU score results can be seen in Table 4 and the semantic similarity results can be seen in Table 5. The label "di -> EN" refers to translation into English from the dialect and "EN di" refers to translation from English into the dialect. The closest language tag was used when possible, such as regional dialect tags for Arabic in NLLB.
6 d
=
nun22 1)
Here, de is the difference in ranking between each word in the list and n is the number of words being compared (100 in this case). If a word ap- pears in one list but not the other, the ranking given is n + 1. The same process is utilized for calcu- lating homogeneity, except both halves are taken from the same corpus. Homogeneity is calculated ensure that there isn't a large degree of internal variation that is influencing the degree of lexical cross-similarity between dialects. While this anal- ysis can be impacted by domain mismatches, by focusing on only the top one hundred words we deal primarily with very common words that are less likely to be specialized vocabulary.
Automatic Speech Recognition Likewise, full results are included for both the raw and finetumed versions of our ASR models. Table 6, the word сттor rates (WER) are provided and in Table 7 the character error rates (CER) are provided. When there is a "+ CV" included in the model title, that refers further finetuning the Common Voice dataset. Additionally, the "spt" and "cnv" titles refer to the scripted and conversational splits of the languages the IARPA Babel dataset. Spanish and Arabic are only evaluated with conversational samples, so there is no data in the scripted column.
Phonetic Similarity The phonetic similarity be- tween dialects is computed in a time-intensive method for this paper, so the results this computa- tion should be considered preliminary until further annotations can be collected. For each language of interest, the primary vowels of that language are determined based on the orthographic choices made in that language. This amounted to three vowels in Arabic (/al, /u/, & fi/) and five vowels in Spanish (/al, /el. fi/, /o/, & /u/). For simplicity. other highly sonorant phonemes, such as /j/, are not included, although some may consider them vowel-like. From this, the set of samples from each dialect where all the language's vowels are present
D Correlations
The plot comparing various metrics with dialect performance can be found in Figure 4. The vertical axis for the MT plots is BLEU %, or the percentage of the standard dialect's BLEU score achieved by the minority dialect. Likewise, the vertical axis for the ASR plots is WER %, the percentage worse WER achieved by the minority dialect, since lower WER is better. This means that higher score on the two left columns better, while a lower score on the two right columns is better. As you can see heuristically, the only obvious corre-

---

Machine Translation
Automatic Speech Recognition Dialects vs. Best- Performing Dialect
Arabic Dialectal Vowel Phonemes f2
Dialects vs. Best- Performing Dialect
1259
Thene
Scripted vs. Conversational Forms
Spanish Dialecial Vowel Phonemes f2- ft
Tal GootatBepp
Swalll Panggoddaa MarkinCanC 10
1503
500
Teag
Augast din Soucia
Mosica Soutak
Parto Rose Socad
Lexical Cinat-imillriyy
Lexical Cass-Simillrity
(a) Lexical Analysis
(b) Phonetic Analysis
Figure 3: Linguistic similarity measures comparing dialects. Left: Lexical similarity, taken either between each dialect and the best-perford best-performing dialect from that family or between spontancous and comversational samples of the same dialect. Right: Vowel positions for the most common vowels in two languages across dialects.
Machine Translation Correlations
Automatic Speech Recognition Correlations
dialect EN
300 EN
dialect
Muklingual
Momolingual/ineete Finetuned
331
150
BIRUU
0.5
CDP
3x31
CDP
1:11
CDP
1412
CDP
1e12
300
300
150
150
BLSU
5333
1000
1500
Sece
10330
15003
I0033
23330
35000
45003
100
30640
20333
33500
450ce
GDP Per Capéi
CDP Per Capio
GDP Pai Cupta
CDP Pa Capia
300
300
150
150
1LEU
WEB
0.22
U/4
3.8
0.5
022
U.4
0.51
1.3
1.0
100
U.a Prpulotton
2.0
Prpribatin
Prgullatio Stnn
Popeibtton
300
100
BLZU
23
BLEU
WER
B-4
0.5
02
100
0.4 Homon Developener Index
Fima Developmert Index
Devetogment
Develgment
300
300
BL.30%
WER
So
Sintileit
Sinrlkekky
Sinlleky
Sintlariy
Google
Geogle USMUSIT
WEB
Helanka Upsalil
XLS-A
Arerage
Anorage
Phreetic Stmilaity
Phearte Similetty
Figure 4: Correlation between various metries and the relative performance of minority dialects, normalised to be relative to the best performing dialect. Note: The MT graphs are with respect to BLEU scores, where higher value means better performance, but the ASR graphs are with respect to WER, where the inverse is true. Linguistic similarity is positively correlated with better performance in both systems.

---

lates are both linguistic measures, which is con- firmed in our calculations.
in the future if ethnicity socioeconomic class was equally able to be designated, what patterns would emerge in consumer systems? These tools may result in maintaining stereotypes regarding subservience and the "place" of certain groups society.
E Concerns on Ethics in Dialect Research
Dialectal research is relatively new field in NLP and as such, it is important to consider the ethical impacts of conducting it. In the past, researchers have examined the ethics of many aspects of NLP as they relate to marginalized and vulnerable pop- ulations, including raising concerns about sources of data (Olson et al., 2023; Rogers et al., 2021; Shmueli et al., 2021), evaluations of representa- tional bias (Hutchinson et al., 2020; Lalor et al., 2022: Sun et al., 2019), and bringing awareness to allocational bias (Aji et al., 2022; Blasi et al., 2022). Additionally, there has been significant work done in the medical domain, where the population be- ing served is particularly vulnerable to both coer- cion and harm from publicized data (Don't Walk et al. 2022; Thompson et al., 2021; Suominen et al., 2007). We can take inspiration from these prior works when examining the ethical pittalls of dialec- tal research, since minority dialect communities face many of the same hurdles other minoritized groups, especially when their language reflects as- pects of their identity that face persecution such as gender, race, or class.
Another aspect to consider is the result of datasets and tools that parse certain identity mark- falling into the wrong hands. A key exam- ple of this in another domain, facial recognition processing, is the Stanford "Gaydar" which was purported to outperform humans in predicting sex- ual orientation from facial structure (Wang and Kosinski, 2018). This system was denounced by multiple prominent LGBTQ organizations for both its methodology and the risk it places on closeted members of the community since it could be used to out members living in unfriendly places (Levin, 2017). Likewise, in China, similar facial recogni- tion systems have already been used to track the movement of Uighurs, a Muslim minority group that has been facing prosecution by the government (Mozur, 2019). As such, the existence of tools to identify minority are unfortunately inextricably linked with the ability to repress that minority. Cur- rently, there are no highly accurate systems that can classify most personal identity features from text alone, but as we work on the identification and usage of dialect data, this may soon change. When conducting this research, it is important to have a deep understanding of the communities impacted by these tools beyond their academic contribution and weigh the consequences of releasing such tools for public against the value of open and collab- orative science.
Currently, the issues that impact minority dialect speakers are largely around representational and allocational bias. Speakers of certain dialects are viewed as less intelligent or more difficult to un- derstand, leading to systems classifying them as such when it comes to usages such as call screen- ing or academic evaluations (Koenecke et al., 2020; Wassink et al., 2022). Important tools in use, such automated emergency service or medical phone systems, may not function well on their dialects, reducing access to necessary care. Additionally, these biases are perpetuated externally towards users as well; female voices are often used when building personal assistants, which continues the societal trend of female systems being "bossed around" or acting in a subservient manner. This is especially the case since there has been no strong correlation in increased trust or comfort when per- sonal assistants are of specific gender, but the vast majority of consumer systems female voices anyway (Tolmeijer al., 2021). As dialect re- search improves, there is unfortunately room for these problems to worsen. Currently, gender is easy to specify when generating synthetic speech, but

---

Language
Dialect
di
Google
EN
NMT
EN
di
di
Meta NLLB
EN
EN
di
di
Helsinki Opus
EN
EN
di
MSA
47.64
22.07
43.71
14,64
26.22
05.72
Iraqi
33.38
03.94
37.37
02.82
03.97
Omani
46.78
19.77
45.07
13.68
20.05
09.40
Qatari
34,84
05.75
37.08
03.03
10:46
01.23
Saudi
39.98
6.79
41.63
04.32
13.67
01.54
Yemeni
35.05
03.84
37.45
02.53
11.28
01.11
Jordanian
37.86
04.75
40.46
02.76
08.00
01.29
Arabic
Palestinian
34.88
04,16
38.62
02.68
07.22
00.94
00.55
Syrian
34.31
03.14
38
02.27
08.63
00.68
Algerian
22.95
04.90
29.76
03.79
09.27
01.38
Moroccan
19.69
02.91
34.99
02.08
06.45
00.71
Libyan
33.03
04,40
38.
03.43
10.93
01.31
Turisiaa
17.29
02.08
30.70
01.35
06.23
00.83
Egyptian
36.44
04.81
39.06
03.12
09.00
01.34
Sudanese
45.62
08.92
46.40
06.20
15.61
03.67
Finnish
Standard
53.58
41.18
54.10
31.95
35.61
21.40
Kven
22.19
05.23
17:24
08.35
20.13
10.38
Mandarin
Mainland
35.82
20.79
27.22
00.90
16.27
01.29
Taiwanese
31.65
13.60
26.23
01.48
15.54
01.64
Standard
50.10
51.01
53.28
44.88
39.78
26.30
German
Law
25.53
Swiss
29.85
02.92
17.02
02.56
18.76
03.02
Standard
47.98
43.61
46.81
35.27
27.86
22.86 86
Malay
Vernacular
33.74
06.95
36.58
06.88
16.76
04.18
Moluccan
17.11
02.04
12.03
09.03
05.89
Brazilian
57.96
$7.09
55.54
$2.98
Portuguese
European
48.08
40.41
47.08
41.66
35.13
29.98
Swahiti
Coastal
50.15
42.94
41.79
32.35
02.52
00.25
Congolese
35.40
20.48
34.11
17.58
02.84
00.54
Table 4: Machine Translation Evaluations: BLEU Scores. Each model is evaluated across both directionalities with either the source or target specified as English. higher BLEU score signifies better performance.
Google di + EN
Meta di - EN
Helsinki OpusMT
I Language
Dialect
EN -?
EN di
di + EN
EN
MSA
0.851
0.890
0.826
0.872
0.590
0.834
Iragi
0.742
0.860
0.750
0.794
0.417
0.771
Onnani
Qatari
0.743
0.814
0.742
0,787
0416
0.768
Saudi
0.792
0.806
0.788
0.800
0.447
0.777
Yemeni
0.733
0.804
0.757
0.793
0.772
Jordanian
0.777
0.785
0.780
0. 0.789
0416
0.767
Arabic
Palestinian
0.746
0.782
0.771
0.776
0.394
0.758
Lebanese
0.681
0.797
0.747
0.765
0.352
0.745
Syrian
0.764
0.788
0.777
0.755
Algerian
0.661
0.780
0.704
0.777
0.378
0.752
Moroccan
0.606
0.799
0.743
0.770
0.339
0.753
Libyan
Tunisian
0.521
0.787
0.685
0.757
0.329
0.747
Egyptian
0.764
0.832
0.773
0.774
0.381
0.749
Sudanese
0.792
0.805
0.782
0.819
0.449
0.796
Finnish
Standard
0.897 0.777
0.842
0.855
0.796
0.728
0.761
Kven
0.754
0.581
0.642
0.692
0.701
Mandarin
Mainland
0.883
0.785
0.637
0.742
Taiwanese
0.849
0.848
0.752
0.762
0.635
0.739
Standard
0.894
0.857
0.886
0.832
0.797
0.764
German
Low
0.528
0.494
0.622
0.489
0.684
0.480
0.669
0.573
0.509
0.522
0.539
Standard
0
0,842
0.839
0.811
0.712
0.756
Malay
Veraauulrr
0.819
0.652
0.805
0.665
0.654
0.627
Moluccan
0.622
0.637
0.580
0.638
0.509
0.618
Portuguese
European
Brazilian
0.933
0.858
0.907
(1.004
0. 889
0.822
0.837
0.820
Swahili
Coastal Congolese
0.857
0.823
0.818 0.732
0.247
0.199
0.728
0.749
0.717
0.257
0,200
Table 5: Machine Translation Evaluations: SentenceBERT Similarity. Each model evaluated across both directionalities with either the source or target specified as English. A higher similarity score signifies better performance.

---

Google USM
OpenAl
Whisper Cnv
Language
Dialect
Spt
Cov
Spt
Spt
Cnv
Argeatinian
26.55%
33.76%
72.56%
Chilcan
25.80%
35.36%
74.95%
Colombian
24.40%
34.32%
67.72%
Spanish
Europcan
25.21%
30.75%
68.24%
Mexican
24.55%
34.79%
68.78%
Peruvian
27.52%
37.41%
71.93%
Puerto Rican
45.21%
Venczuelan
28.68%
35.69%
73.85%
Tragi
54.14%
134.09%
94.49%
Omani
57.81%
157.52%
94.77%
50.51%
120.31
93.12%
Arabic
Emirati
57%13%
154.92%
95 95.01%%
Jordanian
49.57%
149.44%
93.93%
Lebanese
58.14%
136.57%
95.88%
Palestinian
49.56%
121.88%
94.06%
Syrian
48.78%
118.59%
94.59%
Kamrupa
60.98%
51.11%
207 58%
190.83%
103.31%.
101. 16%
Bengali
Radha
58.16%
49.79%
229 59%
188 698.62%
103.06%
100. 800
Varendra
57.92%
52.79%
221 121446%
185.26%
102.83%
100.94%
Georgian
Eastern
49.12%
32.39%
117.03%
132.76%
104.13%
101.11%
Western
50.21% %
39.09%
144.40%
129.93%
104.45%
101.13%
Central
63.91%
65 65.600
128 9666
134.233%
100.500%
101 83%
Madurai
63.69%
70.59%
83 68%
131.25%
100 45%
101 62%
Tamil
Northem
65.12%
08.07%
17%
131.10%
100.01%
101.699
Southern
65.75%
65.75%
102 50%
125 225.66%
100.99%
101.72%
Western
66.10%
68.48%
150.69%
130.37%
100.23%
101.65%
Central
66.34%
65.01%
238. 818
207.66%
105.
102, 602
66.20%
64.24%
317.679
193.24%
104.78%
103.08%
Telugu
Northern
65.37%
24%
320.80%
211.71 1%.
105.57%
103, 27%
Southern
66.53%
65.80%%
254.95%
232.51 %S
104.36%
103.035:
Central
110.49%
90.21%
Tagalog
Northem
116.31%
95,71%
Southern
94.09%
105.98%
Google STT
OpenAl Whisper + CV Spr
Meta XLS-R CV
Language
Dialect
Spt
Cnv
Cnv
Spt
Argentinian
55.36%
44.65%
62.58%
Chilean
47.95%
37.03%
65.22%
Culomhian
49.71 %
34.75%
59.61%
Spanish
European
45.221%.
29.76
57.27%
Mexican
44.78%
Peruvian
$3.38
42.68%
63.51%
Puerto Rican
59.33%
51.63%
76.61%
Venezuelan
53.47%
36.76%
65.82%
Iraqi
72.47%
218.33%
95.39%
Omani
73.93%
217.44%
96. 16%
Saudi
67.39%
185.53%
94.60%
Arable
Emirati
74.39%
216.66%
95.935
Jordanian
72.91 S6
186.66%
94.94%
Lebanese
82.27%
200.33%
96.37%
Palestinian
73.78%
189.31%
95. 15%
Syrian
75.04%
195.73%
95.69%
Kamrupa
64.11%
75.87%
176.37%
73.02%
Bengali
Radha
63.09%
79 85
68.351
144.77%
68. 34%
88 84%
Varendra
65.82%
80.37%
74.17%
153.00%
71.54%
89.59%
Georgian
Western
Eastern
64.64%
66.95%
81.71%
81.44%
165.49%
143.95%
210.67%
222.75%
72.83%
68.61%
88.11%
88.56%
Central
64.97%
86.39's.
73.251%
77.71%
Madurai
63.82%
86.54%
73.0095
109.13%
76.795
98. 16%
Tamil
Northern
64.57%
57.00%
74.72%
112.06%
78.46%
98. 18%
Southern
65. 65.13%%
86.75%
79.97%
110.75%
81.36%
98. 98.25%
Western
64.13%
84.39%
77.67%
106.23%
77.07%
97.58%
Central
63.94%
84.35%
Telugu
Northern
Eastern
63.36%
62.73%
85.63%
841.91 %
Southern
64.62%
Central
58.82%
64. 64.61% S
Tagalog
Northern
58.16%
65. 65.01% %
Southern
57.04%
65.10%
Table 6: Automatic Speech Recognition Evaluations: Word Error Rate. Each language is evaluated on three multilingual models and three monolingual models, with the IARPA Babel samples separated into Scripted (spt) and Conversational (cnv) classes. lower word signifies better performance.

---

Google
OpenAl
Whisper Cov
Language
Dialect
Spt
Cnv
Spt
Spt
Cov
Argentinian
17.54%
22.16%
35.73%
Chilean
16.98%
23. 10%
37.43%
Colomhian
16.61%
22.90%
32.17%
Spanish
European
18.27%
21.75%
32.94%
Mexican
16.71%
23.40%
32.92%
Peruvian
18.52%
25.03%
35.87%
Puerto Rican
25.75%
31.33%
Venezuelan
19.80%
24.22%
37.01%
Iraqi
23. 23.97%
104.15%
53.48%
Omani
27.56%
128.51
55.32%
Saudi
21.94%
91.25%
52.54%
Arabic
Emirati
26.45%
121.32%
55.08%
Jordanian
20,57%
95.77%
52.01%
Lebanese
27.38%
99.87%
54.49%
Palestinian
20.06%
99.87%
52.53%
Syrian
19.83%
82.85%
52.02%
Kamrupa
41 1125
29.56%
231 39%
218.
93.63%%
94. 60%%
Bengali
Radha
36.26%
30.91%
268 22%
213 8918
93.34%
94.43%
Varendra
37.27%
31.28%
220.769
207.11
93.30%
94.25%
Georgian
Eastem
35.15%
13.83%
336.56%
298 0.0%%
72.64%
88.39%
Western
36.11%
18.16%
349.79%
291.62%
74.14%
86.77%
Central
42.42%
29.44%
63.03%
100.791%
$9.91 19.901%
93.049%
Madurai
40.91
38.91%
65.5%
90.71%
55.39%
92. 53%
Tamil
Northem
41.04%
34.72%
78.49%
86.91%
58.23%
47%
Southern
43.58%
31.01%
69.37%
86.06%
59.82%
92.35%
Western
43.42%
37.36%
107.68%
88.58%
56.77%
93.1
Central
39.95%
35,86%
251.07%
275 66%
97.61%
95.46%
Telugu
Eastem
41.20%
33.14%
307.03%
271.42%
96.86%
95.46%
Northemn
40.95%%
37.10%
233.57%
279.64%
96.89%
95.89%
Southem
42.03%
36.015
275.7043 70%%
282.87%
96.70%
95.77%
Central
81 86%
62.82%
Tagalog
Northern
87.30%
66.62%
Southern
66.92%
69.32%
Google STT
OpenAl Whisper + CV Spr
Meta XLS-R CV Spt
Language
Dialect
Spt
Cnv
Cnv
Argentinian
43.08%
27.71%
29.56%
Chilean
33.53%
22.74%
30.89%
Culonbia
38.25%
22.01
27.93%
Spanish
European
35.47%
18.8915.
27.31%
26.99%
Peru
40.95%
26.77%
30.30%
Puerto Rican
44.26%
32.40%
40.68%
Venezuelan
40.83%
22. 96%
31.97%
Iraqi
44.05%
191.00%5
51.91%
Omani
49.02%
189.29%
16%
Saudi
40.39%
152.65%
51.88%
Arable
Emirati
48 28%
34%
54,28%
Jordanian
45, 35.74%
30%
51%
Lebanese
55.58%
170.02%
53.70%
Palestinian
47.21 %
158.14%
51.35%
Syrian
49.19%
168.76%
51.45%
Kamrupa
41.55%
39.3215
30.26%
51.65%
Bengali
Radha
39.86%
67.09%
31 16-6
108.85%
27. 39%
49 09%
Varendra
42.39%
68.01 %
37.11%
117.18%
29.86%
49.92%
Georgian
Western
Eastern
46.48%
44.43%
52.86%
53.26%
110.34%
100.37%
176.89%
183.55%
20.15%
18.16%
35.25%
34.67%
Central
41.54%
65.28%
32.56%
70.98'S.
30.02%
Madurai
40.35%
60.
33.71%
73 86%
26, 98.
59.70%
Tamil
Northern
40.94%
66 66.49%
33.79%
75.13%
29. 69%
59.71%
Southern
41.37%
65, 65.62%
40.76%
72.77%
32.30%
59.17%
Western
40.59%
62.21%
34.51%
69.72%
26.16%
57.29%
Central
40.76%
69.31%
Telugu
Northern
Eastern
40.58%
39.09%
71.41% %
69.53%
Southern
42.03%
71.33%
Central
44.89%
44.
Tagalog
Northern
44.69%
44.85%
Southern
43.85%
44.95%
Table 7: Automatic Speech Recognition Evaluations: Character Error Rate. Each language is evaluated three multilingual models and three monolingual models, with the IARPA Babel samples separated into Scripted (spt) and Conversational (cnv) classes. lower character signifies better performance.


Reviews:
[{"review_id": "J8sWWzIyDH", "reviewer": "Reviewer_rTFa", "summary": "", "questions": "A. Could you elaborate on the unexpected correlations and negative results found in the study? How do these findings fit into the broader context of the field, and what additional analysis might be required to interpret them fully? \n\nB. Expand on Linguistic Analysis of Dialects? \n\nC.  Why was there no qualitative analysis presented?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for reading our paper. We are glad to hear that you find this work to be a detailed and relevant examination that highlights important disparities, especially for languages from under-served parts of the world. We are encouraged that you find our work to be well-written, easy to follow, and well-motivated. Please see our response below:\n\n> How do the unexpected correlations/negative results fit into the broader context?\n\nPrevious work has shown that disparities between languages in NLP technology are loosely correlated with socioeconomic factors such as GDP and population size (Blasi et al., 2021). As such, it would be logical to assume that dialects would follow a similar trend, with more widely spoken or wealthier dialects being better supported. However, this is not the trend we discovered; instead, there is *no significant correlation* between the performance of models on dialects and any socio-economic factors. The only correlations were found to be with linguistic proximity to the “standard” dialect. This demonstrates that **assumptions established in the study of low-resource languages do not necessarily translate to dialects**. Additionally, this means that linguistically informed similarity analysis will be extremely valuable in creating methods for shrinking the dialect performance gap. We will clarify this in the next version of the paper.\n\n> Include some qualitative analysis.\n\nThis is a great idea and inspired us to run an error analysis on languages discussed in Figure 3 to identify whether the presence of hand-selected “dialectal” words seems to lead to lower performance. Examples of sentences with similar English translations but different levels of accuracy across dialects will be included in the final version, with their lexical variations being compared to existing literature on dialectal lexicon.  An example:\nBoth of these sentences are officially translated as “Children need loving.”\nStandard German - Kinder brauchen Liebe.\nNLLB - Children need love.\nSwiss German - Chind bruchet Liebi.\nNLLB - Chind broke my heart.\nIn this example, “chind” is a Swiss German word for child and is not picked up by the model, which assumes it is a proper noun. Additionally, the conjugation of both of the other words is spelled differently, which leads to a completely incorrect interpretation of the sentence. Although there is evidence here that German has variation outside of lexical and phonetic differences, calculating the level of lexical divergence alone is enough to predict the degree to which this model will struggle with the dialect. \n\n> Expand on the linguistic analysis of dialects.\n\nFurther information on the process of linguistic analysis is already included in Appendix B, but this section will be clarified with more details on the similarity computation processes and more background on how these techniques were selected from past work in computational linguistics. \n\n> The figures are difficult to read.\n\nWe agree and with the additional page in the final version of the paper, we will redo the figures to be easier to read and bigger so that the labels are easier to parse. With the addition of more details concerning our linguistic analysis, there will be enough clarification to assist even those without specific background knowledge in linguistics or computer science to understand the analysis. \n\n> There is a lack of rigor and gaps in our analysis.\n\nWe confidently establish in this paper that dialects perform differently from low-resource languages and that linguistic similarity is a valuable starting point for future work. We study the phenomenon of the dialect performance gap across 2 tasks (with 2 variations each), 3 models per task, and 7 languages per task. We find consistent trends across these 84 settings, including a correlation between linguistic proximity to the standard dialect and better performance of a dialect. We also find *no significant correlation* between factors such as wealth, population size, and level of development and the corresponding dialect’s performance in the cases studied. While the analysis of such complex sociological and linguistic phenomenon could fill a much larger work, we are confident in our findings. Our findings are further supported by Section 6, where we use the idea of addressing the linguistic diversity of training data through finetuning as a starting point to address the dialect performance gap.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths:\n\n- The paper provides an extensive examination of linguistic diversity in NLP, highlighting key disparities and areas that need more attention.\n- Paper is well-written, easy to follow and well motivated. \n- The paper's exploration of dialectal differences and under-resourced languages fills a significant gap in the current literature, adding depth to the understanding of language and dialect support in NLP systems.\nThe chosen tasks (MT and ASR) are well suited for the analysis, facilitating a detailed and relevant examination of the subject matter.", "reasons_to_reject": "The unexpected correlations mentioned in the text (such as negative correlation with wealth in ASR or inconsistent findings between MT and ASR) are acknowledged but not deeply analyzed or explained. This leaves gaps in understanding and may limit the applicability of the findings.\nThe presentation of the results, including the graphs, may not be clearly laid out or properly explained, hindering interpretation, especially for readers coming from different fields.\nThere is a noticeable absence of qualitative analysis in the paper.\nThe linguistic analysis of dialects is not properly explained, which can make it difficult for readers to fully grasp this essential aspect of the research.\n\nThe lack of rigor in some experiments, gaps in analysis, unclear presentation of results, and unexplored negative findings may limit the paper's contribution to the field and its applicability to real-world scenarios.", "questions_to_authors": "A. Could you elaborate on the unexpected correlations and negative results found in the study? How do these findings fit into the broader context of the field, and what additional analysis might be required to interpret them fully? \n\nB. Expand on Linguistic Analysis of Dialects? \n\nC.  Why was there no qualitative analysis presented?", "paper_topic_and_main_contributions": "The paper focuses on the challenges related to linguistic diversity and dialectal variations in NLP, particularly in the context of Large Language Models (LLMs). It explores the lack of attention to under-resourced languages and dialects, as well as the complexities in recognizing different speech forms.\n\nMain Contributions:\n1. Highlighting Linguistic Diversity Issues: Emphasizes gaps in research, especially outside Europe and East Asia.\n2. Exploration of Dialectal Models: Discusses the capabilities and limitations of LLMs in handling under-resourced languages and dialects.\n3.Speech Recognition Analysis: Investigates how training data affects the performance of speech recognition models across dialects.\n\nIn essence, the paper offers insights and analysis on linguistic disparities in NLP and suggests directions for developing more inclusive and effective models across languages and speech forms.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "It was very well written."}, "extra_text": {}}, {"review_id": "yz6GKekMEy", "reviewer": "Reviewer_V96V", "summary": "", "questions": "a) Section 4 mentions other ways of capturing linguistic similarity via syntactic, lexical, or morphological features. What was the reason for choosing the lexical and phonetic similarity metrics over one of these feature-based approaches? ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read our paper. We are glad you found our paper interesting, well-organized, and well-argued, with an impressive range of language varieties. We have addressed your comments for improvement below:\n\n> Title should reference “correlates” rather than “causes”.\n\nWe agree, and the title will be updated to reflect that in the final version.\n\n> Are the datasets comparable due to differences in domains and collection practices?\n\nYes, in the way we use them. Within each language, only one dataset was used across dialects, so the comparisons between the performance of dialects within languages are sound. All the performance comparisons in this paper are done *within languages*, with percentage differences between dialects of the same language being used in Figure 2, so even if a specific dataset is “easier,” it would not have an impact on the overall analysis. In general, we are focusing on the relative performance between dialects of each language and comparing these *relative performances* across languages because achieving a controlled and uniform environment when dealing with pre-trained models for multiple languages is difficult. \n\n> Why were lexical and phonetic features used over the feature-based metrics discussed in Section 4?\n\nThe syntactic, morphological, and lexical feature metrics discussed in Section 4 have been used when quantifying the linguistic similarity between languages, but these features do not necessarily apply the same way or to the same extent when examining dialects. Dialects are (usually) linguistically closer together than languages, and as such broader trends such as word order, affixation, etc. will be the same across dialects. Dialects primarily arise due to lexical and phonetic shifts, with syntactic and morphological shifts being less common and usually appearing after a more extended period of divergence. To keep our analysis applicable to all the dialects examined, these two metrics were prioritized.\n\n> Is the lexical variation metric capturing orthographic variation too?\n\nYes, but that’s not a bad thing. Analyzing lexical variation impacts MT and ASR differently, so we will address them individually. In MT, orthographic variations can result from two different sources: lexical shift (where new words or slang are introduced) and phonetic shift (where spelling shifts to match phonetic drift). Both of these are common in dialects, and if MT systems are to work on dialectal text, they will need to function regardless of why the lexicon has changed. So, lexical similarity is a good metric to use even if it captures variation from both of these sources. For ASR, lexical variation usually does not reflect orthographic variation because the annotators of these datasets were instructed to transcribe into the standard form unless the word is unique to the dialect. Here, the lexical similarity metric purely captures lexicon drift so we also utilize our phonetic similarity metric to account for phonetic changes. \n\n> Figure 1a is difficult to understand.\n\nThank you for pointing out this issue. To resolve this problem, the figure will be made larger and more space-efficient in the final version. The figure itself is a violin plot, where each bar is composed of calculations of Spearman’s Rank Correlation Coefficient across 100 random splits of the data as calculated in Appendix B. A higher coefficient translates to high levels of similarity in the lexicon of the two dialects, and a lower variance translates to a lower likelihood that the similarity scores are due to poor sampling. In the final version, more information about the content of the figure will be explained in Section 4 rather than just referencing Dunn (2021), which is where the structure of the figure is pulled from.\n\n> Missing references.\n\nThank you for bringing these two papers to our attention. The first paper (Snæbjarnarson et al., 2023) will be referenced in the future work section as a way to apply what has been established by this paper to improve performance. The second paper (Ranathunga & Silva, 2022) will be referenced in our related works as evidence of unbalanced training sets and may additionally be brought up in Section 6.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The task is an important one which is relevant to current issues in NLP. \n- The findings are interesting and have important implications.\n- The study examines an impressive range of language varieties across language families and across high- and low-resource languages, and examines correlations with social as well as linguistic factors. \n- Experimental design choices (e.g. models, languages, metrics) are well argued for. \n- The paper is clearly written and well organized, with detailed appendices.", "reasons_to_reject": "- The title is misleading; this paper examines correlations, not causes.\n- The multiple datasets include data from different sources and domains (e.g. religious, media, Wikipedia), and likely have different annotation/transcription practices. This limits the comparability across all the datasets and languages. \n- The lexical similarity metric seems like it may also be capturing orthographic similarity between language varieties, which is something quite different from linguistic similarity. \n", "questions_to_authors": "a) Section 4 mentions other ways of capturing linguistic similarity via syntactic, lexical, or morphological features. What was the reason for choosing the lexical and phonetic similarity metrics over one of these feature-based approaches? ", "paper_topic_and_main_contributions": "This paper presents an evaluation of 6 LLMs from Google, Meta, and OpenAI on the tasks of machine translation (MT) and automatic speech recognition (ASR), for at least 30 language varieties across 7 languages per task. The authors find that disparities in performance correlate more with linguistic proximity (i.e. lexical and/or phonetic similarity) to well-resourced language varieties than with GDP, population, or HDI of the speaker base. They also find that training data size and construction procedure is a significant factor but cannot entirely explain performance disparities. Finally, they discuss how to address this gap in performance. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- I was not able to understand Figure 1a, or what it was meant to tell us; please consider elaborating on the explanation in Section 4 or in the caption."}, "extra_text": {"Missing_References": "- This paper shows how transfer performance can be improved by using data/models of closely-related (i.e. linguistically similar) high-resource languages, exploiting the connection between linguistic similarity and transferability of model performance: https://aclanthology.org/2023.nodalida-1.74.pdf \n- This paper looks at correlations between language data resources, inclusion in pretrained multilingual models, and various social factors of speaker bases (e.g. population, GDP): https://arxiv.org/pdf/2210.08523.pdf Although they don’t examine model performance, it seems relevant as it may offer insights into relationships between some of the dependent variables in your paper. "}}, {"review_id": "gwVzMjIe4S", "reviewer": "Reviewer_7UB3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and for your kind words. We are glad that you found our paper comprehensive and thoughtful in its approach. We hope that, as you said, this work will be used as a starting point for data collection and development at the dialectal level in future work. Your grammar and spelling corrections are helpful and will be incorporated into the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is very comprehensive in the analysis of the two chosen tasks, the analysis and explanation of trends as well as the directions for future research. Also, it addresses the important topic of linguistic variety in NLP, not just at the language level, but at the level of dialects. The authors choice of socio-economic and linguistics metrics was thoughtful and a good illustration of the fact that language is less homogeneous than some assume.", "reasons_to_reject": "I do not currently see any reasons why this paper should be rejected.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a study on the performance of LLMs in MT and ASR for diverse dialects, usually a \"major/standard\" and one or more \"minor\" dialects. The performance is correlated with various socio-economic and linguistic metrics to better understand what could be of influence on the performance on minority dialects of the tested LLMs. This leads to sometimes surprising and counterintuitive results - which is an interesting starting point for data development for dialectal NLP. A clear conclusion is that the more lexically or phonetically similar a minority dialect to the standard the better the performance of the tested models.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 008: have been conducted -> have been mainly conducted?\n\nline 024: one-size approach -> one-size-fits-all approach\n\nline 186: socioeconomic -> socio-economic\n\nline 246: BLEU score -> reference?\n\nline 578: makes out system -> makes our system?\n\nline 1107: Appen PTY Ltd, Sydney and Australia -> correct reference to not include Sydney and Australia\n\nline 1153ff: by splitting by taking -> by splitting\n\nReferences: Please check capitalisation in references (e.g.nlp, bert) and correct where necessary.\n"}}, {"review_id": "6EDAxPg4tI", "reviewer": "Reviewer_J4pE", "summary": "", "questions": "A) Line 325 mentions a manual analysis of a random sample of data. Who did the manual analysis and what was their background? How was the data sample to make sure that it was representative of the population, rather than some speakers? Are there existing linguistic resources that this analysis agrees with or is this an entirely novel contribution of the work?\n\nB) For the correlations, how well fit are your regressions on each variable? It's hard to judge the meaning of any of these without metrics (at least in the appendix) for goodness of fit.\n\nC) Many of the economic and population metrics used seem like they likely correlate with dataset size. Did you control for dataset size in this analysis? If not, how likely is it that these trends are just a function of dataset size?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read our paper and leaving such detailed feedback. We are glad you found our work to be interesting in its multidimensional approach and agree that the linguistic metrics we identify as strong correlates are valuable as starting points for improving dialect performance in future work. We address your comments and concerns below:\n\n> Does this paper significantly extend the quantification of the dialect gap beyond prior work?\n\nWhile we build on prior work, we believe that our work differs in several key ways from prior work studying dialects in NLP. This work is unique in its scope - prior work on dialects in NLP largely focuses on a single language or language family. Here, we are instead looking at broader trends in performance for two high-impact tasks: machine translation and automatic speech recognition. **By examining performance across dialects of both low- and high-resource languages, we can extract trends to identify routes for future, language-agnostic work in dialect NLP.** Thank you for the additional references, including other analyses of dialectal performance (Lakew et al., 2018; Nigmatulina et al., 2020; Zamipieri et al., 2020; Sun et al., 2022), methods of improving dialectal performance (Salloum & Habash, 2012; Demszky et al., 2021) and some other related works published at or after the EMNLP deadline (Ziems et al., 2023; Deas et al., 2023; Eisenstein et al., 2023). We will discuss additional related works in the final version of our paper.\n\n> Are these models LLMs?\n\nThere is no single definition of “large language models,” but most of these models (e.g. Whisper) satisfy proposed requirements such as being trained on massive amounts of scraped internet data, having extremely large parameter sizes, and having the ability to apply to multiple tasks without further training . We had a hard time ourselves deciding on the single ‘best-aligned’ track for the work, which is why we indeed submitted this paper to both the LLM track and the multilinguality track: it straddles the line between both, and we agree that it can be argued to belong in either track.\n\n> How was the phonetic analysis (line 325) conducted and is this novel work?\n\nThe formant analysis done in this paper is grounded in linguistic theory as discussed in various linguistic dialectology papers, such as (Ghorshi et al., 2008). This citation will be included in the final version. The analysis was conducted by one of the authors who is a linguist and was decided upon after consulting a phoneticist to validate the methodology. The samples were taken randomly across participants to ensure that no one participant’s speech patterns would significantly impact the results (i.e. when there were greater than 10 participants, each sample was from a unique participant, otherwise equal number of samples (+ or - 1) were taken from each participant). \n\n> How well fit are the correlation regressions?\n \nThank you for raising this point. The Pearson correlation coefficients for the correlations are as follows (where the score axis is inverted for WER (ASR), so negative coefficients indicate a positive correlation in performance):\n\n| | MT →en | MT →di | ASR mult | ASR mono |\n| ---- | ---- | ---- | ---- | ---- |\n| GDP | 0.11 | 0.16 | -0.25 | -0.13 |\n| GPC | 0.16 | 0.31* | 0.44* | 0.16 |\n| POP | 0.04 | -0.13 | -0.30 | -0.00 |\n| HDI | -0.06 | -0.03 | 0.23 | 0.09 |\n| LEX | 0.48* | 0.69* | -0.70* | -0.57* |\n| PHO | - | - | -0.49* | -0.63* |\n*p<0.05\n\nThis demonstrates that **only the correlation between lexical and phonological features and performance is statistically significant**. Thank you for pointing out that these should be included in the paper - the values themselves will be reported in Section 5 with the p-values being included in the Appendix to strengthen the argument. Additionally, we plan to change the title from “...Causes Across Languages” to “...Correlates Across Languages” to better reflect the content of the analysis. \n\n> Are these trends just a factor of dataset size (which might be correlated with socioeconomic factors)?\n\nNo, since we kept dataset sizes uniform across dialects of each language and pre-training was done dialect-agnostically. We are primarily interested in performance relative to the standard dialect rather than absolute performance, so the correlates are taken with regards to % of the standard score. Certain languages and dialects may have more data available at pretraining due to socioeconomic factors, but **our analysis shows that any correlation between these factors and performance is shallow at best and contradictory at worst, as you mentioned**. The only consistently statistically significant correlates are lexical and phonetic similarity as shown by the correlation coefficients calculated above. These trends might be a result of dataset makeup, as discussed in Section 6.2, but that is part of the point of this paper - pretraining datasets need to be better weighted across dialects if we don’t want performance to disproportionately be weighed in benefit of the standard forms.\n\n> Improve Figures 2 and 3.\nThank you for pointing out this important issue, we will fix this in the final version. We'll structure Figure 2 with the information provided above to be more explicitly clear and make it larger with the additional space available in the final version. For Figure 3, the projection lines mostly serve to emphasize the different trends seen across languages and directionalities; following your feedback, we will remove this in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The work looks at discrepancies across both Multilingual and Multidialectal NLP simultaneously. Looking at these dimensions simultaneously is important to understand the effects of language variation at multiple scales and multiple levels of resourcedness. By doing so, the authors get larger samples which could serve as a more effective foundation for causal analysis of what drives the dialect gap.\n\n- The similarity analyses are interesting as a predictor of transfer performance between dialects. Understanding how the similarity metrics the authors use could be used to identify training data from other dialects which might lead to positive transfer could be an interesting pathway to addressing the data size issues the authors discuss later in the work.", "reasons_to_reject": "- The paper does not significantly extend the quantification of the dialect gap beyond existing published work. Several works have quantified the effects of dialect on machine translation systems for different languages. Similarly, several works have quantified the effects of dialect on ASR for different languages. This paper does not introduce any resources that did not exist in prior work (see Appendix A, all resources existed prior to this work). While evaluation of more recent systems is informative, the evaluation does not particularly shift the understanding of dialect disparities in NLP nor offer novel pathways to resolving the gap.\n\n- The section dedicated to causal analysis of the dialectal gap requires a more formalized statistical analysis plan to justify the use of \"causes across languages\" in the title. Only the correlations between performance and different features are evaluated, with many of these features being certainly not directly causal in their relationship with performance. Even at just the level of correlations, some results seem questionable (GDP per Capita and HDI which are themselves colinear show sometimes contradictory trends). I think the work would be stronger if it looked at a smaller set of factors in a more thorough manner, rather than covering a broad range of factors shallowly.\n\n- The paper does not seem particularly well-suited to the theme. None of the models evaluated appear to be LLMs as the term is used to refer to general purpose multi-task pretrained models such as LLaMA, GPT series, FLAN, FALCON, Pythia, etc. Instead, each one is a purpose trained model for a single task (either Machine Translation or Automatic Speech Recognition). The titling suggests that the paper will quantify the dialect gap in this broad general purpose setting, but only covers Machine Translation and ASR which are not new challenges brought with LLMs. The work would be a stronger fit in the Multilinguality track (Subtrack for Dialects and Related Languages).", "questions_to_authors": "A) Line 325 mentions a manual analysis of a random sample of data. Who did the manual analysis and what was their background? How was the data sample to make sure that it was representative of the population, rather than some speakers? Are there existing linguistic resources that this analysis agrees with or is this an entirely novel contribution of the work?\n\nB) For the correlations, how well fit are your regressions on each variable? It's hard to judge the meaning of any of these without metrics (at least in the appendix) for goodness of fit.\n\nC) Many of the economic and population metrics used seem like they likely correlate with dataset size. Did you control for dataset size in this analysis? If not, how likely is it that these trends are just a function of dataset size?", "paper_topic_and_main_contributions": "This paper looks at the performance disparity for dialects inside of modern NLP technology. They quantify this disparity for large-scale industrial Machine Translation and ASR systems between several groups of dialects and related languages using existing benchmarks which cover dialects and related languages in 7 languages for MT and 7 other languages for ASR. They then study the correlations between these performance results and high level economic indicators, population size, and metrics of linguistic similarity. For all languages, variation is shown between languages. Additionally, a correlation is shown between performance for a dialect and it's lexical and phonetic similarity with high-resource dialects.", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The work would benefit from a much more significant literature review of existing NLP methods for dialects and related languages. VarDial has many relevant works, and prior methods have studied the challenges of translating to and from dialects. Contextualizing the work in prior work will help highlight areas where the analysis can be further expanded to add on to existing literature, rather than reiterate it.\n\nRelevant References to Section 2:\nSurafel Melaku Lakew, Aliia Erofeeva, and Marcello Federico. 2018. Neural Machine Translation into Language Varieties. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 156–164, Brussels, Belgium. Association for Computational Linguistics.\n\nIuliia Nigmatulina, Tannon Kew, and Tanja Samardzic. 2020. ASR for Non-standardised Languages with Dialectal Variation: the case of Swiss German. In Proceedings of the 7th Workshop on NLP for Similar Languages, Varieties and Dialects, pages 15–24, Barcelona, Spain (Online). International Committee on Computational Linguistics (ICCL).\n\nZampieri, M., Nakov, P., & Scherrer, Y. (2020). Natural language processing for similar languages, varieties, and dialects: A survey. Natural Language Engineering, 26(6), 595-612.\n\nSun, J., Sellam, T., Clark, E., Vu, T., Dozat, T., Garrette, D., ... & Gehrmann, S. (2022). Dialect-robust evaluation of generated text. arXiv preprint arXiv:2211.00922.\n\nSalloum, W., & Habash, N. (2012, December). Elissa: A dialectal to standard Arabic machine translation system. In Proceedings of COLING 2012: Demonstration Papers (pp. 385-392).\n\nDemszky, D., Sharma, D., Clark, J. H., Prabhakaran, V., & Eisenstein, J. (2021, June). Learning to Recognize Dialect Features. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 2315-2338).\n\nCaleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023. Multi-VALUE: A Framework for Cross-Dialectal English NLP. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 744–768, Toronto, Canada. Association for Computational Linguistics.\n\nDeas, N., Grieser, J., Kleiner, S., Patton, D., Turcan, E., & McKeown, K. (2023). Evaluation of African American Language Bias in Natural Language Generation. arXiv preprint arXiv:2305.14291.\n\nEisenstein, J., Prabhakaran, V., Rivera, C., Demszky, D., & Sharma, D. (2023). MD3: The Multi-Dialect Dataset of Dialogues. arXiv preprint arXiv:2305.11355.\n\n", "Typos_Grammar_Style_and_Presentation_Improvements": "Figure 3(a): It's not explained how these projections are fit to the data and to what degree we should trust them, especially given grokking phenomena common in deep learning models. Explain how they were fit and how they were tested!\n\nFigure 2: It's very difficult to actually extract the correlations from these plots - not to mention they are space inefficient. I think a table or a bar chart of correlations would be more effective ways to show this data as is common for regressions in causal inference.\n\nSomtimes (like table 1) the model is referred to as XLS-R, other times (like figure 3b) it's referred to as XSL-R. Make this consistently the correct XLS-R!\n\n"}}]

Author rebuttal:

Meta-review: This paper evaluates the performance of large language models (LLMs) on machine translation (MT) and automatic speech recognition (ASR) for different languages and dialects. The paper uses six LLMs from Google, Meta, and OpenAI, and tests them on 30 language varieties across 7 languages per task.. The paper measures the performance disparity between major/standard and minor dialects, and analyzes the factors that influence it. The paper finds that linguistic similarity to well-resourced language varieties is more important than socio-economic indicators or training data size. The paper also discusses the challenges and opportunities for improving dialectal NLP and hence has merits for promoting research for dialects.
Decision: Accept-Findings