A NLP Benchmark for Dialects, Varieties, and Closely-Related Languages (2024)

Abstract: Language technologies should be judged on their usefulness in real-world use cases. An often overlooked aspect in natural language processing (NLP) research and evaluation is language variation in the form of non-standard dialects or language varieties (hereafter, varieties). Most NLP benchmarks are limited to standard language varieties. To fill this gap, we propose DIALECTBENCH, the first-ever largescale benchmark for NLP on varieties, which aggregates an extensive set of task-varied variety datasets (10 text-level tasks covering 281 varieties). This allows for a comprehensive evaluation of NLP system performance on different language varieties. We provide substantial evidence of performance disparities between standard and non-standard language varieties, and we also identify language clusters with larger performance divergence across tasks. We believe DIALECTBENCH provides a comprehensive view of the current state of NLP for language varieties and one step towards advancing it further. 1

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14412–14454 August 11-16, 2024 ©2024 Association for Computational Linguistics
1 Introduction
Benchmarking is important for tracking the progress the field of natural language processing (NLP) has made in various tasks. In the past few years, large-scale multilingual benchmarks like XTREME (Hu et al., 2020), XTREME-R (Ruder et al., 2021), and XGLUE (Liang et al., 2020) have played a pivotal role in evaluating the multilingual capabilities of NLP models. These efforts have sought to make model evaluation more accessible to researchers and representative of a variety of languages (Song et al., 2023). However, most benchmarks have focused on the standard varieties of languages, largely neglecting non-standard dialects and language varieties (Blasi et al., 2022).
*Equal contribution. 1More information can be found at the following:
Code/data: https://github.com/ffaisal93/DialectBench Website: https://fahimfaisal.info/DialectBench.io
We refer to non-standard dialects and language varieties simply as varieties, and sometimes include low-resource related languages, writing system variants, and other kinds of variation. Varieties contain subtle but notable variations in vocabulary, pronunciation, orthography and grammar, reflecting regional, social, and cultural differences (Chambers and Trudgill, 1998). The non-standard nature of these language varieties oftentimes contributes to the scarcity of substantial datasets that accurately capture these variations (Hedderich et al., 2021). As a result, they have often been absent from widely adopted benchmarks, even from admirable efforts like XTREME-up (Ruder et al., 2023), GLUECoS (Khanuja et al., 2020) and CreoleVal (Lent et al., 2023), which focus on underresourced, code-switched, and creole languages, respectively. It is currently challenging to accurately test the robustness of multilingual models on a large suite of varieties without establishing an NLP evaluation framework covering multiple language clusters (each of which contains standard languages alongside its closely related varieties).
To this end, we create DIALECTBENCH, a largescale benchmark covering 40 language clusters with 281 varieties, spanning 10 NLP tasks. We observe that the performance disparity between different varieties of the same language cluster becomes
14412
more pronounced when we shift from zero-shot evaluation to fine-tuning on variety data, because of uneven data availability across varieties. Certain language clusters exhibit varying performance across downstream tasks within the same category, due to low-resource limitations. Additionally, we improve the dialectal task coverage for natural language inference by constructing a translate-test evaluation dataset. Putting these all together, DIALECTBENCH serves as a comprehensive suite that attains a multifaceted purpose: identifying broader limitations in dialectal NLP, while reflecting on potential areas for improvement.
2 DIALECTBENCH
DIALECTBENCH is a benchmark created to unify dialectal datasets across different tasks to foster research on language varieties and non-standard dialects. Below we describe the design choices we undertook to achieve this goal. This includes our language variety and task selection procedures, data collation methods, and evaluation principles.
Variety Selection We first looked through papers published in the ACL Anthology2 from the last 10 years to find usable language resources, as well as commonly used online data repositories (Littauer, n.d.). We selected languages that have well-established, high-resourced varieties. Varieties may vary by location, ethnicity, or other factors. We also found instances where varieties are classified by writing system or even by genre (e.g., Twitter). When varying by location, varieties may be classified by different datasets at different levels of granularity, sometimes country, region, or city. In some cases, we found resources with two or more varieties within one dataset (e.g., the UD_Portuguese-Bosque depen-
2https://aclanthology.org
dency treebank (Rademaker et al., 2017) includes examples from both European and Brazilian Portuguese variants. To incorporate all these cases under one paradigm, we formulate a cluster-variety mapping procedure.
Cluster-Variety Mapping We construct several language clusters comprising of both highresourced varieties and their low-resourced counterparts. We use the Glottolog language database (Nordhoff, 2012) to define clusters and assign varieties as outlined in Figure 1. This design choice enables us to keep varieties that are closely related in terms of either mutual intelligibility, phylogenetic similarity or geographic proximity within the same cluster. Hence, all cluster varieties always root back to the closest common linguistic ancestor and the whole cluster maps to an established phylogenetic subtree. For example, Fiji Hindi and Hindi, with Hindustani3 as their closest common ancestor, are placed in the Hindustani cluster.
We primarily use the Glottocode language identification scheme (Hammarström and Forkel, 2022), ensuring a standardized naming scheme across all varieties. For instance, AAE variety from TwitterAAE (Blodgett et al., 2018) dependency parsing dataset is renamed as African American Vernacular English with a corresponding Glottocode afri1276. In cases where Glottocodes are unavailable, like for spoken English from South India, we substitute with the immediate ancestor Glottocode (indi1255) and further categorize the varieties using the following metadata identifiers:
1. Area (a): the region where the variety is spoken or where its dataset was collected.
2. Language register (r): frozen, formal, consultative, casual, and intimate.
3. Language mode (m): written, spoken, and signed language.
4. Orthography (o): In DIALECTBENCH this is only specific to Sinitic varieties. This could be either traditional or simplified.
5. Identifier (i): Dataset-specific metadata, could be domain (eg. twitter).
We encapsulate all this information, into a naming convention, and use the template: {Glottocode name}-(a:{},r:{},m:{},o:{},i:{}). 4
3https://glottolog.org/resource/languoid/id/ hind1270
4For example, mandarin chinese (a:mainland, o:simplified refers to Mandarin Chinese (mand1415) spoken in Mainland China and written in simplified characters.
Cluster Representative Each cluster will often have a high-resourced variety usually with the largest speaker population. We choose this highresourced variety as the cluster representative. This selection might vary across downstream tasks depending on the data availability. We primarily utilize this representative variety to evaluate the performance gap across cluster varieties, and also rely on it for transfer-learning in resource-scarce settings. Sometimes, the members of a cluster are considered closely related languages, and sometimes dialects; to avoid making this distinction, we refer to all the members of a cluster simply as varieties of the cluster representative.
Task and Dataset Selection In selecting tasks, we maintain a balanced approach, promoting task diversity while also including tasks that require diverse levels of textual understanding. In the end, our complete list of tasks are as follows:
1. Dependency parsing (DEP parsing) 2. Parts-of-speech tagging (POS tagging) 3. Named entity recognition (NER) 4. Dialect identification (DId) 5. Sentiment analysis (SA) 6. Topic classification (TC) 7. Natural language inference (NLI) 8. Multiple-choice machine reading comprehen-
sion (MRC) 9. Extractive question answering (EQA)
10. Machine translation (MT) In Table 1, we present the task and dataset details. We mostly keep the datasets in their originally published form (except for varieties renaming). For NLI, we use the existing English test set of XNLI (Conneau et al., 2018) and construct a multilingual dialect-focused translated evaluation dataset. We refer to this as translate-test NLI.
Evaluation Principles On the ground level, we evaluate existing NLP systems on text-based tasks
using standard evaluation metrics (e.g., UAS for parsing, F1 for classification tasks, BLEU for translation). At a global level, we believe a sustainable NLP system should be user-focused while providing substantial (i) linguistic utility and (ii) demographic utility (Song et al., 2023; Blasi et al., 2022). Blasi et al. (2022) defined the utility of a task and language, as the corresponding performance normalized by the best possible performance (usually human-level performance). Demographic utility considers the demand for a language technology within a specific language, where the demand is proportional to the number of speakers of that language. Linguistic utility, on the other hand, asserts that “all languages are created equal” regardless of the number of speakers, and hence all languages in the world should receive identical weights.
Overall, we want to capture the performance gap between language clusters (e.g., Anglic vs. Italian Romance) as well as within language clusters (e.g., Norwegian Bokmål vs. Nynorsk). To attain this, we define the performance gap metrics in §3.3 . We also vary the experimental settings in §3.2, including zero-shot and few-shot cross-lingual transfer, as well as fine-tuning with similar high-resource languages. This is essential given that we lack clean annotated data in many varieties.
3 Experiments
Here, we report the entire process involved in creating and evaluating baselines for all of the tasks and varieties in DIALECTBENCH. Additionally, we define a dialectal gap metric to analyze performance disparities within and across clusters.
3.1 Models We evaluate using two multilingual models: mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) for all tasks except MT. For MT, we do zero-shot evaluation with NLLB (NLLB
Team et al., 2022), using both the 600M and 1.3B variants. In addition, we use Mistral 7B (Jiang et al., 2023) to evaluate the current capability of LLMs on multilingual and dialectal understanding tasks. Our main goal is collating dialectal data across different languages and tasks under a single platform, hence we do not optimize for the best model performance. Rather, we focus on understanding and reporting the current state of performance on all DIALECTBENCH varieties.
3.2 Training and Evaluation Training and evaluation procedures are largely determined by the availability of training or evaluation data for each task.
For any cluster C, let C̄ be the highest-resourced variety (which is usually the cluster representative) of C. In addition, for any variety v ∈ C, we write v̄ for the highest-resourced variety, that is, v̄ = C̄. For any varieties t and v, let St(v) be the raw evaluation score of a system fine-tuned on t and tested on v (higher is better).
We use five general approaches for task-specific model training:
1. In-variety fine-tuning (Sv(v)): In cases where there is available training data for a variety v, we fine-tune the base model on v. This primarily applies to tasks such as POS tagging and dependency parsing. 2. In-cluster fine-tuning (Sv̄(v)): In-variety fine-tuning is quite resource-intensive when we have a large number of varieties within a cluster C. In such cases, we fine-tune the base model on C̄. Then we evaluate this model on each variety v ∈ C. This is the most common setting in our experiments, as it allows us to evaluate the dialectal performance gap without increasing the computation cost. For the DId task, we typically use a dataset of sentences annotated with variety labels to finetune one dialect-identification model for each language cluster. 3. Combined fine-tuning (SL(v)): Unlike in the previous two methods, where each training set contains data from a single language cluster, we fine-tune our baseline question-answering (EQA and MRC) models using the SD-QA (Faisal et al., 2021) and Belebele (Bandarkar et al., 2023) datasets respectively, both of which contain training data in multiple standard varieties only and test data in other varieties. The SD-QA
training data (L in the notation) contains questions in 9 standard varieties (L = {eng, ara, ben, fin, ind, swa, kor, rus, tel}), while Belebele assembles data from 6 distinct multiple-choice QA datasets in standard English (L = {eng}). 4. Zero-shot evaluation (Seng(v)): For certain varieties, obtaining training data even for incluster fine-tuning can be a challenge. Fortunately, English training data is always available for the datasets we study, so we use English to fill the gaps when we lack in-variety, in-cluster, or combined training data. At the same time, we aim to assess the feasibility of using this zero-shot cross-lingual transfer in reducing any existing performance gap across varieties. So we perform zero-shot crosslingual transfer from English to each variety for 6 tasks in total. We only leave out those tasks such as dialect identification that explicitly require in-cluster training data. 5. In-context learning (Sicl(v)): When evaluating large language models, we do not finetune them but instead rely on prompting and in-context learning. For this, we provide instructions and 5 examples in English as exemplars, followed by a prompt for predicting the test examples. Employing Mistral 7B (Jiang et al., 2023), we assess the present effectiveness of a close-to-state-of-the-art LLM on language varieties. The task-specific example prompts are reported in Appendix I.
Table 2 summarizes the task-specific training procedures that we employ based on data availability. Note that, for MT, we perform zero-shot evaluation specifically in the translation direction, (standard variety to English) tested on (dialectal variety to English). But evaluation is a challenge because human-created reference translations into or out of non-standard varieties are usually very limited.
Therefore, we adopt an evaluation protocol from previous work (Alam et al., 2023) that uses pseudoreferences. Given x, a sentence in a variety and x̄, the translation of x into the standard variety, let y be the output of the MT system on input x and ȳ be the output on input x̄. Then we measure the quality of y (using, e.g., BLEU) compared against ȳ as a pseudo-reference.
3.3 Quantifying the Dialectal Gap To quantify the performance disparity across various resource-specific settings, language clusters and varieties, we introduce a dialect performance gap metric Gt(u, v): the relative decrease in performance of a system fine-tuned on variety t, tested on variety v compared to a baseline variety u:
Gt(u, v) = St(u)− St(v)
St(u)
with a special case for in-variety fine-tuning:
Gin-variety(u, v) = Su(u)− Sv(v)
Su(u) .
For the baseline score, we use either the score on the standard variety for each cluster (u = v̄), or, in the zero-shot setting, the score on the language used for fine-tuning, namely English (u = t = eng). Rather than computing an absolute gap, we opt for a relative gap (i.e., dividing by the baseline score). We also indicate whether the training setting is zero-shot (t = eng) or fine-tuned on invariety, in-cluster, or assembled data. Putting all these together, we compute the following three variations of dialectal inequality.
1. Geng(eng, v): We calculate this metric to get a comprehensive measurement of global disparity across all varieties in a resource-limited environment (zero-shot transfer from English). 2. Geng(v̄, v): Using this variation, we keep the setting fixed as zero-shot and calculate the gap between the representative variety and any other variety. 3. Gt(v̄, v): The two aforementioned metrics shed light on the extent of the variety performance gap in a resource-limited setting. To gain a more comprehensive perspective, we additionally compute another metric, this time utilizing the availability of resources. The computation approach remains as straightforward as before. We just use fine-tuning on a variety t instead of zero-shot transfer
from Standard English: t = in-variety for in-variety fine-tuning, t = v̄ for in-cluster fine-tuning, or t is some set of varieties for combined fine-tuning.
For all three G metrics, we compute them at the variety level and then average them at the cluster level:
Gt(u,C) = 1 |C| ∑
v∈C Gt(u, v)
Gin-variety(u,C) = 1 |C| ∑
v∈C Gin-variety(u, v).
4 Results
We, first of all, discuss results by highlighting the highest possible score per variety, aka the maximum obtainable evaluation scores regardless of evaluation method or training data. Next, we extend our discussion further by reporting the existing dialectal disparity across clusters and varieties.
4.1 Maximum Obtainable Scores Here we provide key findings from our evaluation on each task. A task-specific summary is reported in Table 3. Detailed results comprising all tasks, models, language clusters and varieties are reported in Tables 10 to 20 in Appendix E.
Structured prediction We present visualizations for the task-specific maximum scores. We show the one for Dependency Parsing in Fig. 3, where we observe that low-resource varieties from language clusters such as Tupi-Guarani (indigenous South American cluster), Saami and Komi (low-resource Uralic language clusters) have the lowest performance compared to Standard English and other closely related Germanic and Romance clusters. These low-resource varieties are also not included in the pretraining stage of our base language models (eg. mBERT). Furthermore, this trend is evident across all three structured prediction tasks. On the other hand, high-resource Indo-European languages such as Portuguese, French, and Norwegian usually perform better.
Sequence classification For DId and SA, we generally collate different datasets for each language cluster and therefore, report the comparative classification results together. As a result, the locality level (e.g. city/region/country) also varies across clusters. For example, we report city-level DId results for Arabic and High German but country-level
num. num. avg. Category Task cl. var. score Max-score cluster/variety Min-score cluster/variety
Structured prediction DEP parsing 16 40 64.3 sw. shifted romance/brazilian portuguese 94.4 tupi-guarani sg./mbyá guaraní (a:brazil) 9.0 POS tagging 17 51 72.1 norwegian/norwegian bokmål (m:written) 98.7 tupi-guarani sg./mbyá guaraní (a:brazil) 1.9 NER 27 85 70.1 eastern romance/romanian 94.2 anglic/jamaican creole english 0.0
Sequence classification NLI 15 38 64.2 anglic/english 83.4 sotho-tswana (s.30)/southern sotho 34.6 TC 15 38 77.7 sinitic/cmm. sinitic (o:traditional) 89.8 kurdish/central kurdish 19.4 DId 6 49 67.0 sinitic/mandarin chinese (a:taiwan, o:simp.) 98.6 sw. shifted romance/portuguese (m:written) 17.4 SA 1 9 80.3 arabic/tunisian arabic 94.6 arabic/south levantine arabic 58.9
results for Portuguese, Spanish and English. In the case of SA, we have region/country-level results for Arabic varieties. For TC and NLI, we have the same set of clusters and varieties. However, we only report the zero-shot transfer performance from Standard English for NLI using the newly created translate-test NLI dataset.
For TC and NLI, we observe the largest incluster disparity in the Kurdish cluster, with Northern Kurdish outperforming all others. The Sotho varieties consistently perform significantly lower compared to other clusters. For all three sequence classification tasks, we generally find the Chinese cluster performing on par with high-resource Latin counterparts.
Question answering We generally do not see large gaps in performance within varieties in each language cluster. In EQA zero-shot experiments, English and its varieties have the highest performance overall and Korean varieties score the lowest. Combined fine-tuning boosts performance on
all language clusters except in English. It’s important to note that these EQA scores primarily indicate the model’s robustness to accent-level differences and transcription noise, rather than broad dialectal robustness. However, further investigation is needed to determine whether this robustness specifically applies to both accent-level differences and transcription noise, or to any character-level variation up to a certain threshold.
For the MRC task, the performance peaked at 53.4 for Standard English, while the lowest score was 29 for Southern Sotho. More detailed results are presented in Tables 15 and 19.
Machine translation The performance gap here varies widely across and within language varieties. Performance is similar within the Swiss-German cluster, with higher performance (see Figure 5) across regions in Northern Switzerland, which is geographically closer to Germany. The performance gap for Norwegian dialects (Figure 10b) is surprising as we perform zero-shot transfer from
Norwegian Nynorsk (a Western dialect) but obtain better performance on the Eastern dialect. Within Arabic, Riyadh is the highest performer while Sfax performs the worst. For the Bengali cluster, Jessore has the highest performance –this is not surprising since it is one of the dialects from which standard Bengali originated (Alam et al., 2023). The Ethiopian variety of Tigrinya exhibits a higher performance than the Eritrean one, even though Tigrinya is more commonly spoken in Eritrea 5. Amongst the clusters within the Basque cluster, Barkoxe and Maule have the lowest score while Azkaine scores the highest.
4.2 Dialectal Gap Across Language Clusters
In Fig. 4, we plot the zero-shot dialectal gap for three tasks. In the x-axis, we report the aggregated cluster-level gap Geng(eng, v), compared against the fine-tuning variety (standard English) while in y-axis we report Geng(v̄, v), the gap compared against the representative variety of a cluster. In an ideal scenario, we would want both of these gap values to be close to zero. However, this is certainly not the case. The general observed trend is that the low-resource clusters have higher gaps of both Geng(eng, C) and Geng(v̄, C), whereas highresource Germanic and Sinitic language clusters consistently exhibit low dialectal gaps. That said, certain specific high-resource varieties, such as Standard German and its dialectal counterparts like Swiss German, showcase significant within-cluster dialectal gaps (Fig. 4a).
We primarily report dialectal gaps using zero-
5https://en.wikipedia.org/wiki/Tigrinya_ language
shot transfer because the finetuning data available across task and cluster is very disproportionate. Often the in-cluster/variety data is not good enough in terms of data quality and quantity. For example, we have 37 varieties in 13 clusters for dependency parsing but out of these, only 20 varieties have data available for in-variety fine-tuning. This lacking becomes more apparent when we compare the statistics of two types of within-cluster dialectal gaps: zero-shot Geng(v̄, C) against fine-tuning Gv̄/in-variety(v̄, C) in Table 4. In general, the withincluster dialectal disparity is smaller for zero-shot transfer (i.e. Geng(v̄, C) ≤ |Gv̄/in-variety(v̄, C)|). Here, the in-cluster/variety fine-tuning results in a higher performance deviation primarily due to the inconsistent variety-specific finetuning data quality.
5 Discussion
High resource vs. low resource varieties The highest-performing varieties are mostly standard high-resource languages and a few high-resource dialects (Norwegian dialects) whereas, the majority of the lowest-performing language variants are lowresourced varieties. This clear distinction of language varieties points towards the large existence of in-cluster dialectal gaps. Furthermore, this finding correlates with language script differences. We observe that 77.2% of top-10 varieties in terms of maximum obtainable score are written with Latin script. Another finding is the performance instability of low-resource varieties across tasks. For instance, Old Guarani performs better in DEP parsing whereas, Mbyá Guarani (Paraguay) surpasses it in POS tagging even though the dataset remains the same (i.e. UD). For more detailed comparisons, in Appendix Table 21, we report the top-10 highest and lowest-scoring varieties across different tasks.
Model hyperparameter tuning In Table 5, we compare the baseline multilingual models. mBERT was comparatively easier to train than XLM-R using the default learning rates reported in the earlier task experiments. Often for tasks such as MRC, we needed to tune hyperparameters (e.g. learning rate, max. sequence length) in case of XLMR. However, once we identify a hyperparameter configuration that converges in a zero-shot setting, we use the same to train all the languagevariant training for that specific task. As a result, for some low-resource languages, XLM-R does not converge compared to mBERT. For example, XLM-R dependency parsing UAS score for Norwegian-NynorskLIA is 56.08 (zeroshot) and 8.25 (in-variety FT) whereas, we get 78.39 for invariety mBERT fine-tuning. We suspect this hyperparameter tuning issue is one of the contributing factors toward a lower winning rate of XLM-R in few-shot / fine-tuning settings. However, this could be improved further with an extensive parameter grid search and settings specifically tailored for each language cluster.
Positive zero-shot transfer for Latin varieties Low-resource varieties written in Latin script receive greater benefit in zero-shot because of effective transfer from high-resource Standard English. With the presence of In-cluster/variety finetuning data, we effectively diminish this script effect to some extent. For example in the Hindi cluster, Fiji performs better than its Latin non-standard counterparts with in-cluster finetuning for NER (Table 12). In summary, if the standard variety of a cluster is non-latin but high-resource, then the success rate of in-cluster/variety fine-tuning tends to be higher. However, where all dialects are low-resource, Latin script varieties utilizing zero-shot transfer, eventu-
ally surpass others in the performance hierarchy. As an example, we report the zero-shot NER instances where the low-resource varieties perform better than the representative ones in Appendix Table 22 (most of these use Latin script).
LLM evaluation via In-context learning For SA and EQA tasks, we have in-context learning results using the Mistral7B LLM. Comparing the performance against zero-shot and fine-tuning using our encoder-based models, we find dialect performance of LLM is better than zero-shot transfer but falls behind the finetuned results. On top of that, data contamination (Ahuja et al., 2023) during LLM evaluation is another existing issue while considering these few available dialectal resources. Creating translation-based comparable data might be a solution to perform a fair benchmarking of LLMs on low-resource varieties.
Misinterpreting evaluation metrics We also report the cluster-level population-weighted average (i.e. demographic utility) which rewards a system more when it provides increased linguistic utility (eg. raw F1 score) for varieties, spoken by a larger population compared to varieties spoken by a smaller population. Alone, this metric could be misleading if we consider the fact that the performance gap among all varieties from a particular cluster should be minimal. On the other hand, solely looking into the linguistic utility average does not give a clear picture either (e.g. often overshadows the larger performance deviation of certain varieties having extreme scores). So for all clusters and tasks, we report the linguistic utility average as well as the demographic utility average, the minimum score of a cluster, and the standard deviation in Tables 23 to 30.
6 Conclusion
We propose DIALECTBENCH, the first-ever inclusive Dialectal NLP benchmark reporting performance evaluation and disparity across standard and non-standard varieties. This is one step towards the effort of bringing more and more language under the paradigm of NLP technology. We would like to further improve the benchmark, constructing highquality comparable data and expanding the task coverage to speech-based NLP technologies.
Limitations
The data quality and quantity, variety coverage vary significantly across tasks because of data scarcity issues. We avoid full-scale LLMevaluation consciously because of the uncertain data-contamination issue and their well-known lower performance threshold compared to smaller masked-language-modeling-based fine-tuned models. In addition, we focus on text-based NLP tasks for this current iteration. Moreover, we do not claim the representative varieties of each language clusters to be any kind of superior or standardized forms over the other varieties. These varieties are chosen to perform a well-informed comparison among the perceived well-resourced linguistic variety and its counterparts having lesser data availability. At the same time, the mutual intelligibility and phylogenetic similarity of the similar cluster varieties also vary across cluster and this was not selected in a numerically quantifiable manner.
Evaluation Limitations To further improve the evaluation fairness of the current version of DIALECTBENCH, we need (i) Parallel corpus utilization to prepare task-specific data (ii) Translationbased task data generation to perform comparable analysis (iii) Quantifying the resource-supply and demand as well as population-coverage (Song et al., 2023) to identify where a variety stands in the global landscape of linguistic utility. Here, we have accumulated data for diverse varieties across tasks that vary significantly in terms of quality, example count, and domain. However, to perform a perfectly fair comparison of dialectal inequality, we should consider high-quality comparable data (e.g. parallel corpus, similar varieties across tasks) which is not available at this point.
Continuity of DIALECTBENCH Despite our best effort, this benchmark does not include every one of the already published task-specific dialectal
datasets. So, our next steps on this project involve hosting the benchmark on the website that displays the current statistics of the datasets in DIALECTBENCH. We will also encourage researchers to add new and existing datasets for tasks, language clusters that might be currently missing alongside the respective baselines.
Space Limitations Our study encompasses a large set of evaluation result tables, their corresponding visualizations and findings analysis. Due to space limitations, we have to move the detailed reports (Appendix E) and the rest of the visualizations (Appendix D) in the Appendix. To better assist, we include an Appendix Table of Content (Table 6) at the introductory section of Appendix (Section 6).
Ethics Statement
This work is a compilation of existent dialectal datasets across different tasks, including structured prediction and generative tasks. Our experiments do not particularly optimize for the best model performance of these tasks. Therefore we acknowledge that for some of the tasks, the baseline models might not be robust enough to handle dialectal text hence resulting in wrong predictions and generations. We believe that this underscores the need for building models robust to different language variations and future work should focus on this.
Acknowledgements
This material is based upon work supported by the US National Science Foundation under Grants No. IIS-2125466, IIS-2125948, CAREER Grant No. IIS2142739, as well as NSF Grants No. IIS2203097 and IIS-2125201. We gratefully acknowledge support from Alfred P. Sloan Foundation Fellowship. This research is also supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022-22072200004. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
References Adel Abdelli, Fayçal Guerrouf, Okba Tibermacine, and
Belkacem Abdelli. 2019. Sentiment analysis of Arabic Algerian dialect using a supervised method. In 2019 International Conference on Intelligent Systems and Advanced Computing Sciences (ISACS), pages 1–6.
Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021. ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7088–7105, Online. Association for Computational Linguistics.
David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O. Alabi, Yanke Mao, Haonan Gao, and Annie En-Shiun Lee. 2023. SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects.
Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. 2023. Megaverse: Benchmarking large language models across languages, modalities, models and tasks.
Marwan Al Omari, Moustafa Al-Hajj, Nacereddine Hammami, and Amani Sabra. 2019. Sentiment classifier: Logistic regression for Arabic services’ reviews in Lebanon. In 2019 International Conference on Computer and Information Sciences (ICCIS), pages 1–5.
Md Mahfuz Ibn Alam, Sina Ahmadi, and Antonios Anastasopoulos. 2023. CODET: A benchmark for contrastive dialectal evaluation of machine translation.
Khaled Mohammad Alomari, Hatem M. ElSherif, and Khaled Shaalan. 2017. Arabic tweets sentimental analysis using machine learning. In Advances in Artificial Intelligence: From Theory to Practice, pages 602–610, Cham. Springer International Publishing.
Dhuha Alqahtani, Lama Alzahrani, Maram Bahareth, Nora Alshameri, Hend Al-Khalifa, and Luluh Aldhubayi. 2022. Customer sentiments toward Saudi banks during the Covid-19 pandemic. In Proceedings of the 5th International Conference on Natural Language and Speech Processing (ICNLSP 2022), pages 251–257, Trento, Italy. Association for Computational Linguistics.
Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2023. The Belebele benchmark: a parallel reading comprehension dataset in 122 language variants. arXiv preprint arXiv:2308.16884.
Verena Blaschke, Hinrich Schütze, and Barbara Plank. 2023. Does manipulating tokenization aid crosslingual transfer? A study on POS tagging for nonstandardized languages. In Proceedings of the Tenth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 40–54, Dubrovnik, Croatia. Association for Computational Linguistics.
Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. 2022. Systematic inequalities in language technology performance across the world’s languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5486–5505, Dublin, Ireland. Association for Computational Linguistics.
Su Lin Blodgett, Johnny Wei, and Brendan O’Connor. 2018. Twitter Universal Dependency parsing for African-American and mainstream American English. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1415–1425, Melbourne, Australia. Association for Computational Linguistics.
Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Jack K. Chambers and Peter Trudgill. 1998. Dialectology. Cambridge University Press.
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454–470.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating crosslingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of
deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
AbdelRahim Elmadany, ElMoatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. ORCA: A challenging benchmark for Arabic language understanding. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9559–9586, Toronto, Canada. Association for Computational Linguistics.
Fahim Faisal, Sharlina Keshava, Md Mahfuz Ibn Alam, and Antonios Anastasopoulos. 2021. SD-QA: Spoken dialectal question answering for the real world. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3296–3315, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Chayma Fourati, Hatem Haddad, Abir Messaoudi, Moez BenHajhmida, Aymen Ben Elhaj Mabrouk, and Malek Naski. 2021. Introducing a large Tunisian Arabizi dialectal dataset for sentiment analysis. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 226–230, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
Moncef Garouani and Jamal Kharroubi. 2022. MAC: An open and free Moroccan Arabic corpus for sentiment analysis. In Innovations in Smart Cities Applications Volume 5, pages 849–858, Cham. Springer International Publishing.
Mika Hämäläinen, Khalid Alnajjar, Niko Partanen, and Jack Rueter. 2021. Finnish dialect identification: The effect of audio and text. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8777–8783, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Harald Hammarström and Robert Forkel. 2022. Glottocodes: Identifiers linking families, languages and dialects to comprehensive reference information. Semantic Web Journal, 13(6):917–924.
Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Strötgen, and Dietrich Klakow. 2021. A survey on recent approaches for natural language processing in low-resource scenarios. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2545–2568, Online. Association for Computational Linguistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multitask benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International
Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4411–4421. PMLR.
Tommi Jauhiainen, Heidi Jauhiainen, and Krister Lindén. 2022. Italian language and dialect identification and regional French variety detection using adaptive naive Bayes. In Proceedings of the Ninth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 119–129, Gyeongju, Republic of Korea. Association for Computational Linguistics.
Tommi Jauhiainen, Krister Lindén, and Heidi Jauhiainen. 2019. Discriminating between Mandarin Chinese and Swiss-German varieties using adaptive language models. In Proceedings of the Sixth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 178–187, Ann Arbor, Michigan. Association for Computational Linguistics.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7B. arXiv:2310.06825.
Bjarte Johansen. 2019. Named-entity recognition for Norwegian. In Proceedings of the 22nd Nordic Conference on Computational Linguistics, NoDaLiDa.
Anjali Kantharuban, Ivan Vulić, and Anna Korhonen. 2023. Quantifying the dialect gap and its correlates across languages.
Simran Khanuja, Sandipan Dandapat, Anirudh Srinivasan, Sunayana Sitaram, and Monojit Choudhury. 2020. GLUECoS: An evaluation benchmark for code-switched NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3575–3585, Online. Association for Computational Linguistics.
Heather Lent, Kushal Tatariya, Raj Dabre, Yiyi Chen, Marcell Fekete, Esther Ploeger, Li Zhou, Hans Erik Heje, Diptesh Kanojia, Paul Belony, Marcel Bollmann, Loïc Grobol, Miryam de Lhoneux, Daniel Hershcovich, Michel DeGraff, Anders Søgaard, and Johannes Bjerva. 2023. CreoleVal: Multilingual multitask benchmarks for creoles.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020. XGLUE: A new benchmark dataset for cross-lingual pre-training, understanding and generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6008–6018, Online. Association for Computational Linguistics.
Richard Littauer. n.d. Low resource languages: Resources for conservation, development, and documentation of low resource (human) languages. https://github.com/RichardLitt/ low-resource-languages. [Accessed 15-11- 2023].
Salima Medhaffar, Fethi Bougares, Yannick Estève, and Lamia Hadrich-Belguith. 2017. Sentiment analysis of Tunisian dialects: Linguistic ressources and experiments. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 55– 61, Valencia, Spain. Association for Computational Linguistics.
Jamshidbek Mirzakhalov. 2021. Turkic Interlingua: A Case Study of Machine Translation in Low-resource Languages. Ph.D. thesis, University of South Florida.
Jamshidbek Mirzakhalov, Anoop Babu, Duygu Ataman, Sherzod Kariev, Francis Tyers, Otabek Abduraufov, Mammad Hajili, Sardana Ivanova, Abror Khaytbaev, Antonio Laverghetta Jr., Bekhzodbek Moydinboyev, Esra Onal, Shaxnoza Pulatova, Ahsan Wahab, Orhan Firat, and Sriram Chellappan. 2021a. A large-scale study of machine translation in Turkic languages. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5876–5890, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jamshidbek Mirzakhalov, Anoop Babu, Duygu Ataman, Sherzod Kariev, Francis Tyers, Otabek Abduraufov, Mammad Hajili, Sardana Ivanova, Abror Khaytbaev, Antonio Laverghetta Jr, et al. 2021b. A large-scale study of machine translation in turkic languages. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5876–5890.
Mahmoud Nabil, Mohamed Aly, and Amir Atiya. 2015. ASTD: Arabic sentiment tweets dataset. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2515–2519, Lisbon, Portugal. Association for Computational Linguistics.
NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No Language Left Behind: Scaling humancentered machine translation. arXiv:2207.04672.
Sebastian Nordhoff. 2012. Linked data for linguistic diversity research: Glottolog/langdoc and asjp. In Christian Chiarcos, Sebastian Nordhoff, and Sebastian Hellmann, editors, Linked Data in Linguistics. Representing and Connecting Language Data and Language Metadata, pages 191–200. Springer, Heidelberg.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958, Vancouver, Canada. Association for Computational Linguistics.
Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Ji Yoon Han, Jangwon Park, Chisung Song, Junseong Kim, Youngsook Song, Taehwan Oh, Joohong Lee, Juhyun Oh, Sungwon Lyu, Younghoon Jeong, Inkwon Lee, Sangwoo Seo, Dongjun Lee, Hyunwoo Kim, Myeonghwa Lee, Seongbo Jang, Seungwon Do, Sunkyoung Kim, Kyungtae Lim, Jongwon Lee, Kyumin Park, Jamin Shin, Seonghyun Kim, Lucy Park, Lucy Park, Alice Oh, Jung-Woo Ha (NAVER AI Lab), Kyunghyun Cho, and Kyunghyun Cho. 2021. KLUE: Korean language understanding evaluation. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1. Curran.
Alexandre Rademaker, Fabricio Chalub, Livy Real, Cláudia Freitas, Eckhard Bick, and Valeria de Paiva. 2017. Universal Dependencies for Portuguese. In Proceedings of the Fourth International Conference on Dependency Linguistics (Depling), pages 197– 206, Pisa, Italy.
Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Massively multilingual transfer for NER. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 151–164, Florence, Italy. Association for Computational Linguistics.
Anthony Rios. 2020. FuzzE: Fuzzy fairness evaluation of offensive language classifiers on AfricanAmerican English. Proceedings of the AAAI Conference on Artificial Intelligence, 34(01):881–889.
Sebastian Ruder, Jonathan H. Clark, Alexander Gutkin, Mihir Kale, Min Ma, Massimo Nicosia, Shruti Rijhwani, Parker Riley, Jean-Michel A. Sarr, Xinyi Wang, John Wieting, Nitish Gupta, Anna Katanova, Christo Kirov, Dana L. Dickinson, Brian Roark, Bidisha Samanta, Connie Tao, David I. Adelani, Vera Axelrod, Isaac Caswell, Colin Cherry, Dan Garrette, Reeve Ingle, Melvin Johnson, Dmitry Panteleev, and Partha Talukdar. 2023. XTREME-UP: A user-centric scarce-data benchmark for under-represented languages.
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021. XTREME-R: Towards more challenging
and nuanced multilingual evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10215–10245, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Hanna Sababa and Athena Stassopoulou. 2018. A classifier to distinguish between Cypriot Greek and Standard Modern Greek. In 2018 Fifth International Conference on Social Networks Analysis, Management and Security (SNAMS), pages 251–255.
Yves Scherrer, Tanja Samardžić, and Elvira Glaser. 2019. Digitising Swiss German: how to process and study a polycentric spoken language. Language Resources and Evaluation, 53.
Haitham Seelawi, Ibraheem Tuffaha, Mahmoud Gzawi, Wael Farhan, Bashar Talafha, Riham Badawi, Zyad Sober, Oday Al-Dweik, Abed Alhakim Freihat, and Hussein Al-Natsheh. 2021. ALUE: Arabic language understanding evaluation. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, pages 173–184, Kyiv, Ukraine (Virtual). Association for Computational Linguistics.
Yueqi Song, Catherine Cui, Simran Khanuja, Pengfei Liu, Fahim Faisal, Alissa Ostapenko, Genta Indra Winata, Alham Fikri Aji, Samuel Cahyawijaya, Yulia Tsvetkov, Antonios Anastasopoulos, and Graham Neubig. 2023. GlobalBench: A benchmark for global progress in natural language processing.
Hongmin Wang, Yue Zhang, GuangYong Leonard Chan, Jie Yang, and Hai Leong Chieu. 2017. Universal Dependencies parsing for colloquial Singaporean English. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1732–1744, Vancouver, Canada. Association for Computational Linguistics.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022. Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085–5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, and Ayu Purwarianti. 2020. IndoNLU: Benchmark and resources for evaluating Indonesian
natural language understanding. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 843–857, Suzhou, China. Association for Computational Linguistics.
Marcos Zampieri, Kai North, Tommi Jauhiainen, Mariano Felice, Neha Kumari, Nishant Nair, and Yash Bangera. 2023. Language variety identification with true labels.
Daniel Zeman, Joakim Nivre, Mitchell Abrams, Elia Ackermann, Noëmi Aepli, Hamid Aghaei, Željko Agić, Amir Ahmadi, Lars Ahrenberg, Ajede, and et al. 2021. Universal Dependencies 2.9. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University.
Caleb Ziems, Jiaao Chen, Camille Harris, Jessica Anderson, and Diyi Yang. 2022. VALUE: Understanding dialect disparity in NLU. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3701–3720, Dublin, Ireland. Association for Computational Linguistics.
Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023. MultiVALUE: A framework for cross-dialectal English NLP. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 744–768, Toronto, Canada. Association for Computational Linguistics.
Appendix
In this supplementary material, we provide the following: (i) Relevant literature review (ii) Overall results and dataset details that we could not fit into the main body of the paper.
A Related Work
The majority of multilingual benchmarks (Hu et al., 2020; Ruder et al., 2023, 2021; Liang et al., 2020; Wilie et al., 2020; Park et al., 2021) have heavily focused on dominant language varieties. However, the performance disparity between standard languages and their dialectal counterparts has been studied (Kantharuban et al., 2023; Ziems et al., 2022; Rios, 2020; Ziems et al., 2023) and shown to be significantly large across different tasks. Still, the large-scale generalization of this finding comprising numerous task
categories is yet to be done. There have been previous efforts to bridge this gap though. Some work has focused on curating dialectal datasets across several tasks within one language cluster, while others have focused on single tasks across many language clusters. For instance, ORCA (Elmadany et al., 2023), ARLUE (Abdul-Mageed et al., 2021) and ALUE (Seelawi et al., 2021) are dedicated natural language understanding benchmarks focusing on Arabic varieties alone. Multi-VALUE (Ziems et al., 2023) was developed for benchmarking NLP tasks in English varieties. (Mirzakhalov et al., 2021a) is a suite of resources for benchmarking MT in several Turkic languages. There has also been a large body of work on dialect identification across several languages (Jauhiainen et al., 2022; Hämäläinen et al., 2021). Recently CODET (Alam et al., 2023) was released as a contrastive dialectal MT benchmark covering 882 different variations from nine different languages. To the best of our knowledge, we are the first to do a large scale aggregation of dialectal data across several language clusters and tasks.
B Tasks of DIALECTBENCH
DIALECTBENCH includes 10 NLP tasks falling into four broader categories: structured prediction, sentence classification, question answering, and text generation. In Table 1, we present statistics for the datasets for each task and briefly discuss each task below.
Dependency parsing For the dependency parsing task, we include only those Universal Dependencies (UD) (Zeman et al., 2021) languages that have dialectal data. Beyond the data available in UD 2.12, we incorporate two additional datasets for African-American English (AAVE) Twitter data (Blodgett et al., 2018) and Singlish (Wang et al., 2017). To make these two datasets compatible with the UD processing pipeline, we replace the original dependency labels with the labels corresponding to the official UD formalism.
Part of speech (POS) tagging We use the same UD languages for POS tagging that we used for dependency parsing. At the same time, we use the POS data instances from Singlish (Wang et al., 2017). Moreover, we include six Finnish dialects, four Arabic dialects and Occitan through the UPOS label standardized pipeline proposed by Blaschke et al. (2023).
Named entity recognition (NER) We use data from the 176-language version of the Wikiann dataset processed by Rahimi et al. (2019). All these languages provide both training and test data. In addition, we include dialectal data from the original Wikiann dataset (282 languages) (Pan et al., 2017) for evaluation. Moreover, we include three Norwegian dialects (Johansen, 2019) with train, test and validation datasets that use a slightly different set of NER tags (GEO, ORG, OTH, PER) compared to the one we use in Wikiann (LOC, ORG, PER). We leave these levels as it is and do not convert to the Wikiann tags.
Dialect identification We include dialect identification experiments on Arabic, Greek, Portuguese, English, Spanish, and Swiss German dialectal datasets. In these datasets, we find large variations in the level of granularity with which dialects are classified. For instance, the MADAR corpus differentiates Arabic varieties at the city level (Bouamor et al., 2018), whereas our English and Spanish datasets are labeled with country names (Zampieri et al., 2023).
Sentiment classification Here we include several different Arabic varieties. Like other dialectal datasets, these datasets do not follow one standard labeling process. However, all datasets contain two main sentiment types: positive and negative. A number of datasets contain additional labels such as “objective” or “neutral.” In our setting, we perform a further split of data to provide validation data for each dialect. However, we do not remove these extra labeled data for information preservation.
Topic classification We use the SIB-200 dataset (Adelani et al., 2023) for topic classification task SIB-200 was constructed from the FLORES-200 translation datasets. The authors annotated the English dataset of FLORES-200 with 6 topic labels and then further propagated these labels to the translated instances for all other languages. For our case of benchmarking dialectal segments, we use the dialectal and regional varieties from SIB-200.
Natural language inference For the natural language inference task, there is no existing dataset with varieties. So we use the existing English test set of XNLI (Conneau et al., 2018) and construct a multilingual dialect-focused translated evaluation dataset. We use a state-of-the-art machine translation model (NLLB-200 3B) to translate the English test set to 12 language clusters encompassing 40 varieties (Complete data statistics are reported in Table 7). After that, we perform zero-shot cross-lingual transfer from the English finetuned NLI model. We refer to this setting as translate-test.
Multiple-choice machine reading comprehension This task aims to evaluate the capability of multiplechoice question answering given a context passage. The question could be answered by understanding the context passage while the right answer is given at one of the multiple choices. We use the Chinese, Sotho and Arabic clusters data from the recently released Belebele MRC dataset (Bandarkar et al., 2023). This is an evaluation-only dataset.
Extractive question answering This task predicts the answer span given a question and context passage. We use the SD-QA dialectal question-answering dataset (Faisal et al., 2021). SD-QA is an evaluation dataset built on top of TyDiQA (Clark et al., 2020), another well-known typologically diverse questionanswering dataset. SD-QA contains the spoken utterances and transcription of the original TyDiQA question from speakers of English, Bengali, Arabic, Korean, and Swahili. We only use the textual part that contains the transcription of the dialectal spoken question matching the original TyDiQA question text. Note that since the transcriptions of the questions are obtained through automatic speech recognition, they may include both dialectal variations and noise due to ASR transcription errors.
Machine translation We evaluate variety translation using the CODET benchmark (Alam et al., 2023) and the TIL MT corpus (Mirzakhalov et al., 2021b). CODET contains a contrastive dataset of 882 different varieties from nine different languages. We evaluate dialects here at city level for all languages except Italian and Swiss-German and aggregate dialects at the region level for them. The TIL corpus contains parallel translations across 22 Turkic languages, but in our evaluations we only include 8 turkic languages (Turkic, Sakha, Kazakh, Karakalpak, Bashkir, Azerbaijani, Kyrgyz) that have parallel English translations.
C Varieties and Clusters of DIALECTBENCH
C.1 DIALECTBENCH Variety list
Table 8: Varieties represented in DialectBench.
Language cluster Variety name Glottocode mBERT seen UDP POS NER SDQA RCMC NLI TC SC DI MT
albanian albanian alba1267 ✓ ✓ ✓ - - - - - - - -gheg albanian gheg1238 - ✓ ✓ - - - - - - - -
anglic
african american vernacular english
afri1276 - ✓ - - - - - - - - -
australian english aust1314 - - - - ✓ - - - - - - english stan1293 ✓ ✓ ✓ ✓ - ✓ ✓ ✓ - ✓ - english (a:scotland) stan1293 - - - - ✓ - - - - - - english (a:uk) stan1293 - - - - - - - - - ✓ - english (o:controlled) stan1293 - - - ✓ - - - - - - - indian english (a:north) indi1255 - - - - ✓ - - - - - - indian english (a:south) indi1255 - - - - ✓ - - - - - - irish english iris1255 - - - - ✓ - - - - - - jamaican creole english jama1262 - - - ✓ - - - - - - - kenyan english keny1281 - - - - ✓ - - - - - - new zealand english newz1240 - - - - ✓ - - - - - - nigerian english nige1260 - - - - ✓ - - - - - - north american english nort3314 - - - - - - - - - ✓ - old english (ca. 450-1100) olde1238 - - - ✓ - - - - - - - philippine english phil1246 - - - - ✓ - - - - - - singlish sing1272 - ✓ ✓ - - - - - - - - southeast american english sout3300 - - - - ✓ - - - - - - southern african english sout3331 - - - - ✓ - - - - - -
arabic
aleppo alep1241 - - - - - - - - - ✓ ✓ algerian arabic alge1239 - - - - ✓ - - - ✓ ✓ ✓ arabian peninsula arabic (a:yemen) arab1393 - - - - - - - - - ✓ ✓ arabic (a:bahrain) - - - - - ✓ - - - - - - arabic (a:jordan) - - - - - ✓ - - - ✓ - - arabic (a:saudi-arabia) - - - - - ✓ - - - ✓ - -
Continued on next page
Table 8: Varieties represented in DialectBench.
Language clusters Variety name Glottocode mBERT seen UDP POS NER SDQA RCMC NLI TC SC DI MT
egyptian arabic egyp1253 - - ✓ ✓ ✓ ✓ ✓ ✓ ✓ - - egyptian arabic (a:alx) egyp1253 - - - - - - - - - ✓ ✓ egyptian arabic (a:asw) egyp1253 - - - - - - - - - ✓ ✓ egyptian arabic (a:cai) egyp1253 - - - - - - - - - ✓ ✓ egyptian arabic (a:kha) egyp1253 - - - - - - - - - ✓ ✓ fez. meknes fezm1238 - - - - - - - - - ✓ ✓ gilit mesopotamian arabic meso1252 - - - - - - - - - ✓ ✓ gulf arabic gulf1241 - - ✓ - - - - - - - - gulf arabic (a:doh) gulf1241 - - - - - - - - - ✓ ✓ gulf arabic (a:jed) gulf1241 - - - - - - - - - ✓ ✓ gulf arabic (a:mus) gulf1241 - - - - - - - - - ✓ ✓ gulf arabic (a:riy) gulf1241 - - - - - - - - - ✓ ✓ levantine arabic nort3139 - - ✓ - - - - - - - - levantine arabic (a:north) nort3139 - - - - - ✓ ✓ ✓ - - - levantine arabic (a:north-dam) nort3139 - - - - - - - - - ✓ ✓ libyan arabic liby1240 - - - - - - - - - - - libyan arabic (a:ben) liby1240 - - - - - - - - - ✓ ✓ moroccan arabic moro1292 - - - - ✓ ✓ ✓ ✓ ✓ - - najdi arabic najd1235 - - - - - ✓ ✓ ✓ - - - north african arabic nort3191 - ✓ ✓ - - - - - - - - north mesopotamian arabic nort3142 - - - - - ✓ ✓ ✓ - - - north mesopotamian arabic (a:bas) nort3142 - - - - - - - - - ✓ ✓ north mesopotamian arabic (a:mos) nort3142 - - - - - - - - - ✓ ✓ rabat-casablanca arabic raba1252 - - - - - - - - - ✓ ✓ sfax sfax1238 - - - - - - - - - ✓ ✓ south levantine arabic sout3123 - ✓ ✓ - - - ✓ ✓ ✓ - - south levantine arabic (a:south-amm) sout3123 - - - - - - - - - ✓ ✓ south levantine arabic (a:south-jer) sout3123 - - - - - - - - - ✓ ✓ south levantine arabic (a:south-sal) sout3123 - - - - - - - - - ✓ ✓ standard arabic stan1318 ✓ ✓ ✓ ✓ - ✓ ✓ ✓ ✓ ✓ - sunni beiruti arabic sunn1238 - - - - - - - - - ✓ ✓ ta’izzi-adeni arabic taiz1242 - - - - - - ✓ ✓ - - - tripolitanian arabic trip1239 - - - - - - - - - ✓ ✓ tunisian arabic tuni1259 - - - - ✓ - ✓ ✓ ✓ - - tunisian arabic (a:tun) tuni1259 - - - - - - - - - ✓ ✓ tunisian arabic (r:casual) tuni1259 - - - - - - - - ✓ - -
basque
basque (a:amenduze) basq1248 ✓ - - - - - - - - - ✓ basque (a:azkaine) basq1248 - - - - - - - - - - ✓ basque (a:baigorri) basq1248 - - - - - - - - - - ✓ basque (a:barkoxe) basq1248 - - - - - - - - - - ✓ basque (a:donibane) basq1248 - - - - - - - - - - ✓ basque (a:garruze) basq1248 - - - - - - - - - - ✓ basque (a:iholdi) basq1248 - - - - - - - - - - ✓ basque (a:jatsu) basq1248 - - - - - - - - - - ✓ basque (a:jutsi) basq1248 - - - - - - - - - - ✓ basque (a:larzabale) basq1248 - - - - - - - - - - ✓ basque (a:luhuso) basq1248 - - - - - - - - - - ✓ basque (a:sara) basq1248 - - - - - - - - - - ✓ basque (a:senpere) basq1248 - - - - - - - - - - ✓ basque (a:suhuskune) basq1248 - - - - - - - - - - ✓ basque (a:uharte) basq1248 - - - - - - - - - - ✓ navarro-labourdin basque (a:behorlegi) basq1249 - - - - - - - - - - ✓ navarro-labourdin basque (a:bidarrai) basq1249 - - - - - - - - - - ✓ navarro-labourdin basque (a:helete) basq1249 - - - - - - - - - - ✓ navarro-labourdin basque (a:mugerre) basq1249 - - - - - - - - - - ✓ navarro-labourdin basque (a:urruna) basq1249 - - - - - - - - - - ✓ souletin (a:maule) basq1250 - - - - - - - - - - ✓
bengali
vanga (a:barisal) vang1242 - - - - - - - - - - ✓ vanga (a:dhaka) vang1242 ✓ - - - ✓ - - - - - ✓ vanga (a:jessore) vang1242 - - - - - - - - - - ✓ vanga (a:khulna) vang1242 - - - - - - - - - - ✓ vanga (a:kushtia) vang1242 - - - - - - - - - - ✓ vanga (a:west bengal) vang1242 ✓ - - - ✓ - - - - - -
circassian adyghe adyg1241 - - - ✓ - - - - - - -kabardian kaba1278 - - - ✓ - - - - - - -
common turkic
bashkir bash1264 ✓ - - - - - - - - - ✓ central oghuz azer1255 - - - - - - - - - - ✓ central oghuz (m:spoken) azer1255 - - - - - - ✓ ✓ - - - crimean tatar crim1257 - - - ✓ - - - - - - - kara-kalpak kara1467 - - - - - - - - - - ✓ kazakh kaza1248 ✓ - - - - - - - - - ✓ kirghiz kirg1245 ✓ - - - - - - - - - ✓ north azerbaijani nort2697 ✓ - - ✓ - - ✓ ✓ - - - sakha yaku1245 - - - - - - - - - - ✓ south azerbaijani sout2697 ✓ - - ✓ - - ✓ ✓ - - - turkish nucl1301 ✓ - - ✓ - - - - - - ✓ uzbek uzbe1247 ✓ - - - - - - - - - ✓
eastern romance aromanian arom1237 - - - ✓ - - - - - - - moldavian mold1248 - - - ✓ - - - - - - -
Continued on next page
Table 8: Varieties represented in DialectBench.
Language clusters Variety name Glottocode mBERT seen UDP POS NER SDQA RCMC NLI TC SC DI MT
romanian roma1327 ✓ - - ✓ - - - - - - -
eastern-western armenianeastern armenian nucl1235 ✓ ✓ ✓ - - - - - - - -western armenian homs1234 - ✓ ✓ - - - - - - - - farsic dari dari1249 - - - - - - - - - - ✓
frisian ems-weser frisian sate1242 - - - ✓ - - - - - - - northern frisian nort2626 - - - ✓ - - - - - - - western frisian west2354 ✓ - - ✓ - - - - - - -
gallo-italian
emiliano-romagnolo emil1243 - - - ✓ - - - - - - - ligurian ligu1248 - ✓ ✓ ✓ - - ✓ ✓ - - - lombard lomb1257 ✓ - - ✓ - - ✓ ✓ - - - piemontese piem1238 ✓ - - ✓ - - - - - - - venetian vene1258 - - - ✓ - - ✓ ✓ - - -
gallo-rhaetian
anglo-norman angl1258 - - - ✓ - - - - - - - arpitan fran1260 - - - ✓ - - - - - - - french stan1290 ✓ ✓ ✓ ✓ - - - - - - - french (a:paris) stan1290 - ✓ ✓ - - - - - - - - friulian friu1240 - - - ✓ - - ✓ ✓ - - - old french (842-ca. 1400) oldf1239 - ✓ ✓ - - - - - - - - romansh roma1326 - - - ✓ - - - - - - - walloon wall1255 - - - ✓ - - - - - - -
greater panjabic eastern panjabi panj1256 ✓ - - ✓ - - - - - - -western panjabi west2386 ✓ - - ✓ - - - - - - -
greek
apulian greek apul1237 - - - - - - - - - - ✓ cretan cret1244 - - - - - - - - - - ✓ cypriot greek cypr1249 - - - - - - - - - - - cypriot greek (r:casual, m:written, i:fb) cypr1249 - - - - - - - - - ✓ - cypriot greek (r:casual, m:written, i:other) cypr1249 - - - - - - - - - ✓ - cypriot greek (r:casual, m:written, i:twitter) cypr1249 - - - - - - - - - ✓ - modern greek mode1248 ✓ - - ✓ - - - - - - - modern greek (r:casual, m:written, i:fb) mode1248 - - - - - - - - - ✓ - modern greek (r:casual, m:written, i:other) mode1248 - - - - - - - - - ✓ - modern greek (r:casual, m:written, i:twitter) mode1248 - - - - - - - - - ✓ - pontic pont1253 - - - ✓ - - - - - - -
high german
bavarian bava1246 ✓ - - ✓ - - - - - - - central alemannic swis1247 - - - ✓ - - - - - - - central alemannic (a:ag) swis1247 - - - - - - - - - - ✓ central alemannic (a:ai) swis1247 - - - - - - - - - - ✓ central alemannic (a:ar) swis1247 - - - - - - - - - - ✓ central alemannic (a:be) swis1247 - - - - - - - - - ✓ ✓ central alemannic (a:bl) swis1247 - - - - - - - - - - ✓ central alemannic (a:bs) swis1247 - - - - - - - - - ✓ ✓ central alemannic (a:fr) swis1247 - - - - - - - - - - ✓ central alemannic (a:gl) swis1247 - - - - - - - - - - ✓ central alemannic (a:gr) swis1247 - - - - - - - - - - ✓ central alemannic (a:lu) swis1247 - - - - - - - - - ✓ ✓ central alemannic (a:nw) swis1247 - - - - - - - - - - ✓ central alemannic (a:ow) swis1247 - - - - - - - - - - ✓ central alemannic (a:sg) swis1247 - - - - - - - - - - ✓ central alemannic (a:sh) swis1247 - - - - - - - - - - ✓ central alemannic (a:so) swis1247 - - - - - - - - - - ✓ central alemannic (a:sz) swis1247 - - - - - - - - - - ✓ central alemannic (a:tg) swis1247 - - - - - - - - - - ✓ central alemannic (a:ur) swis1247 - - - - - - - - - - ✓ central alemannic (a:vs) swis1247 - - - - - - - - - - ✓ central alemannic (a:zg) swis1247 - - - - - - - - - - ✓ central alemannic (a:zh) swis1247 - ✓ ✓ - - - - - - ✓ ✓ central bavarian cent1967 - - - - - - - - - - ✓ german stan1295 ✓ ✓ ✓ ✓ - - - - - - - kölsch kols1241 - - - ✓ - - - - - - - limburgan limb1263 - - - ✓ - - ✓ ✓ - - - luxemburgish luxe1243 ✓ - - ✓ - - ✓ ✓ - - - pennsylvania german penn1240 - - - ✓ - - - - - - - pfaelzisch-lothringisch pala1330 - - - ✓ - - - - - - - upper saxon uppe1465 - - - - - - - - - - ✓
hindustani fiji hindi fiji1242 - - - ✓ - - - - - - -hindi hind1269 ✓ - - ✓ - - - - - - -
inuit alaskan inupiaq inup1234 - - - ✓ - - - - - - -kalaallisut kala1399 - - - ✓ - - - - - - -
italian romance
continental southern italian neap1235 - ✓ ✓ ✓ - - - - - - - italian ital1282 ✓ ✓ ✓ ✓ - - ✓ ✓ - - - italian (a:abruzzo) ital1282 - - - - - - - - - - ✓ italian (a:basilicata) ital1282 - - - - - - - - - - ✓ italian (a:calabria) ital1282 - - - - - - - - - - ✓ italian (a:campania) ital1282 - - - - - - - - - - ✓ italian (a:emilia-romagna) ital1282 - - - - - - - - - - ✓ italian (a:friuli-venezia giulia) ital1282 - - - - - - - - - - ✓ italian (a:lazio) ital1282 - - - - - - - - - - ✓ italian (a:liguria) ital1282 - - - - - - - - - - ✓ italian (a:lombardia) ital1282 - - - - - - - - - - ✓
Continued on next page
Table 8: Varieties represented in DialectBench.
Language clusters Variety name Glottocode mBERT seen UDP POS NER SDQA RCMC NLI TC SC DI MT
italian (a:marche) ital1282 - - - - - - - - - - ✓ italian (a:molise) ital1282 - - - - - - - - - - ✓ italian (a:piemonte) ital1282 - - - - - - - - - - ✓ italian (a:puglia) ital1282 - - - - - - - - - - ✓ italian (a:sardegna) ital1282 - - - - - - - - - - ✓ italian (a:sicilia) ital1282 - - - - - - - - - - ✓ italian (a:toscana) ital1282 - - - - - - - - - - ✓ italian (a:trentino-alto adige/südtirol) ital1282 - - - - - - - - - - ✓ italian (a:umbria) ital1282 - - - - - - - - - - ✓ italian (a:unknown) ital1282 - - - - - - - - - - ✓ italian (a:veneto) ital1282 - - - - - - - - - - ✓ italian (r:casual, m:written, i:tweet) ital1282 ✓ ✓ ✓ - - - - - - - - italian (r:formal, m:written, i:essay) ital1282 ✓ ✓ ✓ - - - - - - - - italian romance (a:apulia, m:spoken, i:tarantino) - - - - ✓ - - - - - - - sicilian sici1248 ✓ - - ✓ - - ✓ ✓ - - -
komi
komi komi1267 - - - ✓ - - - - - - - komi-permyak komi1269 - ✓ ✓ ✓ - - - - - - - komi-zyrian (m:spoken) komi1268 - ✓ ✓ - - - - - - - - komi-zyrian (m:written) komi1268 - ✓ ✓ - - - - - - - -
korean korean (a:south-eastern,m:spoken) kore1280 - - - - ✓ - - - - - -
seoul (m:spoken) seou1239 ✓ - - - ✓ - - - - - -
kurdish
central kurdish cent1972 - - - ✓ - - ✓ ✓ - - - kurdish kurd1259 - - - ✓ - - - - - - - northern kurdish nort2641 - - - - - - ✓ ✓ - - - sine’i sine1239 - - - - - - - - - - ✓ sorani sora1257 - - - - - - - - - - ✓
latvian east latvian east2282 - - - ✓ - - ✓ ✓ - - -latvian latv1249 ✓ - - ✓ - - ✓ ✓ - - -
mari eastern mari east2328 - - - ✓ - - - - - - -western mari west2392 - - - ✓ - - - - - - -
modern dutch dutch dutc1256 ✓ - - ✓ - - ✓ ✓ - - - western flemish vlaa1240 - - - ✓ - - - - - - - zeeuws zeeu1238 - - - ✓ - - - - - - -
neva
central and north pohjanmaa (a:ostrobothnian)
cent1985 - - ✓ - - - - - - - -
estonian esto1258 ✓ - ✓ - - - - - - - - finnish finn1318 ✓ - ✓ - - - - - - - - häme (a:tavastian) hame1240 - - ✓ - - - - - - - - neva (a:south-west trans) - - - ✓ - - - - - - - - savo (a:savonian) savo1254 - - ✓ - - - - - - - - southeastern finnish (a:southeast) sout1743 - - ✓ - - - - - - - - southwestern finnish (a:southwest) sout2677 - - ✓ - - - - - - - -
norwegian
norwegian norw1258 ✓ - - - - - - - - - - norwegian (a:eastern) norw1258 - - - - - - - - - - ✓ norwegian (a:setesdal) norw1258 - - - - - - - - - - ✓ norwegian (a:southwestern) norw1258 - - - - - - - - - - ✓ norwegian (m:written, i:samnorsk) - - - - ✓ - - - - - - - norwegian bokmål (m:written) norw1259 ✓ ✓ ✓ ✓ - - ✓ ✓ - - - norwegian nynorsk (m:written) norw1262 ✓ ✓ ✓ ✓ - - ✓ ✓ - - - norwegian nynorsk (m:written, i:old) norw1262 - ✓ ✓ - - - - - - - -
saami north saami nort2671 - ✓ ✓ - - - - - - - -skolt saami skol1241 - ✓ ✓ - - - - - - - - sabellic umbrian umbr1253 - ✓ ✓ - - - - - - - - sardo-corsican corsican cors1241 - - - ✓ - - - - - - -sardinian sard1257 - - - ✓ - - ✓ ✓ - - -
serbian-croatian-bosnian bosnian standard bosn1245 ✓ - - ✓ - - - - - - - croatian standard croa1245 ✓ - - ✓ - - - - - - - serbian standard serb1264 ✓ - - ✓ - - - - - - - serbian-croatian-bosnian sout1528 ✓ - - ✓ - - - - - - -
sinitic
cantonese cant1236 - - - ✓ - - ✓ ✓ - - - classical chinese lite1248 - ✓ ✓ ✓ - - - - - - - classical-middle-modern sinitic (a:hongkong, o:traditional) clas1255 - ✓ ✓ - - - - - - - - classical-middle-modern sinitic (o:simplified) clas1255 ✓ ✓ ✓ - - ✓ ✓ ✓ - - - classical-middle-modern sinitic (o:traditional) clas1255 ✓ - - - - ✓ ✓ ✓ - - - hakka chinese hakk1236 - - - ✓ - - - - - - - mandarin chinese mand1415 - - - ✓ - - - - - - - mandarin chinese (a:mainland, o:simplified) mand1415 - - - - - - - - - ✓ -
Continued on next page
Task Cluster Name Cluster Representative
DEP. Parsing albanian albanian arabic standard arabic eastern-western armenian western armenian sinitic classical-middle-modern sinitic (o:simplified) anglic english gallo-rhaetian french high german german tupi-guarani subgroup i.a old guarani italian romance italian (r:formal, m:written, i:essay) komi komi-zyrian (m:spoken) norwegian norwegian bokmål (m:written) southwestern shifted romance brazilian portuguese saami north saami
POS Tagging albanian albanian anglic english arabic standard arabic eastern-western armenian western armenian gallo-rhaetian french high german german italian romance italian (r:formal, m:written, i:essay) komi komi-zyrian (m:spoken) neva finnish norwegian norwegian bokmål (m:written) saami north saami sinitic classical-middle-modern sinitic (o:simplified) southwestern shifted romance brazilian portuguese tupi-guarani subgroup i.a mbyá guaraní (a:paraguay)
NER
anglic english arabic standard arabic circassian adyghe common turkic turkish eastern romance romanian frisian western frisian gallo-italian piemontese gallo-rhaetian french greater panjabic western panjabi greek modern greek high german german hindustani hindi inuit alaskan inupiaq italian romance italian komi komi kurdish kurdish
Continued on next page
DId
anglic english (a:uk) arabic standard arabic greek modern greek (r:casual, m:written, i:twitter) high german central alemannic (a:zh) sinitic mandarin chinese (a:mainland, o:simplified) southwestern shifted romance brazilian portuguese
NLI
anglic english arabic standard arabic common turkic north azerbaijani gallo-italian venetian high german luxemburgish italian romance italian kurdish northern kurdish latvian latvian norwegian norwegian bokmål (m:written) sinitic classical-middle-modern sinitic (o:simplified) sotho-tswana (s.30) northern sotho southwestern shifted romance spanish
D Result Visualizations
D.1 Regional maps with aggregated Machine Translation scores
D.2 Task Specific Plot for Maximum Scores D.3 Dialectal Gap visualizations (zeroshot) E Evaluation results
cluster variety UD-code Zeroshot (mBERT) Zeroshot (XLM-R)
Finetune (mBERT) Finetune (XLM-R)
albanian albanian UD_Albanian-TSA 81.78 83.08 - -gheg albanian UD_Gheg-GPS 38.14 43.50 - -
anglic english UD_English-EWT 91.55 91.10 91.55 91.1 singlish singlish 69.88 68.55 82.36 11.0 african american vernacular english TwitterAAE 50.53 52.53 - -
arabic standard arabic UD_Arabic-PADT 50.87 55.91 88.47 2.48 south levantine arabic UD_South_Levantine_Arabic-
MADAR 49.94 45.75 - -
north african arabic UD_Maghrebi_Arabic_FrenchArabizi 34.33 28.04 5.46 54.7
eastern-western armenian western armenian UD_Western_Armenian-ArmTDP 54.63 59.71 88.84 89.29
eastern armenian UD_Armenian-ArmTDP 53.27 62.63 85.55 87.01
gallo-italian ligurian UD_Ligurian-GLT 50.22 43.78 13.06 9.1
gallo-rhaetian french UD_French-ParTUT 80.87 82.17 93.93 92.09 french (a:paris) UD_French-ParisStories 57.64 61.36 77.54 77.54 old french (842-ca. 1400) UD_Old_French-SRCMF 56.12 46.52 91.51 89.78
high german german UD_German-LIT 72.65 76.50 - -central alemannic (a:zh) UD_Swiss_German-UZH 36.77 34.70 - -
italian romance
italian UD_Italian-PUD 78.99 78.58 - - italian (r:formal, m:written, i:essay) UD_Italian-MarkIT 76.83 79.38 86.34 83.81 italian (r:casual, m:written, i:tweet) UD_Italian-PoSTWITA 61.88 62.13 85.82 86.32 continental southern italian UD_Neapolitan-RB 30.00 50.00 - -
komi komi-zyrian (m:spoken) UD_Komi_Zyrian-IKDP 26.89 32.14 - - komi-permyak UD_Komi_Permyak-UH 26.12 30.91 - - komi-zyrian (m:written) UD_Komi_Zyrian-Lattice 21.01 27.55 - -
norwegian norwegian bokmål (m:written) UD_Norwegian-Bokmaal 79.55 82.95 93.57 93.49 norwegian nynorsk (m:written) UD_Norwegian-Nynorsk 76.45 76.76 93.15 93.14 norwegian nynorsk (m:written, i:old) UD_Norwegian-NynorskLIA 56.58 56.08 78.39 8.25
H Cluster level Result Summaries with Demographic Utility and Standard Deviation report
I In-Context Learning Details
I.1 Prompts We adapt the prompts from Super-NaturalInstructions (Wang et al., 2022) for our in-context learning experiments.
Sentiment Analysis. For sentiment analysis we provide 4 few-shot examples in the prompt. The prompt template is given below:
In this task, you are given a piece of text. Your task is to classify the sentiment of the text based on its content.
Sentence: <Sentence Example 1> Label: <Label for Example 1, Positive, negative, neutral>
· · ·
Sentence: <Sentence Example k>
cluster variety dialect-code support precision (mBERT) precision (XLM-R) recall (mBERT) recall (XLM-R) F1 (mBERT) F1 (XLMR)
anglic english (a:uk) english:en-gb 249 98.10 89.57 83.13 71.59 90.00 79.58north american english english:en-us 349 93.27 88.14 83.38 82.09 88.05 85.01
arabic
aleppo arabic:ale 200 59.50 58.50 59.50 48.35 59.50 52.94 algerian arabic arabic:alg 272 79.00 65.50 58.09 62.68 66.95 64.06 arabian peninsula arabic (a:yemen) arabic:san 177 60.50 55.50 68.36 56.63 64.19 56.06 egyptian arabic (a:alx) arabic:alx 192 70.50 74.50 73.44 66.82 71.94 70.45 egyptian arabic (a:asw) arabic:asw 221 56.00 52.00 50.68 45.02 53.21 48.26 egyptian arabic (a:cai) arabic:cai 130 35.50 36.50 54.62 72.28 43.03 48.50 egyptian arabic (a:kha) arabic:kha 244 63.50 55.50 52.05 44.05 57.21 49.12 fez. meknes arabic:fes 196 60.00 59.50 61.22 56.40 60.61 57.91 gilit mesopotamian arabic arabic:bag 203 57.50 47.50 56.65 49.48 57.07 48.47 gulf arabic (a:doh) arabic:doh 205 50.00 43.50 48.78 45.55 49.38 44.50 gulf arabic (a:jed) arabic:jed 196 58.00 46.00 59.18 40.89 58.59 43.29 gulf arabic (a:mus) arabic:mus 178 38.50 49.50 43.26 42.67 40.74 45.83 gulf arabic (a:riy) arabic:riy 311 62.00 56.50 39.87 37.92 48.53 45.38 levantine arabic (a:northdam) arabic:dam 148 37.50 24.50 50.68 42.98 43.10 31.21 libyan arabic (a:ben) arabic:ben 238 56.50 47.50 47.48 52.78 51.60 50.00 north mesopotamian arabic (a:bas) arabic:bas 186 49.50 39.00 53.23 49.68 51.30 43.70 north mesopotamian arabic (a:mos) arabic:mos 188 71.50 70.00 76.06 69.31 73.71 69.65 rabat-casablanca arabic arabic:rab 153 50.00 40.00 65.36 60.61 56.66 48.19 sfax arabic:sfx 215 62.50 64.50 58.14 48.13 60.24 55.13 south levantine arabic (a:south-amm) arabic:amm 177 40.50 35.00 45.76 35.53 42.97 35.26 south levantine arabic (a:south-jer) arabic:jer 202 48.50 47.00 48.02 40.34 48.26 43.42 south levantine arabic (a:south-sal) arabic:sal 167 46.00 66.50 55.09 59.11 50.14 62.59 standard arabic arabic:msa 244 75.00 98.00 61.48 95.61 67.57 96.79 sunni beiruti arabic arabic:bei 192 58.00 52.50 60.42 68.18 59.18 59.32 tripolitanian arabic arabic:tri 201 66.00 60.00 65.67 60.30 65.84 60.15 tunisian arabic (a:tun) arabic:tun 164 52.50 37.00 64.02 56.49 57.69 44.71
greek cypriot greek (r:casual, m:written, i:other) greek:cg_other 81 74.14 84.48 53.09 56.32 61.87 67.59
cypriot greek (r:casual, m:written, i:twitter) greek:cg_twitter 36 51.11 44.44 63.89 68.97 56.79 54.05 modern greek (r:casual, m:written, i:twitter) greek:smg_twitter 94 89.83 100.00 56.38 53.15 69.28 69.41
high german
central alemannic (a:be) swissdialects:be
389 72.89 51.58 71.21 62.42 72.04 56.48
central alemannic (a:bs) swissdialects:bs
340 73.50 57.83 75.88 61.14 74.67 59.44
central alemannic (a:lu) swissdialects:lu
335 72.91 65.13 75.52 59.47 74.19 62.17
central alemannic (a:zh) swissdialects:zh
359 78.84 73.33 75.77 63.73 77.27 68.19
sinitic
mandarin chinese (a:mainland, o:simplified) mandarin_simplified:m 986 97.90 96.10 99.29 90.66 98.59 93.30 mandarin chinese (a:mainland, o:traditional, i:synthetic) mandarin_traditional:m977 96.80 92.10 99.08 95.74 97.93 93.88 mandarin chinese (a:taiwan, o:simplified) mandarin_simplified:t 1014 99.30 90.10 97.93 95.85 98.61 92.89 mandarin chinese (a:taiwan, o:traditional, i:synthetic) mandarin_traditional:t1023 99.10 95.90 96.87 92.39 97.97 94.11
dialect MBERT_Acc MBERT_F1 XLMR_Acc XLMR_F1 mistral7b lang-group variety
Label: <Label for Example k, Positive, negative, neutral>
Sentence: <Test Example Input> Label:
Extractive Question Answering. We provide 2 few-shot examples i.e. k = 2 due to the long-form nature of text for this task.
This task is about writing a correct answer for the reading comprehension task. Based on the information provided in a given passage, you should identify the shortest continuous text span from the passage that serves as an answer to the given question. Avoid answers that are incorrect or provides incomplete justification for the question.
Passage: <Passage for Example 1> Question: <Question for Example 1> Answer: <Answer for Example 1>
· · ·
Passage: <Passage for Example k> Question: <Question for Example k> Answer: <Answer for Example k>
Passage: <Passage for Test Example> Question: <Question for Test Example> Answer:

heading: Future Work
text: We believe that this underscores the need for building models robust to different language variations and future work should focus on this