Multi-VALUE: A Framework for Cross-Dialectal English NLP
Year: 2023

Abstract:
Dialect differences caused by regional, social, and economic factors cause performance discrepancies for many groups of language technology users. Inclusive and equitable language technology must critically be dialect invariant, meaning that performance remains constant over dialectal shifts. Current systems often fall short of this ideal since they are designed and tested on a single dialect: Standard American English (SAE). We introduce a suite of resources for evaluating and achieving English dialect invariance. The resource is called Multi-VALUE, a controllable rule-based translation system spanning 50 English dialects and 189 unique linguistic features. Multi-VALUE maps SAE to synthetic forms of each dialect. First, we use this system to stress tests question answering, machine translation, and semantic parsing. Stress tests reveal significant performance disparities for leading models on nonstandard dialects. Second, we use this system as a data augmentation technique to improve the dialect robustness of existing systems. Finally, we partner with native speakers of Chicano and Indian English to release new goldstandard variants of the popular CoQA task. To execute the transformation code, run model checkpoints, and download both synthetic and gold-standard dialectal benchmark datasets, see http://value-nlp.org/.

heading: 1 Introduction
text: “[Often, speakers] will not be hampered by the
lack of language technology in their local lan-
guage, but by the lack of support for their variety
of the contact language.”
— Steven Bird (2022)
Global contact languages like English will continue to have an outsized impact on commerce, economics, wellbeing, and equity worldwide. English, like any other language, is subject to variation
Equal contribution.
across time (Yang, 2000) and between speakers or speaker groups (Eckert, 2017; Holmes and Meyerhoff, 2008). Rather than focusing on social status or political power (Stewart, 1968; Chambers and Trudgill, 1998), linguists define dialects as descriptive sets of correlated features common across a group of speakers (Nerbonne, 2009). Current pretraining paradigms employ content filters that can exclude text in English dialects other than Standard American and British (Gururangan et al., 2022), which leads to performance gaps for other varieties. These discrepancies in Natural Language Processing (NLP) cause allocational harms for dialectal speakers in downstream applications (Bender et al., 2021), making dialect robustness a critical need for fair and inclusive language technology.
This disparity is clear in a growing body of empirical work on African American English (Ziems et al., 2022; Halevy et al., 2021; Blodgett et al., 2018; Jurgens et al., 2017; Kiritchenko and Mohammad, 2016). However, there does not yet exist a systematic exploration of robustness across multiple Englishes, nor of models’ ability to transfer knowledge between varieties with similar features, as in multi-lingual NLP. We need new tools to benchmark and achieve dialect robustness.
We introduce Multi-VALUE1 for English dialect robustness. Our feature-based approach leverages decades of field linguistics research to isolate grammatical constructions (Demszky et al., 2021) that vary in regional Englishes (Labov, 1972; Eckert, 1989; Hovy and Yang, 2021). We focus on varieties that (1) are mutually intelligible with Standard American English (SAE); (2) share vocabulary with SAE; and (3) differ from SAE with respect to morphology and syntax. The third criterion defines the critical axis of variation. The first two criteria ensure that our definition of model robustness aligns with the human ability to understand
1Multi-VALUE is a Multi-dialectal VernAcular Language Understanding Evaluation framework (value-nlp.org)
744
other varieties. For example, creoles have their own unique vocabularies and are not easily understood by speakers of other Englishes (Sebba, 1997); they are outside the scope of this study.
First, we provide a controllable (1) rule-based translation system for injecting up to 189 features into SAE text. This will allow researchers and practitioners to build synthetic training data plus on-demand dialect stress tests for nearly any task. We stress test leading models for three challenging tasks and find statistically significant performance gaps. Second, we provide reliable (2) gold standard benchmarks for the CoQA task in two widely-spoken varieties: Chicano and Indian English. We find that, by training models on synthetic data, we improve dialectal robustness. Third, we fine-tune and publish (3) dialect-robust models on the HuggingFace Hub (Wolf et al., 2020), which can be used directly in downstream applications. Figure 1 demonstrates the full project pipeline.
We recognize five advantages in the MultiVALUE approach. Our system is
(A) Interpretable: supports systematic perturbation analyses
(B) Flexible: customized to align with new and evolving dialects by adjusting the density of dialectal features, unlike fixed or static datasets.
(C) Scalable: allows users to mix and match tasks and dialects at scale without the need for costly human annotation.
(D) Responsible: vetted by native speakers to ensure gold standards and synthetic data are dependable for ongoing research.
(E) Generalizable: moves the field beyond single-dialect evaluation, which allows re-
searchers to draw more transferrable findings about cross-dialectal NLP performance.

heading: 2 Related Work
text: Dialect Disparity is an issue of equity and fairness (Hovy and Spruit, 2016; Gururangan et al., 2022; Halevy et al., 2021; Blodgett and O’Connor, 2017). There is mounting evidence of dialect disparity in NLP. Hate speech classifiers have known biases against African American English (Davidson et al., 2019; Mozafari et al., 2020; Rios, 2020; Sap et al., 2019; Zhou et al., 2021). Text from regions with a predominantly Black population are more likely to be classified as hate speech (Mozafari et al., 2020; Sap et al., 2019; Davidson et al., 2019). AAVE performance gaps have also been found across a wide range of core NLP tasks like NLI (Ziems et al., 2022), dependency parsing and POS tagging (Blodgett et al., 2018; Jørgensen et al., 2015), plus downstream applications (Lwowski and Rios, 2021). Still, there does not exist a systematic study on cross-dialectal model performance. We aim to fill this gap, expanding the VernAcular Language Understanding Evaluation (VALUE) framework of Ziems et al. (2022). Where VALUE established a uni-dialectal evaluation harness with 11 perturbation rules, Multi-VALUE now supports multi-dialectal evaluation with 189 different perturbations across 50 English dialects. Our empirical study on dialect disparity is also more expansive than prior work as we consider three separate domains: QA, MT, and semantic parsing.
Multilingual NLP studies how to learn common structures that transfer across languages. These strategies may also yield benefits in multi-dialectal settings. Massively multilingual models (Pires et al., 2019; Conneau et al., 2020; Liu et al., 2020;
Xue et al., 2021) exploit the commonalities between many languages at once, rather than merely achieving pairwise transfer (Lin et al., 2019). Additionally, benchmarking across multiple languages can reveal language discrepancies at the modeling level, even without language-specific feature engineering or training data (Bender, 2011; Ravfogel et al., 2018; Ahmad et al., 2019; Tsarfaty et al., 2020). Multi-VALUE aims to bring these advantages to the study of English dialects.

heading: 3 Multi-VALUE Perturbations
text: There is a clear need for dialect robustness (§2). The challenge is that language is subject to variation and change. This means speakers can contextually modulate the density of features in their grammar, and over time, speakers adopt different features. Shifting language can quickly antiquate training and testing data, and updating such resources can be costly and time-consuming.
In this section, we introduce the first stage of the Multi-VALUE pipeline. We automatically inject structural variation into SAE text using linguistic perturbation rules that alter syntax and morphology but preserve semantics. In this way, perturbations preserve labels. Unlike many black-box translation approaches (Krishna et al., 2020; Sun et al., 2022), label preservation will allow users to convert existing benchmarks directly into dialectal stress tests. Modular, independent perturbation functions give researchers the flexibility to isolate the effects of different features in different combinations.
What distinguishes our work from other syntactic data augmentation methods (Wu et al., 2022) is that our perturbations are grounded in formal language patterns. We operationalize the decades of linguistics research cataloged in the Electronic World Atlas of Varieties of English (eWAVE; Kortmann et al. 2020), a database with 235 features from 75 English varieties, as documented by 87 professional linguists in 175 peer-reviewed publications. eWAVE distinguishes dialects by their unique clusters of linguistic features and the relative pervasiveness of each feature.2 We define a dialect transformation as a sequential application of perturbation rules. Decisions to perturb the text follow the eWAVE heuristic probabilities: 100% for obligatory features; 60% for features neither
2For example, the give passive feature #153 is considered pervasive or obligatory in Colloquial Singapore English, while it is rarely observed in Philippine and Tristan da Cunha English, and it is never seen in any other dialect.
pervasive nor rare; 30% for rare features; 0% for features with no information or an attested absence.
For each rule, we condition the perturbation on morphosyntactic signals from POS tags, noun and verb inflection, and dependency relations using the spaCy 2.1.0 (Honnibal et al., 2020) and inflect 5.5.2 libraries. For the give passive pertubation above in Figure 2, we search for passive constructions with a past participle ROOT (VBN), an nsubjpass patient, and an agent. We construct the new phrase by inflecting the ROOT to its base (VB) form and moving it after the entire agentive noun phrase.
Following the eWAVE organizational scheme, we motivate and present our feature perturbations in 12 grammatical categories: (1) Pronouns, (2) Noun Phrases, (3) Tense and Aspect, (4) Mood, (5) Verb Morphology, (6) Negation, (7) Agreement, (8) Relativization, (9) Complementation, (10) Adverbial Subordination, (11) Adverbs and Prepositions, and finally (12) Discourse and Word Order. For a more detailed breakdown, see Appendix A.
Pronouns are critical for tasks like machine translation and summarization, which depend on coreference resolution (Sukthanker et al., 2020). Our pronoun perturbation rules account for linguistic structure and are not merely surface manipulations. For example, we condition on coreference for referential pronouns and on verb frames to identify benefactive datives. In total, we implement 39 of the 47 pronoun features from eWAVE.
Noun Phrases are the focus of fundamental NLP research in semantic role labeling and named entity recognition as well as downstream tasks like
sentiment analysis, information extraction, summarization, and question answering (Gildea and Jurafsky, 2000). Multi-VALUE has 31 rules that operate on NP constituents.
Tense and Aspect are two grammatical properties that have to do with time. Together, these categories are known to significantly challenge machine translation (Matusov, 2019; Koehn and Knowles, 2017). With 26 rules, Multi-VALUE introduces different kinds of inflections and auxiliary verbs to indicate when an action, event, or state occurred and how it extends over time.
Mood is important for applications in sentiment analysis and opinion mining, including the detection of biased language (Recasens et al., 2013) and framing strategies in political discourse (King and Morante, 2020; Demszky et al., 2019; Ziems and Yang, 2021). Misunderstandings of modality can also challenge NLU systems on tasks like natural language inference (Gong et al., 2018). There are three modal perturbations in Multi-VALUE.
Verb Morphology is expected to affect model understanding of verb frame semantics (Baker et al., 1998), which could impact performance on semantic role labeling, summarization, and machine translation, among other tasks. We implement 16 related perturbations that change verb suffixes, the forms of verb inflection, and the expression of semantic roles using specialized verbal phrases.
Negation is covered by 16 eWAVE features, 14 of which are implemented in Multi-VALUE. Problems with negation account for many of the failure cases in natural language inference (Hossain et al., 2020) and sentiment analysis (Barnes et al., 2021). Our perturbations introduce negative concord, invariant question tags, and new words for negation.
Agreement is a group of 11 rules which have to do with subject-verb agreement and the omission of copula and auxiliary be in different environments. Examples include the invariant present tense in He speak English (feature #170), and the existential dummy word in It’s some food in the fridge (feature #173). Nine of these 11 agreement features are attested in African American English (see Green 2002), which may be linked to the demonstrable performance disparities in AAVE dependency parsing (Blodgett et al., 2018), POS tagging (Jurgens et al., 2017), and NLU tasks (Ziems et al., 2022).
Relativization is a class of perturbations that operates on relativizers, which link relative clauses with their nouns. The purpose of a relative clause is to modify a noun phrase. It’s an important construction for NLU because it can contain a presupposition (Joshi and Weischedel, 1977). Our perturbation rules cover all 14 eWAVE features, operating both on individual relativizer words as well as sentence structure to move the relative clause and build correlative constructions, for example.
Complementation is a set of perturbations that turn dependent clauses into the subject or object of the sentence. Like relative clauses, complementation can contain presuppositions and implicatures (Potts, 2002), which are critical for natural language understanding. They can also convey a speaker’s degree of certainty (Couso and Naya, 2015), which correlates with biased language and framing strategies. We implement all 11 complementation features that are catalogued in eWAVE.
Adverbial Subordination is a set of perturbations that operate on independent clauses with a “conjunctive adverb.” Adverbial conjunctions can express causality (therefore), purpose (so that), sequence (then), contrast (however), comparison (similarly), and various forms of emphasis (indeed). We implement all 5 eWAVE features in this class.
Adverbs and Prepositions are represented by four rules, which can drop prepositions and replace adverbs with their adjectival forms.
Discourse and Word Order has two sides: two discourse features and 9 phrase-based perturbations that move entire constituents in a manner similar to constituency replacement (Sutiono and HahnPowell, 2022). These rules significantly alter the sentence structure, and in this way radically differ from prior token-level data augmentation techniques like synonym replacement (Wei and Zou, 2019). Phrasal movements include fronting and clefting, subject-auxiliary inversion, and a lack of inversion in questions. We also inject the word like to indicate focus or quotation.

heading: 4 Scope and Reliability of Multi-VALUE

heading: 4.1 Scope
text: Multi-VALUE’s scope is extensive. Out of the 235 features documented in eWAVE, Multi-VALUE covers 189, spanning all 50 recorded English dialects. On average, the feature space for any given
dialect is 86.6% implemented, and no dialect is less than 80% implemented (see Appendix A).

heading: 4.2 Recruiting Native Speakers for Validation
text: One key benefit of the Multi-VALUE approach is our ongoing partnership with native speakers to confirm that our theoretically-inspired rules generate plausible and grammatical text. Here, we validate our transformation rules using the linguistic acceptability judgments of native speakers for 10 English dialects.3 We recruit speakers from Amazon Mechanical Turk and screen them using a Dialect Assessment Survey.4 This qualification survey ensures that each speaker’s empirical language patterns align with the literature on the dialect that they had self-reported. At each turn, the speaker considers a sentence in the target dialect and provides a binary grammaticality judgment about that sentence. Sentences come from published linguistics journals. The survey is efficient5 as it implements binary search, dynamically selecting the feature that most evenly partitions the space of candidate dialects.

heading: 4.3 Validating the Multi-VALUE Pipeline
text: To validate our perturbation rules, we use the task from Ziems et al. (2022) in which each annotator
3Chicano (29 annotators), Colloquial American (13), Indian (11), Appalachian (4), Aboriginal (4), North of England (3), Ozark (3), Southeast American Enclave (3), Urban African American (1), and Black South African English (1).
4https://calebziems.com/resources/ value/dialect-quiz.html
5The survey uses an average of 9 questions, but the survey length will depend upon the user’s answers.
Seventy four rules have >95% accuracy, while sixteen have accuracy in [85,95) , and only two are <85% accurate, demonstrating the reliability of our approach.
is shown a pair of sentences: one in SAE, and the other as a dialect transformation: a copy of the first with perturbations corresponding to the target dialect. Annotators see only perturbations corresponding to their native dialect. Annotators mark portions of sentence 1 that were perturbed incorrectly in sentence 2. The interface is shown in in Figure 4 in the Appendix.
A group of 72 annotators evaluate a total of 19k sentence pairs, which were drawn from CoQA and other sources. We use CoQA sentences for our Gold Test Sets (§4.4), and for added syntactic diversity, we pull sentences from three nltk corpora: Reuters (Rose et al., 2002), Sentiment Analysis (Pang and Lee, 2004) and Movie Reviews (Pang and Lee, 2005). Three annotators evaluate each
transformation, marking any pre-highlighted spans where the transformation appeared ungrammatical. This gives us both transformation and perturbationlevel evaluations. The majority vote determines the accuracy of the perturbation rule.6 Perturbation accuracies are given in Table 1. Since there are 55 rules with perfect accuracy, and all perturbation rules achieve above 81%, researchers can feel confident in the linguistic plausibility of the Multi-VALUE transformation pipeline.

heading: 4.4 Gold Test Sets
text: While synthetic Multi-VALUE transformations will be useful for identifying weak points in a model’s performance, this does not ensure the model is ready for the real world. We urge practitioners to heavily test user-facing models with numerous in-domain tests. As a first step, we provide reliable gold standard CoQA datasets in Chicano English (ChcE) and Indian English (IndE). Out of 7,983 CoQA questions, our pipeline made changes to 1,726 ChcE questions (21.6%) and 6,825 IndE questions (85.4%). Human annotators considered only transformed questions and provided their own alternative phrasing for transformations they found ungrammatical. Alternatively, they could simply exclude the erroneous perturbations from the question. ChcE had a total transformation accuracy of 82.7% while IndE had 66.1%. The lower IndE accuracy is due to the higher density of features in this dialect. After rephrasing or removing errors, we were left with 1,498 dialect-transformed ChcE questions and 5,289 IndE questions. Together with any unperturbed questions, these gold questions constitute the gold test sets for evaluation in §6.1.

heading: 5 Using Multi-VALUE
text: With our feature rules written (§3) and handvalidated by native speakers (§4), we can use MultiVALUE to create synthetic data for training dialectrobust models and also for stress testing leading systems on dialect benchmarks. We specifically provide synthetic data for five English dialects: Appalachian (AppE), Chicano English (ChcE), Indian English (IndE), Colloquial Singapore English (CollSgE), and Urban African American English (UAAVE). Three of these dialects are based in the US, where annotators were most abundant for validation, and two are outside the US.
6Accuracy reliably measures strong consensus in the quality of our approach and, unlike kappa scores, it will not suffer from the prevalence problem (Eugenio and Glass, 2004).
To understand models’ ability to transfer knowledge between dialects, we also consider models trained on dialect A and evaluated on dialect B for each dialectal pair (A,B). We can further leverage the strengths of Multi-VALUE as a multidialectal augmentation tool by training on a synthetic pseudo-dialect that contains the union of all feature options (Multi). We hypothesize that models trained on multi-(pseudo)-dialectal data will benefit from robustness. While the Multi-VALUE approach could apply over any task with free-form text, we focus on three domains in particular: conversational question answering, semantic parsing, and machine translation. All three are user-facing tasks where language variation may hinder users’ access to information, resources, and/or the global economy (Blasi et al., 2022; Faisal et al., 2021).
Conversational Question Answering (CoQA; Reddy et al.2019) is a reading comprehension benchmark with 127k question-answer pairs and 8k passages in seven different genres and domains. We use it because it is a challenging task where dialect-induced errors can compound. The primary challenge is that questions are conversational: they contain coreference and pragmatic relations to prior questions. To transform the publicly available training and development sets, we perturb only questions. This is a natural information-retrieval setting: the user submits queries in a low-resource dialect while the underlying corpus is in SAE.
Semantic Parsing is the task of mapping natural language to formal language. This is a critical skill for dialogue systems, information retrieval, code generation, and other user-facing applications where dialect use is likely. We transform Spider (Yu et al., 2018), a widely-used text-to-SQL benchmark. Again, we transform only the natural language query, leaving both the database tables and the SQL query unchanged to simulate interaction with a dialect user. Unlike the question answering setting where knowledge is encoded in free-text SAE passages, the knowledge and query language in Spider are encoded in formal tables and structured language, both of which are dialect-free. Consequently, any performance discrepancies here will be due to a mismatch between the models’ training and testing data rather than a mismatch between the query dialect and that of the knowledge base.
Machine Translation is an interesting test case where challenges can arise from domain mismatch
(Koehn and Knowles, 2017) due to dialect. We especially anticipate challenges with verb morphology (§3), tense and aspect (§3), and pronouns (§3). We use a standard dataset, WMT19, and evaluate translation from each English Dialect to Chinese, German, Gujurati, and Russian. This simulates a user interacting with translation software using their native dialect.

heading: 6 Cross-Dialectal Stress Testing
text: Here we benchmark current models on dialect variants of the three tasks in §5. For each dataset, we use fixed hyperparameters without early stopping and report all performances on dialect variants of the evaluation data, since public test sets are not available for the original datasets. We use the base versions of BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) on dialect variants of the CoQA task, following the Rationale Tagging Multi-Task setup of Ju et al. (2019). For SPIDER, we evaluate BART and T5, since both are near the state of the art in semantic parsing (Xie et al., 2022). For Translation, we evaluate the NLLB Translation Model at two distilled scales: 615M and 1.3B (Costa-jussà et al., 2022). We report hyperparameters and further motivation for model selection in Appendix B.

heading: 6.1 Linking Natural and Synthetic Data
text: While natural data is the gold standard, it is difficult to scale to the number of dialects and tasks we can cover with synthetic data. Thus our broad evaluations are synthetic stress tests. Importantly, we first demonstrate the critical relationship between the gold and synthetic transformations using the gold evaluation sets from §4.4 and the synthetic training data from §5. Table 2 shows the gold standard
CoQA results, which should be compared to the synthetic CoQA results in Table 3.
The synthetic stress test results match the gold performance for Chicano English with only small deviations. The Indian English stress tests slightly overestimate the performance drop of an SAE model on Indian English (70.8% synthetic vs. 72.3% natural IndE with BERT; 76.1% vs. 77.7% with RoBERTa). This is expected, as the synthetic feature density may be higher than some annotators naturally use. Synthetic results are a lower bound on performance for a target dialect. For all treatments, the stress tests are directionally correct: treatments that improve performance on the stress test also improve results on the gold data.
Combined with speaker validation of the patterns themselves in §4.3, this shows that Multi-VALUE can be used to reliably measure the effects of modeling choices on dialectal performance.

heading: 6.2 Synthetic Stress Tests
text: We run 3 stress tests to understand worst-case performances on dialect-shifted data across a suite of models and tasks. Evaluation reveals large and statistically significant performance gaps across each task and across all dialects. This highlights, for the first time, the pervasiveness of English dialect disparity beyond any single dialect.
CoQA + Data Augmentation results are shown in Table 3. As predicted in §6.1, Chicano English (ChcE) does not produce a significant drop in performance (-0.7% BERT; -0.3% RoBERTa) since few of its pervasive features are distinct from SAE (the Manhattan distance between feature vectors for ChcE and Colloquial American English is 0.14, or only half the distance as between CollAmE and CollSgE, IndE, and UAAVE.) On the other hand, Singapore English, which is distant from SAE and therefore has many obligatory features, leads to the largest drop (-25.4% BERT; -18.9% RoBERTa). Appalachian, Indian, and Urban African American English each induce significant but smaller RoBERTa performance drops of -3.4%, -7.5%, and -6.7% respectively.
The data augmentation technique described in §5 successfully closes the dialectal performance gap. Across every dialect but Chicano English, we find that we can improve results by training on data that was transformed to the target dialect. Compared to standard RoBERTa, the RoBERTA model trained on Multi-dialectal data improves
average cross-dialectal performance by 2.7 points. However, multi-dialectal training causes a drop of 1.2 points on SAE, reminiscent of interference in multilingual models (Wang et al., 2019, 2020).
We performed a Qualitative Error Analysis on 30 errors for each transformed dialect. In each error, models trained on SAE flipped from a correct answer in SAE to an incorrect answer in one of the dialect-transformed COQA sets. Fully validated perturbations in tense, inflection, plural marking, phrasal order, and the deletion of pragmaticallyrecoverable pronouns, prepositions, and auxiliaries all lead to significant errors. As expected, these errors can cascade down the conversation, leading to model failure on later unperturbed questions as well. In some cases, erroneous answers still belong to the correct class, like flipping from yes to no in the presence of negative concord. Suprisingly, transformations also frequently cause the model to respond with an erroneous class, like giving a noun phrase or prepositional phrase to a yes/no question
under perturbations like clefting and the omission of auxiliary did, is, and wh-words.
Our analysis also suggests that the noticeably larger drop in performance on Singapore English might be largely due to the higher density of two perturbation types: preposition omissions (feature #198), and the one relativizer (feature #216). Future work can use perturbation analyses (Ziems et al., 2022) to quantitatively measure these sources of error.
Semantic Parsing Table 4 shows that SAE models significantly underperform on all dialectal stress tests, both in terms of Exact Match Accuracy and Execution Accuracy. For both BART and T5, the largest performance gaps appear when we test on the two non-American dialects, CollSgE and IndE (-15.3% and -12.3% exact match accuracy for T53b). The semantic parsing performance gaps here are as large as those in conversational question answering. This supports our claim that the discrepancies are caused by model mismatch, rather
than solely a mismatch between the dialect of the question and that of the knowledge base.
Machine Translation stress test results are shown in Table 5. Except for ChcE, performance drops significantly across all dialects for each language. Interestingly, the size of the average dialectal performance gap is higher when the target language is structurally more similar to English: the largest average drop is from English 7→German (-19.5% on 615M; -18.0% on 1.3B) and the smallest average drop is from English 7→Chinese (-10.6% on 615M; -11.0% on 1.3B). This result cannot be explained simply as a reflection of the model’s SAE translation performance. If it were, we might expect a smaller performance gap for Gujurati, a low-resource IndoEuropean language, since it has low SAE translation performance (21.7 SacreBLEU on 615M), but in fact, English 7→Gujurati has the second largest dialectal translation performance gap (-17.2% on 615M; -15.7% on 1.3B). Our explanation is that Gujurati has syntax that is more similar to English.
Despite both the 1.3B and 615M NLLB models being distilled from the same larger model, we see that the dialectal gap is smaller for German, Gujurati, and Russian. This suggests that model compression may affect low-resource dialects more heavily than SAE, similar to multi-lingual findings for low-resource languages (Ahia et al., 2021).

heading: 7 Conclusion
text: In this work, we introduced Multi-VALUE – a dialect robustness evaluation framework that is interpretable, flexible, scalable, responsible, and generalizable. The rule-based methods form a transparent syntactic translation system that can flexibly adjust to the shifting feature space of living dialects. Additionally, the transformation rules are reliably
sourced from over a decade of linguistics literature and vetted by native speakers. After showing that these transformations predict human-translated dialect benchmark performance, we used them to build dialect benchmarks and training data at scale, without the need for additional annotation efforts. By training and evaluating in a cross-dialectal manner, we demonstrated how Multi-VALUE can be used for more generalizable findings about model performance and dialect transferability.
Multi-VALUE can facilitate a wide range of NLP tasks and applications, such as measuring the relationships between dialect similarity and generalization performance, the scaling laws of dialect disparity, as well as inspiring algorithms on better dialect transfer. Overall, we anticipate that MultiVALUE will continue to support the development of more fair and equitable language technologies.

heading: 8 Limitations
text: Lexical variation is not our focus because it is not well-described by systematic, scalable, and generalizable rules. One can derive lexical distributions from data, but many low-resource dialects lack corpora on which to base these insights. This is an important problem for future research.
Multi-VALUE’s strength is its extensive coverage of English morphosyntacic patterns that have been documented in eWAVE by over 80 linguists. Such comprehensive resources are not available for other languages, but we encourage continued collaborations between computer scientists and linguists to build these resources for dialect-robust NLP systems across languages. As it stands, the current iteration of Multi-VALUE provides global value by serving a global contact language, English, and its 50 most documented varieties.
Despite the scope and precision of eWAVE for
English, its catalog ultimately derives from linguists’ oral interviews with native speakers, and here we can identify some additional limitations. First, the orthographic conventions that linguists use to encode spoken dialect may not always align with the speakers’ own writing conventions and usage. Second, our approach can only cover the variation that linguists observe frequently enough to document, and in canonical forms in which they are documented. This means we may not fully capture variation within each feature.
Finally, dialects should not be treated like deterministic speech patterns, but rather like a range of grammatical options or switches that may be turned on and off and adjusted for frequency in various social and personal contexts. Dialects do not always fit into nicely prescribed categories.

heading: 9 Ethical Considerations
text: This work makes use of human subjects for annotation. All procedures were subject to ethical review and were approved by the authors’ institution. Consent was gathered in accordance with the authors’ institution guidelines and annotators had access to a data use statement when giving consent.
The purpose of Multi-VALUE is to provide tools which enable researchers and practitioners to understand and mitigate dialectal bias in their models. We will release these tools responsibly, ensuring that users sign a Data Use Agreement that forbids the use of Multi-VALUE for deception, impersonation, mockery, discrimination, hate speech, targeted harassment and cultural appropriation.
In the agreement, researchers and practitioners will also acknowledge the Limitations of this work (§8), that Multi-VALUE may not fully or accurately represent the natural usage patterns of all sub-communities of speakers. Multi-VALUE is designed to be easily updatable and configurable such that it can be extended by and for specific sub-communities and updated as dialects evolve over time.

heading: Acknowledgements
text: We are thankful to the members of SALT Lab for their helpful feedback on the draft. Caleb Ziems is supported by the NSF Graduate Research Fellowship under Grant No. DGE-2039655. Part of this work was funded by an Amazon Faculty Research Award on Alexa Fairness in AI to DY.

heading: In Proceedings of the 2020 Conference on Empirical
text: Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.
Zhengxuan Wu, Isabel Papadimitriou, and Alex Tamkin. 2022. Oolong: Investigating what makes crosslingual transfer hard with controlled studies. ArXiv preprint, abs/2202.12312.
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. ArXiv preprint, abs/2201.05966.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics.
Charles D Yang. 2000. Internal and external forces in language change. Language variation and change, 12(3):231–250.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911–3921, Brussels, Belgium. Association for Computational Linguistics.
Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Yejin Choi, and Noah Smith. 2021. Challenges in automated debiasing for toxic language detection. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3143–3155, Online. Association for Computational Linguistics.
Caleb Ziems, Jiaao Chen, Camille Harris, Jessica Anderson, and Diyi Yang. 2022. VALUE: Understanding dialect disparity in NLU. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3701–3720, Dublin, Ireland. Association for Computational Linguistics.
Caleb Ziems and Diyi Yang. 2021. To protect and to serve? analyzing entity-centric framing of police violence. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 957–976, Punta Cana, Dominican Republic. Association for Computational Linguistics.
A Implementation Details
In Table 6, we give summary statistics for the number of features implemented for each of the 50 focus dialects, and the number of such features which were validated by native speakers. On average, the feature space for any given dialect is 86.6% implemented, and no dialect is less than 80% implemented. The reason we did not cover 100% of the eWAVE catalogue is that some features operate with information unavailable to us. For example, in SAE, aspect and mood may not be marked morphosyntactically; these features are outside the scope of current methods. Similarly, we are unable to inject distinct pronouns for groups of 2, 3, and 4+ people [#37], as group size information may not be contained in the focus utterance.
In Tables 7-18, we detail our Multi-VALUE implementations with an enumeration of our implemented dialects and features and examples of each. In the VAL ACC. column we give the validation accuracy (§4.3) as well as tags ChcE or IndE to indicate if the feature appears in the gold Chicano or Indian English CoQA dataset respectively.
A.1 Pronouns There are 47 pronoun features in eWAVE, and we cover 39 of them (83%). While simple regular expressions can cover some pronoun mappings, this is not always possible since English maps the same surface forms to different grammatical roles.7 We overcome this problem by conditioning rules on pronouns’ syntactic roles. We also condition on coreference for referential pronouns [29], and on verb frames to identify benefactive datives [9]. Furthermore, we swap the morphology of possession [20], change reflexive marking [11-16], swap animate pronouns for inanimate objects [1-2], and include additional elements like reduplication [40]. In summary, our pronoun perturbation rules account for linguistic structure and are not merely surface manipulations.
A.2 Noun Phrases Among our 31 noun phrase perturbations, we regularize or modify plural morphology [49] and comparison strategies [80], to drop or modify articles [60], construct phrases for possession [75], and
7For example, her is both the accusative in “give it to her“ and the noun modifier in “her cart,” while the masculine pronouns in “give it to him” and “his cart” differ. This problem was observed but not solved in the rule-based perturbation augmentation of Qian et al. (2022).
adjust the tree adjoining order to create adjective postfixes [87].
A.3 Tense and Aspect Tense and aspect perturbations include alternative inflections and auxiliaries to mark tense [117], including immediate vs. distant future [119], as well as perfect aspect [99].
A.4 Mood Multi-VALUE includes perturbations that inject double modals [121] and quasi-modals [126], change verb inflections under modal scope [123], and introduce auxiliaries to mark the sequential or irrealis mood [106].
A.5 Verb Morphology Verb morphology features include levelling certain finite and non-finite verb forms [130] adding suffixes for transitive verbs [143], and building serial verb phrases (Tallerman, 2019) to mark passive constructions [153], indirect objects [148], or the movement of direct objects [150].
A.6 Negation Multi-VALUE includes rules for building phrases with negative concord [154], and forms of negation with the negation words never, no, not, no more or ain’t, as well as special invariant tags for questions [166].
A.7 Agreement We implement the invariant present tense [170], as well as the existential dummy it [173].
A.8 Relativization These perturbations modify the form of the relativizer [186-190], as well as drop [193] or introduce new shadow pronouns [194], such as double relativizers [191] and phrasal forms [192]. Our perturbations also operate on the sentence structure by forming correlative constructions [196], deleting stranded prepositions [198], and moving the relative clause before the head noun [199].
A.9 Complementation These perturbations can change the form of the complementizer [200, 201], delete [208, 209] or introduce additional complementizer words [203, 204], build existential constructions from complementizer phrases [205, 206], and modify the verb in the non-finite clause complement [210].
A.10 Adverbial Subordination
Our perturbation rules introduce clause-final conjunctions [211, 212] and double conjuctions [214, 215], and remove the adverb in verb-chaining constructions [213], which together represent the five adverbial subordination features in eWAVE.
A.11 Adverbial Prepositions
In this section, we drop prepositions [216] and replace adverbs with their adjectival forms [220, 221]. We also include the word too as a qualifier [222].
A.12 Discourse and Word Order
In discourse, we insert the word like as a focus [234] or quotation marker [235]. Our phrase-based perturbations include fronting and clefting [223, 224], subject–auxiliary inversion in both negation phrases [226] and indirect questions [227], and a lack of inversion in certain questions [228, 229]. These rules significantly alter the sentence structure, and in this way radically differ from prior token-level data augmentation techniques like synonym replacement (Wei and Zou, 2019). Our approach here is most similar to constituency replacement (Sutiono and Hahn-Powell, 2022).

heading: B Models & Hyperparameters
text: CoQA We use the base versions of BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) on dialect variants of the CoQA task, following the Rationale Tagging Multi-Task setup of Ju et al. (2019) to adapt these models to the CoQA setup which includes Yes, No, and Unknown responses in addition to extractive answers. Each model was trained on an Nvidia GeForce RTX 2080 Ti for approximately 6 hours. For each model and dialect, we fine-tune using AdamW (Loshchilov and Hutter, 2019) for 2 epochs with a batch size of 16 and a learning rate 3e− 5.
Semantic Parsing. Following Xie et al. (2022), for T5-base we adopted the AdamW optimizer, while Adafactor was used for T5-3B and the two BART models. We used NVIDIA A100 to train these models with T5-3b, BART-large, T5-base, and BART-base using 8 GPUs for 52 hours, 4 GPUs for 32 hours, 4 GPUs for 4 hours, 4 GPU for 13 hours respectively. We set the learning rate at 5e-5 for T5 models and 1e-5 for BARTs. We fixed the batch size at 32 when fine-tuning T5-BASE and
BARTs. As for the extremely large T5-3B, we configured a batch size of 64 to speed up convergence and utilised DeepSpeed to save memory. Linear learning rate decay was used for all models.
Machine Translation. We evaluate the NLLB Translation Model at two distilled scales: 615M and 1.3B (Costa-jussà et al., 2022). Evaluation was done on an Nvidia GeForce RTX 2080 Ti and takes less than 10 minutes. The NLLB model is designed for many-to-many translation with low-resource language communities and is trained on a large corpus mined from the internet, rather than exclusively human aligned translations. We choose this model to give us an estimate of the performance of large scale translation products available to users.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 8

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 9

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 5

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 9

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 9
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The released datasets are derivatives of CoQA. Our Morphosyntactic patterns could not add additional information about individuals. The annotators were anonymized in accordance with the ethics review body of the authors’ institution.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 5
C 3 Did you run computational experiments? Section 6.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix C
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 6

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We used Bootstrap tests for significance for each run. We state that this is the bootstrap of a single run in the caption of each table.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Figures 4 and 5

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 4

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Section 9

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Section 9

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Section 4

heading: Future Work
text: distant future [119], as well as perfect aspect [99]. A. 4 Mood Multi-VALUE includes perturbations that inject double modals [121] and quasi-modals [126], change verb inflections under modal scope [123], and introduce auxiliaries to mark the sequential or irrealis mood [106]. A. 5 Verb Morphology Verb morphology features include levelling certain finite and non-finite verb forms [130] adding suffixes for transitive verbs [143], and building serial verb phrases (Tallerman, 2019) to mark passive constructions [153], indirect objects [148], or the movement of direct objects [150]. A. 6 Negation Multi-VALUE includes rules for building phrases with negative concord [154], and forms of negation with the negation words never, no, not, no more or ain’t, as well as special invariant tags for questions [166]. A. 7 Agreement We implement the invariant present tense [170], as well as the existential dummy it [173]. A. 8 Relativization These perturbations modify the form of the relativizer [186-190], as well as drop [193] or introduce new shadow pronouns [194], such as double relativizers [191] and phrasal forms [192]. Our perturbations also operate on the sentence structure by forming correlative constructions [196], deleting stranded prepositions [198], and moving the relative clause before the head noun [199]. A. 9 Complementation These perturbations can change the form of the complementizer [200, 201], delete [208, 209] or introduce additional complementizer words [203, 204], build existential constructions from complementizer phrases [205, 206], and modify the verb in the non-finite clause complement [210]. A. 10 Adverbial Subordination
Our perturbation rules introduce clause-final conjunctions [211, 212] and double conjuctions [214, 215], and remove the adverb in verb-chaining constructions [213], which together represent the five adverbial subordination features in eWAVE. A. 11 Adverbial Prepositions
In this section, we drop prepositions [216] and replace adverbs with their adjectival forms [220, 221]. We also include the word too as a qualifier [222]. A. 12 Discourse and Word Order
In discourse, we insert the word like as a focus [234] or quotation marker [235]. Our phrase-based perturbations include fronting and clefting [223, 224], subject–auxiliary inversion in both negation phrases [226] and indirect questions [227], and a lack of inversion in certain questions [228, 229]. These rules significantly alter the sentence structure, and in this way radically differ from prior token-level data augmentation techniques like synonym replacement (Wei and Zou, 2019). Our approach here is most similar to constituency replacement (Sutiono and Hahn-Powell, 2022)