DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rules
Authors: Yanchen Liu
William Barr Held
Diyi Yang
Venue: EMNLP 2023 Main (2023)
Keywords: Dialect Adaptation; Dialect Robustness; Linguistic Diversity; Fairness; Human-Centered NLP
TL;DR: To better address the flexibility of dialects, we propose DADA, a modular approach to imbue SAE-trained models with multi-dialectal robustness from a finer-grained perspective, by composing adapters which handle specific linguistic features.

Abstract: Existing large language models (LLMs) that mainly focus on Standard American English (SAE) often lead to significantly worse performance when being applied to other English dialects. While existing mitigations tackle discrepancies for individual target dialects, they assume access to high-accuracy dialect identification systems. The boundaries between dialects are inherently flexible, making it difficult to categorize language into discrete predefined categories. In this paper, we propose DADA (Dialect Adaptation via Dynamic Aggregation), a modular approach to imbue SAE-trained models with multi-dialectal robustness by composing adapters which handle specific linguistic features. The compositional architecture of DADA allows for both targeted adaptation to specific dialect variants and simultaneous adaptation to various dialects. We show that DADA is effective for both single task and instruction finetuned language models, offering an extensible and interpretable framework for adapting existing LLMs to different English dialects.

DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rules
Yanchen Liu
William Held
Diyi Yang
Harvard University,
Georgia Institute of Technology, Stanford University
yanchenliu@g. hww.arad.dddd..... wheld3@gatech. edu, diyiyecs. stanford. stt.yyesssttaffrd...om edu
Abstract
Dialect Adaptation via Dynamic Aggregation
Existing large language models (LLMs) that mainly focus Standard American English (SAE) often lead to significantly worse perfor- mance when being applied to other English dialects. While existing mitigations tackle dis- crepancies for individual target dialects, they assume access high-aceuracy dialect identi- fication systems. The boundaries between di- alects are inherently flexible, making difficult categorize language into discrete predefined categories. In this work, we propose DADA (Dialect Adaptation via Dynamic Aggregation), modular approach to imbue SAE-trained mod- els with multi-dialectal robustness by compos- ing adapters which handle specific linguistic features. The compositional architecture of DADA allows for both targeted adaptation to specific dialect variants and simultaneous adap- tation to various dialects. We show that DADA is effective for both single task and instruction finetuned language models, offering exten- sible and interpretable framework for adapting existing LLMs to different English dialects.'
drop, AAVE allows copula deletion and other auxiliary dropping.
Adapter Fusion
Linguistic Rule
Feature Adapter
Adapter Training
Feature Adapters
Frazen
Figure 1: DADA dynamically composes adapters that handle specific features of dialectal variation to adapt an SAE model to various dialects by leveraging their commonality. We train nearly 200 feature adapters capture the linguistic differences between SAE and its di- alect variants. These feature adapters can composed flexibly and arbitrarily target different dialects.
2018; 18; Blodgett et al., 2018). These challenges ex- tend automated speech recognition used by vir- tual assistants (Koenecke et al., 2020) and hate speech detection employed by online media plat- forms (Davidson et al., 2019; Sap et al., 2019; Rios, 2020; Mozafari et al., 2020; Halevy et al., 202 la; Zhou et al., 2021). Notably, even large language models not exempt from these limitations (Bom- masani et al., 2021: Solaiman and Dennison, 2021; Rae et al., 2022; Liang al., 2022). Such perfor- mance disparities raise ethical and moral concerns regarding the potential for racial disparities the seemingly expeditious development of language technologies (Hovy and Spruit, 2016; Blodgett and O' O'Connor, 2017; Halevy et al., 2021b).
1 Introduction
As Natural Language Processing (NLP) becomes even more impactful, the equitable distribution of its benefits becomes increasing concern. Specif- ically, NLP tooling is often trained and evaluated on dominant language variants, such as Standard American English (SAE). This results a signifi- cant decline the performance when these tools are applied to non-SAE dialects. Studies have re- vealed that SAE models tested on African American Vernauular English (AAVE) encounter difficulties in language identification (Jurgens et al., 2017a) as well as various other natural language tasks (Jør- gensen et al., 2016a; 6a; Kiritchenko and Mohammad,
Existing research mitigate this disparity has mainly focused dialectal adaptation targeting individual dialects of interest (Ziems al., 2022: Garcia and Firat, 2022; Ziems et al., 2023; Sun et al., 2022). This approach a powerful first step. but has key limitations of missing connectedness
'lll the code, synthetic datasets, and trained adapters in this work are available at https: //gi thub. com/SAL SALT-NLP/ DADA.

---

among dialects; for instance, English alone has 77 recognized variants that vary internally (Koenecke et at., 2020; Demszky et al., 2021). Prior adaptation methods also require highly accurate dialect identi- fication systems for real-world uses, leading to the development of separate systems for different di- alects. Such separate systems are not yet available for many dialects and related languages (Malmasi al., 2016; Aepli et al., 2023; Chakravarthi et al., 2021; Aepli et al., 2022). Alternative approaches train models using combination of various dialect variants in a multi-task learning manner (Caruana, 1997; Liu et al., 2019a). However, this approach re- quires training new models for dialectal NLP from scratch simultaneously with data from all desired dialects. This training process is prohibitive, es- pecially given the trend towards larger language models with costs upwards of millions of dollars?. Thus, there is a pressing need for an effective and extensible approach that can adapt existing models to the multi multi-dialectal setting.
neous adaptation multiple dialects. As a result of its compositional nature, DADA also makes it easy to re-use feature adapters regardless of dialect, speaker, or time variations in feature usage. The modular architecture ensures interpretability by en- abling analysis of the components responsible for the improvement in performance. To sum up, our work contributes the following:
We propose a modular approach DADA to adapt the standard SAE model to dialect vari- ants via dynamic aggregation of different linguistic features. (Sec. 3)
We train nearly 200 feature adapters, which can be flexibly composed to target different di- alects. Moreover, we demonstrate that DADA with all the trained feature adapters can con- sistently improve model performance across five English dialects. (Sec. 4)
DADA exhibits strong interpretability. Using AAVE as an example, we illustrate that DADA possesses the capability to detect the relevant linguistic features for a given input and sub- sequently activate the corresponding feature adapters. (Sec. 5)
Previous linguistic works have developed a col- lection of lexical and morphosyntactic features that describe the differences between SAE and vari- ous other English dialects (Kortmann et al., 2020; Ziems et al., 2023). Many dialects can be described by this common set of features or linguistic rules, with each dialect expressing subset of the feature space. In addition, dialects are not deterministic speech patterns but rather ranges of acceptable use of these features that speakers adjust based on so- cial contexts (Ziems al., 2023; Koenecke al., 2020; Demszky et al., 2021). As a result, dialects do not neatly fit into predefined categories.
We show that DADA improves dialectal robust- ness in task-agnostic instruction-tuned LLMs using FLAN-TS (Chung et al., 2022) (Sec. 6), which highlights the capability of DADA in learning task-agnostic features that can be ap- plied to newer general-purpor models.
Related Work
Dialect NLP research tends to focus primarily on dominant dialects represented in "textbook" gram- mar, such as Standard American English (SAE), over lower-resource dialects. The performance dis- parity in resulting models is pervasive (Koenecke et al., 2020; Davidson et al., 2019: Sap et al., 2019; Rios, 2020; Mozafari et al., 2020; Halevy et al., 2021a; Zhou et al., 2021: Ziems et al., 2022, 2023; Sun et al., 2022). The existence of such perfor- mance disparities raises ethical and moral concerns where NLP can potentially exacerbate the marginal- ization of the speakers of these dialects (Blodgett and O' O'Connor, 2017: Halevy al., 2021b). Lack- ing common dialectal evaluation, NLP can inforce existing power discrepancies (Hovy and Spruit, 2016; Bommasani et al., 2021). Existing works on English dialects have mainly focused on adapting models to individual dialects, such as
To this end, develop model which han- dles this reality by accommodating the diversity of English variants at a fine-grained level (linguistic features or linguistic rules). Concretely, we pro- pose Dialect Adaptation via Dynamic Aggregation (DADA): modular approach to adapt an estab- lished model trained on SAE to dialect variants by composing linguistic features. DADA captures and encapsulates each feature using adapters (Houlsby al., 2019) trained on individual feature rules. Feature adapters dynamically aggregate test time using adapter fusion (Pfeiffer et al., 2021), which enables the SAE model to flexibly adapt to dialects. The modular design of DADA enables targeted adaptation to specific dialect variants or simulta-
htttp:://anbdalabs.oo htttp::/landaaas..c//bbgg demystifying-gatt 3

---

Step1: Synthetic Datasets
Step2: Feature Adapter Training
Step3: Dynamic Aggregation
Linguistic Rule:
For each linguistic feature, we train feature adapter.
This particular instance of stinks, Hypothesis: It is a terrible situation.
Premise:
Adapter Fusion
Premisa: This particular instance stinks, Hypothesis: It a terrible situation.
nearly 200
Feature Adaipter
Tronsform
null reic
010100
verbal ing. sug_suffi
Norm
010100
TransForm
drop
Original Dataset
plural. pastaosed
Feed Forward
been, done
Linguistic Rule: been, done
Festure Adapter
Premise: She had spoken with no trace of foreign accent.
Add & Norm
Hypothesis: The woman had never spoken before.
Multi-eea Attention
Premisa: She had spoken with of foreign Hypothesis: The woman done never spoken before.
Feature Adapters Pool
Figure 2: The overall process of DADA. We first construct a synthetic dataset D. by applying each linguistic transformation rule T, E T. such as drop_aux: "AAVE allows copula deletion and other auxiliary dropping" ", to each individual training example within the original training dataset D (taking MNLI as example). Then we develop a feature adapter A; for each linguistic rule T; by training it on the corresponding synthetic dataset D;. We select the backbone model trained on the original SAE task datasets to enable the feature adapter capture linguistic differences while disregarding the task-specific information.
African American Vernacular English (AAVE) (Jør- gensen et al., 2016b; Blevins et al., 2016; Ziems et al., 2022). However, for real-world use, such systems would require another system to recognize these dialects so that the appropriate model can be used for each input. This task itself is challenging. with wfate-oftte of-the-art systems showing relatively low accuracy even when distinguishing high-resource dialects of English (Zampieri et al., 2023). Our work avoids this flaw by modeling multiple dialects at once using multidialcctal training data. Multidi- alectal training data has been shown to potentially increase robustness across all dialects in multiple prior works around data collection (Jurgens et al., 2017b) and augmentation (Ziems et al., 2023).
learning. Besides, several research efforts have been carried out to ensemble parameter-efficien components for multi-task learning and domain adaptation. Pfeiffer et al. (2021) propose to ag- gregate adapters trained on source tasks with an attentional layer to transfer acquired knowledge to target task. Asai al. (2022) introduce sim- ilar method, while using soft prompts instead of adapters. To improve robustness against dataset shorteuts, Liu et al. (2023) combine adapters with gating networks. Recently, Held et al. (2023) use adapter stacking for task-agnostic robustness tar- geting individual dialects. However, given the in- herent flexibility of dialects, there arises necessity for method to enhance multi-dialectaa robustness. Moreover, the existence of well-defined transfor- mation rules between dialects, which is uncommon in other domains, allows us to achieve finer-graine adaptation by aggregating linguistic features.
Garameter-Efficient Learning To efficiently transfer pretrained language models downstream tasks, several techniques (He et al., 2022) have been proposed to update only a small number of extra parameters while keeping most parameters frozen. For example, udapter tuning (Houlsby et al., 2019; Pfeiffer et al., 2020) adapts models using small bottleneck modules. Prefix tuning (Li and Liang, 2021) and prompt tuning (Lester al., 2021) prepend additional tunable prefix tokens to input hidden layers. Brown al. (2020); Liu al. (2022a,b) prompt language models for specific tasks without any parameter updates by in-context
Instruction Tuning Inspired by the success of prompting LLMs to adapt to various tasks (Brown et al., 2020), instruction tuning (Sanh et al., 2022; Wei et al., 2022; Ouyang et al., 2022) propose finetune language models on variety of tasks de- scribed through instructions to achieve the multi- task capability and to enhance zero-shot perfor- mance on unseen tasks. Since instruction tuning in- volves prompting the language models at the input

---

level, our approach is orthogonal to it and can be employed in conjunction enhance model's multi- task and multi-dialect abilities simultaneously.
3.2 Feature Adapter
Adapter tuning is known for its ability to adapt quickly to new tasks without catastrophic forgetting (Pfeiffer et al., 2021). Given these benefits and the inherent modularity of adapters,
3 DADA
develop
feature adapter A& for each of the N línguistic transformation rules T; E by training it on the corresponding synthetic dataset D, created in Sec. 3.1. We insert an adapter module after each feed- forward layer 4 of the backbone model M that has been trained on the original SAE task datasets, in order to target specific lexical and morphosyntactic differences between SAE and its dialect variants.
We introduce Dialect Adaptation via Dynamic Aggregation (DADA), modular method for adapt- ing an existing model trained on the Standard American English (SAE) to accommodate dialect variants at a finer- mner-grained level. Our proposed method deploys a dynamic aggregation of feature adapters, which characterize the divergence of lin- guistic features between SAE and its dialect vari- ants. Specifically, DADA involves the creation of synthetic training dataset for each individual fea- ture using transformation rules (Ziems et al., 2023). These synthetic datasets are used to train respective adapters for each linguistic feature. Finally, we compose these feature adapters to create a single model via an additional fusion layer.
3.3 Dynamic Aggregation
In Sec. 3.2, we described the process of training feature adapter A, for each linguistic transforma- tion rule to capture a specific type of linguistic dif- ference between SAE and its dialect variants. How- ever, it is common for multiple linguistic differ- ences to co-oceur within single sentence in real- world scenarios, thereby necessitating the model to simultaneously consider these distinct linguistic features to varying degrees.
3.1 Synthetic Datasets
Previous works have discerned a series of linguistic divergences and devised Mul ti-VALUE, a collection of lexical and morphosyntactic transformation rules between SAE and its 50 dialect variants (Ziems et al., 2022, 2023), including Appalachian English (AppE), Chicano English (ChcE), Colloquial Sin- gapore English (CollSgE), Indian English(IndE), and African American Vernacular English (AAVE), among others. For instance, a well-known linguis- tic feature of AAVE is the use of Negative Concord, where two negative morphemes are employed to convey a single negation (Martin and Wolfram, 2021). This transformation rule is sensitive to the verb-object dependency structure and neces- sitates an indefinite noun object (Green, 2002). As an example, the SAE sentence "He doesn't 't have camera" " could be rendered as "He don't 't have no camera" " in AAVE.
Therefore, we propose to dynamically aggre- gate the N trained feature adapters, denoted as A = [Ai, A2,...A }. into the SAE-trained back- bone model M via an additional fusion layer (Pfeif- fer et al., 2021). For this purpose, first construct super-synthetic training dataset D, employing the same approach as deseribed Sec. 3.1, but with all lexical and morphosyntactic transformation rules
T - [Ti, T..TT applied. After incorporating the N trained feature adapters A and a fusion layer into each layer of the backbone model, we train the fusion layers using the super-synthetic training dataset D, while keeping the feature adapters A and the backbone model M frozen.
Following Pfeiffer et al. (2021), we define the fusion layer as a composition of Key, Value and Query matrices at each layer of the transformer, denoted by K, V, and Q: respectively. The output of the feedforward layer h, is taken as the query vector and the output of each feature adapter A,, denoted a is used input to both the value and key transformations. With this attention-like fusion layer (Vaswani et al., 2017), the outputs of
Let (Ti, T2.......T TN) denote the set of trans- formation rules between SAE and its dialect variants. For each transformation rule T₃ E T, we can gener- ate a corresponding synthetic dataset D; by apply- ing the respective rule to each individual training example within the original training dataset D.
"There are different implementation variants for adapter tuning, and in our work. we follow Pfeiffer et al. (2020) by only inserting adapter modules after each feed-forward layer, while in some other works, adapters are inserted after multi- head attention layers as well.
*See Appendix A for detailed description of each trans- formation rule and the statistics of the corresponding synthetic training dataset.

---

all feature adapters are combined as followed:
For each transformation rule, we generate syn- thetic dataset by applying only that specific trans- formation rule to each example in the original MNLI training dataset. We only retain examples that differ from the original example, i.e., examples that have been transformed. Afterward, we train feature adapters using these synthetic datasets, described in Sec. 3.2. To aggregate trained fea- ture adapters into the backbone model, train large fusion layer for 5 epochs on a synthetic dataset that applies all dialectal variations simulta- neously, termed Multi, ail Additionally, we include a null adapter that remains as the identity function. This is kept for purely SAE inputs. In Appendix B, we report full hyperparameters along with the training details. We evaluate DADA on five English dialects: AppE, ChcE, CollSgE, IndE, AAVE and re- port the results in Table 1. Followed by Ziems al. (2022, 2023), we construct each dialect-specific MNLI dataset by utilizing a subset of transforma- tion rules that correspond to the respective dialect.
S/ so ftmaar (hf Q. Ki, {I,.., N),
ai. = ai. Vi, (I,..,,} N
A B/.....a/. Th.....a. .l,
O - s/ A/
where [] indicates the concatenation of vec- tors and of is the output of the 1-th fusion layer. Through training on the super-synthetic dataset D, a parameterized compositional mixture of feature adapters can be learned to identify the applied lin- guistic features for a given input and activate the corresponding feature adapters, thereby facilitating the effective addressing of linguistic discrepancies between SAE and its dialect variants.
To sum up, the compositionality of DADA en- ables targeted adaptation to specific dialect variants by selecting appropriate feature adapters. DADA uses modularity and compositionality to adapt a model to linguistic features present at test time since the pervasiveness of a feature can vary greatly based on its applicability and density (Demszky et al., 2021). This allows DADA to simultancously adapt various dialects by using a comprehensive set of feature adapters. We explore this property further in Sec. 5, using its interpretability to study individual feature adaptations utilized (see Sec. 5).
4.2 Results
Compared to the standard SAE model trained on the original MNLI dataset (SAE baseline), DADA demonstrates significant performance improve- ments across all evaluated dialects and even on SAE, with an average improvement of 2.16% More- over, DADA delivers comparable performance the strong baseline provided by individual further fine-tuning or adapter tuning on the SAE trained model with dialect-speciffe training data (Single Finetuning and Single Adapter). However, while these two approaches require a perfect dialect iden- tification system and D models, our approach uses single model and therefore does not rely on di- alect identification. This makes DADA a simpler and more realistic method for use when the target dialect distribution is unknown.
4 Multi-Dialeet Adaptation
this section, we demonstrate how DADA can enable the adaptation of an existing SAE model to multiple dialect variants, taking Multi-Genre Natural Language Inference (MNLI; Williams et al. (2018)) task as an example.
4.1 Experimental Setup and Evaluation
As deseribed in Sec. 3.2, we train feature adapter for each transformation rule from Ziems et al. (2023), the collection of lexical and morphosyn- tactic transformation rules between SAE and its di- alect variants. In total, we train nearly 200 feature adapters for downstream use. Here, we demon- strate that these features can be flexibly composed in DADA to improve model performance across multiple dialects simultaneously. We evaluate on five representative dialects: AppE, ChcE, CollSgE. IndE, AAVE. We employ RoBERTa Base (Liu et al., 2019b) that has been finetuned on the original SAE MNLI training dataset as the backbone model.
Compared to additional finetuning or adapter tuning Multi on standard SAE model (Multi Fine- tuning and Multi Adapter), DADA brings aver- age improvement of 0.32% and 0.47%, respectively. Morcover, tunes fewer parameters during a single training run compared to Multi ti Finetuning. We confirm that the empirically strong performance of DADA stems from the effective use of the correct individual feature adapters in Sec. 5.
Note that with DADA, in instances where a new dialect arises, the integration of this new dialect can be achieved through the identification of the

---

Dialect Adaptation Details
Evaluation Performance
Dialect
Total
Dialect Params.
Method
AppE
Col1SgE
IndE
AAVE
Nean
Data
Params.
SAE Baseline
125.W
0
83,70
84,91
80.62
82.00
83.95
82.77
86.57
Finetuning
"Muli
125M
126M
85.72
86 33
85.00
85.09
85, 85.30gam1
86.72
Adapter
Multi
126M
1.5.M
85.68
86.38
84,26
84.76
84.66
85, 85.15
86.73
DAAD
HuTEi
316M
19217
36.00 (siaa)
86. 70 (
84,59 (eam))
85.32 (nour)
85.50
85. 85.622na1
87.16 (1938)
DalA
Multi
315M
190M
86 14 (-244)
86.61 (4173)
84,25 (4000)
84,80
85.74 (41.9)
85, 83.49
87.00 (uars)
Finetoning
single
D 125M
D- 125MM
85.74
86.45
84.84
85.11
86.15
85.56
86.57
Adapter
single
D 126M
D7 1.5.
86.23
86.53
84.85
85.40
86.26
85.63
86.57
Table Multi-Dialect Adaptation results of SAE RoBERTa Base (Liu et al., 2019b) model for five English dialects: AppE, ChcE, CollSgE, IndE and AAVE. Due to submission limitations the GLUE benchmark, the results are reported on the validation mismatched split. The significance bars of the mean accuracies are determined through paired bootstrap test conducted on concatenation of each individual dialect dataset. is the number of target dialects for dialect adaptation. DADA outperforms the standard SAE baseline on five dialects and SAE (marked as (+)), with an averge of 2.16% improvement. Most importantly, DADA achieves comparahle performance and even surpasses (underlined) that of individual models.
linguistic transformation rules that govern the shift from SAE to the new dialect, followed by the train- ing of a feature adapter for each new transformation rule, and finally the retraining of the fusion layer. Furthermore, the potential for reusability of trained feature adapters is significant many dialects of- ten share common linguistic features.
DADA
Multi Finetuning
Mean Accuracy
85.30
null adapter For SAE inputs, every adapter has the potential to incorrectly change the model's orig- inal predictions. Therefore, we introduce a null adapter that which preserves the output of the orig- inal SAE model at each layer. We conduct an ab- lation study to evaluate the necessity of the mull adapter by comparing with models where it is ex- cluded. We denote this variant as DADAwlo null- As shown Table 1, excluding the null adapter results in a slight drop in performance for SAE.
25
50
75
100
125
150
175
Number of Feature Adapters
Figure 3: The mean accuracy of DADA shows an overall upward trend with the number of feature adapters.
5.1 Analyses Setup and Results
Number of feature adapters We analyze the av- erage performance of DADA on 5 evaluated English dialects, considering different numbers of feature adapters (k) ranging from I to all. For each k. we select the top k feature adapters with the best performance on the evaluation set. The results in Figure 3 demonstrate an overall increasing trend, indicating that each feature adapter incorporated in DADA can contribute to performance improvement, rather than relying solely on a select few.
Here, we use the AAVE dialect and MNLI task as an examplc. To adapt a standard MILI-finetuned RoBERTa Base model to target the AAVE dialect, we only need to take into account the 10 transfor- mation rules between SAE and AAVE proposed by Ziems et al. (2022). We select the corresponding feature adapters from collection and dynam- ically aggregate them by training fusion layer on AAVE training set for epochs with a learning rate 5e-5 and batch size 64. We evaluate the re- sulting model on the test split of the AAVE matched MNLI dataset as shown in Table 2. In comparison to the standard SAE model, DADA demonstrates 3. 33 and 1.4% improvement on AAVE and SAE, respectively. Morcover, DADA outperforms simple additional finetuning and adapter tuning of AAVE on SAE model by 0.4% and 0.5%, respectively, achieving the best performance of 86.6% on AAVE. These results demonstrate the superior performance of DADA over all other methods evaluated.
5 Interpretability
As discussed in Sec. 3, DADA can implicitly iden- tify the relevant linguistic features for a given in- put and activate the corresponding feature adapters. We validate this by investigating the correlation be- tween attention scores within each layer of DADA and the presence of linguistic features, to deter- mine whether the contributing feature adapters are relevant to the features present.

---

Dialect Adaptation Details
Test
Acc.
possible to combine the two approaches easily, In this section, we demonstrate how DADA can be employed to instruction-tura LLMs to improve their task-agnostic performance on dialects.
Backbone
Method
Data
AAVE
SAE
FT
SAE
83.4
86.2
pretrained
FT
SAE AAVE
84.8
85.6
FT
AAVE
86.2
87.4
SAE finetueed
Adapter DADA
AAVE
86.1
87.4
6.1
Experimental Setup
AAVE
86.6V
87.6 v
Using AAVE dialect as a case study, to demonstrate the effectiveness of our method in adapting the SAE model across multiple tasks, we include the tasks from the AAVE transformed version (Ziems et al., 2022) of the GLUE Benchmark (Wang al., 2018), including CoLA, MNLI, QNLI, QQP, SST- 2. and STS-B. For our backbone model, we employ a FLAN-T5 Base (Chung et al., 2022). Despite the original paper incorporates GLUE within the FLAN-T5's training data, we retrain the model on these specific tasks to enhance its suitability.
Table 2: AAVE Adaptation results of RoBERTa Base (Liu et al., 2019b), pretrained denotes the pretrained RoBERTa Base model, while SAL finetuned denotes the RoBERTa Base model that has been finetuned on the origi- nal SAE MNLI dataset. FT refers "fine-tuning", DADA demonstrates superior performance AAVE and SAE com- pared to baselines (marked V).
5.2 Correlation Analysis of Fusion Activation
We perform correlation analysis of these 10 fea- ture adapters for the linguistic features applied to the input data. For each transformation rule, we calculate the softmax activation for each adapter, for each input to which the specific linguistic fea- ture applies, and average over all activations within the same layer calculated over all instances in the AAVE MNLI test set. For better clarity, our final metrics takes the average utilization score of each feature adapter for the entire dataset and then sub- tracts the average utilization score associated with each transformation rule.
6.2 Multi-task training
For each transformation rule of AAVE dialect, we construct synthetic training data following the pro- cedure described in Sec. 3.1. However, in the case of multi-tase model, we construct a synthetic dataset for each task considered and utilize the mix- ture to train the corresponding feature adapter. Sub- sequently, we proceed to fuse these feature adapters by training fusion layer on the super-synthetic dataset Multi-Task AAVE, which is constructed by applying all the AAVE transformation rules. In Appendix D, we provide the templates used to train the FLAN-T5 model. In Appendix B, we report full hyperparameters along with the training details. We assess the performance DADA on AAVE trans- formed version of the GLUE Benchmark, and com- pare its results with the SAE baseline and Adapter Tuning with Multi-Task AAVE.
We plot the results for layers 1, 7, 11 in Figure 4. We found that significant correlations in utiliza- tion on the lower layers (0-3) are observed, while those the middle and higher layers found to be negligible. This is consistent with our intuition, as the primary distinction between SAE and its di- alect variants lies in their linguistic features (lexical and morphosyntactic). which are mainly captured by the lower layers of the model This analysis demonstrates that DADA has the capability to detect which linguistic features are relevant to the given input, and subsequently trigger the corresponding feature adapters. This highlights the interpretability of DADA with regard to the underlying factors that contribute to performance improvement.
6.3 Results
It is surprising note that although single Adapter Tuning with Multi-Task AAVE demonstrates im- provements in 4 out of 7 tasks, the overall average performance is even inferior to that of the SAE baseline. In contrast, DADA consistently outper- forms both the SAE baseline and Adapter Tuning acToss all evaluated tasks, resulting in an overall im- provement of 1 1.80/1.9 1.92 points on the AAVE GLUE benchmark, respectively. Specifically, on the rel- atively large datasets, DADA achieves a notable accuracy improvement of 2.0% on MNLI- mm, 0.9%/1. .2% on QNLI, and 1.5%/0.9% on QQP when compared to the SAE Baseline and Adapter Tuning, respectively. These results demonstrate
6 Multi-Task Dialect Adaptation
Recent LLMs such as FLAN-T5 (Chung et al., 2022) and InstructGPT (Ouyang et al., 2022) are instruction-tuner (Wei et al., 2022) for various tasks, which is orthogonal to method, making
'The linguistic fcature differences are subtle. Although the absolute values of the correlation cocfficients are not large, they are sufficient to indicate the existence of correlations.

---

-6
-2
Layer
Layer 3
Layer
Layer 11
bd
dl
da
nc
ni
ng
nr
bo
di
9
bo rD
da
Figure 4: Correlation Coefficients for AAVE adaptation between the feature adapters (column) and the inputs to which specific linguistic features (row) apply in layers 1, 3, 7, 11. (See Appendix for other layers.) Significant correlations can he observed within the lower layers (0-3), whereas there appear to be little to no correlations in the middle and higher layers. We use abbreviations for certain terms, such as "nc" for negative_concont
AAVE GLUE Performance
Method
CoLA
MNLI-m
MNLI-mm
QNLI
QQP
SST2
STS-B
Mean
SAH Baseline
21.1
83.2
82.6
90.6
87.1
92.1
86.4
77.59
Adapter Tuning
18.2
84.1
83.6
90.3
87.7
92.9
85.5
77.47
DADA
26.3 V
84.4 /
84.6 V
91.3 V
88.6 /
93.7 V
86.6
79.39
ChatGPT
26.33
59.60
63.00
82.00
72.40
95.00
80.15
68.35
ChatGPT + "Native Speaker"
18.24 L
56.00
57.20
73.60 |
67.60
91.60
48.91
59.02
Table 3: Multi-Task AAVE Adaptation results SAE FLAN-T5 Base (Chung et al., 2022) (Matthew's 's Corr. for CoLA; Pearson-Spearman Corr. for STS-B; Accuracy for all others). SAE Baseline denotes the FLAN-T5 Base model that has been finetuned using the original SAE mixture of task datasets. In comparison to both the SAE Baseline and Adapter Tuning with Multi-Task AAVE, DADA consistently exhibits superior performance across evaluated tasks (marked with v). Due budget constraints, the results of ChatGPT reported randomly sampled 500 example subset of the development sets. Prompt based interventions do not improve ChatGPT's performance on AAVE. On the contrary, can even result in further degraded performance (marked with J).
that our proposed approach, DADA, is not limited to single-task applications but can be easily scaled up accommodate various tasks for use with the increasingly common multi-task instruction-tumie -unning setup using in popular large-scale industrial sys- tems (Ouyang et al., 2022; OpenAI, 2023a; Anil et al., 2023; OpenAl, 2023b).
prompts (Odena et al., 2021: Shi et al., 2022), we incorporate a "Native Speaker" Prompt for Chat- GPT:
native [DIALECT_NAME] English speaker, and here is your task:" However, ChatGPT "Native Speaker" Prompt does not yield improved results and, in fact, per- forms even worse than the vanilla ChatGPT on all evaluated tasks. This highlights that dialect adap- tation is not solved with trivial prompt-based inter- ventions while being simultaneously less grounded in expert linguistic resources than DADA.
"You
In Table 3, we also present the results obtained with C (OpenAI, 2023a). Due to budget constraints,
only able to evaluate randomly sampled 500 examples from the development each task. However, even with this limited eval- uation, we can still gain insights that ChatGPT performs significantly worse than the SAE FLAN- T5 Base model on 5 out of 7 tasks. This empha- sizes that merely scaling up the model is inade- quate for tackling the challenge of dialect dispari- ties. These limitations persist even in the context of large language models. Inspired by "expert"
7 Conclusion
In this paper, we present Dialect Adaptation via Dynamic Aggregation (DADA), fine-grained and modular approach designed adapt an established model trained on Standard American English its dialect variants through the compositional ag- gregation of linguistic features. Our experiments demonstrate that the compositionality of DADA en-
"Engine: t-t-bo.. We conducted our ChatGPT ex- periments on May 16, 2023.

---

ables targeted adaptation to specific dialects, and demonstrated improved robustness across multiple evaluated dialects, including AppE, CheE, Co IndE, and AAVE. Our analysis also highlights the interpretability of DADA, as shown through its capability to identify relevant linguistic features for a given input and trigger the corresponding adapters. Furthermore, our experiments on FLAN- T5 illustrate the potential of applying DADA task- agnostic instruction-tume tuned large language models, showeasing its generalizability.
guistic features present in a given sentence. We learn parameterized compositional mixture of the dialectal features automatically, rather than rely- ing on static assumptions of density. This avoids what we view as a major issue: it is often difficult to determine the dialect of an input since dialects themselves vary depending context and speaker. The density of dialectal feature represents ap- proximate of density across the entire dialect, but may not be accurate specific speaker and con- text (Koenecke al., 2020). On the other hand, DADA can dynamically recognize the applicable dialectal features for a given input and activate the corresponding feature adapters. It remains to be explored in future work how the density of dialectal features, as captured in the linguistic literature, re- lates to the compositional mixture of these features as learned in the fusion layer of DADA.
Limitations
DADA involves the training for feature adapters and the fusion layer, which can make computa- tionally expensive, especially when dealing with a substantial number of linguistic rules. However, each training run only requires small number of parameters to be learned, and parallelization is feasible for feature adapter training. More impor- tantly, these trained feature adapters exhibit signif- icant reusability; the same set of feature adapters can be reused and employed for multiple dialects, though the fusion layer would need to be retrained for these dialects. However, if a use case does not involve significant reuses, this aspect may indeed remain limitation. We will release our trained feature adapters so that future studies will not need reincur the up-front training cost.
Ethics Statement
Previous linguistic works on dialectal features may not fully or accurately document the natural us- age patterns of all existing dialects in terms of their linguistic rules. As a result, we acknowledge that our proposed method DADA, which relies on these dialectal features from prior literature. may not take some undocumented features associated with dialects into account. However, by curating more dialectal features, our method can be easily extended to a broader range of dialects. Addition- ally, as DADA is task-agnostic when applied to instruction-tuned models (Sec 6), malicious indi- viduals might misuse it. To address this concern, we will release DADA with license that explicitly prohibits its usage for purposes of deception, im- personation, mockery, discrimination, hate speech, targeted harassment, and cultural uppropriation tar- geting dialect-speaking communities.
Furthermore, while DADA has the flexibility to utilize any linguistic rules, in our experiments, we specifically employed these linguistic transforma- tion rules that are well-established in prior work for English (Ziems et al., 2022, 2023). These rules were chosen because they were curated by lin- guists, validated by dialect speakers, and because English has many globally relevant dialects (Bird, 2022). However, evaluating DADA for other lan- guage groups and broader sets of lexical variation is key area for future work.
A Acknowledgement
While DADA mainly relies on Multi-VALUE (Ziems et al., 2022, 2023). they are orthogonal processes with different assumptions about dialect use. For each dialect, VALUE defines the density of a dialectal feature as the probability of the feature occurring when it is applicable, well as the probability of the corresponding perturbation to be used in converting sentence from SAE into that dialect. However, the actual prevalence of feature heavily depends also on applicability.
We would like to thank the anonymous review- ers and SALT lab members for their valuable feedback. This work was partially sponsored by the Defense Advanced Research Project Agency (DARPA) grant HR001 (2290103/HHB)) 1260656, and NSF grant IIS-2247357 and IIS-2308994.
References
Noëmi Aepli, Antonios Anastasopoulos, Gabriel Chifu, William Domingues, Fahim Faisal, Mihaela
DADA instead focuses on adapting to the lin-

---

Gaman, Radu Tudor lonescu, and Yves Scherrer. 2022. Findings of the VarDial evaluation campaign 2022. In Proceedings of the Ninth Workshop NLP for Similar Languages, Varieties and Dialects, pages 1-13, Gyeongju, Republic of Korea. Association for Computational Linguistics.
other high- resource scenarios. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 7817-7829, Dublin, Ireland. Association for Compu- tational Linguistics.
Terra Blevins, Robert Kwiatkowski, Jamie MacBeth, Kathleen McKeown, Desmond Patton, and Owen Rambow, 2016. Automatically processing tweets from gang-involved youth: Towards detecting loss and aggression. Proceedings of COLING 2016, the 26th International Conference on Computational Lin- guistics: Technical Papers. pages Japan. The COLING 2016 Organizing Committee.
Noëmi Aepli, Çagri Çoltekin, Rob van der Goot, T. Jauhiainen, Mourhaf Kazzaz, Nikola Ljubesic, Kai North, Barbara Plank, Yves Scherrer, and Marcos Zampieri. 2023. Findings of the vardial evaluation campaign 2023. In Workshop NI. for Similar Languages, Varieties and Dialects.
Osaka,
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang. Kathy Hellssern,,,, Gau- rav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay. Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury. Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev. Jacob Devlin, Mark Díaz. Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng. Vlad Fienber, Markus Freitag, Xavier Gar- cia, Sebastian Gehrmann, Lucas Gonzalez. Guy Gur- Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hur- witz, Michael Isard, Abe Ittycheriah, Matthew Jagiel- ski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Ben- jamin Lee, Eric Li. Music Li, Wei Li, YaGuang Li. Jian Li. Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nys- trom, Álicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley. Alex Castro Ros, Au- rko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone. Daniel Smilkov, David So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang. Zirui Wang, Tao Wang, John Wiet- ing, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. 2023. Palm technical report.
Su Lin Blodgett and Brendan T. O'Connor. 2017. Racial disparity in natural language processing: A case study of social media african-american english. ArXiv, abs/1707.00065
Su Lin Blodgett, Johnny Wei, and Brendan O' O'Connor. 2018. Twitter Universal Dependency parsing for African-American and mainstream American En- glish. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1415-1425, Melhourne, Australia. Association for Computational Linguistics.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson. S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji. Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Dem- szky, Chris Donahue, Moussa Doumbouya, Esin Dur- mus, Stefano Ermon, John Ftchemendy, Kawin Etha- yarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lau- ren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho. Jenny Hong. Kyle Hsu, Jing Huang. Thomas F. Ieard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khat- tab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li. Xuechen Li, Tengyu Ma, Ali Ma- lik, Christopher Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa. Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan. J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadim- itriou, Joon Sung Park, Chris Picch, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Re- ich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e. Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramer, Rose Wang. William Wang. Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Ya- sunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang. Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021.
Akari Asai, Mohammadreza Salehi, Matthew Pe- ters, and Hannaneh Hajishirzi. 2022. ATTEMPT: Parameter-efficien multi-task tuning via attentional mixtures of soft prompts. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 6655-6672. Abu Dhabi, United Arab Emirates. Association for Com- putational Linguisties.
Steven Bird. 2022. Local languages, third spaces, and

---

On the opportunities and risks of foundation models. ArXiv.
Xavier Garcia and Orhan Firat. 2022. Using natural language prompts for machine translation. arXiv preprint arXiv: 22X.iv200..11 11822.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark. Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Lisa J. Green. 2002. African American English: A Linguistic Introduction. Cambridge University Press.
Matan Halevy, Camille Harris, Amy Bruckman, Diyi Yang, and Ayanna Howard. 202 Mitigating racial biases in toxic language detection with an equity- based ensemble framework. In Equity and Ac- cess in Algorithms, Mechanisms, and Optimization, EAAMO '21, New York, NY, USA. Association for Computing Machinery.
Matan Halevy, Camille Harris, Amy Bruckman, Diyi Yang, and Ayanna M. Howard. 202 Mitigating racial biases in toxic language detection with an equity-based ensemble framework. Equity and Ac- cexs in Algorithms, Mechanisms, and Optimization.
Rich Caruana. 1997. Multitask learning. Mach. Leurn., 28(1):41-75.
Damiel Cer, Mona Diab, Eneko Agirre, lñigo Lopez- Gazpio, and Lucia Specia. 2017. SemEval-201 task 1; Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) pages 1-14, Vancouver, Canada, Association for Computational Linguisties.
Junxian Hc, Chunting Zhou, Xuezhe Ma. Taylor Berg- Kirkpatrick, and Graham Neubig, 2022. Towards unified view of parameter-efficieer transfer learning. In International Conference on Learning Representa- tions.
Will Held, Caleb Ziems, and Diyi Yang. 2023. Tada: Task-agnostic dialect adapters for english.
Bharathi Raja Chakravarthi, Gaman Mihaela, Radu Tu- dor Ionescu, Heidi Jauhiainen, Tommi Jauhiainen, Krister Lindén, Nikola Ljubešić, Niko Partanen, Ruba Priyadharshini, Christoph Purschke, Eswari Ra- jagopal, Yves Scherrer, and Marcos Zampieri. 2021. Findings of the VarDial evaluation campaign 2021. In Proceedings of the Eighth Workshop on NLP for Similar Languages, Varieties and Dialects, pages 1- 11, Kiyv, Ukraine. Association for Computational Linguistics.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NI.P. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani. Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Ximyun Chen, Aakanksha Chowdhery, Sharan Narang. Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instruction-finetume language mod- cls.
Dirk Hovy and Shannon L. Spruit. 2016. The social impact of natural language processing. In Proceed- ings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- pers), pages 591-598, Berlin, Germany. Association for Computational Linguistics.
Anna Jørgensen, Dirk Hovy, and Anders Søgaard. 6a. Learning POS tagger for AAVE-like lan- guage. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 1115-1120, San Diego California. Association for Computational Linguistics.
Thomas Davidson, Debasmita Bhattacharya, and Ing- mar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. In Proceedings of the Third Workshop on Abusive Language Online, pages 25-35, Florence, Italy. Association for Com- putational Linguistics.
Anna Jørgensen, Dirk Hovy, and Anders Søgaard. 2016b. Learning POS tagger for AAVE-like lan- guage. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 1115-1120, San Diego, California. Association for Computational Linguistics.
Dorottya Demszky, Devyani Sharma, Jonathan H Clark, Vinodkumar Prabhakaran, and Jacob Eisenstein. 2021. Learning to recognize dialect features. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 2315-2338.
David Jurgens, Yulia Tsvetkov, and Dan Jurafsky. 201 7a. Incorporating dialectal variability for socially equi- table language identification, In Proceedings of the

---

55th Annual Meeting of the Association for Compu- tational Linguistics ( (Volume 2: Short Papers), pages 51-57, Vancouver, Canada. Association for Compu- tational Linguistics.
of the 57th Annual Meeting of the Association for Computational Linguisties, pages 4487-4496, Flo- rence, Italy. Association for Computational Linguis- tics.
David Jurgens, Yulia Tsvetkov, and Dan Jurafsky. 2017b. Incorporating dialectal variability for socially equi- table language identification. In Proceedings of the 55th Annual Meeting of the Association for Compu- tational Linguisties ( (Volume 2: Short Papers). pages 51-57, Vancouver, Canada. Association for Compu- tational Linguistics.
Yanchen Liu, Timo Schick, and Hinrich Schütze. 2022b. Semantic-oriented unlabeled priming for large-scale language models.
Yanchen Liu, Jing Yan, Yan Chen, Jing I Liu, and Hua Wu. 2023. SMoA: Sparse mixture of adapters to mitigate multiple dataset biases.
Svetlana Kiritchenko and Saif Mohammad. 2018. Ex- amining gender and race bias in two hundred senti- ment analysis systems. In Proceedings of the Sev- enth Joint Conference on Lexical and Computational Semantics, pages 43-53, New Orleans, Louisiana. Association for Computational Linguisties.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining ap- proach.
Shervin Malmasi, Marcos Zampieri, Nikola Ljubešić, Preslav Nakov, Ahmed Ali, and Jörg Tiedemann. 2016. Discriminating between similar languages and Arabic dialect identification: A report on the third DSL shared task. In Proceedings of the Third Work- shop on NLP for Similar Languages, Varieties and Dialects ( (VarDial3), pages 1-14, Osaka, Japan. The COLING 2016 Organizing Committee.
Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey. Zion Mengesha, Connor Toups. John R. Rickford, Dan Jurafsky, and Sharad Goel. 2020. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences. 117(14):7684-760 7689.
Bernd Kortmann, Kerstin Lunkenheimer, and Katharina Ehret, editors. 2020. eWAVE.
Stefan Martin and Walt Wolfram. 2021. The sentence in african-america american vernacular english. In African- American English, pages 11-40. Routledge.
Brian Lester, Rami AI-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Marzich Mozafari, Reza Farahbakhsh, and Noël Crespi 2020. Hate speech detection and racial bias mitiga- tion in social media based on bert model. PLoS ONE, 15.
Augustus Odena, Charles Sutton, David Martin Do- han, Ellen Jiang, Henryk Michalewski, Jacob Austin, Maarten Paul Bosma, Maxwell Nye, Michael Terry, and Quoe V. Le. 2021. Program synthesis with large language models. n/a, page n/a, n/a, N/a.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582- 4597, Online. Association for Computational Lin- guistics.
OpenAl. 2023a. ChatGPT. https://openai.coo research/chatgpt.
OpenAL. 2023b. Gpt-4 technical report.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang. Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, et al. 2022. Holistic evaluation of language models. arXiv preprint ar XiX::2211.001111
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems.
Jaachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022a. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (Deel10 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100-1 14, Dublin, Ireland and Online. Associa- tion for Computational Linguistics.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. AdapterFusion: Non-destructive task composition for transfer learning. Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 487-503, Online, Association for Computational Lin- guistics.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jian- feng Gao. 2019a. Multi-task deep neural networks for natural language understanding. In Proceedings

---

Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić. Sebastian Ruder, Kyunghyun Cho, and Tryna Gurevych. 2020. AdapterHub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing: System Demonstrations, pages 46-54, Online. Association for Computational Linguistics.
Freda Shi. Mirac Suzgun, Markus Freitag, Xuezhi Wang. Suraj Srivats, Soroush Vosoughi, Hyung Won Chung. Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2022. Language models are multilingual chain-of-thought reasoners. arXiv preprint urXiv:2210.03057.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D. Manning, Andrew Ng. and Christopher Potts. 2013. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference Empiri- cal Methods in Natural Language Processing, pages 1631 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Jack W. Rac. Sebastian Borgeaud, Trevor Cai. Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young. Eliza Rutherford, Tom Hennigan, Ja- cob Menick, Albin Cassirer, Richard Powell, George van den Driessche. Lisa Anne Hendricks, Mari- beth Rauh, Po-Sen Huang, Amelia Glaese, Jo- hannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Anto- nia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Bud- den, Esme Sutherland, Karen Simonyan, Michela Pa- ganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, (can-Baptiss Lespiau. Maria Tsim- poukelli, Nikolai Grigorev, Doug Fritz, Thibault Sot- tiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autume, Yujia Li. Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy. Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, lason Gabriel, William Isaac. Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Ko- ray Kavukcuoglu, and Geoffrey Irving. 2022. Scaling language models: Methods, analysis & insights from training gopher.
Irene Solaiman and Christy Dennison. 2021, Process for adapting language models to society (PALMS) with values-targeted datasets. In Advances in Neural Information Processing Systems.
Jiao Sun, Thibault Sellam, Elizabeth Clark, Tu Vu, Tim- othy Dozat, Dan Garrette, Aditya Siddhant, Jacob Eisenstein, and Sebastian Gehrmann. 2022. Dialect- robust evaluation of generated text. arXiv preprint arXiv:2211. 11.0022..
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30, Curran Associates, Inc.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
Anthony Rios. 2020. Fuzze: Fuzzy faimness evaluation of offensive language classifiers on african-american english. Proceedings of the AAAI Conference on Artificial Intelligence, 34(01):881-889.
Alex Warstadt, Amanpreet Singh, and Samuel R Bow- man. 2018. Neural network acceptability judgments. arXiv preprint arXiv: 100:. 12471.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica. Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang. Tr- ishala Neeraj. Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Ieo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompted training enables zero- zero-shot task gener- alization. In International Conference on Learning Representations.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. 2022. Finetuned language mod- els are zero-shot learners. In International Confer- ence on Learning Representations.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias hate speech detection. Annual Meeting of the Association for Computational Linguistics.
Marcos Zampieri, Kai North, T. Jauhiainen, Mariano Felice, Neha Kumari, N.U. Nair, and Yash Mahesh Bangera. 2023. Language variety identification with true labels. ArXiv, abs/2303.01490.

---

Xuhui Zhou, Maarten Sap, Swabha Swayamdipta. Noah A. Smith, and Yejin Choi. 2021. Challenges in automated debiasing for toxic language detection. ArXiv,
A Tranformation Rules Details
Ziems al. (2022, 2023) developed a collection of lexical and morphosyntactic transformation rules that account for the differences in linguistic fea- tures between SAE and its various dialect variants. In our study, we build upon this work by train- ing transformation adapters for each rule in this collection. In their original paper, they present comprehensive overview of each transformation rule in Appendix B. In Tables 9-21, they provide detailed Multi- VALUE implementations, including an enumeration of the implemented dialects and features, accompanied by illustrative examples for each.
Caleb Ziems, Jiaao Chen, Camille Harris, Jessica An- derson, and Diyi Yang. 2022. VALUE: Understand- ing dialect disparity in NLU. In Proceedings of the 60th Annual Meeting of the Association for Compu- lational Linguistics (Volume 1: Long Papers), pages 3701-3720, Dublin, Ireland. Association for Compu- tational Linguistics.
Caleb Ziems, William Held, Jingfeng Yang. Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023. Multi- VALUE: A framework for cross-dialectal English NLP. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume I: Long Papers), Toronto, Canada. Association for Computational Linguistics.
Furthermore, we provide detailed statisties for the respective synthetic training datasets (for MNLI task) associated with each linguistic rule for the AAVE dialect in Table 4. While do present statistics for every linguistic feature for all dialects across all evaluated tasks, we release our code, all synthetic datasets and the trained adapters, to fur- ther improve the reproducibility.
Linguistic Rule
Size
Eval Acc
been_done dey it drop_aux
48,515
84.46
33,927
84.41
78.157
84.06
got
25,203
83.41
lexical
331,784
86. 1
negative_comm
49,529
84.41
negative_inversion inversioo
658
83.03
null genetive
50,122
84.11
null relcl
45,899
83.70
uninflect
124,447
84.64
Table 4: Linguistic Rules, Dataset Size, and Feature Adapter Accuracy for AAVE dialect (MNLI task).
B Training Details
Multi-Dialect Adaptation We train feature adapters for each transformation rule using syn- thetic datasets, as described in Sec. 3.2, with learn- ing rate 3e-4 and batch size 64 followed by Houlsby et al. (2019). To prevent significant performance differences among the trained feature adapters due to varying sizes of synthetic datasets, we fix the number of training steps to 10,000. For each fea- ture adapter, we choose the checkpoint with the highest accuracy on the validation matched split of a synthetic dataset that applies all dialectal varia- tions simultaneously. termed Multi. For dynamic

---

aggregation, we train large fusion layer for epochs Multi. We set the learning rate 2.5c-5 and the batch size to 64.
D FLAN-T5 Templates
We provide here the templates used in Sec. 6 train the FLAN-T5 model for each task. In the orig- inal paper by Chung et al. (2022), they defined 10 templates for each task and randomly applied them to each training example to enhance the model's robustness to varying instruction wordings. How- ever, in our study, our goal is to demonstrate the generalizability of our proposed method DADA to instruction-tunce models, rather than focusing improving the model's instruction-followii capa- bility. Therefore, for each task, fix the usage of the first template from the set of templates designed the original paper.
Multi-Task Dialect Adaptation For feature adapter training, we set the learning rate to le-3 and fix the number of training steps 50000. To fuse these feature adapters, train fusion layer for epochs using a learning rate of 8c-5.
Throughout the process of model training (in- cluding finetuning, adapter tuning, DADA training etc.), we consistently employ the standard train- ing objectives specific to the tasks, such as cross- entropy loss for classification tasks.
C Utilization Correlation Coefficients Plots
CoLA The Corpus Linguistic Acceptability (CoLA; Warstadt et al. (2018)) task is a widely used benchmark that focuses on grammatical ac- ceptability judgments. It aims to assess the ability of models determine whether given sentence is syntactically and semantically correct not. For the CoLA task, we adopt the following template:
In Sec. 5, we showcase the effectiveness of DADA in adapting the RoBERTa Base (Liu et al., 2019b) model that has been finetuned on the original SAE MNLI training dataset to AAVE. To demonstrate the interpretability of DADA, conduct an analysis of the utilization correlation among the aggregated 10 transformation adapters. We present utilization correlation coefficient plots for all layers Figure
Sentence: (sentence) Would linguist rate this sentence to be acceptable linguistically?
5 and
I think the answer is (answer)
MNLI The Multi-Genre Natural Language Infer- ence (MNLI; Williams et al. (2018)) is designed to assess the model's ability to comprehend and reason. MNLI involves determining the logical relationship entailment, contradiction, or neutral- ity between a given premise and corresponding hypothesis. For the MNLI task, we adopt the fol- lowing template:
Premise: (premise)
Hypothesis: (hypothesis)
Does the premise entail the hypothesis?
(answer)
QNLI The Question-answering Natural Lan- guage Inference (QNLI; (Wang et al., 2018)) task is a prominent benchmark that focuses on assessing

---

Layer
Layer
been done
6:15
4.73
4,31
4.33
-0.0 -2.4 0.29
4.2
0.14
-0.3
6.09
-0.9
-1.17
-2.43
14.27
-2.4
-0.63
40.2
0.15
-1.52
dey_t
drop_ aux
0.43
0.15
5.09
0.02
0.41
0.53 0.29
0.31
0.01
0.46
0.28
1.73
4.07
0.15
0.24
0.1
0.05
0.19
0.17
got
2.93
+0.35
4.81
4.26
6.72
0.18
4.37
2.24
+0.62
-0.35
-0.16
6.43
4.33
10.12
-0.45
+3.37
Texical
negalive_er concori soncord
-0.37
-0.18
4..9
4.53
0.91
4.46
0.76
-3.R1
<0.16
-0.00
.0.7
0.21
-1.77
-5.57
-13
3.01
143
4.85
0.39
-3.41
rnneriiv_inversion
0.25
3.98
4:57
1.36
1.35
null geleseeeevve
0.13
-0.26
40.0
5.06
.0.2 40.62
4.04
-0.1
0.24
-0.69
0.31
42.30
-2.23
-0.37
0.34
-0.41
ruld
0:35
0.26
0.47
3.18
0.53
0.09
0.35
4.00
-0188
0.61
0.21
0.29
0.43
0.77
15.00
0.25
0.13
0.84
uninflect
Layer
Layer
been done
5:19
0.45
1.18
0.58
1.2
0.45 0.45
0.1
0.25
0.53
5.32
0.65
1.31
0.5
0.96
0.4
0.36
0.45
0.08
0.8
dey_R
drop_ aux
.09
40.00
1.29
4.13
9.22
0.07
0.58
0.05
0.81
-0.3
2.14
-2.05
4.61
42.5
0.21
4.26
6.12
-3.49
got
.0.79
-0.52
4.81
1.61 1.42 4.51 0.14
-2.81
4.52
-0.9
-122
6.31
4.44
.0.58
-0.53
4.59
-1.33
lexical
0.09
0.06
0.05
0.85
0.05
negative concord
rocative_cooncrr
0.84
.0.00
4..7
5.16
0.13
4.07
0.04
-2.28
0.17
-0.55
-0.5
-0.22
-1.26
-2.25
409
4:95
-0.34
40.45
40.33
-0.91
nega negative_inversion
null_genetive
0.23
0.26
0.13
0.68
0.25
0.29
1.48 0.25
0.14
0.2
0.39
0.48
0.02
0.36
0.49
0.12
0.15
0.52
0.04
reld
8.5
.033
-0.3
0.27
0.33
-0.32
0.21
4.75
0.01 -0.2
unintlect
0.14
5.27
0.55
-02 0.21
0.24
0.36
4.2
0.15
-.14
Layer
Layer
been_done
2.81 .63
4.28
6.75
4.00 0.37
-0.33
-0.1
-0.05 0.85
-2.08
40.9 0.1 -0.6
0.28
0.61
doy_F
drop_rn
0.82
4.17
2.96
4.17 0.55 0.64 02 4.31 0.14
0.15
-533
0.29
3.11
-5.27 0.77 0.63 0.29
0.28
0.15
0.42
goc .06
-0.21
0.27
0.22
0.00
00.11
-0.12
40.35
18.02
0.11
lexkal .0.11
40.05
44.04
-0.11
0.02
40.05
-0.02
-0.03
-0.32
+2.00
4,50
0.05 00002
0.18
regative_cooncrr concord
0.55
0.49
1.67
0.47
1.26
5:38
0.43 0.46
0.3
0.97
0.38
0.41
1.45 0.38
1.32
5.71
0.36 0.41
0.48
0.71
Inversion
mull genetivee
.02 -0.11
6.71 .5.2 ..M -0.5 0.12 5.14 ..11 0.69
.0.13
-0.13 0.16
-5.17 404 10.35 -0.16 4.12
0.68
6.61
reld
0.21
0.08 0.22
0.14
0.25 0.17
unintlect
<0.14
<0.11
.4.11
.0.32
40.1
4.12
2.04
0.19
0.57
-2.07
-0.08 -0.25
-2.09
4..7
.2.00
-2.1
40.08
0.50
0.51
meiL been done
miil dey,
mri Jevical
mni pogedive concerd
miil pogative Jnversion
mmil genetive
mmll mull reld
mnl Uninleec
moil boen done
mnl gat
mrli Jostral
mell pega connord
mnlu negetive Inversion
genetive
mill mull pald
mnl smintlect
mes
mail
Figure 5: Correlation Coefficients between the transformation adapters (column) and the inputs to which specific transformation rules (row) apply in layers 0-5.
the ability of models to perform sentence-level se- mantic matching and reasoning. In this task, given a question and corresponding sentence, the objec-
tive is to determine whether the sentence contains the answer to the question, considering both lin- guistic and logical entailment. For the QNLI task,

---

Layer
Layer
been done
1.5
<4.18
4.37
4.00
-0.3
-0.1 00.6 -.11
0.21
0.28
0.1
-049
-0.24
-2.00
-0.11
-0.07
-002
0.23
0.43 0.03
dey_t
drop_ aux
0.33
0.04
1.3
0.01
0.15
933
0.05
4.82
0.05
0.29
0.15
0.03
0.22
0.53
0.03
0.12
0.09
0.07
got
40.5
+0.05
4.2
0.08
4.57
4.00
-0.2
-0.35
+2.02
0.01
0.01
0.27
Texical
negalive_eo
senccone_conoor soncord
40.3
-4.18
4.57
-.4
0.08
4.5
0.35
4.4
-0.25
-0.6
.0.16
5.04
-0.24
0.1
-..06 0.01
-0.0
4.15
0.33
0.17
negative rnveriiv_inversion
0.27
3:11
0.05
null segnettntve
.01
-0.
40.7 5.08
-0.31
-0.27
-01
0.48
-0.12
-0.1
-0.31
-2.07 .2.3 -2.02 40.03
0.45
0.11
ruld
0.15
46.15
0.22
8.03
0.44
0.08
0.36
4.31
0.46
.0.19
-0.08
-0.11
-0.07
0.39
00.00
00.07
0.02
0.02
0.18
uninflect
Layer
Layer
boen done
0.67
0.07
0.57 0.08 0.21 0.07 0.08 0.05
0.04
0.56
0.07
0.04 0.03
8.0L
0.01
0.02
0.11
0.01
6.08
0.12
dey_
drop aux
0.24
0.01
0.52
4.00
0.22
2203
-003
40.23
-2.01
4.05
0.02
0.00
4002
4.03
0.14
got
00.05
5.13
0.14
40.1
2.09
0.45
-0.03
-0.01
-0.01
00.01
0.09
4.03
4.81
0.01
lexical
0.01
0.03
0.02
0.01
0.04
recative_cooncrr concord 40.22 -0.12
4.12
1.73 0.34 40.12
0.02 5.18
-0.11
-0.2
0.26
0.02 0.03
0.03
0.02 0.04
0.11 0.02
0.00
-0.23
nega segalivv__nvvee
null_genetive 0.07 0.87
0.67
6.07
0.2 0.07 0.05 0.00 0.06
0.77
0.08
0.05
0.05 0.01 0.04 0.04
0.05
0.01
0.04
0.26
ruld .0.07 48.87
4..0
0.05
-3.02
-0.05
+2.01
40.51 +0.03
0.12
unintlect 0.06 -0.56 0.06 0.17 0.00 0.03
.0.05
-0.03
-5.01 4.52 00.02
0.14
Layer
Layer
been_done .0.57 -5.12 4.13
5.79 8.02 .012
02
..12
0.41
-0.01
-0.05
0.49
.0.44
-3.09 0.03
4.15
doy_F
drop_aus
0.03
02 5.18 0.69
0.25
0.35
0.52
0.09
-0.22
0.45 4,01 4.1 0..0 0.26
0.06
0.00
got 0.06 -0.00
0.03
0.09 0.11 0.0L
0.14
0,09
0.00
lexkal ..41 +0.02 4.06 5.06 0.01 4.02
0.25
-0.02
-0.51
0.33
4,00 -2.00 0.01
-0.00
negative_concee concord
0.14 6.00 0.84 8.5L 0.12 0.14 0.54
4.09
6.1
0.43
8.1
0.08
0.79
8.07
0.66
2.05
0.49
0.08 0.04
mull genetive
0.62 -0.11
.22 5.17 8.12 4.68
6.1 4.M
.0.09
6.15
.0.0
-0.02
+0.31
0.25
4.2
0.11 4.11 4.00 .000
ruld
0.06
0.05
0.08
unintlect .0.22 .0.00
5.18
0.08
40.08
4.05
2.01
.0.08
0.18
0.34
40.03
-2.07
0.02
4.06
-2.01
meiL been done
dey.
mri Jevical
mni pogetive concerd
mii pogative Jnversion
mmil genelive
mmlt mull reld
mnl Uninleec
mail boen done
mnl got
mrij Jostral
mull pega waweteee_ooooooo connord
mnil negative Inversion
genetive
mill mull pldd
mnl smintlect
med
Figure 6: Correlation Coefficients between the transformation adapters (column) and the inputs to which specific transformation rules (row) apply in layers 6-11.
adopt the following template:
Does the sentence (sentence) answer the question (question)
(answer)

---

QQP The Quora Question Pairs? (QQP) task is widely recognized benchmark that focuses on question sentence similarity. The task involves determining whether a pair of questions asked on the Quora platform semantically equivalent or not. For the QQP task, we adopt the following template:
[question1]
{question2]
Would you say that these questions are the same? [answer]
SST-2 The SST-2 (Stanford Sentiment Treebank; Socher et al. (2013)) task is a widely used bench- mark for sentiment analysis. It involves classifying the sentiment of given sentence as either posi- tive or negative. For the SST-2 task, we adopt the following template:
Review:
(sentence)
Is this movie review sentence negative or posi- tive?
The answer is: (answer)
The Semantic Textual Similarity Bench- mark (STS-B; Cer et al. (2017)) task is widely recognized benchmark that evaluates the ability of models to assess the semantic similarity between pairs of sentences. The task involves assigning similarity score pairs of sentences based on their semantic equivalence. For the STS-B task, we adopt the following template:
STS-B
[sentencel] [sentence2]
Rate the textual similarity of these two sentences scale from 0 to 5, where 0 is "no meaning overlap" and 5 "means the same thing".
(answer)


Reviews:
[{"review_id": "AtbiMdntjw", "reviewer": "Reviewer_ybzc", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review of our paper. We extend our gratitude for your insightful evaluation of our paper and your recognition of **its distinctions from the prevailing \"macro-language\" perspective in NLP**, underscoring the uniqueness of our approach DADA. Thank you for identifying the unique perspective and contributions this work takes!\n\nIn addition, we are grateful for your suggestion to include some more **qualitative analysis**. In Fig. 4, we already plotted the correlation coefficients between the feature adapter activations and the linguistic features for AAVE dialect. But as you suggested, it would be interesting to extend the analysis to the multi-dialect setting. We have supplemented this analysis, where we show the activations for each feature adapter across different dialects, and analyze their correlation with the pervasiveness of each linguistic feature within different dialects. Here are some results:\n| Dialect   | Pearson Coefficient |\n|-----------|----------------------|\n| AppE      | 0.1968               |\n| ChcE      | 0.2016               |\n| CollSgE   | 0.2724               |\n| IndE      | 0.2610               |\n| AAVE      | 0.2148               |\nAs we mentioned in the paper, because linguistic feature differences are very subtle, and, in this analysis, we incorporated all of the nearly 200 linguistic features that we discuss in the paper. Although the correlation coefficient values are not large, they are sufficient to provide evidence to demonstrate that DADA can indeed identify linguistic features within the input text, activate the corresponding feature adapters, and thereby allow it to accommodate different dialects. Additionally, we will incorporate supplementary results and plots into the revised version of the paper.\n\nAnd finally, we will meticulously review and correct any existing typos to ensure our bibliography's correctness. And we are committed to releasing all the code of our paper, as well as making all of our trained adapters available as assets for the entire community to use, which also stands as a main contribution of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I'd like this paper to be accepted because it goes beyond the current \"macro-language\" perspective in NLP and delves into dialects and non-standard varieties. Relevant studies (also cited in the paper) focus on the transformation of linguistic rules and various approaches using adaptors. Although the current study is very much in line with the previous work, it also generates synthetic data and dynamically aggregates the feature adapters which enhances the model’s multitasking and multi-dialect abilities.", "reasons_to_reject": "Although the study is thorough and supports all its claims, it would have been interesting to have some qualitative analysis on the dialects and the linguistic features that make them different in the fine-tuned models.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on dialectal robustness and puts forward an approach to handle specific linguistic features of Standard American English dialects by composing adapters. The methodology is compelling and the findings are very interesting.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The bibliography is to be checked for uncapitalized proper names (like english) and incomplete references (like like 923)."}, "extra_text": {}}, {"review_id": "5GH8m0TfYI", "reviewer": "Reviewer_KiYC", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review of our paper. We greatly appreciate your interest in applying our DADA to languages other than English. DADA's potential extends far beyond English and there is great potential in utilizing DADA for other languages. We think broad support of linguistic variation like dialects is key to supporting global and diverse speaker populations, making methods like DADA extremely valuable.\n\nAdditionally, it's important to highlight that we have indicated in our paper the sources of all the linguistic features we are currently considering - these features are curated by linguists (eWAVE: https://ewave-atlas.org/) and are implemented as transformation rules by Multi-Value [1]. \n\nIt is notable that the evolution of languages/dialects is a slow process, hence the infrequent emergence of new linguistic rules. And the curation of these rules is a one-time effort; once curated, they can be consistently employed. Furthermore, we should emphasize that linguistic rule curation is much more efficient than constructing distinct datasets for every dialect variant.\n\nHowever, we recognize that this is still a complex and costly process. Therefore, we are currently engaged in a new project where we are leveraging LLMs to assist in this process. We're exploring ways in which LLMs can aid linguists or autonomously identify new linguistic rules from text corpora, aiming to streamline and enhance this process.\n\nFurthermore, we are committed to releasing all the code of our paper, as well as making all of our trained adapters available as assets for the entire community to use, which also stands as a main contribution of our work.\n\n[1] Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023. Multi-VALUE: A Framework for Cross-Dialectal English NLP. ACL 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents an interesting approach to dialect adaptation, and the results of the mentioned experiments are promising. I would be curious to see this approach applied to other languages than English. ", "reasons_to_reject": "The dynamic aggregation process relies on having a set of lexical and morphosyntactic rules, but there was no mention of how these rules are created. Are they manually curated or automatically generated from existing data, since if this is a manual process it will present an expensive entry threshold.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents an approach to dialect adaptation of a number of English variants. It demonstrates that through training features adapters they can improve the performance of language models. The paper also claims that introducing new dialects can be achieved by identifying the linguistic transformation rules from SAE to the new dialect. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "F9eosjong2", "reviewer": "Reviewer_XXyA", "summary": "", "questions": "A: Can we be certain that the feature adapters adequately learn to adapt to the linguistic features they should? What about cases when there are very few training instances due to the rarity of a certain features? What about when the difference is very subtle between \"dialectal\" and standard sentences?\n\nB: The authors analyse the impact feature adapters have on the mean accuracy on the MNLI task. However, despite their claim that there is an overall increasing correlation with the number of adapters added, performance markedly decreases between 75 and 100 adapters. Is there something specific about the adapters added in that stage? Additionally, the figure also does not have numbers on the y axis, making it difficult to read the exact impact of the adapters.\n\nC: When testing the various setups on MNLI, the authors use a so-called \"validation mismatched split\" to evaluate on. Can they clear up what they mean by that?\n\nD: What is the training objective the various feature adapters are trained on? What data sizes are used for training the individual adapters?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review of our paper. We extend our gratitude for your thorough and insightful evaluation of our paper. And we greatly appreciate your recognition of the most significant advantage of DADA: enhancing multi-dialect robustness without relying on a high-accuracy dialect-identification system, a challenge that is still far from being resolved. We also greatly appreciate your interest in applying our DADA to languages other than English and multilingual NLP.\n\nWe agree that evaluating **robustness** across dialects and across tasks simultaneously would solidify our confidence in the generalizability of our method. However, we believe that our current two experiments are complementary, and when combined, there is sufficient evidence to demonstrate that DADA can enhance multi-dialect robustness across multiple tasks. And to further strengthen this, we will release our code and all our trained feature adapters to the public later and this will enable people to verify by themselves.\n\nRegarding the **efficiency** of DADA compared to individual dialect adapters, it's crucial to acknowledge that the latter approach assumes the presence of a highly accurate dialect identification system and a prior knowledge of the specific dialect associated with the input. However, this assumption is not feasible in real-world scenarios. Furthermore, flexible boundaries and the evolution of dialects means that while feature adapters could be reusable, fixed dialect adapters might need more frequent retraining.\n\nRegarding the **reproducibility**, we provide additional experimental details as follows: Throughout the process of model training (including finetuning, adapter tuning, DADA training etc.), we consistently employed the standard training objectives specific to the tasks, for example cross-entropy loss for classification: $- \\sum_{i=1}^{N} y_i \\cdot \\log(p_i)$. In order to ensure robustness, we conducted a total of 5 runs for each experiment (except for time or resources intensive ones). The reported final results were based on the average values derived from these runs. Regarding the experiments conducted with ChatGPT, all were executed with model \"gpt-3.5-turbo\" on May 16 2023, and for each task, we randomly sampled 500 examples for the evaluation.\n\nRegarding the **statistics** of the training datasets, we had a table depicting the size of each training dataset, which we later opted to remove due to space constraints. For your reference, here are some statistics regarding the training datasets for AAVE adaptation. \n\n[Table: Linguistic Rules, Dataset Size, and Feature Adapter Accuracy for AAVE Dialect]\n| Linguistic Rule      | Size    | Eval Acc |\n|----------------------|---------|----------|\n| been_done            | 48,515  | 84.46    |\n| dey_it               | 33,927  | 84.41    |\n| drop_aux             | 78,157  | 84.06    |\n| got                  | 25,203  | 83.41    |\n| lexical              | 331,784 | 86.11    |\n| negative_concord     | 49,529  | 84.41    |\n| negative_inversion   | 658     | 83.03    |\n| null_genetive        | 50,122  | 84.11    |\n| null_relcl           | 45,899  | 83.70    |\n| uninflect            | 124,447 | 84.64    |\n\nUnfortunately, due to limitations in space, we are unable to present the statistics for all the linguistic features. However, we plan to re-include this information in the appendix of the revised version of our paper. To further improve the reproducibility, we are committed to release our code and all the trained adapters later.\n\n\n\nBelow are the responses to your supplementary questions:\n\n-**QA:** For DADA, the availability of training instances is not an issue, even though the linguistic features are quite infrequent. This is because the training samples we utilize are synthetic data - dialect text transformed from standard text. And it has been shown that DADA can deal well with very subtle linguistic features. Among the linguistic features we are currently considering, there are already some very subtle linguistic features, and DADA has adapted to them quite effectively. For example, within the linguistic feature 'regularized_reflexives,' the difference is often only a letter (e.g., 'myself' and 'meself').\n\n-**QB:** We don’t think anything particularly noteworthy happened here; it is possible that this is merely some noise. Also the decrease is only minor and not significantly marked. And because our experimentation involved only every 25 adapters, experiments with finer granularity may help diminish such noise.\n\n-**QC:** There are two validation sets of MNLI - “mismatched” and “matched”. The \"mismatched\" split means that the text comes from a different distribution than the training set. Following previous works, we evaluate the model on the validation matched set during training and report the final performance on the mismatched set.\n\n-**QD:** See above.\n\nFurthermore, we are committed to releasing all the code of our paper, as well as making all of our trained adapters available as assets for the entire community to use, which also stands as a main contribution of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper proposes a promising methodology which is novel and uses linguistic information for dialectal adaptation.\n* It reasons well for the need for such adaptation on ethical, moral and societal grounds.\n* It reasons well for the advantages of its methodology in terms of difficulties of dialectal identification which makes targetted adaptation to dialects challenging.\n* It provides a extensive literature review on dialect adaptation and parameter-efficient learning.\n* Its framework is flexible enough to allow adaptation to a range of potential target dialects across a number of various tasks.\n* The aspects of interpretability are promising.", "reasons_to_reject": "* The authors compare their DADA with other setups across dialects only on a single task, MNLI, and reserve the other tasks for a single dialect. This might limit how generalisable the results are.\n\n* It would be an asset if the authors could release the feature adapters to the public, currently this is missing.\n\n* Relative efficiency of the method compared to maintaining individual dialectal adapters is questionable. There are presumably much fewer English dialects than linguistic features that set them apart and it is not true that for fine-tuning adapters we would need as many separate models as many dialects.\n\n* There are concerns with reproducibility of the paper. The authors do not report what training objective they use to create their feature adapters and the size of the training datasets for each. They also do not state whether they carry out multiple experimental runs to make their results more robust. Finally, they compare DADA on various GLUE tasks against ChatGPT. Due to limitations, they limit the development set to 500 examples, but do not state either the version number of ChatGPT, nor the basis on which they make this selection.", "questions_to_authors": "A: Can we be certain that the feature adapters adequately learn to adapt to the linguistic features they should? What about cases when there are very few training instances due to the rarity of a certain features? What about when the difference is very subtle between \"dialectal\" and standard sentences?\n\nB: The authors analyse the impact feature adapters have on the mean accuracy on the MNLI task. However, despite their claim that there is an overall increasing correlation with the number of adapters added, performance markedly decreases between 75 and 100 adapters. Is there something specific about the adapters added in that stage? Additionally, the figure also does not have numbers on the y axis, making it difficult to read the exact impact of the adapters.\n\nC: When testing the various setups on MNLI, the authors use a so-called \"validation mismatched split\" to evaluate on. Can they clear up what they mean by that?\n\nD: What is the training objective the various feature adapters are trained on? What data sizes are used for training the individual adapters?", "paper_topic_and_main_contributions": "In this paper, the authors propose **DADA**, a flexible and efficient framework of language model adaptation to non-standard (non-SAE) dialects of English. They train individual adapters corresponding to linguistic features that are characteristic of various dialects, and ensemble them using AdapterFusion in order to improve language model performance on the target varieties. Using various tasks from the GLUE benchmark, including MNLI and SST2, the paper shows that DADA can outperform the baseline of fine-tuning or adapter-tuning a language model such as RoBERTa and FLAN-T5 on SAE data on a range of dialects, for instance, African-American Vernacular English, Appalachian English, and Indian English. \n\nIt is true that the performance improvements are relatively minor, and that DADA generally underperforms the baseline of training adapters on dialectal datasets. However, the authors' method circumvents the need for reliable dialect-identification which is far from a solved issue. Additionally, DADA is a highly interpretable method where the activation of adapters corresponding to individual linguistic features can be indicative of the features present in a target text or variety. Furthermore, DADA has the potential to inspire a wide range of follow-up papers experimenting with feature adapters for language model adaptation for multilingual NLP and varieties of other languages. \n\nOverall, it is a strong paper that I would recommend for acceptance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a dialect adaptation to a number of English dialects without the need of a high-accuracy dialect identification systems. They propose DADA, a modular approach to adapt Standard American English (SAE) trained models to different dialect variants by composing adapters which handle specific linguistic features obtained from several linguistic rules. They trained on nearly 200 feature adapters that distinguish SAE and other English dialects. 

All the reviewers appreciated the incorporation of the linguistic rules to the model, and the promising experiments, and the modularity of adapters incorporated in the model. There are a few concerns about how they obtained the 200 linguistic features which the authors adequately responded to. Another issue was the small improvement in performance. Overall, this is a great contribution for incorporating other english dialects.
Decision: Accept-Main