Task-Agnostic Low-Rank Adapters for Unseen English Dialects
Authors: Zedian Xiao
William Barr Held
Yanchen Liu
Diyi Yang
Venue: EMNLP 2023 Main (2023)
Keywords: Low-resource
Dialects
Hypernetworks
LoRA
Cross-dialectal Alignment

Abstract: Large Language Models (LLMs) are trained on corpora  disproportionally weighted in favor of Standard American English. As a result, speakers of other dialects experience significantly more failures when interacting with these technologies. In practice, these speakers often accommodate their speech to be better understood. Our work shares the belief that language technologies should be designed to accommodate the diversity in English dialects and not the other way around. However, prior work on dialect struggle with generalizing to evolving and emerging dialects in a scalable manner. To fill this gap, our method, HyperLoRA, leverages expert linguistic knowledge to enable resource-efficient adaptation via hypernetworks. By disentangling dialect-specific and cross-dialectal information, HyperLoRA improves generalization to unseen dialects in a task-agnostic fashion. Not only is HyperLoRA more scalable in the number of parameters, but it also achieves the best or most competitive performance across 5 dialects in a zero-shot setting. In this way, our approach facilitates access to language technology for billions of English dialect speakers who are traditionally underrepresented.

Task- Low-Rank Adapters for Unseen English Dialects
Zedian Xiao
William Held
Yanchen Liu
Diyi Yang
Stanford University Georgia Institute of Technology
Harvard University
markxiao@stanford.c edu, wheld3@gatech. edu, yanchenl iu@g harvard.
diyiyêcs. stanford.edu
Abstract
Unseen
Methods
Tasks
Dialects
Large Language Models (LLMs) trained on corpora disproportionally weighted in fa- vor Standard American English. As re- sult, speakers of other dialects experience sig- nificantly more failures when interacting with these technologies. practice, these speakers ofien accommodate their speech to be better understood. Our work shares the belief that language technologies should be designed to accommodate the diversity in English dialects not the other way around. However, prior works dialect struggle with generalizing to evolving and emerging dialects in scalable manner. To fill this gap, our method, Hyper- LoRA, leverages expert linguistic knowledge to enable resource-efficie pemetworks. By disentangling spalect-specific and cross-dialectal information, HyperLoRA improves generalization to unseen dialects in task-agnostic fashion. Not only is HyperLoRA more scalable in the number of parameters, but also achieves best or most competitive performance across dialects in a zero-shot setting. In this way, our approach facilitates access to language technology for billions of English dialect speakers who traditionally underrepresented.
Jørgensen et (2016)
Blodgett et al. (2018)
X
Multi-VALUE Ziems et al.
X
(2023b)
TADA Held et al. (2023) HyperL.oRA
V
x
/
Table I: Comparison of previous work in dialectal ro- bustness under zero-shot shot transfer capabilities new tasks and new dialects.
are under-represented (Gururangan et al., 2022) in the training corpus of these technologies (Hovy and Spruit, 2016; Blodgett and O' O'Connor, 2017; Halevy al., 2021a). Understanding and mitigat- ing these discrepancies are particularly important in avoiding harmful and undesired consequences, which can range from denial care in commer- cial healthcare systems (Obermeyer et al., 2019) to racial biases in hate speech detection (Davidson et al., 2019; Sap et al., 2019; Rios, 2020; Halevy et al., 202 lb; Zhou et al., 2021).
adaptation via hy-
Previously, dialectal robustness methods have primarily focused on filling the lack of dialect data via manual (Blevins al., 2016; Blodgett et al.. 2018) and weak forms of supervision (Jørgensen et al., 2016; Jurgens et al., 2017), more recently via synthetic data augmentation MMulti-VLLU Ziems et al., 2022, 2023b). shared limitation of these methods their assumption of available dialectal data for all downstream tasks. In prac- tice, this is unrealistic, as it is already challeng- ing to find annotators in all dialects (Ziems et al., 2023b). Recent work has started to reduce the burden task-specific dialectal data, such as by training task-agnostic adapters via cross-dialectal alignment (Held al., 2023). While new dialects are emerging and existing dialects are evolving, the need for data all dialects remains, as well as
1 Introduction
Dialectal diversity stems from racial, cultural, re- ligious, ethnic, regional, socio-cconomic, and age- related differences. Considering the increasingly widespread integration of LLMs (Dai et al., 2019; Liu et al., 2019; Raffel et al., 2020) in daily tools, these LLMs should be made invariant to dialec- tal differences. This is not yet the case, in fact, a significant gap in the performance of LLMs is observed when they are applicd English dialects linguistically distant from Standard American En- glish (SAE) (Jurgens et al., 2017; Blodgett al., 2018; Kiritchenko and Mohammad, 2018; Ziems et al., 2023b). These discrepancies raise racial, eth- nic, and socio-economic concerns for groups that

---

adaptation methods that are resource-efficient and lask-agnostic.
prompting LLMs can further degrade the perfor- mance on these dialects (Ziems et al., 2023a; Liu et al.. 2023). These discrepancies can further rein- force existing power imbalances (Hovy and Spruit, 2016; Bommasani et al., 2021) and bring alloca- tional harm to specific racial, ethnic, and socio- economic communities. This is precisely why the development of dialect robust methods are cur- rently of the utmost importance.
To this end, we propose HyperLoRA, an effi- cient adaptation method to new dialects without the need for additional dialect annotations. In re- moving this dependency on dialect data, we turn to existing expert knowledge on dialects. Bird (2022) claims that we do not need to bridge the gap in data in settings where expert knowledge is read- ily available. This assumption of having access to expert knowledge is reasonable because the cost of having a single expert identify the dialect of a speaker is much lesser than hiring annotators from all dialects. Previously, the use of typological fea- tures has been successful in removing this gap in the multilingual setting (Ansell et al., 2021). In- spired by this, our work investigates whether this expert knowledge and typological features can be leveraged for dialects as well.
Transfer Learning Transfer Learning has be- come the dominant paradigm in specializing mod- els to target languages and tasks. To this effect, many parameter-efficient fine-tuning (PEFT) (Hu et al., 2021; Houlsby et al., 2019; Zaken et al., 2022) modules have been designed for efficiently adapting Large pretrained Language Models to downstream applications (Pfeiffer et al., 2023). MAD-X (Pfeiffer et al., 2020b) shows that sep- arate task and language adapters can be composed to achieve multi-task cross-lingual transfer. Like MAD-X, TADA (Held et al., 2023) trains dialect- specific adapters separately from task adapters, al- lowing the adaptation of the SAF-trained model to different dialects in a task aankosgc manner. These works, however, are limited by the need to train an adapter for each language/dialect. To address this shortcoming, several works make use of hypernet- works generate language-specifie adapters from language typological vectors (Ansell et al., 2021) and language identifiers (Üstün et al., 2022), effec- tively removing the need to train over hundreds of language adapters. In addition to adapters, hyper- networks have also been applied to prompt-tuning (He et al., 2022) and LoRA (Phang et al., 2022). While prior work mainly generates modules for lan- guage adaptation, work is the first to perform dialect adaptation via hyperettworss.
A natural solution in leveraging this expert knowledge is via hypernetworks (Ha et al., 2016), which have exhibited remarkable generalization ca- pabilities in computer vision and NLP (Knyazev et al., 2021; Üstún et al., 2022). Using a hyper- network, we modulate dialect-specific LoRA (Hu et al.. 2021) adapters using typological features for adaptation to target dialects. By isolating the complexity of the typological space to the hyper- network and by generating dialect-specifie LoRA adapters, we minimize the cross-dialectal interfer- ence (Wang et al., 2020) in the main model. The hypernetwork is trained on parallel corpora to opti- mize a morphosyntactic aligmment objective in the representation space, which allows HyperLoRA to learn to adapt to dialects independently of the downstream application. This alignment objective is novel, principled, and easy to compute. Most importantly, find that effectively using expert knowledge can account for 250 annotations per di- alect. Finally, we design metric to evaluate the coverage of dialect features, in order to better un- derstand the limitations of using hypernetworks for zero-shot generalization to dialects.
Cross-lingual alignment Cross-lingual align- ment has been observed in learned representations of multilingual language models (Pires et al., 2019). Alignment is a particularly desirable property en- abling task-adapter modules to be shared across languages. Furthermore, cross-lingun alignment methods (Conneau et al., 2018, 2020) are partic- ularly effecttve when working with highly simi- lar languages, making them suitable for the cross- dialectal setting. Surprisingly, this cross-dialectal setting remains underexplored. In most settings, token-to-tun level supervision for alignment is unavailable. Prior works have addressed this by
2 Related Work
Dialectal NLP When applied to other English di- alects, existing language models that primarily fo- cus on Standard American English (SAE) often demonstrate significantly lower performance (Sap et al., 2019; Rios, 2020; Halevy et al., 2021b; Zhou et al., 2021). Previous research has revealed that

---

tion where the possessive marker is appended at the end of the noun phrase. To linguists, this is known as a línguistic feature or linguistic rule that dialect speakers employ at different rates and in different contexts. Experts have found that this feature is not unique to the East Anglian dialect and can be found in many dialects geographically close to the East Anglian dialect, or even in Indian English and in Hong Kong English, with lower levels of per- vasiveness. Experts have long studied the intra- and cross-dialectal variations in the lens of these typological features. We follow the intuition of Nerbonne (2009), defining dialects by their unique sets of correlated dialect features. These typolog- ical feature vectors are readily available on the Electronic Atlas of Varieties of English (eWAVE; Kortmann al., 2020)' Multi VALUE applies fea- ture transformations probabilisticially according their attestation in WAVE at the following rates: 100% for obligatory features, 60% for features nei- ther pervasive nor rare, 30% for rare features and 0% for no information or attested absence. We fol- low this procedure. More specifically, we model the space of linguistic features jointly with their aggregation patterns using neural network and investigate its usefulness for cross-dialectal gener- alization.
hypernetork
former
Meight Vector
edapter
Feesforsard
Re
Add Nom
Mu Bhiti-hue Ti-heac Attentine
Dialect Features
grop_auk
Figure I: HyperLoRA Architecture During training, only hypernetwork weights are updated and there is no task adapter in the main model. inference, the task adapter and its classification head are added.
developing unsupervised methods. In this line of work, few methods perform cross-lingual align- ment by minimizing an approximate Wasserstein distance (Arjovsky et al., 2017; Romanov et al., 2019). Alternatively, prior work has shown that di- rectly optimizing for a relaxed Wasserstein distance using Sinkhorn's 's Divergence can also be effective for cross-lingual alignment when sufficiently re- liable representations are available (Zhang et al., 2017). In our setting, Multi VALUE provides us with an abundance of pseudo-dialectal training data, which makes it possible for us to design mor- phosyntactic alignment objective.
3.2 Hyper
We leverage hypernetworks for Low-Rank Adap- tation (LoRA). LoRA (Hu et al., 2021) fine- tuning approach that keeps the full model parame- ters fixed and instead updates low-rank decompo- sition of the attention matrices. Instead of updating LoRA weights directly, our approach learns the weights of a hypernetwork (Ha et al., 2016), which is then used to generate the appropriate LoRA weights. To our knowledge, we are the first to generate LoRA adapters with a hypernetwork for domain adaptation. We give a detailed outline in Figure for this novel hypernetwork architecture for generating LoRA parameters. Concretely, we lay out the notation for our hypernetwork archi- tecture as follows. Let D Uk denote the layer k low-rank projections associated with the query, and D Uk those associated with the value. We use hypernetwrrss g taking input concat(d, it ((aw)) where d E [0, 1]th features is the dialect feature vec- tor and dk i/g,w) {0, 2 x # blocks) the posi-
3 HyperLoRA
As a first step towards dialectal robustness, Hyper- LoRA enables resource-efficient adaptation to new dialects task-agnostic manner. Our approach relies on 4 key ingredients: (1) we support low- resource dialects with expert linguistic knowledge whose information is modeled by (2) hypernet- work that learns shared linguistic feature space across dialects. The hypernetwork is trained to gen- erate (3) lightweight LoRA modules with (4) the objective align dialect and SAE representations by finding the optimal transport plan. Under this optimal transport plan, we can dircctly plug the LoRA modules into any downstream task.
3.1 Dialectal Typology as Expert Knowledge
"The man met's girlfriend is real beauty", is what an East Anglian dialect speaker would say instead of "The girlfriend of the man I met is a real beauty". The East Anglian speaker uses a construc-
'These vectors can be found at https//gittub.co//AAIT NLP/multi-value

---

Training
Inference
acapter NgE generated
die lect LoRt
hypernstwork
Transport Map
Model
CLS HEAD
Transforer (tyer
SAE
aaypern tacrk
EndE
AMVE
CALISGE
Lhee
Rig
Inout Sequence
Figure 2: Training and Inference pipelines: During training, the hypernetwokk learns mapping from the dialect feature vector to the LoRA adapter weights that perform alignment. During inference, the same hypernetwork is used generate dialect-specific LoRA adapters from dialect features. At both the training and inference time, concatenate positional encoding to the dialect feature to differentiate between transformer blocks, and between the query and the value LoRA adapters.
tional embedding that differentiates between layers and between queries and keys. We use separate hypernetworks for D Dgg.3 and U (g.e3- Each hy- pemeeworr parameterized by weights Wa. Wa denoting the down and up projections respectively. Finally, for Dia.v) (similarly for U(awz) the hyper- network equations can be written as:
across and within dialects. However, comprehen- sive examination of PEFT modules for dialects is needed, which leave for future work.
3,4 Morphosyntactic Alignment
While there an abundance of sentence parallel bitexts originating from machine translation used for cross-lingual alignment, the equivalent does not exist for English dialects. As a remedy, we employ the rule-based translation system of Multi- VALUE (Ziems et al., 2023b) to generate paral- lel corpora for all source dialects. While Multi- VALUE evaluation real-world performance (Ziems al., 2023b), the synthetic nature of this evaluation is a limiation of our work discussed further in the Limitations section.
concat(d, 1 ((aw))
(1)
D Daw Uk (g.)) g(x)
(2)
and more specifically:
D Plao MM(ReLU((M((. Wa),, W.))
(3)
shown to be predictive of
where MM stands for matrix multiplication. Equa- tion also applies Uh (q with its respective weights via a similar calculation. Training Hy- perLoRA shown in Figure and Algorithm 1.
The VALUE transformed corpora are only aligned the sentence level. However, the differ- ences we tackle lie at the morphosyntactic level, which calls for a token-level alignment. To this end, we leverage unsupervised alignment methods discussed in previous work (Zhang et al., 2017;
3.3 Dialect-Specift Low-Rank Adaptation
Previous cross-lingual adaptation methods have fo- cused on a variety of different bottleneck adapter configurations applied after the multi-head atten- tion in the transformer layer (Lialin et al., 2023; Pfeiffer et al., 2020b, 2023). Building upon these efforts, hypothesize that adaptation at the at- lention level can be effective for the morphosyntac- tic variations present in dialects. This hypothesis stems from the observation that the self-attention mechanism, known for its sensitivity to syntac- tic nuances, can better serve syntactical variations
Al Alvarez-Melis and Jaakkola, 2018). We measure token-level variations via the earth mover's dis- tance, denoted ([PDIAL, PSAE), where IP represents the distribution of dialect last layer rep- resentations, while PAA corresponds the dis- tribution for SAE. The earth mover's distance, or Wasserstein's distance (W), can be approxi-

---

mated via Sinkhorn's divergence (Feydy et al., 2018) which interpolates between the Wasserstein Distance, and the Maximum Mean Discrepancy (MMD) via the equation:
been the primary focus of previous works in di- alectal robustness. IndE and NgE are widely used dialects by over a hundred million of speakers. CollSgE has shown to be a particularly difficult dialectal shift (Ziems et al., 2023b) sharing little linguistic features with mainstream SAE, and with many unique features in CollSgÉ alone. ChcE on the other hand is particularly close to SAE. In our experiments, we focus on these 5 dialects. Later, in our ablation studies, we will explore training HyperLoRA on other dialects closer to CollSgE to study the impact of dialccts used at training time.
1 Sa(a, def We(a, 8) / We a) W (3,
Here We is the computationalll--efiiicenn entropy regularized Wasserstein distance (Cuturi, 2013), which is defined as follows:
Wa(a, B) def min [ dx, (2)
=) xxy
Training Details For all experiments, we use a pretrained RoBERTa Base (Liu et al., 2019) as the backbone model. For the training of Hyper- LoRA, we use 1000 WiC (Pilchvar and Camacho- Collados, 2019) examples from each source di- alect. At inference time, we plug the generated LoRA module from the learned hypernetwork in the backbone model with appropriate task-specific adapters and their associated classification heads. We train HyperLoRA with source dialects using the Adam (Kingma and Ba, 2017) optimizer with a learning rate of 3e-5, with a linear scheduler, and using a batch size of 16 for 50 epochs. We load the model with the lowest loss at the end of training, These hyperparameters have been selected via a grid search over learning rates of 1e-5, 3e-5, and le-4, batch sizes of 16, 32, and 64, and between 30 and 50 epochs. For task-specific adapters, we directly utilize readily available GLUE adapters from Adapterhub (Pfeiffer et al., 2020a). In all of our experiments, HyperLoRA is trained and evalu- ated in a zero-shot fashion. For each unseen dialect (e.g., A), we train HyperLoRA using the remain- ing dialects (B, C. D, E) and evaluate its dialectal robustness against A. For example, in Figure 2, HyperLoRA is trained on AAVE, NgE, ChcE, and IndE, and evaluated on the target dialect CollSgE.
+ KKLLLTTT (
where and y are the last layer dialect and SAE rep- resentations respectively. And similarly X' and y are the feature spaces for last layer dialect and SAE representations, respectively. π the coupling that minimizes the cost e of moving mass from distribu- tions a to B. To compute the Sinkhorn divergence, we use the solver provided by Feydy et al. (2018) with E 0.05 and c as the squared error.
Algorithm HyperLoRA Training
Input: features {dahses, sentences {Iahees,
SAE representations hSAE Initialize M # Main model Initialize g # Hypernetwork for training step do
S Sample dialect
B. ~ { # Sample batch
0, + (d,) # LoRA adapter
for x. E B, do
hg + M Mx.;;3) 02) # last hidden states
end for loss (t hSAE)
backpropagate loss in g
end for Return: g
Baselines In benchmarking HyperLoRA, we evaluate its (1) resource efficiency against current task-agnostic dialect methods, its (2) dialectal ro- bustness against models trained for SAE, and its (3) ability to effectively utilize expert knowledge for adapting to new dialects. For each of these search questions, we establish suitable baseline. To address the resource efficiency of our method, we compare HyperLoRA with TADA (Held et al., 2023) trained on varying numbers of examples from the target dialect. More specifically, for each
4 Experimental Setup
Datasets We evaluate our method on 5 di- alect transformed variants of GLUE using Multi- VALUE (Ziems et al., 2023b). We choose African American Vernacular English (AAVE), Indian En- glish (IndE), Nigerian English (NgF), Colloquial Singaporean English (CollSgE), and Chicano En- glish (ChcE) as our dialects of focus. AAVE has

---

ChcE
IndE
NgE
CollSgE
38.1
92
213
05
Huundar Anouazad Sanpaies
Hunbo dendutte Semilo
Mambor dnsstted Sampla
Mambbrr Arrocated Fambiee
QQP Performance under Few-shot Evaluation: target dialect being used the model on a log scale. We use a blue star (and the scattered line) to denote the performance of HyperLoRA, while the orange line curve shows the performance of TADA using increasingly more annotated samples. The cost of training scales linearly with the number of annotated samples.
Figure 3:
The x-axis denotes the number of examples of the
k [10, 25, 50, 250, 500, 1000], train TADA & WiC samples. We follow TADA to use 1000 examples and keep the remaining training details unchanged. To highlight the dialectal ro- bustness of HyperLoRA, we implement a simple adapter baseline, which we denote by SAE. Us- ing RoBERTa-BBE Base as our backbone, we add task- specific adapters trained on the original GLUE dataset. Similarly to HyperLoRA, this is zero- shot baseline. Finally, we establish a baseline that does not utilize expert knowledge. To do this, we re- move the hypernetwork component of HyperL.RRA, A, keeping LoRA modules and our alignment loss. As opposed to HyperLoRA, these LoRA modules are cross-dialectal. We train and evaluate both Hyper- LoRA and LoRA in the same zero-shot manner.
the potential emergence of new ones. Acquiring 250 annotated samples for each dialect can be pro- hibitively expensive and challenging in terms of finding suitable annotators (Ziems et al., 2023b). We acknowledge that while HyperLoRA may not completely bridge the performance gap, it effec- tively addresses the trade-off between performance and resource constraints without any dialect exam- ples. Consequently, it provides valuable degree of robustness at an almost negligible cost.
5.2 Zero-Shot Transfer Results
To evaluate the dialectal robustness of HyperLoRA, we compare HyperLoRA to the SAE baseline across all 5 dialects in Table 2. We observe that our method generally achieves higher performance over the SAE baseline, with the exception of RTE. Noticeably, HyperLoRA achieves higher perfor- mance on more than 4 out of tasks. In analyzing these results, we find that COLA, RTE, and SST2 suffer from large variability in performance. On the remaining tasks, that is MNLI, QNLI, QQP, and STSB, HyperLoRA achieves the best or competi- tive performance. As a whole, there is an improve- ment of 1.7% in mean performance for AAVE and 0.8% in mean performance for NgE.
5 Experimental Results
5.1 Efficient Adaptation Unseen Dialects
First, we highlight the efficiency of using expert knowledge in adapting to new dialects. For the sake of simplicity, we restrict our evaluation to Quora Question Pairs (QQP), which is one of the tasks with the least variability in performance.
In Figure 3, we show QQP performances across all 5 dialects. HyperLoRA finds competitive perfor- mance at a much lower cost, showing comparable performance to TADA trained on R 250 dialect samples. For AAVE, and CollSgE, this is lower, around 50 and 25 respectively. This observation highlights the value of expert linguistic knowledge for dialect adaptation, as can be equivalent to having 250 annotated samples per dialect-a sub- stantial benefit. The significance of this finding becomes evident when considering the vast num- ber of existing dialects, which exceeds 70, and
In the case of CheF., our approach fails to bring a mean performance improvement. It is worth not- ing that the authors of VALLi- (Ziems et al., 2023b) also encountered similar outcome when training ChcE instead of SAE. This lack of im- provement can be attributed the striking simi- larities between ChcE and Colloquial American English. This set of experiments takes into account the potential variability the differences between the source dialects used to train HyperLoRA and

---

COLA
MNLI
QNLI
RTE
QQP
SST2
STSB
Mean
Unseen Dialect
Orig.
Ours
Ori..
Ours
Orig.
Ours
Orig.
Ours
Orig.
Ours
Orig.
Ours
Orig.
Ours
Orig.
Ours
AAVE
-0.02
10.5
83.7
83.8
90.5
90.5
68.9
68.4
87.0
87.2
92.8
93.5
88.5
88.7
73.0
74.7
ChcE
30.7
31.0
86.3
86.3
93.0
93.1
68.5
66.8
89.6
89.8
93.5
93.1
90.1
90.1
78.8
78.6
IndE
19.4
18.9
82.6
82.9
89.4
89.3
64.2
65.0
86.1
86.3
92.0
92.2
88.1
88.7
74.5
74.8
NgE
24.7
26.6
84.6
84.7
90.8
91.0
64.2
66.1
88.2
88.3
92.0
92.6
89.5
89.5
76.2
77.0
CollSgH
4.5
8.0
82.0
82.2
88.3
88.2
66.4
64.6
85.0
85.8
91.6
91.1
87.5
87.7
72.1
72.5
Table 2: Zero-shot performance on GLUE. For cach task, we report the SAE Task Adapter performance (Orig.) and the HyperLoRA performance (Ours). We report Matthew's Correlation score for COLA, the -pesonnSpeerman correlation score for STS-B, and accuracy for the rest. Via a paired bootstrap test at 0.05, we label significant improvements for each task with +. There was no significant drop in performance.
Methods
CollSgE Glue Performance
Model
COLA
MNLI
QNLI
RTE
QQP
SST2
STS-B
Mean
SAE
4.5
82.0
88.3
66.4
85
91.6
87.5
72.1
LoRA
0.7
82.0
88.3
64.6
85
91.0
87.5
71.3
HyperLoRA
8.0 (+7.3)
82.2
88.2
64.6
85.8 +1 (+0.8)
91.1
87.7
72.5
Table 3: CollSgÉ GLUE Performance With RoBERTa Base base model, we compare adding SAB-trained task adapters, adding SAE task adapters and LoRA, and adding SAE task adapters and HyperLoRA. Both LoRA and HyperLoRA are trained on AAVE, Chicano English, Nigerian English, and Indian English. For each task, we run paired bootstrap test with a 0.05 and label significant improvements w.r.t. the SAE Task Adapter with and w.r.t. the LoRA baseline with t. There was no significant drop in performance.
the dialects HyperLoRA is evaluated on. This vari- ability can explain the differences in performance gain across dialects.
aging expert knowledge, proves to be an effective strategy for improving cross- cioss-iillettal transfer.
As a boug-and-play module that can be readily used by any community, HyperLoRA has the poten- tial improve the robustness of the SAE-trained backbone model regardless of dialect.
Ablation Analyses
6.1 Morphosyntactic Alignment
To understand the effectiveness of our morphosyn- tactic objective, we return to TADA's setup and modify its aligmment objective to our Sinkhorn Di- vergence. For both TADA and our alignment objec- tive, train dialect-specific adapters for AAVE using 1000 parallel samples from the SAE WiC dataset and the Multi- Vilti-VALLEEE transformed AAVE Wic Dataset. We evaluate these adapters on the GLUE benchmark and report the results in Table 4.
5.3 Effectiveness of Expert Knowledge
In order to validate the contribution of expert knowledge, we compare HyperLoRA with the LoRA baseline. We report the results in Table 3. We observe that training cross-dialectal LoRA adapters can negatively impact GLUE performance. When compared to the naive SAE baseline, LoRA demonstrates poorer performance with decrease of 3.8% and 1.8% on COLA and RTE, respec- tively. For HyperLoRA, we have found that al- though there is still a slight decrease in RTE per- formance -1.8%, it proves to be superior over the SAE baseline. Specifically, HyperLoRA A brings im- provements of 3.5% on COLA and 0.8% on QQP. Through a paired bootstrap test, we verify that the drop in RTE performance is not statistically sig- nificant, while the improvements on COLA and QQP are statistically significant. In conclusion, our findings suggest that employing a hypernetwork to minimize negative interference, along with lever-
We observe that both TADA and our alignment objective outperform the naive SAE task adapter. While our strategy achieves +0.7% on COLA and -1.8% on RTE comparatively to TADA, we verify through a paired bootstrap test and find that these differences are not statistically significant. There- fore, with no significant difference in performance, our Sinkhorn divergence- based morphosyntactic alignment objective presents well-founded opti- mization problem that can be efficiently solved. It offers desirable convergence guarantees, eliminat- ing the necessity for additional heuristics employed in the adversarial training approach in TADA.

---

Methods
Adapters
AAVE
Glue Performance
Model
Dialect
Task
COLA
MNLI
QNLI
RTE
QQP
SST2
STS-B
Mean
SAE
-0.02
83.7
90.5
68.9
87.0
92.8
88.5
73.0
TADA
24.5
84.8
91.7
70.4
88.1
93.0
89.6
77.4
Sinkhorn
25.2
84.7
91.5
68.6
88.1
93.3
89.4
77.3
Table 4: Alignment Objectives: We compare the cross-dialectal alignment objective TADA with our objective based on the Sinkhorn divergence. For both objectives, we train dialect adapters on AAVE data, and evaluate it AAVE GLUE tasks. We run a paired bootstrap test at a = 0.05 but find no significant difference between TADA and Sinkhorn performances.
CollSgE Glue Performance
Source Dialect SAE MalaF, MaltE, JamE, IndSÄE CapeE, FijiAE, MaltE, SriLE NgF, AAVE, IndF, CheF
LI dist Coverage
COLA
MNLI
QNLI
RTE
QQP
SST2
STS-B
Mean
4.5
82.0
88.3
66.4
85.0
91.6
87.5
72.1
0.219
87.8
7.6
82.1
88.2
67.2
85.7
90.7
88.0
72.8
0.209
65.7
7.4
82.2
88.3
65.7
85.74
90.7
72.6
0.257
81.3
8.0
82.2
88.2
64.6
85.8
91.1
87.7
72.5
Table 5: Impact of Source Dialects: We compare CollSgE performance when training HyperLoRA on different source dialects. Typically low average LI distance and higher coverage indicate that the source dialects are closer to the target dialect. We label significant improvements in performance over SAB with +.
6.2 Impact of Source Dialects
CollSgE, we find that the set (CapeF, FijiAE, MaltE, SriLE) attains the lowest Manhattan dis- tance, but also low coverage score. Moving up in the pareto frontier, the set (MalaE, MaltE, JamE, IndSAE) attains a low Manhattan distance, but high coverage score. We train HyperLoRA for these two sets source dialects and compare the perfor- mance to our previous experiment (Section 5), We report the results in Table 5.
We study the impact of the source dialects more closely by analyzing the distinctiveness of the new dialect at test time with respect to the source di- alects used for training. This distinctiveness of di- alect feature sets is natural, in fact, it is commonly known in dialectology that some features contra- dict each other (Nerbonne, 2009). Commonly used metrics to measure dialect differences the geo- graphical distance and the Manhattan distance ap- plied dialect feature vectors (Ziems al., 2023b). However, these metrics are not directly suited for the multi-source setting. To this effect, we develop a metric for feature coverage, as follows. We hy- pothesize that HyperLoRA performs best on new dialects when most of the linguistic features of the new dialect have been seen during training.
We find that both lower average Manhattan dis- tance and larger feature coverage can contribute to performance improvement on the target dialect. Specifically, simultaneously decreasing the Man- hattan distances and improving the feature cover- age can lead to an improvement of +2.6% on RTE (from NgE, AAVE, IndE, ChcE to MalaE, MaltE, JamE, IndSAE). Overall, when the new dialect is particularly close in Manhattan distance and largely covered by the source dialects, we observe Hyper- LoRA can lead to the highest performance, with an improvement of +0.7% on mean performance, compared to the SAE baseline.
Coverage = I[(E da) d
(4)
d
where de and de are the linguistic feature vectors for a source dialect and the target dialect i, re- spectively. S represents the set of souree dialects. Our metric effectively computes the percentage of weighted features in the target dialect that are cov- ered by dialects in S.
Based on these findings, we demonstrate that when computational resources are limited, employ- ing these heuristics offers straightforward and efficient strategy for selecting the source dialects when addressing evolving dialects.
To measure the impact of source dialects, compute the average Manhattan distance and the coverage score for all combinations of 4 dialects that are different from the target dialect. For
Conclusion
In this paper, we propose HyperLoRA, a task- agnostic, light-weighted, and highly scalable di-

---

alect adaptation method. Where only accessing ex- pert knowledge about dialects, we show that Hyper- LoRA can lead to robustness improvement against unseen dialects on the GLUE benchmark, across five dialects. At inference time, HyperLoRAA does not require any dialect data, which makes it widely applicable in resource and compute-constrained set- tings. Furthermore. HyperLoRA is trained with data volume that can be easily replaced by manu- ally translated dialect corpora. This resource and computational efficiency greatly facilitate the ap- propriation of language technologies within small but diverse communities". Finally, by generating LoRA adapters using a lightweight hypernetwork, our approach is highly portable to LLMs with less than 0.5% additional parameters and without any additional inference latency. These aspects enable HyperLoRA to achieve favorable tradeoff be- tween the training and inference cost and dialectal robustness. To sum up. HyperLoRA holds great po- tential to enable billions of traditionally underrepre- sented English dialect speakers to access language technology using their preferred languages.
periment gap where we are unable to verify the per- formance of our method on encoder-decoder, and decoder-only architectures. Future work should fill the gap and further explore task-agnostic dialect adaptation solutions for models with these alternate architectures.
Ethies Statement
As highlighted in our limitations, we acknowledge that we are unable to offer guarantees regarding the usage of HyperLoRA in communitics where intra- dialectal variations are prevalent. This limitation stems from the fact that dialects are not uniform en- tities and encompass diverse variations. Therefore, it crucial for members of these dialect commu- nities to take necessary precautions when applying HyperLoRA to their use cases.
Acknowledgement
We would like thank the anonymous review- ers and SALT lab members for their valuable feedback. This work was partially sponsored by the Defense Advanced Research Project Agency (DARPA) grant HR001 1229112222122 100000003/000011 1260656, and NSF grant 2IS-2247357 and I1S-2308994.
Limitations
HyperLoRA is trained on pseudo-dialects obtained using the Multi-VALUP (Ziems et al., 2023b) trans- formation rules, which are synthetic dialectal shifts that focus on morphology and syntax- differ- ences. It is important to note that these shifts do not encompass the entirety of possible variations found in real-world dialects. Therefore, we encourage fu- ture research to address this limitation and explore other naturally occuring variations associated with dialects such as lexical differences, topical shifts and register shifts. Additionally, while HyperLoRA can utilize any linguistic vector that provides a more detailed characterization of dialects during the testing phase, we did not conduct a sensitivity analysis for these vectors. This lack of guarantee can pose challenges since real-world dialectal vari- ations are often much more nuanced and intricate. Furthermore, our work does not include comprehensive comparison various parameter- efficient fine-tune fne-tuning techniques for dialect adapta- tion. We encourage further research to delve into this area and explore it.
References
David Alvarez-Melis and Tommi Jaakkola. 2018. Gromov- Fromov-Wasseee Wasserstein alignment of word embedding spaces. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing. pages 1881-1890, Brussels, Belgium. Association for Computational Linguistics.
Alan Ansell, Edoardo Maria Ponti, Jonas Pfeiffer, Se- bastian Ruder, Goran Glavaš, Ivan Vulić, and Anna Korhonen, 2021. MAD-G: Multilingual adapter gen- eration for efficient cross-lingual transfer. In Find- ings of the Association for Computational Linguis- ticx: EMNLP 2021, pages 4762-4781, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein gan.
Steven Bird. 2022. Local languages, third spaces, and other high-resource scenarios. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (ooluee 1: Long Papers), pages 7817-7829, Dublin, Ireland. Association for Compu- tational Linguistics.
Finally, all of our experiments primarily focus on encoder-only LLMs. As result, this creates ex-
Terra Blevins, Robert Kwiatkowski, Jamie MacBeth, Kathleen McKeown, Desmond Patton, and Owen Rambow, 2016. Automatically processing tweets
"You can find our https//gthua..co//zcttaa/hppppop
implementation
at

---

from sang-involver youth: Towards detecting loss and aggression. In Proceedings of COLING 2016, the Computational Lin- guistics: Technical Papers, pages 2196-2206, Osaka, Japan. The COLING 2016 Organizing Committee.
8451, Online. Association for Computational Lin- guistics.
26th International Conference
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross- lingual sentence representations. In Proceedings of the 2018 Conference on Empiriaal Methods in Nat- ural Language Processing, pages 2475-2485, Brus- sels, Belgium. Association for Computational Lin- guistics.
Su Lin Blodgett and Brendan O' O'Connor. 2017. Racial
disparity in natural language processing: A case study of social media african-ameriia english.
Su Lin Blodgeu, Johnny Wei, and Brendan O' O'Connor.
2018. Twitter Universal Dependency parsing for African-American and mainstream American En-
Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.
glish. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1415-1425, Melbourne, Australia, Association for Computational Linguistics.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Car- bonell, Quoe Le, and Ruslan Salakhutdinov. 2019. Transformer-XL. Attentive language models beyond fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-29999999 2988, Florence, Italy. Asso- ciation for Computational Linguistics.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel. Jared Davis, Dora Dem- szky, Chris Donahue, Moussa Doumbouya, Esii Dur- mus, Stefano Ermon, John Etchemendy, Kawin Fiha- yarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lau- ren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang. Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khami, O. Khat- tab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee. Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li. Xuechen Li, Tengyu Ma, Ali Ma- lik. Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa. Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadim- itriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Re- ich, Hongyu Ren, Frieda Rong. Yusuf H. Roohani, Camilo Ruiz. Jack Ryan, Christopher R'e. Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Ya- sunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang. Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021. On the opportunities and risks of foundation models. ArXiv.
Thomas Davidson, Dehasmita Bhattacharya, and Ing- mar Weber. 2019. Racial bias hate speech and abusive language detection datasets. In Proceedings of the Third Workshop on Abusive Language Online, pages 25-35, Florence, Italy, Association for Com- putational Linguistics.
Jean Feydy, Thibault Séjourné, François-Xaviee Vialard, Shun ichi Amari, Alain Trouvé, and Gabriel Peyré. 2018. Interpolating between optimal transport and mmd using sinkhorn divergences.
Suchin Gururangan, Dallas Card, Sarah K. Dreier, Emily K. Gade, Leroy Z. Wang, Zeyu Wang. Luke Zettlemoyer, and Noah A. Smith. 2022. Whose lan- guage counts as high quality? measuring language ideologies in text data selection.
David Ha, Andrew Dai, and Quoc V. Le. 2016. Hyper- networks.
Matan Halevy, Camille Harris, Amy Bruckman, Diyi Yang, and Ayanna Howard. 202 Mitigating racial biases in toxic language detection with an equity- based ensemble framework. In Equity and Access in Algorithms, Mechanisms, and Optimization. ACM.
Matan Halevy, Camille Harris, Amy Bruckman, Diyi Yang. and Ayanna M. Howard. 2021b. Mitigating racial biases in toxic language detection with an equity-based ensemble framework. Equity and Ac- cess in Algorithms, Mechanisms, and Optimization.
Yun Hc, Huaixiu Steven Zheng. Yi Tay, Jai Gupta, Yu Du, Vamsi Aribandi, Zhe Zhao, YaGuang Li. Zhao Chen, Donald Metzler, Heng-Tze Cheng, and Ed H. Chi. 2022. Hyperprompt: Prompt-based task- conditioning of transformers.
Alexis Conncau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440-
Will Held, Caleb Ziems, and Diyi Yang. 2023. Tada: Task-agnostic dialect adapters for english.

---

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parfmeter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. 2021. Parameter- efficient multi-task fine-tuning for transformers via shared hyperetworks.
John Nerbonne. 2009. Data-driven dialectology. Lan- guage and Linguistics Compass, 3(1):175-198;
Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. Dissecting racial bias in an algorithm used to manage the health of popula- tions. Science, 366(64444):47745333
Dirk Hovy and Shannon L. Spruit. 2016. The social impact of natural language processing. In Proceed- ings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- pers). pages 591-598, Berlin, Germany. Association for Computational Linguistics.
Jonas Pfeiffer. Sebastian Ruder, Ivan Vulié, and Edoardo Maria Ponti. 2023. Modular deep learning.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulié, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020a. Adapterhub: A framework for adapting transformers.
Anna Jørgensen, Dirk Hovy, and Anders Søgaard. 2016. Learning a POS tagger for AAVE-like language. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 11 15-, San Diego, California. Associa- tion for Computational Linguistics.
Jonas Pfeiffer, Ivan Vulié, Iryna Gurevych, and Sebas- tian Ruder. 2020b. Mad-x: An adapter- based frame- work for multi-task cross-lingual transfer.
Jason Phang, Yi Mao, Pengcheng He, and Weizhu Chen, 2022. Hypertuning: Toward adapting large language models without back-propagation.
David Jurgens. Yulia Tsvetkov, and Dan Jurafsky. 2017. Incorporating dialectal variability for socially equi- table language identification. In Proceedings of the 55th Annual Meeting of the Association for Compu- lational Linguistics (Volume 2: Short Papers), pages 51-57, Vancouver, Canada. Association for Compu- tational Linguistics.
Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: the word-in-contex dataset for evalu- ating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 1267-1273, Minncapolis. Minnesota. Association for Computa- tional Linguistics.
Diederik P. Kingma and Jimmy Ba. 2017. Adam: A method for stochastic optimization.
Svetlana Kiritchenko and Saif Mohammad. 2018. Ex- amining gender and race bias in two hundred senti- ment analysis systems. In Proceedings of the Sev- enth Joint Conference on Lexical and Computational Semantics, pages 43-53, New Orleans, Louisiana. Association for Computational Linguistics.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceed- ings of the 57th Annual Meeting the Association for Computational Linguistics, pages 4996-5001, Flo- rence, Italy. Association for Computational Linguis- tics.
Boris Knyazev, Michal Drozázal, Graham W. Taylor, and Adriana Homero-Soriana, 2021. Parameter pre- diction for unseen deep architectures.
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former.
Bernd Kortmann, Kerstin Lunkenheimer, and Katharina Ehret, editors. 2020. eWAVE.
Anthony Rios. 2020. Fuzze: Fuzzy fairess evaluation of offensive language classifiers on english. In AAAI Conference on Artificial Intelli- gence.
Vladislav Lialin. Vijeta Deshpande, and Anna Rumshisky, 2023. Scaling down to scale up: A guide to parfmeter-effiiiet fine-tuning.
Yanchen Liu, William Held, and Diyi Yang. 2023. Dada: Dialect adaptation via dynamic aggregation of lin- guistic rules.
Alexey Romanov, Anna Rumshisky, Anna Rogers, and David Donahue. 2019. Adversarial decomposition
of text representation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach.
Minneapolis, Minnesota. Association for Computational Linguistics.

---

Maarten Sap. Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668-1678, Florence, Italy. Asso- ciation for Computational Linguistics.
max IE [fade] max PaAL [Adv(d: 8)]
Adv
ESSPPAE [Adv(s; 0)]
Cédric Villani. 2008. Optimal transport: Old and new.
assume Adv is K-Lipschitz,
Zirui Wang, Zachary C. Lipton, and Yulia Tsvetkov. 2020. On negative interference in multilingual mod- els: Findings and meta-learning treatment. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), Online. Association for Computa- tional Linguisties.
sup Ed-Ppas [Adv(d;0) 0)] R3 K ||Ad Ae.
-ESWPSAE [Adv(s; 0)
pages
When c E2.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2022. Bitfit: Simple parameter-efficient fine-tuning for transformer-basee masked language-models.
=W(F PSAE)
Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Earth mover's s distance minimization for unsupervised bilingual lexicon induction. In Proceed- ings of the 2017 Conference Empirical Methods Natural Language Processing, pages 1934-1945, Copenhagen, Denmark. Association for Computa- tional Linguisties.
Under the Eg ground distance, the last step follows from the centtrrovic-Ruuinsttee duality (Villani, 2008). Briefly, when the objective of the critic reaches optimality, it approximates the Wasserstein distance up to scaling factor K, while the generator minimizes this approximate distance. As such, we have shown that TADA aims to minimize the same mathematical objective.
Xubui Zhou, Maarten Sap, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2021. Challenges in automated debiasing for toxic language detection. ArXiv,
Our alignment objective is independent of the chosen ground distance, as opposed to the dual problem used by WGAN that only holds when the ground distance is the & distance. Using Sinkhorn's 's divergence, we do not need to intro- duce adversarial training procedure that relies the optimization and the approximation power of a critic network. We understand that this is not direct comparison as TADA also includes a con- trastive sequence loss, thus putting a higher weight on the CLS token.
Caleb Ziems, Jiaao Chen, Camille Harris, Jessica An- derson, and Diyi Yang. 2022. VALUE: Understand- ing dialect disparity in NLU. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 3701-3720, Dublin, Ireland. Association for Compu- tational Linguistics.
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023a. Can large language models transform computational social sci- ence?
B Dialectal Differences
Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023b. Multi- value: A framework for cross-dialectal english nlp.
To quantify how much of our test sets be- ing modified by applying Multi-VALUE we com- pute the percentage of entries that have been trans- formed for each test set in figure On average, for each dialect, we have over 88% transformed entries except for Chicano English. This is expected, as Chicano English shares many similarities with Col- loquial American English. In the case of Colloquial Singaporean English, the entries almost always transformed by
Ahmet Üstün, Arianna Bisazza, Gosse Bouma, Gertjan van Noord, and Sebastian Ruder. 2022. Hyper-x: A unified hypemneworr for multi-task multilingual transfer.
A Alignment losses
As attempt explain the closeness in perfor- mance in our alignment objective and TADA's alignment objective, we take a closer look at TADA's morphosyntactic alignment objective. TADA solves the alignment problem via adversar- ial training, where the critic optimizes:
is difficult in prac- tice get a precise estimate of these differences dialect variations do not fit in deterministic baskets, instead different features are utilized at different rates.

---

Models
CollSgE STS-B
Source Dialects
1l dist
Coverage
Performance
0.9
CapeE, FijiAE.
0.228
0.866
87.97
STS-8 Performance
FijiBE. MalaE
85,
JamE, aave.
0.287
0.815
87.84
AppE. FijiBE
88
JamE, CupelE,
0.231
0.837
88.03
07.3 98
MaltE, AbEng
JamE, FijiAE,
0.238
0.844
87.99
Coverage
IndSAE, AbEng
SriLE, aave,
0.257
0.871
87.98
0.75
MalaE, AbEng
SriLE, IndE,
0.246
0.675
87.87
AppE, FijiAE SriL.E, IndF,
0.7
0.253
0.744
87.89
AppE, IndSAE
SriLE, NgH,
0.268
0.777
87.89
AppE, FijiBH MalaE, MaltE,
Distance
0.219
0.878
88.03
JamE, IndSAB
CapeE, FijiAE,
0.209
0.657
87.88
Figure 4: Performance of HyperLoRA trained dif- ferent source dialects with respect to LI distance and Coverage metric
MaltE, SriLE
Table 6: CollSgE STS-B Performance with Hyper- LoRA trained on different source dialects. We report both the LI distance and coverage metric.
Model
Approach
Params
Fine-tuning
d t x 125M d L 1.2M
MultiVALUE
Adapter
Furthermore, the applied features to the test sets are diverse. table 9, we find that across all di- alects, a large majority of rules are being applied to the test sets.
TADA Adapter
d 1.5M
HyperLoRA HyperLoRA
225K
Table 7: Computational efficiency with respect to the number of trainable parameters for MultiVAL.TT, TADA, and HyperLoRA. All these reported values use RoBERTa Base model base model. TADA cludes critic network.
C Different Source Dialects
Our ablation study focuses on few source dialect combinations. As result, drawing correlations risk being misleading given the relatively small sample of experiments we have the moment. We report additional experiments for our ablation study on the impact of source dialects in table and figure 4, In these additional experiments for CollSgE STS- B, training on source dialects with high coverage score and low L1 distance maintains overall best performance.
HyperLoRA extremely lightweight. We have experimented with more complex architectures which did not show further improvements in per- formance. We hypothesize this is due to the fact that the space of linguistic features is both simple and has low intrinsic dimension.
E LoRA Adapters
In a previous iteration of the paper, we investigated the use of Hyperformer++ (Mahabadi et al., 2021) as our hypernetoork instead of HyperLoRA. We present our results in table 10. What we find is that bottleneck adapters typically worse than LoRA adapters in the zero-shot setting.
D Computational and Parameter Efficiency
Parameter costs for HyperLoRA are reported in Ta- ble 7. d and t mark the dependence on the number of dialects and the number of tasks, respectively. We compare HyperLoRA Multi-VALUE (Ziems al., 2023b) and TADA (Held al., 2023). The Multi-VALUUE models are standard fine-tuning and adapter tuning methods applied directly to the di- alect transformed task data.

---

Percentage of Transformed Entries
Dialect
COLA
MNLI
QNLI
RTE
QQP
SST2
STS-B
Mean
AAVE
95.2
93.6
95.0
99.3
97.0
93.2
91.8
95.0
CheE
55.3
59.0
26.4
74.0
41.4
57.9
37.1
50.1
IndE
98.8
96.8
99.6
100
98.8
97.1
99.6
98.7
NgE
82.6
87.5
84.5
98.6
82.2
91.3
91.2
88.3
CollSgE
99.7
97.6
99.5
100
99.7
97.1
99.8
99.1
Table Dialectal Differences Percentage of transformed entries for each test set.
Number of Applied Features
Dialect
Total Features
COLA
MNLI
QNLI
RTE
QQP
SST2
STS-B
AAVE
118
92
109
91
86
110
89
92
CheE
30
23
28
24
25
28
22
24
IndE
90
71
85
77
68
84
74
73
NgE
45
34
42
32
35
42
34
32
CollSgE
67
58
63
54
51
63
54
55
Table 9: Dialectal Differences Number of applied transformations for each test set.
Methods
NgE Glue Performance
Model
Trainable Params.
COLA
MNLI
QNLI
RTE
QQP
SST2
STS-B
Mcan
SAE Task Adapter
0
24.7
84.6
90.8
64.2
88.2
92.0
89.5
76.2
Adapter
1.1M
23.8
83.8
89.9
66.7
86.9
91.7
89.0
75.9
LoRA
295K
25.6
84.6
90.8
65.3
88.2
92.4
89.4
76.6
Hyperforme+++
IM
20.3
83.2
87.6
63.9
88.3
91.9
88.7
74.8
HyperLoRA
225K
26.6
84.7
91.0
66.1
88.3
92.5
89.5
77.0
Table 10: Zero-shot NgE GLUE Performance RoBERTa SaaSRTa--Bae model adapters and LoRA. We compare adapter models LoRA models, and in particular, Hyperformer++ to HyperLoRA, LoRA, Adapter, Hyperformer++, and HyperLoRA are trained using our alignment objective.


Reviews:
[{"review_id": "le5s2WYwU1", "reviewer": "Reviewer_g9kY", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. Your concern on the applicability of this method to models other than RoBERTa-Base is important to us. Experiments for HyperLoRA are currently limited to encoder-only models and do not account for encoder decoder, or decoder-only models. We did not find the time to complete these experiments. We leave the evaluation of HyperLoRA and the development of dialect adaptation methods on these other architectures as future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novel to combine hypernetwork with LoRA for the domain adaptation \n2. Very clear writing, demonstration, and math expression\n3. The results demonstrate HyperLoRA can be extendable (works on unseen dialects), and very efficient (w/o slowing inference)", "reasons_to_reject": "1. The paper only works on RoBERTa-Base model and lack of the proof of scalability.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes HyperLoRA, which combines the hypernetwork for the dialect topology and low-rank adaptation (LoRA) for efficient fine-tuning, to implement efficient dialect adaptation with improved generalization and good results on zero-shot dialect experiments.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "E99aQU2kaI", "reviewer": "Reviewer_i59w", "summary": "", "questions": "Question A) Related to my main point in Reasons To Reject, is there additional evidence you can provide that testing on synthetic data in this case still allows us to draw conclusions about a hypothetical real-world scenario?\n\nQuestion B) The introduction mentions dialects without standardized writing systems as a specific use-case where zero-shot methods are especially beneficial. Are any of the dialects you experimented with examples of this? I imagine that those with non-standardized writing systems may have lower token overlap with SAE, which could make it more difficult to leverage the LLM's pretraining.\n\nQuestion C) Your technique is clearly applicable in the setting where no labeled data is available, but what about when a small amount is? If your method is able to make use of some labeled examples, it would be interesting to see that compared to Figure 3.\n\nQuestion D) Did you investigate how much your various test sets were actually modified by applying Multi-VALUE? There are many ways to do so; one would be calculating a BLEU score of the transformed corpus against the SAE corpus. I think this could be useful to get an approximate idea of available headroom: if only p% of examples were actually modified by Multi-VALUE then I think the theoretical headroom over an SAE baseline should be p% absolute accuracy, and probably much less in practice if the modifications were minor. Also, a low change rate may indicate that some aspect of the test sets (such as domain, topic, or possibly the task itself) doesn't have real-world value to native speakers: for example, if dialectal differences only show up in colloquial text, then there's little-to-no value testing on formal text.\n\nQuestion E) In Table 2, how does HyperLoRA have fewer parameters than LoRA? My understanding was that your LoRA model uses a single set of matrices for all dialects, while your HyperLoRA model uses a a set of hypernetworks whose output shape is the same as the LoRA matrices. It's possible I've misunderstood the architecture, but it seems to me that you would need strictly more parameters for HyperLoRA.\n\nQuestion F) Could you provide more detail about the feature representation for the expert dialectal knowledge? I looked at eWAVE and it appears to have 235 features, but each can take five values (that I saw). How do you encode these as input to the model, and for your coverage metric?\n\nQuestion G) Table 5 seems to feature multiple dialects never mentioned in the text. It would be good to include their names somewhere and reword the Datasets portion of Section 4 to indicate that you don't exclusively use the 5 mentioned dialects.\n\nQuestion H) It seems that the three rows in Table 5 are not the only combinations you tested; are there any useful summary statistics you could calculate over the whole set of experiments? For example, instead of only providing two anecdotal combinations, it would be interesting to see how well L1 distance and Coverage each correlate with downstream performance.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the effort you put in writing this detailed and thoughtful review. We really appreciate the questions and your interest in our work. Based on your feedback, we will revise the manuscript to better address your concerns on drawing conclusions about real-world performance from synthetic data and on the presence of dialectal differences in our test sets. First, we provide a detailed explanation and cite the authors of Multi-VALUE [1] on why it is reasonable to draw conclusions about modeling choices on real-world performance in the datasets section of our experimental setup. Secondly, we include additional statistics on the presence of dialectal differences in all test sets in the appendix.\n\nYour “single largest issue” highlights the gap in the current literature of a rich and diverse natural dataset to benchmark dialect robustness for English. While the use of pseudo-dialects can fail to exactly reflect real world performance, it does not impede on drawing conclusions about modeling choices. We see our paper as a first step towards providing some dialectal robustness and we encourage the community to improve the toolbox around dialectal variations. Whenever possible, we urge users to properly validate user-facing models with real-world in-domain data.\n\n**Question A: evidence that testing on synthetic data still allows us to draw conclusions about a hypothetical real-world scenario.**\n\nThe use of synthetic data generated by Multi-VALUE to draw conclusions about modeling choices for dialectal performance is justified [1]. First of all, the authors of Multi-VALUE find that modifications to the model resulting in performance improvements on the synthetic data of Indian English and Chicano English also improve results on parallel gold standard natural dialect data provided by native speakers of Indian English and Chicano English. This implies that the results from testing on synthetic data are directionally correct, allowing us to draw conclusions about real-world performance. \n\nFurthermore, Multi-VALUE synthetic data has been validated by native speakers to be representative of their unique grammar constructions. (1) The dialect features found in Multi-VALUE are based on eWAVE, a database of observed morphosyntactic variations in 77 english varieties, compiled by a team of 84 professional linguists from descriptive materials, naturalistic corpus data, and native speaker knowledge [2]. This grounds Multi-VALUE in appropriate expert knowledge directly extracted from native speakers and naturalistic texts. (2) Multi-VALUE validated 92 of these rules by a group of 72 native speakers. With above 81% accuracy validated by native speakers and on average 86.6% implemented features per dialect, Multi-VALUE provides us with a reasonable level of reliability to draw conclusions on real-world dialect performance.\n\nWe thank the reviewer again for pointing out this key concern and we will update the paper to better address this concern.\n\n[1] Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023. Multi-VALUE: A Framework for Cross-Dialectal English NLP. ACL 2023.\n\n[2] Bernd Kortmann, Kerstin Lunkenheimer, and Katharina Ehret, editors. 2020. eWAVE.\n\n**Question B: The introduction mentions dialects without standardized writing systems as a specific use-case where zero-shot methods are especially beneficial.**\n\nOur approach does not address potential orthographic variations that arise from non-standard writing systems. As you point it out, these orthographic variations, much like lexical variations, give rise to additional difficulties in harnessing language technologies. To this effect, we revise this part of the introduction to focus on non-standard grammars. Our work is an important first step in quickly adapting to these new standards which vary in their grammatical rules. \n\nTo answer your question, all dialects in our experiments have standard writing systems.\n\n**Question C: Your technique is clearly applicable in the setting where no labeled data is available, but what about when a small amount is?**\n\nThis is an insightful remark, we thank you for the suggestion. At the moment, we focus on highlighting the use of expert knowledge encoded in dialect features, but we will be happy to extend the applicability of our work to small amounts of dialect data in the future. \n\nOne way this can be done is by adding small amounts of data from many dialects to the HyperLoRa training set. We didn’t have time to add the experiment here, but the hypernetwork flexibly allows you to mix any amount of data from any number of dialects.\n\n**Question D: It is not clear that the test sets contain enough dialectal differences to estimate real-world benefit to native speakers.**\n\nIn the table below, you can find the % of total entries in the test sets which have been modified by Multi-VALUE. On average, for each dialect, we have over 88% transformed entries except for Chicano English. This is expected, as Chicano English shares many similarities with Colloquial American English. In the case of Colloquial Singaporean English, the entries are almost always transformed by Multi-VALUE. It is difficult in practice to get a precise headroom estimate as dialect variations do not fit in deterministic baskets, instead different features are utilized at different rates. In our case, a large proportion of the test set is affected, and we expect theoretical headroom estimates to be large overestimates.\n\n| | CoLA (%) | MNLI (%) | QNLI (%) | RTE (%) | QQP (%) | SST2 (%) | STSB (%) | Mean (%) |\n| --- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| AAVE | 95.2 | 93.6 | 95.0 | 99.3 | 97.0 | 93.2 | 91.8 | 95.0 |\n| ChcE | 55.3 | 59.0 | 26.4 | 74.0 | 41.4 | 57.9 | 37.1 | 50.1 |\n| IndE | 98.8 | 96.8 | 99.6 | 100 | 98.8 | 97.1 | 99.6 | 98.7 |\n| NgE | 82.6 | 87.5 | 84.5 | 98.6 | 82.2 | 91.3 | 91.2 | 88.3 |\n| CollSgE | 99.7 | 97.6 | 99.5 | 100 | 99.7 | 97.1 | 99.8 | 99.1 |\n\nFurthermore, the applied features across the test set are diverse. In the following table, you can find the number of processed rules per dialect and task. Across all dialects, a large majority of rules are being applied to the test sets.\n\n| | Total Feat | CoLA | MNLI | QNLI | RTE | QQP | SST2 | STSB |\n| --- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| AAVE | 118 | 92 | 109 | 91 | 86 | 110 | 89 | 92 |\n| ChcE | 30 | 23 | 28 | 24 | 25 | 28 | 22 | 24 |\n| IndE | 90 | 71 | 85 | 77 | 68 | 84 | 74 | 73 |\n| NgE | 45 | 34 | 42 | 32| 35 | 42 | 34 | 32 |\n| CollSgE | 67 | 58 | 63 | 54 | 51 | 63 | 54 | 55 |\n\nYour comment highlighting the dependence of feature rates on the specific context is important. For instance, Nigerian pidgin is typically used in informal conversation, whereas Nigerian English is vastly used in politics, science, and media. Our work takes a first step towards task-agnostic and resource-efficient dialect adaptation, we think that further adaptation to topical and register shifts can be an interesting area of future research. \n\n**Question E: In Table 2, how does HyperLoRA have fewer parameters than LoRA?**\n\nYour understanding of HyperLoRA is correct, the total number of parameters in HyperLoRA is strictly greater than LoRA by the number of parameters of the hypernetwork. This is actually a typo from an earlier version where we wanted to report the number of trainable parameters. This will be removed in the revision to avoid confusion.\n\n**Question F: Could you provide more detail about the feature representation for the expert dialectal knowledge?**\n\nFor each of those 235 features, eWAVE specifies different heuristic probabilities: 100% for obligatory features, 60% for features neither pervasive nor rare, 30% for rare features and 0% for no information or attested absence. These probabilities are used by MultiVALUE to generate synthetic dialect data. We directly use these probabilities to encode expert dialectal knowledge. For both the model and our coverage metric these feature vectors take values in {1, 0.6, 0.3, 0}^{# features}. We will revise the manuscript to include these details.\n\n**Question G: Table 5 seems to feature multiple dialects never mentioned in the text.**\n\nThank you for catching this! We will carefully mention the dialects used in this ablation study in our dataset section in the revised manuscript.\n\n**Question H: are there any useful summary statistics you could calculate over the whole set of experiments?**\n\nWe agree that it would be interesting to visualize potential correlation between downstream performance and the two metrics. However, such correlations risk being misleading given the relatively small sample of experiments we have at the moment. We will finalize these experiments by the end of the discussion period and adjust the appendix according to our findings.\n\n** Other **\n\nThank you for catching those issues with our grammar and presentation style! We have revised the paper to address these stylistic problems. \n- prior works on dialect struggle\n- a, b: these are typos (should be \\alpha, \\beta respectively)\n- \\mathcal{X}, \\mathcal{Y}: feature spaces for last layer dialect and SAE representations respectively\n- Following your comment, we have removed the color-coding and simply mention that “The cost of training scales linearly with the number of annotated samples”\n\nIn addressing these questions, we hope that we were able to bring clarity to important issues you had. We would be happy to answer any additional concerns and we greatly appreciate any further feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Dialect data is generally scarce, so zero-shot methods like this one are widely applicable.\n- Dialects are, in my opinion, under-researched.\n- The baselines and ablations are thoughtfully constructed and thorough.\n- The application of hypernetworks to LoRA appears to be novel.\n- The paper acknowledges its own limitations and includes a thoughtful ethics statement, which should be encouraged.", "reasons_to_reject": "- The test data is synthetic, generated by a rule-based system; we can't be sure that any of the results (baseline or experimental) accurately reflect performance on real data gathered from native speakers of these dialects. Lines 276-280 and the Limitations section discuss this, but in my opinion the discussion is inadequate. I would need to see citations and a specific, well-reasoned explanation for why it is valid to draw conclusions about real-world performance based on synthetic test data. This is my single largest issue with this paper.\n- Many of the results appear to be disappointing; there is not a clear trend of HyperLoRA significantly improving over baseline performance.\n- It is not clear that the test sets contain enough dialectal differences to estimate real-world benefit to native speakers.\n- Some details about the method were unclear from the text (though I acknowledge that the promised public code release mitigates this somewhat).", "questions_to_authors": "Question A) Related to my main point in Reasons To Reject, is there additional evidence you can provide that testing on synthetic data in this case still allows us to draw conclusions about a hypothetical real-world scenario?\n\nQuestion B) The introduction mentions dialects without standardized writing systems as a specific use-case where zero-shot methods are especially beneficial. Are any of the dialects you experimented with examples of this? I imagine that those with non-standardized writing systems may have lower token overlap with SAE, which could make it more difficult to leverage the LLM's pretraining.\n\nQuestion C) Your technique is clearly applicable in the setting where no labeled data is available, but what about when a small amount is? If your method is able to make use of some labeled examples, it would be interesting to see that compared to Figure 3.\n\nQuestion D) Did you investigate how much your various test sets were actually modified by applying Multi-VALUE? There are many ways to do so; one would be calculating a BLEU score of the transformed corpus against the SAE corpus. I think this could be useful to get an approximate idea of available headroom: if only p% of examples were actually modified by Multi-VALUE then I think the theoretical headroom over an SAE baseline should be p% absolute accuracy, and probably much less in practice if the modifications were minor. Also, a low change rate may indicate that some aspect of the test sets (such as domain, topic, or possibly the task itself) doesn't have real-world value to native speakers: for example, if dialectal differences only show up in colloquial text, then there's little-to-no value testing on formal text.\n\nQuestion E) In Table 2, how does HyperLoRA have fewer parameters than LoRA? My understanding was that your LoRA model uses a single set of matrices for all dialects, while your HyperLoRA model uses a a set of hypernetworks whose output shape is the same as the LoRA matrices. It's possible I've misunderstood the architecture, but it seems to me that you would need strictly more parameters for HyperLoRA.\n\nQuestion F) Could you provide more detail about the feature representation for the expert dialectal knowledge? I looked at eWAVE and it appears to have 235 features, but each can take five values (that I saw). How do you encode these as input to the model, and for your coverage metric?\n\nQuestion G) Table 5 seems to feature multiple dialects never mentioned in the text. It would be good to include their names somewhere and reword the Datasets portion of Section 4 to indicate that you don't exclusively use the 5 mentioned dialects.\n\nQuestion H) It seems that the three rows in Table 5 are not the only combinations you tested; are there any useful summary statistics you could calculate over the whole set of experiments? For example, instead of only providing two anecdotal combinations, it would be interesting to see how well L1 distance and Coverage each correlate with downstream performance.", "paper_topic_and_main_contributions": "This paper proposes HyperLoRA, a parameter-efficient technique to adapt LLMs to unseen dialects of English using a feature vector obtained from expert knowledge instead of any annotated data in the target dialect. The core technique is using hypernetworks to generate dialect-specific LoRA modules in a way that can be applied to unseen dialects. Similarly to LoRA, the technique is widely applicable.\n\nThe paper provides a number of baselines and ablations that test various hypothetical benefits of their approach.\n\nThe authors have promised to publicly release their code. \n\nThe paper takes steps to call out and address some limitations, and includes a thoughtful ethics statement.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "In line 12 of the abstract, \"prior work on dialect struggle\" should either use \"works\" or \"struggles\".\n\nIn the equation on line 302, what are a, b, curly X, and curly Y don't appear to be defined anywhere.\n\nThe color-coding of the points in Figure 3 seem to repeat information encoded by their location on the x-axis. Consider removing the color-coding; also note that it will not be noticeable in black-and-white."}}, {"review_id": "57CJgs8Pu5", "reviewer": "Reviewer_ZQT3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the acknowledgement of HyperLoRA’s role in “alleviating the need for human-annotated dialectal data”, for recognizing that our ablation study “simplifies the applicability and broadens the scope” of our method, and for your appreciation of our contributions.\n\nWe agree with you that the paper should be self-contained. To this effect, we will revise (section 3.2) to include HyperNetworks and Low Rank Adaptation. And we elaborate:\n\n> We leverage hypernetworks for Low-Rank Adaptation .“LoRA [1] is a fine-tuning approach that keeps the full model parameters fixed and instead updates a low-rank decomposition of the attention matrices. Instead of updating LoRA weights directly, our approach learns the weights of a hypernetwork [2], which is then used to generate the appropriate LoRA weights.”\n\n[1] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.\n\n[2] David Ha, Andrew Dai, and Quoc V. Le. 2016. Hypernetworks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) Experimental Setup: The experimental section is detailed and the hyperparameters are clearly specified. The results are tested for statistical significance and the code will be released.\n\n2) The method alleviates the need for human-annotated samples which is relatively harder to obtain and presents a way to leverage linguistic features instead. In general, the method of leveraging hypernetworks to learn LoRa weights is unique to the best of my knowledge.\n\n3) Ablations and discussion: The ablations section has interesting insights, for example, Section 6.2 where the paper discusses how a larger feature coverage and proximity to a source dialect improves performance. While intuitive, its nice to quantify it and confirm the intuition. It also simplifies the applicability of the method and broadens its scope. ", "reasons_to_reject": "I don't have major reasons of rejection and these are minor points : \n\n1) Related Work & Writing: The related work section might benefit by including information on hypernetworks and LoRA (maybe in the transfer learning section?). Even though I know of these concepts, it took revisiting to understand the framework presented in the paper. In general, its a good idea to make the paper self-contained, and including information strictly relevant to the paper might help with that. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper deals with the task of adapting language models to new unseen dialects in a resource-efficient way. The method uses hypernetworks to learn a mapping from linguistic dialectal features to LoRA adapter weights during training. By doing this, it alleviates the need for human-annotated dialectal data. The paper evaluates the developed framework on 5 dialectal variations of the GLUE benchmark and uses pre-trained RoBERTa as its backbone. The framework is mostly comparable or slightly better than using task adapters, and is resource-efficient in that it does not need human-annotated data. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HF5J1j9TcH", "reviewer": "Reviewer_wWbU", "summary": "", "questions": "1) Can you provide more insights into the limitations and potential challenges of the proposed method, especially regarding the use of pseudo-dialects and the coverage of linguistic features, what kind of risk might have with the pseudo-dialects?\n\n2) Have you explored the effectiveness of HyperLoRA on models with different architectures? It would be helpful to improve the depth of the ablation studies to evaluate the contribution of each component of the model.\n", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback and for highlighting in detail the strengths of our paper: “addresses an important problem”, “the approach is innovative”, “promising results”, and “comprehensive analysis”.\n\n**Question 1: Can you provide more insights into the limitations and potential challenges of the proposed method, especially regarding the use of pseudo-dialects and the coverage of linguistic features, what kind of risk might have with the pseudo-dialects?**\n\nPseudo-dialects obtained via Multi-VALUE [1] are limited to variations in syntactic structure and morphology. This presents important challenges when the dialect of interest contains significant lexical variations. This is the case of pidgins and creoles where it is common to borrow words from another language, or to use slang. As these words are not featured in the vocabulary of these pseudo-dialects. \n\nBecause Multi-VALUE is compiled via oral interviews with native speakers, the employed orthographic conventions by linguists may not accurately represent the speaker’s own conventions. This matter becomes even more complex when the dialect is not standardized.\n\nThese unaccounted variations create discrepancies in the performance of pseudo-dialects and real dialects. We encourage the development of better tools to account for these differences. Whenever possible, we urge users to properly validate user-facing models with real-world data.\n\n[1] Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023. Multi-VALUE: A Framework for Cross-Dialectal English NLP. ACL 2023.\n\n**Question 2: Have you explored the effectiveness of HyperLoRA on models with different architectures? It would be helpful to improve the depth of the ablation studies to evaluate the contribution of each component of the model.**\n\nThe applicability of this method to models other than RoBERTa-Base is important to us and we understand that it currently is a limitation in our work. Experiments for HyperLoRA are currently limited to encoder-only models and do not account for encoder decoder, or decoder-only models. We did not find the time to complete these experiments. We leave the evaluation of HyperLoRA and the development of dialect adaptation methods on these other architectures as future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper addresses an important problem of dialectal diversity in language models, as the disproportionality in the representation of Standard American English in LLMs is a well-known issue and it's crucial for ensuring inclusivity and fairness in natural language processing applications.\n\n2) The proposed method, HyperLoRA, leverages expert knowledge and hyper-networks to achieve resource-efficient adaptation to unseen dialects. This approach is innovative and shows promising results and effectively deals with the scarcity of annotated dialect data.\n\n3) The paper provides a solid evaluation of HyperLoRA on the GLUE benchmark, demonstrating its effectiveness in improving dialectal robustness across multiple tasks and dialects, the comparison with relevant baselines strengthens the validity of the proposed method.\n\n4) Comprehensive analysis of the method's task-agnostic nature and its implications for scalability and generalizability to unseen dialects.\n\n5) The paper is well-written and clearly presents the problem, methodology, and experimental results. The Figures are drawn in a nice clear style.\n\n", "reasons_to_reject": "1) The paper lacks a comprehensive comparison with other dialect adaptation methods, which could provide a better understanding of the state-of-the-art in this area.\n2) The paper does not explore the effectiveness of HyperLoRA on models with different architectures. Ablation study is not conducted to evaluate the contribution of each component, which could limit the generalizability of the proposed method.\n", "questions_to_authors": "1) Can you provide more insights into the limitations and potential challenges of the proposed method, especially regarding the use of pseudo-dialects and the coverage of linguistic features, what kind of risk might have with the pseudo-dialects?\n\n2) Have you explored the effectiveness of HyperLoRA on models with different architectures? It would be helpful to improve the depth of the ablation studies to evaluate the contribution of each component of the model.\n", "paper_topic_and_main_contributions": "The paper addresses the problem of dialectal diversity in large language models (LLMs) and proposes a task-agnostic adaptation method called HyperLoRA. The main contribution of the paper is the use of expert linguistic knowledge and hyper-networks to enable resource-efficient adaptation (by LoRA) to unseen dialects. The paper shows that HyperLoRA achieves competitive performance across multiple dialects in a zero-shot setting.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores how to use linguistic knowledge to enable resource-efficient adaptation via hyper-networks on 5 English dialects. The most interesting part of this paper is the usage of expert knowledge instead of labeled data. The paper sheds light on an interesting and under-studied phenomena: dialects. In general the reviewers agree that the proposed method is strong, as well as the experimental setup and the used baselines. Additionally ablation studies and a in-depth analysis was also performed. 

The limitations of this paper are: that the method is only tested on RoBERTa-base (g9kY);  that it is tested on synthetic data and that this dataset is not well describe and might be insufficient to measure the linguistical diversity (i59w) ; and a lack of proper comparison with other dialectic adaptation methods (wWbU). 

Overall the reviewers have strong scores for this paper. However, I would also take in consideration the weaknesses pointed out.
Decision: Accept-Main