Venue: NAACL 2025

# Predicting the Target Word of Game-playing Conversations using a

Low-Rank Dialect Adapter for Decoder Models

 Dipankar Srirag\({}^{\circledR}\)  Aditya Joshi\({}^{\circledR}\)  Jacob Eisenstein\({}^{\circledR}\)

University of New South Wales, Sydney   Google DeepMind

{d.srirag, aditya.joshi}@unsw.edu.au jeisenstein@google.com

###### Abstract

Dialect adapters that improve the performance of LLMs for NLU tasks on certain sociolects/dialects/national varieties ('dialects' for the sake of brevity) have been reported for encoder models. In this paper, we extend the idea of dialect adapters to decoder models in our architecture called LoRDD. Using MD-3, a publicly available dataset of word game-playing conversations between dialectal speakers, our task is Target Word Prediction (TWP) from a masked conversation. LoRDD combines task adapters and dialect adapters where the latter employ contrastive learning on pseudo-parallel conversations from MD-3. Our experiments on Indian English and Nigerian English conversations with two models (mistral and gamma) demonstrate that LoRDD outperforms four baselines on TWP. Additionally, it significantly reduces the performance gap with American English, narrowing it to 12% and 5.8% for word similarity, and 25% and 4.5% for accuracy, respectively. The focused contribution of LoRDD is in its promise for dialect adaptation of decoder models using TWP, a simplified version of the commonly used next-word prediction task.

## 1 Introduction

Dialect adaptation of language models refers to approaches that improve their performance for different dialects of a language Joshi et al. (2025). Past work proposes dialect adaptation for encoder models Held et al. (2023); Xiao et al. (2023) or encoder-decoder models Liu et al. (2023). This paper extends it to decoder models, via a novel architecture called **L**ow-**R**ank **D**iacl robustness for **D**ecoder Models (LoRDD). To demonstrate the effectiveness of LoRDD, we use MD-3 Eisenstein et al. (2023), a dataset of manually transcribed dialectal dialogues between speakers of either Indian English (en-IN) or Nigerian English (en-NG) or US English (en-US) playing the word-guessing game of taboo1. We select MD-3 conversations where the guesser correctly identifies the target word/phrase ('target word' for the sake of brevity) and mask the target word (using [MASK]; as shown in Figure 1). Our task then is to predict the target word in a masked conversation, _i.e._, target word prediction (TWP). TWP represents a simplified version of next-word generation utilised by decoder models. Since decoder models are adopt in tasks involving causal language modeling, TWP is a reasonable task choice. Upon observing that the TWP performances for en-IN and en-NG are lower than those of en-US, the objective of LoRDD is to improve the TWP performances for en-IN and en-NG. LoRDD employs a combination of two LoRA-based Hu et al. (2022) adapters. The first is a task-specific adapter that uses instruction fine-tuning Wei et al. (2022) on an augmented set of en-US and en-IN/en-NG conversations. The second is a dialect adapter that uses contrastive learning on a pseudo-parallel corpus between en-US and

Figure 1: Illustrative example of Target Word Prediction on an en-IN conversation. The inaccurate output from the in-dialect fine-tuned model (left) is corrected by the model trained using LoRDD (right).



-en-IN/en-NG conversations about a specific target word. We release the code for training LoRDD adapters on Github.

Our work is novel in two ways: (A) LoRDD is the first methodology for dialect adaptation of decoder models, and outperforms one in-dialect and three cross-dialect baselines, (B) We leverage an existing dataset MD-3 to create a pseudo-parallel corpus of natural dialectal conversations, as opposed to past work that relies on synthetically transformed dialectal corpora.

## 2 Architecture of LoRDD

The architecture of LoRDD employs two parameter-efficient adapters: task adapter and dialect adapter, as shown in Figure 2.

### Task Adapter

We define \({\bf x}\) and \({\bf t}\) as lists of tokens in the masked conversation and the target word respectively. For a batched input of \(N\) pairs of masked conversations and corresponding target words, we train the task adapters to output the correct target word using maximum likelihood estimation - a standard learning objective for causal language modeling Jain et al. (2023).

\[\mathcal{L_{\textbf{Task}}}=-\frac{1}{N}\sum_{j=1}^{N}\left\{\sum_{i=|{\bf x} ^{j}|+|{\bf t}^{j}|}^{|{\bf x}^{j}|}\log p({\bf x}^{j}_{i}|{\bf x}^{j}_{<i})\right\}\]

Here, \({\bf x}^{j}_{<i}=[{\bf x}^{j}_{1},\ldots,{\bf x}^{j}_{i-1}]\) denotes the subsequence before \({\bf x}^{j}_{i}\) and \(|\cdot|\) is the number of tokens.

### Dialect Adapter

To train the dialect adapter, we use a pseudo-parallel corpus between en-US and en-IN/en-NG conversations. This corpus consists of both positive and negative pairs of masked conversations. We consider a masked conversation pair as a positive example if both conversations pertain to the same target word, and a negative example if they pertain to a different target word. We then perform contrastive learning between the frozen representation of the masked en-US conversation ([MASK]\({}_{\text{US}}\)) and the trainable representation of the masked en-IN/en-NG conversation ([MASK]\({}_{\text{X}}\)), using cosine embedding loss. This allows the adapters to learn from both positive and negative examples present in the pseudo-parallel corpus.

Here, X represents dialect in focus (either en-IN or en-NG), sim(\(\cdot\)) calculates the cosine similarity, 'd' is the margin, and 'y' is the label (1 for a positive example, and -1 otherwise).

In contrast to the task adapter, the dialect adapter is trained to output standard dialect representations for an input text. Hence, LoRDD stacks the task adapter on top of the dialect adapter (as shown in Figure 2), allowing the models to predict the target word as required for TWP.

## 3 Experiment Setup

We experiment with two open-weight de-coder models namely, Mistral-7B-Instruct-v0.2 (mistral; Jiang et al., 2023) and Gemma2-9B-Instruct (gemma; Geuma Team, 2024). LoRDD is trained as follows:

* The task adapter is trained by fine-tuning the model for 20 epochs, with a batch size of 32, Paged 8-bit AdamW Dettmers et al. (2022) as the optimiser and learning rate of 2e-4.
* To train the dialect adapter, we perform contrastive learning for 10 epochs, with a batch size of 8, AdamW as the optimiser, a learning rate of 2e-5, and a margin of 0.25.

We inject adapter matrices at all linear layers, as recommended by Dettmers et al. (2023). Training either adapter for a single experiment takes approx. 25 minutes on an A100 GPU. We compare LoRDD with one in-dialect and three cross-dialect baselines. The in-dialect baseline involves finetuning a model on the training set of en-IN/en-NG. The cross-dialect baselines are:

#### en-Us

Fine-tune the model on train set of en-US.

In-MV/NG-MVWe use Multi-VALUE Ziems et al. (2023) to transform en-US conversations into en-IN. IN-MV is fine-tuned on these synthetically created conversations.

\begin{table}
\begin{tabular}{l c c c} \hline \hline Subset & Train & Valid & Test \\ \hline en-US & 62 & 41 & 311 \\ en-IN & 31 & 21 & 160 \\ en-NG & 38 & 25 & 194 \\ IN-MV & 57 & 39 & 296 \\ NG-MV & 57 & 39 & 296 \\ IN-TR & 25 & 17 & 132 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Data statistics.

 

In-TrWe prompt GPT-4 Turbo (OpenAI, 2024) to _transform_ en-IN by removing dialectal information, resulting in IN-TR, and use it to fine-tune a model.

**Note:** We do not perform similar transformations on the en-NG subset due to the high API pricing at the time of writing.

We consider the in-dialect fine-tuned model as a strong baseline, while cross-dialect models are weak baselines. We compare all baselines and LoRDD with in-dialect fine-tuned models on en-US conversations, which serves as our skyline result.

Tables 1 and 2 report the statistics of the extended MD-3 dataset and the pseudo-parallel corpus respectively. Additional details including prompt used to create TR-IN and corpus examples are in Appendix A. All evaluations are on the test set of the en-IN or en-NG subsets for the baselines and LoRDD, and on the test set of the en-US dataset for the skyline. We report two metrics: (a) Similarity (average cosine similarity between the Sentence-BERT Reimers and Gurevych (2019) embeddings of the reference and generated target word); and (b) Accuracy (the proportion of conversations where the model generates the correct target word).

## 4 Evaluation

Our results address three questions: (a) What is the current gap in the task performance between en-US and en-IN/en-NG?; (b) How well does LoRDD help bridge the gap?; (c) How essential is each component in LoRDD to bridge the gap?

Table 3 compares the performance of LoRDD with the baselines and the skyline. On the similarity and accuracy, LoRDD achieves average scores of 59.9 and 35.7, respectively, when evaluated on en-IN, and 63.5 and 41.9, respectively, when evaluated on en-NG. On average, LoRDD improves on the performances of the en-IN in-dialect baseline by 13.4% on similarity and 28.1% on accuracy. Similarly, it improves on the en-NG in-dialect baseline by 11.4% on similarity and 33.8% on accuracy.

As expected, the skyline achieves the highest performance for the task. However, LoRDD significantly narrows the initial performance gaps. For en-IN, the gap in similarity is reduced from 27.3% to 12%, and the gap in accuracy is reduced from 64.7% to 25%. For en-NG, the gap in similarity is reduced from 17.9% to 5.8%, and the gap in accuracy is reduced from 43.1% to 4.5%.

Table 4 shows the results from an ablation study that evaluates both adapters in LoRDD. We compare LoRDD with three variants: (a) the dialect adapter trained on other parallel corpora, (b) LoRDD without the dialect adapter, within which we also compare, (c) the task adapters trained on other augmented data. Compared to LoRDD, all other variants report a degradation in their performances. Training the dialect adapter on synthetic parallel corpora (en-US II IN-MV, en-IN I IN-TR and en-US II NG-MV) results in degradation ranging from 1.0 to 2.3 on similarity and 2.5 to 4.8

\begin{table}
\begin{tabular}{l c c c} \hline \hline llCorpus & Samples & Positive & Negative \\ \hline en-US II en-IN & 144 & 11 & 133 \\ en-US II en-NG & 168 & 13 & 155 \\ en-US II RN-MV & 197 & 97 & 100 \\ en-US II NG-MV & 197 & 97 & 100 \\ en-IN NI IN-TR & 142 & 42 & 100 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Data statistics of the pseudo-parallel corpus.

Figure 2: Architecture of LoRDD.

 

on accuracy. Removing the dialect adapter results in a further degradation ranging from 1.5 to 8.7 on similarity and 3.0 to 12.2 on accuracy. The worst-performing variants are the models that only train the task adapter on synthetically augmented data (en-US + IN-MV, en-IN + IN-TR and en-IN + NG-MV). While the degraded performances of these models show the importance of the dialect adapter, the lower performances on variants involving synthetic conversations further solidify the use of natural conversations in LoRDD. We provide additional results, such as ablations on proportion of conversations in augmented data, in Appendix B.

Finally, we manually analyse erroneous en-IN instances from LoRDD, and categorise them into types of en-IN dialect features given by Lange (2012) and Demszky et al. (2021). Figure 3 shows that extraneous article (_"It's @ one word"_) is the most common feature associated with these conversations. The definitions of all identified dialect features with examples are in Table 5.

**Note:** We do not perform error analysis for en-NG instances due to lack of similar labelled features for the dialect.

\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{Method} & \multirow{2}{*}{Training Data} & \multirow{2}{*}{llCorpus} & \multicolumn{2}{c}{mistral} & \multicolumn{2}{c}{gamma} & \multicolumn{2}{c}{\(\mu\)} \\ \cline{4-9}  & & Similarity & Accuracy & Similarity & Accuracy & Similarity & Accuracy \\ \hline \hline \multicolumn{9}{c}{(a) Tested on en-IN} \\ \hline LoRDD & en-US + en-IN & en-US \(\parallel\) en-NG & **55.9** & **30.0** & **63.9** & **41.3** & **59.9** & **35.7** \\ \hline \multirow{2}{*}{\(\leftrightarrow\)LoRPD} & en-US + en-IN & en-US \(\parallel\) IN-MV & 55.6 & 28.1 & 62.0 & 37.5 & 58.8 (1.1) & 32.8 (2.9) \\  & en-US + en-IN & en-IN \(\parallel\) IN-TR & 54.9 & 27.5 & 62.8 & 38.8 & 58.9 (1.0) & 33.2 (2.5) \\ \hline \multirow{2}{*}{\(-\mathcal{L}_{\textbf{oial}}\)} & en-US + en-IN & 54.4 & 26.9 & 62.3 & 37.5 & 58.4 (1.5) & 32.2 (3.5) \\  & en-IN + IN-MV & Not Used & 51.6 & 23.1 & 57.1 & 31.9 & 54.4 (5.5) & 27.5 (8.2) \\  & en-IN + IN-TR & & 44.8 & 18.1 & 57.5 & 28.8 & 51.2 (8.7) & 23.5 (12.2) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Ablation on LoRDD based on parallel corpus (\(\leftrightarrow\) llCorpus), dialect adapter (\(\mathcal{L}_{\textbf{oial}}\)) and data augmentation. For each model, we report Similarity and Accuracy when tested on (a) en-IN and (b) en-NG. The best performance is shown in **bold**. \(\mu\) is the average of the metrics across both models. The degradation on the ablations compared to LoRDD is shown in (number).

\begin{table}
\begin{tabular}{c l c c c c c c c} \hline \hline \multirow{2}{*}{Method} & \multirow{2}{*}{Training Data} & \multirow{2}{*}{llCorpus} & \multicolumn{2}{c}{mistral} & \multicolumn{2}{c}{gamma} & \multicolumn{2}{c}{\(\mu\)} \\ \cline{4-9}  & & Similarity & Accuracy & Similarity & Accuracy & Similarity & Accuracy \\ \hline \hline \multicolumn{9}{c}{(a) Tested on en-IN} \\ \hline LoRDD & en-US + en-IN & en-US \(\parallel\) en-IN & **55.9** & **30.0** & **63.9** & **41.3** & **59.9** & **35.7** \\ \hline \multirow{2}{*}{\(\leftrightarrow\)LoRPD} & en-US + en-IN & en-US \(\parallel\) IN-MV & 55.6 & 28.1 & 62.0 & 37.5 & 58.8 (1.1) & 32.8 (2.9) \\  & en-US + en-IN & en-IN \(\parallel\) IN-TR & 54.9 & 27.5 & 62.8 & 38.8 & 58.9 (1.0) & 33.2 (2.5) \\ \hline \multirow{2}{*}{\(-\mathcal{L}_{\textbf{oial}}\)} & en-US + en-IN & 54.4 & 26.9 & 62.3 & 37.5 & 58.4 (1.5) & 32.2 (3.5) \\  & en-IN + IN-MV & Not Used & 51.6 & 23.1 & 57.1 & 31.9 & 54.4 (5.5) & 27.5 (8.2) \\  & en-IN + IN-TR & & 44.8 & 18.1 & 57.5 & 28.8 & 51.2 (8.7) & 23.5 (12.2) \\ \hline \hline \multicolumn{9}{c}{(b) Tested on en-NG} \\ \hline LoRDD & en-US + en-NG & en-US \(\parallel\) en-NG & **62.4** & **40.5** & **64.5** & **43.2** & **63.5** & **41.9** \\ \hline \multirow{2}{*}{\(\leftrightarrow\) ILCorpus} & en-US + en-NG & en-US \(\parallel\) EN-M & 60.4 & 35.6 & 61.9 & 38.5 & 61.2 (2.3) & 37.1 (4.8) \\  & en-US + en-NG & & 61.3 & 39.7 & 62.4 & 38.1 & 61.9 (1.6) & 38.9 (3.0) \\ \hline \multirow{2}{*}{\(-\mathcal{L}_{\textbf{oial}}\)} & en-IN + NG-MV & Not Used & 58.6 & 33.6 & 60.7 & 33.1 & 59.7 (3.8) & 33.4 (8.5) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Performance comparison between the skyline, baselines and LoRDD on TWP. For each model, we report Similarity and Accuracy when tested on (a) en-IN and (b) en-NG. \(\mu\) is the average of the metrics across both evaluation models. LoRDD (represented in **bold**) improves the performance on all baselines. The percentage improvement over the in-dialect baseline and the percentage degradation compared to the skyline are shown in (number) and (number) respectively.

 



## 5 Related Work

Language technologies need to be equitable to dialects/sociolelects/national varieties Joshi et al. (2025); Blodgett et al. (2020). Dialect adaptation involves strategies to improve the performance of non-mainstream dialects. These strategies range from introducing dialectal information at the pretraining phase Sun et al. (2023) to adapter-based approaches. Adapters are explored to be viable and efficient in improving dialect robustness Liu et al. (2023) or cross-lingual transfer Pfeiffer et al. (2020). In particular, we derive from this line of work by training a low-rank dialect adapter like Xiao et al. (2023) using a contrastive learning objective like Held et al. (2023). While past approaches adapt encoder models, we distinguish ourselves by proposing LoRDD as an architecture to adapt decoder models. Similarly, past work uses frameworks like VALUE Ziems et al. (2022) and Multi-VALUE Ziems et al. (2023) to create synthetic dialectal variants of standard US English benchmarks. In contrast, we use a pseudo-parallel corpus of naturally occurring dialectal conversations from MD-3 Eisenstein et al. (2023). Our task of target word prediction is closely similar to Chalamasetti et al. (2023), who generate word game conversations using LLMs and evaluate their ability to predict the target word. Target word prediction is also utilised by Srirag et al. (2025), who evaluate dialect-robustness of language models using masked MD-3 conversations. Finally, our cross-dialect baselines on corpora created using Multi-VALUE and GPT-4 discuss the shortcomings of synthetic datasets for dialect adaptation for dialogues, as also noted in Faisal et al. (2024).

## 6 Conclusion

This paper focused on a simplistic causal language modeling task, called target word prediction, using masked game-playing conversations between two dialectal speakers of English (en-US, en-IN and en-NG). The task was to predict the target word from a masked conversation. From our initial experiments with fine-tuned decoder models, the in-dialect baseline (en-IN and en-NG) reported a performance degradation on TWP, when compared with the skyline (en-US). To address the gap in the case of en-IN and en-NG, we proposed LoRDD as a novel architecture using low-rank adapters. LoRDD extends past work in dialect adaptation for encoder models to decoder models by employing contrastive learning via a pseudo-parallel corpus of real conversations. LoRDD outperformed one in-dialect baseline and three cross-dialect baselines, while also bridging the gap with the skyline to 12% (down from 27.3%) and 25% (down from 64.7%) on similarity and accuracy respectively for en-IN. For en-NG, the gap is reduced to 5.8% (down from 17.9%) on similarity and 4.5% (down from 43.1%) on accuracy. Through ablation tests on LoRDD, we validated the effectiveness of its components.

Although TWP works with a restricted dataset and utilises turn-based dialogue, LoRDD sets up the promise for dialect adaptation of decoder models. Our error analysis also highlights the scope for future improvement. A potential future work is to evaluate LoRDD on other causal language modeling tasks, including seq2seq tasks, and other dialects. Similarly, an extension to LoRDD would eliminate the requirement of naturally occurring conversations in multiple dialects.

### Limitations

While previous approaches have proposed dialect adapters as task-agnostic, our study does not make the same claim. We use target word prediction as

Figure 3: Percentage count of dialect features in erroneous instances from LoRDD.



the task of predicting the last word of a conversation which was the word that the described was attempting to convey to the guesser. This task is a simplistic version of causal language modeling. However, we do not verify that LoRDD works for causal language modeling because there is no suitable parallel dataset of turn-aligned conversations, to the best of our knowledge. Held et al. (2023) use bottleneck adapters based on their ability for cross-lingual transfer, but we do not explore these types of adapters due to the lack of support for our choice of models at the time of writing the paper. The choice of en-IN and en-NG as the dialects of interest is solely based on the availability of the dataset.

## Ethics Statement

We use a publicly available dataset of conversations consisting of human players engaged in a game of taboo. The topics discussed in the dataset are fairly general and are unlikely to cause distress. One of the authors of the paper performed the error analysis. The synthetic conversation created using GPT-4 may contain biased output, arising due to the properties of the model. We do not expect any reasonably significant risks arising as a result of the project.

## References

* S. L. Blodgett, S. Barocas, H. Daume III, and H. Wallach (2020)Language (technology) is power: A critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 5454-5476. External Links: Link Cited by: SS1.
* K. Chalamalasetti, J. Gotze, S. Hakimov, B. Madureira, P. Sadler, and D. Schlangen (2023)CLDBench: Using game play to evaluate chat-optimized language models as conversational agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 11174-11219. External Links: Link Cited by: SS1.
* D. Dzsmzky, D. Sharma, J. Clark, V. Prabhakaran, and J. Eisenstein (2021)Learning to recognize dialect features. In American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, pp. 2315-2338. External Links: Link Cited by: SS1.
* T. Dettmers, M. Lewis, S. Shleifer, and L. Zettlemoyer (2022)8-bit optimizers via block-wise quantization. In International Conference on Learning Representations, Cited by: SS1.
* T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer (2023)QLoRA: efficient finetuning of quantized LLMs. In Thirty-seventh Conference on Neural Information Processing Systems, Cited by: SS1.
* J. Eisenstein, V. Prabhakaran, C. Rivera, D. Demszky, and D. Sharma (2023)ML3: the multi-dialect dataset of dialogues. In INTERSPEECH, Cited by: SS1.
* F. Faisal, O. Ahia, A. Srivastava, K. Ahuja, D. Chiang, Y. Tsvetkov, and A. Anastasopoulos (2024)DIALECTBENCH: an nlp benchmark for dialects, varieties, and closely-related languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangokok, Thailand, pp. 14412-14454. External Links: Link Cited by: SS1.
* G. Team (2024)Gamma 2: Improving open language models at a practical size. Preprint arXiv:2408.00118. Cited by: SS1.
* W. Held, C. Ziems, and D. Yang (2023)TADA : Task agnostic dialect adapters for English. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, pp. 813-824. External Links: Link Cited by: SS1.
* E. J. Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen (2022)LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, Cited by: SS1.
* N. Jain, D. Zhang, W. Uddin Ahmad, Z. Wang, F. Nan, X. Li, M. Tan, R. Nallapati, B. Ray, P. Bhatia, X. Ma, and B. Xiang (2023)ContraCLM: contrastive learning for causal language model. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 6436-6459. External Links: Link Cited by: SS1.
* A. Q. Jiang, A. Sablayrolles, A. Mengch, C. Bamford, D. Singh Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lamplep, L. Saulnier, L. Renard Lavaud, M. Lachaux, P. Stock, T. Le Scao, Thibaut Lavril, T. Wang, Timothee Lacroix, and William El Sayed (2023)MIST 7b. PreprintarXiv:2310.06825. External Links: Link Cited by: SS1.
* A. Joshi, R. Dabre, D. Kanojia, Z. Li, H. Zhan, G. Haffari, and D. Dipplold (2025)Natural language processing for dialects of a language: a survey. ACM Comput. Surv.. Note: [https://doi.org/10.1111/j.acm.2025](https://doi.org/10.1111/j.acm.2025) Cited by: SS1.

 

C. Lange (2012)The syntax of spoken Indian English. Vol. 3, pp. 495-506. External Links: Document Cited by: SS1.
* C. Lange (2012)The syntax of spoken indian English. Vol. 1, pp. . External Links: Link Cited by: SS1.
* Y. Liu, W. Held, and D. Yang (2023)DADA: dialect adaptation via dynamic aggregation of linguistic rules. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, Singapore, pp. 13776-13793. External Links: Document Cited by: SS1.
* OpenAI (2024)Gpt-4 technical report. Preprint arXiv:2303.08774. External Links: Link Cited by: SS1.
* J. Pfeiffer, I. Vulic, I. Gurevych, and S. Ruder (2020)Mad-x: an adapter-based framework for multi-task cross-lingual transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7654-7673. External Links: ISBN 978-3-319-50958-3, Link, Document Cited by: SS1.
* N. Reimers and I. Gurevych (2019)Sentence-BERT: sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 3982-3992. External Links: Link Cited by: SS1.
* D. Srirag, N. Ranjan Sahoo, and A. Joshi (2025)Evaluating dialect robustness of language models via conversation understanding. In Proceedings of the Second Workshop on Scaling Up Multilingual & Multi-Cultural Evaluation, Abu Dhabi, Cited by: SS1.
* J. Sun, T. Sellam, E. Clark, T. Vu, T. Dozat, D. Garrette, A. Siddhant, J. Eisenstein, and S. Gehrmann (2023)Dialectrobust evaluation of generated text. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 6010-6028. External Links: Link Cited by: SS1.
* J. Wei, M. Bosma, V. Zhao, K. Guu, A. Wu Yu, B. Lester, N. Du, Andrew M. Dai, and Q. V. Le (2022)Finetuned language models are zero-shot learners. In International Conference on Learning Representations, Cited by: SS1.
* Z. Xiao, W. Held, Y. Liu, and D. Yang (2023)Task-agnostic low-rank adapters for unseen english dialects. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 7857-7870. External Links: Link Cited by: SS1.
* C. Ziems, J. Chen, C. Harris, J. Anderson, and D. Yang (2022)VALUE: understanding dialect disparity in hcl. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, pp. 3701-3720. External Links: Link Cited by: SS1.
* C. Ziems, W. Held, J. Yang, J. Dhamala, R. Gupta, and D. Yang (2023)Multi-value: a framework for cross-dialactical engish nlp. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. . External Links: Link Cited by: SS1.

 

Dataset Construction

Table 6 describes the example conversations from en-IN and en-US subsets along with their respective transformed IN-TR and IN-MV conversations. We utilise the following prompt used in the evaluation study by Srirag et al. (2025) to create IN-TR.

_'Normalise the conversation. Remove all exaggerations and dialectal information. Return a neutral response.'_

The conversations are then masked by replacing the target word with the [MASK] token and pruning the rest of the conversation, as described in the Table 7.

Table 8 describes examples from the pseudo-parallel corpus: en-US \(\|\) en-IN. The conversations in a positive pair, while dissimilar in the syntax of the conversation, pertain to the same target word. For example, the conversation pair labelled as 'positive' in the Table 8 describe the same target word-_Washing Machine_. The conversation pair labelled as 'negative' describe different target words; the en-US conversation describes _Justin Bieber_, while en-IN conversation describes _Washing Machine_.

## Appendix B Additional Ablations

We conducted additional ablation studies on LoRDD to address the following question: Can the performance improvement of LoRDD be attributed to the increased training data from data augmentation?

Table 9 compares the performance of the proposed combination of LoRDD with variations that exclude data augmentation. Training the task adapter solely on en-IN results in significantly lower performance, with similarity scores dropping by 5.9 to 7.0 and accuracy scores decreasing by 8.2 to 9.7.

Table 10 examines the effect of varying the proportion of en-US conversations in the augmented training data (en-US + en-IN). The best performance is observed when LoRDD is trained with augmented data containing only 50% en-US conversations. While this configuration outperforms the proposed full-proportion combination, determining the optimal proportions is challenging and limits generalisability across models. More particularly, Table 10 also reveals that mistral is highly sensitive to such changes in the training data composition, whereas gemma is more robust.

These ablation results, combined with the findings in Table 4, further reinforce our proposed methodology. Specifically, training the task adapter on fully proportioned augmented data (en-IN + en-US) and the dialect adapter on a parallel corpus constructed from natural conversations (en-US \(\|\) en-IN) proves to be a more effective and generalisable approach.

\begin{table}
\begin{tabular}{l l l} \hline \hline  & en-IN & IN-TR \\ \hline Describer: (Uh). What do you call if we, what will be there in the water? & **Describer:** (\(\varnothing\))What do you call \\ Water? & **De**creatures in the water? \\ Guesser: Fish(es) & **Guesser:**Fish(\(\varnothing\)). \\ Describer: Who will catch that? & **Describer:**Who catches them? \\ Guesser: Fisherman. & **Guesser:**Fishermen. \\ \hline \multicolumn{3}{c}{en-US} & IN-MV \\ \hline Describer: Perfect. Oh! (We) earn this. We go tour jobs. & **Describer:**Perfect. Oh! (\(\varnothing\)) [are] \\ \multicolumn{3}{c}{earn[ing] this. We [are] going] to} \\ \multicolumn{3}{c}{our jobs.} \\ Guesser: Money & **Guesser:**Money \\ \hline \hline \end{tabular}
\end{table}
Table 6: Example _transformations_ of en-IN to IN-TR, and en-US to IN-MV. We utilise GPT-4 Turbo to generate IN-TR, and Multi-VALUE to create IN-MV. The text in parentheses refers to the omission/removal of certain filler and exaggerated words, and the text such as this, refers to the words or sentences that were rephrased to convey the original meaning, and the text such as [this], refers to the dialectical features added using Multi-VALUE.

 



\begin{table}
\begin{tabular}{c c l} \hline \hline Target & \multicolumn{1}{l}{en-IN} & Masked en-IN \\ Word & & \\ \hline  & Describer: Uh. What do you call if & Describer: Uh. What do you call if \\ Fisherman & we, what will be there in the water? & we, what will be there in the water? \\ Guesser: Fishes & Guesser: Fishes \\  & Describer: Who will catch that? & Describer: Who will catch that? \\  & Guesser: Fisherman. & Guesser: [MASK] \\ \hline Target & \multirow{2}{*}{
\begin{tabular}{c} Target \\ Word \\ \end{tabular} } & \multirow{2}{*}{en-US} & Masked en-US \\ \cline{3-4}  & & & \\ \hline \multirow{2}{*}{Planet} & Describer: These are hard words. um & Describer: These are hard words. um \\
**Planet** & Okay. So there’s. the Sun and the & Okay. So there’s. the Sun and the \\ Moon and all the rest of them. & Moon and all the rest of them. \\ Guesser: And all the planets? & Guesser: [MASK] \\
**(Describer: Yes.)** & Guesser: [MASK] \\ \hline \hline \end{tabular}
\end{table}
Table 7: Masking conversations from the extended MD-3. The text such as this represents the target word utterance by the guesser which is masked (represented by, [MASK] in the final version of the conversation. The rest of the original conversation is pruned as represented text in parentheses.

\begin{table}
\begin{tabular}{p{56.9pt} p{80.0pt} p{113.8pt} p{113.8pt} p{113.8pt}} \hline \hline Label & \multicolumn{2}{c}{en-US} & \multicolumn{2}{c}{en-IN} \\ \hline Positive & \multicolumn{1}{p{113.8pt}}{Describer: Good job. Okay. Um. How we clean our clothes.} & \multicolumn{1}{p{113.8pt}}{Describer: Yeah here I got a thing uhh much most of us daily use that to wash our clothes.} & \multicolumn{1}{p{113.8pt}}{Describer: [MASK]} \\  & \multicolumn{1}{p{113.8pt}}{Guesser: [MASK]} & \multicolumn{1}{p{113.8pt}}{[MASK]} & \multicolumn{1}{p{113.8pt}}{[MASK]} \\  & \multicolumn{1}{p{113.8pt}}{Disccriber: this. What? All right all rights so.} & \multicolumn{1}{p{113.8pt}}{[MASK]} & \multicolumn{1}{p{113.8pt}}{[MASK]} \\ Negative & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} \\  & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} \\  & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} \\  & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} \\  & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} \\  & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} \\  & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} \\  & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} & \multicolumn{1}{p{113.8pt}}{} \\ \hline \hline \end{tabular}
\end{table}
Table 8: Example conversation pairs from the pseudo-parallel corpus: en-US || en-IN. A positive example contains conversations describing the same target word, while the negative example contains conversations pertaining to two different target words.

\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline Method & Training Data & \(\mathbf{I_{Corpus}}\) & \multicolumn{2}{c}{Mistral} & \multicolumn{2}{c}{gamma} & \multicolumn{2}{c}{\(\mu\)} \\ \cline{4-9}  & & Similarity & Accuracy & Similarity & Accuracy & Similarity & Accuracy \\ \hline LoRDD & en-US + en-IN & en-US || en-IN & **55.9** & **30.0** & **63.9** & **41.3** & **59.9** & **35.7** \\ \(\leftrightarrow\) & & en-US || en-IN & 52.0 & 23.1 & 53.7 & 28.8 & 52.9 (7.0) & 26.0 (9.7) \\ \(\leftrightarrow\) & & en-IN (No Augmentation) & en-IN || IN-TR & 52.0 & 23.8 & 54.1 & 28.8 & 53.0 (6.9) & 26.3 (9.4) \\  & & en-US || IN-MV & 53.3 & 25.0 & 54.6 & 30.0 & 54.0 (5.9) & 27.5 (8.2) \\ \hline \hline \end{tabular}
\end{table}
Table 9: Ablation on LoRDD based on parallel corpus (\(\leftrightarrow\) || Corpus) and data augmentation. For each model, we report Similarity and Accuracy when tested on en-IN. The best performance is shown in **bold**. \(\mu\) is the average of the metrics across both models. The degradation on the ablations compared to LoRDD is shown in (number).





\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \multirow{2}{*}{Method} & \multirow{2}{*}{llCorpus} & \multirow{2}{*}{\% of en-US} & \multicolumn{3}{c}{MISTAL} & \multicolumn{2}{c}{GEMMA} & \(\mu\) \\ \cline{5-10}  & & & Similarity & Accuracy & Similarity & Accuracy & Similarity & Accuracy \\ \hline \multirow{4}{*}{LoRDD} & & 0\% & 52.0 & 23.1 & 53.7 & 28.8 & 52.9 & 26.0 \\  & & 25\% & 53.8 & 31.9 & 61.2 & 35.4 & 57.5 & 33.7 \\  & & 50\% & **58.8** & **33.8** & **64.1** & **41.8** & **61.5** & **37.8** \\  & & 75\% & 54.6 & 30.6 & 63.4 & 40.8 & 59.0 & 35.7 \\  & & 100\%* & 55.9* & 30.0* & 63.9* & 41.3* & 59.9* & 35.7* \\ \multirow{4}{*}{\(-\mathcal{L_{\text{bial}}}\)} & & 0\% & 51.0 & 24.4 & 54.6 & 30.0 & 52.8 & 27.2 \\  & & 25\% & 52.0 & 29.4 & 60.5 & 34.4 & 56.3 & 31.9 \\ \multirow{4}{*}{\(-\mathcal{L_{\text{bial}}}\)} & & 50\% & 55.3 & 29.4 & 61.4 & 35.6 & 58.4 & 32.2 \\  & & 75\% & 52.5 & 27.5 & 61.6 & 35.6 & 57.1 & 31.6 \\ \multirow{4}{*}{\(-\mathcal{L_{\text{bial}}}\)} & & 100\% & 54.4 & 26.9 & 62.3 & 37.5 & 58.4 & 32.2 \\ \hline \end{tabular}
\end{table}
Table 10: Ablation on LoRDD based on dialect adapter (\(\mathcal{L_{\text{bial}}}\)) and proportion of en-US conversations in augmented data (en-US + en-IN). For each model, we report Similarity and Accuracy when tested on en-IN. The best performance is shown in **bold**, and the proposed combination is represented by number*. \(\mu\) is the average of the metrics across both models.