A Multi-Agent Framework for Mitigating Dialect Biases in Privacy Policy Question-Answering Systems
Authors: Đorđe Klisura, Astrid Bernaga Torres, Anna Karen Gárate, Rajesh Roshan Biswal, Ke Yang, Hilal Pataci, Anthony Rios
Accepted at: acl25

A Multi-Agent Framework for Mitigating Dialect Biases in Privacy Policy Question-Answering Systems

Đorde Klisura1, Astrid R Bernaga Torres2, Anna Karen Gárate-Escamilla2, Rajesh Roshan Biswal2, Ke Yang1, Hilal Pataci1, Anthony Rios1
1University of Texas at San Antonio
2Tecnológico de Monterrey
{Dorde.Klisura, Anthony.Rios}@utsa.edu

Abstract
Privacy policies inform users about data collection and usage, yet their complexity limits accessibility for diverse populations. Existing Privacy Policy Question Answering (QA) systems exhibit performance disparities across English dialects, disadvantaging speakers of non-standard varieties. We propose a novel multi-agent framework inspired by human-centered design principles to mitigate dialectal biases. Our approach integrates a Dialect Agent, which translates queries into Standard American English (SAE) while preserving dialectal intent, and a Privacy Policy Agent, which refines predictions using domain expertise. Unlike prior approaches, our method does not require retraining or dialect-specific fine-tuning, making it broadly applicable across models and domains. Evaluated on PrivacyQA and PolicyQA, our framework improves GPT-4o-mini’s zero-shot accuracy from 0.394 to 0.601 on PrivacyQA and from 0.352 to 0.464 on PolicyQA, surpassing or matching few-shot baselines without additional training data. These results highlight the effectiveness of structured agent collaboration in mitigating dialect biases and underscore the importance of designing NLP systems that account for linguistic diversity to ensure equitable access to privacy information.

1 Introduction
Privacy policies are essential documents that outline how organizations collect, use, and share personal data. Yet, their effectiveness is undermined by excessive length, legal complexity, and inaccessible language, making it difficult for users to understand their rights and risks (Ravichander et al., 2019; Ahmad et al., 2020). Privacy Policy Question Answering (QA) systems aim to bridge this gap by providing users with concise, query-driven insights. However, existing systems remain largely indifferent to linguistic diversity, particularly the nuanced variations in English dialects, thereby constraining equitable access to privacy information.

![Illustration of dialect-based disparities in Privacy Question Answering (QA). The QA model correctly answers a query phrased in Standard American English (SAE) but produces an incorrect response when the same query is asked in African American Vernacular English (AAVE).](page_1012_370_388_246.png)

Figure 1: Illustration of dialect-based disparities in Privacy Question Answering (QA). The QA model correctly answers a query phrased in Standard American English (SAE) but produces an incorrect response when the same query is asked in African American Vernacular English (AAVE).

This oversight is especially consequential in real-world deployments, where dialectal differences fundamentally shape how users parse and interpret complex legal and technical content. Specifically, the Electronic Privacy Information Center (EPIC) states the following on their website (Electronic Privacy Information Center, 2025):

Marginalized communities are disproportionately harmed by data collection practices and privacy abuses from the both the government and private sector. Communities of color are especially targeted, discriminated against, and exploited through surveillance, policing, and algorithmic bias.
- EPIC

From a privacy QA perspective, if all groups cannot ask questions to help protect their information effectively, those groups are at risk. We illustrate this issue in Figure 1.

The challenge of dialectal bias in NLP has been extensively documented, with non-standard
dialects such as African American Vernacular English (AAVE), Chicano English, and Aboriginal English often receiving subpar performance compared to Standard American English (SAE) (Ziems et al., 2023; Blodgett and O’Connor, 2018). This disparity disproportionately affects marginalized communities, amplifying existing inequities and limiting access to language technologies for non-dominant speakers (Sap et al., 2019; Davidson et al., 2019). While frameworks like Multi-VALUE have been developed to evaluate and mitigate dialect biases in general NLP tasks (Ziems et al., 2023), no work has explored how such biases manifest in domain-specific applications like privacy policy QA.

Furthermore, much of the recent work on question-answering has focused on large language models (LLMs) and, in particular, prompting-based methods (Lee and Lee, 2022; Yu et al., 2023). These systems are developed to work well generally for a wide audience. However, they struggle with geographical/cultural (Lwowski et al., 2022; Liu et al., 2024; Naous et al., 2024) and dialectal biases (Lwowski and Rios, 2021; Faisal et al., 2024) when used by specific communities. Hence, a fundamental question is, “How can we tune the prompting procedures of LLMs to perform well for minority communities/dialects without collecting large amounts of training data from these communities to fine-tune models, which may be difficult, particularly in sensitive application domains?”

To address these limitations, we introduce a novel multi-agent1 collaboration framework for dialect-sensitive privacy policy QA. Our method integrates two specialized agents: a Dialect Agent and a Privacy Policy Agent. The Dialect Agent processes user queries in diverse dialects by translating them into SAE, providing relevant judgments, and explaining their reasoning. The Privacy Policy Agent further refines these outputs by leveraging domain-specific expertise to validate and improve predictions. This collaborative design allows us to mitigate dialectal biases without requiring task-specific retraining or extensive dialectal datasets, addressing the scalability challenges of previous approaches.

We evaluate our framework on the PrivacyQA and PolicyQA datasets, which include queries across a wide range of dialects generated using the Multi-VALUE framework. Our method significantly improves fairness and accuracy, reducing performance disparities across dialects by up to 82% as measured by the maximum difference in F1 scores between dialects. Furthermore, our approach achieves state-of-the-art performance in privacy policy QA, highlighting its robustness, scalability, and real-world applicability in mitigating dialectal biases while enhancing accessibility to critical privacy information. Overall, we make the following contributions in this paper:

• We perform an exhaustive benchmark of dialect biases for state-of-the-art LLMs applied to privacy question-answering datasets.

• We introduce a novel multi-agent framework that introduces direct knowledge about the dialect and/or minority group to mitigate biases and improve overall performance.

• We perform a comprehensive ablation and error analysis. Moreover, we provide implications for deploying this approach in practice.

2 Related Work

NLP and Privacy. NLP research in privacy policy extends beyond QA, tackling the structural and interpretive challenges of privacy policies. To address this, various datasets have been developed to facilitate privacy policy research (Wilson et al., 2016; Ramanath et al., 2014; Srinath et al., 2021; Amos et al., 2021; Manandhar et al., 2022). Notable efforts include OPP-115, which focuses on classifying privacy practices within policies (Chi et al., 2023). Similarly, PolicyIE enables semantic parsing by identifying intents and filling slots related to privacy practices (Ahmad et al., 2021). Named Entity Recognition (NER) tasks, such as PI-Extract, identify specific data types mentioned in privacy policies, supporting better automatic understanding (Bui et al., 2021). The PLUE benchmark consolidates these tasks, providing a comprehensive evaluation framework for privacy policy language understanding (Chi et al., 2023). These initiatives have broadened the scope of privacy policy NLP by addressing tasks like classification, semantic parsing, and NER, creating a foundation for advanced applications in this domain.

Privacy policy QA has emerged as a critical area of study, aiming to streamline user interactions with these documents by retrieving concise and relevant answers to user queries. PrivacyQA introduced a

1We use the term multi-agent to describe structured prompt-based collaboration between distinct roles invoked via large language models, rather than autonomous agents in classical multi-agent systems.
sentence-level evidence retrieval framework, highlighting the inherent challenges of answerability and relevance (Ravichander et al., 2019). PolicyQA advanced this approach by framing the task as span extraction, emphasizing the need for short and precise answers to improve accessibility (Ahmad et al., 2020). PLUE expanded the evaluation framework to include QA as one of its core tasks, demonstrating the value of domain-specific pre-training in improving QA accuracy (Chi et al., 2023). Despite significant progress, open challenges persist, particularly in addressing ambiguities, improving robustness to linguistic diversity, and ensuring fairness across user demographics, as well as mitigating emerging security concerns in deploying large language models (Klisura and Rios, 2024).

Dialectal NLP. Dialect NLP research highlights significant performance disparities between dominant dialects, such as standard American English (SAE), and lower-resource dialects such as African American Vernacular English (AAVE), Chicano English and Indian English, raising concerns about fairness and equity in language technology (Ziems et al., 2023; Blodgett and O’Connor, 2018; Jurgens et al., 2017). These disparities, evident in tasks such as dependency analysis, sentiment analysis, and hate speech detection, disproportionately affect marginalized communities (Sap et al., 2019; Davidson et al., 2019; Jørgensen et al., 2016). The lack of robust dialectal evaluation frameworks exacerbates these issues, reinforcing existing power imbalances in NLP systems (Bender et al., 2021; Hovy and Spruit, 2016). Existing work, such as Multi-VALUE, addresses these gaps by creating rule-based perturbations and stress tests to evaluate model robustness across 50 English dialects (Ziems et al., 2023; Kortmann et al., 2020). Frameworks like DADA and TADA employ modular and task-agnostic approaches, enabling fine-grained adaptation and cross-dialectal robustness without requiring extensive task-specific data (Liu et al., 2023b; Held et al., 2023). These advancements are complemented by efforts to incorporate sociolinguistic insights into model development, addressing morphosyntactic variations and promoting scalable, equitable solutions for dialectal NLP (Sun et al., 2023; Demeszky et al., 2019). Together, these approaches underscore the critical need for inclusive NLP systems that mitigate dialectal biases and ensure equitable access to language technologies (Blodgett and O’Connor, 2018; Sap et al., 2019; Davidson et al., 2019). This paper uses the Multi-Value dialectal testing framework to evaluate biases in privacy QA tasks. Moreover, we overcome some of the limitations of prior dialectal technologies that require dialect-aware training frameworks (Liu et al., 2023b; Held et al., 2023). Instead, our framework only requires some initial (minimal) dialect information supplied as a prompt, minimizing some of the complexities in implementing prior work.

Multi-agent Modeling. Multi-agent systems (MAS) have become increasingly prominent in NLP for coordinating specialized agents to handle complex and large-scale tasks. LongAgent (Zhao et al., 2024a) addresses long-document QA by distributing text across agents and using iterative communication to reduce hallucinations and ensure consistent answers. Recent MAS work has also emphasized collective decision-making (CDM), with systems like GEDI (Zhao et al., 2024b) applying voting methods such as ranked pairs and plurality to improve fairness and robustness. Beyond QA, MAS have proven effective in multi-turn reasoning (Chen et al., 2023), knowledge retrieval (Liu et al., 2023a), and structured prediction (Xu et al., 2023), showcasing their versatility. These frameworks highlight how inter-agent collaboration and feedback loops can enhance performance, reliability, and inclusivity in a range of NLP applications.

LLM-based multi-agent systems. Recent work has explored LLM-based multi-agent systems that differ from classical approaches by coordinating agents through natural language rather than fixed protocols (Li et al., 2024). These systems assign roles like planner, critic, or explainer to individual models and enable them to collaborate via structured, prompt-based dialogue. Frameworks like CAMEL (Li et al., 2023), AutoAgents (Chen et al., 2024), and ChatDev (Qian et al., 2023) show how role-based agents can dynamically negotiate, critique, and refine their outputs to complete complex tasks like software development, multi-hop reasoning, or policy interpretation. While classical MAS emphasized distributed algorithms and communication protocols, LLM-based systems focus on emergent cooperation through language, enabling more flexible task decomposition and iterative problem-solving. Our work builds on this paradigm by prompting specialized agents (the Dialect and Privacy Policy agents) to engage in structured collaboration through role-specific prompting and iterative refinement.
Figure 2: Our multi-agent framework for mitigating dialect biases in privacy QA. The Dialect Agent translates queries into Standard American English (SAE) and validates responses. The Privacy Policy Agent generates answers based on policy text. Disagreements trigger refinement, ensuring accurate and inclusive responses across dialects.

3 Methodology

Our primary objective is to reduce performance disparities in privacy policy QA across multiple large language models when queries are posed in diverse English dialects. To formally define the task, let \( q_d \) be a question in dialect \( d \in \mathcal{D} \) and let \( p \) be a corresponding privacy policy snippet. A QA model \( f \) produces an answer \( A = f(p, q_d) \), which is compared to a ground-truth answer \( A^* \). We measure correctness using a metric \( \Phi \). For a given dialect \( d \), the average performance of \( f \) is denoted by \( \Phi_{d_i}(f) \). We define the overall performance disparity \( \Delta(f) \) as:

\[
\Delta(f) = \max_{d_i, d_j \in \mathcal{D}} |\Phi_{d_i}(f) - \Phi_{d_j}(f)|.
\]

The goal is to design a QA framework \( F \) that minimizes \( \Delta(f) \) while maintaining average accuracy on privacy policy questions.

To achieve this, we introduce a multi-agent collaboration framework. Figure 2 provides a high-level overview of our approach. The framework mirrors a human-centered design (Cooley, 2000) approach by prioritizing usability, fairness, and inclusivity in PrivacyQA systems. It leverages two specialized agents: a Dialect Agent and a Privacy Agent, designed to adapt to user needs and linguistic diversity. The Dialect Agent is an intermediary that translates “non-standard” dialect questions into SAE while preserving the user’s query’s original intent and cultural nuances. This, again, is based on human-centered design, where we try to add user information about the dialect they speak to the model to improve performance. This ensures that speakers of diverse dialects are not disadvantaged when interacting with privacy policy information because they are explicitly addressed in the model.

Meanwhile, the Privacy Agent interprets privacy policy segments\(^2\) and generates accurate, policy-oriented answers that remain accessible and relevant across different linguistic backgrounds. By structuring the system as a collaborative process that integrates dialect-aware adaptation (from the Dialect Agent) and domain expertise (from the Privacy Agent), our approach embodies human-centered design principles—ensuring adequate performance on dialects beyond SAE. We describe the agents below.

Step 1: Dialect Agent. The Dialect Agent is prompted to act as an expert in diverse English dialects. Before processing any user query, it is given a concise yet detailed summary of a particular dialect’s key linguistic properties, including (very brief) phonetic, grammatical, lexical, and cultural aspects. Please see Appendix C with examples. This setup enables the Dialect Agent to translate a user’s dialectal question into SAE accurately and, subsequently, to validate whether the final answer aligns with the user’s original intent.

When a user provides a privacy policy segment and a question in a non-standard dialect, the question first goes to the Dialect Agent. Its task is to translate the query into clearly understandable

\footnotetext{2Privacy policies typically encompass ten major categories of data practices. These include First Party Collection (FP), Third Party Sharing/Collection (TP), Data Retention (DR), and Data Security (DS), which explain how and why first and third parties collect, process, store, share, and protect customer data. User rights are addressed through categories like User Choice/Control (UCC), User Access, Edit, Deletion (UAED), and Do Not Track (DNT)(Wilson et al., 2016).}
SAE using its background knowledge about the dialect. Specifically, it is provided with the following prompt:3

Prompt
“You are an expert linguist specializing in the following dialect: {dialect_info}. Your task is to translate the following question from this dialect into clear, standard American English. Ensure that the translation is easily understandable to a general audience.”

where dialect_info is the dialect information for that particular dialect. The output of this step is a standardized version of the user’s question, ready to be processed by the Privacy Agent.

Step 2a: Privacy Agent. Once the dialectal query has been translated to SAE, it is handed over to the Privacy Agent along with the relevant segment of the privacy policy. The Privacy Agent is prompted as a domain expert, possessing comprehensive knowledge of typical privacy policy structures and terminologies.

The Privacy Agent uses the translated question and the given policy snippet to craft an initial response. The focus is on extracting accurate, succinct information from the policy segment that addresses the user’s query. The general prompt looks as follows:

Prompt
“You are a privacy policy expert. Review the provided policy segment and answer the following question in a concise manner, ensuring factual accuracy. Base your response solely on the information in the policy segment.”

The Privacy Agent outputs both the initial answer and a brief rationale, indicating how the policy text justifies that answer.

Steps 2b and 2c: Evaluation by Dialect Agent. Next, we provide the dialect agent with the original dialectal question, the policy segment, and the Privacy Agent’s proposed answer to the Dialect Agent. The Dialect Agent then evaluates whether the answer sufficiently captures the user’s intent and does not overlook subtle dialect-specific nuances. To do this, we provide the dialect agent the following prompt:

<table>
  <tr>
    <th></th>
    <th>PrivacyQA Mobile Apps</th>
    <th>PolicyQA Websites</th>
  </tr>
  <tr>
    <td># Policies</td>
    <td>35</td>
    <td>115</td>
  </tr>
  <tr>
    <td># Questions</td>
    <td>1,750</td>
    <td>714</td>
  </tr>
  <tr>
    <td># Annotations</td>
    <td>3,500</td>
    <td>25,017</td>
  </tr>
</table>

Table 1: Statistics for Privacy Policy QA datasets.

Prompt
“Based on your understanding of the dialect’s linguistic and cultural nuances, determine whether the Privacy Agent’s answer fully addresses the user’s original question. Are there any discrepancies or misunderstandings that arise from the dialectal phrasing?”

If the Dialect Agent confirms the answer is satisfactory, this output is accepted as final and step 2c is followed to return the final answer. If it flags potential inaccuracies or misunderstandings (for instance, the Privacy Agent missed the user’s intended meaning due to unique dialectal expressions), the process moves into a reconsideration stage (Step 2b) instead.

Upon receiving negative feedback from the Dialect Agent, the Privacy Agent revisits its initial answer. It is prompted to update or refine its response based on the Dialect Agent’s observations regarding the original question’s intent. The prompt is defined as follows:

Prompt
“You received feedback indicating that certain elements of the user’s dialectal query were not fully addressed. Please revise your previous answer to incorporate the Dialect Agent’s insights and ensure the user’s intent is accurately captured.”

The Privacy Agent will then return another answer and rationale to the Dialect Agent. We will repeat this process until the agreement is met or a maximum number of iterations is met (we only loop a maximum of 2 times). This loop ensures that dialect nuances are not lost while improving the correctness of policy-based answers. Note that in few-shot settings, we use a total of 8 examples per prompt for each agent. These examples reflect diverse dialects, question types, and policy scenarios, helping the agents generalize across linguistic and contextual variation.

4 Evaluation

Data. We use two privacy QA datasets: PrivacyQA and PolicyQA. We provide the dataset statistics in Table 1 for complete details. Priva-
<table>
  <tr>
    <th>Model</th>
    <th>SAE (†)</th>
    <th>RAAVE (†)</th>
    <th>Jamaican (†)</th>
    <th>Aboriginal (†)</th>
    <th>Welsh (†)</th>
    <th>SWE (†)</th>
    <th>AVG (†)</th>
    <th>AVG Diff (‡)</th>
    <th>Max Diff (‡)</th>
  </tr>
  <tr>
    <td>GPT-4o-mini Zero</td>
    <td>.394</td>
    <td>.344</td>
    <td>.332</td>
    <td>.329</td>
    <td>.312</td>
    <td>.301</td>
    <td>.335</td>
    <td>.022</td>
    <td>.093</td>
  </tr>
  <tr>
    <td>GPT-4o-mini Few</td>
    <td>.605</td>
    <td>.573</td>
    <td>.562</td>
    <td>.555</td>
    <td>.547</td>
    <td>.547</td>
    <td>.565</td>
    <td>.016</td>
    <td>.058</td>
  </tr>
  <tr>
    <td>GPT-4o-mini Multi-agent-zero (ours)</td>
    <td><b>.601</b></td>
    <td><b>.588</b></td>
    <td><b>.578</b></td>
    <td><b>.587</b></td>
    <td><b>.592</b></td>
    <td><b>.576</b></td>
    <td><b>.587</b></td>
    <td><b>.007</b></td>
    <td><b>.025</b></td>
  </tr>
  <tr>
    <td>GPT-4o-mini Multi-agent-few (ours)</td>
    <td><b>.611</b></td>
    <td><b>.595</b></td>
    <td><b>.596</b></td>
    <td><b>.602</b></td>
    <td><b>.594</b></td>
    <td><b>.594</b></td>
    <td><b>.595</b></td>
    <td><b>.005</b></td>
    <td><b>.019</b></td>
  </tr>
  <tr>
    <td>Llama 3.1 Zero</td>
    <td>.469</td>
    <td>.349</td>
    <td>.370</td>
    <td>.325</td>
    <td>.356</td>
    <td>.336</td>
    <td>.365</td>
    <td>.025</td>
    <td>.144</td>
  </tr>
  <tr>
    <td>Llama 3.1 Few</td>
    <td>.546</td>
    <td>.463</td>
    <td>.469</td>
    <td>.448</td>
    <td>.485</td>
    <td>.446</td>
    <td>.476</td>
    <td>.026</td>
    <td>.100</td>
  </tr>
  <tr>
    <td>Llama 3.1 Multi-agent-zero (ours)</td>
    <td><b>.549</b></td>
    <td><b>.527</b></td>
    <td><b>.520</b></td>
    <td><b>.524</b></td>
    <td><b>.523</b></td>
    <td><b>.526</b></td>
    <td><b>.528</b></td>
    <td><b>.007</b></td>
    <td><b>.029</b></td>
  </tr>
  <tr>
    <td>Llama 3.1 Multi-agent-few (ours)</td>
    <td><b>.555</b></td>
    <td><b>.525</b></td>
    <td><b>.523</b></td>
    <td><b>.529</b></td>
    <td><b>.522</b></td>
    <td><b>.528</b></td>
    <td><b>.530</b></td>
    <td><b>.008</b></td>
    <td><b>.033</b></td>
  </tr>
  <tr>
    <td>DeepSeek-R1 Zero</td>
    <td>.532</td>
    <td>.510</td>
    <td>.547</td>
    <td>.529</td>
    <td>.532</td>
    <td>.512</td>
    <td>.527</td>
    <td>.011</td>
    <td>.037</td>
  </tr>
  <tr>
    <td>DeepSeek-R1 Few</td>
    <td>.581</td>
    <td>.549</td>
    <td>.547</td>
    <td>.517</td>
    <td>.556</td>
    <td>.541</td>
    <td>.549</td>
    <td>.014</td>
    <td>.064</td>
  </tr>
  <tr>
    <td>DeepSeek-R1 Multi-agent-zero (ours)</td>
    <td><b>.582</b></td>
    <td><b>.579</b></td>
    <td><b>.583</b></td>
    <td><b>.579</b></td>
    <td><b>.566</b></td>
    <td><b>.573</b></td>
    <td><b>.577</b></td>
    <td><b>.005</b></td>
    <td><b>.017</b></td>
  </tr>
  <tr>
    <td>DeepSeek-R1 Multi-agent-few (ours)</td>
    <td><b>.533</b></td>
    <td><b>.506</b></td>
    <td><b>.585</b></td>
    <td><b>.581</b></td>
    <td><b>.557</b></td>
    <td><b>.569</b></td>
    <td><b>.572</b></td>
    <td><b>.019</b></td>
    <td><b>.073</b></td>
  </tr>
</table>

Table 2: Performance comparison on PrivacyQA across dialects. Our multi-agent framework (bold) improves accuracy and reduces disparities (AVG Diff and Max Diff) compared to baseline models (GPT-4o-mini, Llama 3.1, and DeepSeek-R1). Results are shown for Standard American English (SAE), Rural African American Vernacular English (RAAVE), Jamaican English, Aboriginal English, Welsh English, and Southwest England Dialect (SWE).

<table>
  <tr>
    <th>Model</th>
    <th>SAE (†)</th>
    <th>RAAVE (†)</th>
    <th>Jamaican (†)</th>
    <th>Aboriginal (†)</th>
    <th>Welsh (†)</th>
    <th>SWE (†)</th>
    <th>AVG (†)</th>
    <th>AVG Diff (‡)</th>
    <th>Max Diff (‡)</th>
  </tr>
  <tr>
    <td>GPT-4o-mini Zero</td>
    <td>.352</td>
    <td>.343</td>
    <td>.332</td>
    <td>.338</td>
    <td>.331</td>
    <td>.323</td>
    <td>.337</td>
    <td>.008</td>
    <td>.029</td>
  </tr>
  <tr>
    <td>GPT-4o-mini Few</td>
    <td>.478</td>
    <td>.422</td>
    <td>.458</td>
    <td>.452</td>
    <td>.444</td>
    <td>.438</td>
    <td>.449</td>
    <td>.014</td>
    <td>.055</td>
  </tr>
  <tr>
    <td>GPT-4o-mini Multi-agent-zero (ours)</td>
    <td><b>.464</b></td>
    <td><b>.444</b></td>
    <td><b>.451</b></td>
    <td><b>.458</b></td>
    <td><b>.447</b></td>
    <td><b>.445</b></td>
    <td><b>.452</b></td>
    <td><b>.006</b></td>
    <td><b>.020</b></td>
  </tr>
  <tr>
    <td>GPT-4o-mini Multi-agent-few (ours)</td>
    <td><b>.484</b></td>
    <td><b>.460</b></td>
    <td><b>.475</b></td>
    <td><b>.473</b></td>
    <td><b>.469</b></td>
    <td><b>.467</b></td>
    <td><b>.471</b></td>
    <td><b>.006</b></td>
    <td><b>.024</b></td>
  </tr>
  <tr>
    <td>Llama 3.1 Zero</td>
    <td>.310</td>
    <td>.260</td>
    <td>.268</td>
    <td>.231</td>
    <td>.237</td>
    <td>.289</td>
    <td>.266</td>
    <td>.023</td>
    <td>.079</td>
  </tr>
  <tr>
    <td>Llama 3.1 Few</td>
    <td>.412</td>
    <td>.332</td>
    <td>.360</td>
    <td>.357</td>
    <td>.393</td>
    <td>.370</td>
    <td>.371</td>
    <td>.021</td>
    <td>.080</td>
  </tr>
  <tr>
    <td>Llama 3.1 Multi-agent-zero (ours)</td>
    <td><b>.381</b></td>
    <td><b>.374</b></td>
    <td><b>.368</b></td>
    <td><b>.358</b></td>
    <td><b>.372</b></td>
    <td><b>.368</b></td>
    <td><b>.370</b></td>
    <td><b>.006</b></td>
    <td><b>.023</b></td>
  </tr>
  <tr>
    <td>Llama 3.1 Multi-agent-few (ours)</td>
    <td><b>.400</b></td>
    <td><b>.380</b></td>
    <td><b>.391</b></td>
    <td><b>.385</b></td>
    <td><b>.394</b></td>
    <td><b>.372</b></td>
    <td><b>.387</b></td>
    <td><b>.008</b></td>
    <td><b>.028</b></td>
  </tr>
  <tr>
    <td>DeepSeek-R1 Zero</td>
    <td>.455</td>
    <td>.436</td>
    <td>.429</td>
    <td>.437</td>
    <td>.422</td>
    <td>.422</td>
    <td>.434</td>
    <td>.009</td>
    <td>.033</td>
  </tr>
  <tr>
    <td>DeepSeek-R1 Few</td>
    <td>.446</td>
    <td>.483</td>
    <td>.488</td>
    <td>.472</td>
    <td>.492</td>
    <td>.477</td>
    <td>.481</td>
    <td>.011</td>
    <td>.046</td>
  </tr>
  <tr>
    <td>DeepSeek-R1 Multi-agent-zero (ours)</td>
    <td><b>.451</b></td>
    <td><b>.480</b></td>
    <td><b>.474</b></td>
    <td><b>.483</b></td>
    <td><b>.485</b></td>
    <td><b>.481</b></td>
    <td><b>.472</b></td>
    <td><b>.010</b></td>
    <td><b>.032</b></td>
  </tr>
  <tr>
    <td>DeepSeek-R1 Multi-agent-few (ours)</td>
    <td><b>.474</b></td>
    <td><b>.476</b></td>
    <td><b>.494</b></td>
    <td><b>.480</b></td>
    <td><b>.487</b></td>
    <td><b>.480</b></td>
    <td><b>.482</b></td>
    <td><b>.006</b></td>
    <td><b>.020</b></td>
  </tr>
</table>

Table 3: Performance comparison on PolicyQA across dialects. Our multi-agent framework (bold) improves accuracy and reduces disparities (AVG Diff and Max Diff) compared to baseline models (GPT-4o-mini, Llama 3.1, and DeepSeek-R1). Results are shown for Standard American English (SAE), Rural African American Vernacular English (RAAVE), Jamaican English, Aboriginal English, Welsh English, and Southwest England Dialect (SWE).

cyQA (Ravichander et al., 2019) is a dataset designed for answer sentence selection on mobile app privacy policies. It contains 1,750 privacy-related questions with over 3,500 expert-annotated answers from 35 policies. Given a question and a set of possible answers (sentences from the policy), a model must determine which, if any, correctly answers the question. Specifically, each answer candidate is classified as “correct” or “incorrect.” The dataset includes answerable and unanswerable questions, reflecting real-world challenges in understanding privacy policies. For example, for the question “Will my data be sold to advertisers?”, a model must determine if the sentence “We do not sell your personal information.” is a valid answer.

PolicyQA (Ahmad et al., 2020) is a dataset for question answering (QA) on website privacy policies. It includes 25,017 question-answer pairs from 115 privacy policies, helping users find clear answers to privacy-related questions. Instead of returning long text passages, PolicyQA provides short, precise answers. For example, given the question “Is my information shared with others?”, the dataset might provide the answer “We do not give that business your name and address.” This makes it easier for users to find the information they need quickly.

We use the Multi-VALUE (Ziems et al., 2023) framework to translate both PrivacyQA and PolicyQA into the dialects it supports (e.g., African American Vernacular English). The Multi-VALUE framework is a rule-based translation system designed to enhance cross-dialectal NLP by systematically transforming SAE into synthetic forms of 50 different English dialects. It applies 189 linguistic perturbation rules informed by dialectology research to modify syntax and morphology while preserving semantics, enabling stress testing and data augmentation for NLP models. In the main text, we report results for five dialects that exhibited the lowest average performance across baseline models: Rural African American Vernacular English (RAAVE), Jamaican English, Aboriginal English, Welsh English, and Southwest England Dialect (SWE). Complete results for all evaluated dialects are provided in Appendix F.

Evaluation Metrics. We evaluate model performance using different metrics suited to each dataset.
<table>
  <tr>
    <th rowspan="2">Setting</th>
    <th colspan="2">PrivacyQA</th>
    <th colspan="2">PolicyQA</th>
  </tr>
  <tr>
    <th>Initial (↑)</th>
    <th>Final (↑)</th>
    <th>Initial (↑)</th>
    <th>Final (↑)</th>
  </tr>
  <tr>
    <td>Zero-shot</td>
    <td>.53</td>
    <td>.59</td>
    <td>.43</td>
    <td>.45</td>
  </tr>
  <tr>
    <td>Few-shot</td>
    <td>.58</td>
    <td>.61</td>
    <td>.47</td>
    <td>.48</td>
  </tr>
</table>

Table 4: Ablation on Initial vs. Final answers for GPT-4o-mini before completing multiple back-and-forths between the Dialect and Policy Agents. Scores are averaged across all English dialects.

For PrivacyQA, we use the F1 score at the answer classification level. This metric is appropriate since PrivacyQA is framed as a sentence selection task, where models must determine whether a given sentence correctly answers a privacy-related question.

For PolicyQA, we adopt a token-level F1 score, commonly used in extractive question-answering tasks. This metric calculates the overlap between predicted answer spans and ground-truth answers at the token level. This approach ensures a fair assessment of partial matches, as PolicyQA requires extracting precise answer spans from privacy policy text rather than classifying entire sentences. We also compare the average difference between SAE and the other dialects and the maximum difference for both datasets.

Baselines. We evaluate three models in this paper: Llama 3.1 8B (Dubey et al., 2024), DeepSeek-R1-Distill-Qwen-14B (Guo et al., 2025), and GPT-4o-mini (Hurst et al., 2024). All models are evaluated in zero- and few-shot settings. Moreover, we evaluate them with our multi-agent framework with and without few-shot examples.

Results. We evaluate our multi-agent framework on the PrivacyQA and PolicyQA datasets across SAE and five non-standard English dialects: Rural African American Vernacular English (RAAVE), Jamaican English, Aboriginal English, Welsh English, and Southwest England Dialect (SWE).

Table 2 presents the PrivacyQA results. Our multi-agent framework consistently improves performance across all dialects compared to baseline models. Notably, the GPT-4o-mini Multi-agent-few model achieves the highest average accuracy (0.598), outperforming its few-shot baseline (0.565). The average performance disparity (AVG Diff) is also reduced, with our multi-agent framework achieving a minimum AVG Diff of 0.005, compared to 0.016 in the best-performing baseline. This reduction in disparity underscores the framework’s ability to generalize linguistic fairness across dialects, not just improve raw performance. A similar trend is observed for Llama 3.1 and DeepSeek-R1, where our framework yields notable improvements. The Llama 3.1 Multi-agent-few improves overall performance to 0.530 while reducing AVG Diff to 0.008. DeepSeek-R1 Multi-agent-zero achieves the lowest Max Diff (0.017) among all models, indicating improved fairness across dialects.

These improvements are not limited to low-resource dialects. We observe that even performance on SAE increases slightly in the multi-agent setup, suggesting that the collaborative refinement process benefits all users, not only those using non-standard varieties. Additionally, the DeepSeek-R1 Multi-agent-few model, while showing a slight drop in SAE, achieves substantial gains on challenging dialects like RAAVE (+.0967 over zero-shot) and Jamaican English (+.038), demonstrating the framework’s ability to reallocate capacity toward fairness without large performance trade-offs.

Table 3 shows results for PolicyQA. Our framework again enhances both overall performance and fairness. The GPT-4o-mini Multi-agent-few model achieves an average accuracy of 0.471, improving over the best baseline model (0.449). The disparity across dialects is also reduced, with our framework achieving an AVG Diff of 0.006, compared to 0.014 in the best baseline. For Llama 3.1, our framework improves overall accuracy from 0.371 (few-shot baseline) to 0.387 (multi-agent-few), reducing Max Diff from 0.080 to 0.028. Similarly, DeepSeek-R1 Multi-agent-few achieves an AVG Diff of 0.006, marking a substantial improvement in fairness.

In contrast to PrivacyQA, where zero-shot mod-

![Comparison of the few-shot baseline performance (grey) F_1 scores with the improvements achieved by our method (colored bars) for each model on PrivacyQA. We compare SAE with the two highest-performing dialects for each model.](page_1012_180_388_246.png)

Figure 3: Comparison of the few-shot baseline performance (grey) \( F_1 \) scores with the improvements achieved by our method (colored bars) for each model on PrivacyQA. We compare SAE with the two highest-performing dialects for each model.
<table>
  <tr>
    <th>Approach</th>
    <th>Initial (↑)</th>
    <th>Final (↑)</th>
  </tr>
  <tr>
    <td>With Dialect Info</td>
    <td>.5772</td>
    <td>.5966</td>
  </tr>
  <tr>
    <td>No Dialect Info</td>
    <td>.5210</td>
    <td>.5894</td>
  </tr>
</table>

Table 5: Average \( F_1 \) across dialects on PrivacyQA dataset, comparing With vs. Without dialect-specific background information.

<table>
  <tr>
    <th>Metric</th>
    <th>Zero-shot</th>
    <th>Few-shot</th>
  </tr>
  <tr>
    <td>Disagreements (Overrides)</td>
    <td>22.99%</td>
    <td>31.75%</td>
  </tr>
  <tr>
    <td>Beneficial among Disagreements</td>
    <td>63.4%</td>
    <td>72.1%</td>
  </tr>
  <tr>
    <td>Detrimental among Disagreements</td>
    <td>24.1%</td>
    <td>18.7%</td>
  </tr>
</table>

Table 6: Frequency and impact of Dialect Agent overrides on PrivacyQA

els struggled more, PolicyQA exhibits overall tighter performance bands, making fairness improvements particularly notable. For example, the Llama 3.1 Multi-agent-few model reduces the Max Diff by more than half (from 0.080 to 0.028) while also achieving the highest gains on dialects such as Jamaican and Aboriginal English, with improvements of +.031 and +.028, respectively. DeepSeek-R1 similarly benefits, achieving a high average accuracy of 0.482 with one of the lowest disparities (AVG Diff = 0.006), which demonstrates that the benefits of our multi-agent design generalize across various question formats and task setups.

One of the most striking findings is that the zero-shot performance of our multi-agent framework matches or even surpasses that of the few-shot baselines across multiple models. This demonstrates the ability of our approach to enhance performance without requiring additional in-context examples, making it highly effective in settings where labeled data is limited.

Across both datasets, our multi-agent framework substantially reduces performance disparities between SAE and non-standard dialects. Compared to baseline models, it consistently lowers Max Diff values, demonstrating improved fairness. At the same time, it improves absolute accuracy across all dialects, highlighting its effectiveness in mitigating dialectal biases in privacy-related QA systems.

Ablations and Analysis. In Table 4, we present an ablation focused on the benefit of the iterative collaboration between the Dialect Agent and the Privacy Policy Agent for GPT-4o-mini. We compare system performance at the initial stage—where a translated query is passed to the Privacy Policy Agent for a single-pass answer—against the Final stage, where the Dialect Agent evaluates the initial answer and provides feedback for refinement. We observe consistent improvements in both PrivacyQA (from .53 to .59 \( F_1 \) in zero-shot and .58 to .61 in few-shot) and PolicyQA (.43 to .45 in zero-shot and .47 to .48 in few-shot). These improvements underscore that a single-pass translation of dialectal queries does not fully capture users’ linguistic nuances. While the dialect information helps a lot initially, once the Dialect Agent reviews the Privacy Policy Agent’s answer, it corrects subtle misunderstandings (e.g., colloquial phrasing, dialect-specific grammatical structures), leading to more accurate final predictions. Notably, improvements persist in both zero-shot and few-shot settings, suggesting that agents’ collaboration is effective even without additional in-context examples.

Figure 3 shows how our multi-agent framework improves performance compared to the few-shot baseline on PrivacyQA. The grey bars represent the few-shot baseline, while the colored bars show the improvements from our method. We compare SAE for each model to the top two performing dialects on each model. Overall, we find that our approach improves the top-performing dialects as well. It does not only improve dialects the model does not perform well on (e.g., we see an improvement for SAE). We also find one interesting phenomenon, i.e., DeepSeek-R1 performs best on the Hong Kong English dialect, not SAE.

Next, we investigate the impact of removing dialect-specific background information (e.g., grammar and phonetic features) from the Dialect Agent’s prompt. Intuitively, we may not have access to or even know the dialectal information in complete detail. Hence, here we just prompt with “You are a linguistics expert in English dialects,” without even the dialect name. As shown in Table 5, omitting these linguistic details leads to performance declines at the Initial stage (single-pass answer), dropping from 0.5772 to 0.5210 in average \( F_1 \). Although the Final stage (after iterative refinement) still yields an improvement (up to 0.5894), the performance remains below that of the fully informed system, which reaches 0.5966. Still, even without dialect metadata, the Final stage model improves over the best-performing single-agent baseline (0.5602), yielding +2.9 \( F_1 \). This highlights that explicit knowledge of dialect-specific characteristics is critical for accurately interpreting user queries in non-standard English variants.
Even with iterative agent collaboration, the absence of tailored dialect information constrains how effectively the system can capture nuanced morphological or syntactic cues, eventually reducing the correctness of privacy-policy answers. Please see Appendix A for a complete error analysis.

Finally, we quantify how often the Dialect Agent intervenes and the effect of those interventions. As shown in Table 6, the Dialect agent overrides the Privacy Policy Agent’s initial answer in 22.99% of zero-shot cases and 31.75% in few-shot cases. Among these overrides, 63.4% are beneficial in the zero-shot setting (i.e., correcting an initial error), while 24.1% are detrimental (i.e., introducing a new error). In few-shot, the success rate improves further, with 72.1% of overrides helping and only 18.7% hurting. These results suggest that the Dialect Agent plays a valuable corrective role, refining the output in most cases and contributing meaningfully to the overall performance improvements of the system.

We also observe that override rates vary across dialects, ranging from 14% to 33% (zero-shot) and 16% to 43% (few-shot). Roughly 9% of overrides were neutral, where both initial and final responses were incorrect. These findings highlight the Dialect Agent’s consistent corrective role, particularly for dialects with greater divergence from SAE.

Finally, to assess the quality of these standardized translations (the final translation by the Dialect Agents), we compare them against the original human-authored references in the dataset. The translations achieve a BLEU score of 46.5 and a ROUGE-L score of 80.5, indicating that the Dialect Agent produces fluent and semantically faithful paraphrases of the original dialectal queries. Representative examples of these translations are provided in Appendix D.

Implications. Our results highlight the critical role of incorporating dialect and cultural context in NLP systems. We demonstrate that even when no training data is available for a given dialect, providing minimal but targeted information about the dialect in the prompt can substantially improve model performance. This underscores the importance of designing NLP systems with a deep understanding of their potential users, ensuring that prompts account for linguistic and cultural variations.

Additionally, dialect-aware prompting strategies can serve as lightweight, scalable interventions for fairness in settings where large-scale data collection is infeasible or ethically complex, such as healthcare, legal reasoning, education, or multilingual customer service. In such domains, user trust and accessibility hinge on a system’s ability to reflect users’ linguistic identities.

We acknowledge that explicit dialect labels may not always be available; future work should explore privacy-preserving, unsupervised methods to infer dialectal features directly from user queries. Responsible AI development must extend beyond model selection and fine-tuning. Practitioners must carefully consider how their models interact with diverse user populations and adapt their prompting strategies accordingly. The success of our approach suggests that small, well-informed modifications to prompting strategies can have a meaningful impact, even in zero-shot settings. Looking ahead, we encourage future research on automated dialect detection, richer cultural representations in prompts, and end-to-end integration of multi-agent reasoning to build truly inclusive NLP systems.

5 Conclusion

This work introduces a multi-agent framework to mitigate dialectal biases in privacy question-answering systems. Our approach reduces performance disparities across dialects while improving overall accuracy, demonstrating that incorporating dialect and cultural awareness can enhance NLP model fairness without requiring additional training data. By leveraging targeted prompts, our method achieves results comparable to or better than few-shot baselines in a zero-shot setting, underscoring the potential of structured prompting for equitable NLP applications.

These findings highlight the importance of accounting for linguistic diversity when designing NLP systems. Making language models accessible to users from diverse backgrounds requires prompting strategies that reflect dialectal variation. Future work should explore extending this approach to high-stakes domains such as healthcare, legal AI, and financial services, where language accessibility is critical. It is also important to investigate how dynamically adapting prompts based on user dialect can enhance real-time interactions with LLMs. Finally, exploring automated dialect detection mechanisms (e.g., in multicultural households) and integrating multi-agent collaboration into broader NLP pipelines could further advance fairness and inclusivity in large-scale language models.
Acknowledgments

This material is based upon work supported by the National Science Foundation (NSF) under Grant No. 2145357.

Limitations

While our multi-agent framework effectively mitigates dialect biases in privacy policy QA, it has several limitations. First, our approach relies on synthetic dialectal data generated using rule-based transformations, which may not fully capture the nuances of naturally occurring dialect variations. Future work should evaluate performance on real-world dialectal data and user-generated queries to ensure robustness. Second, while our framework reduces performance disparities, some dialects still exhibit lower accuracy compared to Standard American English (SAE). This suggests that further refinements in the Dialect Agent’s translation capabilities may be needed to preserve contextual nuances more effectively. Third, our method depends on accurate dialect metadata to select the appropriate linguistic adaptation strategy. In cases where dialect information is unavailable or ambiguous, performance gains may be limited. Finally, our study focuses on English dialects, and it remains an open question how well this framework generalizes to other languages with diverse linguistic variations.

Ethical Implications

Our work highlights important ethical considerations in the development of NLP systems, particularly for high-stakes applications like privacy policy QA. By reducing dialectal disparities, our framework improves access to critical privacy information for speakers of non-standard English varieties, promoting fairness and inclusivity. However, dialect adaptation raises concerns about linguistic representation and cultural preservation. While translation into SAE may improve comprehension, it may also reinforce dominant linguistic norms at the expense of dialectal authenticity. Future research should explore methods that balance accessibility with dialectal preservation, ensuring that speakers of all linguistic backgrounds feel represented in NLP systems. Additionally, our study underscores the broader need for AI systems to consider sociolinguistic diversity in their design. Developers must be mindful of biases in training data, evaluation metrics, and system outputs to avoid perpetuating inequities in AI-driven decision-making. Further, our approach requires transparency in how dialect adaptation decisions are made, emphasizing the need for user agency in interacting with privacy policy QA systems.

References

Wasi Ahmad, Jianfeng Chi, Tu Le, Thomas Norton, Yuan Tian, and Kai-Wei Chang. 2021. Intent classification and slot filling for privacy policies. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4402–4417, Online. Association for Computational Linguistics.

Wasi Ahmad, Jianfeng Chi, Yuan Tian, and Kai-Wei Chang. 2020. PolicyQA: A reading comprehension dataset for privacy policies. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 743–749, Online. Association for Computational Linguistics.

R. Amos et al. 2021. Privacy policies over time: Curation and analysis of a million-document dataset. In Proceedings of the Web Conference, pages 2165–2176.

Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623.

Su Lin Blodgett and Brendan O’Connor. 2018. Twitter universal dependency parsing for african-american english. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics, pages 1405–1415.

Viet Dung Bui et al. 2021. Pi-extract: A dataset for extracting personal information from privacy policies. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1234–1245.

Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje F. Karlsson, Jie Fu, and Yemin Shi. 2024. Autoagents: A framework for automatic agent generation. Preprint, arXiv:2309.17288.

Hao Chen, Yi Zhou, and Qi Zhang. 2023. Multi-agent reasoning for complex task decomposition in nlp. In Findings of the Association for Computational Linguistics, pages 4567–4582.

Jianfeng Chi, Wasi Uddin Ahmad, Yuan Tian, and Kai-Wei Chang. 2023. PLUE: Language understanding evaluation benchmark for privacy policies in English. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume
2: Short Papers), pages 352–365, Toronto, Canada. Association for Computational Linguistics.

Mike Cooley. 2000. Human-centered design. Information design, pages 59–81.

Thomas Davidson, Debashmita Bhattacharya, and Ingmar Weber. 2019. Racial bias in hate speech and abusive language detection datasets. In Proceedings of the Third Workshop on Abusive Language Online, pages 25–35, Florence, Italy. Association for Computational Linguistics.

Dorotyya Demeszký, Weijie Gong, and Diyi Yang. 2019. Analyzing bias and framing in news articles on artificial intelligence: The case of sentiment analysis. In Proceedings of the 2019 Annual Conference on Empirical Methods in Natural Language Processing, pages 1655–1661.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Electronic Privacy Information Center. 2025. Privacy and Racial Justice. Accessed: 2025-02-16.

Fahim Faisal, Orevaghenne Ahia, Aarohi Srivastava, Kabir Ahuja, David Chiang, Yulia Tsvetkov, and Antonios Anastasopoulos. 2024. DIALECTBENCH: An NLP benchmark for dialects, varieties, and closely-related languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14412–14454, Bangkok, Thailand. Association for Computational Linguistics.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.

William Held, Caleb Ziemis, and Diyi Yang. 2023. TADA : Task agnostic dialect adapters for English. In Findings of the Association for Computational Linguistics: ACL 2023, pages 813–824, Toronto, Canada. Association for Computational Linguistics.

Dirk Hovy and Shannon L. Spruit. 2016. Social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 591–598.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Anna Jørgensen, Dirk Hovy, and Anders Søgaard. 2016. Learning a POS tagger for AAVE-like language. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1115–1120, San Diego, California. Association for Computational Linguistics.

David Jurgens, Tim Finethy, James McCorriston, Yi Tian Xu, and Derek Ruths. 2017. Geolocation prediction in twitter using social networks: A critical analysis and review of current practice. In Proceedings of the Ninth International Conference on Web and Social Media, pages 188–197.

Dorde Klisura and Anthony Rios. 2024. Unmasking database vulnerabilities: Zero-knowledge schema inference attacks in text-to-sql systems. arXiv preprint arXiv:2406.14545.

Bernd Kortmann et al. 2020. ewave: Electronic world atlas of varieties of english. In Linguistic Atlas Frameworks.

Seungyeon Lee and Minho Lee. 2022. Type-dependent prompt CycleQAG : Cycle consistency for multi-hop question generation. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6301–6314, Gyongju, Republic of Korea. International Committee on Computational Linguistics.

Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizhbullin, and Bernard Ghanem. 2023. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991–52008.

Xinyi Li, Sai Wang, Siqi Zeng, Yu Wu, and Yi Yang. 2024. A survey on llm-based multi-agent systems: workflow, infrastructure, and challenges. Viciagearth, 1(1):9.

Chen Cecilia Liu, Iryna Gurevych, and Anna Korhonen. 2024. Culturally aware and adapted nlp: A taxonomy and a survey of the state of the art. ArXiv preprint, abs/2406.03930.

Wei Liu, Hongjun Xiong, and Xinyu Peng. 2023a. Knowledge-augmented multi-agent systems for efficient document retrieval. In Proceedings of the 2023 Annual Meeting of the Association for Computational Linguistics, pages 894–910.

Yanchen Liu, William Held, and Diyi Yang. 2023b. DADA: Dialect adaptation via dynamic aggregation of linguistic rules. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13776–13793, Singapore. Association for Computational Linguistics.

B Lwowski and A Rios. 2021. The risk of racial bias while tracking influenza-related content on social media using machine learning. Journal of the American Medical Informatics Association: JAMIA, 28(4):839–849.

Brandon Lwowski, Paul Rad, and Anthony Rios. 2022. Measuring geographic performance disparities of offensive language classifiers. In Proceedings of the
29th International Conference on Computational Linguistics, pages 6600–6616, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.

S. Manandhar et al. 2022. Smart home privacy policies demystified: A study of availability, content, and coverage. In 31st USENIX Security Symposium, pages 3521–3538.

Tarek Naous, Michael J Ryan, Alan Ritter, and Wei Xu. 2024. Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16366–16393, Bangkok, Thailand. Association for Computational Linguistics.

Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. 2023. Chatdev: Communicative agents for software development. arXiv preprint arXiv:2307.07924.

R. Ramanath, F. Liu, N. Sadeh, and N. A. Smith. 2014. Unsupervised alignment of privacy policies using hidden markov models. In Proceedings of ACL.

Abhilasha Ravichander, Alan W Black, Shomir Wilson, Thomas Norton, and Norman Sadeh. 2019. Question answering for privacy policies: Combining computational and legal perspectives. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4947–4958, Hong Kong, China. Association for Computational Linguistics.

Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678, Florence, Italy. Association for Computational Linguistics.

M. Srinath, S. Wilson, and C. L. Giles. 2021. Privacy at scale: Introducing the privaseer corpus of web privacy policies. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6829–6839.

Zhen Sun, Haoyue Liu, and Yi Zhang. 2023. Dialect-aware machine translation for chinese regional variants. In Proceedings of the 2023 Annual Meeting of the Association for Computational Linguistics, pages 115–129.

S. Wilson et al. 2016. The creation and analysis of a website privacy policy corpus. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1330–1340.

Jing Xu, Lin Wang, and Xuanjing Huang. 2023. Structured decision-making in multi-agent nlp frameworks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1412–1425.

Fangyi Yu, Lee Quartey, and Frank Schilder. 2023. Exploring the effectiveness of prompt engineering for legal reasoning tasks. In Findings of the Association for Computational Linguistics: ACL 2023, pages 13582–13596, Toronto, Canada. Association for Computational Linguistics.

Jun Zhao, Can Zu, Hao Xu, Yi Lu, Wei He, Yiwen Ding, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024a. Longagent: Achieving question answering for 128k-token-long documents through multi-agent collaboration. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16310–16324.

Xiutian Zhao, Ke Wang, and Wei Peng. 2024b. An electoral approach to diversify llm-based multi-agent collective decision-making. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 2712–2727.

Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023. Multi-VALUE: A framework for cross-dialectal English NLP. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 744–768, Toronto, Canada. Association for Computational Linguistics.

A Error Analysis

Our error analysis indicates that performance variations across dialects likely stem from training data biases, as less-represented dialects consistently yielded lower final F1 scores, suggesting challenges in capturing subtle linguistic nuances. In some cases, the multiagent framework’s refinement process yielded marginal improvements, yet in other examples, adjustments introduced new errors, particularly for dialects with complex or idiomatic expressions.

In the PolicyQA task, for instance, one error involved the segment

Last Updated on May 22, 2015

paired with the question “Do you take the user’s opinion before or after making changes in policy,” where the annotated answer was “Last Updated on May 22, 2015“. This example shows how the model mistakenly extracted meta-information as the answer rather than identifying the procedural detail requested by the question. In another example, the question “Does the privacy policy mention anything about children,” was paired with a lengthy segment
You can jump to specific areas of our Privacy Policy by clicking on the links below, or you can read on for the full Privacy Policy: Information We Collect How We Use Personal Information We Collect How We May Disclose Personal Information We Collect How We May Use or Disclose Other Information We Collect Your Options How We Protect Your Personal Information Cookies Social Networking and Third Party Sites California Users’ Privacy Rights Children’s Online Privacy International Contact Us

and the annotated answer was “Children’s.” Here, the generative models’ tendency to provide longer, more contextually diffuse answers led them to miss the succinct, targeted answer. These examples underscore a common issue with large language models: their inclination to generate overly verbose responses, which highlights the need for more targeted fine-tuning and improved context disambiguation for precise answer extraction.

B Prompts for Dialect and Privacy Policy Agents

To implement our multi-agent framework, we designed two specialized agents: the Dialect Agent and the Privacy Policy Agent. The Dialect Agent is responsible for translating user queries from a given dialect into Standard American English (SAE) while preserving the original intent. Additionally, it plays a critical role in validating the responses generated by the Privacy Policy Agent. The Privacy Policy Agent processes the translated queries, retrieving relevant information from a given privacy policy and determining whether a policy segment is Relevant or Irrelevant to the question.

The following subsections describe the prompts used to guide each agent at different stages of our method.

B.1 Dialect Agent Prompts

B.1.1 Initial Translation Prompt

The Dialect Agent first translates a user’s query from a non-standard English dialect into Standard American English (SAE). This translation ensures that downstream processing by the Privacy Policy Agent is not negatively impacted by dialectal variations.

Dialect Agent: Initial Translation

SYSTEM PROMPT
You are an expert linguist specializing in the following dialect:
{dialect_info}
Your task is to translate the following question from this dialect into clear, Standard American English. Ensure that the translation is easily understandable to a general audience. Please provide only the translated question and do not include any additional text.
USER MESSAGE
{question}

At this stage, no feedback from the Privacy Policy Agent is available. The Dialect Agent simply returns the translated question.

B.1.2 Responding to Expert Feedback

After the Privacy Policy Agent classifies a privacy policy segment as Relevant or Irrelevant, the Dialect Agent evaluates whether the classification is consistent with the original intent of the user’s question in their dialect.

Dialect Agent: Evaluating Privacy Agent’s Response

SYSTEM PROMPT
You are an expert linguist specializing in the following dialect, with expertise in privacy policies.
Previously, you translated a question from this dialect into Standard American English. Now, you need to critically assess whether the Privacy Policy Agent’s classification accurately reflects the meaning of the original question in the dialect.
Privacy Policy Segment:
{privacy_policy_segment}
Original Question in Dialect:
{question}
The Privacy Policy Agent has classified the policy segment as '{Classification}' with the following reasoning:
{reasoning}
Based on your understanding of the dialect and its nuances, analyze the expert’s classification and reasoning. Do you find any discrepancies or misunderstandings? Please provide a detailed explanation and conclude with either 'Agree' if you concur with the classification or 'Disagree' if you do not.

If the Dialect Agent disagrees, the Privacy Policy Agent will be prompted to reconsider its classification based on the Dialect Agent’s insights.

B.2 Privacy Policy Agent Prompts

B.2.1 Initial Classification Prompt

The Privacy Policy Agent is responsible for determining whether a privacy policy segment is relevant to a user’s question. In PrivacyQA, this classification is binary (Relevant or Irrelevant), while in PolicyQA, the Privacy Policy Agent provides a direct answer based on the policy text.
Privacy Policy Agent: Initial Classification

SYSTEM PROMPT
You are a privacy policy expert. Your task is to determine whether the provided privacy policy segment is 'Relevant' or 'Irrelevant' to the question, based on the following definitions:

Definitions:
- Relevant: The policy segment directly addresses the question.
- Irrelevant: The policy segment does not directly address the question.
Please analyze the material below and provide: 1. A brief explanation of your reasoning. 2. Conclude only with 'Label: Relevant' or 'Label: Irrelevant'.

USER MESSAGE
Privacy Policy Segment:
{privacy_policy_segment}
Question:
{translated_question}

In this zero-shot setup, the Privacy Policy Agent classifies the segment and explains its decision.

B.2.2 Reconsideration Prompt (After Dialect Feedback)

If the Dialect Agent disagrees with the Privacy Policy Agent’s classification, the Privacy Policy Agent is asked to reevaluate its decision. This step ensures that dialectal nuances are reflected in the final classification.

Privacy Policy Agent: Reconsideration After Dialect Feedback

SYSTEM PROMPT
You are a privacy policy expert. Previously, you classified the privacy policy segment as '{previous_classification}' regarding the question, with the following reasoning:
{previous_reasoning}
However, the Dialect Agent has provided additional insights and disagrees with your classification. Their reasoning is as follows:
{dialect_reasoning}
Please reconsider your initial decision in light of this new information. Provide: 1. A brief explanation of your reconsidered decision. 2. Conclude with 'Final Label: Relevant' or 'Final Label: Irrelevant'.

If the Dialect Agent’s feedback indicates a misclassification, the Privacy Policy Agent revises its response to better match the user’s intent; if the classification is correct, it retains its decision and provides additional justification.

C Dialect Details

In this section, we provide examples of the dialect information we give to the LLMs to help them better understand linguistic variations. Each dialect entry includes key phonetic, grammatical, and vocabulary differences compared to Standard American English (SAE), along with cultural context. This information helps the model accurately translate dialectal queries while preserving their meaning. For example, Indian English includes retroflex consonants and distinct grammatical patterns, while Jamaican English (Patois) features non-rhotic pronunciation and unique verb structures. By incorporating these details, our framework improves the model’s ability to handle dialect-specific nuances in privacy policy question-answering.

Here is an example of the Indian English prompt:

Indian Dialect
Key Features of Indian English

Phonetics and Pronunciation:
- Retroflex consonants influenced by Indian languages.
- Variable stress and intonation patterns.
- Vowel pronunciation often closer to native Indian languages.

Grammar:
- Use of present continuous for habitual actions (e.g., 'I am knowing').
- Omission of articles and prepositions in certain contexts.
- Use of Indian syntax and sentence structures.

Vocabulary:
- Incorporation of Hindi, Tamil, Bengali, and other Indian language terms
- Unique expressions and idioms specific to Indian culture.

Cultural Notes:
- Reflects India’s diverse linguistic landscape.
- Widely used in Indian media, education, and business.

Here is an example of the Jamaican English prompt:

Jamaican English
Key Features of Jamaican English (Jamaican Patois)

Phonetics and Pronunciation:
- Non-rhotic pronunciation with 'r' often not pronounced.
- Use of tone and pitch influenced by African languages.
- Simplified consonant clusters and vowel shifts.

Grammar:
- Use of particles like 'fi' (to) and 'a' (progressive aspect).
- Simplified tense markers and verb forms.
- Use of double negatives for emphasis.

Vocabulary: - Extensive borrowing from West African languages, Spanish, and English.
- Unique slang and expressions reflecting Jamaican culture.

Cultural Notes: - Central to Jamaican music genres like reggae and dancehall.
- Reflects the island’s history and multicultural influences.
<table>
  <tr>
    <th>Dialectal Input (AAVE)</th>
    <th>Dialect Agent Translation (SAE)</th>
  </tr>
  <tr>
    <td>It is access to my information?<br>gon for me test results be shared with any third party?<br>what information it is access to that collaborators ?<br>which information, if any, do that app sell to other people?<br>do the app need any special permission for to run ?</td>
    <td>Who is going to have access to my information?<br>Will my test results be shared with any third-party?<br>What information do the collaborators have access to?<br>What information, if any, does that app sell to others?<br>Does the app need any special permissions to run?</td>
  </tr>
</table>

Table 7: Examples of AAVE queries and their SAE translations produced by the Dialect Agent. No hallucinated content was observed across over 500 spot-checked samples.

D  Dialect Translation Examples

To evaluate the reliability of the Dialect Agent’s output, we manually inspected over 500 SAE translations produced by the agent when translating dialectal queries (e.g., AAVE) from the Multi-VALUE benchmark. We found no instances of hallucination, i.e., the agent did not invent new content, facts, or answer components. This outcome is expected given the bounded task design: translating sentence-level questions from dialectal English into Standard American English (SAE), often involving paraphrasing rather than generation from scratch.

Table 7 shows representative examples of AAVE queries and the Dialect Agent’s SAE translations. These illustrate how the agent improves clarity while maintaining user intent and factual fidelity.

E  Resources

All experiments were trained on a server with two NVIDIA A6000 GPUs.

F  Full Results

This section shows all of the results for all 50 dialects generated using the Multi-Value framework. See Tables 8, 9, 10, and 11. Table 12 shows the full dialect results without any specific dialect (they are a general dialect expert) information is passed directly to the dialect agent.
Table 8: Baseline Results for GPT-4, Llama 3.1, and DeepSeek-R1 on PrivacyQA (PQA) and PolicyQA (PoQA). “PQA 0” = PrivacyQA Zero-shot, “PQA F” = PrivacyQA Few-shot, “PoQA 0” = PolicyQA Zero-shot, “PoQA F” = PolicyQA Few-shot.

<table>
  <tr>
    <th rowspan="2">Dialect</th>
    <th colspan="4">GPT-4</th>
    <th colspan="4">Llama 3.1</th>
    <th colspan="4">DeepSeek-R1</th>
  </tr>
  <tr>
    <th>PQA 0</th>
    <th>PQA F</th>
    <th>PoQA 0</th>
    <th>PoQA F</th>
    <th>PQA 0</th>
    <th>PQA F</th>
    <th>PoQA 0</th>
    <th>PoQA F</th>
    <th>PQA 0</th>
    <th>PQA F</th>
    <th>PoQA 0</th>
    <th>PoQA F</th>
  </tr>
  <tr><td>Standard American Dialect</td><td>.394</td><td>.605</td><td>.352</td><td>.478</td><td>.469</td><td>.546</td><td>.310</td><td>.412</td><td>.532</td><td>.581</td><td>.455</td><td>.446</td></tr>
  <tr><td>Kenyan Dialect</td><td>.386</td><td>.595</td><td>.337</td><td>.439</td><td>.430</td><td>.465</td><td>.247</td><td>.380</td><td>.536</td><td>.570</td><td>.425</td><td>.466</td></tr>
  <tr><td>Sri Lankan Dialect</td><td>.386</td><td>.595</td><td>.336</td><td>.447</td><td>.438</td><td>.453</td><td>.256</td><td>.371</td><td>.531</td><td>.571</td><td>.435</td><td>.500</td></tr>
  <tr><td>Scottish Dialect</td><td>.385</td><td>.594</td><td>.315</td><td>.454</td><td>.420</td><td>.473</td><td>.285</td><td>.375</td><td>.539</td><td>.585</td><td>.439</td><td>.487</td></tr>
  <tr><td>Malaysian Dialect</td><td>.380</td><td>.592</td><td>.333</td><td>.451</td><td>.403</td><td>.488</td><td>.239</td><td>.364</td><td>.532</td><td>.567</td><td>.421</td><td>.486</td></tr>
  <tr><td>Indian Dialect</td><td>.379</td><td>.591</td><td>.333</td><td>.433</td><td>.376</td><td>.487</td><td>.208</td><td>.340</td><td>.535</td><td>.557</td><td>.408</td><td>.473</td></tr>
  <tr><td>Chicano Dialect</td><td>.379</td><td>.580</td><td>.320</td><td>.441</td><td>.456</td><td>.467</td><td>.287</td><td>.365</td><td>.532</td><td>.591</td><td>.458</td><td>.498</td></tr>
  <tr><td>Cameron Dialect</td><td>.378</td><td>.580</td><td>.342</td><td>.430</td><td>.390</td><td>.484</td><td>.246</td><td>.346</td><td>.541</td><td>.539</td><td>.435</td><td>.473</td></tr>
  <tr><td>Irish Dialect</td><td>.377</td><td>.584</td><td>.349</td><td>.451</td><td>.469</td><td>.510</td><td>.248</td><td>.353</td><td>.535</td><td>.551</td><td>.437</td><td>.468</td></tr>
  <tr><td>Nigerian Dialect</td><td>.375</td><td>.582</td><td>.324</td><td>.463</td><td>.469</td><td>.487</td><td>.240</td><td>.375</td><td>.540</td><td>.592</td><td>.426</td><td>.478</td></tr>
  <tr><td>Appalachian Dialect</td><td>.375</td><td>.583</td><td>.320</td><td>.436</td><td>.439</td><td>.462</td><td>.244</td><td>.365</td><td>.538</td><td>.560</td><td>.458</td><td>.487</td></tr>
  <tr><td>White South African Dialect</td><td>.373</td><td>.584</td><td>.320</td><td>.439</td><td>.423</td><td>.487</td><td>.257</td><td>.386</td><td>.551</td><td>.557</td><td>.444</td><td>.461</td></tr>
  <tr><td>Channel Islands Dialect</td><td>.372</td><td>.581</td><td>.324</td><td>.438</td><td>.431</td><td>.465</td><td>.263</td><td>.376</td><td>.538</td><td>.559</td><td>.409</td><td>.456</td></tr>
  <tr><td>Southeast American Enclave Dialect</td><td>.372</td><td>.579</td><td>.328</td><td>.444</td><td>.391</td><td>.370</td><td>.262</td><td>.370</td><td>.551</td><td>.563</td><td>.432</td><td>.473</td></tr>
  <tr><td>Ugandan Dialect</td><td>.372</td><td>.578</td><td>.331</td><td>.449</td><td>.433</td><td>.470</td><td>.246</td><td>.363</td><td>.551</td><td>.578</td><td>.422</td><td>.453</td></tr>
  <tr><td>Liberian Settler Dialect</td><td>.371</td><td>.577</td><td>.326</td><td>.444</td><td>.377</td><td>.478</td><td>.270</td><td>.376</td><td>.553</td><td>.545</td><td>.417</td><td>.481</td></tr>
  <tr><td>Cape Flats Dialect</td><td>.370</td><td>.576</td><td>.328</td><td>.444</td><td>.440</td><td>.465</td><td>.257</td><td>.381</td><td>.535</td><td>.570</td><td>.411</td><td>.468</td></tr>
  <tr><td>Tristan Dialect</td><td>.368</td><td>.575</td><td>.324</td><td>.439</td><td>.393</td><td>.466</td><td>.251</td><td>.339</td><td>.540</td><td>.549</td><td>.447</td><td>.466</td></tr>
  <tr><td>Ozark Dialect</td><td>.368</td><td>.574</td><td>.328</td><td>.442</td><td>.410</td><td>.502</td><td>.290</td><td>.381</td><td>.530</td><td>.559</td><td>.453</td><td>.446</td></tr>
  <tr><td>Australian Dialect</td><td>.367</td><td>.574</td><td>.321</td><td>.434</td><td>.436</td><td>.521</td><td>.250</td><td>.355</td><td>.543</td><td>.557</td><td>.416</td><td>.461</td></tr>
  <tr><td>Tanzanian Dialect</td><td>.366</td><td>.573</td><td>.333</td><td>.452</td><td>.401</td><td>.482</td><td>.264</td><td>.382</td><td>.536</td><td>.569</td><td>.446</td><td>.509</td></tr>
  <tr><td>Peri-Acrolect</td><td>.364</td><td>.571</td><td>.333</td><td>.445</td><td>.406</td><td>.500</td><td>.288</td><td>.382</td><td>.557</td><td>.570</td><td>.458</td><td>.475</td></tr>
  <tr><td>Fijian Dialect</td><td>.364</td><td>.571</td><td>.338</td><td>.460</td><td>.344</td><td>.506</td><td>.228</td><td>.381</td><td>.547</td><td>.518</td><td>.448</td><td>.441</td></tr>
  <tr><td>Pakistani Dialect</td><td>.364</td><td>.569</td><td>.319</td><td>.430</td><td>.392</td><td>.427</td><td>.260</td><td>.359</td><td>.533</td><td>.574</td><td>.428</td><td>.447</td></tr>
  <tr><td>Philippine Dialect</td><td>.363</td><td>.568</td><td>.349</td><td>.471</td><td>.370</td><td>.506</td><td>.240</td><td>.366</td><td>.552</td><td>.548</td><td>.440</td><td>.479</td></tr>
  <tr><td>White Zimbabwean Dialect</td><td>.363</td><td>.567</td><td>.330</td><td>.449</td><td>.425</td><td>.465</td><td>.260</td><td>.352</td><td>.537</td><td>.582</td><td>.433</td><td>.468</td></tr>
  <tr><td>Newfoundland Dialect</td><td>.362</td><td>.566</td><td>.319</td><td>.428</td><td>.394</td><td>.508</td><td>.264</td><td>.374</td><td>.526</td><td>.556</td><td>.420</td><td>.493</td></tr>
  <tr><td>Orkney Shetland Dialect</td><td>.362</td><td>.565</td><td>.335</td><td>.454</td><td>.452</td><td>.494</td><td>.250</td><td>.380</td><td>.530</td><td>.561</td><td>.443</td><td>.490</td></tr>
  <tr><td>East Anglican Dialect</td><td>.361</td><td>.564</td><td>.319</td><td>.422</td><td>.412</td><td>.466</td><td>.246</td><td>.374</td><td>.527</td><td>.559</td><td>.422</td><td>.478</td></tr>
  <tr><td>Early African American Vernacular</td><td>.358</td><td>.563</td><td>.319</td><td>.423</td><td>.393</td><td>.465</td><td>.231</td><td>.373</td><td>.549</td><td>.560</td><td>.430</td><td>.478</td></tr>
  <tr><td>Falkland Islands Dialect</td><td>.358</td><td>.562</td><td>.333</td><td>.451</td><td>.439</td><td>.475</td><td>.268</td><td>.365</td><td>.535</td><td>.574</td><td>.453</td><td>.484</td></tr>
  <tr><td>Australian Vernacular</td><td>.357</td><td>.561</td><td>.329</td><td>.448</td><td>.398</td><td>.479</td><td>.240</td><td>.387</td><td>.537</td><td>.579</td><td>.453</td><td>.468</td></tr>
  <tr><td>Black South African Dialect</td><td>.356</td><td>.560</td><td>.311</td><td>.420</td><td>.381</td><td>.461</td><td>.228</td><td>.364</td><td>.541</td><td>.551</td><td>.455</td><td>.497</td></tr>
  <tr><td>Colloquial American Dialect</td><td>.354</td><td>.559</td><td>.320</td><td>.447</td><td>.375</td><td>.486</td><td>.276</td><td>.361</td><td>.526</td><td>.572</td><td>.439</td><td>.471</td></tr>
  <tr><td>Indian South African Dialect</td><td>.353</td><td>.558</td><td>.316</td><td>.454</td><td>.374</td><td>.467</td><td>.207</td><td>.352</td><td>.541</td><td>.554</td><td>.447</td><td>.489</td></tr>
  <tr><td>New Zealand Dialect</td><td>.353</td><td>.557</td><td>.344</td><td>.464</td><td>.387</td><td>.494</td><td>.241</td><td>.345</td><td>.550</td><td>.567</td><td>.434</td><td>.473</td></tr>
  <tr><td>Bahamian Dialect</td><td>.352</td><td>.556</td><td>.325</td><td>.441</td><td>.345</td><td>.458</td><td>.241</td><td>.352</td><td>.537</td><td>.526</td><td>.448</td><td>.473</td></tr>
  <tr><td>Hong Kong Dialect</td><td>.351</td><td>.555</td><td>.336</td><td>.455</td><td>.406</td><td>.503</td><td>.237</td><td>.342</td><td>.566</td><td>.596</td><td>.465</td><td>.497</td></tr>
  <tr><td>Colloquial Singapore Dialect</td><td>.350</td><td>.554</td><td>.346</td><td>.464</td><td>.384</td><td>.463</td><td>.210</td><td>.370</td><td>.538</td><td>.529</td><td>.434</td><td>.434</td></tr>
  <tr><td>Manx Dialect</td><td>.349</td><td>.553</td><td>.337</td><td>.457</td><td>.403</td><td>.513</td><td>.242</td><td>.386</td><td>.534</td><td>.551</td><td>.436</td><td>.466</td></tr>
  <tr><td>African American Vernacular</td><td>.348</td><td>.552</td><td>.325</td><td>.441</td><td>.376</td><td>.441</td><td>.269</td><td>.362</td><td>.539</td><td>.560</td><td>.438</td><td>.491</td></tr>
  <tr><td>Southeast England Dialect</td><td>.348</td><td>.551</td><td>.328</td><td>.445</td><td>.433</td><td>.455</td><td>.245</td><td>.372</td><td>.548</td><td>.580</td><td>.436</td><td>.477</td></tr>
  <tr><td>Rural African American Vernacular</td><td>.344</td><td>.550</td><td>.343</td><td>.463</td><td>.349</td><td>.463</td><td>.260</td><td>.332</td><td>.510</td><td>.549</td><td>.436</td><td>.483</td></tr>
  <tr><td>Maltese Dialect</td><td>.342</td><td>.549</td><td>.343</td><td>.463</td><td>.348</td><td>.492</td><td>.242</td><td>.352</td><td>.525</td><td>.548</td><td>.446</td><td>.480</td></tr>
  <tr><td>Irish Dialect</td><td>.337</td><td>.547</td><td>.335</td><td>.454</td><td>.368</td><td>.502</td><td>.222</td><td>.368</td><td>.542</td><td>.529</td><td>.403</td><td>.483</td></tr>
  <tr><td>Jamaican Dialect</td><td>.332</td><td>.545</td><td>.332</td><td>.450</td><td>.370</td><td>.469</td><td>.268</td><td>.360</td><td>.541</td><td>.547</td><td>.429</td><td>.468</td></tr>
  <tr><td>Portuguese Dialect</td><td>.329</td><td>.543</td><td>.338</td><td>.455</td><td>.325</td><td>.448</td><td>.231</td><td>.357</td><td>.529</td><td>.517</td><td>.437</td><td>.472</td></tr>
  <tr><td>North England Dialect</td><td>.328</td><td>.541</td><td>.325</td><td>.442</td><td>.379</td><td>.467</td><td>.234</td><td>.369</td><td>.550</td><td>.565</td><td>.427</td><td>.454</td></tr>
  <tr><td>St Helena Dialect</td><td>.322</td><td>.539</td><td>.349</td><td>.472</td><td>.382</td><td>.506</td><td>.249</td><td>.360</td><td>.536</td><td>.539</td><td>.426</td><td>.472</td></tr>
  <tr><td>Welsh Dialect</td><td>.312</td><td>.537</td><td>.331</td><td>.449</td><td>.356</td><td>.485</td><td>.237</td><td>.393</td><td>.532</td><td>.556</td><td>.422</td><td>.492</td></tr>
  <tr><td>Southwest England Dialect</td><td>.301</td><td>.535</td><td>.323</td><td>.436</td><td>.336</td><td>.446</td><td>.289</td><td>.370</td><td>.512</td><td>.541</td><td>.422</td><td>.477</td></tr>
</table>
Table 9: MultiAgent Framework Results for GPT-4 on PrivacyQA and PolicyQA

<table>
  <tr>
    <th>Dialect</th>
    <th colspan="2">PrivacyQA Zero-shot</th>
    <th colspan="2">PrivacyQA Few-shot</th>
    <th colspan="2">PolicyQA Zero-shot</th>
    <th colspan="2">PolicyQA Few-shot</th>
  </tr>
  <tr>
    <th></th>
    <th>Initial</th>
    <th>Final</th>
    <th>Initial</th>
    <th>Final</th>
    <th>Initial</th>
    <th>Final</th>
    <th>Initial</th>
    <th>Final</th>
  </tr>
  <tr><td>Standard American Dialect</td><td>.532</td><td>.610</td><td>.608</td><td>.611</td><td>.444</td><td>.464</td><td>.481</td><td>.484</td></tr>
  <tr><td>Tanzanian Dialect</td><td>.531</td><td>.586</td><td>.580</td><td>.588</td><td>.437</td><td>.457</td><td>.478</td><td>.481</td></tr>
  <tr><td>Manx Dialect</td><td>.531</td><td>.581</td><td>.572</td><td>.600</td><td>.442</td><td>.460</td><td>.478</td><td>.481</td></tr>
  <tr><td>Orkney Shetland Dialect</td><td>.527</td><td>.579</td><td>.574</td><td>.602</td><td>.442</td><td>.457</td><td>.474</td><td>.478</td></tr>
  <tr><td>New Zealand Dialect</td><td>.527</td><td>.576</td><td>.576</td><td>.598</td><td>.440</td><td>.461</td><td>.474</td><td>.478</td></tr>
  <tr><td>Nigerian Dialect</td><td>.532</td><td>.588</td><td>.573</td><td>.600</td><td>.441</td><td>.462</td><td>.474</td><td>.478</td></tr>
  <tr><td>East Anglican Dialect</td><td>.528</td><td>.587</td><td>.578</td><td>.604</td><td>.427</td><td>.455</td><td>.474</td><td>.478</td></tr>
  <tr><td>African American Vernacular</td><td>.529</td><td>.570</td><td>.577</td><td>.598</td><td>.424</td><td>.460</td><td>.473</td><td>.477</td></tr>
  <tr><td>Early African American Vernacular</td><td>.527</td><td>.583</td><td>.577</td><td>.587</td><td>.433</td><td>.459</td><td>.472</td><td>.476</td></tr>
  <tr><td>Black South African Dialect</td><td>.533</td><td>.594</td><td>.577</td><td>.598</td><td>.421</td><td>.451</td><td>.471</td><td>.475</td></tr>
  <tr><td>Jamaican Dialect</td><td>.529</td><td>.578</td><td>.577</td><td>.596</td><td>.426</td><td>.451</td><td>.471</td><td>.475</td></tr>
  <tr><td>Newfoundland Dialect</td><td>.528</td><td>.581</td><td>.576</td><td>.600</td><td>.436</td><td>.452</td><td>.471</td><td>.475</td></tr>
  <tr><td>Australian Vernacular</td><td>.528</td><td>.604</td><td>.579</td><td>.601</td><td>.423</td><td>.454</td><td>.470</td><td>.475</td></tr>
  <tr><td>Irish Dialect</td><td>.526</td><td>.575</td><td>.577</td><td>.589</td><td>.433</td><td>.450</td><td>.470</td><td>.474</td></tr>
  <tr><td>Fiji Basilect</td><td>.525</td><td>.586</td><td>.576</td><td>.596</td><td>.427</td><td>.451</td><td>.469</td><td>.474</td></tr>
  <tr><td>North England Dialect</td><td>.525</td><td>.584</td><td>.579</td><td>.601</td><td>.437</td><td>.450</td><td>.469</td><td>.474</td></tr>
  <tr><td>Scottish Dialect</td><td>.529</td><td>.580</td><td>.576</td><td>.602</td><td>.427</td><td>.456</td><td>.469</td><td>.474</td></tr>
  <tr><td>St Helena Dialect</td><td>.529</td><td>.597</td><td>.581</td><td>.602</td><td>.425</td><td>.449</td><td>.468</td><td>.473</td></tr>
  <tr><td>Aboriginal Dialect</td><td>.528</td><td>.587</td><td>.581</td><td>.602</td><td>.418</td><td>.458</td><td>.468</td><td>.473</td></tr>
  <tr><td>Pakistani Dialect</td><td>.529</td><td>.597</td><td>.576</td><td>.597</td><td>.420</td><td>.451</td><td>.468</td><td>.472</td></tr>
  <tr><td>Malaysian Dialect</td><td>.529</td><td>.581</td><td>.576</td><td>.598</td><td>.436</td><td>.449</td><td>.468</td><td>.472</td></tr>
  <tr><td>Ghanaian Dialect</td><td>.529</td><td>.590</td><td>.576</td><td>.597</td><td>.428</td><td>.454</td><td>.468</td><td>.472</td></tr>
  <tr><td>Southeast England Dialect</td><td>.526</td><td>.585</td><td>.577</td><td>.595</td><td>.433</td><td>.451</td><td>.468</td><td>.472</td></tr>
  <tr><td>Bahamian Dialect</td><td>.530</td><td>.578</td><td>.576</td><td>.596</td><td>.420</td><td>.450</td><td>.467</td><td>.472</td></tr>
  <tr><td>Colloquial Singapore Dialect</td><td>.526</td><td>.573</td><td>.578</td><td>.599</td><td>.421</td><td>.454</td><td>.467</td><td>.472</td></tr>
  <tr><td>Falkland Islands Dialect</td><td>.529</td><td>.585</td><td>.578</td><td>.592</td><td>.419</td><td>.454</td><td>.467</td><td>.472</td></tr>
  <tr><td>Southeast American Enclave Dialect</td><td>.532</td><td>.588</td><td>.576</td><td>.587</td><td>.435</td><td>.455</td><td>.467</td><td>.471</td></tr>
  <tr><td>Welsh Dialect</td><td>.529</td><td>.592</td><td>.577</td><td>.592</td><td>.433</td><td>.447</td><td>.465</td><td>.469</td></tr>
  <tr><td>Australian Dialect</td><td>.531</td><td>.582</td><td>.574</td><td>.602</td><td>.435</td><td>.449</td><td>.465</td><td>.469</td></tr>
  <tr><td>White Zimbabwean Dialect</td><td>.528</td><td>.590</td><td>.574</td><td>.597</td><td>.425</td><td>.451</td><td>.464</td><td>.469</td></tr>
  <tr><td>Ozark Dialect</td><td>.530</td><td>.589</td><td>.578</td><td>.597</td><td>.423</td><td>.451</td><td>.464</td><td>.469</td></tr>
  <tr><td>Channel Islands Dialect</td><td>.530</td><td>.584</td><td>.579</td><td>.589</td><td>.425</td><td>.450</td><td>.463</td><td>.468</td></tr>
  <tr><td>Chicano Dialect</td><td>.530</td><td>.604</td><td>.582</td><td>.611</td><td>.419</td><td>.445</td><td>.463</td><td>.468</td></tr>
  <tr><td>Cape Flats Dialect</td><td>.528</td><td>.581</td><td>.577</td><td>.590</td><td>.421</td><td>.447</td><td>.463</td><td>.468</td></tr>
  <tr><td>Colloquial American Dialect</td><td>.528</td><td>.577</td><td>.578</td><td>.600</td><td>.421</td><td>.447</td><td>.463</td><td>.468</td></tr>
  <tr><td>Kenyan Dialect</td><td>.525</td><td>.593</td><td>.582</td><td>.592</td><td>.415</td><td>.449</td><td>.462</td><td>.467</td></tr>
  <tr><td>White South African Dialect</td><td>.529</td><td>.588</td><td>.577</td><td>.604</td><td>.430</td><td>.444</td><td>.462</td><td>.467</td></tr>
  <tr><td>Ugandan Dialect</td><td>.532</td><td>.601</td><td>.580</td><td>.590</td><td>.421</td><td>.444</td><td>.462</td><td>.467</td></tr>
  <tr><td>Southwest England Dialect</td><td>.527</td><td>.576</td><td>.581</td><td>.594</td><td>.415</td><td>.445</td><td>.462</td><td>.467</td></tr>
  <tr><td>Appalachian Dialect</td><td>.527</td><td>.589</td><td>.575</td><td>.595</td><td>.416</td><td>.449</td><td>.461</td><td>.466</td></tr>
  <tr><td>Tristan Dialect</td><td>.526</td><td>.584</td><td>.575</td><td>.592</td><td>.429</td><td>.443</td><td>.460</td><td>.465</td></tr>
  <tr><td>Indian Dialect</td><td>.531</td><td>.585</td><td>.577</td><td>.600</td><td>.414</td><td>.443</td><td>.459</td><td>.465</td></tr>
  <tr><td>Cameroon Dialect</td><td>.527</td><td>.590</td><td>.580</td><td>.585</td><td>.420</td><td>.440</td><td>.458</td><td>.463</td></tr>
  <tr><td>Hong Kong Dialect</td><td>.528</td><td>.594</td><td>.577</td><td>.601</td><td>.410</td><td>.439</td><td>.458</td><td>.463</td></tr>
  <tr><td>Indian South African Dialect</td><td>.527</td><td>.590</td><td>.577</td><td>.596</td><td>.415</td><td>.444</td><td>.457</td><td>.463</td></tr>
  <tr><td>Rural African American Vernacular</td><td>.527</td><td>.588</td><td>.573</td><td>.595</td><td>.424</td><td>.444</td><td>.454</td><td>.460</td></tr>
  <tr><td>Maltese Dialect</td><td>.529</td><td>.592</td><td>.576</td><td>.597</td><td>.408</td><td>.441</td><td>.454</td><td>.460</td></tr>
</table>
Table 10: MultiAgent Framework Results for Llama 3.1 on PrivacyQA and PolicyQA

<table>
  <tr>
    <th rowspan="2">Dialect</th>
    <th colspan="2">PrivacyQA Zero-shot</th>
    <th colspan="2">PrivacyQA Few-shot</th>
    <th colspan="2">PolicyQA Zero-shot</th>
    <th colspan="2">PolicyQA Few-shot</th>
  </tr>
  <tr>
    <th>Initial</th>
    <th>Final</th>
    <th>Initial</th>
    <th>Final</th>
    <th>Initial</th>
    <th>Final</th>
    <th>Initial</th>
    <th>Final</th>
  </tr>
  <tr><td>Standard American Dialect</td><td>.514</td><td>.549</td><td>.424</td><td>.555</td><td>.310</td><td>.381</td><td>.379</td><td>.400</td></tr>
  <tr><td>St Helena Dialect</td><td>.493</td><td>.514</td><td>.488</td><td>.536</td><td>.241</td><td>.368</td><td>.335</td><td>.392</td></tr>
  <tr><td>Kenyan Dialect</td><td>.506</td><td>.559</td><td>.502</td><td>.543</td><td>.264</td><td>.355</td><td>.352</td><td>.361</td></tr>
  <tr><td>Scottish Dialect</td><td>.508</td><td>.535</td><td>.510</td><td>.545</td><td>.260</td><td>.360</td><td>.350</td><td>.385</td></tr>
  <tr><td>Ozark Dialect</td><td>.498</td><td>.519</td><td>.505</td><td>.552</td><td>.268</td><td>.382</td><td>.357</td><td>.372</td></tr>
  <tr><td>New Zealand Dialect</td><td>.493</td><td>.512</td><td>.480</td><td>.503</td><td>.228</td><td>.352</td><td>.317</td><td>.384</td></tr>
  <tr><td>Ugandan Dialect</td><td>.502</td><td>.533</td><td>.507</td><td>.549</td><td>.242</td><td>.351</td><td>.332</td><td>.374</td></tr>
  <tr><td>Early African American Vernacular</td><td>.505</td><td>.540</td><td>.510</td><td>.523</td><td>.257</td><td>.374</td><td>.344</td><td>.387</td></tr>
  <tr><td>Indian South African Dialect</td><td>.495</td><td>.546</td><td>.501</td><td>.519</td><td>.231</td><td>.372</td><td>.326</td><td>.374</td></tr>
  <tr><td>Falkland Islands Dialect</td><td>.495</td><td>.511</td><td>.496</td><td>.524</td><td>.246</td><td>.374</td><td>.342</td><td>.387</td></tr>
  <tr><td>Colloquial Singapore Dialect</td><td>.514</td><td>.527</td><td>.501</td><td>.513</td><td>.251</td><td>.366</td><td>.344</td><td>.377</td></tr>
  <tr><td>Welsh Dialect</td><td>.497</td><td>.523</td><td>.491</td><td>.522</td><td>.290</td><td>.372</td><td>.371</td><td>.394</td></tr>
  <tr><td>Indian Dialect</td><td>.496</td><td>.536</td><td>.492</td><td>.509</td><td>.210</td><td>.377</td><td>.310</td><td>.397</td></tr>
  <tr><td>Malaysian Dialect</td><td>.506</td><td>.529</td><td>.497</td><td>.532</td><td>.244</td><td>.386</td><td>.345</td><td>.364</td></tr>
  <tr><td>Irish Dialect</td><td>.497</td><td>.521</td><td>.494</td><td>.507</td><td>.248</td><td>.376</td><td>.346</td><td>.377</td></tr>
  <tr><td>White Zimbabwean Dialect</td><td>.513</td><td>.537</td><td>.501</td><td>.536</td><td>.237</td><td>.358</td><td>.328</td><td>.390</td></tr>
  <tr><td>African American Vernacular</td><td>.488</td><td>.527</td><td>.502</td><td>.525</td><td>.242</td><td>.374</td><td>.338</td><td>.385</td></tr>
  <tr><td>Tristan Dialect</td><td>.510</td><td>.521</td><td>.511</td><td>.534</td><td>.208</td><td>.369</td><td>.314</td><td>.382</td></tr>
  <tr><td>Jamaican Dialect</td><td>.492</td><td>.520</td><td>.476</td><td>.523</td><td>.240</td><td>.368</td><td>.324</td><td>.391</td></tr>
  <tr><td>Newfoundland Dialect</td><td>.512</td><td>.545</td><td>.521</td><td>.539</td><td>.260</td><td>.355</td><td>.352</td><td>.389</td></tr>
  <tr><td>White South African Dialect</td><td>.532</td><td>.539</td><td>.518</td><td>.522</td><td>.285</td><td>.380</td><td>.358</td><td>.379</td></tr>
  <tr><td>Appalachian Dialect</td><td>.501</td><td>.532</td><td>.497</td><td>.529</td><td>.246</td><td>.360</td><td>.330</td><td>.386</td></tr>
  <tr><td>Ghanaian Dialect</td><td>.517</td><td>.549</td><td>.512</td><td>.542</td><td>.239</td><td>.364</td><td>.319</td><td>.381</td></tr>
  <tr><td>Australian Vernacular</td><td>.501</td><td>.528</td><td>.497</td><td>.524</td><td>.289</td><td>.372</td><td>.366</td><td>.388</td></tr>
  <tr><td>Channel Islands Dialect</td><td>.529</td><td>.550</td><td>.527</td><td>.548</td><td>.263</td><td>.369</td><td>.342</td><td>.369</td></tr>
  <tr><td>Hong Kong Dialect</td><td>.507</td><td>.525</td><td>.485</td><td>.520</td><td>.222</td><td>.364</td><td>.311</td><td>.396</td></tr>
  <tr><td>Black South African Dialect</td><td>.507</td><td>.530</td><td>.483</td><td>.516</td><td>.245</td><td>.374</td><td>.345</td><td>.366</td></tr>
  <tr><td>Maltese Dialect</td><td>.534</td><td>.564</td><td>.499</td><td>.525</td><td>.231</td><td>.381</td><td>.329</td><td>.374</td></tr>
  <tr><td>Rural African American Vernacular</td><td>.489</td><td>.523</td><td>.496</td><td>.538</td><td>.207</td><td>.374</td><td>.309</td><td>.380</td></tr>
  <tr><td>Southeast England Dialect</td><td>.530</td><td>.548</td><td>.514</td><td>.536</td><td>.257</td><td>.370</td><td>.341</td><td>.374</td></tr>
  <tr><td>Pakistani Dialect</td><td>.522</td><td>.560</td><td>.514</td><td>.536</td><td>.240</td><td>.351</td><td>.334</td><td>.387</td></tr>
  <tr><td>Fiji Accent</td><td>.502</td><td>.530</td><td>.494</td><td>.534</td><td>.270</td><td>.373</td><td>.348</td><td>.372</td></tr>
  <tr><td>Southeast American Enclave Dialect</td><td>.497</td><td>.520</td><td>.500</td><td>.527</td><td>.250</td><td>.357</td><td>.337</td><td>.377</td></tr>
  <tr><td>East Anglican Dialect</td><td>.487</td><td>.502</td><td>.480</td><td>.510</td><td>.260</td><td>.356</td><td>.340</td><td>.388</td></tr>
  <tr><td>Orkney Shetland Dialect</td><td>.513</td><td>.540</td><td>.515</td><td>.520</td><td>.265</td><td>.351</td><td>.350</td><td>.370</td></tr>
  <tr><td>Bahamian Dialect</td><td>.503</td><td>.521</td><td>.488</td><td>.510</td><td>.234</td><td>.368</td><td>.329</td><td>.368</td></tr>
  <tr><td>Manx Dialect</td><td>.486</td><td>.506</td><td>.489</td><td>.532</td><td>.287</td><td>.383</td><td>.355</td><td>.377</td></tr>
  <tr><td>Cameroon Dialect</td><td>.521</td><td>.545</td><td>.481</td><td>.511</td><td>.228</td><td>.374</td><td>.324</td><td>.388</td></tr>
  <tr><td>North England Dialect</td><td>.518</td><td>.539</td><td>.495</td><td>.525</td><td>.249</td><td>.369</td><td>.337</td><td>.377</td></tr>
  <tr><td>Colloquial American Dialect</td><td>.496</td><td>.520</td><td>.506</td><td>.530</td><td>.241</td><td>.384</td><td>.329</td><td>.394</td></tr>
  <tr><td>Australian Dialect</td><td>.506</td><td>.537</td><td>.488</td><td>.519</td><td>.237</td><td>.359</td><td>.323</td><td>.389</td></tr>
  <tr><td>Fiji Basilet</td><td>.491</td><td>.536</td><td>.468</td><td>.505</td><td>.269</td><td>.381</td><td>.344</td><td>.374</td></tr>
  <tr><td>Nigerian Dialect</td><td>.498</td><td>.547</td><td>.495</td><td>.524</td><td>.262</td><td>.377</td><td>.345</td><td>.368</td></tr>
  <tr><td>Philippine Dialect</td><td>.505</td><td>.537</td><td>.498</td><td>.515</td><td>.247</td><td>.361</td><td>.341</td><td>.387</td></tr>
  <tr><td>Sri Lankan Dialect</td><td>.530</td><td>.556</td><td>.512</td><td>.544</td><td>.256</td><td>.373</td><td>.348</td><td>.373</td></tr>
  <tr><td>Liberian Settler Dialect</td><td>.507</td><td>.531</td><td>.492</td><td>.509</td><td>.264</td><td>.381</td><td>.356</td><td>.381</td></tr>
  <tr><td>Tanzanian Dialect</td><td>.517</td><td>.542</td><td>.498</td><td>.533</td><td>.276</td><td>.377</td><td>.346</td><td>.371</td></tr>
  <tr><td>Cape Flats Dialect</td><td>.511</td><td>.521</td><td>.514</td><td>.544</td><td>.268</td><td>.378</td><td>.358</td><td>.374</td></tr>
</table>
Table 11: MultiAgent Framework Results for DeepSeek-R1 on PrivacyQA and PolicyQA

<table>
  <tr>
    <th>Dialect</th>
    <th colspan="2">PrivacyQA Zero-shot</th>
    <th colspan="2">PrivacyQA Few-shot</th>
    <th colspan="2">PolicyQA Zero-shot</th>
    <th colspan="2">PolicyQA Few-shot</th>
  </tr>
  <tr>
    <th></th>
    <th>Initial</th>
    <th>Final</th>
    <th>Initial</th>
    <th>Final</th>
    <th>Initial</th>
    <th>Final</th>
    <th>Initial</th>
    <th>Final</th>
  </tr>
  <tr><td>Kenyan Dialect</td><td>.517</td><td>.569</td><td>.446</td><td>.585</td><td>.446</td><td>.488</td><td>.428</td><td>.498</td></tr>
  <tr><td>St Helena Dialect</td><td>.543</td><td>.587</td><td>.456</td><td>.569</td><td>.416</td><td>.468</td><td>.460</td><td>.491</td></tr>
  <tr><td>Scottish Dialect</td><td>.529</td><td>.580</td><td>.440</td><td>.535</td><td>.419</td><td>.481</td><td>.464</td><td>.485</td></tr>
  <tr><td>Ozark Dialect</td><td>.515</td><td>.575</td><td>.478</td><td>.581</td><td>.404</td><td>.470</td><td>.421</td><td>.498</td></tr>
  <tr><td>New Zealand Dialect</td><td>.530</td><td>.583</td><td>.439</td><td>.569</td><td>.406</td><td>.465</td><td>.422</td><td>.494</td></tr>
  <tr><td>Ugandan Dialect</td><td>.535</td><td>.578</td><td>.437</td><td>.563</td><td>.401</td><td>.475</td><td>.430</td><td>.477</td></tr>
  <tr><td>Early African American Vernacular</td><td>.526</td><td>.584</td><td>.481</td><td>.580</td><td>.401</td><td>.475</td><td>.460</td><td>.491</td></tr>
  <tr><td>Indian South African Dialect</td><td>.523</td><td>.578</td><td>.436</td><td>.573</td><td>.411</td><td>.488</td><td>.451</td><td>.473</td></tr>
  <tr><td>Falkland Islands Dialect</td><td>.532</td><td>.569</td><td>.440</td><td>.535</td><td>.437</td><td>.480</td><td>.443</td><td>.483</td></tr>
  <tr><td>Colloquial Singapore Dialect</td><td>.498</td><td>.570</td><td>.436</td><td>.572</td><td>.417</td><td>.488</td><td>.433</td><td>.495</td></tr>
  <tr><td>Indian Dialect</td><td>.532</td><td>.583</td><td>.460</td><td>.584</td><td>.416</td><td>.459</td><td>.455</td><td>.479</td></tr>
  <tr><td>Malaysian Dialect</td><td>.501</td><td>.569</td><td>.439</td><td>.552</td><td>.434</td><td>.488</td><td>.461</td><td>.506</td></tr>
  <tr><td>Irish Dialect</td><td>.504</td><td>.560</td><td>.445</td><td>.578</td><td>.431</td><td>.468</td><td>.449</td><td>.500</td></tr>
  <tr><td>African American Vernacular</td><td>.512</td><td>.551</td><td>.462</td><td>.578</td><td>.436</td><td>.485</td><td>.464</td><td>.490</td></tr>
  <tr><td>Jamaican Dialect</td><td>.517</td><td>.583</td><td>.447</td><td>.585</td><td>.437</td><td>.474</td><td>.455</td><td>.494</td></tr>
  <tr><td>Standard American Dialect</td><td>.501</td><td>.562</td><td>.460</td><td>.533</td><td>.422</td><td>.451</td><td>.456</td><td>.474</td></tr>
  <tr><td>Newfoundland Dialect</td><td>.531</td><td>.575</td><td>.448</td><td>.557</td><td>.437</td><td>.468</td><td>.433</td><td>.475</td></tr>
  <tr><td>Appalachian Dialect</td><td>.519</td><td>.560</td><td>.470</td><td>.567</td><td>.414</td><td>.453</td><td>.451</td><td>.488</td></tr>
  <tr><td>Ghanaian Dialect</td><td>.514</td><td>.561</td><td>.468</td><td>.564</td><td>.448</td><td>.482</td><td>.424</td><td>.486</td></tr>
  <tr><td>Australian Vernacular</td><td>.550</td><td>.602</td><td>.434</td><td>.561</td><td>.434</td><td>.467</td><td>.413</td><td>.481</td></tr>
  <tr><td>Channel Islands Dialect</td><td>.507</td><td>.574</td><td>.466</td><td>.554</td><td>.429</td><td>.458</td><td>.428</td><td>.475</td></tr>
  <tr><td>Hong Kong Dialect</td><td>.507</td><td>.579</td><td>.448</td><td>.557</td><td>.434</td><td>.485</td><td>.419</td><td>.502</td></tr>
  <tr><td>Black South African Dialect</td><td>.515</td><td>.571</td><td>.445</td><td>.590</td><td>.440</td><td>.474</td><td>.420</td><td>.471</td></tr>
  <tr><td>Maltese Dialect</td><td>.512</td><td>.576</td><td>.451</td><td>.565</td><td>.440</td><td>.469</td><td>.436</td><td>.504</td></tr>
  <tr><td>Rural African American Vernacular</td><td>.534</td><td>.579</td><td>.476</td><td>.606</td><td>.420</td><td>.480</td><td>.467</td><td>.476</td></tr>
  <tr><td>Pakistani Dialect</td><td>.507</td><td>.568</td><td>.452</td><td>.579</td><td>.411</td><td>.469</td><td>.422</td><td>.493</td></tr>
  <tr><td>Fiji Acrolect</td><td>.551</td><td>.579</td><td>.486</td><td>.573</td><td>.403</td><td>.472</td><td>.451</td><td>.485</td></tr>
  <tr><td>Southeast American Enclave Dialect</td><td>.510</td><td>.582</td><td>.463</td><td>.593</td><td>.424</td><td>.452</td><td>.422</td><td>.505</td></tr>
  <tr><td>East Anglican Dialect</td><td>.523</td><td>.593</td><td>.459</td><td>.569</td><td>.444</td><td>.467</td><td>.452</td><td>.498</td></tr>
  <tr><td>Orkney Shetland Dialect</td><td>.511</td><td>.563</td><td>.456</td><td>.572</td><td>.447</td><td>.456</td><td>.417</td><td>.487</td></tr>
  <tr><td>Bahamian Dialect</td><td>.517</td><td>.574</td><td>.449</td><td>.572</td><td>.410</td><td>.470</td><td>.425</td><td>.506</td></tr>
  <tr><td>Manx Dialect</td><td>.551</td><td>.580</td><td>.450</td><td>.580</td><td>.445</td><td>.482</td><td>.416</td><td>.471</td></tr>
  <tr><td>Cameroon Dialect</td><td>.517</td><td>.575</td><td>.470</td><td>.573</td><td>.432</td><td>.472</td><td>.462</td><td>.484</td></tr>
  <tr><td>North England Dialect</td><td>.522</td><td>.577</td><td>.464</td><td>.587</td><td>.448</td><td>.471</td><td>.430</td><td>.504</td></tr>
  <tr><td>Colloquial American Dialect</td><td>.530</td><td>.586</td><td>.465</td><td>.577</td><td>.444</td><td>.487</td><td>.454</td><td>.494</td></tr>
  <tr><td>Australian Dialect</td><td>.525</td><td>.580</td><td>.465</td><td>.577</td><td>.400</td><td>.467</td><td>.445</td><td>.471</td></tr>
  <tr><td>Fiji Basilet</td><td>.532</td><td>.571</td><td>.476</td><td>.605</td><td>.449</td><td>.482</td><td>.414</td><td>.501</td></tr>
  <tr><td>Nigerian Dialect</td><td>.522</td><td>.571</td><td>.467</td><td>.551</td><td>.411</td><td>.479</td><td>.468</td><td>.484</td></tr>
  <tr><td>Philippine Dialect</td><td>.522</td><td>.580</td><td>.464</td><td>.566</td><td>.422</td><td>.472</td><td>.428</td><td>.477</td></tr>
  <tr><td>Sri Lankan Dialect</td><td>.537</td><td>.555</td><td>.468</td><td>.563</td><td>.428</td><td>.459</td><td>.469</td><td>.505</td></tr>
  <tr><td>Liberian Settler Dialect</td><td>.547</td><td>.583</td><td>.455</td><td>.572</td><td>.433</td><td>.478</td><td>.469</td><td>.502</td></tr>
  <tr><td>Tanzanian Dialect</td><td>.534</td><td>.584</td><td>.456</td><td>.574</td><td>.400</td><td>.486</td><td>.413</td><td>.503</td></tr>
  <tr><td>Cape Flats Dialect</td><td>.537</td><td>.596</td><td>.443</td><td>.548</td><td>.434</td><td>.451</td><td>.424</td><td>.478</td></tr>
</table>
Table 12: Few-shot MultiAgent Framework Results for GPT-4o-mini on PrivacyQA(No Dialect Info)

<table>
  <tr>
    <th>Dialect</th>
    <th>Initial F1</th>
    <th>Final F1</th>
  </tr>
  <tr><td>StHelenaDialect</td><td>0.529</td><td>0.555</td></tr>
  <tr><td>KenyanDialect</td><td>0.533</td><td>0.600</td></tr>
  <tr><td>ScottishDialect</td><td>0.521</td><td>0.603</td></tr>
  <tr><td>OzarkDialect</td><td>0.518</td><td>0.583</td></tr>
  <tr><td>NewZealandDialect</td><td>0.516</td><td>0.600</td></tr>
  <tr><td>UgandanDialect</td><td>0.535</td><td>0.597</td></tr>
  <tr><td>EarlyAfricanAmericanVernacular</td><td>0.532</td><td>0.558</td></tr>
  <tr><td>IndianSouthAfricanDialect</td><td>0.516</td><td>0.590</td></tr>
  <tr><td>FalklandIslandsDialect</td><td>0.527</td><td>0.564</td></tr>
  <tr><td>ColloquialSingaporeDialect</td><td>0.508</td><td>0.600</td></tr>
  <tr><td>WelshDialect</td><td>0.524</td><td>0.610</td></tr>
  <tr><td>IndianDialect</td><td>0.518</td><td>0.578</td></tr>
  <tr><td>MalaysianDialect</td><td>0.510</td><td>0.565</td></tr>
  <tr><td>IrishDialect</td><td>0.501</td><td>0.556</td></tr>
  <tr><td>WhiteZimbabweanDialect</td><td>0.527</td><td>0.574</td></tr>
  <tr><td>AfricanAmericanVernacular</td><td>0.534</td><td>0.576</td></tr>
  <tr><td>TristanDialect</td><td>0.534</td><td>0.553</td></tr>
  <tr><td>JamaicanDialect</td><td>0.513</td><td>0.600</td></tr>
  <tr><td>StandardAmericanDialect</td><td>0.518</td><td>0.614</td></tr>
  <tr><td>NewfoundlandDialect</td><td>0.512</td><td>0.604</td></tr>
  <tr><td>WhiteSouthAfricanDialect</td><td>0.516</td><td>0.567</td></tr>
  <tr><td>AppalachianDialect</td><td>0.530</td><td>0.605</td></tr>
  <tr><td>GhanaianDialect</td><td>0.520</td><td>0.603</td></tr>
  <tr><td>AustralianVernacular</td><td>0.534</td><td>0.595</td></tr>
  <tr><td>ChannelIslandsDialect</td><td>0.508</td><td>0.596</td></tr>
  <tr><td>HongKongDialect</td><td>0.522</td><td>0.605</td></tr>
  <tr><td>BlackSouthAfricanDialect</td><td>0.512</td><td>0.564</td></tr>
  <tr><td>MalteseDialect</td><td>0.496</td><td>0.606</td></tr>
  <tr><td>RuralAfricanAmericanVernacular</td><td>0.501</td><td>0.604</td></tr>
  <tr><td>SoutheastEnglandDialect</td><td>0.518</td><td>0.565</td></tr>
  <tr><td>PakistaniDialect</td><td>0.523</td><td>0.599</td></tr>
  <tr><td>FijiAcrolect</td><td>0.526</td><td>0.582</td></tr>
  <tr><td>SoutheastAmericanEnclaveDialect</td><td>0.539</td><td>0.612</td></tr>
  <tr><td>EastAnglicanDialect</td><td>0.514</td><td>0.591</td></tr>
  <tr><td>OrkneyShetlandDialect</td><td>0.521</td><td>0.622</td></tr>
  <tr><td>BahamianDialect</td><td>0.508</td><td>0.592</td></tr>
  <tr><td>ManxDialect</td><td>0.514</td><td>0.575</td></tr>
  <tr><td>CameroonDialect</td><td>0.526</td><td>0.566</td></tr>
  <tr><td>NorthEnglandDialect</td><td>0.531</td><td>0.565</td></tr>
  <tr><td>ColloquialAmericanDialect</td><td>0.513</td><td>0.573</td></tr>
  <tr><td>AustralianDialect</td><td>0.526</td><td>0.587</td></tr>
  <tr><td>FijiBasillect</td><td>0.536</td><td>0.619</td></tr>
  <tr><td>NigerianDialect</td><td>0.531</td><td>0.603</td></tr>
  <tr><td>PhilippineDialect</td><td>0.522</td><td>0.595</td></tr>
  <tr><td>SriLankanDialect</td><td>0.525</td><td>0.622</td></tr>
  <tr><td>LiberianSettlerDialect</td><td>0.518</td><td>0.584</td></tr>
  <tr><td>TanzanianDialect</td><td>0.521</td><td>0.615</td></tr>
  <tr><td>CapeFlatsDialect</td><td>0.530</td><td>0.594</td></tr>
</table>

Reviewer paper summaries: This paper proposes a multi-agent collaborative framework to tackle the issue of uneven performance across multiple English dialects under a privacy policy question-answering setting. The framework consists of a Dialect Agent and a Privacy Agent. The Dialect Agent would first translate the dialect back into Standard American English, then the translated query is passed into the Privacy Agent to obtain a response. The Dialect Agent would then determine whether the Privacy Agent response is satisfactory; if it is not satisfactory, the revision process is then triggered. The authors utilize the PrivacyQA and PolicyQA datasets. Each query is first translated into multiple dialects using the MultiVALUE framework. The proposed framework improves performance across all dialects and reduces the disparity between different dialects.

This paper proposes a LLM-based multi-agent setup, consisting of two agents (Dialect Agent, Privacy QA Agent) which cooperate to translate and answer questions about privacy policies where the questions might be given in non-standard American English dialects. 
The proposed system outperforms its single-model counterparts and reduces performance disparities between different English dialects, thereby enabling more fair and equitable QA system for speakers of such dialects.

The paper is clearly written, easy to follow and the empirical results support the main claims of the work. Some more information about the performance of the Dialect Agent would be helpful to comprehend the overall performance of the system and where it might go wrong (adding additional agents always comes at the risk of introducing a new source of error). Further, some relevant information about experimental settings and task description should be added to improve readability and reproducibility.
Summary of strengths: * The proposed framework is effective in improving model performance across different dialects and in reducing disparity in performance
* The authors examined a good variety of models with different sizes
* The framework can improve performance significantly even in a zero-shot setting
* The work in improving fairness among different dialect groups is important and socially meaningful

- strong motivation and a clear, easy-to-follow writing style
- presents an extensive set of experiments, which supports the empirical validity of the work
Summary of weaknesses: * The dialect versions of questions are created through synthetic approaches. It would be better to collect organic dialect questions by asking native speakers of these dialects to translate the original SAE utterances.
* The validity of the Dialect Agent itself is not thoroughly validated in the paper; is the Dialect Agent capable of translating dialectal utterances faithfully?

- real-world applicability of the approach is somewhat limited due to its dependence on dataset-specific metadata
- there are instances of missing information and minor formatting issues (see comments)
Comments, suggestions and typos: ## Presentation Suggestions
1. Please move Figure 3 to Page 8 for easier referencing.
2. Missing decimal point before 533 for DeepSeek R1 results in Table 2.

## Comments
1. In Table 2, your GPT-4o-mini-Few outperforms your GPT-4o-mini Multi-Agent Zero-shot, so the result for GPT-4o-mini-Few should be bolded.

## Questions
1. When you were translating dialect queries back to SAE, did the final result match the initial query? If not, are there patterns in the mismatches?
2. Have you done an experiment where you ablate different components of the dialect info you provide to the Dialect Agent? I would imagine information about phonetics may not be strictly relevant in a mostly textual setting.

- Two evaluations appear to be missing from the current work. First, to better assess the real-world applicability of the system, it would be valuable to evaluate how accurately the system identifies the dialect from the user input. Second, it remains unclear in how many cases the Dialect Agent incorrectly overrides or corrects the initial responses generated by the Privacy Policy Agent.
- In the related work section, it would be helpful to clarify that the discussion focuses specifically on LLM-based multi-agent systems. The term “multi-agent” has a long-standing and rich history in the broader AI literature, independent of large language models. Making this distinction more explicit would help position the contribution more clearly within the relevant body of work. 
- In Figure 2, it would be helpful to clarify whether the dialect information—according to the documentation—is explicitly provided as part of the agent’s input. Since this metadata is essential for selecting the appropriate system prompt, it would be beneficial to include it in the listed input variables for completeness and clarity.
- Including a concrete example from the dataset would help clarify the structure of the input and expected output. This would enhance the reader’s understanding of how the method operates in practice and how the data is represented.
- l. 445: How many few-shot examples are used?
- l 416: "main paper" is misleading; reference the table directly instead
- I recommend considering combining tables 2 and 3 to save space of duplicate figure descriptions
- l 485-492: I recommend making it more explicit in how many comparisons the zero-shot setting actually outperforms the few-shot setting. At first glance, this appears to be the case in only a minority of direct comparisons (approximately 20%), without a consistent pattern across models or dialects. While the conclusion that the approach is “highly effective in settings where labeled data is limited” is promising, it may currently appear somewhat optimistic based on the presented results.
Overall assessment scores: 3.5

4