Investigating Language Preference of Multilingual RAG Systems
Authors: Jeonghyun Park, Hwanhee Lee
Accepted at: acl25

Investigating Language Preference of Multilingual RAG Systems

Jeonghyun Park and Hwanhee Lee†
Department of Artificial Intelligence, Chung-Ang University, Seoul, Korea
{tom0365, hwanheelee}@cau.ac.kr

Abstract
Multilingual Retrieval-Augmented Generation (mRAG) systems enhance language models by integrating external multilingual information to produce context-aware responses. However, mRAG systems struggle with retrieving relevant information due to linguistic variations between queries and documents, generating inconsistent responses when multilingual sources conflict. In this work, we systematically investigate language preferences in both retrieval and generation of mRAG through a series of experiments. Our analysis indicates that retrievers tend to prefer high-resource and query languages, yet this preference does not consistently improve generation performance. Moreover, we observe that generators prefer the query language or Latin scripts, leading to inconsistent outputs. To overcome these issues, we propose Dual Knowledge Multilingual RAG (DKM-RAG), a simple yet effective framework that fuses translated multilingual passages with complementary model knowledge. Empirical results demonstrate that DKM-RAG mitigates language preference in generation and enhances performance across diverse linguistic settings. Code is available at https://github.com/jeonghyunpark2002/LanguagePreference.git

1 Introduction
Multilingual Retrieval-Augmented Generation (mRAG) extends traditional Retrieval-Augmented Generation (RAG) (Lewis et al., 2020) by leveraging multilingual external sources to generate accurate, contextually and linguistically aware responses. However, mRAG systems face challenges in retrieving relevant information due to linguistic discrepancies between queries and documents (Wu et al., 2024a). Moreover, conflicts among multilingual sources can lead to inconsistencies in the generated responses (Chataigner et al., 2024).

Figure 1: Failure cases of multilingual RAG system showing degraded generation ability because of language preference of retriever and generator in mRAG pipeline. docrel in the Korean (KO) document represents the relevant document to the given query that can be utilized to generate a final answer.

Beyond retrieval challenges and source conflicts, language preference is another critical issue in mRAG systems, often leading to inaccurate outputs. As illustrated in Case 1 of Figure 1, the retriever may prioritize particular languages—especially high-resource or query-language documents—at the expense of truly relevant information in low-resource languages. Consequently, the Large Language Model (LLM) either produces an incorrect answer or deems the query unanswerable due to irrelevant content in the documents. Likewise, in Case 2, even if relevant documents are retrieved, the generator might favor passages in the query language or Latin scripts, ignoring essential evidence
in lower-resource languages and resulting in inaccurate outputs. These preferences ultimately limit the effectiveness of mRAG, yielding biased rankings, reduced answer quality, and inconsistencies across languages (Sharma et al., 2024).

Prior studies (Yang et al., 2024a; Telemala and Suleman, 2022; Sharma et al., 2024) have investigated this issue by introducing language fairness metrics to assess whether documents from different languages are ranked equitably via statistical equivalence testing, by proposing Language-Preference-Based Re-ranking for multilingual information retrieval, and investigating LLM’s linguistic preference in across-language RAG-based information search setting. However, these approaches primarily focus on a limited set of languages and fail to reflect the true ranking dynamics of documents across languages.

In this work, we aim to understand language preference phenomena in mRAG systems comprehensively. We focus on the following three key research questions:

• **RQ1 (§4): Which languages does the retriever prefer?**

• **RQ2 (§5): Which languages does the generator prefer, and how do these preferences correlate with mRAG performance?**

• **RQ3 (§6): How can we mitigate language preference in mRAG?**

To address these questions, we present a comprehensive evaluation of language preferences throughout the entire mRAG pipeline across multiple languages. To systematically investigate the language preference problem of multilingual retrievers, we propose MultiLingualRankShift (MLRS), a novel metric that quantifies language preference at the retriever level by measuring the shift in document rankings when non-query-language passages are translated into the query language. Our extensive experiments with diverse language combinations demonstrate that the retriever strongly prefers documents that are in high-resource languages and also share the same language as the query, confirming the presence of significant preference (§4).

At the generator level, we evaluate language preference by generating responses in multiple languages for the same query and the same retrieved document set, measuring their semantic similarity. Our results show that the generator favors both query languages and Latin script languages, with a relatively modest preference for query languages. This ultimately results in a decline in answer quality. Moreover, we uncover a nuanced relationship between language preference and overall mRAG performance. We observe that a strong preference for high-resource languages does not always lead to improved mRAG performance (§5). This occurs because the retriever may retrieve high-resource but irrelevant documents so that the generator cannot generate accurate answers from them. Therefore, language preference can degrade performance by overlooking lower-resource but relevant documents, thereby causing inconsistencies.

Finally, we propose Dual Knowledge Multilingual RAG (DKM-RAG), a simple yet effective framework that mitigates the language preference of mRAG. DKM-RAG enhances mRAG by combining externally retrieved, translated passages with internally rewritten passages enriched by the model’s knowledge. Empirical results demonstrate that DKM-RAG significantly reduces language preference issues in the generation process, leading to improved performance across a range of linguistic settings (§6).

2 **MultiLingualRankShift**

To evaluate the language preference of a multilingual retriever in the mRAG system, we introduce *MultiLingualRankShift* (MLRS), a novel metric that quantifies how much the ranking of retrieved documents improves when non-query language documents are translated into the query language. As shown in Figure 2, MLRS is computed in three stages: (i) retrieving documents across multiple languages, (ii) translating documents that are not in the query language into the query language, and (iii) re-ranking the translated documents to measure rank improvements.

**2.1 Stage 1: Initial Document Retrieval**

For each query \( q \in Q \) (where \( Q \) is the set of all queries), we retrieve a ranked list of documents \( D_q \) from multilingual datastores. Each document \( d \in D_q \) is assigned an initial rank \( r_d^{\text{init}} \) (with 1 being the highest rank). Let \( L_q \) denote the language of the query and \( L_d \) the language of document \( d \).

**2.2 Stage 2: Translation of Non-Query Language Documents**

To ensure language consistency when assessing ranking improvements, we extract documents
Figure 2: Overall framework of calculating MLRS. For simplicity, we only consider three documents to calculate the MLRS score.

whose language differs from that of the query. Formally, we define:

\[
D_q^{\text{diff}} = \{ (d, r_d^{\text{init}}) \mid d \in D_q, L_d \neq L_q \}.
\]

Each document in \( D_q^{\text{diff}} \) is then translated into the query language \( L_q \), resulting in the set:

\[
D_q^{\text{Translated}} = \{ d \mid d \text{ has been translated into } L_q \}.
\]

2.3 Stage 3: Re-Ranking and MLRS Score Computation

The translated documents in \( D_q^{\text{Translated}} \) are re-ranked using retrievers in conjunction with the original query. Let \( r_d^{\text{re-rank}} \) denote the new rank of document \( d \) after re-ranking. To capture ranking improvements, we compute the rank difference for each document \( d \) as:

\[
\Delta r_d = \max(r_d^{\text{init}} - r_d^{\text{re-rank}}, 0).
\]

A positive value of \( \Delta r_d \) indicates that the document has moved up in the ranking. For each query \( q \), the total observed improvement is given by:

\[
\Delta r_q = \sum_{d \in D_q^{\text{Translated}}} \Delta r_d.
\]

To normalize this improvement, we first define the maximum possible improvement for each document as:

\[
\Delta r_d^{\text{max}} = r_d^{\text{init}} - 1,
\]

and then compute the total maximum improvement for query \( q \):

\[
\Delta r_q^{\text{max}} = \sum_{d \in D_q^{\text{Translated}}} \Delta r_d^{\text{max}}.
\]

The query-specific MLRS score is then calculated as:

\[
\text{MLRS}_q = \begin{cases}
\frac{\Delta r_q}{\Delta r_q^{\text{max}}} \times 100, & \text{if } \Delta r_q^{\text{max}} > 0, \\
0, & \text{otherwise.}
\end{cases}
\]

Finally, the overall MultiLingualRankShift is obtained by averaging the scores over all queries:

\[
\text{MLRS} = \frac{1}{|Q|} \sum_{q \in Q} \text{MLRS}_q.
\]

3 General Setup

3.1 Dataset

By following previous study (Chirkova et al., 2024), we use MKQA (Longpre et al., 2021) dataset, a multilingual open domain question answering evaluation set in our experiments. MKQA consists of 10k examples from the Natural Questions (NQ) dataset (Kwiatkowski et al., 2019), translated into 25 languages. This dataset is therefore parallel between languages and grounds knowledge primarily in English Wikipedia. In our experiments, we also select a subset of 2.7K samples, overlapping between MKQA and KILT NQ datasets *, thus recovering relevant documents information from KILT NQ.

3.2 Models

Multilingual Retrievers Following previous work (Chirkova et al., 2024), we use a strong and publicly available BGE-m3 (Chen et al., 2024) as our multilingual retriever which can

*https://huggingface.co/datasets/facebook/kilt_tasks
encode various languages we consider in our experiments. Consistent with the retriever, we use BGE-m3 (Chen et al., 2024) as a re-ranking encoder for computing MLRS. In addition, we use two other Sentence-BERT series re-ranking encoders (Reimers and Gurevych, 2019), paraphrase-multilingual-MiniLM-L12-v2 and paraphrase-multilingual-mpnet-base-v2 to generalize the experimental results. We abbreviate them as p-mMiniLM and p-mMpNet for better visibility of the table.

Generators We use a recently released strong multilingual LLM, aya-expanse-8b (Dang et al., 2024) that can deal with various languages well. Also, we use strong LLMs, qwen 2.5-7B-Instruct (Team, 2024) and Phi-4 14B (Abdin et al., 2024), Llama-3.1-8B-Instruct (Dubey et al., 2024) as our generators.

3.3 Other Implementation Details

Translation model We utilize a robust translation model for various languages, NLLB-200-distilled-600M (Costa-jussà et al., 2022) in our experiments.

Datastore Following previous study (Chirkova et al., 2024), we use Wikipedia as our datastore documents collection. In most of our experiments, we retrieve from two main Wikipedia sources: the KILT version of English Wikipedia† and the Wikipedia edition in the user’s native language‡. For detailed statistics, please refer to Appendix D.

Baseline We conduct several experiments to measure the language preference of mRAG based on Bergen (Chirkova et al., 2024). Bergen explores the components and adjustments necessary to develop an effective mRAG pipeline, serving as a robust baseline for future research.

4 Language Preference of Retrievers

In this section, we examine two factors that may affect the retriever’s language preference: (i) the relationship between the query language (\( L_q \)) and the document language (\( L_d \)), and (ii) the resource availability of the languages involved.

4.1 Effect of the \( L_q \) and \( L_d \) Relationship

4.1.1 Experimental Setup

We evaluate eight language pairs under two scenarios: (1) monolingual settings where \( L_q = L_d \), and (2) cross-lingual settings where \( L_q \neq L_d \). In each case, our primary metric is *MLRS* (Multi-LingualRankShift), computed using three re-ranker encoders (bge-m3, p-mMiniLM, and p-mMpNet). For example, if the query is in English (en) but the target translation is in Korean (ko), we translate all non-English passages into Korean and then measure the rank changes with MLRS.

4.1.2 Results for Monolingual Settings (\( L_q = L_d \))

**Strong Preference When the Query and Document Languages Match.** As shown in the left-most column of Table 1, when the query and document languages are identical, the retriever shows a high preference. This is expected, as direct linguistic alignment avoids the complexities of cross-lingual mapping and translation, thereby yielding stronger preference.

4.1.3 Results for Cross-Lingual Settings (\( L_q \neq L_d \))

**Lower Overall MLRS in Cross-Lingual Matching.** When \( L_q \neq L_d \), the retriever performs cross-lingual matching, which typically results in lower MLRS values than in monolingual cases. As indicated by the right-hand columns in Table 1 (highlighted in blue), cross-lingual setups are generally less preferred than their monolingual counterparts—except in cases involving English.

**English as a Dominant Target Language.** We observe that when the translated document language \( L_d \) is English, the retriever exhibits nearly the highest language preference as stated in the English column (en) in Table 1. In fact, English often outperforms even monolingual configurations, likely due to the abundance of English data in pre-training, which biases the model towards stronger English representations.

**Influence of Language Family Similarity.** Language family and geographic proximity also play a role. For example, Romance languages (fr, it, pt, es) share extensive lexical and structural similarities, which help maintain a relatively high cross-lingual preference and narrow the performance gap with monolingual setups, as illustrated by the joint \( L_q \)

†https://huggingface.co/datasets/facebook/kilt_wikipedia
‡https://huggingface.co/datasets/wikimedia/wikipedia
<table>
  <tr>
    <th rowspan="2">Query Lang.</th>
    <th rowspan="2">Encoder</th>
    <th colspan="6">L_q = L_d</th>
    <th colspan="6">L_q \neq L_d</th>
  </tr>
  <tr>
    <th>en</th>
    <th>ko</th>
    <th>zh</th>
    <th>fr</th>
    <th>ja</th>
    <th>it</th>
    <th>pt</th>
    <th>es</th>
  </tr>
  <tr>
    <td rowspan="3">en</td>
    <td>bge-m3</td>
    <td>56.03</td>
    <td>–</td>
    <td>33.02 (±2.01)</td>
    <td>33.10 (±2.93)</td>
    <td>36.61 (±19.42)</td>
    <td>33.36 (±22.67)</td>
    <td>35.89 (±20.44)</td>
    <td>35.86 (±20.73)</td>
    <td>36.62 (±19.41)</td>
  </tr>
  <tr>
    <td>p-mMiniLM</td>
    <td>56.85</td>
    <td>–</td>
    <td>34.34 (±22.51)</td>
    <td>34.61 (±22.46)</td>
    <td>38.17 (±18.80)</td>
    <td>34.52 (±22.33)</td>
    <td>37.15 (±19.70)</td>
    <td>36.73 (±20.12)</td>
    <td>37.96 (±18.89)</td>
  </tr>
  <tr>
    <td>p-mMpNet</td>
    <td>57.49</td>
    <td>–</td>
    <td>34.45 (±25.06)</td>
    <td>34.27 (±22.32)</td>
    <td>37.94 (±19.85)</td>
    <td>34.67 (±22.83)</td>
    <td>37.34 (±20.81)</td>
    <td>37.02 (±20.47)</td>
    <td>37.90 (±19.95)</td>
  </tr>
  <tr>
    <td rowspan="3">ko</td>
    <td>bge-m3</td>
    <td>41.15</td>
    <td>43.49 (±2.34)</td>
    <td>–</td>
    <td>34.42 (±4.73)</td>
    <td>36.42 (±4.73)</td>
    <td>37.18 (±3.97)</td>
    <td>35.72 (±5.43)</td>
    <td>35.30 (±5.85)</td>
    <td>35.93 (±5.22)</td>
  </tr>
  <tr>
    <td>p-mMiniLM</td>
    <td>42.95</td>
    <td>44.62 (±1.67)</td>
    <td>–</td>
    <td>36.04 (±4.91)</td>
    <td>37.08 (±5.87)</td>
    <td>38.47 (±4.48)</td>
    <td>36.07 (±6.88)</td>
    <td>36.18 (±6.77)</td>
    <td>36.45 (±6.59)</td>
  </tr>
  <tr>
    <td>p-mMpNet</td>
    <td>42.53</td>
    <td>44.98 (±2.45)</td>
    <td>–</td>
    <td>35.85 (±4.68)</td>
    <td>37.20 (±5.33)</td>
    <td>39.01 (±5.52)</td>
    <td>36.21 (±6.32)</td>
    <td>35.63 (±6.88)</td>
    <td>36.34 (±6.19)</td>
  </tr>
  <tr>
    <td rowspan="3">zh</td>
    <td>bge-m3</td>
    <td>44.98</td>
    <td>45.26 (±0.26)</td>
    <td>34.52 (±10.46)</td>
    <td>–</td>
    <td>36.34 (±8.44)</td>
    <td>36.05 (±8.93)</td>
    <td>35.86 (±9.12)</td>
    <td>35.73 (±9.29)</td>
    <td>36.45 (±8.59)</td>
  </tr>
  <tr>
    <td>p-mMiniLM</td>
    <td>46.18</td>
    <td>45.39 (±0.79)</td>
    <td>35.46 (±10.72)</td>
    <td>–</td>
    <td>36.98 (±9.20)</td>
    <td>36.77 (±9.41)</td>
    <td>36.38 (±9.80)</td>
    <td>36.05 (±10.19)</td>
    <td>36.85 (±9.31)</td>
  </tr>
  <tr>
    <td>p-mMpNet</td>
    <td>46.27</td>
    <td>45.41 (±0.86)</td>
    <td>35.21 (±11.06)</td>
    <td>–</td>
    <td>36.87 (±9.40)</td>
    <td>36.71 (±9.56)</td>
    <td>36.28 (±9.99)</td>
    <td>35.94 (±10.33)</td>
    <td>36.78 (±9.49)</td>
  </tr>
  <tr>
    <td rowspan="3">fr</td>
    <td>bge-m3</td>
    <td>43.18</td>
    <td>47.23 (±4.05)</td>
    <td>33.29 (±9.99)</td>
    <td>33.58 (±9.60)</td>
    <td>–</td>
    <td>34.07 (±9.11)</td>
    <td>36.70 (±6.48)</td>
    <td>36.30 (±6.88)</td>
    <td>37.25 (±5.99)</td>
  </tr>
  <tr>
    <td>p-mMiniLM</td>
    <td>44.09</td>
    <td>48.15 (±4.06)</td>
    <td>34.54 (±9.55)</td>
    <td>34.52 (±9.57)</td>
    <td>–</td>
    <td>34.83 (±9.28)</td>
    <td>37.65 (±6.44)</td>
    <td>37.05 (±7.04)</td>
    <td>38.03 (±6.09)</td>
  </tr>
  <tr>
    <td>p-mMpNet</td>
    <td>43.96</td>
    <td>48.14 (±4.18)</td>
    <td>34.25 (±9.71)</td>
    <td>34.37 (±9.59)</td>
    <td>–</td>
    <td>34.61 (±9.35)</td>
    <td>37.59 (±6.37)</td>
    <td>36.93 (±7.03)</td>
    <td>38.01 (±5.99)</td>
  </tr>
  <tr>
    <td rowspan="3">ja</td>
    <td>bge-m3</td>
    <td>45.03</td>
    <td>45.18 (±0.15)</td>
    <td>35.45 (±9.98)</td>
    <td>34.86 (±10.17)</td>
    <td>36.71 (±8.23)</td>
    <td>–</td>
    <td>36.11 (±9.02)</td>
    <td>35.88 (±9.51)</td>
    <td>36.56 (±8.47)</td>
  </tr>
  <tr>
    <td>p-mMiniLM</td>
    <td>45.80</td>
    <td>45.54 (±0.26)</td>
    <td>35.90 (±9.96)</td>
    <td>35.57 (±10.23)</td>
    <td>37.18 (±8.42)</td>
    <td>–</td>
    <td>36.53 (±9.27)</td>
    <td>36.25 (±9.55)</td>
    <td>36.91 (±8.89)</td>
  </tr>
  <tr>
    <td>p-mMpNet</td>
    <td>45.67</td>
    <td>45.39 (±0.28)</td>
    <td>35.73 (±9.94)</td>
    <td>35.30 (±10.37)</td>
    <td>36.94 (±8.73)</td>
    <td>–</td>
    <td>36.24 (±9.43)</td>
    <td>35.98 (±9.69)</td>
    <td>36.62 (±9.06)</td>
  </tr>
  <tr>
    <td rowspan="3">it</td>
    <td>bge-m3</td>
    <td>41.06</td>
    <td>46.63 (±5.57)</td>
    <td>33.30 (±7.76)</td>
    <td>33.47 (±7.59)</td>
    <td>37.92 (±3.14)</td>
    <td>33.86 (±7.20)</td>
    <td>–</td>
    <td>36.44 (±4.62)</td>
    <td>37.68 (±3.38)</td>
  </tr>
  <tr>
    <td>p-mMiniLM</td>
    <td>42.11</td>
    <td>47.49 (±5.58)</td>
    <td>34.57 (±7.54)</td>
    <td>34.59 (±7.52)</td>
    <td>39.07 (±3.04)</td>
    <td>34.80 (±7.31)</td>
    <td>–</td>
    <td>37.55 (±4.56)</td>
    <td>38.83 (±3.29)</td>
  </tr>
  <tr>
    <td>p-mMpNet</td>
    <td>41.98</td>
    <td>47.59 (±5.61)</td>
    <td>34.48 (±7.50)</td>
    <td>34.68 (±7.30)</td>
    <td>38.94 (±3.04)</td>
    <td>34.67 (±7.31)</td>
    <td>–</td>
    <td>37.27 (±4.71)</td>
    <td>38.67 (±3.31)</td>
  </tr>
  <tr>
    <td rowspan="3">pt</td>
    <td>bge-m3</td>
    <td>39.19</td>
    <td>46.44 (±7.45)</td>
    <td>33.37 (±5.32)</td>
    <td>33.46 (±5.73)</td>
    <td>37.83 (±3.30)</td>
    <td>34.02 (±5.17)</td>
    <td>37.13 (±2.96)</td>
    <td>–</td>
    <td>38.61 (±3.81)</td>
  </tr>
  <tr>
    <td>p-mMiniLM</td>
    <td>40.17</td>
    <td>47.75 (±7.58)</td>
    <td>34.67 (±5.50)</td>
    <td>34.91 (±5.26)</td>
    <td>39.02 (±3.15)</td>
    <td>35.03 (±5.14)</td>
    <td>38.25 (±1.92)</td>
    <td>–</td>
    <td>39.68 (±1.49)</td>
  </tr>
  <tr>
    <td>p-mMpNet</td>
    <td>39.91</td>
    <td>47.30 (±7.39)</td>
    <td>34.68 (±5.23)</td>
    <td>34.50 (±5.41)</td>
    <td>38.70 (±3.21)</td>
    <td>34.72 (±5.19)</td>
    <td>38.01 (±1.90)</td>
    <td>–</td>
    <td>39.35 (±1.56)</td>
  </tr>
  <tr>
    <td rowspan="3">es</td>
    <td>bge-m3</td>
    <td>40.76</td>
    <td>46.93 (±6.17)</td>
    <td>33.36 (±7.40)</td>
    <td>33.42 (±7.34)</td>
    <td>37.73 (±3.05)</td>
    <td>33.87 (±6.89)</td>
    <td>37.22 (±3.54)</td>
    <td>36.88 (±3.88)</td>
    <td>–</td>
  </tr>
  <tr>
    <td>p-mMiniLM</td>
    <td>41.81</td>
    <td>47.90 (±6.09)</td>
    <td>34.63 (±7.18)</td>
    <td>34.52 (±7.29)</td>
    <td>38.86 (±2.95)</td>
    <td>34.76 (±7.05)</td>
    <td>38.33 (±3.48)</td>
    <td>37.84 (±3.97)</td>
    <td>–</td>
  </tr>
  <tr>
    <td>p-mMpNet</td>
    <td>41.33</td>
    <td>47.34 (±6.01)</td>
    <td>34.39 (±6.84)</td>
    <td>34.19 (±7.14)</td>
    <td>38.34 (±2.99)</td>
    <td>34.39 (±6.94)</td>
    <td>37.73 (±3.60)</td>
    <td>37.25 (±4.08)</td>
    <td>–</td>
  </tr>
</table>

Table 1: Language preference measured by MLRS with different re-ranking encoders for various query–document language pairs. The \( L_q = L_d \) column shows scores for matching query and document languages, while the remaining columns represent cross-lingual scenarios. Parentheses indicate the change from the \( L_q = L_d \) column (positive for improvement, negative for decline). The highest score per row is in bold, and the second highest is underlined.

and \( L_d \) pairs in Table 1. Similarly, East Asian languages (ko, ja, zh) tend to show moderate declines in cross-lingual scenarios compared to the monolingual baseline, although they still lag behind the highest scores.

4.2 Impact of Language Resource Availability

4.2.1 Experimental Setup
We also investigate whether the volume of available language resources affects MLRS. We categorize languages into three groups based on their distribution in the pre-training corpus of recent LLMs: high-resource (e.g., English), mid-resource (e.g., Spanish), and low-resource (e.g., Korean). We use the same query set across all setups while systematically varying \( L_q \) and \( L_d \).

4.2.2 Results
Limited Impact of Query Language Resources.
The resource level of the query language (\( L_q \)) has a limited effect on cross-lingual preference. As shown in Table 1, when \( L_q \) is high-resource (e.g., English), strong preference is observed only if \( L_d \) also matches a high-resource language. Otherwise, the MLRS scores remain similar regardless of whether \( L_q \) is high-, mid-, or low-resource.

Document Language Resources Are More Influential.
In contrast, the language resource level of the document language (\( L_d \)) has a pronounced impact on MLRS. As shown in Table 1, documents from high-resource languages consistently achieve the highest preference scores, followed by mid-resource and then low-resource languages. This trend (High > Mid > Low) suggests that extensive pre-training on high-resource languages enables stronger alignment, yielding higher MLRS even across diverse query languages. Conversely, low-resource datastores typically produce lower MLRS scores unless the query language also corresponds to that low-resource setting.

Overall, our results indicate that the resource availability of \( L_d \) critically influences the language preference of the retriever. These findings lay the groundwork for further investigation into the language dynamics within mRAG systems.

4.3 Impact of Translation Quality
To investigate the effect of translation quality on language preference as measured by the MLRS metric, we conduct a small-scale experiment by randomly sampling 100 queries out of the full set
Figure 3: Language Preference of the Generators. In each figure, "aya" represents aya-expanse-8B, "llama" represents Llama-3.1-8B-Instruct, and "gpt" represents gpt-4o-mini. The red dotted line indicates the average generator preference.

of 2,827. We translate these queries using GPT-4o-mini, which demonstrated highly robust translation quality. We evaluate five languages (English, Korean, Spanish, Chinese, and French) and re-ranked the top-10 retrieved documents using the BGE-M3 encoder, and compute rank shifts within the top-10 to derive MLRS for monolingual (\( L_q = L_d \)) and cross-lingual (\( L_q \neq L_d \)) settings. The results, shown in Table 2, are consistent with prior MLRS findings and further reveal that GPT-based translation amplifies the distinctness of language preference trends, as indicated by larger inter-language preference gaps.

<table>
  <tr>
    <th rowspan="2">Query Lang.</th>
    <th rowspan="2">\( L_q = L_d \)</th>
    <th colspan="5">\( L_q \neq L_d \)</th>
  </tr>
  <tr>
    <th>en</th>
    <th>ko</th>
    <th>zh</th>
    <th>fr</th>
    <th>es</th>
  </tr>
  <tr>
    <td>en</td>
    <td>58.29</td>
    <td>34.98 (±23.11)</td>
    <td>36.10 (±22.16)</td>
    <td>38.49 (±30.09)</td>
    <td>40.07 (±31.25)</td>
  </tr>
  <tr>
    <td>ko</td>
    <td>47.15</td>
    <td>42.59 (±4.86)</td>
    <td>37.50 (±6.05)</td>
    <td>39.31 (±7.48)</td>
    <td>38.73 (±6.05)</td>
  </tr>
  <tr>
    <td>zh</td>
    <td>47.42</td>
    <td>46.66 (±6.70)</td>
    <td>36.89 (±8.17)</td>
    <td>38.24 (±6.84)</td>
    <td>38.10 (±6.05)</td>
  </tr>
  <tr>
    <td>fr</td>
    <td>47.35</td>
    <td>50.76 (±4.74)</td>
    <td>34.90 (±12.46)</td>
    <td>36.10 (±11.25)</td>
    <td>–</td>
    <td>38.90 (±4.45)</td>
  </tr>
  <tr>
    <td>es</td>
    <td>43.98</td>
    <td>50.73 (±4.79)</td>
    <td>35.06 (±10.02)</td>
    <td>36.18 (±8.80)</td>
    <td>40.52 (±5.40)</td>
    <td>–</td>
  </tr>
</table>

Table 2: MLRS scores for monolingual (\( L_q = L_d \)) and cross-lingual (\( L_q \neq L_d \)) settings on 100 randomly sampled queries.

5 Language Preference of Generators

In this section, we explore LLM generators’ language preferences in mRAG and their impact on overall performance.

5.1 Do LLMs Prefer Certain Languages for Contextual Knowledge?

5.1.1 Experimental Setup

To assess the generator’s language preference, we measure multilingual answer consistency across eight languages: English (en), Korean (ko), Chinese (zh), French (fr), Japanese (ja), Italian (it), Portuguese (pt), and Spanish (es). As shown in Table 13, for a given query the generator produces responses in each language using the same set of retrieved documents from the multilingual datastore. We then compute the embedding similarity between each pair of generated answers, resulting in an 8×8 similarity matrix. We define the preference for a specific language as the average similarity score of the responses in that language.

We use LaBSE for measuring multilingual semantic similarity (Feng et al., 2022). And we use aya-expanse-8B, Llama-3.1-8B-Instruct, and GPT-4o-mini as our generators. We use language-specific prompts that incorporate the retrieved passages to induce responses in the target language, enabling us to capture the generator’s inherent language preference.

5.1.2 Results

Strong Preference for Latin Script Languages.
Figure 3 indicates that the generator produces more consistent responses in languages that use Latin scripts (e.g., en, fr, it, pt, es) compared to non-Latin languages (e.g., ko, zh, ja). This suggests that the model benefits from structural advantages in token alignment when processing Latin-based languages.

Modest Preference for the Query Languages.
In addition, the generator shows a slight increase in consistency when the output language matches a query language. For instance, Korean (ko) queries yield somewhat more consistent responses when the generator replies in Korean rather than when the query is in English. However, this improvement is marginal, suggesting that the overall preference toward Latin scripts remains influential. Despite the modest gain, the model still demonstrates some capacity to handle multilingual queries effectively, indicating that matching the query language can provide a small but measurable benefit in non-Latin contexts.
5.2 Correlation between Language Preference and mRAG Performance

5.2.1 Experimental Setup
To examine how language preference impacts overall mRAG performance, we isolate language effects by providing generators with retrieved passages unified in a single target language—selected from the eight candidates (en, ko, zh, fr, ja, it, pt, es). We retrieve data from multilingual sources, enabling a direct comparison between language preference (measured by MLRS, as shown in Table 1) and performance across three query languages (en, ko, zh).

We evaluate four generators (aya-expans-8B, Phi-4, Qwen2.5-7B-Instruct, and Llama3.1-8B-Instruct) using character 3-gram recall (Chirkova et al., 2024) under three configurations. In the first configuration, passages are retrieved from multilingual resources, denoted as all. In the second, all retrieved passages are unified into a single target language (single-language document set). Finally, in the third configuration, we employ our proposed DKM-RAG framework (detailed in Section 6) to mitigate language preference. We also compute the average MLRS score (across different query languages) for each language to indicate its overall preference.

5.2.2 Results and Analysis
Strong Correlation for English Queries. As stated in Table 3, for queries in English (\( L_q = \text{en} \)), RAG performance shows a strong correlation with language preference. English achieves the best results—likely due to its high-resource availability and the model’s familiarity with it. In this setting, the all strategy is particularly effective, as it leverages cross-lingual knowledge fusion. We observe an exception for Japanese (ja), where performance is lower despite moderate preference, possibly due to challenges with non-Latin scripts and complex morphology.

Weaker Correlation for Non-English Queries. When \( L_q \neq \text{en} \), the relationship between language preference and performance becomes less pronounced. Although the generator generally prefers English passages overall, it achieves optimal performance when it receives retrieved passages that directly match the query language. In these cases, translating all passages into English does not enhance performance; instead, maintaining language consistency between the query and passages yields better results. This finding underscores the importance of linguistic compatibility in mRAG systems.

Optimal mRAG Strategy. Based on our experiments, different strategies depending on the query language prove more effective. As stated in Table 3, for English queries, employing the all strategy capitalizes on the high cross-lingual preference for English. In contrast, for non-English queries, translating retrieved passages into the query language \( L_q \) bridges the comprehension gap and ensures better alignment between query intent, passage semantics, and output language. This targeted approach ultimately leads to improved RAG performance by accommodating the specific language dynamics of the generator.

![Overall flow of proposed DKM-RAG.](page_1012_1092_388_388.png)

Figure 4: Overall flow of proposed DKM-RAG.

6 Dual Knowledge Multilingual RAG

Translating retrieved documents into the query language benefits mRAG, but it may also reflect retrieval outputs from high-resource languages including irrelevant content. Therefore, leveraging the LLM’s internal knowledge can help filter inaccuracies and enrich the retrieved information with more reliable content. So we rewrite translated passages to refine the relevancy of documents by leveraging LLM’s internal information.

Based on this insight, we propose Dual Knowledge Multilingual RAG (DKM-RAG), a framework
<table>
  <tr>
    <th></th>
    <th>all</th>
    <th>en</th>
    <th>zh</th>
    <th>ko</th>
    <th>fr</th>
    <th>ja</th>
    <th>it</th>
    <th>pt</th>
    <th>es</th>
    <th>DKM-RAG</th>
  </tr>
  <tr>
    <th colspan="11">\( L_q = \text{en} \)</th>
  </tr>
  <tr>
    <td>aya-expanse-8b</td>
    <td>80.09</td>
    <td><b>79.34</b></td>
    <td>63.08</td>
    <td>64.46</td>
    <td>76.13</td>
    <td>61.20</td>
    <td>75.47</td>
    <td>75.65</td>
    <td>76.32</td>
    <td><b>82.60</b></td>
  </tr>
  <tr>
    <td>Phi-4</td>
    <td>79.69</td>
    <td><b>78.89</b></td>
    <td>63.06</td>
    <td>52.30</td>
    <td>74.43</td>
    <td>48.86</td>
    <td>74.02</td>
    <td>74.39</td>
    <td>75.32</td>
    <td><b>82.59</b></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>80.15</td>
    <td><b>79.11</b></td>
    <td>50.31</td>
    <td>64.90</td>
    <td>76.28</td>
    <td>62.62</td>
    <td>75.47</td>
    <td>75.97</td>
    <td>76.54</td>
    <td><b>82.60</b></td>
  </tr>
  <tr>
    <td>Llama3.1-8B-Instruct</td>
    <td>80.25</td>
    <td><b>79.28</b></td>
    <td>61.99</td>
    <td>65.81</td>
    <td>76.40</td>
    <td>62.58</td>
    <td>75.89</td>
    <td>76.09</td>
    <td>76.47</td>
    <td><b>82.57</b></td>
  </tr>
  <tr>
    <th colspan="11">\( L_q = \text{zh} \)</th>
  </tr>
  <tr>
    <td>aya-expanse-8b</td>
    <td>32.55</td>
    <td>25.62</td>
    <td><b>38.31</b></td>
    <td>26.64</td>
    <td>24.00</td>
    <td>25.27</td>
    <td>23.63</td>
    <td>23.63</td>
    <td>23.79</td>
    <td><b>44.57</b></td>
  </tr>
  <tr>
    <td>Phi-4</td>
    <td>16.75</td>
    <td>17.57</td>
    <td><b>36.76</b></td>
    <td>17.50</td>
    <td>18.15</td>
    <td>17.56</td>
    <td>18.19</td>
    <td>17.89</td>
    <td>18.44</td>
    <td><b>44.56</b></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>34.28</td>
    <td>27.33</td>
    <td><b>38.31</b></td>
    <td>27.91</td>
    <td>25.15</td>
    <td>27.78</td>
    <td>25.90</td>
    <td>25.37</td>
    <td>25.30</td>
    <td><b>44.70</b></td>
  </tr>
  <tr>
    <td>Llama3.1-8B-Instruct</td>
    <td>28.50</td>
    <td>24.36</td>
    <td><b>38.48</b></td>
    <td>23.84</td>
    <td>22.48</td>
    <td>23.78</td>
    <td>23.18</td>
    <td>23.32</td>
    <td>23.02</td>
    <td><b>44.51</b></td>
  </tr>
  <tr>
    <th colspan="11">\( L_q = \text{ko} \)</th>
  </tr>
  <tr>
    <td>aya-expanse-8b</td>
    <td>40.60</td>
    <td>38.08</td>
    <td>26.01</td>
    <td><b>49.66</b></td>
    <td>25.37</td>
    <td>26.82</td>
    <td>24.98</td>
    <td>25.26</td>
    <td>25.51</td>
    <td><b>55.01</b></td>
  </tr>
  <tr>
    <td>Phi-4</td>
    <td>26.80</td>
    <td>20.24</td>
    <td>17.54</td>
    <td><b>49.25</b></td>
    <td>19.03</td>
    <td>17.91</td>
    <td>18.93</td>
    <td>19.19</td>
    <td>19.19</td>
    <td><b>54.82</b></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>36.50</td>
    <td>22.87</td>
    <td>20.08</td>
    <td><b>49.44</b></td>
    <td>21.79</td>
    <td>20.94</td>
    <td>21.65</td>
    <td>21.44</td>
    <td>21.52</td>
    <td><b>54.85</b></td>
  </tr>
  <tr>
    <td>Llama3.1-8B-Instruct</td>
    <td>37.18</td>
    <td>26.48</td>
    <td>22.88</td>
    <td><b>49.87</b></td>
    <td>24.46</td>
    <td>24.86</td>
    <td>25.23</td>
    <td>24.87</td>
    <td>25.22</td>
    <td><b>54.99</b></td>
  </tr>
  <tr>
    <td>MLRS (Preference)</td>
    <td>-</td>
    <td><b>47.70</b></td>
    <td>35.90</td>
    <td>35.47</td>
    <td>37.94</td>
    <td>37.59</td>
    <td>37.66</td>
    <td>37.15</td>
    <td><b>37.97</b></td>
    <td>-</td>
  </tr>
</table>

Table 3: Performance comparison between DKM-RAG and single/all language retrieval settings, showing character 3-gram recall scores for three query languages (\( L_q \in \{\text{en}, \text{ko}, \text{zh}\} \)) and eight passage languages. The bottom row shows average preference (MLRS) scores. We highlight the cells corresponding to matching query and passage languages with a yellow background. The highest score per row is in bold, and the second highest is underlined.

that leverages both external translated passages and internal knowledge as shown in Figure 4. First (#1), we retrieve documents for a given query from the all strategy and re-rank them. Next (#2), we obtain external translated passages, \( P_{\text{translated}} \) by translating into the query language. And (#3), the rewriter LLM refines each translated passage in the context of the given query to produce refined passages, \( P_{\text{refined}} \). This refining process utilizes a prompt to guide the model in integrating its internal knowledge, removing redundancy, and highlighting relevant information in a coherent and consistent style. For detailed prompts, please refer to Appendix B. Finally (#4), We concatenate the two sets to form the final passage set as input to the generator LLM, ensuring that responses are both contextually enriched and linguistically aligned with the query.

Results. As shown in Table 3, DKM-RAG outperforms other document-based generator settings. For non-English queries (\( L_q \neq \text{en} \)), it leverages translated passages and enriched content to handle linguistic diversity. Even for English queries (\( L_q = \text{en} \)), it surpasses the all baseline, highlighting the importance of integrating translated and refined knowledge.

Ablation Study. To prove the effectiveness of concatenating translated passages and refined passages in the DKM-RAG framework, we provide an ablation study of each component in DKM-RAG. As stated in Table 4, removing any component from DKM-RAG decreases performance, highlighting that every part is crucial to its effectiveness.

7 Related Works

7.1 Multilingual RAG

Researchers explore challenges in mRAG, such as the problem of cross-lingual dense passage retrieval for low-resource languages (Wu et al., 2024a), and propose various techniques to address key challenges in mRAG, such as enhancing the performance of language models in low-resource languages (Deshpande et al., 2024), resolving low-resource scenarios (Zhang et al., 2024), and adapting language models for multilingual reasoning tasks (Yoon et al., 2024). Benchmarks like MMTEB (Enevoldsen et al., 2025) enable systematic evaluation of multilingual retrieval.

Earlier mRAG systems frequently focus on high-resource languages (e.g., English), but a growing body of research aims to make advanced Natural Language Processing (NLP) technology accessible across a wide spectrum of linguistic contexts. Proposed solutions include code-mixed prompts for in-context learning (Shankar et al., 2024) and self-distillation from resource-rich to low-resource languages (Zhang et al., 2024).
7.2 Language Preference in mRAG

Despite significant progress, language preference—a systematic tendency to favor certain languages—remains a critical issue in mRAG systems. This preference arises from imbalances in training data, tokenization mismatches, script differences, and uneven resource availability (Sharma et al., 2024; Wu et al., 2024b). Studies show that high-resource languages (e.g., English) often overshadow relevant content in lower-resource languages during retrieval (Yang et al., 2024b; Chirkova et al., 2024), leading to suboptimal evidence retrieval (Yang et al., 2024a) and causing inconsistencies or hallucinations in outputs (Chataigner et al., 2024). These disparities also raise broader fairness concerns in multilingual NLP, as pre-trained models exhibit group fairness issues across languages (Cabello Piqueras and Søgaard, 2022; Ramesh et al., 2023).

Researchers propose several methods to counteract language preferences, including language-preference-based re-ranking (Telemala and Suleman, 2022), evaluate knowledge consistency across languages (Qi et al., 2023), and specialized datasets designed to detect such imbalances (Li et al., 2024). However, these approaches often focus on a single mRAG stage or overlook the actual ranking of retrieved documents (Sharma et al., 2024; Yang et al., 2024a). We introduce a metric that quantifies language preference in retrieval via ranking differences and propose a simple framework to mitigate these preferences across the entire mRAG pipeline.

<table>
  <tr>
    <th></th>
    <th>DKM-RAG</th>
    <th>w/o \( P_{\text{refined}} \)</th>
    <th>w/o \( P_{\text{translated}} \)</th>
  </tr>
  <tr>
    <th colspan="4">\( L_q = en \)</th>
  </tr>
  <tr>
    <td>aya-expanse-8b</td>
    <td>82.60</td>
    <td>79.34</td>
    <td>81.10</td>
  </tr>
  <tr>
    <td>Phi-4</td>
    <td>82.59</td>
    <td>78.89</td>
    <td>81.08</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>82.60</td>
    <td>79.11</td>
    <td>81.06</td>
  </tr>
  <tr>
    <td>Llama3.1-8B-Instruct</td>
    <td>82.57</td>
    <td>79.28</td>
    <td>81.19</td>
  </tr>
  <tr>
    <th colspan="4">\( L_q = zh \)</th>
  </tr>
  <tr>
    <td>aya-expanse-8b</td>
    <td>44.57</td>
    <td>38.31</td>
    <td>39.44</td>
  </tr>
  <tr>
    <td>Phi-4</td>
    <td>44.56</td>
    <td>36.76</td>
    <td>38.95</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>44.70</td>
    <td>38.31</td>
    <td>39.78</td>
  </tr>
  <tr>
    <td>Llama3.1-8B-Instruct</td>
    <td>44.51</td>
    <td>38.48</td>
    <td>39.35</td>
  </tr>
  <tr>
    <th colspan="4">\( L_q = ko \)</th>
  </tr>
  <tr>
    <td>aya-expanse-8b</td>
    <td>55.01</td>
    <td>49.66</td>
    <td>46.15</td>
  </tr>
  <tr>
    <td>Phi-4</td>
    <td>54.82</td>
    <td>49.25</td>
    <td>45.24</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>54.85</td>
    <td>49.44</td>
    <td>45.32</td>
  </tr>
  <tr>
    <td>Llama3.1-8B-Instruct</td>
    <td>54.99</td>
    <td>49.87</td>
    <td>45.55</td>
  </tr>
</table>

Table 4: Ablation study on DKM-RAG. “DKM-RAG” denotes the DKM-RAG setting (i.e., the DKM-RAG column in Table 3), “w/o \( P_{\text{refined}} \)” indicates the performance corresponding to the highlighted cells, and “w/o \( P_{\text{translated}} \)” represents the results using only refined passages.

8 Conclusion

In this work, we investigate language preferences in mRAG systems. We propose a metric that measures the language preference of retrievers by checking the rank difference between the translated passage and the original one. Our experiments reveal that retrievers prefer high-resource and query language but do not always yield better generation performance. We also find that generators often favor the query language or Latin scripts, resulting in inconsistent outputs. To address this, we propose DKM-RAG which integrates translated passages with internal knowledge. Empirical results show that DKM-RAG consistently enhances mRAG performance across diverse languages.

Limitations

Our approach involves translating documents to measure rank shifts and unify linguistic representations. This process relies heavily on the quality of the translation model employed. Errors or inaccuracies in translation can distort the original meaning of passages and potentially introduce noise into both the retrieval and generation stages.

MLRS entails translation and re-ranking steps. While this approach offers a principled way to quantify language preference, it also adds latency and computational cost, especially when dealing with large-scale multilingual corpora or real-time systems.

DKM-RAG framework which combines external translated passages and parametric (internal) knowledge, improves performance yet remains relatively straightforward. Future work could explore more sophisticated techniques for merging external and internal knowledge (e.g., trainable fusion mechanisms, dynamic weighting) to further reduce preferences and enhance overall system capabilities.

Lastly, our experiments focus on Wikipedia-based datasets in a specific set of languages, which may not generalize to all linguistic varieties or specialized domains. Future research should examine broader contexts, including low-resource languages not present in widely available corpora or
domain-specific retrieval settings, to fully assess how language preferences manifest across diverse real-world scenarios.

Ethics Statement

We conduct our experiments using publicly available, multilingual dataset and models that follow recognized research and data-sharing guidelines. These resources are widely utilized in the academic community and are distributed with the intent to minimize harmful biases, inappropriate content, or stereotypes. However, they may still not fully represent the diversity of all languages and cultural contexts. We adhere strictly to the usage protocols and license agreements set forth by the original providers, who have taken steps to ensure compliance with established ethical standards.

Acknowledgments

We would like to thank Byeongjeong Kim for his comments and feedback about our figures. We also thank Gyutae Park for his minor corrections to this work. This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [RS-2021-II211341, Artificial Intelligent Graduate School Program (Chung-Ang University)]. This research was supported by the Chung-Ang University Graduate Research Scholarship in 2025.

References

Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojgan Jahareri, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weisheng Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. 2024. Phi-4 technical report.

Laura Cabello Piqueras and Anders Søgaard. 2022. Are pretrained multilingual models equally fair across languages? In Proceedings of the 29th International Conference on Computational Linguistics, pages 3597–3605, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.

Cléa Chataigner, Afaf Taïk, and Golnoosh Farnadi. 2024. Multilingual hallucination gaps in large language models.

Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation.

Nadezhda Chirkova, David Rau, Hervé Déjean, Thibault Formal, Stéphane Clinchant, and Vassilina Nikoulina. 2024. Retrieval-augmented generation in multilingual settings. In Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024), pages 177–188, Bangkok, Thailand. Association for Computational Linguistics.

Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672.

John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, Acyr Locatelli, Hangyu Lin, Dwarkan Talupuru, Bharat Venkitesh, David Cairuz, Bowen Yang, Tim Chung, Wei-Yin Ko, Sylvie Shang Shi, Amir Shukayev, Sammie Bae, Aleksandra Piktus, Roman Castagné, Felipe Cruz-Salinas, Eddie Kim, Lucas Crawhall-Stein, Adrien Morisot, Sudip Roy, Phil Blunsom, Ivan Zhang, Aidan Gomez, Nick Frosts, Marzieh Fadaee, Beyza Ermiş, Ahmet Üstün, and Sara Hooker. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier.

Tejas Deshpande, Nidhi Kowtal, and Raviraj Joshi. 2024. Chain-of-translation prompting (cotr): A novel prompting technique for low resource languages. arXiv preprint arXiv:2409.04512.

Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzeminski, Genta Indra Winata, et al. 2025. Mmtcb: Massive multilingual text embedding benchmark. The International Conference on Learning Representations.

Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Ariazhagan, and Wei Wang. 2022. Language-agnostic bert sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878–891.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. Curran Associates Inc.

Bryan Li, Samar Haider, Fiona Luo, Adwait Agashe, and Chris Callison-Burch. 2024. BordIRlines: A dataset for evaluating cross-lingual retrieval augmented generation. In Proceedings of the First Workshop on Advancing Natural Language Processing for Wikipedia, pages 1–13, Miami, Florida, USA. Association for Computational Linguistics.

Shayne Longpre, Yi Lu, and Joachim Daiber. 2021. MKQA: A linguistically diverse benchmark for multilingual open domain question answering. Transactions of the Association for Computational Linguistics, 9:1389–1406.

Jirui Qi, Raquel Fernández, and Arianna Bisazza. 2023. Cross-lingual consistency of factual knowledge in multilingual language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10650–10666, Singapore. Association for Computational Linguistics.

Krithika Ramesh, Sunayana Sitaram, and Monojit Choudhury. 2023. Fairness in language models beyond English: Gaps and challenges. In Findings of the Association for Computational Linguistics: EACL 2023, pages 2106–2119, Dubrovnik, Croatia. Association for Computational Linguistics.

Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.

Bhavani Shankar, Preethi Jyothi, and Pushpak Bhattacharyya. 2024. In-context mixing (ICM): Code-mixed prompts for multilingual LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4162–4176, Bangkok, Thailand. Association for Computational Linguistics.

Nikhil Sharma, Kenton Murray, and Ziang Xiao. 2024. Faux polyglot: A study on information disparity in multilingual large language models.

Qwen Team. 2024. Qwen2.5: A party of foundation models.

Joseph P. Telemala and Hussein Suleman. 2022. Language-preference-based re-ranking for multilingual swahili information retrieval. In Proceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR ’22, page 144–152, New York, NY, USA. Association for Computing Machinery.

Jie Wu, Zhaochun Ren, and Suzan Verberne. 2024a. What are the limits of cross-lingual dense passage retrieval for low-resource languages?

Suhang Wu, Jialong Tang, Baosong Yang, Ante Wang, Kaidi Jia, Jiawei Yu, Junfeng Yao, and Jinsong Su. 2024b. Not all languages are equal: Insights into multilingual retrieval-augmented generation.

Eugene Yang, Thomas Jänich, James Mayfield, and Dawn Lawrie. 2024a. Language fairness in multilingual information retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page 2487–2491, New York, NY, USA. Association for Computing Machinery.

Jinrui Yang, Fan Jiang, and Timothy Baldwin. 2024b. Language bias in multilingual information retrieval: The nature of the beast and mitigation methods. In Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024), pages 280–292, Miami, Florida, USA. Association for Computational Linguistics.

Dongkeun Yoon, Joel Jang, Sungdong Kim, Seungone Kim, Sheikh Shafayat, and Minjoon Seo. 2024. LangBridge: Multilingual reasoning without multilingual supervision. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7502–7522, Bangkok, Thailand. Association for Computational Linguistics.

Yuanchi Zhang, Yile Wang, Zijun Liu, Shuo Wang, Xiaolong Wang, Peng Li, Maosong Sun, and Yang Liu. 2024. Enhancing multilingual capabilities of large language models through self-distillation from resource-rich languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11189–11204, Bangkok, Thailand. Association for Computational Linguistics.
A Implementation Details

When retrieving from the datastore in all languages, we utilize the approach outlined in (Chirkova et al., 2024) as our baseline. Specifically, we employ the basic_translated_langspec prompt template, as detailed in Table 6 to generate our final mRAG answer from the generator. In our method, we retrieve and re-rank the top-50 documents for each query, and then use only the top-5 documents to generate the final answer. The document retrieval and re-ranking are carried out using bge-m3. We do not translate documents already in query language in the framework of DKM-RAG to reduce costs.

We conduct our experiments using an AMD EPYC 7313 CPU (3.0 GHz) paired with four NVIDIA RTX 4090 GPUs. We use Python 3.11.5 and PyTorch 2.3.1 for the software environment.

<table>
  <tr>
    <th>Language</th>
    <th>Passage Count (M)</th>
    <th>Percentage (%)</th>
  </tr>
  <tr>
    <td>ja</td>
    <td>27</td>
    <td>20.53</td>
  </tr>
  <tr>
    <td>en</td>
    <td>25</td>
    <td>19.00</td>
  </tr>
  <tr>
    <td>de</td>
    <td>14</td>
    <td>10.64</td>
  </tr>
  <tr>
    <td>fr</td>
    <td>13</td>
    <td>9.88</td>
  </tr>
  <tr>
    <td>zh</td>
    <td>11</td>
    <td>8.36</td>
  </tr>
  <tr>
    <td>es</td>
    <td>10</td>
    <td>7.60</td>
  </tr>
  <tr>
    <td>ru</td>
    <td>8.6</td>
    <td>6.54</td>
  </tr>
  <tr>
    <td>it</td>
    <td>8.2</td>
    <td>6.23</td>
  </tr>
  <tr>
    <td>pt</td>
    <td>4.7</td>
    <td>3.57</td>
  </tr>
  <tr>
    <td>th</td>
    <td>3.7</td>
    <td>2.81</td>
  </tr>
  <tr>
    <td>ar</td>
    <td>3.3</td>
    <td>2.51</td>
  </tr>
  <tr>
    <td>ko</td>
    <td>1.6</td>
    <td>1.22</td>
  </tr>
  <tr>
    <td>fi</td>
    <td>1.5</td>
    <td>1.14</td>
  </tr>
</table>

Table 5: Language distribution of wikipedia we use in our experiment.

B Prompts

As shown in Table 6, we provide the prompts used to generate our final answer with the retrieved documents in our mRAG baseline. Docs refers to retrieved documents and question refers to the current query. We also provide prompts during the passage rewriting phase in the DKM-RAG framework as stated in Table 7. We only provide english prompts for simplicity. And we provide prompts to measure the language preference of GPT-4o-mini, regarding answering in the specific languages as stated in Table 8.

C Language Notation

In this work, we use standard ISO 639-1 language codes to represent the various languages involved in our experiments. Specifically, en denotes English, ko represents Korean, ar corresponds to Arabic, zh refers to Chinese (Simplified), fi indicates Finnish, fr stands for French, de represents German, ja corresponds to Japanese, it refers to Italian, pt denotes Portuguese, ru stands for Russian, es represents Spanish, and th corresponds to Thai. These concise notations facilitate the identification and processing of language-specific data across datasets and models in multilingual NLP research.

D Dataset Statistics

We present the statistics of the datasets used in our experiments. MKQA serves as the primary dataset, and its details, including the number of examples and the median lengths of questions and answers, are summarized in Table 10. Additionally, we utilize Wikipedia as the external source for the retriever datastore, with its statistics (number of passages and median lengths) also provided in Table 10. And we provide the number of passages in each language and the ratio of them in Table 5. These details offer a clear overview of the data resources supporting our experiments.

Language Distribution of Pre-trained LLM
We provide language distribution in the pre-training corpus of Llama-2. As stated in Table 9, we use English (EN) as a high-resource, Spanish (ES) as a mid-resource, and Korean (KO) as a low-resource language in our experiment based on their ratios.

E Language Preference of Other Languages

We also perform additional experiments to explore language preferences for languages not covered in Table 1, using the MLRS score that we propose. As shown in Table 11, similar to the results in Table 1, the highest preferences are typically observed when \( L_q = L_d \) across all query languages. English is also the most preferred language. For clarity, we omit results for other languages.

For most languages, such as Arabic, Finnish, German, and Russian, switching to a cross-lingual setup leads to a significant drop in MLRS. For example, Arabic queries using the bge-m3 encoder achieve a monolingual score of 40.39, but cross-lingual retrieval (e.g., with Thai) results in a 6.80-point decrease.

Interestingly, for Thai queries, some cross-lingual pairs show a slight improvement over the monolingual baseline (as indicated by the positive
<table>
  <tr>
    <th>System</th>
    <th>Prompt</th>
  </tr>
  <tr>
    <td>With Documents</td>
    <td>You are a helpful assistant. Your task is to extract relevant information from provided documents and to answer to questions as short as possible. Please reply in English.<br>user: f"Background:{docs}\n\nQuestion:{question}"</td>
  </tr>
  <tr>
    <td>Without Documents</td>
    <td>You are a helpful assistant. Answer the questions as short as possible. Please reply in English.<br>user: f"Question:{question}"</td>
  </tr>
</table>

Table 6: System prompts with and without documents. The table outlines how instructions and prompts differ when documents are provided or omitted.

Prompt

Original Passage: {passage}

Question: {question}

Please create an independent document according to the following requirements:

1) Utilize known facts (parametric knowledge) related to the question.

2) Seamlessly combine with the original passage by removing redundant or unnecessary sentences. No additional explanations are allowed.

3) All content must be written smoothly and concisely in English.

Table 7: The prompt used for generating \( P_{\text{refined}} \) based on the passage and question. The instructions guide the generator to combine parametric knowledge with the original passage while ensuring clarity and conciseness.

differences in red), suggesting that for low-resource languages like Thai, cross-lingual signals might sometimes offer complementary benefits

F Similarity Matrices

We provide similarity matrix measured by LaBSE for each query language en, zh, ko and each generator in Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 10, Figure 11, Figure 12 and Figure 13. Each entry represents the embedding similarity score between answers generated in different languages, with the diagonal values all equal to 1 (i.e., comparing an answer with itself). Moreover, the values shown in Figure 3 are computed by averaging over the rows or columns for each language.

G Case study

MLRS We provide an example of a document that improved MLRS score, where the rank of a relevant document significantly increases after translation. In Table 12, the user query "영국 캐리비안에 언제 노예제가 폐지됐나요?" (When was slavery abolished in the British Caribbean?)" is in Korean, whereas the original passage is in English. Initially, the document's rank (\( r_d^{\text{init}} = 34 \)) was relatively low, but after translating the passage into Korean and re-ranking (\( r_d^{\text{rank}} = 2 \)), the document moved much closer to the top. This demonstrates how cross-lingual alignment can substantially improve retrieval performance in a multilingual setting. Notably, even if the passage content is semantically the same, language preference in the model can lead to poor alignment when the query and document are in different languages, adversely affecting retrieval. Translating the document into the query language effectively mitigates this issue.

Answer Generation in Language Preference of Generator We also provide an example of generated answers in different languages with a generator, GPT-4o-mini as shown in Table 13. The
<table>
  <tr>
    <th>Role</th>
    <th>Content</th>
  </tr>
  <tr>
    <td><b>System Message</b></td>
    <td>You are a highly capable multilingual assistant.<br>Here are some reference documents: {top5_passages}<br>The user wants answers in multiple languages. Please follow these rules strictly:<br>1) Return your final answer as a valid JSON object.<br>2) The JSON object must contain <b>exactly</b> these keys: {TARGET_LANGUAGES}.<br>3) Each field’s value must be the answer written in that respective language.<br>4) Do not include any additional text outside the JSON (e.g. no Markdown or explanations).<br>5) Ensure it is valid JSON with correct format.</td>
  </tr>
  <tr>
    <td><b>User Message</b></td>
    <td>Question: {question}<br>Please provide the answers in JSON form for each of the following languages: {TARGET_LANGUAGES}.</td>
  </tr>
</table>

Table 8: Prompts used for measuring language preference of GPT-4o-mini in the mRAG pipeline.

preference score in the rightmost column of Table 13 indicates that the generator prefers the query language and Latin-script languages over other languages.

Unified Document of DKM-RAG Additionally, we provide a sample of \( P_{translated} \) and \( P_{refined} \) obtained via our proposed DKM-RAG framework in Table 14. This example illustrates how the crucial answer component, “the executive branch”, which is not apparent from the translated passage alone, emerges through the model’s internal knowledge. Consequently, this shows that DKM-RAG can effectively leverage additional knowledge sources that is not included in the translated passage to achieve better performance.

G.1 Failure Case

MLRS We present a failure case of the MLRS metric in Table 15. Due to the difficulty of translating documents in low-resource languages, repetitive phrases such as *Changing the line-up* appear in the translated passage. This repetition causes the re-ranker to misinterpret the content, leading to an improvement in the rank even though the content is irrelevant.

DKM-RAG We also present a failure case of DKM-RAG in Table 16. The retriever retrieves an English document that is irrelevant to the query due to its language preference. Additionally, the LLM lacks relevant knowledge related to the query, resulting in a failed generation.

H Language Preference of Generators in average

We provide language preference of generators in terms of average as shown in Figure 14. Consistent with the result of each query language in Figure 3, the generator shows preferences for Latin-script languages. And GPT-4o-mini shows more consistent outputs than other generators. This is because it is a larger model than the others, providing more stable answers regardless of language preference. Between Llama and Aya, Aya produces slightly more consistent outputs, demonstrating its multilingual capability in handling diverse linguistic contexts.

I MLRS Analysis

We prove the effectiveness of our proposed language preference metric, MLRS by comparing language preference between MLRS score and the average document language ratio of retrieved documents for each dataset. As stated in Table 17, the tendency of average language ratio of retrieved documents and MLRS score is similar. To prove it, we also report Pearson and Spearman correlation coefficients and each p-value between them. Pearson value (0.98558) indicates a very strong positive linear correlation between the average MKQA language distribution values (mkqa_avg) and the MLRS (Preference) scores. The p-value (7.75e-10) is extremely small, showing that the probability of observing such a strong correlation by chance is almost negligible. In short, there is a statistically significant, nearly perfect linear relationship between these two sets of values. Similarly, the Spearman value (0.86264) also indicates a strong association, and the corresponding p-value (1.47e-4) confirms that this correlation is statistically significant. By these results, we prove that MLRS is efficient for measuring language preference of retriever.
<table>
  <tr>
    <th>Language</th>
    <th>Percentage</th>
  </tr>
  <tr>
    <td><b>EN</b></td>
    <td><b>89.70%</b></td>
  </tr>
  <tr>
    <td>Unknown</td>
    <td>8.38%</td>
  </tr>
  <tr>
    <td>DE</td>
    <td>0.17%</td>
  </tr>
  <tr>
    <td>FR</td>
    <td>0.15%</td>
  </tr>
  <tr>
    <td>SV</td>
    <td>0.15%</td>
  </tr>
  <tr>
    <td><b>ES</b></td>
    <td><b>0.13%</b></td>
  </tr>
  <tr>
    <td>ZH</td>
    <td>0.15%</td>
  </tr>
  <tr>
    <td>RU</td>
    <td>0.12%</td>
  </tr>
  <tr>
    <td>NL</td>
    <td>0.11%</td>
  </tr>
  <tr>
    <td>IT</td>
    <td>0.11%</td>
  </tr>
  <tr>
    <td>JP</td>
    <td>0.11%</td>
  </tr>
  <tr>
    <td>PL</td>
    <td>0.09%</td>
  </tr>
  <tr>
    <td>PT</td>
    <td>0.09%</td>
  </tr>
  <tr>
    <td>VI</td>
    <td>0.08%</td>
  </tr>
  <tr>
    <td>RO</td>
    <td>0.03%</td>
  </tr>
  <tr>
    <td>SR</td>
    <td>0.04%</td>
  </tr>
  <tr>
    <td>CA</td>
    <td>0.04%</td>
  </tr>
  <tr>
    <td><b>KO</b></td>
    <td><b>0.06%</b></td>
  </tr>
  <tr>
    <td>UK</td>
    <td>0.07%</td>
  </tr>
  <tr>
    <td>Other</td>
    <td>0.21%</td>
  </tr>
</table>

Table 9: Language distribution in the pre-training corpus of Llama-2. Unknown represents languages we cannot know because of closed-source access of model and other denotes other languages.

<table>
  <tr>
    <th>Dataset</th>
    <th>en</th>
    <th>ar</th>
    <th>es</th>
    <th>fi</th>
    <th>fr</th>
    <th>de</th>
    <th>ja</th>
    <th>it</th>
    <th>ko</th>
    <th>pt</th>
    <th>ru</th>
    <th>zh</th>
    <th>th</th>
  </tr>
  <tr>
    <th colspan="14">MKQA</th>
  </tr>
  <tr>
    <td># examples</td>
    <td>2827</td>
    <td>2827</td>
    <td>2827</td>
    <td>2827</td>
    <td>2827</td>
    <td>2827</td>
    <td>2827</td>
    <td>2827</td>
    <td>2827</td>
    <td>2827</td>
    <td>2827</td>
    <td>2827</td>
    <td>2827</td>
  </tr>
  <tr>
    <td>len question.</td>
    <td>43</td>
    <td>38</td>
    <td>48</td>
    <td>46</td>
    <td>49</td>
    <td>47</td>
    <td>26</td>
    <td>48</td>
    <td>22</td>
    <td>45</td>
    <td>42</td>
    <td>16</td>
    <td>41</td>
  </tr>
  <tr>
    <td>len answer.</td>
    <td>11</td>
    <td>10</td>
    <td>11</td>
    <td>11</td>
    <td>11</td>
    <td>11</td>
    <td>8</td>
    <td>11</td>
    <td>6</td>
    <td>11</td>
    <td>12</td>
    <td>6</td>
    <td>12</td>
  </tr>
  <tr>
    <th colspan="14">Wikipedia</th>
  </tr>
  <tr>
    <td># ex. (M)</td>
    <td>25</td>
    <td>3.3</td>
    <td>10</td>
    <td>1.5</td>
    <td>13</td>
    <td>14</td>
    <td>27</td>
    <td>8.2</td>
    <td>1.6</td>
    <td>4.7</td>
    <td>8.6</td>
    <td>11</td>
    <td>3.7</td>
  </tr>
  <tr>
    <td>len passage.</td>
    <td>624</td>
    <td>585</td>
    <td>619</td>
    <td>833</td>
    <td>627</td>
    <td>720</td>
    <td>208</td>
    <td>650</td>
    <td>431</td>
    <td>619</td>
    <td>721</td>
    <td>206</td>
    <td>217</td>
  </tr>
</table>

Table 10: Statistics of the datasets used in our experiments. MKQA Number of examples and median lengths of questions and answers (in Unicode characters). Wikipedia: Number of passages (in millions) and their median lengths.
<table>
  <tr>
    <th rowspan="2">Query Lang.</th>
    <th rowspan="2">Encoder</th>
    <th colspan="1">\( L_q = L_d \)</th>
    <th colspan="5">\( L_q \neq L_d \)</th>
  </tr>
  <tr>
    <th>ar</th>
    <th>fi</th>
    <th>de</th>
    <th>ru</th>
    <th>th</th>
  </tr>
  <tr>
    <td rowspan="3">ar</td>
    <td>bge-m3</td>
    <td>40.39</td>
    <td>–</td>
    <td>34.10 (<u>-6.29</u>)</td>
    <td>35.91 (<u>-4.48</u>)</td>
    <td>
36.22
 (<u>-4.17</u>)</td>
    <td>33.59 (<u>-6.80</u>)</td>
  </tr>
  <tr>
    <td>p-mMiniLM</td>
    <td>41.25</td>
    <td>–</td>
    <td>34.90 (<u>-6.35</u>)</td>
    <td>36.58 (<u>-4.67</u>)</td>
    <td>
37.13
 (<u>-4.12</u>)</td>
    <td>34.46 (<u>-6.79</u>)</td>
  </tr>
  <tr>
    <td>p-mMpNet</td>
    <td>41.34</td>
    <td>–</td>
    <td>34.64 (<u>-6.70</u>)</td>
    <td>36.34 (<u>-5.00</u>)</td>
    <td>
36.87
 (<u>-4.47</u>)</td>
    <td>34.36 (<u>-6.98</u>)</td>
  </tr>
  <tr>
    <td rowspan="3">fi</td>
    <td>bge-m3</td>
    <td>36.65</td>
    <td>33.47 (<u>-3.18</u>)</td>
    <td>–</td>
    <td>36.33 (<u>-0.32</u>)</td>
    <td>35.42 (<u>-1.23</u>)</td>
    <td>33.07 (<u>-3.58</u>)</td>
  </tr>
  <tr>
    <td>p-mMiniLM</td>
    <td>37.37</td>
    <td>34.60 (<u>-2.77</u>)</td>
    <td>–</td>
    <td>
37.14
 (<u>-0.23</u>)</td>
    <td>36.48 (<u>-0.89</u>)</td>
    <td>34.12 (<u>-3.25</u>)</td>
  </tr>
  <tr>
    <td>p-mMpNet</td>
    <td>37.27</td>
    <td>34.41 (<u>-2.86</u>)</td>
    <td>–</td>
    <td>
36.92
 (<u>-0.35</u>)</td>
    <td>36.28 (<u>-0.99</u>)</td>
    <td>34.12 (<u>-3.15</u>)</td>
  </tr>
  <tr>
    <td rowspan="3">de</td>
    <td>bge-m3</td>
    <td>39.81</td>
    <td>33.21 (<u>-6.60</u>)</td>
    <td>34.16 (<u>-5.65</u>)</td>
    <td>–</td>
    <td>
34.63
 (<u>+5.18</u>)</td>
    <td>32.95 (<u>-6.86</u>)</td>
  </tr>
  <tr>
    <td>p-mMiniLM</td>
    <td>40.80</td>
    <td>34.62 (<u>-6.18</u>)</td>
    <td>35.25 (<u>-5.55</u>)</td>
    <td>–</td>
    <td>
35.94
 (<u>+4.86</u>)</td>
    <td>34.18 (<u>-6.62</u>)</td>
  </tr>
  <tr>
    <td>p-mMpNet</td>
    <td>40.92</td>
    <td>34.81 (<u>-6.11</u>)</td>
    <td>35.33 (<u>-5.59</u>)</td>
    <td>–</td>
    <td>
36.13
 (<u>+4.79</u>)</td>
    <td>34.37 (<u>-6.55</u>)</td>
  </tr>
  <tr>
    <td rowspan="3">ru</td>
    <td>bge-m3</td>
    <td>45.05</td>
    <td>33.84 (<u>-11.21</u>)</td>
    <td>34.20 (<u>-10.85</u>)</td>
    <td>35.63 (<u>-9.42</u>)</td>
    <td>–</td>
    <td>33.24 (<u>-11.81</u>)</td>
  </tr>
  <tr>
    <td>p-mMiniLM</td>
    <td>46.08</td>
    <td>34.85 (<u>-11.23</u>)</td>
    <td>35.18 (<u>-10.90</u>)</td>
    <td>
36.73
 (<u>-9.35</u>)</td>
    <td>–</td>
    <td>34.23 (<u>-11.85</u>)</td>
  </tr>
  <tr>
    <td>p-mMpNet</td>
    <td>45.82</td>
    <td>34.63 (<u>-11.19</u>)</td>
    <td>34.83 (<u>-10.99</u>)</td>
    <td>
36.28
 (<u>-9.54</u>)</td>
    <td>–</td>
    <td>34.12 (<u>-11.70</u>)</td>
  </tr>
  <tr>
    <td rowspan="3">th</td>
    <td>bge-m3</td>
    <td>34.52</td>
    <td>33.68 (<u>-0.84</u>)</td>
    <td>34.11 (<u>-0.41</u>)</td>
    <td>
35.99
 (<u>+1.47</u>)</td>
    <td>35.60 (<u>+1.08</u>)</td>
    <td>–</td>
  </tr>
  <tr>
    <td>p-mMiniLM</td>
    <td>35.38</td>
    <td>34.65 (<u>-0.73</u>)</td>
    <td>34.77 (<u>-0.61</u>)</td>
    <td>
36.63
 (<u>+1.25</u>)</td>
    <td>36.40 (<u>+1.02</u>)</td>
    <td>–</td>
  </tr>
  <tr>
    <td>p-mMpNet</td>
    <td>34.73</td>
    <td>34.10 (<u>-0.63</u>)</td>
    <td>34.14 (<u>-0.59</u>)</td>
    <td>
36.08
 (<u>+1.35</u>)</td>
    <td>35.84 (<u>+1.11</u>)</td>
    <td>–</td>
  </tr>
</table>

Table 11: Language preference measured by MLRS with various re-ranking encoders for various query and document language combinations in a multilingual retriever. The \( L_q = L_d \) column reports the diagonal scores where the query language matches the translated document language, while the remaining columns represent cross-lingual scenarios (i.e., where the query language differs from the document language). Scores in parentheses indicate the difference from the diagonal value (positive for an improvement, negative for a decline). The highest score for each row is highlighted in bold, and the second highest is underlined.

<table>
  <tr>
    <th>Field</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>query</td>
    <td>영국 캐리비안에 언제 노예제가 폐지됐나요? (When was slavery abolished in the British Caribbean?)</td>
  </tr>
  <tr>
    <td>gold answer</td>
    <td>1834-08-01</td>
  </tr>
  <tr>
    <td>doc id</td>
    <td>kilt-100w_6947054 (English)</td>
  </tr>
  <tr>
    <td>r<sub>d</sub><sup>att</sup></td>
    <td>34</td>
  </tr>
  <tr>
    <td>r<sub>d</sub><sup>re-rank</sup></td>
    <td>2</td>
  </tr>
  <tr>
    <td>d (content)</td>
    <td><i>History of the Caribbean. Empire remained slaves, however, until Britain passed the Slavery Abolition Act in 1833. When the Slavery Abolition Act came into force in 1834, roughly 700,000 slaves in the British West Indies immediately became free; other enslaved workers were freed several years later after a period of forced apprenticeship. Slavery was abolished in the Dutch Empire in 1814. Spain abolished slavery in its empire in 1811, with the exceptions of Cuba, Puerto Rico, and Santo Domingo; Spain ended the slave trade to these colonies in 1817, after being paid £400,000 by Britain. Slavery itself was not abolished in Cuba until 1886.</i></td>
  </tr>
  <tr>
    <td>d (translated)</td>
    <td>1834년 노예제 폐지법이 시행되자, 영국 서인도 제도에서 약 700,000명의 노예가 즉시 해방되었고, 다른 노예 노동자들은 강제 연습 생활을 한 후 몇 년 후에 해방되었다. 1814년 네덜란드 제국에서 노예제는 폐지되었다. 1811년 스페인은 쿠바, 푸에르토리코, 산토 도밍고를 제외하고는 제국에서 노예제를 폐지했다. 1817년 영국이 400만원을 지불한 후 스페인은 이들 식민지에서의 노예 무역을 종식시켰다. 노예제는 1886년까지 쿠바에서 폐지되지 않았다.</td>
  </tr>
</table>

Table 12: An example of an improved MLRS case. After translating the document into Korean, its rank improved from 34 to 2, illustrating language preference of retriever.
<table>
  <tr>
    <th>Language</th>
    <th>Answer</th>
    <th>Preference Score</th>
  </tr>
  <tr>
    <td>en</td>
    <td>Low pressure is associated with warm air rising.</td>
    <td>0.9179</td>
  </tr>
  <tr>
    <td>ko</td>
    <td>따뜻한 공기가 상승하는 것과 관련된 공기 압력은 저압입니다.</td>
    <td>0.9060</td>
  </tr>
  <tr>
    <td>zh</td>
    <td>暖空上升相的空力是低 °</td>
    <td>0.8853</td>
  </tr>
  <tr>
    <td>fr</td>
    <td>La pression basse est associée à l’ascension de l’air chaud.</td>
    <td>0.9231</td>
  </tr>
  <tr>
    <td>ja</td>
    <td>暖かい空が上昇することに連するは低です °</td>
    <td>0.9187</td>
  </tr>
  <tr>
    <td>it</td>
    <td>La bassa pressione è associata all’aria calda che sale.</td>
    <td>0.9256</td>
  </tr>
  <tr>
    <td>pt</td>
    <td>A pressão baixa está associada ao ar quente que sobe.</td>
    <td>0.9316</td>
  </tr>
  <tr>
    <td>es</td>
    <td>La presión baja está asociada con el aire caliente que asciende.</td>
    <td>0.9317</td>
  </tr>
</table>

Table 13: An example of generated answers in different languages with gpt-4o-mini. Also, we report the average similarity score between each pair of answers.

<table>
  <tr>
    <th>Field</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>query</td>
    <td><i>in the united states the president is the head of which branch of government?</i></td>
  </tr>
  <tr>
    <td>gold answer</td>
    <td>the executive branch</td>
  </tr>
  <tr>
    <td>doc id</td>
    <td>kilt-100w_5089743 (English)</td>
  </tr>
  <tr>
    <td>\( P_{\text{translated}} \)</td>
    <td><i>President of the United States. President of the United States (POTUS) is the head of state and head of government of the United States of America. The president is the commander-in-chief of the United States Armed Forces. In contemporary times, the president is looked upon as one of the world’s most powerful political figures as the leader of the only remaining global superpower. The role includes responsibility for the world’s most expensive military, which has the second largest nuclear arsenal. The president also leads the nation with the largest economy.</i></td>
  </tr>
  <tr>
    <td>\( P_{\text{refined}} \)</td>
    <td><i>The president of the United States is the head of the <span style="color:red;">executive branch</span> of the federal government. The president directs the <span style="color:red;">executive branch</span> and is the commander-in-chief of the United States Armed Forces/ In contemporary times, the president is looked upon as one of the world’s most powerful political figures as the leader of the only remaining global superpower. The role includes responsibility for the world’s most expensive military, which has the second-largest nuclear arsenal. The president also leads the nation with the largest economy.</i></td>
  </tr>
</table>

Table 14: A DKM-RAG case study illustrating how \( P_{\text{translated}} \) and \( P_{\text{refined}} \) correspond to the retrieved passage (translated into the query language) and the rewritten passage leveraging parametric knowledge, respectively. The overlap with the gold answer is highlighted in red.
<table>
  <tr>
    <th>Field</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>query</td>
    <td>연속으로 가장 많은 자유투 기록 (who holds the record for most free throws made in a row)</td>
  </tr>
  <tr>
    <td>gold answer</td>
    <td>톰 설정</td>
  </tr>
  <tr>
    <td>doc id</td>
    <td>wiki-100w-ja_8993041</td>
  </tr>
  <tr>
    <td>P<sub>raw</sub></td>
    <td>31</td>
  </tr>
  <tr>
    <td>P<sub>rank</sub></td>
    <td>5</td>
  </tr>
  <tr>
    <td>d (content)</td>
    <td>' 맥도채소를 변 change ${NEWLINE} 記 ${NEWLINE} イニング最多連 四球 :5 (日本記) ia 1946年4月29日 ${NEWLINE} 第一年第2球で勝利 :1948年で史上3人目に ベイ最多失点:14（セレグ記） 1950年6月7日 ${NEWLINE} 背番'...</td>
  </tr>
  <tr>
    <td>d (translated)</td>
    <td>Changing the line-up, Changing the line-up, Changing the line-up, Changing the line-up, Changing the line-up, Changing the line-up, Changing the line-up, Changing the line-up, Changing the line-up, Changing the line-up, Changing the line-up, Changing the line-up, (...)</td>
  </tr>
</table>

Table 15: A failure case of MLRS because bad translation quality due to difficulty in translating low-resource language.

<table>
  <tr>
    <th>Field</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>query</td>
    <td>who is the first british governor in india?</td>
  </tr>
  <tr>
    <td>gold answer</td>
    <td>William Bentinck, Warren Hastings</td>
  </tr>
  <tr>
    <td>doc id</td>
    <td>kilt-100w_10189364 (English)</td>
  </tr>
  <tr>
    <td>P<sub>raw</sub></td>
    <td>Sir John Child, 1st Baronet. Sir John Child, 1st Baronet (died 1690) was a governor of Bombay, and de facto (although not officially) the first governor-general of the British settlements in India. Born in London, Child was sent as a young boy to his uncle, the chief of the factory at Rajapour. On 27 October 1681, he was appointed chief of the East India Company's affairs at Surat and Bombay, while at the same time his namesake—stated to be unrelated by the Oxford Dictionary of National Biography—Sir Josiah Child, was governor of the company at home. The two</td>
  </tr>
  <tr>
    <td>P<sub>refined</sub></td>
    <td>Sir John Child, 1st Baronet was the first governor of Bombay, and de facto (although not officially) the first governor-general of the British settlements in India. He was born in London but later moved to India where he became involved in the East India Company’s affairs. On 27th October 1681, he took up the position of Chief of the East India Company’s affairs at both Surat and Bombay. This made him the first British Governor in ….</td>
  </tr>
</table>

Table 16: A failure case of DKM-RAG because of preference of retriever so that high-resource but irrelevant document is retrieved.
<table>
  <tr>
    <th></th>
    <th>en</th>
    <th>ko</th>
    <th>ar</th>
    <th>zh</th>
    <th>fi</th>
    <th>fr</th>
    <th>de</th>
    <th>ja</th>
    <th>it</th>
    <th>pt</th>
    <th>ru</th>
    <th>es</th>
    <th>th</th>
  </tr>
  <tr>
    <td>mkqa_en</td>
    <td>44.12</td>
    <td>1.60</td>
    <td>1.19</td>
    <td>1.30</td>
    <td>2.54</td>
    <td>10.03</td>
    <td>6.90</td>
    <td>1.44</td>
    <td>8.32</td>
    <td>7.67</td>
    <td>4.85</td>
    <td>9.90</td>
    <td>0.13</td>
  </tr>
  <tr>
    <td>mkqa_ko</td>
    <td>23.07</td>
    <td>17.35</td>
    <td>1.99</td>
    <td>4.81</td>
    <td>2.04</td>
    <td>7.90</td>
    <td>5.96</td>
    <td>10.36</td>
    <td>6.16</td>
    <td>5.06</td>
    <td>6.85</td>
    <td>6.85</td>
    <td>1.58</td>
  </tr>
  <tr>
    <td>mkqa_ar</td>
    <td>24.93</td>
    <td>3.30</td>
    <td>15.29</td>
    <td>4.07</td>
    <td>2.10</td>
    <td>8.30</td>
    <td>6.53</td>
    <td>6.64</td>
    <td>6.80</td>
    <td>5.71</td>
    <td>7.78</td>
    <td>7.65</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>mkqa_zh</td>
    <td>24.70</td>
    <td>3.17</td>
    <td>1.76</td>
    <td>23.22</td>
    <td>2.01</td>
    <td>7.47</td>
    <td>6.17</td>
    <td>6.27</td>
    <td>6.08</td>
    <td>5.24</td>
    <td>6.37</td>
    <td>7.27</td>
    <td>0.27</td>
  </tr>
  <tr>
    <td>mkqa_fi</td>
    <td>30.32</td>
    <td>2.27</td>
    <td>1.63</td>
    <td>2.33</td>
    <td>7.92</td>
    <td>11.11</td>
    <td>8.20</td>
    <td>3.78</td>
    <td>8.77</td>
    <td>7.18</td>
    <td>6.51</td>
    <td>9.42</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>mkqa_fr</td>
    <td>29.90</td>
    <td>1.48</td>
    <td>1.25</td>
    <td>1.55</td>
    <td>2.50</td>
    <td>21.44</td>
    <td>6.96</td>
    <td>2.06</td>
    <td>9.40</td>
    <td>7.96</td>
    <td>4.77</td>
    <td>10.55</td>
    <td>0.19</td>
  </tr>
  <tr>
    <td>mkqa_de</td>
    <td>32.54</td>
    <td>1.46</td>
    <td>1.17</td>
    <td>1.44</td>
    <td>2.96</td>
    <td>11.40</td>
    <td>15.12</td>
    <td>1.89</td>
    <td>9.09</td>
    <td>7.69</td>
    <td>4.83</td>
    <td>10.17</td>
    <td>0.24</td>
  </tr>
  <tr>
    <td>mkqa_ja</td>
    <td>24.56</td>
    <td>4.80</td>
    <td>1.69</td>
    <td>3.99</td>
    <td>2.19</td>
    <td>7.97</td>
    <td>5.99</td>
    <td>22.55</td>
    <td>6.38</td>
    <td>5.66</td>
    <td>6.49</td>
    <td>7.45</td>
    <td>0.28</td>
  </tr>
  <tr>
    <td>mkqa_it</td>
    <td>28.72</td>
    <td>1.59</td>
    <td>1.30</td>
    <td>1.58</td>
    <td>2.52</td>
    <td>12.30</td>
    <td>6.97</td>
    <td>1.95</td>
    <td>17.46</td>
    <td>8.47</td>
    <td>5.26</td>
    <td>11.70</td>
    <td>0.17</td>
  </tr>
  <tr>
    <td>mkqa_pt</td>
    <td>28.82</td>
    <td>1.71</td>
    <td>1.40</td>
    <td>1.63</td>
    <td>2.60</td>
    <td>11.92</td>
    <td>6.74</td>
    <td>2.23</td>
    <td>10.24</td>
    <td>13.78</td>
    <td>5.38</td>
    <td>13.33</td>
    <td>0.24</td>
  </tr>
  <tr>
    <td>mkqa_ru</td>
    <td>27.02</td>
    <td>2.53</td>
    <td>1.92</td>
    <td>1.98</td>
    <td>2.45</td>
    <td>8.83</td>
    <td>6.44</td>
    <td>2.71</td>
    <td>7.36</td>
    <td>6.24</td>
    <td>23.83</td>
    <td>8.43</td>
    <td>0.26</td>
  </tr>
  <tr>
    <td>mkqa_es</td>
    <td>29.45</td>
    <td>1.73</td>
    <td>1.27</td>
    <td>1.60</td>
    <td>2.66</td>
    <td>11.85</td>
    <td>6.93</td>
    <td>1.83</td>
    <td>10.55</td>
    <td>9.33</td>
    <td>5.27</td>
    <td>17.36</td>
    <td>0.16</td>
  </tr>
  <tr>
    <td>mkqa_th</td>
    <td>32.39</td>
    <td>3.10</td>
    <td>2.10</td>
    <td>2.96</td>
    <td>2.53</td>
    <td>10.00</td>
    <td>7.40</td>
    <td>4.43</td>
    <td>8.06</td>
    <td>7.43</td>
    <td>6.80</td>
    <td>9.70</td>
    <td>3.10</td>
  </tr>
  <tr>
    <td><b>mkqa_avg</b></td>
    <td><b>29.27</b></td>
    <td><b>3.55</b></td>
    <td><b>2.61</b></td>
    <td><b>4.04</b></td>
    <td><b>2.85</b></td>
    <td><b>10.81</b></td>
    <td><b>7.41</b></td>
    <td><b>5.24</b></td>
    <td><b>8.82</b></td>
    <td><b>7.49</b></td>
    <td><b>7.31</b></td>
    <td><b>9.98</b></td>
    <td><b>0.62</b></td>
  </tr>
  <tr>
    <td><b>MLRS (Preference)</b></td>
    <td><b>47.70</b></td>
    <td><b>35.47</b></td>
    <td><b>35.59</b></td>
    <td><b>35.90</b></td>
    <td><b>35.13</b></td>
    <td><b>37.94</b></td>
    <td><b>37.20</b></td>
    <td><b>37.59</b></td>
    <td><b>37.66</b></td>
    <td><b>37.15</b></td>
    <td><b>37.99</b></td>
    <td><b>37.97</b></td>
    <td><b>34.09</b></td>
  </tr>
</table>

Pearson correlation coefficient: 0.98558 (p-value: 7.75e-10)
Spearman correlation coefficient: 0.86264 (p-value: 1.47e-4)

Table 17: Language distribution ratios of documents retrieved from datasets composed of each query language. The table lists the raw MKQA language distribution values (without the percent sign) for each dataset. The row mkqa_avg shows the average distribution across all MKQA datasets for each language, while the row MLRS (Preference) provides the corresponding MLRS scores. Additionally, we report Pearson and Spearman correlation coefficients between MLRS and mkqa_avg.
<table>
  <tr>
    <th></th>
    <th>en</th>
    <th>ko</th>
    <th>zh</th>
    <th>fr</th>
    <th>ja</th>
    <th>it</th>
    <th>pt</th>
    <th>es</th>
  </tr>
  <tr>
    <th>en</th>
    <td>1.0000</td>
    <td>0.7596</td>
    <td>0.7614</td>
    <td>0.7888</td>
    <td>0.7648</td>
    <td>0.7974</td>
    <td>0.7760</td>
    <td>0.7966</td>
  </tr>
  <tr>
    <th>ko</th>
    <td>0.7596</td>
    <td>1.0000</td>
    <td>0.7502</td>
    <td>0.7738</td>
    <td>0.7784</td>
    <td>0.7711</td>
    <td>0.7743</td>
    <td>0.7787</td>
  </tr>
  <tr>
    <th>zh</th>
    <td>0.7614</td>
    <td>0.7502</td>
    <td>1.0000</td>
    <td>0.7731</td>
    <td>0.7546</td>
    <td>0.7641</td>
    <td>0.7665</td>
    <td>0.7736</td>
  </tr>
  <tr>
    <th>fr</th>
    <td>0.7888</td>
    <td>0.7738</td>
    <td>0.7731</td>
    <td>1.0000</td>
    <td>0.7776</td>
    <td>0.7919</td>
    <td>0.8061</td>
    <td>0.8129</td>
  </tr>
  <tr>
    <th>ja</th>
    <td>0.7648</td>
    <td>0.7784</td>
    <td>0.7546</td>
    <td>0.7776</td>
    <td>1.0000</td>
    <td>0.7741</td>
    <td>0.7782</td>
    <td>0.7841</td>
  </tr>
  <tr>
    <th>it</th>
    <td>0.7974</td>
    <td>0.7711</td>
    <td>0.7641</td>
    <td>0.7919</td>
    <td>0.7741</td>
    <td>1.0000</td>
    <td>0.7884</td>
    <td>0.8015</td>
  </tr>
  <tr>
    <th>pt</th>
    <td>0.7760</td>
    <td>0.7743</td>
    <td>0.7665</td>
    <td>0.8061</td>
    <td>0.7782</td>
    <td>0.7884</td>
    <td>1.0000</td>
    <td>0.8068</td>
  </tr>
  <tr>
    <th>es</th>
    <td>0.7966</td>
    <td>0.7787</td>
    <td>0.7736</td>
    <td>0.8129</td>
    <td>0.7841</td>
    <td>0.8015</td>
    <td>0.8068</td>
    <td>1.0000</td>
  </tr>
</table>

Figure 5: LaBSE Similarity Matrix of aya-expanse-8b (en).
<table>
  <tr>
    <th></th>
    <th>en</th>
    <th>ko</th>
    <th>zh</th>
    <th>fr</th>
    <th>ja</th>
    <th>it</th>
    <th>pt</th>
    <th>es</th>
  </tr>
  <tr>
    <th>en</th>
    <td>1.0000</td>
    <td>0.7682</td>
    <td>0.7790</td>
    <td>0.7834</td>
    <td>0.7625</td>
    <td>0.7730</td>
    <td>0.7727</td>
    <td>0.7869</td>
  </tr>
  <tr>
    <th>ko</th>
    <td>0.7682</td>
    <td>1.0000</td>
    <td>0.7952</td>
    <td>0.7678</td>
    <td>0.7806</td>
    <td>0.7834</td>
    <td>0.7621</td>
    <td>0.7800</td>
  </tr>
  <tr>
    <th>zh</th>
    <td>0.7790</td>
    <td>0.7952</td>
    <td>1.0000</td>
    <td>0.7795</td>
    <td>0.7855</td>
    <td>0.8091</td>
    <td>0.7682</td>
    <td>0.7906</td>
  </tr>
  <tr>
    <th>fr</th>
    <td>0.7834</td>
    <td>0.7678</td>
    <td>0.7795</td>
    <td>1.0000</td>
    <td>0.7680</td>
    <td>0.7703</td>
    <td>0.7971</td>
    <td>0.7986</td>
  </tr>
  <tr>
    <th>ja</th>
    <td>0.7625</td>
    <td>0.7806</td>
    <td>0.7855</td>
    <td>0.7680</td>
    <td>1.0000</td>
    <td>0.7751</td>
    <td>0.7629</td>
    <td>0.7747</td>
  </tr>
  <tr>
    <th>it</th>
    <td>0.7730</td>
    <td>0.7834</td>
    <td>0.8091</td>
    <td>0.7703</td>
    <td>0.7751</td>
    <td>1.0000</td>
    <td>0.7650</td>
    <td>0.7855</td>
  </tr>
  <tr>
    <th>pt</th>
    <td>0.7727</td>
    <td>0.7621</td>
    <td>0.7682</td>
    <td>0.7971</td>
    <td>0.7629</td>
    <td>0.7650</td>
    <td>1.0000</td>
    <td>0.7900</td>
  </tr>
  <tr>
    <th>es</th>
    <td>0.7869</td>
    <td>0.7800</td>
    <td>0.7906</td>
    <td>0.7986</td>
    <td>0.7747</td>
    <td>0.7855</td>
    <td>0.7900</td>
    <td>1.0000</td>
  </tr>
</table>

Figure 6: LaBSE Similarity Matrix (zh) of aya-expanse-8b.
<table>
  <tr>
    <th></th>
    <th>en</th>
    <th>ko</th>
    <th>zh</th>
    <th>fr</th>
    <th>ja</th>
    <th>it</th>
    <th>pt</th>
    <th>es</th>
  </tr>
  <tr>
    <th>en</th>
    <td>1.0000</td>
    <td>0.7609</td>
    <td>0.7594</td>
    <td>0.7755</td>
    <td>0.7565</td>
    <td>0.7613</td>
    <td>0.7693</td>
    <td>0.7791</td>
  </tr>
  <tr>
    <th>ko</th>
    <td>0.7609</td>
    <td>1.0000</td>
    <td>0.7445</td>
    <td>0.7697</td>
    <td>0.7993</td>
    <td>0.8153</td>
    <td>0.7858</td>
    <td>0.7970</td>
  </tr>
  <tr>
    <th>zh</th>
    <td>0.7594</td>
    <td>0.7445</td>
    <td>1.0000</td>
    <td>0.7630</td>
    <td>0.7487</td>
    <td>0.7502</td>
    <td>0.7534</td>
    <td>0.7559</td>
  </tr>
  <tr>
    <th>fr</th>
    <td>0.7755</td>
    <td>0.7697</td>
    <td>0.7630</td>
    <td>1.0000</td>
    <td>0.7663</td>
    <td>0.7737</td>
    <td>0.7915</td>
    <td>0.7910</td>
  </tr>
  <tr>
    <th>ja</th>
    <td>0.7565</td>
    <td>0.7993</td>
    <td>0.7487</td>
    <td>0.7663</td>
    <td>1.0000</td>
    <td>0.7970</td>
    <td>0.7794</td>
    <td>0.7880</td>
  </tr>
  <tr>
    <th>it</th>
    <td>0.7613</td>
    <td>0.8153</td>
    <td>0.7502</td>
    <td>0.7737</td>
    <td>0.7970</td>
    <td>1.0000</td>
    <td>0.7922</td>
    <td>0.8000</td>
  </tr>
  <tr>
    <th>pt</th>
    <td>0.7693</td>
    <td>0.7858</td>
    <td>0.7534</td>
    <td>0.7915</td>
    <td>0.7794</td>
    <td>0.7922</td>
    <td>1.0000</td>
    <td>0.8014</td>
  </tr>
  <tr>
    <th>es</th>
    <td>0.7791</td>
    <td>0.7970</td>
    <td>0.7559</td>
    <td>0.7910</td>
    <td>0.7880</td>
    <td>0.8000</td>
    <td>0.8014</td>
    <td>1.0000</td>
  </tr>
</table>

Figure 7: LaBSE Similarity Matrix (ko) of aya-expanse-8b.
![Cross-Lingual Similarity Matrix (en)](page_324_671_1092_527.png)

Figure 8: LaBSE Similarity Matrix (en) of Llama-3.1-8B-instruct.
![Cross-Lingual Similarity Matrix (zh)](page_324_671_1092_563.png)

Figure 9: LaBSE Similarity Matrix (zh) of Llama-3.1-8B-instruct.
![Cross-Lingual Similarity Matrix (ko)](page_324_671_1092_563.png)

Figure 10: LaBSE Similarity Matrix (ko) of Llama-3.1-8B-instruct.
<table>
  <tr>
    <th></th>
    <th>en</th>
    <th>ko</th>
    <th>zh</th>
    <th>fr</th>
    <th>ja</th>
    <th>it</th>
    <th>pt</th>
    <th>es</th>
  </tr>
  <tr>
    <th>en</th>
    <td>1.0000</td>
    <td>0.8752</td>
    <td>0.8668</td>
    <td>0.9463</td>
    <td>0.8723</td>
    <td>0.9523</td>
    <td>0.9378</td>
    <td>0.9504</td>
  </tr>
  <tr>
    <th>ko</th>
    <td>0.8752</td>
    <td>1.0000</td>
    <td>0.8420</td>
    <td>0.8668</td>
    <td>0.8996</td>
    <td>0.8693</td>
    <td>0.8721</td>
    <td>0.8693</td>
  </tr>
  <tr>
    <th>zh</th>
    <td>0.8668</td>
    <td>0.8420</td>
    <td>1.0000</td>
    <td>0.8551</td>
    <td>0.8483</td>
    <td>0.8569</td>
    <td>0.8376</td>
    <td>0.8567</td>
  </tr>
  <tr>
    <th>fr</th>
    <td>0.9463</td>
    <td>0.8668</td>
    <td>0.8551</td>
    <td>1.0000</td>
    <td>0.8601</td>
    <td>0.9716</td>
    <td>0.9643</td>
    <td>0.9700</td>
  </tr>
  <tr>
    <th>ja</th>
    <td>0.8723</td>
    <td>0.8996</td>
    <td>0.8483</td>
    <td>0.8601</td>
    <td>1.0000</td>
    <td>0.8639</td>
    <td>0.8652</td>
    <td>0.8628</td>
  </tr>
  <tr>
    <th>it</th>
    <td>0.9523</td>
    <td>0.8693</td>
    <td>0.8569</td>
    <td>0.9716</td>
    <td>0.8639</td>
    <td>1.0000</td>
    <td>0.9675</td>
    <td>0.9749</td>
  </tr>
  <tr>
    <th>pt</th>
    <td>0.9378</td>
    <td>0.8721</td>
    <td>0.8376</td>
    <td>0.9643</td>
    <td>0.8652</td>
    <td>0.9675</td>
    <td>1.0000</td>
    <td>0.9746</td>
  </tr>
  <tr>
    <th>es</th>
    <td>0.9504</td>
    <td>0.8693</td>
    <td>0.8567</td>
    <td>0.9700</td>
    <td>0.8628</td>
    <td>0.9749</td>
    <td>0.9746</td>
    <td>1.0000</td>
  </tr>
</table>

Figure 11: LaBSE Similarity Matrix (en) of gpt-4o-mini.
<table>
  <tr>
    <th></th>
    <th>en</th>
    <th>ko</th>
    <th>zh</th>
    <th>fr</th>
    <th>ja</th>
    <th>it</th>
    <th>pt</th>
    <th>es</th>
  </tr>
  <tr>
    <th>en</th>
    <td>1.0000</td>
    <td>0.8820</td>
    <td>0.8655</td>
    <td>0.9432</td>
    <td>0.8797</td>
    <td>0.9502</td>
    <td>0.9350</td>
    <td>0.9484</td>
  </tr>
  <tr>
    <th>ko</th>
    <td>0.8820</td>
    <td>1.0000</td>
    <td>0.8433</td>
    <td>0.8745</td>
    <td>0.9053</td>
    <td>0.8766</td>
    <td>0.8792</td>
    <td>0.8751</td>
  </tr>
  <tr>
    <th>zh</th>
    <td>0.8655</td>
    <td>0.8433</td>
    <td>1.0000</td>
    <td>0.8565</td>
    <td>0.8453</td>
    <td>0.8575</td>
    <td>0.8365</td>
    <td>0.8575</td>
  </tr>
  <tr>
    <th>fr</th>
    <td>0.9432</td>
    <td>0.8745</td>
    <td>0.8565</td>
    <td>1.0000</td>
    <td>0.8686</td>
    <td>0.9722</td>
    <td>0.9661</td>
    <td>0.9716</td>
  </tr>
  <tr>
    <th>ja</th>
    <td>0.8797</td>
    <td>0.9053</td>
    <td>0.8453</td>
    <td>0.8686</td>
    <td>1.0000</td>
    <td>0.8719</td>
    <td>0.8735</td>
    <td>0.8693</td>
  </tr>
  <tr>
    <th>it</th>
    <td>0.9502</td>
    <td>0.8766</td>
    <td>0.8575</td>
    <td>0.9722</td>
    <td>0.8719</td>
    <td>1.0000</td>
    <td>0.9668</td>
    <td>0.9747</td>
  </tr>
  <tr>
    <th>pt</th>
    <td>0.9350</td>
    <td>0.8792</td>
    <td>0.8365</td>
    <td>0.9661</td>
    <td>0.8735</td>
    <td>0.9668</td>
    <td>1.0000</td>
    <td>0.9737</td>
  </tr>
  <tr>
    <th>es</th>
    <td>0.9484</td>
    <td>0.8751</td>
    <td>0.8575</td>
    <td>0.9716</td>
    <td>0.8693</td>
    <td>0.9747</td>
    <td>0.9737</td>
    <td>1.0000</td>
  </tr>
</table>

Figure 12: LaBSE Similarity Matrix (zh) of gpt-4o-mini.
<table>
  <tr>
    <th></th>
    <th>en</th>
    <th>ko</th>
    <th>zh</th>
    <th>fr</th>
    <th>ja</th>
    <th>it</th>
    <th>pt</th>
    <th>es</th>
  </tr>
  <tr>
    <th>en</th>
    <td>1.0000</td>
    <td>0.8890</td>
    <td>0.8854</td>
    <td>0.9480</td>
    <td>0.8891</td>
    <td>0.9538</td>
    <td>0.9392</td>
    <td>0.9531</td>
  </tr>
  <tr>
    <th>ko</th>
    <td>0.8890</td>
    <td>1.0000</td>
    <td>0.8589</td>
    <td>0.8833</td>
    <td>0.9155</td>
    <td>0.8850</td>
    <td>0.8884</td>
    <td>0.8844</td>
  </tr>
  <tr>
    <th>zh</th>
    <td>0.8854</td>
    <td>0.8589</td>
    <td>1.0000</td>
    <td>0.8770</td>
    <td>0.8633</td>
    <td>0.8788</td>
    <td>0.8583</td>
    <td>0.8794</td>
  </tr>
  <tr>
    <th>fr</th>
    <td>0.9480</td>
    <td>0.8833</td>
    <td>0.8770</td>
    <td>1.0000</td>
    <td>0.8816</td>
    <td>0.9758</td>
    <td>0.9695</td>
    <td>0.9753</td>
  </tr>
  <tr>
    <th>ja</th>
    <td>0.8891</td>
    <td>0.9155</td>
    <td>0.8633</td>
    <td>0.8816</td>
    <td>1.0000</td>
    <td>0.8841</td>
    <td>0.8868</td>
    <td>0.8827</td>
  </tr>
  <tr>
    <th>it</th>
    <td>0.9538</td>
    <td>0.8850</td>
    <td>0.8788</td>
    <td>0.9758</td>
    <td>0.8841</td>
    <td>1.0000</td>
    <td>0.9689</td>
    <td>0.9779</td>
  </tr>
  <tr>
    <th>pt</th>
    <td>0.9392</td>
    <td>0.8884</td>
    <td>0.8583</td>
    <td>0.9695</td>
    <td>0.8868</td>
    <td>0.9689</td>
    <td>1.0000</td>
    <td>0.9756</td>
  </tr>
  <tr>
    <th>es</th>
    <td>0.9531</td>
    <td>0.8844</td>
    <td>0.8794</td>
    <td>0.9753</td>
    <td>0.8827</td>
    <td>0.9779</td>
    <td>0.9756</td>
    <td>1.0000</td>
  </tr>
</table>

Figure 13: LaBSE Similarity Matrix (ko) of gpt-4o-mini.
![Line graph showing Generator Language Preference (avg) for llama, aya, and gpt across en, ko, zh, fr, ja, it, pt, es.](page_324_654_1097_682.png)

Figure 14: Average Generator Preference for three query languages: en, zh, ko.

Reviewer paper summaries: This paper investigates language preference issues in multilingual Retrieval-Augmented Generation (mRAG) systems. It introduces MultiLingualRank (MLR), a novel metric that quantifies retriever-level language bias by measuring ranking shifts after translating non-query-language passages. Experiments reveal that retrievers strongly favor high-resource and query-language documents, though this does not consistently improve generation quality. Additionally, generators exhibit a bias toward the query language or Latin scripts, leading to inconsistent outputs. To address these challenges, the paper proposes Dual Knowledge Multilingual RAG (DKM-RAG), a framework that combines translated multilingual passages with model knowledge. DKM-RAG effectively mitigates language biases and enhances generation performance across diverse linguistic settings.
Summary of strengths: 1. Novel Research Topic: This paper explores a novel and timely topic by focusing on language preferences in both retrieval and generation within multilingual Retrieval-Augmented Generation (mRAG) systems.

2. Comprehensive Experiments: The paper conducts extensive experiments, revealing that retrievers strongly favor high-resource and query-language documents, while generators exhibit a bias toward the query language or Latin scripts. The proposed DKM-RAG framework effectively mitigates these language biases and improves generation performance across diverse linguistic settings.

3. Valuable Findings for the Community: The insights presented in this paper are highly beneficial for researchers in multilingual RAG, as they can inform the development of more effective and balanced multilingual retrieval-augmented generation systems.
Summary of weaknesses: 1. Limited Novelty: Although the research task topic is novel, the paper introduces MultiLingualRank (MLR)—a metric to quantify retriever-level language bias—and proposes Dual Knowledge Multilingual RAG (DKM-RAG)—a framework that integrates translated multilingual passages with model knowledge—both contributions lack sufficient innovation. The metric is not rigorously justified, and the framework is overly simplistic and straightforward.

2. Insufficient Justification of MultiLingualRank (MLR): In Section 2, the authors define MLR in three stages but fail to rigorously justify its effectiveness in evaluating ranking quality. Moreover, it lacks consideration of document relevance score. For example, in Figure 2, if the initial ranking is 7/20/36 and positions #1 and #3 are swapped during re-ranking, the Δ𝑟𝑑 values would be 0 (−29)/0/+29, resulting in MLR = 29/(6+19+35). If the positions remain unchanged (Δ𝑟𝑑 = 0/0/0), then MLR = 0/(6+19+35). In both cases, documents #1, #2, and #3 are all relevant, so the ranking performance should be considered equal. However, the MLR values differ, suggesting an inconsistency. In contrast, well-known metrics like NDCG offer a more principled evaluation by considering ideal ranking and document relevance score. NDCG remains consistent as long as the relevant documents are the same, even if their order changes slightly as shown in the above example.

3. Presentation Needs Improvement: The presentation of the paper requires refinement. The discussion of limitations in related work (e.g., lines 067–070) is vague. While the authors criticize prior work for focusing on a limited set of languages, their own experiments are also conducted on only eight selected languages, without clearly explaining how their setup is more general. Additionally, the paper refers to "ranking dynamics of documents across languages" but fails to provide a clear definition or conceptual explanation of this term. The organization of the paper—following the sequence of MLR, Generator analysis, and DKM-RAG—makes it feel like a collection of three loosely connected sub-studies, rather than a coherent narrative with a clearly defined central contribution.
Comments, suggestions and typos: 1. Line 322: Please clarify the rationale for choosing LaBSE as the embedding model. It would be helpful to explain its advantages or relevance in the context of multilingual retrieval and how it impacts performance.

2. Line 190: The authors adopt NLLB-200-distilled-600M (Costa-jussà et al., 2022) as the translation model, but its translation quality on the MKQA and KILT NQ datasets across different languages is not reported, leaving uncertainty about the reliability of translated passages used in the experiments.

3. Line 160: The authors select a subset of 2.7K samples overlapping between MKQA and KILT NQ, but the criteria or methodology for sample selection is not explained, making it unclear whether the subset is representative or introduces any biases.
Overall assessment scores: 2.5