Retrieval-augmented generation in multilingual settings (2024)

Abstract: Retrieval-augmented generation (RAG) has recently emerged as a promising solution for incorporating up-to-date or domain-specific knowledge into large language models (LLMs) and improving LLM factuality, but is predominantly studied in English-only settings. In this work, we consider RAG in the multilingual setting (mRAG), i.e. with user queries and the datastore in 13 languages, and investigate which components and with which adjustments are needed to build a well-performing mRAG pipeline, that can be used as a strong baseline in future works. Our findings highlight that despite the availability of high-quality offthe-shelf multilingual retrievers and generators, task-specific prompt engineering is needed to enable generation in user languages. Moreover, current evaluation metrics need adjustments for multilingual setting, to account for variations in spelling named entities. The main limitations to be addressed in future works include frequent code-switching in non-Latin alphabet languages, occasional fluency errors, wrong reading of the provided documents, or irrelevant retrieval. We release the code for the resulting mRAG baseline pipeline at https: //github.com/naver/bergen1.

heading: 1 Introduction
text: Retrieval-augmented generation (RAG) (Ram et al., 2023) has recently emerged as a promising solution for incorporating up-to-date or domainspecific knowledge into large language models (LLMs) and improving LLM factuality, especially in knowledge-intensive tasks such as open-domain question answering or fact-checking. RAG augments user queries with relevant context retrieved from the Internet or a given collection and
∗Work done while at Naver Labs Europe. 1Documentation: https://github.com/naver/bergen/ blob/main/documentations/multilingual.md Correspondence to: [nadia.chirkova, vassilina.nikoulina]@naverlabs.com
177
then passes the result to an LLM to generate a knowledge-grounded response. Recent works focus on improving various components of the complex RAG pipeline, e.g. generator (Yoran et al., 2024) or search query processor (Ma et al., 2023), as well as addressing fragility of the RAG approach, e.g. filtering irrelevant retrieved context (Wang et al., 2023; Xu et al., 2023; Kim et al., 2024) or dynamically deciding for which user queries retrieval is actually needed (Jiang et al., 2023; Asai et al., 2024).
Unfortunately, all listed efforts are focusing on English as the data language in their experiments, i.e. the language of the user queries and of the knowledge datastore. In this work, we argue for the importance of considering multilingual settings in RAG experiments and advancing multilingual RAG (mRAG), as it has clear advantages for both English and non-English speakers. On the one side, enabling access to RAG advances for nonEnglish speakers requires testing the applicability of approaches proposed in the literature for non-English queries, and possibly developing special multilinguality-oriented RAG methodologies. On the other side, considering non-English knowledge datastores ensures access to local or culturespecific information for all future users of RAG models, as such information is often available only in non-English. In the similar way retrieving from English may be beneficial for non-English queries e.g. about US or British culture.
Enabling high-quality RAG in multilingual settings requires access to strong multilingual retrievers and generators, as well as high-quality multilingual evaluation. The retriever should be able to map queries in the user language to the documents in the same or different language. The generator should be able to generate fluently and correctly in the user language, but also to understand documents in various languages and to follow instructions specified in the prompt. While recent advances in natural language processing and information retrieval made appropriate candidate components available, the entire multilingual RAG pipeline was not evaluated in the literature before.
The main contribution of our work is (1) building a publicly available baseline mRAG pipeline, to foster research on multilingual RAG in a zeroshot setting, and (2) conducting an initial study of mRAG in open question answering with user queries and retrieval datastores in 13 languages.
We aim to answer the following research questions:
• does RAG bring same performance improvements in knowledge-intensive tasks in nonEnglish as in English?
• which components are needed for effective mRAG and which adaptations are required?
• what are the main limitations of the existing components that can be addressed in future work?
Our key findings can be summarized as follows:
• Retrieval: recent off-the-shelf multilingual retrievers and rerankers perform reasonably well in both cases when queries and documents are in the same or different language, and also handle well retrieval from multilingual datastores (Tables 1 and 7);
• Generation: achieving high performance across all languages requires a strong multilingually pretrained and tuned LLM, coupled with advanced prompting, e.g. translating prompts into user languages and instructing the LLM to generate responses in the user language (Tables 2, 5 and 6);
• Evaluation: evaluation metrics need adjustment to take into account the zero-shot scenario, e.g. variations in spelling named entities in cross-lingual settings (Table 3);
• The main limitations to be addressed in future works include frequent code-switching2
in non Latin alphabet languages, occasional fluency errors, wrong reading of the provided documents, or irrelevant retrieval (Table 8).

heading: 2 Related Work
text: Despite mRAG being not well studied in the literature, some of the individual components of the RAG pipeline were rather well developed for multilingual settings, e.g. multilingual retrievers and generator LLMs; we discuss them in Section 3.
The closest line of work to ours is multilingual open question answering (Asai et al., 2021b; Muller et al., 2022; Sorokin et al., 2022; Asai et al., 2022) defined as a the task of answering non-English questions from a large collection of multilingual
2Code-switching refers to inserting fragments in other languages when generating in a given language.
documents, as introduced in (Asai et al., 2021b). Those aforementioned works train task-specific models combining cross-lingual retrievers and multilingual generation models, e.g. with iterative extension of annotated data used in the CORA approach (Asai et al., 2021b). The key difference of our work is that we compose the mRAG system in a zero-shot manner, using off-the-shelf components without dedicated training. This approach, dominating nowadays in the literature, is enabled by recent advances in LLMs and retrieval and makes the system more robust and easy-to-extend. It’s important to note that our goal is not to outperform the mentioned models such as CORA, but to evaluate the state of the described zero-shot mRAG setting, understand its open problems, and provide an experimental ground for future development of mRAG.
Another related and orthogonal effort is (Thakur et al., 2024) which release a NoMIRACL dataset for evaluating LLM robustness in mRAG across 18 typologically diverse languages.

heading: 3 Multilingual RAG pipeline
text: The high-level illustration of the mRAG pipeline is presented in Figure 1. The input is represented by a user query q in language Lq. This could be an arbitrary user request to an LLM. Following the common practice of testing RAG systems on open-domain question answering, we assume q is an information-seeking question. The model is expected to output response r which correctly answers the given question. An important (and reasonable) expectation is that the model replies in the user language, i.e. r is written in Lq.
Step 1: retrieval. The first step in mRAG is retrieving context c relevant to the query q from the Internet or a particular collection C, using the retriever system R: c = R(q̃, C), q̃ = Q(q). Here Q denotes an optional query generation model which infers a search query q̃ from a user query c, e.g. it can be an LLM prompted to reformulate the query, or simply copying the user query q. Following a standard practice in testing RAG systems, we use Wikipedia as our collection C. In most of the experiments we assume monolingual C in language LC (English or user language), but we also experiment with retrieving from the multilingual C.
The retriever system R usually consists of two stages. The first stage ranker R1 encodes queries q and documents d ∈ C independently: hq =
R1(q̃) ∈ Rn, hd = R1(d) ∈ Rn, allowing to precompute document representations offline and enabling fast search over large collections, e.g. c̃ = top-Kd∈Ch T q hd, K denotes the number of retrieved documents. The second-stage reranker R2 processes a (small) subset c̃ of documents from C retrieved by R1 and encodes documents together with queries: hq,d = R2(q̃, d) ∈ R, enabling semantically richer representations and selecting k most relevant documents: c = top-kd∈c̃hq,d. Both R1 and R2 are often based on BERT-like models and trained on retrieval datasets such as MSMARCO (Nguyen et al., 2016). In our work we rely on retrievers and rerankers developed specifically for the multilingual setting.
Step 2: generation. The second stage of mRAG pipeline consists of generating a response r based on the user query q and retrieved relevant context c with a generator LLM: r = LLM(q, c). State-ofthe-art LLMs follow the wide-spread paradigm of pretraining a decoder-only Transformer model on a large set of unsupervised data and then tuning it for instruction following and alignment with user preferences. This second step of instruction tuning and alignment often introduces a template, representing formatting rules for passing data into the LLM. Template usually contains placeholders for user queries q, model responses r and also for a system prompt, which is put in the beginning of the template and describes the task / role for the LLM. A simplest example of the system prompt is “You are a helpful assistant.”. In our work we study several generator LLMs and experiment extensively with various prompting strategies for mRAG.
Below we describe how we instantiate different components of our mRAG pipeline.
Multilingual retrievers. The described problem setting requires strong monolingual and crosslingual rankers and rerankers, for cases when Lq = LC and Lq ̸= LC , correspondingly. We pick a strong recently released and publicly available BGE-m33 (Chen et al., 2024) which provides all listed functionalities and includes all languages we consider in its training data. We also consider a baseline including query translation, where query generator Q translates q from Lq to LC . We employ the NLLB-600M translation model4 (Team
3Retriever: https://huggingface.co/BAAI/bge-m3 (dense version). Reranker: https://huggingface.co/ BAAI/bge-reranker-v2-m3.
4https://huggingface.co/facebook/ nllb-200-distilled-600M
et al., 2022).
Multilingual generation. Most of current stateof-the-art LLMs are either English-centric or support a limited set of languages, possibly due to under-investigated effects of the "curse of multilinguality" for large models (Conneau et al., 2020), i.e. it is yet unclear how many languages LLMs can fit without hurting performance, or due to limited availability of multilingual instruction tuning and alignment datasets. At the same time, it was shown that even English-centric LLMs, which were pretrained and finetuned mostly on English data, may exhibit good multilingual capabilities due to the occasional presence of multilingual data in pretraining (Ye et al., 2023; Chirkova and Nikoulina, 2024). As such, we experiment with both strong English-centric and recent multilingual models. Among English-centric models we pick commonly-used LLaMA-2-7B-chat (Touvron et al., 2023) and state-of-the-art SOLAR-10.7B (Kim et al., 2023), and among multilingual models we pick Mixtral-8x7B (Jiang et al., 2024) and Command-R-35B5. All models were instructiontuned. Command-R-35B was developed with keep-
5https://huggingface.co/CohereForAI/ c4ai-command-r-v01
ing RAG application in mind and officially supports 11 languages6, including most of our considered languages, and also includes 13 more languages (incl. Russian) in pretraining but not instruction tuning. Mixtral-8x7B was pretrained on the multilingual data with 5 languages7, we use it’s instruction-tuned version.
System prompt. In our preliminary experiments we noticed that models sometimes reply in English even for non-English user queries. This is not an expected behavior and substantially reduces metrics, calculated over groundtruth answers in user languages. To tackle this, we study various strategies for defining the system prompt, e.g. including an explicit instruction to reply in the user language, see Table 2 for all the system prompts that we consider. Some strategies include translation of the prompts into user languages: we used Google Translate and asked native or fluent speakers of considered languages, employed in our research laboratory, to check and correct the generated translations8.
Multilingual QA datasets. We follow (Asai et al., 2021b) and use MKQA (Longpre et al., 2021) and XOR-TyDi QA (Asai et al., 2021a) datasets for evaluation in our experiments. MKQA consists of 10k examples from the Natural Questions (NQ) dataset (Kwiatkowski et al., 2019), translated into
6Command-R official languages: Arabic, Brazilian Portuguese, English, French, German, Italian, Japanese, Korean, Simplified Chinese, and Spanish
7Mixtral official languages: English, French, Italian, German, and Spanish
8Issues raised when controlling prompt translation include (1) wrong semantics of the assistant’s task in translations which is highly undesirable; (2) choosing between formal and informal register – we chose informal style for all cases; (3) complications with translating field-specific terms such as “named entities”; (4) absence of the direct translation of the phrase "You are a helpful assistant" in some languages.
25 languages. This dataset is therefore parallel between languages and grounds knowledge primarily in English Wikipedia. In our experiments we select a subset of 2.7K samples, overlapping between MKQA and KILT NQ datasets9, thus recovering relevant documents information from KILT NQ. XOR-TyDi QA comprises 40K informationseeking questions in 7 languages (of which we us 3K validation questions) and grounds questions in Wikipedia in the same language as the question or in English. To provide English for comparison, we include results for English on the TyDi QA dataset (Clark et al., 2020). Though both datasets come with oracle contexts, questions are contextindependent, meaning that they can be understood without context and the answers are “universal” and not specific to the provided contexts. This property is not held for many other multilingual QA datasets, e.g. some reading comprehension datasets.
Statistics of the used datasets (number of examples, average lengths) are presented in Table 4. We select a diverse set of user languages (ULs) to experiment with, including Latin and non Latin script ones (see Table 1).
Evaluation. Both MKQA and XOR-TyDi QA contain mostly short answer labels, e.g. a person name, a date etc. Following common RAG evaluation practice and Asai et al. (2021b), we use lexical matching metrics, i.e. whether ground-truth or its tokens are contained in the generated answer. One key difference with (Asai et al., 2021b) is that we generate answers with off-the-shelf LLMs in a zero-shot setting, which tend to produce verbose answers, mostly consisting of full sentences rather than single-phrase outputs. While this is not a weakness, it requires adjusting metrics for reliable evaluation, e.g. prioritize recall over precision and measure which percentage of tokens contained in the ground-truth label are contained in the response generated by the model.
In our preliminary experiments we noticed a pattern arising sometimes in the scenario with crosslingual retrieval, when models generate a transliteration of named entities in other languages different from the one contained in the ground-truth label. This is again not a weakness of the system, but needs to be accounted in the evaluation metric. Since word-level matching fails to capture similarity in the described case, we propose to evaluate
9NQ dataset in KILT benchmark available at https:// huggingface.co/datasets/kilt_tasks
recall on character n-gram level. We first split ground-truth labels into tokens, extract all character 3-grams from each token and evaluate which percentage of such ngrams is present in the modelgenerated response, see Table 3 for illustration.
In addition to the task metric, we also control the correct language rate, CLR, which measures which percentage of model outputs are written in the user language. We detect languages using fasttext library (Joulin et al., 2017, 2016) and its lid.176.bin model10. Due to high erroneous level of language identification for short sequences, we only evaluate the CRL metric for model responses longer than 20 characters.

heading: 4 Experimental details
text: Retrieval. We follow (Asai et al., 2021b) and (Karpukhin et al., 2020) and construct passages by splitting Wikipedia article into chunks of 100 words (or 100 Unicode characters for non whitespace separated languages, namely Chinese, Japanese, and Thai) and prepending the article title to each chunk. In most of the experiments we retrieve either from English Wikipedia (KILT version11) or Wikipedia in the user language12, but we also experiment with retrieving from concatenation of two mentioned Wikipedias and from Wikipedia in all considered languages. For each question in the evaluation data, we retrieve 50 relevant passages and pass them to the reranker to select top-5 relevant ones which will be inserted in the LLM context during generation.
Generation. We use greedy decoding, limit generation to maximum 128 new tokens and run all experiments with model quantized into int4.
Evaluation. We rely on the commonly-used SQUAD evaluation script13, but use it on the character 3-gram level, as discussed in Section 3 and illustrated in Table 3. We preprocess both groundtruth labels and predicted responses by lowercasing them, removing punctuation and articles.
10https://fasttext.cc/docs/en/ language-identification.html
11https://huggingface.co/datasets/facebook/ kilt_wikipedia
12https://huggingface.co/datasets/wikimedia/ wikipedia
13https://github.com/allenai/bi-att-flow/blob/ master/squad/evaluate-v1.1.py

heading: 5 Results and discussion
text: Table 1 summarizes the results across different languages on MKQA and XOR TyDi QA datasets. We observe a high performance improvement brought by RAG for all languages, but in many cases there is an important gap in performance in English and
non-English. In what follows we present multiple ablation studies to demonstrate steps needed to achieve shown results, to better understand the reasons behind the gap with English, and identify future research directions. We study the effect of the system prompt, generator model, retrieval system and language. We run ablations on three languages: French, Korean, and Russian.
Prompting strategy: importance of translating the system prompt into target languages and specifying the desired language of the response. Table 5 summarizes an impact of prompt formulation (defined in Table 2) on RAG performance with English-centric SOLAR-10.7B and multilingual Command-R-35B models.
The left part reporting Correct Language
Rate (CLR) allows us to assess how often the model replies in the user language. Due to multilingual pretraining and instruction tuning, Command-R-35B, equipped with the default system prompt ("Reply short (EN)"), replies in the user language in most, but not all, cases. Importantly, it gets "distracted" by the English context when retrieving from English Wikipedia and replies in English for around 50% of non-English user queries. English-centric SOLAR-10.7B, provided with the default system prompt, also often replies in English. These results demonstrate the need for using more advanced language-related prompting strategies for both models.
Explicitly specifying an instruction to reply in the given user language, while keeping the system prompt itself in English ("+ reply in UL (EN)"), substantially alleviates the problem of generation in English and correspondingly increases recall, but still does not enable correct language rate (CRL) close to 100%. In Appendix Table 9, we also consider a more generic prompt with a "metainstruction" to reply in the same language as the input language (+ reply in same lang (EN)) and find that it leads to considerably lower CRL than explicit language specification.
The further improvement in CRL (and thus recall) for both models is enabled by translating the system prompt into user languages. With the system prompt which includes explicit specification to generate in the given user language and is also written in the user language, both models achieve CRL > 95% in most cases (except SOLAR-10.7B for Korean). Such an approach is however less convenient in practice, as it requires language expertise to control the quality of translating prompts (see footnote 8) and dynamic selection of the system prompt based on the user query. We believe that enabling multilingual LLMs to follow instructions within mixed-language prompts is an interesting research direction that would help to eliminate the need for the described ad-hoc prompting.
The high CLR is necessary but not sufficient for high overall performance, as LLMs may use code-switching and tend to insert English named entities in their responses in user languages. In Appendix Table 9 we attempt to alleviate this issue by augmenting the system prompt with an explicit instruction to write all named entities in ULs ("+NE in ULs"). While it does slightly improve character
3-gram recall for Command-R in many cases, it does not solve the issue fully. We believe that addressing the described code-switching problem is an important direction for future research.
Generator model: importance of using a strong multilingual base model. Table 6 compares four considered generator LLMs with and without retrieval. We find that Command-R-35B is the only model which consistently achieves high CLR and highest ranges of recall for all considered languages (with advanced prompts discussed above). Another considered multilingual-by-design model, Mixtral-8x7B, reaches consistently high CLR and recall only for French which was present in its pretraining. English-centric LLAMA-2-chat-7B most often replies in English. Interestingly, Englishcentric SOLAR-10.7B reaches high CLR and recall for French and Russian (with advanced prompts). This could be attributed to its strong capabilities in prompt understanding and accidental multilingual data present in pretraining.
Despite Command-R-35B being a leader model for non-English, its recall in English is much lower than of English-centric SOLAR-10.7B which is possibly due to the "curse of multilinguality" effect. This highlights the need for future models which would be fluent and accurate in both English and non-English.
Retrieval: high performance of off-the-shelf multilingual retrievers in the in-domain setting. In our work we rely on a strong multilingual retriever and reranker, BGE-m3, which was shown by its authors to outperform other approaches on multilingual retrieval benchmarks. In Table 7 we evaluate its performance in the cross-lingual setting (documents in English and user queries in nonEnglish), by comparing to the baselines involving query translation from user languages to English. We find that BGE-m3 outperforms a strong English model, SPLADE, used with translated queries. We note that BGE-m3 was trained on the datasets which also use Wikipedia as the document datastore, therefore in our experiments it is used in the in-domain setting. The retrieval performance in the multilingual setting with domain-shift is yet to be explored.
Which language to retrieve from: highest performance with retrieving from multilingual Wikipedia. Table 1 compares retrieval from English Wikipedia, Wikipedia in the user language,
their union, and also Wikipedia in all considered languages. In the latter two cases with run retrieval over the embeddings of passages in multiple languages, so that the selected passages may be also in multiple languages.
Comparing retrieval from English and user language, we observe different behavior on the two considered datasets. On the MKQA dataset, retrieval from English is more beneficial, which is expected since questions in MKQA were initially written by relying on the English Wikipedia and then translated into other languages. At the same time, XOR-TyDi QA includes questions grounded in both English and user languages (see statistics in Table 2, Longpre et al., 2021), and we observe that retrieval from Wikipedia in the user language is more beneficial.
Overall, we find that BGE-m3 also successfully manages to retrieve from the concatenated multilingual Wikipedia and thus dynamically choose the more appropriate datastore, often reaching performance higher than with any of the two monolingual Wikipedias.
Best performing configuration to be used as a strong baseline. Based on the previous experiments, we highlight our best configuration, including Command-R-35B generator, BGE-m3 retriever and reranker, the system prompt ‘Reply short in UL (UL)‘, and retrieval from the concatenation of Wikipedia in various languages.
Manual inspection of errors. To better analyze failure cases, we perform a manual analysis of pre-
dictions in French, Chinese, and Russian and report results in Table 8. We find that system improvements can be made at all steps, including retrieval, reading from the retrieved documents, addressing issues with code-switching and occasional fluency issues in non-English generation. Table 7 confirms gap in retrieval quality between English and nonEnglish. Many examples are characterized by different transliteration of named entities which we take into account in evaluation, by computing lexical match metrics on the character n-gram level. We underline that the possibility of various possible transliterations and code switching should be also kept in mind in the future development of evaluation metrics. Finally, we notice several issues with evaluation data, including ambiguous questions and incomplete or wrong labels, as well as typos or fluency errors in questions.

heading: 6 Conclusion
text: In this work we study RAG in multilingual settings and build a strong pipeline to be used as a baseline in future works. Better understanding of mRAG would enable reliable information access across different languages and cultures. We analyze an impact of each mRAG component impact on overall performance and provide guidelines and future research direction to further improve it.
Possible research directions include:
• The need for stronger multilingual LLMs and decoding strategies. Our study highlights multilingual generation as a weakest part of the mRAG pipeline, especially with mixed-language context. We show that even strongest available multilingual LLMs can get distracted by the language of the prompt, and require ad-hoc prompting to enable consistent generation in the user language. Even then, they are still prone to code-switching especially when writing named entities. We believe listed limitations could be addressed by including mixed-language examples in instruction tuning or by developing specific decoding strategies.
• LLM-based evaluation in multilingual settings. In our work we rely on the lexical matching-based metrics due to their transparency and interpretability. At the same time, recent works use LLM-based evaluation which captures better semantic similarities but
is currently underexplored in multilingual settings.
• Multi-domain multilingual retrieval. Current multilingual retrievers and rerankers are predominantly trained on Wikipedia-based data which could limit their applicability to other domains.
Limitations
Following common practice in RAG and as a first step in mRAG, we run evaluation on the open question answering task and with Wikipedia as the datastore. Important next steps include considering other tasks and domains.
Some of the standard practice in RAG which we left out of the scope of this study include query reformulation component and context postprocessing (e.g. filtering irrelevant passages). These components are less relevant for the question answering datasets we studied, but will be more relevant for other tasks, and should be included in future work.
We only considered single retriever and reranker model (Chen et al., 2024) since this is the strongest open-source multilingual retrieval system available at the moment of our work, covering many different languages withing a single model.
Ethics Statement
We do not anticipate negative societal impact from our work and on the reverse hope that it will help to broaden the accessibility of modern NLP to other languages.

heading: 7 Acknowledgments
text: We gratefully appreciate the help of Shuai Wang, Inyoung Kim, Salah Ait Mokhtar, Carlos Lassance, Beomseok Lee, Tomi Silander, and Riccardo Volpi.

heading: Future Work
text: In this work we study RAG in multilingual settings and build a strong pipeline to be used as a baseline in future works. Better understanding of mRAG would enable reliable information access across different languages and cultures. We analyze an impact of each mRAG component impact on overall performance and provide guidelines and future research direction to further improve it. Possible research directions include:
• The need for stronger multilingual LLMs and decoding strategies. Our study highlights multilingual generation as a weakest part of the mRAG pipeline, especially with mixed-language context. We show that even strongest available multilingual LLMs can get distracted by the language of the prompt, and require ad-hoc prompting to enable consistent generation in the user language. Even then, they are still prone to code-switching especially when writing named entities. We believe listed limitations could be addressed by including mixed-language examples in instruction tuning or by developing specific decoding strategies. • LLM-based evaluation in multilingual settings. In our work we rely on the lexical matching-based metrics due to their transparency and interpretability. At the same time, recent works use LLM-based evaluation which captures better semantic similarities but
is currently underexplored in multilingual settings. • Multi-domain multilingual retrieval. Current multilingual retrievers and rerankers are predominantly trained on Wikipedia-based data which could limit their applicability to other domains. Limitations
Following common practice in RAG and as a first step in mRAG, we run evaluation on the open question answering task and with Wikipedia as the datastore. Important next steps include considering other tasks and domains. Some of the standard practice in RAG which we left out of the scope of this study include query reformulation component and context postprocessing (e. g. filtering irrelevant passages). These components are less relevant for the question answering datasets we studied, but will be more relevant for other tasks, and should be included in future work. We only considered single retriever and reranker model (Chen et al. , 2024) since this is the strongest open-source multilingual retrieval system available at the moment of our work, covering many different languages withing a single model