Venue: NAACL 2025

# Mirage-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems

Nandan Thakur\({}^{1}\), Suleman Kazi\({}^{2}\), Ge Luo\({}^{2}\), Jimmy Lin\({}^{1}\), Amin Ahmad\({}^{2}\)

\({}^{1}\) University of Waterloo, Canada \({}^{2}\) Vectara, USA

{nandan.thakur,jimmylin}@uwaterloo.ca

{suleman,rogger,amin}@vectara.com

###### Abstract

Traditional retrieval-augmented generation (RAG) benchmarks evaluate systems using heuristic-based metrics, but these require human preferences as the ground truth for reference. In contrast, arena-based benchmarks, where systems compete against each other, require an expensive large language model (LLM) as a judge for a reliable evaluation. We present a simple efficient technique to combine the best of both worlds. The idea is to train a surrogate judge using heuristic metrics as input, to output the LLM as a judge prediction. In our work, we develop Mirage-Bench,1 a synthetic arena-based RAG benchmark for 18 diverse languages on Wikipedia focused on multilingual answer generation evaluation. It extensively couples both heuristic features and LLM as a judge for evaluation. We benchmark 19 multilingual LLMs, and observe a high correlation (Kendall Tau (\(\tau\)) = 0.909) using our surrogate judge and between GPT-4o as a teacher using the Bradley-Terry framework. Our results show proprietary and large open-source LLMs currently dominate on Mirage-Bench. Our code and datasets are made publicly available here: [https://github.com/vectara/mirae-bench](https://github.com/vectara/mirae-bench).

Footnote 1: Mirage-Bench has been coined as (Miracel+ rag + Evaluation + Benchmark).

## 1 Introduction

Large language models (LLMs) have recently gained popularity for information-seeking queries leading to the widespread adoption of retrieval-augmented generation (RAG) Guu et al. (2020); Lewis et al. (2020); Izacard and Grave (2021); Borregaud et al. (2022). The naive RAG setup traditionally includes a retrieval and a generation stage, conducted sequentially. RAG systems such as Bing Search Microsoft (2023) provide grounded responses, i.e., statements that include citations to one or more retrieved passages. The citations reduce factual hallucinations with easy verifiability and improve faithfulness to passages provided within context Khandelwal et al. (2020); Lewis et al. (2020); Gao et al. (2023a,b); Liu et al. (2024). However, existing RAG benchmarks are English-centric, due to uneven and scarce data available across multiple languages Thakur et al. (2024c). So far, it is unclear how existing LLMs perform in multilingual RAG, i.e., where queries and passages are non-English and the LLM generates the answer in the same language. An example of a naive RAG pipeline in Hindi (hn) is shown in Figure 1.

RAG benchmarks can be broadly classified as either (i) _heuristic-based_, where benchmarks hand-craft evaluation metrics (e.g., faithfulness or fluency) to evaluate systems on multiple dimensions Gao et al. (2023a); Chen et al. (2024c); Yang et al. (2024b) or (ii) _arena-based_: where systems compete each other in a tournament and an LLM-based teacher is used as the judge for evaluation

Figure 1: Multilingual naive RAG pipeline in Hindi (hn). In Mirage-Bench, we reuse the oracle retrieval set (query and oracle judged passages) from Miracal CLZhang et al. (2023) and focus on evaluating the answer generation stage with multilingual LLMs.



(Rackauckas et al., 2024; Pradeep et al., 2024). Heuristic-based benchmarks are computationally cheaper to evaluate but require human preferences as gold truth for reference. They also face challenges in aggregating different features into a ranking order for models. On the other hand, arena-based benchmarks require a high-performance or reasoning-intensive LLM as a teacher for accuracy Zheng et al. (2023), which makes exhaustive pairwise comparisons expensive for a large set of models. For instance, evaluating a single query on 19 models involves \({19\choose 2}=171\) comparisons and costs between $5-10 USD with GPT-4o OpenAI (2023).

In our work, we get the best of both worlds with a surrogate judge, a learning to rank model, e.g. random forest Ho (1995), using heuristic features to estimate an arena-based leaderboard obtained with a Bradley-Terry model Hunter (2004) from pair-wise evaluations using an LLM as a judge Zheng et al. (2023). We use bootstrapping to obtain confidence bounds for better statistical estimates. After training, the learning to rank model can be used to estimate the performance of newer released models reliably in the future without the expensive LLM as a judge. It also provides better interpretability and is easily retrainable with a different or newer set of heuristic features.

We develop Mirage-Bench, a RAG benchmark across 18 languages for multilingual generation evaluation on Wikipedia. Mirage-Bench data is sourced from miracl Zhang et al. (2023), a multilingual retrieval dataset containing human-generated queries and human-labeled relevance judgments for Wikipedia passages. We benchmark 19 frontier multilingual LLMs in our experiments. Our evaluation flowchart adopted in Mirage-Bench is shown in Figure 2. We evaluate seven heuristic features: (i) language detection, (ii) citation quality, (iii) support, (iv) reranker score, (v) answer overlap (traditional), (vi) answer overlap (LLM-measured), and (vii) fluency (LLM-measured). We use GPT-4o as the judge to evaluate our pairwise RAG comparisons on a smaller subset consisting of 100 queries. Next, we train a random forest model as a surrogate judge, using the heuristic features as input, and learn to output the Bradley-Terry model logits as output. Finally, during inference, we use our surrogate judge to produce a "synthetic" ranking leaderboard for all baselines across every language.

More specifically, we ask the following research questions in our work:

* Can we avoid the LLM as a judge cost and combine heuristic-based evaluation?
* How do frontier multilingual-focused LLMs perform in multilingual answer generation?
* Does fine-tuning on Mirage-Bench training dataset improve LLM performance?

Our experimental results show that: (i) random forest model (surrogate judge) learned rankings highly correlates with GPT-4o as a judge, achieving an average Kendall-Tau (\(\tau\)) score of 0.909. (ii) proprietary and large open-source LLMs (\(\geq\)70B parameters) achieve the topmost ranks in the Mirage-Bench leaderboard. (iii) Mirage-Bench training data, synthetically constructed using GPT-4o, is beneficial to improve smaller open-source LLMs (7-8B parameters). The main contribution of work is building Mirage-Bench and benchmark 19 frontier multilingual LLMs to accelerate development in the area of multilingual RAG.

Figure 2: The Mirage-Bench evaluation flowchart consists of three steps: (i) heuristic-based features evaluating the baseline model response across several dimensions; (ii) exhaustive pairwise comparisons with GPT-4o as a judge on a small subset of queries to train our surrogate judge. (iii) After training, we utilize our surrogate judge to output the model ranking on the whole subset of queries, to construct the synthetic RAG arena-based leaderboard.

 



## 2 Related Work

Prior work on RAG evaluation has been conducted exclusively in English. For example, benchmarks such as ALCE Gao et al. (2023), FreshLLM Vu et al. (2024), ClapN Rosenthal et al. (2025), HAGRID Kamalloo et al. (2023) and CRAG Yang et al. (2024b), all include long-form answers for English-only queries and are based on collections containing documents from either the English Wikipedia, MS MARCO Bajaj et al. (2016) or NQ Kwiatkowski et al. (2019). Similarly, TREC 2024 RAG, a TREC competition for RAG evaluation is focused on evaluating queries in English.2

Footnote 2: TREC 2024 RAG: [https://trec-rag.github.io/](https://trec-rag.github.io/)

**Multilingual RAG.** On the multilingual side, RAG has not been well studied in prior literature. The RGB benchmark Chen et al. (2024c) is limited in language scope covering only one additional language: Chinese (zh). NeuCLIR 2024 track Mayfield et al. (2024) evaluates long-form report generation from participants, but is limited to three languages. A concurrent work is BERGEN Chirkova et al. (2024), which evaluates the multilingual open-domain QA setting on 13 languages. In contrast, in Mirage-Bench, we (i) evaluate the generation task in the multilingual RAG pipeline on 18 languages, (ii) provide multilingual instruction-tuned data for RAG fine-tuning, and (iii) utilize high-quality native-speaker multilingual queries generated in miracl Zhang et al. (2023).

**Learning to rank.** A supervised learning technique, where models are trained to rank items within a list similar to training data Liu (2010); Turnbull (2017). Models are trained in either a pointwise, pairwise, or listwise objective Cao et al. (2007). In our work, we experiment with simple models such as random forest to train to rank the Bradley-Terry model coefficient produced by LLM as a judge. We keep complex models such as LambdaAMRT Burges (2010) for future work.

## 3 An Overview of Mirage-Bench

We select 18 languages in Mirage-Bench as the starting point, representing an appropriate crossection of the diversity of the languages spoken worldwide at this point. Mirage-Bench is a comprehensive multilingual RAG benchmark focusing on the generation task evaluation. As shown in Table 1, the benchmark includes 11,195 evaluation pairs and 39,763 training pairs across 18 languages.

### Distinction and Extension from miracl

miracl introduced in Zhang et al. (2023) is a monolingual retrieval dataset, which evaluates the retrieval task, i.e., given a user query and a passage corpus, retrieve the ranked list of passages as output. miracl contains human-annotated relevance judgments to evaluate retrieval and re-ranker models, e.g., lexical models like BM25 Robertson and Zaragoza (2009) or bi-encoders like mDPR Karpukhin et al. (2020), or late-interaction models like ColBERT Khattab and Zaharia (2020).

In contrast, Mirage-Bench evaluates the generation task in RAG, requiring LLMs to generate a summarized answer given the query and context available from oracle-judged passages. In our work, we reuse the queries and relevance-juddments from miracl, but solely evaluate the multilingual generation task in RAG measuring answers using both heuristics and LLM as a judge. Since Mirage-Bench evaluates the generation task containing the oracle context, independent of the retrieval task, the chances of contamination in Mirage-Bench evaluation from fine-tuning miracl is less overall.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c c c c c} \hline  & **ar** & **bn** & **de** & **cn** & **cs** & **ls** & **B** & **by** & **ll** & **ld** & **ja** & **ke** & **ru** & **sw** & **te** & **th** & **yo** & **fh** \\ \hline \multicolumn{11}{r}{**Minack-ExecExec Evaluation Dataset**} \\ \# Queries** & 1501 & 411 & 304 & 787 & 617 & 632 & 183 & 343 & 509 & 939 & 771 & 213 & 1247 & 481 & 150 & 730 & 119 & 391 \\ \# Avg. Query Tokens & 10.2 & 17.6 & 10.4 & 9.1 & 11.4 & 14.1 & 12.2 & 10.4 & 17.3 & 10.1 & 14.6 & 14.2 & 14.3 & 12.0 & 16.1 & 19.6 & 13.0 & 8.0 \\ \cline{1-1}
**Avg. Ref. Passages / Q** & 2.0 & 2.1 & 2.6 & 2.8 & 4.3 & 2.1 & 2.0 & 2.1 & 2.1 & 3.1 & 2.2 & 2.6 & 2.8 & 1.9 & 1.2 & 1.8 & 1.2 & 2.5 \\ \cline{1-1}
**Avg. Non Rat. Passages / Q** & 8.1 & 8.0 & 7.7 & 7.7 & 5.6 & 8.3 & 8.1 & 7.9 & 7.8 & 7.0 & 8.2 & 11.8 & 7.6 & 8.7 & 4.9 & 8.5 & 8.8 & 7.5 \\ \hline \multicolumn{11}{r}{**Max-ExecExec Training Dataset**} \\ \# Queries & 3468 & 1624 & — & 2857 & 21.59 & 2104 & 2878 & 1137 & 1165 & 4054 & 3466 & 859 & 4567 & 1866 & 3283 & 2965 & — & 1311 \\ \# Avg. Query Tokens & 10.2 & 17.4 & – & 8.7 & 11.2 & 14.2 & 11.8 & 10.4 & 17.1 & 9.9 & 14.5 & 14.3 & 14.3 & 12.0 & 16.1 & 19.3 & — & 7.9 \\ \hline
**Avg. Rel. Passages / Q** & 1.8 & 2.3 & 2.5 & 5.6 & 2.0 & 1.9 & 2.0 & 2.0 & 2.9 & 1.9 & 1.2 & 2.1 & 2.0 & 1.4 & 1.2 & 1.6 & 2.3 \\ \cline{1-1}
**Avg. Non Rat. Passages / Q** & 5.5 & 7.9 & — & 7.5 & 5.3 & 8.3 & 5.0 & 8.0 & 7.9 & 7.1 & 7.9 & 12.4 & 5.2 & 3.5 & 4.3 & 5.6 & — & 7.6 \\ \cline{1-1}
**Avg. GPT-to Context Tokens** & 105.5 & 1445 & — & 137.9 & 203.4 & 133.8 & 129.4 & 180.1 & 149.4 & 108.0 & 140.2 & 136.4 & 90.0 & 106.4 & 54.6 & 86.0 & — & 137.9 \\ \cline{1-1}
**Avg. GPT-to Answer Tokans** & 35.7 & 22.9 & — & 20.6 & 55.3 & 51.6 & 30.3 & 56.0 & 26.6 & 22.6 & 24.1 & 2.3 & 16.1 & 18.4 & 11.4 & 20.4 & — & 40.6 \\ \hline \end{tabular}
\end{table}
Table 1: Dataset statistics for 18 languages in Mirage-Bench; All tokens are calculated using the GPT-4o tokenizer OpenAI (2023); (Rel.) denotes relevancy; (# Avg GPT-4o) counts the tokens in the GPT-4o generated context and answer; Training data is not available for two languages (denoted by —): German (de) and Yoruba (yo).





### Mirage-Bench Evaluation Dataset

miracl queries are high-quality and generated by native speakers (Zhang et al., 2023). The annotation procedure in miracl is identical to TyDI-QA Clark et al. (2020). The passage collection is constructed from language-specific Wikipedia corpora and parsed using WikiExtractor. The Mirage-Bench evaluation dataset is constructed re-using the queries and oracle-judged passages available in the miracl development split.3

Footnote 3: We did not utilize the test split in MIRACL as the relevance judgments are not publicly available.

We incorporate two changes: (i) In Arabic (ar), we randomly sample a smaller subset of 1,501 out of 2,896 queries for uniformity in the number of queries available across other languages. (ii) We filter out a small subset of queries with zero non-relevant passages, i.e., we always include queries with hard passages from miracl to make the Mirage-Bench evaluation challenging.4

Footnote 4: An exception is Telugu (te), where 78 queries have at least one non-relevant passage. Therefore, we randomly sample 72 additional queries with only relevant judged passages.

### Mirage-Bench Training Dataset

The Mirage-Bench training dataset is developed from the miracl training dataset (Zhang et al., 2023). In Mirage-Bench, we reuse all the miracl training pairs available in 16 languages (except German (de) and Yoruba (yo)) and convert it using a simple recipe into a multilingual instruction-tuned RAG dataset Zhang et al. (2024); Niu et al. (2024). In our recipe, we first keep only the relevant passages as context along with the input query to generate a zero-shot RAG output by prompting strong teachers such as GPT-4o OpenAI (2023), Llama 3 (70B) Dubey et al. (2024) and Mistral (8\(\times\)22B) Jiang et al. (2024). After generation, we include non-relevant passages within our prompt as "distracting and noisy context", to help improve the quality of the training dataset. Since we convert a retrieval dataset, we do not have human-annotated answers for questions in Mirage-Bench.

## 4 Multilingual RAG Evaluation

### Heuristic-based Evaluation

Answer generation in RAG requires evaluation on various dimensions. For example, whether a system's response provides the correct final answer or cites the relevant documents, a single metric alone is _not sufficient_ to capture the comprehensive evaluation required for RAG systems. Inspired by other recent works Kiela et al. (2021); Santhanam et al. (2023); Gao et al. (2023), we introduce five _deterministic_ features and two _LLM-measured_ features for evaluation in our work. We rely on features that are explainable, cheap, and fast to compute. We explain each heuristic feature in Appendix A.

**Language detection.** We compute the probability of a system's response in the required target language with langid Lui and Baldwin (2012). We compute two metrics: language detection (target language) and English detection.

**Citation quality.** Using passage-level relevance judgments for all queries (or qrels) information available in miracl, we evaluate whether the system's response cites the relevant passages, crucial for measuring faithfulness. We compute and evaluate: Recall@\(k\) and MAP@\(k\), where \(k\) = 10, as we have a maximum of 10 passages per query.

**Support.** Grounding is necessary to avoid hallucinations in the system's response. Support evaluation Gao et al. (2023) checks whether each sentence is supported by cited passages using a multilingual NLI model He et al. (2023).5 We compute the probability of the _entailment_ and _neutral_ score, macro-averaged across the sentence-citation pairs.

Footnote 5: MoritzLaurer/mDeBERTx-v3-base-xnli-multilingual-nli-2mi17

**Reranker Chen et al. (2024)b: BAAI/bge-reranker-v2-m3**



**Fluency (LLM-measured).** It measures for grammatical correctness and idiomatic word choices in the system's response. As previously mentioned, we use Llama-3 (8B) Dubey et al. (2024) in a pointwise setup, proving an integer score in \([1,5]\). The fluency prompt description is listed in Figure 12.

### Arena-based Evaluation

Heuristic-based evaluation metrics often rely on a gold standard for evaluation. Tasks such as text retrieval Bajaj et al. (2016); Thakur et al. (2021) require human-labeled relevance judgments, and similarly, NLP tasks such as machine translation Stahlberg (2020), require human-annotated translations. As human preferences are seldom available in numerous applications, using LLM as a judge Zheng et al. (2023); Chen et al. (2024); Chiang et al. (2024) is becoming a de facto approach for arena-based evaluation of LLMs.

We evaluate pointwise, listwise, and pairwise LLM as judge evaluations. We anecdotally observe the pointwise judge, which is efficient \(O(n)\), but is not good for ranking LLMs as it provides similar scores for a wide range of models (e.g., 4/5 score for 16 out of 18 LLMs evaluated). Similarly, the listwise judge finds it difficult to rank all 19 models in the correct order. Therefore, although suboptimal in complexity, \(O(n^{2})\), where \(n\) is the number of models evaluated, we choose a pairwise evaluation in our work.

**Pairwise LLM as a judge.** Following prior works on arena-based evaluation in RAG Rackauckas et al. (2024); Pradeep et al. (2024), we evaluate two system's responses in a head-on comparison by computing pairwise judgments with LLM as a judge. We reuse the RAGElo prompt template Rackauckas et al. (2024) with minor additions. The prompt template is listed in Figure 13. LLM as a judge evaluator includes three types of biases Zheng et al. (2023): (i) verbosity bias Wu and

Figure 3: Lollipop plots denoting the average heuristic-based feature scores achieved by LLM baselines for each language in Mirage-Bench. \(x\)-axis denotes the 18 languages; whereas \(y\)-axis plots every heuristic feature score. Models in the same LLM family are represented in the same color in a lollipop (as multiple circles). Figure 9 in the Appendix provides lollipop plots for all eleven heuristic-based features used in our work.



Aji, 2025) (ii) self-enhancement bias Xu et al. (2024); Panickssery et al. (2024) and (iii) position bias Wang et al. (2024). We avoid the verbosity bias, as RAG evaluation has fixed evaluation criteria requiring sentence-level citations and answers Pradeep et al. (2024a) and the position bias by randomly swapping the position of two models.

### Learning to Approximate Rankings

There is no predefined way to aggregate the heuristic features to provide an overall leaderboard ranking in Mirage-Bench Averaging the scores is too simplistic as features measure different aspects of RAG evaluation. On the other hand, areanabased evaluations provide ranked leaderboards but are computationally expensive to compute with a strong teacher model. To avoid computational costs, smaller models as teachers have been proposed Thakur et al. (2024a); Ni et al. (2024).

Motivated by similar observations, we train a surrogate judge to effectively emulate an arena-based leaderboard without incurring the expensive LLM as a judge pairwise cost. We find the random forest model to serve as a scalable and cost-effective judge that can be trained within minutes on small training datasets without expensive computation. Therefore, we train a random forest (learning to rank model) as a surrogate judge to approximate the Bradley-Terry model coefficients Hunter (2004) learned from an arena-based evaluation that uses GPT-4o as a judge for pairwise judgments.

**Learning to rank model.** While the heuristic features introduced in Section 4.1 can be computed efficiently and without the reliance on proprietary LLMs, inducing a ranking from pairwise comparisons via a Bradley-Terry model is computationally expensive and requires access to a high-performance LLM. Furthermore, as we demonstrate in Section 6.3, the ranking accuracy, measured by the Kendall-Tau (\(\tau\)) coefficient, degrades rapidly when subsampling tournament matches.

The procedure, detailed in Algorithm 1, simulates \(N_{t}\) tournaments, each involving a total of \(N_{l}\) models and \(N_{q}\) queries. For each query, judgments are obtained for all \(\binom{N_{l}}{2}\) pairings of models. We employ bootstrapping on the query selection process to estimate the variance in the \(R^{2}\) metric in the learning to rank models' approximations of the Bradley-Terry coefficients over a randomly-sampled holdout set, lLM\({}_{predict}\).

We randomly select two models, Lemma 1.1 (B) and Llama-3 (70B) as holdout models, i.e., we do not train on the features for holdout models. For English, we observe an average \(\bar{R}^{2}=0.971\) with a 95% confidence interval of \([0.905,0.999]\). On the other hand, for Bengali, we observe \(\bar{R}^{2}=0.937\) with a 95% confidence interval of \([0.766,0.998]\). \(\bar{R}^{2}\) scores for all 18 languages with 95% confidence intervals are listed in Table 5. Taken together, these results indicate that the training procedure is fairly robust with \(N_{q}=100\).

```
1:for\(i\in[N_{t}]\)do
2:\(M_{BT}^{i}\leftarrow\textsc{Tournament}(N_{q})\)
3:\(X_{t},Y_{t}\leftarrow\textsc{Dataset}(\texttt{llm}_{train},M_{BT}^{i})\)
4:\(X_{p},Y_{p}\leftarrow\textsc{Dataset}(\texttt{llm}_{predict},M_{BT}^{i})\)
5:\(M_{reg}^{i}\leftarrow\textsc{Fit}(X_{t},Y_{t})\)
6:\(R_{2}^{i}\gets M_{reg}^{i}(X_{p},Y_{p})\)
7:endfor
8:\(M_{BT}\leftarrow[M_{BT}^{1};M_{BT}^{2};...;M_{BT}^{N_{t}}]\)
9:\(M_{reg}\leftarrow[M_{reg}^{1};M_{reg}^{2};...;M_{reg}^{N_{t}}]\)
10:\(R_{2}\leftarrow[R_{2}^{1};R_{2}^{2};...;R_{2}^{N_{t}}]\)
```

**Algorithm 1** Simulate Tournaments and Fit Models

## 5 Experimental Settings

### Multilingual Baselines

Existing frontier LLMs are either English-only or support a limited set of languages, predominantly due to the _curse of multilinguality_ for large models Conneau et al. (2020). It is unclear how well existing LLMs perform on RAG on a wide variety of languages, due to the scarce availability of multilingual instruction tuning datasets. We experiment with LLMs from seven different families, containing proprietary and open-source LLMs. Wherever possible, we evaluate the _instruction-tuned_ version if available. Refer to Appendix B for more details.

* **OpenAI:** GPT-3.5-turbo, GPT-4, and GPT-4o OpenAI, 2023) using the Azure OpenAI service.
* **Mistral:** Mistral-Instruct-v0.2 and v0.3 (7B) Jiang et al. (2023) and Mixtral-Instruct-v0.1 (8\(\times\)7B) and (8\(\times\)22B) versions Jiang et al. (2024).
* **Cohere:** Command-R (35B), Command-R+ 

(104B) and Aya-23 (35B) (Aryabumi et al., 2024).
* **Gemma:** Gemma 1.1 instruct (2B) and (7B) models Mesnard et al. (2024).
* **Llama-3:** Llama-3 instruct (8B) and (70B) models Dubey et al. (2024).
* **Phi-3:** Phi 3 instruct series: Medium (14B), Small (7B), and Mini (3.8B) Abdin et al. (2024).
* **Qwen-2:** Qwen-2-instruct series: 1.5B and 7B Yang et al. (2024).

Prompt templateWe internally optimized6 the ChatQA prompt template Liu et al. (2024), to include in-text citations of the context passages following the IEEE format Kamallo et al. (2023). In Mirage-Bench, we have about \(10\) passages annotated in the oracle setting. Therefore, we trim each passage available and take the first 800 tokens to fit all passages within a fixed context length of 8192 tokens. Following prior work in Shi et al. (2023), the prompt requires the LLM to explain the multilingual generation task starting with "#Reason" and the answer itself starting with "##Answer". Utilizing this output format has its advantages in easily parsing the generated answer and the rationale behind the answer. The prompt template for multilingual generation is shown in Figure 10.

Footnote 6: A majority of the prompt optimization was internal and based on eye-balling RAG responses across LLMs.

## 6 Experimental Results

### Heuristic-based Results

Figure 3 shows lollipop plots indicating the average heuristic-feature value (\(y\)-axis) distribution for each language (\(x\)-axis). In English detection (higher the worse), smaller LLMs such as Gemma-1.1 (2B) do not generate output in the required target language but rather generate reasoning and answers in English. For citation quality, support, and reranker score features, LLMs from OpenAI and Llama-3 family achieve high Recall\(@10\) and entailment scores (except Llama-3 (70B) for a few languages), indicating generated answers include grounded citations from relevant passages. In contrast, LLMs from the Qwen-2 or Gemma-1.1 family, tend to under-cite passages in their answers.

Furthermore, we observe LLMs from the OpenAI family achieve the highest word overlap in the ROUGE-L metric (GPT-4 used as ground truth, which could be a potential bias) and Llama-3 (8B) as a judge, while we observe less variance across other LLMs. In Fluency, we observe a majority of the LLMs are rather fluent in generation, except Bengali (bn), Telugu (te), and Yoruba (yo).

\begin{table}
\begin{tabular}{c c|c|c|c|c|c|c} \hline \multicolumn{1}{l|}{**Lang.**} & \(\tau\) & **Lang.** & \(\tau\) & **Lang.** & \(\tau\) & **Lang.** & \(\tau\) \\ \hline ar & 0.951 & bn & 0.874 & de & 0.825 & en & 0.835 \\ es & 0.876 & fa & 0.924 & fi & 0.949 & fr & 0.914 \\ hi & 0.946 & id & 0.896 & ja & 0.892 & ko & 0.950 \\ ru & 0.849 & sw & 0.958 & te & 0.938 & tn & 0.946 \\  & yo & 0.906 & rh & 0.941 & & \\ \hline \end{tabular} Avg. Kendall Tan (\(\tau\)) on 18 languages = **0.909**

\end{table}
Table 2: Kendall \(\tau\) rank correlation between pairwise GPT-4o as a judge Bradley-Terry model and the synthetic arena-based ranking leaderboard generated using our surrogate judge in Mirage-Bench.

Figure 4: Mirage-Bench arena-based leaderboards: (left heatmap) Bradley-Terry model coefficients with GPT-4o as a pairwise judge for a subset of 100 sampled queries; (right heatmap) Synthetic rankings using heuristic-based features and a random forest model as a surrogate judge on all queries. Each highlighted cell denotes the rank of the LLM (lower the better). LLMs are sorted by lowest to highest average rank across all 18 languages.

 



### Arena-based Results

Figure 4 (left heatmap) is the arena-based leaderboard with the Bradley Terry model conducting 200 tournaments and bootstrapping 100 matches per tournament on a subset of 100 queries using GPT-4o as a pairwise judge. We observe that proprietary LLMs such as GPT-4o and GPT-4, and larger open-source LLMs such as Llama-3 (70B) and Miktral (8\(\times\)22B) perform better than other LLMs. LLM rankings across languages are usually stable; with a few notable exceptions such as Gemma-1.1 (7B) which achieves a rank of 4 in Telugu. Command R (35B) performs poorly in low-resource languages such as Bengali (rank 13) or Swahili (rank 14). The complete scores including model coefficient logits and 95% confidence intervals (error bars) are provided in Table 7 and Table 8 in the Appendix.

**Synthetic rankings using random forest.** Figure 4 (right heatmap) is the learned synthetic leaderboard rankings on all queries using heuristic-based features trained with a random forest model. The synthetic leaderboard generated using a surrogate judge, highly correlates to the GPT-4o as a pair-wise judge, achieving an average Kendall-Tau (\(\tau\)) rank correlation = **0.909**, by training on 17 LLMs during training and keeping 2 LLMs as a holdout for every language. Individual language-specific Kendall-Tau rank correlation scores are listed in Table 2. This provides evidence of the efficacy of training a random forest model as a surrogate judge. In Appendix D, we extend the evaluation for Llama-3.1 and Gemma-2 series of LLMs.

**Heuristic feature importance.** In Figure 5, we plot the average feature importance achieved by our random forest model as a surrogate judge. Using Llama-3 (8B) as a judge, for fluency and answer overlap are the important heuristic features. Similarly, deterministic answer overlap and reranking-based metrics are equally important. Some heuristic features such as language detection, and neutral score in support evaluation obtain the least importance. We observe all answer-related heuristic features achieve a high importance indicating that the generated "answer" portion in an LLM's response is crucial and required to learn rankings from GPT-4o as a pairwise judge.

### Ablations & Discussion

To better understand the gaps observed during training of the random forest model as a surrogate judge, we conduct further ablations on a subset of seven languages including Arabic (ar), Bengali (bn), Finnish (fi), Japanese (ja), Korean (ko), Russian (ru) and Telugu (te):

**Learning to rank model choice.** We compare different learning to rank models as choices for learning the Bradley-Terry model coefficients. We conduct our experiments on the train set, where models contain pairwise judgments, and on a randomly sampled holdout set, a realistic scenario, with no available training data. We evaluate the following choices: Random Forest, Linear Regression, MLP Regressor, XGB Regressor, and SVR. All models are implemented via scikit-learn.8 The results are shown in Table 3. Random forest achieves the best \(R^{2}\) metric on the holdout subset for 4 out of 7 languages. SVR also achieves a similar \(R^{2}\) metric on the holdout subset, however, underperforms random forest on the training subset. Other baselines, such as XGB Regressor and MLP Regressor show signs of significant overfitting on the training subset, thereby underperforming random forest on the holdout subset.

Footnote 8: [https://scikit-learn.org/stable/supervised_learning.html](https://scikit-learn.org/stable/supervised_learning.html)

Figure 5: Boxplot with the feature importance value (averaged across 18 languages in Mirage-Bench) observed by the learning to rank (random forest) model.





**Non-exhaustive pairwise comparisons lead to performance degradation.** Exhaustive pairwise comparisons across a subset of 19 models in Mirage-Bench using GPT-4o for all queries are quite expensive. To avoid this, we investigate whether all pairwise exhaustive comparisons are necessary during training. We utilize two sampling techniques: (i) full pairwise judgments on a subsample of 100 queries, e.g., 20 or 50 queries; (ii) partially judge a non-exhaustive random sample of the pairwise judgments across 100 queries, e.g., only 50% of all the exhaustive pairwise combinations. Both results are shown in Figure 6. we observe that Kendall-Tau (\(\tau\)) correlations increase linearly with queries and pairwise judgments. In summary, an exhaustive pairwise comparison and a sufficient number of queries, such as 100, are necessary without impacting the leaderboard rankings.

**All heuristic features are not necessary.** We experiment with features used for random forest model training as a surrogate judge. We evaluate four training configurations: (i) all features (ii) without LLM-measured features (iii) without language detection and support, i.e., the low-correlation features observed in Figure 5, and (iv) including only LLM-measured features. From Table 4, we observe that removing low-correlated heuristic RAG features helps train the random forest model better leading to a conclusion that not necessarily all heuristic features are important. Removing the LLM-measured features completely or only using them for training the model decreases the Kendall-Tau (\(\tau\)) correlation score.

**Fine-tuning on Mirage-Bench training data.** We evaluate three variants of the Mirage-Bench training dataset using two smaller open-source LLMs: Mistral-v0.2 (7B) and Llama-3 (8B). We fine-tune the Mirage-Bench training datasets using (i) both on GPT-4o, (ii) Llama-3 (8B) on Llama-3 (7OB), and (iii) Mistral-v0.2 (7B) on Mixtal (8\(\times\)22B). From Figure 7, we observe that GPT-4o is a strong teacher, Mistral-v0.2 (7B) fine-tuned on GPT-4o distilled training data achieves rank 2 outperforming Llama-3 (7OB). This shows that Mirage-Bench training data is useful for improving the RAG answer generation task quality.

## 7 Conclusion

We present Mirage-Bench, a multilingual RAG benchmark for 18 languages aimed at evaluating the multilingual generation part within RAG and aggregate traditional heuristic-based features to train a lightweight learning to rank model as a surrogate judge to learn a Bradley Terry model with GPT-4o pairwise judgments. Our results indicate a strong correlation between our surrogate judge and GPT-4o as a pairwise judge. This demonstrated the effectiveness of our efficient, cheap, and easy-to-retrain random forest model as a surrogate judge trained using only computationally cheap heuristic features for arena-based leaderboard ranking by achieving a 0.909 Kendall \(\tau\). On Mirage-Bench, we observe that most proprietary and open-source larger LLMs currently dominate, whereas smaller open-source LLMs continue to struggle. Instruction tuning on Mirage-Bench training data helps improve smaller open-source LLMs, e.g. instruction-tuned Mistral-v0.2 (7B) on GPT-4o distilled training data can outperform Llama 3 (7OB) on Mirage-Bench.

Figure 6: Sampling experiments to reduce computation cost. (left) reduces the number of queries whereas (right) reduces the pairwise judgments.

\begin{table}
\begin{tabular}{l c r r r r r r r} \hline \hline Features / Language & wf & ur & bn & fi & ja & ko & rn & vq \\ \hline Allly Features & 11 & Osss & **0.850** & 0.921 & 0.851 & 0.921 & 0.826 & 0.918 \\ (Wy) LLM As a Judge & 9 & 0.912 & 0.866 & 0.985 & 0.853 & 0.917 & 0.811 & 0.904 \\ (Wy) Low Correlation & 7 & **0.951** & **0.867** & **0.923** & **0.885** & **0.929** & 0.829 & **0.940** \\ (Oky) LLM As Judge & 2 & 0.948 & 0.728 & 0.907 & 0.854 & 0.916 & **0.851** & 0.872 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Kendall Tau (\(\tau\)) scores using different features for training the random forest regression model.

Figure 7: Approximate rankings using heuristic features after fine-tuning Llama-3 (8B) and Mistral-v0.2 (7B) on Mirage-Bench dataset across four configurations.





## 8 Limitations

Mirage-Bench is one of the first large-scale multilingual RAG benchmarks. Although not perfect, we below discuss a set of limitations in our work:

* In Mirage-Bench, we focused on benchmarking the generation task in RAG with oracle passages, we did not consider the retrieval task and its error propagating on the generation task.
* Due to budget constraints, we were unable to evaluate diverse LLMs as teachers such as Claude-3.5 (sonnet) Anthropic [20] or Gemini Pro Anil et al. [2023]. We evaluated using GPT-4o which can cause self-enhancement bias towards LLMs in the OpenAI family.
* In our heuristic evaluation, we only considered a smaller subset of features. We did not explore more recent hand-crafted features such as nugget-based recall and precision Pradeep et al. [2024b; Farzi and Dietz 2024; Arabzadeh and Clarke 2024; Lin and Demner-Fushman [2005].
* Lastly, Mirage-Bench does not provide human-labeled answers for queries across all languages and is limited to Wikipedia as the source.

## Acknowledgements

This research is supported by the Canada First Research Excellence Fund and the Natural Sciences and Engineering Research Council (NSERC) of Canada. We thank Ka Wong for helping us out with bootstrapping during the training of the random forest model as a surrogate judge.

## References

* A. Abdin, S. Jacoobs, A. Awan, J. Aneja, A. Awadallah, H. Awadalla, N. Bach, A. Bahree, A. Bakhtiari, H. S. Behl, A. Benhaim, M. Bilenko, J. Bjorck, S. Bubeck, M. Cai, C. Cesar Teodoro Mendes, W. Chen, V. Chaudhary, P. Chopra, A. Del Giorno, G. de Rosa, M. Dixon, R. Eldan, D. Iter, A. Garg, A. Goswami, S. Guanesakker, E. Haider, J. Hao, R. I. Hewett, J. Huyhn, Q. Javaherij, X. Jin, P. Kauffmann, N. Karampatziakis, D. Kim, M. Khademi, L. Kurlienko, J. R. Lee, Y. T. Lee, Y. Li, C. Liang, W. Liu, E. Lin, Z. Lin, P. Madan, A. Mitra, H. Modi, A. Nguyen, B. Norick, B. Patra, D. Perez-Becker, T. Portel, R. Pyramid, H. Roy, O. Ruwase, O. Saarikivii, A. Saied, A. Salim, M. Santacroce, S. Shah, N. Shang, H. Sharma, X. Song, M. Tanaka, X. Wang, R. Ward, G. Wu, J. Yu, K. Xu, J. Xu, F. Yang, Z. Yan, S. Zhang, Y. Zhang, C. Zheng, Y. Zhou, and X. Zhou (2024)Phi-3 technical report: a highly capable language model locally on your phone. CoRRabs/2404.14219. External Links: Link Cited by: SS6.
* R. Anil, S. Borgesaud, Y. Wu, J. Alayrac, J. Yu, R. Sorcitu, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, D. Silver, S. Petrov, M. Johnson, I. Antonoglou, J. Schrittwieser, A. Glaese, J. Chen, E. Pitler, T. P. Lillicrap, A. Lazaridou, O. Firat, J. Molloy, M. Isard, P. Randal Barham, T. Hennigan, B. Le, F. Viola, M. Reynolds, Y. Xu, R. Doherty, E. Collins, C. Meyer, E. Rutherford, E. Moreira, K. Ayoub, M. Goel, G. Tucker, E. Piperas, M. Krikhun, I. Barr, N. Savinov, I. Danika, B. Roelofs, A. White, A. Andreassen, T. von Glehm, L. Yagati, M. Kazemi, L. Gonzalez, M. Khalman, J. Sygnowski, and et al. (2023)Gemini: a family of highly capable multimodal models. CoRRabs/2312.11805. External Links: Link Cited by: SS6.
* N. Arabzadeh and C. L. A. Clarke (2024)A comparison of methods for evaluating generative IR. CoRRabs/2404.04044. External Links: Link Cited by: SS6.
* V. Aryabumi, J. Dang, D. Talupuru, S. Dash, D. Cairuz, H. Lin, B. Venkitesh, M. Smith, J. A. Campos, Y. Chen Tan, K. Marchisio, M. Bartolo, S. Ruder, A. Locatelli, J. Kreutzer, N. Frosst, A. N. Gomez, P. Blunsom, M. Fadaee, A. Ustun, and S. Hooker (2024)Aya 23: open weight releases to further multilingual progress.. CoRRabs/2405.15032. External Links: Link Cited by: SS6.
* P. Bajaj, D. Campos, N. Crasswell, L. Deng, J. Gao, X. Liu, R. Majumder, A. McNamara, B. Mitra, T. Nguyen, M. Rosenberg, X. Song, A. Stoica, S. Tiwary, and T. Wang (2016)MS MARCO: a human generated machine reading comprehension dataset. CoRRabs/1611.09268. External Links: Link Cited by: SS6.
* S. Borgead, A. Marsch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. van den Driessche, J. Lespiau, B. Damoc, A. Clark, D. de Las Casas, A. Guy, J. Menick, R. Ring, T. Hennigan, S. Huang, L. Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganni, G. Irving, O. Vinyals, S. Osindero, K. Simonyan, J. W. Bae, E. Elsen, and L. Sifre (2022)Improving language models by retrieving from trillions of tokens. In International Conference on

Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_, volume 162 of _Proceedings of Machine Learning Research_, pages 2206-2240. PMLR.
* Burges (2010) Chris J.C. Burges. 2010. From RankNet to LambdaMark to LambdaMART: An overview. Technical Report MSR-TR-2010-82.
* Cao et al. (2007) Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007. Learning to rank: from pairwise approach to listwise approach. In _Proceedings of the 24th International Conference on Machine Learning_, ICML '07, page 129-136, New York, NY, USA. Association for Computing Machinery.
* Chen et al. (2024a) Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024a. Humans or LLMs as the judge? a study on judgement bias. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing_, pages 8301-8327, Miami, Florida, USA. Association for Computational Linguistics.
* Chen et al. (2024b) Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024b. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 2318-2335, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
* Chen et al. (2024c) Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024c. Benchmarking large language models in retrieval-augmented generation. In _Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada_, pages 17754-17762. AAAI Press.
* Chiang et al. (2024) Wei-Lin Chiang, Tianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: An open platform for evaluating LLMs by human preference. In _Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024_. OpenReview.net.
* Chirkova et al. (2024) Nadezhda Chirkova, David Rau, Herve Dejean, Thibault Formal, Stephane Clinchant, and Vassilina Nikoulina. 2024. Retrieval-augmented generation in multilingual settings. In _Proceedings of the 1st Workshop on Towards Knowledgeable Language Models (KnowLLM 2024)_, pages 177-188, Bangkok, Thailand. Association for Computational Linguistics.
* Clark et al. (2023) Elizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez, Roee Aharoni, Vitaly Nikolaev, Thibault Sellam, Aditya Siddharn, Dipanjan Das, and Ankur P. Parikh. 2023. SEAHORSE: A multilingual, multifaceted dataset for summarization evaluation. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_, pages 9397-9413. Association for Computational Linguistics.
* Clark et al. (2020) Jonathan H. Clark, Jenninaria Palomaki, Vitaly Nikolaev, Enusol Choi, Dan Garrette, Michael Collins, and Tom Kwiatkowski. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. _Trans. Assoc. Comput. Linguistics_, 8:454-470.
* Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 8440-8451. Association for Computational Linguistics.
* Dao (2024) Tri Dao. 2024. FlashAttention-2: Faster attention with better parallelism and work partitioning. In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.
* Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aoho Yang, Archit Mitra, Archic Saravan Kumar, Artem Korenev, Arthur Hinnsvark, Arnu Rao, Aston Zhang, Augreien Rodriguez, Austen Gregerson, Avgataru, Baptiste Roziere, et al. 2024. The Llama 3 herd of models.
* Fan et al. (2019) Angela Fan, Yaccine Jernite, Ethan Perez, David Granier, Jason Weston, and Michael Auli. 2019. EL1S: long form question answering. In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_, pages 3558-3567. Association for Computational Linguistics.
* Farzi and Dietz (2024) Naghmheh Farzi and Laura Dietz. 2024. Pencis down! automatic rubric-based evaluation of retrieve/generate systems. In _Proceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval_, ICTIR '24, page 175-184, New York, NY, USA. Association for Computing Machinery.
* Gao et al. (2023a) Tianyu Gao, Howard Yen, Jiatop Yu, and Danqi Chen. 2023a. Enabling large language models to generate text with citations. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_, pages 6465-6488. Association for Computational Linguistics.
* Gao et al. (2023b) Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, 

Meng Wang, and Haofen Wang. 2023b Retrieval-augmented generation for large language models: A survey. CoRRabs/2312.10997.
* Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3929-3938. PMLR.
* He et al. (2023) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
* 15, 1995, Montreal, Canada. Volume I, pp. 278-282. IEEE Computer Society.
* Hu et al. (2022) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.

* 23, 2021_, pages 874-880. Association for Computational Linguistics.
* Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lelio Rehard Lavaud, Marie-Anne Lachaux, Pierre Stock, Thomas Wang, Timothe Lacroix, and William El Sayed. 2023. Mitral 7b. _CoRR_, abs/2310.06825.
* Jiang et al. (2024) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lelio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Theophile Gervet, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. 2024. Mixstral of experts. CoRRabs/2401.04088.
* Kamalloo et al. (2023)Ehsan Kamalloo, Aref Jarafari, Xinyu Zhang, Nandan Thakur, and Jimmy Lin. 2023. HAGRID: A human-llm collaborative dataset for generative information-seeking with attribution. CoRRabs/2307.16883.
* Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, Online. Association for Computational Linguistics.
* Khandelwal et al. (2020) Urvashl Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
* Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and effective passage search via contextualized late interaction over BERT. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, page 39-48, New York, NY, USA. Association for Computing Machinery.
* Kiela et al. (2021) Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amamperti Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedical, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynabench: Rethinking benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_, pages 4110-4124. Association for Computational Linguistics.
* Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: a benchmark for question answering research. Trans. Assoc. Comput. Linguistics, 7:452-466.
* Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, pages 611-626. ACM.
* Lewis et al. (2023) Patrick S. H. Lewis, Ethan Perez, Aleksandra Picktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kutler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, Sebastian Riedel, and Douwe 

Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, pp. 748-759. Cited by: SS1.
* C. Lin (2004)ROUGE: a package for automatic evaluation of summaries. In Text Summarization Branches Out, Barcelona, Spain, pp. 74-81. External Links: Link, Document Cited by: SS1.
* J. Lin and D. Demner-Fushman (2005)Automatically evaluating answers to definition questions. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, Vancouver, British Columbia, Canada, pp. 931-938. External Links: ISBN 978-3-319-142-3, Link, Document Cited by: SS1.
* T. Liu (2010)Learning to rank for information retrieval. In Proceeding of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2010, Geneva, Switzerland, July 19-23, 2010, pp. 904. External Links: Link, Document Cited by: SS1.
* Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu (2023)G-Eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 2511-2522. External Links: Link, Document Cited by: SS1.
* Z. Liu, W. Ping, R. Roy, P. Xu, A. Lee, M. Shoeybi, and B. Catanzaro (2024)ChatQA: surpassing GPT-4 on conversational QA and raga. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, Cited by: SS1.
* M. Lui and T. Baldwin (2012)Iangidj.py: an off-the-shelf language identification tool. In The 50th Annual Meeting of the Association for Computational Linguistics, Proceedings of the System Demonstrations, July 10, 2012, Jeju Island, Korea, pp. 25-30. External Links: Link, Document Cited by: SS1.
* S. Mangrukar, S. Gugger, L. Debut, Y. Belkada, S. Paul, and B. Bossan (2022)PEFT: state-of-the-art parameter-efficient fine-tuning methods. Note: [https://github.com/huggingface/peft](https://github.com/huggingface/peft) Cited by: SS1.
* J. Mayfield, E. Yang, D. J. Lawrie, S. MacAvaney, P. McNamee, D. W. Oard, L. Soldaini, I. Soboroff, O. Weller, F. Selin Kayi, K. Sanders, M. Mason, and N. Hibbler (2024)On the evaluation of machine-generated reports. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, pp. 1904-1915. External Links: Link, Document Cited by: SS1.
* T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Riviere, M. Sanjay Kale, J. Love, P. Tafti, L. Hussenot, A. Chowdhory, A. Roberts, A. Barua, A. Bottev, A. Broutse, A. Botev, A. Toschetti, A. Bulanova, A. Paterson, B. Tsai, B. Shahriari, C. Le Lan, C. A. Ahoquette-Choo, C. Crey, D. Cer, D. Depolito, D. Reid, E. Buchatskaya, E. Niezic, N. Doleng Yan, G. Tucker, G. Muraru, G. Rozhdestvenskiy, H. Michalewski, I. Tenney, I. Grishchenko, J. Austin, J. Keeling, J. Labanowski, J. Lespiau, J. Stanway, J. Brennan, J. Chen, J. Ferret, J. Chiu, and et al. (2024)GEMA: open models based on Gemini research and technology. CoRRabs/2403.08295. External Links: Link, 2403.08295 Cited by: SS1.
* M. (2023)Reinforventing search with a new AI-powered Microsoft Bing and Edge, your copilot for the web. External Links: Link Cited by: SS1.
* J. Ni, F. Xue, X. Yue, Y. Deng, M. Shah, K. Jain, G. Neubig, and Y. You (2024)MixEval: deriving wisdom of the crowd from LLM benchmark mixtures. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, Cited by: SS1.
* C. Niu, Y. Wu, J. Zhu, S. Xu, K. Shum, R. Zhong, J. Song, and T. Zhang (2024)RAGTruth: a hallucination corpus for developing trustworthy retrieval-augmented language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangok, Thailand, pp. 10862-10878. External Links: Link, Document Cited by: SS1.
* Q. Li (2023)GPT-4 technical report. CoRRabs/2303.08774. External Links: Link, 2303.08774 Cited by: SS1.
* A. Panicksery, S. R. Bowman, and S. Feng (2024)LLM evaluators recognize and favor their own generations. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, Cited by: SS1.
* K. Papineni, S. Roukos, T. Ward, and W. Zhu (2002)BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, pp. 311-318. External Links: Link, Document Cited by: SS1.
* K. Pillutta, S. Swayamdipta, R. Zellers, J. Thickstun, S. Welleck, Y. Choi, and Z. Harchaoui (2021)MAVE: measuring the gap between neural text and human text using divergence frontiers. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 4816-4828. External Links: Link, 2102.0248 Cited by: SS1.
* R. Pradeep, N. Thakur, S. Shariffymoghadam, E. Zhang, R. Nguyen, D. Campos, N. Craswell, and J. Lin (2024)RagnAROK: a 

reusable RAG framework and baselines for TREC 2024 retrieval-augmented generation track. CoRRabs/2406.16828.
* Pradeep et al. (2024) Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. 2024b. Initial nugget evaluation results for the TREC 2024 RAG track with the automwegeitzer framework. _CoRR_, abs/2411.09607.
* Rackauckas et al. (2024) Zackary Rackauckas, Arthur Camara, and Jakub Zaverl. 2024. Evaluating RAG-fusion with RAGElo: an automated elo-based framework. In _Proceedings of The First Workshop on Large Language Models for Evaluation in Information Retrieval (LLM4Eval 2024) co-located with loth International Conference on Online Publishing (SIGIR 2024), Washington D.C., USA, July 18, 2024_, volume 3752 of _CEUR Workshop Proceedings_, pages 92-112. CEUR-WS.org.
* Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. _Foundations and Trends in Information Retrieval_, 3(4):333-389.
* Rosenthal et al. (2025) Sara Rosenthal, Avirup Sil, Radu Florian, and Salim Roukos. 2025. CLAPnq: Cohesive long-form answers from passages in natural questions for RAG systems. _Transactions of the Association for Computational Linguistics_, 13:53-72.
* Santhanam et al. (2023) Keshav Santhanam, Jon Saad-Falcon, Martin Franz, Omar Khattab, Avi Sil, Radu Florian, Md Arafat Sultan, Salim Roukos, Matei Zaharia, and Christopher Potts. 2023. Moving beyond downstream task accuracy for information retrieval benchmarking. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 11613-11628, Toronto, Canada. Association for Computational Linguistics.
* Shi et al. (2023) Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipjanjan Das, and Jason Wei. 2023. Language models are multilingual chain-of-thought reasoners. In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
* Stahlberg (2020) Felix Stahlberg. 2020. Neural machine translation: A review. _J. Artif. Intell. Res._, 69:343-418.
* Team et al. (2024) Gemma Team et al. 2024. Gemma 2: Improving open language models at a practical size.
* Thakur et al. (2024) Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. 2024a. Judging the judges: Evaluating alignment and vulnerabilities in LLMs-as-Judges. _CoRR_, abs/2406.12624.
* Thakur et al. (2024) Nandan Thakur, Luiz Bonifacio, Crystalline Zhang, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Boxing Chen, Mehdi Rezagholizadeh, and Jimmy Lin. 2024b. "Knowing When You Don't Know": A multilinear relevance assessment dataset for robust retrieval-augmented generation. In _Findings of the Association for Computational Linguistics: EMNLP 2024_, pages 12508-12526, Miami, Florida, USA. Association for Computational Linguistics.
* Thakur et al. (2024c) Nandan Thakur, Jianmo Ni, Gustavo Hernandez Abrego, John Wieting, Jimmy Lin, and Daniel Cer. 2024c. Leveraging LLMs for synthesizing training data across many languages in multilingual dense retrieval. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 7699-7724, Mexico City, Mexico. Association for Computational Linguistics.
* Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Ruckle, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlIPS Datasets and Benchmarks 2021, December 2021, virtual_.
* Tunstall et al. (2023) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alvaro Bartolome, Alexander M. Rush, and Thomas Wolf. 2023. The Alignment Handbook.
* Turnbull (2017) Doug Turnbull. 2017. Learning to rank 101 -- linear models.
* Vu et al. (2024) Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry W. Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denmy Zhou, Quoc V. Le, and Thang Luong. 2024. FreshLTMs: Refreshing large language models with search engine augmentation. In _Findings of the Association for Computational Linguistics, ACL 2024, Bangskok, Thailand, and virtual meeting, August 11-16, 2024_, pages 13697-13720. Association for Computational Linguistics.
* Wang et al. (2024) Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. 2024. Large language models are not fair evaluators. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangskok, Thailand, August 11-16, 2024_, pages 9440-9450. Association for Computational Linguistics.
* Wu and Aji (2025) Minghao Wu and Alham Fikri Aji. 2025. Style over substance: Evaluation biases for large language models. In _Proceedings of the 31st International Conference on Computational Linguistics_, pages 297-312, Abu Dhabi, UAE. Association for Computational Linguistics.
* Xu et al. (2024) Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Wang. 2024. Pride and prejudice: LLM amplifies self-bias in self-refinement. In _Proceedings of the 62nd Annual Meeting of the 

Association for Computational Linguistics (Volume I: Long Papers), ACL 2014, Bangokol, Thailand, August 11-16, 2024_, pages 15474-15492. Association for Computational Linguistics
* Yang et al. [2023] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chenspeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Kemling Lu, Kejian Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruiz Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Yuancheng Ren, Xuejing Liu, Yang Fan, Yanyo Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. 2024a. Qwen2 technical report.
* comprehensive RAG benchmark. In _The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
* November 4, 2018_, pages 2369-2380. Association for Computational Linguistics.
* Zhang et al. [2023] Tianjun Zhang, Shishir G Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E. Gonzalez. 2023. RAFT: Adapting language model to domain specific RAG. In _First Conference on Language Modeling_.
* Zhang et al. [2023] Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiagoquang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. 2023. MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages. _Transactions of the Association for Computational Linguistics_, 11:1114-1131.
* 16, 2023_.
 

Heuristic-based Evaluation: Features & Additional Details

**1. Language Identification:** In a multilingual RAG system, the output response should ideally be in the same language in which the user asked their query. To capture this feature, we attempt to identify which natural language the output response is in. We use langid(Lui and Baldwin, 2012), an off-the-shelf language detecting Python library for detecting the language of the long-form RAG answer. We use the probability of the target language detected as the score for language identification, i.e., \(\hat{p}=langid(a,t)\), where \(t\) denotes the target language and \(a\) denotes the long-form answer.

**2. Citation Quality:** A multilingual RAG system must cite information from the relevant passages within their answers, to improve faithfulness and reduce hallucinations. We capture whether the passages (using relevance judgments provided in miracl(Zhang et al., 2023)) are cited in the multilingual generation task. For scoring, we compute the Recall\(@k\) and MAP\(@k\) score, where Recall\(@k\) is 1.0 for a generated answer \(a\), if and only if \(a\) cites all available relevant passages, Similarly, the MAP\(@10\) score measures the percentage of relevant passages within the top-\(k\) cited passages.

**3. Support:** RAG systems have been shown to hallucinate across retrieval-augmented generation tasks, especially when provided with non-relevant contexts Thakur et al. (2024b). Grounding is necessary to avoid hallucinations. We compute the grounding score of every sentence \(s_{j}\) in generated answer \(A\) along with the cited context \(c_{j}\) using the multilingual NLI model, which computes the similarity score as a probability of either _entailment_, _neutral_ or _contradiction_. The entailment denotes the generated sentence in the long-form answer, which entails the cited passage within its response.

**4. Reranker Score:** The reranker score measures the semantic similarity between the user query and the cited passages in the system's response. If the cited passages are relevant in answering the query, the reranker model would output a higher similarity score. We utilize a multilingual open-source reranker, namely BGE-M3 for our evaluation. We compute the reranker score across each cited passage \(p^{j}_{i}\) included in the long-form answer along with the user query \(q_{i}\).

**5. Answer Overlap:** Existing open-domain question answering datasets such as Natural Questions Kwiatkowski et al. (2019), HotpotQA Yang et al. (2018) or ELI5 Fan et al. (2019) all include a human-labeled answer, assisting in evaluation using text overlap metrics such as Exact Match (EM) or F1. However, user queries in RAG systems potentially generate long-form answers, requiring metrics such as SacreBLEU or ROUGE-L to evaluate text-generation tasks. Automatic metrics are fairly quick and cheap to compute. For this reason, we include two metrics, SacreBLEU and ROUGE-L, for evaluating the RAG-generated answer. As we do not have human-labeled answers in MirageBench, we consider the GPT-4 generated answer as the gold truth for evaluation.

**6. Answer overlap (LLM-measured):** To capture semantic overlap between answers, we use the Llama-3 (8B) model as the judge for evaluation in a pointwise setup, where the LLM as a judge outputs a score between 1 to 5.

**7. Fluency (LLM-measured):** Fluency measures for grammatical correctness and idiomatic word choices in long-form answer generation. Evaluating fluency in multilingual long-form generation answers is not straightforward. While existing techniques are available for English such as MAUVE Pillutla et al. (2021), only a few models evaluate multilingual summarization Clark et al. (2023). Inspired by recent works in G-EVAL Liu et al. (2023), we evaluate fluency using open-source LLM such as Llama-3 (8B) as the judge. Our reason for choosing open-source models lies in reducing the expense, of running an expensive proprietary LLM such as GPT-4. Our LLM as a judge setup outputs a score between 1 to 5.

## Appendix B Baselines: Additional Details

In this section, we briefly describe each of the 19 multilingual-focused models utilized in our Mirage-Bench evaluation experiments:

1. [leftmargin=*]
2. **GPT-3.5-turbo:**OpenAI, 2023) is evaluated using the Azure OpenAI service.9 We set the temperature parameter to \(0.1\) for a deterministic output. It utilizes the cl100k_base BPE-based tokenizer in the tiktoken10 repository. Footnote 9: [https://learn.microsoft.com/en-us/azure/ai-services/openai/](https://learn.microsoft.com/en-us/azure/ai-services/openai/)
3. **GPT-4:**OpenAI, 2023) is also evaluated using the Azure OpenAI service. We use a temperature setting of \(0.1\) for a deterministic output and the cl100k_base BPE-based tokenizer.

 

3. **GPT-4o:**(OpenAI, 2023) is also evaluated using the Azure OpenAI service. We use a temperature setting of \(0.1\) for a deterministic output and the o200k_base BPE-based tokenizer.
4. **Mistral-7B-Instruct-v0.2**(Jiang et al., 2023) is the v0.2 of the instruct-version model containing 7B parameters.11 It is an English-centric model, i.e., not instruction fine-tuned with any multilingual data. We used the multiple GPU inference using the vllm repository Kwon et al. (2023). We set the temperature parameter to 0.1.
5. **Mistral-7B-Instruct-v0.3**(Jiang et al., 2023) is an extension of the instruct-version 0.2 model containing 7B parameters.12 We set inference parameters similar to the previous model. Footnote 11: mistralai/Mistral-7B-Instruct-v0.2 mistralai/Mistral-7B-Instruct-v0.3 mistralai/Mistral-8x7B-Instruct-v0.1 mistralai/Mistral-8x22B-Instruct-v0.1
6. **Mistral-8\(\times\)7B-Instruct-v0.1**(Jiang et al., 2024) is a pretrained generative sparse Mixture of Experts (MoE), containing 8\(\times\)7B parameters. It has been pretrained in 5 languages including English, French, Italian, German, and Spanish.13 As the model is computationally not feasible to evaluate due to resource constraints, We use the model API endpoint available in the Anyscale platform ([https://www.anyscale.com/](https://www.anyscale.com/)), with a temperature setting of 0.1.
7. **Mistral-8\(\times\)22B-Instruct-v0.1**(Jiang et al., 2024) is a pretrained generative sparse Mixture of Experts (MoE), containing 8\(\times\)22B parameters. Similar to before, it has pretrained on 5 languages including English, French, Italian, German, and Spanish.14 We utilize the model API endpoint available in the Anyscale platform ([https://www.anyscale.com/](https://www.anyscale.com/)), with a temperature setting of 0.1. Footnote 11: CabereForAll/cdai-command-r-plus
8. **Command R** is developed keeping RAG in mind and officially supports 11 languages: Arabic, Brazilian, Portuguese, English, French, German, Italian, Japanese, Korean, Chinese, and Spanish. The model contains 35 billion parameters.15 We utilize the model API available in the Cohere platform ([https://cohere.com/](https://cohere.com/)), with a temperature setting of 0.1, and using the chat template format. Footnote 11: CabèreForAll/cdai-command-r-plus
9. **Command R+** is also developed keeping RAG in mind and officially supports 10 languages: English, French, Spanish, Italian, German, Brazilian Portuguese, Japanese, Korean, Arabic, and Chinese. The model contains 105 billion parameters.16 We utilize the model API available in the Cohere platform ([https://cohere.com/](https://cohere.com/)), with a temperature setting of 0.1, and using the chat template format. Footnote 17: CabèreForAll/aya-23-3EB
10. **Aya-23-35B**(Araybumi et al., 2024) is an instruction fine-tuned model with highly advanced multilingual capabilities. The model officially supports 23 languages: Arabic, Chinese (simplified & traditional), Czech, Dutch, English, French, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Korean, Persian, Polish, Portuguese, Romanian, Russian, Spanish, Turkish, Ukrainian, and Vietnamese. The model contains 35 billion parameters.17 We utilize the model API available in the Cohere platform ([https://cohere.com/](https://cohere.com/)), with a temperature setting of 0.1, and using the chat template format. Footnote 18: google/gamma-1.1–2b-it
11. **Gamma 1.1 (2B) it**(Mesnard et al., 2024) is an instruction fine-tuned model trained using the RLHF method containing 2 billion parameters.19 We used the multiple GPU inference using the vl1lm repository Kwon et al. (2023). We set the temperature parameter to 0.1. Footnote 19: google/gamma-1.1-7b-it
12. **Gamma 1.1 (7B) it**(Mesnard et al., 2024) is an instruction fine-tuned model trained using the RLHF method containing 7 billion parameters.20 We used the multiple GPU inference using the vl1lm repository Kwon et al. (2023). We set the temperature parameter to 0.1. Footnote 20: meta-lama/Meta-Llama-3-8B-Instruct
13. **Meta-Llama-3-70B-Instruct**(Dubey et al., 2024) is an English-only instruction fine-tuned model containing 8 billion parameters.21 We used the multiple GPU inference using the vl1m repository Kwon et al. (2023). We set the temperature parameter to 0.1. Footnote 21: meta-lama/Meta-Llama-3-70B-Instruct
14. **Meta-Llama-3-70B-Instruct**(Dubey et al., 2024) is an instruction fine-tuned model containing 70B parameters.22 As the model is computationally not feasible to evaluate due to resource constraints, We use the model API endpoint available in the Anyscale platform ([https://www.anyscale.com/](https://www.anyscale.com/)), with a temperature setting of 0.1. Footnote 22: meta-lama/Meta-Llama-3-8B-Instruct
15. **Phi-3 (mini)**(Abdin et al., 2024) is an English-focused instruction fine-tuned model trained 

model containing 3.8 billion parameters.22 We used the multiple GPU inference using the vllm repository Kwon et al. (2023). We set the temperature parameter to 0.1. Footnote 22: microsoft/Phi-3-mini-128k-instruct
* **Phi-3 (small)**Abdin et al. (2024) is a multilingual instruction fine-tuned model trained model containing 8 billion parameters.23 There is no available information on the number of languages covered by the model. We used the multiple GPU inference using the vlln repository Kwon et al. (2023). We set the temperature parameter to 0.1. Footnote 23: microsoft/Phi-3-small-8k-instruct
* **Phi-3 (medium)**Abdin et al. (2024) is a multilingual instruction fine-tuned model trained model containing 14 billion parameters.24 There is no available information on the number of languages covered by the model. We used the multiple GPU inference using the vlln repository Kwon et al. (2023). We set the temperature parameter to 0.1. Footnote 25: Qwen2-7B-Instruct
* **Qwen2-1.5B-Instruct**Yang et al. (2024a) is an English-focused instruction fine-tuned model trained model containing 1.5 billion parameters.26 We used the multiple GPU inference using the vllm repository Kwon et al. (2023). We set the temperature parameter to 0.1. Footnote 25: Qwen/Qwen2-1.5B-Instruct
* **Qwen2-7B-Instruct**Yang et al. (2024a) is an English-focused instruction fine-tuned model trained model containing 7 billion parameters.26 We used the multiple GPU inference using the vllm repository Kwon et al. (2023). We set the temperature parameter to 0.1. Footnote 26: Qwen/Qwen2-7B-Instruct

## Appendix C MIRAGE Fine-tuning Details

For multilingual RAG fine-tuning, we use teacher models to _distill_ synthetic knowledge directly within smaller open-source models. We first generate RAG outputs on the Mirage-Bench training dataset using three high-performing teacher models: (i) GPT-4o, (ii) Llama-3 (70B), and (iii) Mixtral (8\(\times\)22B), and generate RAG output for queries in Mirage-Bench training dataset using only relevant passages, i.e., without distracting the model with information from non-relevant passages. We filter out the teacher model responses and curate them to create the training dataset.

Next, using supervised fine-tuning (SFT) with LoRA Hu et al. (2022), we fine-tune two open-source models: (i) Llama-3 (8B) and (ii) Mistralv0.2 (7B). Our hyperparameter choices are listed in Table 6. We use PEFT Mangrulkar et al. (2022) and the alignment-handbook27 Tunstall et al. (2023) for supervised LoRA fine-tuning. We finetune four variants of models: (i) Mistral-v0.2 (7B) distilled using GPT-4o as a teacher, (ii) Mistral-v0.2 (7B) distilled using Mikxtral (8\(\times\)22B) and itself as a teacher, (iv) Lima-3 (8B) distilled using GPT-4o as a teacher, and (iv) Lima-3 (8B) distilled using Lima-3 (70B) and itself as a teacher. After fine-tuning, first all heuristic features are computed, using the already trained learning to rank model (using the baselines) is used to compute inference for the fine-tuned models and compared against upper-bound baselines, GPT-4o, Llama-3 (70B), and Mixtral (8\(\times\)22B) and lower-bound baselines, Mistral-v0.2 (7B) and Lima-3 (8B).

Footnote 27: [https://github.com/huggingface/alignment-handbook](https://github.com/huggingface/alignment-handbook)

Footnote 28: meta-llama/Meta-Llama-3.1-SB-Instruct

### Extending MIRAGE Evaluation

As a holdout experiment, we evaluate newer versions of models, (i) Llama-3.1 series Dubey et al. (2024): Llama-3.1 (8B)29 and Llama-3.1 (70B)30 instruct versions, and (ii) Gemma-2 series Team et al. (2024): Gemma-2 (9B)31 and Gamma-2 (27B)31 instruct versions. For both models, we used the API versions of the model provided by NVIDIA ([https://build.nvidia.com/](https://build.nvidia.com/)) by setting the temperature parameter to 0.1. The maximum sequence length of Gemma-2 models is 4096 tokens.

Footnote 29: meta-llama/Meta-Llama-3.1-70B-Instruct

Footnote 30: pagele/gemma-2-9b-it

**Experimental results.** From Figure 8, we observe that the Gemma-2 (27B) and Llama-3.1 (70B) are strong baselines, by achieving an overall rank of 4 and 5 in the Mirage-Bench dataset. German-2 (27B) improves the previously best Lemma-1.1 (7B) by 13 ranks, whereas Llama-3.1 (70B) continues to underperform the best Llama-3 (70B) by 2 ranks. These results indicate newer models are improving, as reported using the surrogate judge on the synthetic Mirage-Bench leaderboard.

 



\begin{table}
\begin{tabular}{c c} \hline \hline
**Hyper-parameter** & **Choice** \\ \hline Attention & FlashAttention-2 (Dao, 2024) \\ Batch Size & 32 \\ Epochs & 1 \\ Learning Rate & 2e-4 \\ Max Sequence Length & 6144 \\ Lora rank (_r_) & 16 \\ Lora alpha (\(\alpha\)) & 16 \\ Lora dropout & 0.05 \\ Lora Modules & [q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj] \\ \hline \hline \end{tabular}
\end{table}
Table 6: Hyperparameter settings set during supervised fine-tuning of Mistral-v0.2 (7B) and Llama-3 (8B) on the Mirage-Bench training dataset.

Figure 8: Approximate rankings using heuristic features including the newer models, Llama-3.1 (Dubey et al., 2024) and Gemma-2 (Team et al., 2024) on Mirage-Bench dataset across all 18 languages. Gemma-2 (27B) and Llama-3.1 (7OB) achieve a strong rank of 4 and 5 respectively in the Mirage-Bench evaluation dataset.





## References

* [1] A. B. Krizhevsky, S. M. Girshick, and J. L. Friedman, "Theoretical Computer Science of the New York: The Case of Data Mining," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 249-258, 2003.
* [2] D. C. Bradley, R. G. Davis, and P. W. Anderson, "A survey of machine learning algorithms for image recognition," in _IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 67-74, 2003.
* [3] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-113, 2003.
* [4] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-113, 2003.
* [5] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [6] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [7] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [8] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [9] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [10] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [11] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [12] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [13] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [14] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [15] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [16] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [17] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [18] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [19] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [20] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [21] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [22] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [23] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [24] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [25] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [26] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [27] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [28] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [29] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [30] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [31] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [32] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [33] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [34] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [35] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.
* [36] D. C. Bradley, R. G. Davis, and P. W. Anderson, "Image classification with deep convolutional neural networks," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 101-116, 2003.





## References

* [1] A. B. Klimp, M. C. R. Douglas and J. H. Schwarz, Phys. Rev. D **48**, 2357 (1993); A. B. Klimp, M. C. R. Douglas and J. H. Schwarz, Phys. Rev. Lett. **60**, 1239 (1993); A. B. Klimp, M. C. R. Douglas and J. H. Schwarz, Phys. Rev. Lett. **60**, 1239 (1993); A. B. Klimp, M. C. R. Douglas and J. H. Schwarz, Phys. Rev. Lett. **60**, 1239 (1993); A. B. Klimp, M. C. R. Douglas and J. H. Schwarz, Phys. Rev. Lett. **60**, 1249 (1993); A. B. Klimp, M. C. R. Douglas and J. H. Schwarz, Phys. Rev. Lett. **60**, 1259 (1993); A. B. Klimp, M. C. R. Douglas and J. H. Schwarz, Phys. Rev. Lett. **60**, 1269 (1993); A. B. Klimp, M. C. R. Douglas and J. H. Schwarz, Phys. Rev. Lett. **60**, 1279 (1993); A. B. Klimp, M. C. R. Douglas and J. H. Schwarz, Phys. Rev. Lett. **60**, 1289 (1993); A. B. Klimp, M. C. R. Douglas and J. H. Schwarz, Phys. Rev. Lett. **60**, 129 (1993); A.





## Appendix A

Figure 9: Lollipop plots denoting the average heuristic-based feature scores achieved by baselines in MirageBench for all eleven heurisitc-based features. \(x\)-axis denotes the languages in Mirage-Bench; whereas \(y\)-axis plots every heuristic feature value. Multiple LLMs in the same family are represented as a single color lollipop (multiple circles).



Question: What was the first newspaper ever printed in the U.K.?

* The very first newspaperser to be published in North America by the Icelandic immigrant population was handwritten by Jon Gudmundsson in 1876 ... "[1956416#2]" The New York Times Magazine
- Its first issue was published on September 6, 1896, and contained the first photographs ever printed in the newspaper ... "[662134#6]" Letterpress printing
- Letterpress printing was introduced in Canada in 1752 in Halifax, Nova Scotia by John Bushell in the newspaper format. This paper was named the Halifax Gazette and became Canada's first newspaper ....

...

"[22112840#15]" Newspaper - The emergence of the new media in the 17th century has to be seen in close connection with the spread of the printing press from which the publishing press derives its name.....

Instruction: Provide an answer to the question using the information provided in contexts written in ({language}). Additionally, provide a step-by-step explanation of your reasoning, demonstrating how you arrived at your answer in ({language}). Cite parts of your reasoning within brackets [] using the IEEE format based on the provided contexts. Please respond in ({language}) using the format: ##Reason: {reason} ##Answer: {answer}.

Figure 11: Prompt template used by Llama-3 (8B) model as a judge to evaluate the answer overlap heuristic feature. We include a grading rubric within the prompt template.[ {Label}] is a placeholder for the gold truth answer provided using the GPT-4; {[ language}] is a placeholder for the target language; {{Question}} is a placeholder for the Mirage-Bench query; {{Documents}} is a placeholder for both Mirage-Bench relevant and non-relevant passages concatenated together; {{Response}} is a placeholder for RAG model output.

 

You will be given one summary written for a question and documents from Wikipedia in ({language}). Your task is to rate the summary on one metric. Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.

Evaluation Criteria: Fluency (1-5) - the collective quality of all sentences. We align this dimension with the DUC quality question of structure and fluency whereby "the summary should be well-structured and well-organized. The summary should not just be a heap of related information, but should build from sentence to sentence to a coherent body of information about a topic." Evaluation Steps:  1. Read the question and Wikipedia documents in ({language}) carefully and identify the main topic and key points.  2. Read the summary and check whether it answers the question. Check if the summary covers the main topic and key points required to answer the question, and if it presents them in a clear and logical order.  3. Assign a rating for fluency on a scale of 1 to 5 and provide an explanation, where 1 is the lowest and 5 is the highest based on the Evaluation Criteria. Example: Question in [{language}]: {Question} Documents in ({language}): {Documents} Summary: {Summary} Rate the fluency of the summary on a scale of 1 to 5 and explain your rating. Please use the format of: #Rating: {rating} ##Explanation: {explanation}. Figure 12: Prompt template used by Llama-3 (8B) model as a judge to evaluate the fluency of a RAG response. We first explain the criteria for evaluation and the model outputs an explanation and score between [1,5] indicating the fluency of the output. { [ language] } is a placeholder for the target language; {{Question} } is a placeholder for the Mirage-Bench query; {{Documents} } is a placeholder for both Mirage-Bench relevant and non-relevant passages concatenated together; {{Summary} } is a placeholder for RAG model output.

 

Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants tasked to answer the question displayed below, based on a set of documents retrieved by a search engine. You should choose the assistant that best answers the user question based on a set of reference documents that may or not be relevant referenced in the IEEE format. Your evaluation should consider factors such as the correctness, helpfulness, completeness, accuracy, depth, and level of detail of their responses. Details are only useful if they answer the user's question. If an answer contains non-relevant details, it should not be preferred over one that only uses relevant information. Begin your evaluation by explaining why each answer correctly answers the user question. Then, you should compare the two responses and provide a short explanation on their differences. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Be as objective as possible. After providing your explanation, output your final verdict by strictly following this format: "[L[A]]" if assistant A is better, "[L[B]]" if assistant B is better, and "[L[C]]" for a tie.

"[User Question]" {query} "[Reference Documents]" {documents}

"[The Start of Assistant A's Answer]" {answer_a} "[The End of Assistant A's Answer]"

"[The Start of Assistant B's Answer]" {answer_b} "[The End of Assistant B's Answer]"

Figure 13: Prompt template used by LLM as a judge to evaluate the RAG response in a pairwise evaluation involving a head-to-head battle. The template is taken and modified from RAGEval (Rackauckas et al., 2024). We explain the evaluation criteria and ask the judge to evaluate two RAG responses based on multiple factors, including correctness, helpfulness, completeness, accuracy, depth, and level of detail. The Judge provides a justification for their model choice and at the end of the response indicates as either “[I[A]]”, “[I[B]]”, or “[I[C]]” denoting a tie. {query} is a placeholder for the input Mirage-Bench query; {{documents}} is a placeholder for both Mirage-Bench relevant and non-relevant passages concatenated together; {{answer_a}} is a placeholder for the output response of model A; {answer_b} is a placeholder for the output response of model B.