Introduction
Large Language Models (LLMs) have recently gained popularity for information-seeking queries leading to the widespread adoption of Retrieval-Augmented Generation (RAG) Guu et al. ( 2020 ); Lewis et al. ( 2020 ); Izacard and Grave ( 2021 ); Borgeaud et al. ( 2022 ) .
The naive RAG setup traditionally includes a retrieval and a generation stage, conducted sequentially. RAG systems such as Bing Search Microsoft ( 2023 ) provide grounded responses, i.e., statments include citations to one or more retrieved passages. The citations reduce factual hallucinations with easy verifiability and improve support or faithfulness to passages provided within context Khandelwal et al. ( 2020 ); Lewis et al. ( 2020 ); Gao et al. ( 2023a , b ); Liu et al. ( 2024 ) . However, existing RAG benchmarks are English-centric, due to uneven and scarce data available across multiple languages Thakur et al. ( 2024b ) . So far, it is unclear how well existing LLMs perform in multilingual RAG, i.e., where queries and passages are non-English and the LLM generates a response in the same language.
An example of RAG in Hindi language (hn) is shown in Figure 1 .

Existing RAG benchmarks can be broadly classified as either (i) heuristic-based , where benchmarks design multiple evaluation metrics to evaluate systems across multiple dimensions Gao et al. ( 2023a ); Chen et al. ( 2024c ) or (ii) arena-based : where systems compete each other in a tournament and an LLM-based teacher is used as the judge Rackauckas et al. ( 2024 ); Pradeep et al. ( 2024 ) .
Heuristic-based benchmarks are computationally cheaper to evaluate but require human preferences as gold truth for reference.
They also face challenges in aggregating different metrics into a ranking order for models.
On the other hand, arena-based benchmarks require a high performance LLM as a teacher Zheng et al. ( 2023 ) , which makes direct pairwise comparisons expensive for a large set of models. For example, using GPT-4o OpenAI ( 2023 ) to exhaustively evaluate a single query on 19 models requires \binom{19}{2}=171 comparisons and costs between $5 and $10 USD.

In our work, we get the best of both worlds by training a surrogate judge, a learning to rank model, e.g. random forest Ho ( 1995 ) , using heuristic features to estimate an arena-based leaderboard obtained with a Bradley-Terry model Hunter ( 2004 ) from pairwise evaluations using an LLM as a judge Zheng et al. ( 2023 ) .
We use boostrapping to obtain confidence bounds for better statistical estimates.
After training, the learning to rank model can be used to estimate the performance of newer released models relaibly in the future without the expensive LLM as a judge. It also provides better interpretability and is easily retrainable with a different or newer set of heuristic features.

We develop \mirage , a RAG benchmark across 18 languages for multilingual generation evaluation on Wikipedia. \mirage is constructed from miracl Zhang et al. ( 2023 ) , a retrieval dataset containing human-generated queries and human-labeled relevance judgments for Wikipedia passages. We benchmark 19 of the latest multilingual LLMs for use in multilingual RAG settings. Our evaluation flowchart adopted in \mirage is listed in Figure 2 .
We evaluate seven heuristic features: (i) language detection, (ii) citation quality, (iii) support, (iv) reranker score, (v) answer overlap (traditional), (vi) answer overlap (LLM-measured), and (vii) fluency (LLM-measured). Next, we use GPT-4o as a Judge OpenAI ( 2023 ) to evaluate our head-on pairwise RAG comparisons on a sampled set of a 100 queries for all languages.
After this, we train a learning to rank model using random forest for each language, using the heuristic features as input and learn to output the Bradley-Terry logits as output. After training, we generate a “synthetic” arena-based leaderboard using the random forest model as a surrogate judge.

Our experimental results show that: (i) learning-to-rank model correlates well with GPT-4o arena-based leaderboards, achieving an average Kendall-Tau ( \tau ) score of 0.909. (ii) Proprietary and large (> 70B parameters) open-source models dominate the generation task by achieving the top-ranks in the \mirage leaderboard. (iii) \mirage training data, synthetically constructed using strong teachers such as GPT-4o, is beneficial to improve smaller (7 or 8B parameters) open-source models on \mirage .

The main contributions of our work are (i) building \mirage , a benchmark to foster research towards multilingual RAG, incorporating a heuristic-based evaluation with an arena-based leaderboard using a trainable learning to rank model as a surrogate judge, and (ii) benchmarking 19 diverse multilingual-focused LLMs for the generation task in multilingual RAG.