RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Related Work
Prior work on RAG evaluation has been conducted exclusively in English. For example,benchmarks such as ALCE Gao et al. ( 2023a ) , FreshLLM Vu et al. ( 2024 ) , ClapNQ Rosenthal et al. ( 2024 ) , HAGRID Kamalloo et al. ( 2023 ) and CRAG Yang et al. ( 2024b ) , all include long-form answers for English-only queries and are based on collections containing documents from either the English Wikipedia, MS MARCO Bajaj et al. ( 2016 ) or NQ Kwiatkowski et al. ( 2019 ) . Similarly, TREC 2024 RAG, an ongoing TREC competition for RAG evaluation is focused on English. 2 2 2 TREC 2024 RAG track: https://trec-rag.github.io/

Multilingual RAG. On the multilingual side, RAG has not been well studied in prior literature. The RGB benchmark Chen et al. ( 2024c ) , is limited in language scope as it covers only one additional language: Chinese (zh). NeuCLIR Mayfield et al. ( 2024 ) evaluates long-form report generation from participants in the upcoming 2024 track; but is limited to three languages. A concurrent work is BERGEN Chirkova et al. ( 2024 ) , which evaluates multilingual open-domain QA settings across 13 languages. In contrast, \mirage evaluates the generation task 3 3 3 As \mirage is built from miracl , in our work we focus on the generation task in RAG using oracle retrieved passages, inspired by the “AG track” in the TREC 2024 RAG competition ( https://trec-rag.github.io/ ). in the multilingual RAG pipeline on 18 languages, provides multilingual instruction-tuned data for RAG fine-tuning, and evaluates on high-quality queries in miracl .

Learning to rank. It is a supervised learning technique Liu ( 2010 ) , where models are trained to provide an ordering between items in each list. The goal of constructing the ranking model is to rank new, unseen lists in a similar way to rankings in the training data Turnbull ( 2017 ) . Models are trained in either a pointwise, pairwise or listwise objective Cao et al. ( 2007 ) . In our work, we experiment with simpler approaches such as random forest as a surrogate judge to successfully learn to Bradley-Terry model coefficient produced by LLM as a judge. We keep experimentation with complex approaches such as LambdaMART Burges ( 2010 ) as a surrogate judge for future work.
\mirage : A Multilingual RAG Benchmark
We select 18 languages in \mirage as the starting point, representing an appropriate cross-section of the diversity of the languages spoken worldwide at this point. \mirage serves as a comprehensive multilingual RAG benchmark focusing on the generation task evaluation.
As shown in Table 1 , \mirage includes 11,195 evaluation pairs and 39,763 training pairs across 18 languages. We discuss the \mirage evaluation and training datasets and highlight differences from the miracl dataset below:
\mirage Evaluation Dataset
miracl introduced in Zhang et al. ( 2023 ) , is a monolingual retrieval dataset, i.e., queries and passages are both in the same language for passage retrieval. Queries are high-quality and generated by native speakers Zhang et al. ( 2023 ) . The annotation procedure in miracl is identical to TyDI-QA Clark et al. ( 2020 ) . The passage collection is constructed from language-specific Wikipedia corpora and parsed using WikiExtractor.
The \mirage evaluation dataset is constructed re-using the queries and oracle-judged passages available in the miracl development split. 4 4 4 We did not utilize the test split in miracl as the relevance judgments are not publicly available.

We incorporate two changes: (i) In Arabic (ar), we randomly sample a smaller subset of 1,501 out of 2,896 queries for uniformity in the number of pairs available across other languages. (ii) We filter out queries with zero non-relevant passages, i.e., we always include non-relevant passages, i.e., hard negatives, from miracl to make the \mirage evaluation task challenging. 5 5 5 An exception here is Telugu (te), where only 78 queries have at least a single non-relevant passage, we sample and include 72 additional queries with only relevant passages.
\mirage Training Dataset
The \mirage training dataset is developed from the miracl Zhang et al. ( 2023 ) training dataset. miracl only contains information on queries and relevant and non-relevant passages. In \mirage , we reuse all the miracl training pairs available in 16 languages, except German (de) and Yoruba (yo). We convert the training dataset using a simple recipe into a multilingual instruction-tuned RAG dataset Zhang et al. ( 2024 ); Niu et al. ( 2024 ) . In our recipe, we filter the relevant passages, and keep them along with the input query to generate a zero-shot RAG output using strong teachers available including GPT-4o OpenAI ( 2023 ) , Llama 3 (70B) Dubey et al. ( 2024 ) and Mixtral (8x22B) Jiang et al. ( 2024 ) . After generation, we include non-relevant passages within our prompt as “distracting and noisy information”, to help improve the quality of the \mirage training dataset. Note that, since we convert a retrieval dataset, we do not have human-annotated answers for questions in \mirage .
Distinction and Extension from MIRACL
miracl introduced in Zhang et al. ( 2023 ) is a monolingual retrieval dataset, which evaluates the “retrieval task”, i.e. given a user query and a Wikipedia corpus, miracl contains human-annotated relevance judgements to evaluate for query-passage level relevancy using retrieval models, e.g., sparse models like BM25 Robertson and Zaragoza ( 2009 ) or bi-encoders like mDPR Karpukhin et al. ( 2020 ) , or late-interaction models like ColBERT Khattab and Zaharia ( 2020 ) .

In contrast, \mirage evaluates the “generation task” in RAG, requiring LLMs to generate a summarized answer given the query and context available from retrieved passages. In our work, we re-use the queries and oracle relevance-judgments from miracl and focus solely on evaluating the generation task in RAG measuring answers using both heuristics and LLM as a judge.
Heuristic-based Evaluation
Multilingual generation in RAG requires the evaluation of multiple dimensions. For example, whether a system’s response provides the correct final answer or cites the relevant documents, a single metric alone is not sufficient to capture the comprehensive evaluation required for RAG systems. Inspired by other recent works Kiela et al. ( 2021 ); Santhanam et al. ( 2023 ); Gao et al. ( 2023a ) , we introduce five deterministic features and two LLM as a judge features for evaluation. We additionally provide details about each feature in Appendix A .

Language detection. We compute the probability of a system’s response in the required target language with langid Lui and Baldwin ( 2012 ) . We compute two metrics: language detection (target language) and English detection.

Citation quality. Using passage-level relevance judgments for all queries (or qrels) information available in miracl , we evaluate whether the system’s response cites the relevant passages, crucial for measuring faithfulness. We compute and evaluate: Recall @k and MAP @k , where k = 10.

Support. Grounding is necessary to avoid hallucinations in the system’s response. Support evaluation Gao et al. ( 2023a ) checks whether each sentence is supported by cited passages using a multilingual NLI model. 6 6 6 Fine-tuned mDeBERTa-v3-base XNLI model He et al. ( 2023 ) : MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 We compute the probability of the entailment and neutral score, macro-averaged across the sentence-citation pairs.

Reranker score. The reranker score measures the average similarity (can be greater than 1.0) between the query and the passages cited within the system’s response. We compute the reranker score using a multilingual reranker model, 7 7 7 Reranker Chen et al. ( 2024b ) : BAAI/bge-reranker-v2-m3 macro-averaged across the query-passage pairs.

Answer overlap. Having the correct answer is crucial in the RAG system’s response. Since \mirage does not include a human-labeled answer, we use the generated answer from GPT-4 OpenAI ( 2023 ) as the gold truth. We compute two traditional answer overlap metrics: SacreBLEU Papineni et al. ( 2002 ) and ROUGE-L Lin ( 2004 ) to measure the lexical word overlap between the gold truth answer (here, GPT-4 answer) and the system’s response.

Answer overlap (LLM-measured). In addition, we evaluate using Llama-3 (8B) Dubey et al. ( 2024 ) , an open-source LLM as a judge evaluator in a pointwise setup, providing a semantic word overlap integer score in the range [1,5] . The answer overlap prompt description is listed in Figure 11 .

Fluency (LLM-measured). It measures for grammatical correctness and idiomatic word choices in the system’s response. As previously mentioned, we use Llama-3 (8B) Dubey et al. ( 2024 ) in a pointwise setup, proving an integer score in [1,5] .
The fluency prompt description is listed in Figure 12 .
Arena-based Evaluation
Heuristic-based evaluation metrics often rely on a gold standard for evaluation. Tasks such as text retrieval Bajaj et al. ( 2016 ); Thakur et al. ( 2021 ) require human-labeled relevance judgments, and similarly, NLP tasks such as machine translation Stahlberg ( 2020 ) , require human-annotated translations. As human preferences are seldom available in numerous applications, using LLMs as a judge Zheng et al. ( 2023 ); Chen et al. ( 2024a ); Chiang et al. ( 2024 ) is becoming a defacto approach for arena-based evaluation of models.

Pairwise LLM as a judge. Following prior works on arena-based evaluation in RAG Rackauckas et al. ( 2024 ); Pradeep et al. ( 2024 ) , we evaluate two system’s responses in a head-on battle by computing pairwise judgments with LLM as a judge. We reuse the prompt template from RAGElo Rackauckas et al. ( 2024 ) with minor additions. LLM as a judge evaluator includes three types of biases Zheng et al. ( 2023 ) : (i) verbosity bias Wu and Aji ( 2023 ) (ii) self-enhancement bias Xu et al. ( 2024 ); Panickssery et al. ( 2024 ) and (iii) position bias Wang et al. ( 2024 ) . We successfully avoid the verbosity bias, as the RAG evaluation has fixed evaluation criteria requiring sentence-level citations and answers Pradeep et al. ( 2024 ) and the position bias by randomly swapping the position of two models. Computing exhaustive pairwise comparisons with GPT-4o OpenAI ( 2023 ) as a teacher is expensive, we keep it as future work to include better and more teachers to eliminate the self-enhancement bias.
Learning to Approximate Rankings
There is no predefined way to aggregate the heuristic features to provide an overall leaderboard ranking in \mirage . Averaging the scores is too simplistic, as features measure different aspects of RAG evaluation. On the other hand, arena-based evaluations provide ranked leaderboards but are computationally expensive to compute with a strong teacher model. To avoid computational costs, smaller models as teachers have been proposed Thakur et al. ( 2024a ); Ni et al. ( 2024 ) . Motivated by similar observations, we train a learning to rank model as a surrogate judge to approximate the Bradley-Terry model coefficients Hunter ( 2004 ) learned from an arena-based evaluation that uses GPT-4o for pairwise judgments.

Regression model. While the heuristic RAG features introduced in Section 4.1 can be computed efficiently and without the reliance on proprietary models, inducing a ranking from pairwise comparisons via a Bradley-Terry model is computationally expensive and requires access to a high-performance LLM. Furthermore, in Section 6.3 , we demonstrate that the ranking accuracy, measured by the Kendall-Tau ( \tau ) coefficient, degrades rapidly when subsampling tournament matches. Therefore we investigate whether a regression model can be successfully trained to approximate the Bradley-Terry logits using heuristic features.

Note: Refer to Section 4.3 for a definition of each of the variables. The Tournament function runs a battle arena, sampling q queries, and returning the learned Bradley-Terry model. The Dataset function accepts a set of LLMs and a learned Bradley-Terry model. It returns X , the heuristic RAG feature values, and Y , the Bradley-Terry coefficients for each LLM model. After simulating N_{t} tournaments, the array R_{2} contains the R^{2} errors for each of the N_{t} regression models.

The procedure, detailed in Algorithm 1 , simulates N_{t} tournaments, each involving a total of N_{l} models and N_{q} queries. For each query, judgments are obtained for all \binom{N_{l}}{2} pairings of models. We employ bootstrapping on the query selection process to estimate the variance in the R^{2} metric in the regression models’ approximations of the Bradley-Terry coefficients over a randomly-sampled holdout set, \textsc{llm}_{predict} .

We randomly selected two models, Gemma 1.1 (2B) as Llama-3 (70B) as holdout models. For English, we observed an average \bar{R}^{2}=0.971 with a 95% confidence interval of [0.905,0.999] , while for Bengali, we observed \bar{R}^{2}=0.937 with a 95% confidence interval of [0.766,0.998] . All scores with 95% confidence intervals are listed in Table 5 . Taken together, these results indicate that the training procedure is fairly robust with N_{q}=100 .
Multilingual Baselines
Existing state-of-the-art LLMs are either English-only or support a limited set of languages, predominantly due to the curse of multilinguality for large models Conneau et al. ( 2020 ) , i.e., it is unclear how well-existing LLMs perform on RAG across a wide variety of languages, due to scarce availability of multilingual instruction tuning datasets. We experiment with models from seven different families, containing proprietary and open-sourced models. Wherever possible, we benchmark the instruction-tuned version if available. Please refer to Appendix B for more details on baselines.

OpenAI: We evaluate the GPT-3.5-turbo, GPT-4, and GPT-4o models OpenAI ( 2023 ) using Azure OpenAI service.

Mistral: We evaluate the Mistral-Instruct-v0.2 and v0.3 (7B) Jiang et al. ( 2023 ) , Mixtral (8x7B) and Mixtral (8x22B) Jiang et al. ( 2024 ) .

Cohere: We evaluate the Command-R (35B), Command-R+ (104B) and Aya-23 (35B) models Aryabumi et al. ( 2024 ) .

Gemma: We evaluate the Gemma 1.1 instruct (2B) and (7B) models Mesnard et al. ( 2024 ) .

Llama-3: We evaluate the Llama-3 instruct (8B) and (70B) models Dubey et al. ( 2024 ) .

Phi-3: We evaluate all models from the Phi 3 instruct series: Medium (14B), Small (7B), and Mini (3.8B) Abdin et al. ( 2024 ) .

Qwen-2: We evaluate two versions of the Qwen-2 model: 1.5B and 7B Yang et al. ( 2024a ) .
Evaluation Details
Prompt template. We internally optimized 8 8 8 A majority of the prompt optimization was internal and based on eye-balling RAG responses across LLMs. the ChatQA prompt template Liu et al. ( 2024 ) , to include in-text citations of the context passages following the IEEE format Kamalloo et al. ( 2023 ) . In \mirage , we have about 10 passages annotated in the oracle setting. Therefore, we trim each passage available and take the first 800 tokens to fit all passages within a fixed context length of 8192 tokens. following prior work in Shi et al. ( 2023 ) , the prompt requires the LLM to explain the multilingual generation task starting with “ ##Reason ” and the answer itself starting with “ ##Answer ”. Utilizing this output format has its advantages in easily parsing the generated answer and the rationale behind the answer. The prompt template is shown in Figure 10 . For GPT-4o as a judge pairwise evaluation, we modified the prompt template available in RAGEval Rackauckas et al. ( 2024 ) . The prompt template is shown in Figure 13 .
Heuristic-based Results
Figure 3 shows lollipop plots indicating the average heuristic-feature value ( y -axis) distribution across all languages ( x -axis). In English detection, smaller LLMs such as Gemma-1.1 (2B) do not generate output in the required target language, but rather rely on English. Next, for citation quality and support evaluation, OpenAI models achieve better Recall @10 and Entailment scores (except Llama-3 (70B) for a few languages), indicating baseline responses include grounded citations from relevant passages. In contrast, models such as Qwen-2 or Gemma-1.1 tend to under-cite in their response. Similar trends are observed for the reranker score.

Furthermore, we observe OpenAI models achieve a higher word overlap in the ROUGE-L metric (GPT-4 used as ground truth) and for Llama-3 (8B) as a judge, we observe rather less variance in scores across models. In Fluency, we observe a majority of the baselines are rather fluent, except Bengali (bn), Telugu (te), and Yoruba (yo).
Arena-based Results
Figure 4 (left) shows the arena-based leaderboard using bootstrapping and Bradley-Terry modeling after conducting 200 tournaments and sampling 100 matches per tournament on a subset of 100 queries using GPT-4o pairwise comparisons. We observe that proprietary models such as GPT-4o and GPT-4, and larger models such as LLAMA-3 (70B) and Mixtral (8x22B), are slightly better than other baselines on \mirage . Baseline rankings across languages are usually stable; with a few notable exceptions such as Gemma-1.1 (7B) which achieves a rank of 4 in Telugu. Command R (35B) performs poorly in low-resource languages such as Bengali (rank 13) or Swahili (rank 14). The complete bradley-terry model coefficient logits and 95% confidence intervals across all 18 languages in \mirage are provided in Table 7 and Table 8 in the Appendix.

Synthetic rankings using the learning to rank model. Figure 4 (right) plots the overall synthetic average rankings on all queries using heuristic-based features trained with a random forest learning to rank model in \mirage . The learned-ranking leaderboard highly correlates to the GPT-4o as a judge leaderboard, achieving an average Kendall-Tau ( \tau ) rank correlation = 0.909 , by training on 17 models during training and 2 models as holdout for every language.
Individual language-specific Kendall-Tau rank correlation scores are listed in Table 2 .
This provides evidence of the efficacy in training a random forest model as a surrogate judge on heuristic features with bootstrapping to approximately learn the Bradley-Terry logits.

Heuristic feature importance. In Figure 5 , we plot the average feature importance achieved by our Random Forest regression model across all 18 languages. We observe using Llama-3 (8B) as a judge, for fluency and answer overlap on average are most important heuristic features. Similarly, deterministic answer-overlap and reranking-based metrics are equally important. Some heuristic features such as language detection, and neutral score in support evaluation obtain the least importance. We potentially observe all answer overlap heuristic features achieve a high score indicating that the “answer” portion in the system’s RAG response is crucial for evaluation and is highly correlated with GPT-4o as a judge for pairwise evaluations.
Conclusion
We present \mirage , a multilingual RAG benchmark for 18 languages aimed towards accelerating research on multilingual RAG. In \mirage , we evaluate the multilingual generation part within RAG and aggregate traditional heuristic-based features to train a lightweight learning to rank model as a surrogate judge to learn a Bradley Terry model with GPT-4o pairwise judgments. Our results indicate a strong correlation between our surrogate judge and original LLM as a judge arena-based leaderboard demonstrating the effectiveness of our random forest model trained using RAG-based heuristic features to emulate a much expensive LLM as a judge pariwse rankings.

Using our arena-based leaderboard, we observed a majority of proprietary and open-sourced larger models currently dominate on the \mirage benchmark. Instruction tuning on \mirage training data helps improve the performance of open-sourced and smaller models on \mirage . Instruction-tuned Mistral-v0.2 (7B) is able to outperform the Llama 3 (70B) baseline on \mirage .
Limitations
is one of the first holistic multilingual RAG benchmarks. Although not perfect, we below discuss a set of limitations in our work:

In \mirage , we focused on benchmarking the generation task in RAG, we did not evaluate the multilingual retrieval task and its error propogating on the multilingual generation task.

In the future, we wish to evaluate diverse LLMs as teachers such as Claude-3.5 (sonnet) Anthropic ( 2024 ) or Gemini Pro Anil et al. ( 2023 ) , currently we only included a single teacher, GPT-4o, as a judge for evaluation. This can cause self-enhancement bias in our experiments towards GPT-4o and similar models in the family (GPT-4 and GPT-3.5).

In our heuristic evaluation, we only considered a small subset of features. We did not explore more recent RAG evaluation techniques such as nugget-based evaluation Lin and Demner-Fushman ( 2005 ) .

Lastly, \mirage dataset does not provide human-labeled answers for queries across all languages and is limited to Wikipedia, unlike recent RAG benchmarks which use the human answer for RAG-based evaluation or cover a wider variety of domains in English.
Acknowledgements
This research is supported by the Canada First Research Excellence Fund and the Natural Sciences and Engineering Research Council (NSERC) of Canada. We thank Ka Wong for helping with bootstrapping during the regression model’s training.
Figure 1: Multilingual naive RAG pipeline in Hindi (hn). In \mirage , we reuse the oracle retrieval set (query, judged relevant and non-relevant passages) in miracl Zhang et al. ( 2023 ) , and extend evaluation for the multilingual answer generation stage with LLMs.; Multilingual naive RAG pipeline in Hindi (hn). In \mirage , we reuse the oracle retrieval set (query, judged relevant and non-relevant passages) in miracl Zhang et al. ( 2023 ) , and extend evaluation for the multilingual answer generation stage with LLMs.
Figure 2: The \mirage evaluation flowchart consisting of three steps: (i) Heuristic-based features on different dimensions for the baseline model response. (ii) Pairwise comparisons involving GPT-4o as a judge on a few queries to train a learning to rank model as a surrogate judge. (iii) After training, we utilize the learning to rank model to output the baseline model rankings on the complete set of queries in a synthetic arena-based leaderboard.; The \mirage evaluation flowchart consisting of three steps: (i) Heuristic-based features on different dimensions for the baseline model response. (ii) Pairwise comparisons involving GPT-4o as a judge on a few queries to train a learning to rank model as a surrogate judge. (iii) After training, we utilize the learning to rank model to output the baseline model rankings on the complete set of queries in a synthetic arena-based leaderboard.
Figure 3: Lollipop plots denoting the average heuristic-based feature scores achieved by baselines in \mirage . x -axis denotes the languages in \mirage ; whereas y -axis plots every heuristic feature value. models in the same family are represented as the same color in a lollipop (as multiple circles). Figure 9 provides lollipop plots for all eleven heuristic-based features.; Lollipop plots denoting the average heuristic-based feature scores achieved by baselines in \mirage . x -axis denotes the languages in \mirage ; whereas y -axis plots every heuristic feature value. models in the same family are represented as the same color in a lollipop (as multiple circles). Figure 9 provides lollipop plots for all eleven heuristic-based features.
Figure 4: \mirage arena-based leaderboards: (left) Bradley-Terry model coefficients with rankings with GPT-4o as a judge pairwise judgments on a subset of 100 sampled queries; (right) Synthetic rankings using heuristic-based features and a learning to rank model. Each highlighted value is the rank of the model on the language (the lower the better). Models have been sorted by lowest to highest average rank across all 18 languages.; \mirage arena-based leaderboards: (left) Bradley-Terry model coefficients with rankings with GPT-4o as a judge pairwise judgments on a subset of 100 sampled queries; (right) Synthetic rankings using heuristic-based features and a learning to rank model. Each highlighted value is the rank of the model on the language (the lower the better). Models have been sorted by lowest to highest average rank across all 18 languages.