MEMERAG: A Multilingual End-to-End Meta-Evaluation Benchmark for Retrieval Augmented Generation
Authors: María Andrea Cruz Blandón, Jayasimha Talur, Bruno Charron, Dong Liu, Saab Mansour, Marcello Federico
Accepted at: acl25

MEMERAG: A Multilingual End-to-End Meta-Evaluation Benchmark for Retrieval Augmented Generation

María Andrea Cruz Blandón1*, Jayasimha Talur* Bruno Charron,
Dong Liu, Saab Mansour, Marcello Federico
1Tampere University, Amazon

Abstract
Automatic evaluation of retrieval augmented generation (RAG) systems relies on fine-grained dimensions like faithfulness and relevance, as judged by expert human annotators. Meta-evaluation benchmarks support the development of automatic evaluators that correlate well with human judgement. However, existing benchmarks predominantly focus on English or use translated data, which fails to capture cultural nuances. A native approach provides a better representation of the end user experience.

In this work, we develop a Multilingual End-to-end Meta-Evaluation RAG benchmark (MEMERAG). Our benchmark builds on the popular MIRACL dataset, using native-language questions and generating responses with diverse large language models (LLMs), which are then assessed by expert annotators for faithfulness and relevance. We describe our annotation process and show that it achieves high inter-annotator agreement. We then analyse the performance of the answer-generating LLMs across languages as per the human evaluators. Finally we apply the dataset to our main use-case which is to benchmark multilingual automatic evaluators (LLM-as-a-judge). We show that our benchmark can reliably identify improvements offered by advanced prompting techniques and LLMs 1.

1 Introduction
Retrieval augmented generation (RAG) is emerging as a popular application of large language models (LLMs) and a powerful paradigm to improve LLMs factuality (Gao et al., 2024). A RAG pipeline first retrieves relevant documents from an index based on a query and then composes a response using an LLM. Grounding the LLM response on retrieved knowledge helps mitigate outdated knowledge, lack of domain expertise and reduce hallucinations (Lewis et al., 2020; Gao et al., 2024). Collecting benchmarking data for RAG is challenging due to the complexity of the pipeline that includes information retrieval and text generation. Text generation, our focus in this paper, has, in general, two modes of automatic evaluation: reference-based and reference-free, which differ in the availability of human-generated gold references for each model input. Both modes can either leverage single (e.g. BERTScore (Zhang et al., 2020)) or multidimensional (e.g. autoMQM (Fernandes et al., 2023)) scores. Multidimensional evaluation (Burchardt, 2013) provides more comprehensive understanding of text generation systems and is the de facto standard in the machine translation (MT) community2.

In a reference-free evaluation setup, gold multidimensional judgements (factuality, relevance, etc) of model generations can be leveraged in a meta-evaluation framework. In this framework, automated evaluators are evaluated against human judgement to measure correlation. The automated evaluators can then be applied to measure the performance of new models’ output.

Previous work for RAG meta-evaluation mainly focused on English (Fan et al., 2024) or leveraged human or machine translation of English datasets (Sharma et al., 2024). Multilingual meta-evaluation is important to reliably measure performance across languages which can vary depending on language characteristics (low vs. high resource, complex morphology, etc.) and scripts (Latin vs. non-Latin). Translation-based benchmarks, while permitting cross-language comparisons, suffer from translationese phenomena such as introducing simpler syntax and lexical choices

*Work done by first author during an internship at Amazon. First two authors contributed equally to this work. Corresponding author: Jayasimha Talur email talurj@amazon.com.
1We release https://github.com/amazon-science/MEMERAG our benchmark to support the community developing accurate evaluation methods for multilingual RAG systems.
2Since 2021 in the WMT metrics shared task https://www2.statmt.org/wmt24/metrics-task.html
(Baker et al., 1993; Graham et al., 2020), thus leading to data distributionally different from native data and not necessarily reflecting native users preferences (Chen et al., 2024). Our position is that translation-based (parallel) benchmarks should be complemented by native multilingual benchmarks.

To bridge those gaps, we propose a native meta-evaluation multilingual benchmark for RAG systems. Our benchmark is built on top of the popular MIRACL (Zhang et al., 2023) dataset3 that includes native questions across 18 languages and relevance judgements of retrieved passages for multilingual retrieval evaluation. We extend MIRACL by generating answers in five languages with a diverse set of LLMs, and collecting judgements on the faithfulness and relevance of the answers using native expert human annotators. For the latter, we devised a structured annotation process that achieved a high rate of inter-annotator agreement. To evaluate the benchmark and set reference baseline results for others to compare against, we run LLM-as-a-judge experiments with various prompting techniques and state-of-the-art LLMs.

To summarize, our main contributions are:
• We built and publicly release the first (to the best of our knowledge) native multilingual meta-evaluation RAG benchmark.
• We developed a rigorous flow chart-based annotation process to achieve high inter-annotator agreement rate for both faithfulness and relevance judgements.
• We evaluated the quality of the benchmark on three multi-lingual meta-evaluation aspects: prompt selection, model selection, and fine-grained analysis.
• We establish reference baselines of multilingual automatic evaluators on our benchmark, showcasing performance improvements when using advanced prompting and LLMs.

2 Related Work

Due to the pipeline approach of RAG systems, the evaluation can be split into 3 main components: 1) retriever metrics to identify relevant chunks of information to the input typically measured with recall/precision@K (Manning et al., 2008); 2) generator metrics to identify the “usefulness” of the generated answers in relation to the input, this is done across fine grained dimensions such as faithfulness and relevance, either leveraging references answers (Es et al., 2024); 3) end-to-end (overall) metrics that take into account additional components such as preprocessing, chunking, query reformulation and cascading errors. Retrieval metrics have been extensively studied in the information retrieval community, hence recent work focused on the text generation performance of RAG systems. This is also the focus of our work. One important difference between generation with and without retrieved documents is the conflict between the parametric “world knowledge” and the non-parametric retrieved documents knowledge, hence the distinction between faithfulness against the retrieved documents (RAG specific) and factuality according to general knowledge (Maynez et al., 2020; Wu et al., 2024).

Recent studies have investigated the importance of various components within multilingual RAG systems. (Chirkova et al., 2024) utilized existing multilingual QA datasets to evaluate different combinations of retrievers and generator models, finding that task-specific prompt engineering is crucial for high-quality multilingual generation. In another study, (Thakur et al., 2024) extend MIRACL dataset to develop “MIRAGE-BENCH” a synthetic arena-based benchmark for ranking multilingual LLM generators. They employed preference judgments from a GPT-4o “judge” to train a ranking model. However, an important limitation of such synthetic benchmarks is the potential for self-preference bias (Panickssery et al., 2024), where the LLM judge may favor its own generations.

In this work, we focus on the faithfulness and relevance aspects to ensure meaningful results. These dimensions are typically assessed by human evaluators based on model-generated outputs, highlighting the need for developing automatic evaluation metrics that correlate well with human judgment—a process known as meta-evaluation. This need has led to a recent trend in the English-language research community of publishing meta-evaluation datasets and developing automated evaluators (Es et al., 2024; Saad-Falcon et al., 2024).

To the best of our knowledge, no multilingual meta-evaluation benchmark for RAG systems currently exists. In this work, we address this gap by developing such a benchmark. Our dataset facilitates the creation of multilingual automatic evaluators that correlate well with human judgments. This, in turn, enables comprehensive end-to-end benchmarking of RAG systems. We believe our dataset is the first to offer this capability in a multi-

3https://huggingface.co/datasets/miracl/miracl
<table>
  <tr>
    <th>Lang.</th>
    <th>Question</th>
    <th>LLM response</th>
    <th colspan="2">Human judgement</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th>Faithful</th>
    <th>Relevant</th>
  </tr>
  <tr>
    <td>EN</td>
    <td>When did Julia Child write her book Mastering the Art of French Cooking?</td>
    <td>She co-authored it with Simone Beck and Louisette Bertholle.</td>
    <td>✓ Logical conclusion</td>
    <td>✓ Adds context to the answer</td>
  </tr>
  <tr>
    <td>DE</td>
    <td>Welche bekannte Universität gibt es in Pennsylvania?</td>
    <td>Hier sind einige Beispiele:</td>
    <td>X Contradiction</td>
    <td>X Unrelated to the question</td>
  </tr>
  <tr>
    <td>ES</td>
    <td>¿Cómo la ciudad de Chicago tiñe de verde el río para celebrar San Patrick?</td>
    <td>Ahora se utiliza un tinte vegetal seguro y no perjudicial para los organismos vivos en el río.</td>
    <td>✓ Direct paraphrase</td>
    <td>✓ Adds context to the answer</td>
  </tr>
  <tr>
    <td>FR</td>
    <td>Quels sont les trains français avec la vitesse la plus haute?</td>
    <td>Les trains S-103 et CRH 3, tous deux de type Velaro, sont les trains avec la vitesse d’exploitation la plus élevée.</td>
    <td>X Adds new information</td>
    <td>✓ Directly answers the question</td>
  </tr>
  <tr>
    <td>HI</td>
    <td>कैंसर के उपचार में प्रयुक्त उत्पाद गैस कोणसी हैं?</td>
    <td>दिए गए पाठशाला में कैंसर के उपचार के लिए किसी गैस का उल्लेख नहीं है।</td>
    <td>✓ Logical conclusion</td>
    <td>✓ Directly answers the question</td>
  </tr>
</table>

MEMERAG (ours) Meta-evaluation benchmark

Figure 1: Examples from our Multilingual End-to-end Meta-Evaluation for RAG (MEMERAG) dataset. We select MIRACL native multilingual questions (for a subset of 5 languages), generate responses using diverse LLMs, and annotate each generated sentence with human experts for faithfulness and relevance. The annotation includes both coarse-grained (\( \checkmark, X \)) and fine-grained labels. Our dataset forms a meta-evaluation benchmark where automated evaluators can be developed and assessed on their correlation to human judgement. Due to space constraints we omit the retrieved documents (context) for the question and show only one sentence per LLM response.

lingual context.

3 Dataset Construction

Meta-evaluation datasets enable the development of reliable automatic evaluators. In a RAG setup, the input to the evaluator is composed of a question \( q \) (user input), a context \( c \) (set of passages automatically retrieved to the question) and an answer \( a \) generated by a language model to answer the question based on the context. The end-to-end evaluator then needs to judge the quality of the answer \( a \) given the context and the question (\( c, q \)). Following previous work (Saad-Falcon et al., 2024; Es et al., 2024) we focus on two quality dimensions:

<table>
  <tr>
    <th>Faithfulness</th>
    <td>Is the answer grounded on the context, regardless of your world knowledge?</td>
  </tr>
  <tr>
    <th>Relevance</th>
    <td>Is the answer relevant to the question, regardless of the context?</td>
  </tr>
</table>

We build a multilingual end-to-end meta-evaluation RAG (MEMERAG) dataset by extending the MIRACL dataset (Zhang et al., 2023) to include <i>model-generated answers and human-based quality judgements</i>. More precisely, we select relevant question-context pairs, generate answers using various language models and gather expert human annotations on the quality of those answers. Our dataset encompasses 5 languages: English (EN), German (DE), Spanish (ES), French (FR), and Hindi (HI), which represent multiple language families and both high- and low-resource languages. Figure 1 shows examples from the dataset, with LLM-generated answers and coarse- to fine-grained human-assigned labels for the faithfulness and relevance dimensions.

The MIRACL dataset is composed of questions written by humans in their <i>native</i> languages, one or more passages automatically retrieved from the Wikipedia, and human annotations about the relevance of each passage. Building a dataset starting from native questions in each language allows to evaluate RAG pipelines without resorting to (machine) translations, thus avoiding limitations and biases associated with translation. Note, however, that as questions were elicited from native speakers independently across different languages, the resulting data set is not parallel.

3.1 Question Selection

The questions in the MIRACL dataset were generated by humans based on prompts. This leads to questions that may be answerable by the prompt but may have ambiguity outside of that context. In particular, we identified as problematic the questions
for which the right answer can change over time. For example “Who is the president of Spain?” or “How old is Drake Hogestyn?”. In a RAG setting, different passages may have been written at different times and provide conflicting context. Additionally, the time of reference is usually not explicit in the question. To remove those complications we automatically filtered out time-dependent questions across all languages4. We combined the train and dev splits of the MIRACL dataset, which corresponds to a total of 3,662, 305, 2,810, 1,486 and 1,519 questions respectively for EN, DE, ES, FR and HI. Of those, 244, 13, 120, 64 and 86 were identified as time-dependent and filtered out.

3.2 Context Selection
The MIRACL dataset has an average of 10.3 passages per question, which corresponds to an average of 1,218 words of context in the English train and dev splits, with similar numbers in other languages. To reduce the cognitive load on human annotators, we limit the context per query to 5 passages.

The source dataset provides human-annotated binary relevance labels for passages. To more accurately simulate an automated retrieval process, we rank the passages for each question using BM25 (Schütze et al., 2008), as implemented in (Lù, 2024). We then select the top-5 ranked passages for each question. If these top-5 passages do not contain any human-annotated relevant passages, we replace the lowest-ranked passage with the highest-ranked relevant passage from the full set. This approach ensures that each question has at least one relevant passage in its context, avoiding scenarios where annotators would evaluate responses without any relevant information.

It is worth noting that simulating scenarios where no relevant passages exist is straightforward (e.g., by including only irrelevant passages). In such cases, for faithfulness evaluation, we would expect responses like "The provided documents do not contain a relevant answer." Our method focuses on faithfulness while efficiently utilizing human annotation efforts by ensuring that each evaluated case has at least some relevant context.

3.3 Answer Generation
After question and passage selection, we generate an answer for each question-context pair and each of five state-of-the-art LLMs. We generated answers with Claude 3 Sonnet, Llama3 70B, Llama3 8B, Mistral 7B, and GPT-4o mini. Those LLMs were selected to cover a range of model sizes, open weight and proprietary models. We prompted all the models in English5, asking to answer the question based only on the given context, and requesting the answer to be provided in the same language as the context and question. For all models, we set the temperature to 0.1, and maximum number of output tokens to 1000.

We thus produced answers for more than 1000 questions per language, except for German for which MIRACL only contains 305 questions. As our focus is on long-form answers, we further filtered out questions for which any of the 5 models generated an answer shorter than 10 words.

3.4 Annotation Guidelines
The task of annotating answers with faithfulness is challenging due to several factors. First, it involves some subjectivity which might impact Inter-Annotator Agreement (IAA) (Kryscinski et al., 2020; Tang et al., 2024b). Then, its label space is not precisely defined in the literature (Tang et al., 2024b; Laban et al., 2023; Malaviya et al., 2024). Finally, although faithfulness should be ideally evaluated for atomic facts, it is generally evaluated at the sentence or even document level, due to annotation costs.

Starting with the factuality error taxonomy introduced in Tang et al. (2024b), we ran a number of annotation pilots to refine the label space and guidelines.

Finally, we converged to three coarse-grained labels (Supported, Not supported, Challenging to determine), explained through 10 fine-grained labels. To increase the consistency of the annotation (IAA), we guide the annotation process through a flow chart (documented in Figure 3 in Appendix A). For relevance, which is significantly less ambiguous to evaluate, we device a simple annotation process with three labels: Directly answers the question, Adds context to the answer, and Unrelated to the question. Note that the first two labels can be used to describe "relevant" sentences, while the last label identifies "irrelevant" sentences. (See Appendix A

4See Appendix F for all prompts used during dataset construction.

5There is evidence in the literature for better model accuracy when the models are prompted to process in English (Lai et al., 2023; Liu et al., 2024), though under particular scenarios other languages could perform better (Behzad et al., 2024). We leave additional prompting experiments for future work.
<table>
  <tr>
    <th>Lang</th>
    <th>#Q</th>
    <th>#S</th>
    <th>Avg. #W</th>
    <th>Context Avg. #W</th>
  </tr>
  <tr>
    <td>EN</td>
    <td>250</td>
    <td>400</td>
    <td>30.3</td>
    <td>613.5</td>
  </tr>
  <tr>
    <td>DE</td>
    <td>250</td>
    <td>468</td>
    <td>27.3</td>
    <td>455.0</td>
  </tr>
  <tr>
    <td>ES</td>
    <td>250</td>
    <td>563</td>
    <td>52.1</td>
    <td>522.3</td>
  </tr>
  <tr>
    <td>FR</td>
    <td>250</td>
    <td>540</td>
    <td>48.7</td>
    <td>478.3</td>
  </tr>
  <tr>
    <td>HI</td>
    <td>250</td>
    <td>351</td>
    <td>23.8</td>
    <td>571.5</td>
  </tr>
  <tr>
    <td>Total</td>
    <td>1,250</td>
    <td>2,322</td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 1: General statistics of the MEMERAG dataset. #Q: number of questions, #S: number of sentences, #W: number of words. Each answer is annotated at the sentence level, leading to 2,322 total sentences annotated by experts for faithfulness and relevance.

for more details.)

3.5 Annotation Process

From the question-context-answer triplets obtained in Section 3.3, we randomly sampled 250 questions per language (50 per answer-generating model, without overlapping questions for diversity). We employed a professional vendor with native annotators6 to gather annotations for each sentence 7 of the 250 answers per language. The statistics of the annotated dataset are presented in Table 1. Among the 250 answers per language, a random subset of 10 were assigned to 3 annotators for computing the IAA and the rest to a single annotator (see Table 2).

The annotations were gathered via a web-based tool that implemented the flow chart of the annotation guidelines (see Appendix A for details). To further enhance IAA, we drew upon the findings of Krishna et al. (2023), which demonstrated that highlighting relevant information aids annotators in performing tasks and reaching consensus. Thus, we utilized the Llama 3 70B LLM to identify sentences within the retrieved passages that could potentially serve as supporting information to the answer sentences (see Appendix F).

The English annotations required approximately 25 hours of total annotation time, averaging 5.5 minutes per question. This covered 250 questions, including 10 that were annotated by three different annotators for quality control. Similar time investments were observed for the other four languages.

Table 2 summarizes the IAA per language for faithfulness and relevance labels assigned by 3 annotators. We report IAA using Gwet’s AC1 (Gwet, 2008) and Fleiss Kappa (Fleiss, 1971). We observe high agreement for faithfulness (0.84-0.93 Gwet’s AC1 and 0.70-0.88 Fleiss Kappa) and even higher agreement for relevance (0.95-1.0 Gwet’s AC1 and 0.63-1.0 Fleiss Kappa). This shows that the annotators are aligned and indicates a high quality of the annotations. Comparing to previous work, Tang et al. (2024b) report a Fleiss Kappa of 0.34-0.42 on faithfulness labels which they deem fair to moderate agreement. Note that a direct comparison with this work is not possible as they deal with different tasks, nevertheless the high IAA we are reporting is a testament of the effectiveness of our flow chart-based annotation design. Further details on IAA with fine-grained explanatory labels, and an extended version of the dataset (MEMERAG-Ext) with five annotators for 150 answers per language can be found in Appendix B.

<table>
  <tr>
    <th rowspan="2">Lang</th>
    <th colspan="2">Faithfulness</th>
    <th colspan="2">Relevance</th>
  </tr>
  <tr>
    <th>Gwet’s AC1</th>
    <th>Fleiss Kappa</th>
    <th>Gwet’s AC1</th>
    <th>Fleiss Kappa</th>
  </tr>
  <tr>
    <td>EN</td>
    <td>0.93</td>
    <td>0.77</td>
    <td>1.00</td>
    <td>1.00</td>
  </tr>
  <tr>
    <td>DE</td>
    <td>0.84</td>
    <td>0.81</td>
    <td>0.95</td>
    <td>0.73</td>
  </tr>
  <tr>
    <td>ES</td>
    <td>0.91</td>
    <td>0.76</td>
    <td>1.00</td>
    <td>1.00</td>
  </tr>
  <tr>
    <td>FR</td>
    <td>0.89</td>
    <td>0.88</td>
    <td>0.93</td>
    <td>0.63</td>
  </tr>
  <tr>
    <td>HI</td>
    <td>0.89</td>
    <td>0.70</td>
    <td>1.00</td>
    <td>1.00</td>
  </tr>
</table>

Table 2: Inter-annotator agreement (IAA) on the faithfulness and relevance dimensions with 3 annotators. Annotations are at the answer sentence level.

4 Annotation Results

We present in this section the results of the human annotations for the 2,322 sentences of the MEMERAG dataset.

Table 3 shows the distribution of faithfulness and relevance labels across the five languages. The distribution of labels is consistent across all languages, with a few exceptions. On faithfulness, German and Hindi show higher percentages of Supported answers. On relevance, Spanish presents a significantly higher percentage of labels Adds context to the answer compared to other languages while English had a very low share of labels Unrelated to the question. As a partial explanation for this, we note that Spanish questions generated the largest numbers of output sentences with 52.1 words vs., for example, 30.3 words for English (see Table 1).

6Annotators were compensated with a competitive hourly rate that is benchmarked against similar roles in their country of residence.
7Sentences were segmented using the pySBD(Sadvilkar and Neumann, 2020) package.
<table>
  <tr>
    <th rowspan="2">Lang</th>
    <th colspan="3">Faithfulness</th>
    <th colspan="3">Relevance</th>
  </tr>
  <tr>
    <th>✓</th>
    <th>×</th>
    <th>?</th>
    <th>✓</th>
    <th>×</th>
    <th>×</th>
  </tr>
  <tr>
    <td>EN</td>
    <td>65.2</td>
    <td>31.5</td>
    <td>3.2</td>
    <td>65.2</td>
    <td>32.5</td>
    <td>2.2</td>
  </tr>
  <tr>
    <td>DE</td>
    <td>71.2</td>
    <td>26.7</td>
    <td>2.1</td>
    <td>61.3</td>
    <td>26.5</td>
    <td>12.2</td>
  </tr>
  <tr>
    <td>ES</td>
    <td>65.7</td>
    <td>32.9</td>
    <td>1.4</td>
    <td>48.8</td>
    <td>43.9</td>
    <td>7.3</td>
  </tr>
  <tr>
    <td>FR</td>
    <td>62.0</td>
    <td>37.8</td>
    <td>0.2</td>
    <td>63.3</td>
    <td>29.3</td>
    <td>7.4</td>
  </tr>
  <tr>
    <td>HI</td>
    <td>73.8</td>
    <td>25.6</td>
    <td>0.6</td>
    <td>68.9</td>
    <td>21.4</td>
    <td>9.7</td>
  </tr>
</table>

Table 3: Label distribution in the benchmark. Percentage of sentences labelled as supported (✓), not supported (×), challenging to determine (?) for faithfulness, and as directly answers the question (✓), adds context to the answer (✓), unrelated to the question (×) for relevance, per language.

<table>
  <tr>
    <th>Label</th>
    <th>EN</th>
    <th>DE</th>
    <th>ES</th>
    <th>FR</th>
    <th>HI</th>
  </tr>
  <tr>
    <td>Direct paraphrase</td>
    <td>8.5</td>
    <td>41.7</td>
    <td>25.0</td>
    <td>33.7</td>
    <td>40.7</td>
  </tr>
  <tr>
    <td>Logical conclusion</td>
    <td>41.2</td>
    <td>28.2</td>
    <td>30.4</td>
    <td>28.0</td>
    <td>5.7</td>
  </tr>
  <tr>
    <td>Other</td>
    <td>15.5</td>
    <td>1.3</td>
    <td>10.3</td>
    <td>0.4</td>
    <td>27.4</td>
  </tr>
  <tr>
    <td>Adds new info</td>
    <td>7.0</td>
    <td>9.6</td>
    <td>16.0</td>
    <td>15.0</td>
    <td>14.8</td>
  </tr>
  <tr>
    <td>Contradiction</td>
    <td>4.5</td>
    <td>11.3</td>
    <td>8.3</td>
    <td>5.9</td>
    <td>7.1</td>
  </tr>
  <tr>
    <td>Mis-referencing</td>
    <td>1.5</td>
    <td>3.0</td>
    <td>2.3</td>
    <td>3.7</td>
    <td>0.3</td>
  </tr>
  <tr>
    <td>Nuance shift</td>
    <td>6.8</td>
    <td>0.6</td>
    <td>4.3</td>
    <td>5.6</td>
    <td>1.7</td>
  </tr>
  <tr>
    <td>Opinion as fact</td>
    <td>0.5</td>
    <td>0.6</td>
    <td>0.2</td>
    <td>2.2</td>
    <td>0.3</td>
  </tr>
  <tr>
    <td>Wrong reasoning</td>
    <td>10.0</td>
    <td>0.6</td>
    <td>1.4</td>
    <td>1.9</td>
    <td>0.3</td>
  </tr>
  <tr>
    <td>Other</td>
    <td>1.2</td>
    <td>0.9</td>
    <td>0.4</td>
    <td>3.5</td>
    <td>1.1</td>
  </tr>
  <tr>
    <td>Challeng. to determ.</td>
    <td>3.2</td>
    <td>2.1</td>
    <td>1.4</td>
    <td>0.2</td>
    <td>0.6</td>
  </tr>
</table>

Table 4: Fine-grained faithfulness label distribution in the benchmark. Percentage of sentences with each label per language. The three sections correspond to the coarse-grained labels Supported/Not supported/Challenging to determine.

Hence, we expect that the relevance statistics reflect the tendency of Spanish answers to be more verbose.

For a more granular insights into the annotations, we show in Table 4 the distribution of the explanatory fine-grained faithfulness labels for each language. As label names are quite self-explanatory, we refer for their precise meaning to Figure 3 in Appendix A. Table 4 is split into three blocks, respectively, addressing fine-grained labels for Supported (top), Not supported (middle), and Challenging to determine (bottom) answers. We observe remarkable differences across languages. For example, supported answers for English are prominently under form of a logical conclusion from the context (41.2%), for German and Hindi from direct paraphrasing of information in the context (41.7% and 40.7%), while for French and Spanish from a more balanced combination of the two reasons. On the side of unsupported answers, the main mistake type in English is Wrong reasoning (10%), i.e. answers are non logical conclusions from the context, while this type or error is significantly rarer (under 2%) for all other languages. The rate of Adds new information errors, a.k.a. hallucinations, ranges from 7% for English to 16% for Spanish. The observed cross-linguistic variations in the distribution of labels can be attributed to the different nature of the questions and accuracy of the models across the 5 languages.

5 Dataset Applications

The MEMERAG dataset is designed to support the development of reliable automatic evaluation methods. For that purpose, we describe in this section how our dataset can be used as a benchmark to enable various meta-evaluation use cases. We focus on two applications: 1) Prompt selection: The ability of our benchmark to effectively select prompts for automatic evaluation, 2) Model selection: The effectiveness of our benchmark to compare and select models. By concentrating on these aspects, we can evaluate the benchmark’s utility as a comprehensive tool for multilingual model assessment. While we provide reference baselines for each application, our focus is on showcasing the effectiveness of the benchmark rather than the underlying capabilities of the LLMs.

5.1 Experimental Setup

Benchmark Tasks Our benchmark is composed of multiple tasks defined by the annotation dimension and subset considered. On the annotation dimension, we focus our experiments on the coarse-grained faithfulness dimension. This dimension is more challenging than relevance as highlighted by the lower IAA, while retaining a high-enough IAA to make for a trustworthy benchmark. We invite benchmark users to also experiment on the other dimensions provided by the dataset depending on their use case. Note that for this experiment, we remove the sentences labelled as Challenging to determine by human annotators. On the annotation subset, we first distinguish the multilingual task which uses the full dataset and the monolingual task, which only considers a single language. Those tasks can then be further broken down at the fine-grained level by considering the subset of sentences with a certain fine-grained label. Performance is evaluated with Balanced Accuracy
(BAcc), with equal weights on each coarse-grained label and language. We conduct significance testing using permutation tests (Good, 2013), with further details in Appendix E.

Reference Prompts To demonstrate how the benchmark can be used to select the appropriate prompt, we experiment with multiple prompting strategies from simple to advanced techniques, starting with zero-shot prompting, where LLMs directly classify statements as Supported or Not supported. We then implement chain-of-thought (COT) prompting (Wei et al., 2022), which incorporates an intermediate reasoning step. While these basic prompting strategies provide a good starting point for evaluation, our initial experiments revealed limitations in their ability to capture the nuanced requirements of our specific task. Without explicit guidelines in the prompt, automatic evaluators rely on their “world knowledge”, which may not align with the specific requirements of the evaluation task. To overcome this, we add instructions from the annotation guidelines (AG) in the prompt. Adding annotation guidelines provides clear criteria for what constitutes Supported versus Not Supported sentences, reducing ambiguity in the evaluation process. The various prompts are presented in Appendix G.

Reference Models We experiment with four LLMs with varying model sizes and capabilities: GPT-4o mini, Qwen 2.5 32B, and two versions of Llama 3.2 (11B and 90B)8. In case an LLM does not produce one of the required labels, we repeatedly prompt the LLM up to five times with temperature and top_p equal to 0.1 to get a valid label. If the LLM fails to generate a label after five retries, we treat the datapoint as an error (wrong label).

<table>
  <tr>
    <th>Prompt</th>
    <th>GPT-4o mini</th>
    <th>Qwen 2.5 32B</th>
    <th>Llama 3.2 90B</th>
    <th>Llama 3.2 11B</th>
  </tr>
  <tr>
    <td>ZS</td>
    <td>59.7</td>
    <td>66.7</td>
    <td>58.0</td>
    <td>55.4</td>
  </tr>
  <tr>
    <td>COT</td>
    <td>61.4</td>
    <td>68.8</td>
    <td>59.9</td>
    <td><b>62.5</b></td>
  </tr>
  <tr>
    <td>AG</td>
    <td>71.6†</td>
    <td><b>72.6</b></td>
    <td>62.8†</td>
    <td>57.9</td>
  </tr>
  <tr>
    <td>AG+COT</td>
    <td><b>71.7</b></td>
    <td>71.8†</td>
    <td><b>64.4</b></td>
    <td>61.6†</td>
  </tr>
</table>

Table 5: Reference baselines for the multilingual task on coarse-grained faithfulness. Balanced accuracy (BAcc) of the automatic evaluators averaged across the 5 languages (EN, DE, ES, FR, HI) using zero-shot (ZS), chain-of-thought (COT), annotation guidelines (AG) and AG+COT prompting strategies. Bold indicates best performance for the column, † indicates results not statistically different from the best (p > 0.05). Additional results on monolingual tasks and standard errors can be found in Appendix E, Tables 10-12.

Table 5 demonstrates the benchmark’s ability to compare different prompting approaches across languages. The benchmark reveals consistent patterns, showing how different prompt designs impact evaluation quality. As expected, adding a reasoning step (COT) improves over zero-shot prompting. In addition, adding annotation guidelines (AG) helps align automated evaluators with human judgments across all languages. Comparing the two best models GPT-4o mini and Qwen 2.5 32B, Qwen 2.5 32B excels in the zero-shot and COT setups, showcasing higher “out-of-the-box” alignment with human judgements. GPT-4o mini achieves similar performance once the annotation guidelines are added to the prompt.

Figure 2, shows the performance per language of various automatic evaluators with a fixed prompt (AG + COT), which allows us to select the best model for each language. We observe that GPT-4o mini performs best in English. For the rest of the languages Qwen 2.5 32B performs the best however, the results are not statistically different from GPT-4o mini. Our benchmark also provides users with the capability to conduct detailed, fine-grained analyses of model performance across various dimensions of faithfulness. The breakdown of automatic evaluation performance by error type is shown in Appendix I.

5.2 Experimental Results

Our benchmark enables systematic evaluation of different approaches to automated faithfulness evaluation. To illustrate this, we examine how the benchmark can surface the effectiveness of various automatic evaluation models and prompting strategies.

8In comparison to the LLMs selected for answer generation, see Section 3.3, we upgraded Llama from 3 to 3.2 as the context length of 8K tokens was not sufficient for all prompts. We picked GPT-4o mini as representative of proprietary models. Additionally, Mistral was excluded as it performed poorly in initial experiments.

6 Conclusions

We introduced a high-quality and challenging multilingual end-to-end meta-evaluation benchmark for RAG (MEMERAG). Our carefully designed flow-chart-based annotation achieved a high inter-
Figure 2: Reference baselines for the monolingual task on coarse-grained faithfulness. Balanced Accuracy (BAcc) of the automatic evaluators using various LLMs across five languages: EN, DE, ES, FR and HI. Each plot compares the performance of four models: Llama 3.2 11B, Qwen 2.5 32B, Llama 3.2 90B, and GPT-4o mini using AG + COT prompt. The best-performing model for each language is highlighted with a darker blue bar. Bars with diagonal hatching indicate results not statistically different from the best (p > 0.05).

annotator agreement rate supporting the reliability of the benchmark. The introduced MEMERAG dataset opens the door for multiple application scenarios, including but not limited to the demonstrated cases, i.e. prompt selection and model selection.

For the meta-evaluation setup, we develop and compare various LLMs-as-a-judge and observe that automatic evaluators performance varies across the languages, influenced by language characteristics, native-question complexity and LLM generation nuances. These variations underscore the importance of our testbed, which demonstrated consistent results when comparing the prompting techniques (COT+guidelines > COT > zero-shot) and provides a foundation for developing better multilingual evaluators.

7 Limitations

Due to time and cost constraints, our annotations and experiments are limited in terms of prompting techniques, LLMs we experimented with and languages we annotated. Nevertheless, we diversified our LLMs across size and “openness” while the languages represent two families and low and high resource ones. In addition, there exists in the literature fine-tuned factuality evaluators for English, though we expect those to not work as well on non-English languages. Another method is to approximate factuality through entailment tasks (i.e. XNLI dataset) though such methods were shown (for English) to be inferior to multi-task training and distillation and data augmentation from LLMs (Tang et al., 2024a). Fine-tuning multilingual evaluators and examining transfer learning across languages is interesting but is left for future work that can leverage our dataset for this purpose.

As we advocate for a native testing approach, the questions across the languages are not parallel, which could introduce a dimension of different questions and LLM generations complexities across the different language test data. The data
we collected presents different challenges which are captured according to our fine grained error labels (Table 4). Future work could balance the challenges and complexities by collecting data for specific challenging phenomena. Note that this balancing is not straightforward, it can be done on the question side though this is insufficient as it does not control for the answer complexity. Controlling for the answer complexity is a challenging problem as the answer side is model generated (one method is to generate many answers and select for certain phenomena with human in the loop which is costly).

References

Mona Baker, Gill Francis, and Elena Tognini-Bonelli. 1993. Corpus linguistics and translation studies: Implications and applications. In In Text and Technology: In Honour of John Sin- clair, Netherlands. John Benjamins Publishing Company.

Shabnam Behzad, Amir Zeldes, and Nathan Schneider. 2024. To ask LLMs about English grammaticality, prompt them in a different language. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 15622–15634, Miami, Florida, USA. Association for Computational Linguistics.

Aljoscha Burchardt. 2013. Multidimensional quality metrics: a flexible system for assessing translation quality. In Proceedings of Translating and the Computer 35, London, UK. Aslib.

Pinzhen Chen, Simon Yu, Zhicheng Guo, and Barry Haddow. 2024. Is it good data for multilingual instruction tuning or just bad multilingual evaluation for large language models? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9706–9726, Miami, Florida, USA. Association for Computational Linguistics.

Nadezhda Chirkova, David Rau, Hervé Déjan, Thibault Formal, Stéphane Clinchant, and Vassilina Nikoulina. 2024. Retrieval-augmented generation in multilingual settings. Preprint, arXiv:2407.01463.

Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. RAGAs: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158, St. Julians, Malta. Association for Computational Linguistics.

Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’24, page 6491–6501, New York, NY, USA. Association for Computing Machinery.

Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André F. T. Martins, Graham Neubig, Ankush Garg, Jonathan H. Clark, Markus Freitag, and Orhan Firat. 2023. The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation. arXiv preprint. ArXiv:2308.07286 [cs].

Joseph Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76:378–.

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv preprint. ArXiv:2312.10997 [cs].

Phillip Good. 2013. Permutation tests: a practical guide to resampling methods for testing hypotheses. Springer Science & Business Media.

Yvette Graham, Barry Haddow, and Philipp Koehn. 2020. Statistical power and translationese in machine translation evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 72–81, Online. Association for Computational Linguistics.

Kilem Li Gwet. 2008. Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1):29–48.

Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. 2023. LongEval: Guidelines for human evaluation of faithfulness in long-form summarization. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1650–1669, Dubrovnik, Croatia. Association for Computational Linguistics.

Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332–9346, Online. Association for Computational Linguistics.

Philippe Laban, Wojciech Kryscinski, Divyansh Agarwal, Alexander Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. 2023. SummEdits: Measuring LLM Ability at Factual Reasoning Through The Lens of Summarization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9662–9676, Singapore. Association for Computational Linguistics.

Viet Dac Lai, Nghia Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and
Thien Huu Nguyen. 2023. ChatGPT beyond English: Towards a comprehensive evaluation of large language models in multilingual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13171–13189, Singapore. Association for Computational Linguistics.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.

Chaoqun Liu, Wenxuan Zhang, Yiran Zhao, Anh Tuan Luu, and Lidong Bing. 2024. Is translation all you need? a study on solving multilingual tasks with large language models. Preprint, arXiv:2403.10258.

Xing Han Lü. 2024. Bm25s: Orders of magnitude faster lexical search via eager sparse scoring. Preprint, arXiv:2407.03618.

Chaitanya Malaviya, Subin Lee, Sihaio Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. 2024. ExpertQA: Expert-Curated Questions and Attributed Answers. arXiv preprint. ArXiv:2309.07852 [cs].

Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press, Cambridge, UK.

Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics.

Arjun Panickksery, Samuel R. Bowman, and Shi Feng. 2024. Llm evaluators recognize and favor their own generations. Preprint, arXiv:2404.13076.

Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2024. ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems. arXiv preprint. ArXiv:2311.09476 [cs].

Nipun Sadvilkar and Mark Neumann. 2020. PySBD: Pragmatic sentence boundary disambiguation. In Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS), pages 110–114, Online. Association for Computational Linguistics.

Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan. 2008. Introduction to information retrieval, volume 39. Cambridge University Press Cambridge.

Nikhil Sharma, Kenton Murray, and Ziang Xiao. 2024. Faux polyglot: A study on information disparity in multilingual large language models. Preprint, arXiv:2407.05502.

Liyan Tang, Philippe Laban, and Greg Durrett. 2024a. MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents. arXiv preprint. ArXiv:2404.10774 [cs].

Liyan Tang, Igor Shalyminov, Amy Wing-mei Wong, Jon Burnsky, Jake W. Vincent, Yu’an Yang, Siffi Singh, Song Feng, Hwanjun Song, Hang Su, Lijia Sun, Yi Zhang, Saab Mansour, and Kathleen McKewown. 2024b. TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization. arXiv preprint. ArXiv:2402.13249 [cs].

Nandan Thakur, Suleman Kazi, Ge Luo, Jimmy Lin, and Amin Ahmad. 2024. Mirage-bench: Automatic multilingual benchmark arena for retrieval-augmented generation systems. Preprint, arXiv:2410.13716.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.11903.

Kevin Wu, Eric Wu, and James Zou. 2024. Clasheval: Quantifying the tug-of-war between an LLM’s internal prior and external evidence. In The Thirty-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.

Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. 2023. MIRACL : A Multilingual Retrieval Dataset Covering 18 Diverse Languages. Transactions of the Association for Computational Linguistics, 11:1114–1131.
A  Human Annotation Guidance

Before conducting large-scale annotations, we conducted a pilot with 10 English RAG outputs and 3 annotators. We asked annotators to evaluate faithfulness at the sentence level either as Supported or with a subset of the factuality mistakes typology in Tang et al. (2024b) (developed for summarization): Contradiction, Hallucination, Misreferencing, Nuance meaning shift, Opinion stated as fact, Wrong reasoning, to which was an Other mistake label was added to account for unforeseen mistakes in the RAG setting. As this led to very low IAA, we conducted another round reducing the non-supported labels to Stating opinion as fact, Drawing wrong conclusions, Other mistake but this still resulted in low IAA (Gwet’s AC1 0.45).

Upon careful analysis of the annotator disagreements in the pilot and further rounds of calibration, we developed the annotation workflow shown in Figure 3. The key improvements were: (i) add a Challenging to determine label, (ii) have two levels of labels, a coarse-grained level (Supported, Not supported, Challenging to determine) and a fine-grained level for precision on the mistakes, (iii) enforce annotators to follow a specific reasoning with a flow chart, (iv) use numbers for the fine-grained level rather than labels which could be misinterpreted. Those guidelines allowed to reach significantly higher IAA on faithfulness (Gwet’s AC1 0.81 coarse-grained). Likewise we iterated on the relevance labels, starting from Must have, Nice to have, or Irrelevant and converging to the more explicit Directly answers the question, Adds context to the answer, or Unrelated to the question. This also increased the IAA significantly. The screenshot of the user interface used by human annotators is shown in Figure 4.

<table>
  <tr>
    <th rowspan="2">Lang</th>
    <th rowspan="2">#S</th>
    <th colspan="2">IAA (Gwet’s AC1)</th>
    <th colspan="2">MEMERAG</th>
  </tr>
  <tr>
    <th>Faithfulness<br>Coarse</th>
    <th>Fine</th>
    <th>Relevance<br>Coarse</th>
    <th>Fine</th>
  </tr>
  <tr>
    <td>EN</td>
    <td>13</td>
    <td>0.93</td>
    <td>0.63</td>
    <td>1</td>
    <td>0.92</td>
  </tr>
  <tr>
    <td>DE</td>
    <td>31</td>
    <td>0.84</td>
    <td>0.47</td>
    <td>0.95</td>
    <td>0.91</td>
  </tr>
  <tr>
    <td>ES</td>
    <td>20</td>
    <td>0.91</td>
    <td>0.3</td>
    <td>1</td>
    <td>0.93</td>
  </tr>
  <tr>
    <td>FR</td>
    <td>23</td>
    <td>0.89</td>
    <td>0.58</td>
    <td>0.93</td>
    <td>0.92</td>
  </tr>
  <tr>
    <td>HI</td>
    <td>17</td>
    <td>0.89</td>
    <td>0.18</td>
    <td>1</td>
    <td>1</td>
  </tr>
</table>

Table 6: Inter-annotator agreement (IAA) on the faithfulness and relevance dimensions with 3 annotators (MEMERAG subset), for coarse-grained and fine-grained levels. Annotations are at the answer sentence level (#S number of answer sentences is provided).

<table>
  <tr>
    <th rowspan="2">Lang</th>
    <th rowspan="2">#S</th>
    <th colspan="2">IAA (Gwet’s AC1)</th>
    <th colspan="2">MEMERAG-Ext</th>
  </tr>
  <tr>
    <th>Faithfulness<br>Coarse</th>
    <th>Fine</th>
    <th>Relevance<br>Coarse</th>
    <th>Fine</th>
  </tr>
  <tr>
    <td>EN</td>
    <td>226</td>
    <td>0.83</td>
    <td>0.41</td>
    <td>1.00</td>
    <td>0.9</td>
  </tr>
  <tr>
    <td>DE</td>
    <td>272</td>
    <td>0.75</td>
    <td>0.47</td>
    <td>0.92</td>
    <td>0.79</td>
  </tr>
  <tr>
    <td>ES</td>
    <td>276</td>
    <td>0.91</td>
    <td>0.39</td>
    <td>1.00</td>
    <td>0.97</td>
  </tr>
  <tr>
    <td>FR</td>
    <td>370</td>
    <td>0.72</td>
    <td>0.53</td>
    <td>0.99</td>
    <td>0.8</td>
  </tr>
  <tr>
    <td>HI</td>
    <td>208</td>
    <td>0.91</td>
    <td>0.33</td>
    <td>0.98</td>
    <td>0.92</td>
  </tr>
</table>

Table 7: Inter-annotator agreement (IAA) on the faithfulness and relevance dimensions with 5 annotators (MEMERAG-Ext), for coarse-grained and fine-grained levels. Annotations are at the answer sentence level (#S number of answer sentences is provided).

due to ambiguity in the annotation guidelines. The fact that such ambiguity remained after significant work on refining the guidelines and guiding annotators with UI elements hints at the irreducible subjectivity of the task. Subjectivity manifests itself as a non-trivial probability distribution of the annotation label obtained by drawing a random human annotator for a given question. It would be of interest in the development of automated annotators to compare this distribution to that of the annotations of a stochastic LLM-based annotator.

To investigate those aspects, we created an extended dataset, called MEMERAG-Ext. The dataset is composed of 150 question-context pairs per language, extracted similarly to MEMERAG with the process described in Section 3. The questions are disjoint between the two datasets, except for German where there is some overlap due to the limited number of questions in MIRACL. The annotation process then differed in two ways: 1) annotations were obtained for all question from 5
Figure 3: Annotation guideline for faithfulness labelling by human

Figure 4: User interface used by human annotators for labelling faithfulness and relevance.

expert annotators, and 2) The Challenging to determine label was not an option for annotators to force meaningful labels even at the cost of disagreements. IAA metrics for the MEMERAG-Ext dataset can be found in Table 7. Those results are in line with the IAA observed in MEMERAG, which confirms its quality. Detailed investigation of the disagreements between human annotators, and comparison to automated annotators are left as future work.

We release this extended dataset along with the main dataset to promote related work by the research community.

C  Detailed numbers on the MEMERAG dataset

In this section, we look at the detailed statics of faithfulness and relevance in the MEMERAG
<table>
  <tr>
    <th>Lang</th>
    <th>#Q</th>
    <th>#S</th>
    <th>Answer Avg. #W</th>
    <th>Context Avg. #W</th>
  </tr>
  <tr>
    <td>EN</td>
    <td>150</td>
    <td>226</td>
    <td>27.5</td>
    <td>591.4</td>
  </tr>
  <tr>
    <td>DE</td>
    <td>150</td>
    <td>272</td>
    <td>30.0</td>
    <td>456.8</td>
  </tr>
  <tr>
    <td>ES</td>
    <td>150</td>
    <td>276</td>
    <td>39.1</td>
    <td>490.8</td>
  </tr>
  <tr>
    <td>FR</td>
    <td>150</td>
    <td>370</td>
    <td>48.1</td>
    <td>453.2</td>
  </tr>
  <tr>
    <td>HI</td>
    <td>150</td>
    <td>208</td>
    <td>23.0</td>
    <td>576.6</td>
  </tr>
  <tr>
    <th>Total</th>
    <th>750</th>
    <th>1352</th>
    <th></th>
    <th></th>
  </tr>
</table>

Table 8: General statistics of the MEMERAG-Ext dataset. #Q: number of questions, #S: number of sentences, #W: number of words. Each answer is annotated at the sentence level, leading to 1,352 total sentences annotated by experts for faithfulness and relevance.

dataset itself. Overall, the faithfulness and relevance in the dataset variate a lot from one language to another language and from one generator model to another. There is usually a range of 10-20 percentage points between the language with lowest percentage and the language with highest percentage of supported sentences. This supports our assumption of variable behaviour/performance across different languages.

Given the large variation, the percentage of supported answers is larger than that of unsuported answers. Similarly, the ratio of relevant answers is in general larger than that of other relevance buckets (except for ES with LLM-D; EN, ES and FR with LLM-A). Interestingly, when analysing the various results per model and language, we did not find general pattern. This is counter intuitive to our assumption of having more supported sentences for English language. Surprisingly, even though Hindi is a low-resource language the models are producing 70% supported sentences, except for LLM-D that produced only around 62% supported sentences.

D Fine grained automatic annotation analysis

Figure 5 shows a heatmap representing the failure modes of automatic evaluators. We observe that improving the prompting strategy from ZS to AG + COT, reduces automatic evaluation errors across all error categories, except for Llama 3.2 11B, where the model makes more errors in the “Logical conclusion category”. We also observe that “Wrong reasoning”, “Nuance shift” and “Logical conclusion” are the top error categories for all the models tested. Future work could explore prompting or fine-tuning techniques designed to handle specific error types.

E Statistical Significance Test Details

In Table 5 and Figure 2, we aim to determine whether the scores (in terms of BAcc) for the best prompt and best model was significantly different from the other scores in the table. To test statistical significance, we use permutation test (Good, 2013), a non-parametric method for comparing two related samples. The null hypothesis for this test suggests that there is no significant difference between the performance of the best-performing prompt/models and the other prompts/models, while the alternative hypothesis suggests a significant difference exists. We consider \( \alpha = 0.05 \) as significance level. Consequently, when p > 0.05, we are not able to reject the null hypothesis, indicating that prompts and models have similar performance.

F Prompts used for dataset construction

In section we describe the various prompts used in our pipeline. Out prompts are written as Jinja2\(^9\) templates.

F.1 Time-dependent answer filtering

During internal pilots, we identified answers that relied on current date / current affairs that could be challenging to determine their faithfulness. For example, to the question "How old is Yann LeCun?" is the answer correct if the LLM uses the time when it was trained? To avoid such cases we used Llama3 70B to filter out those cases using the prompt shown below:

Time-dependent Answer filtering

You are an NLP assistant that helps to identify if a question requires to know when is today (day, or month, or year), current affairs, or up-to-date information. Give the step by step of how to answer the question in between the labels <rationale></rationale>. Then verify if the steps included to know any information about the current time and give your answer in between the tags: <label></label>. Please only use 'yes' or 'no' in your final answer. You will be given the question in {{language}}. Provide your rationale and label in English.

\footnotetext{9https://jinja.palletsprojects.com/en/stable/}
<table>
  <tr>
    <th>Model</th>
    <th>EN</th>
    <th>DE</th>
    <th>ES</th>
    <th>FR</th>
    <th>HI</th>
  </tr>
  <tr>
    <td>LLM-A (615)</td>
    <td>52.7 / 43.8 / 3.5</td>
    <td>74.3 / 17.1 / 8.6</td>
    <td>58.6 / 41.4 / 0</td>
    <td>59.6 / 40.4 / 0</td>
    <td>73.2 / 26.8 / 0</td>
  </tr>
  <tr>
    <td><i>Faithfulness</i></td>
    <td>42.5 / 51.7 / 5.8</td>
    <td>60.9 / 37.4 / 1.7</td>
    <td>42.3 / 54.5 / 3.2</td>
    <td>48 / 50.3 / 1.7</td>
    <td>46.8 / 41.3 / 11.9</td>
  </tr>
  <tr>
    <td>LLM-B (315)</td>
    <td>76.1 / 17.9 / 6</td>
    <td>88.4 / 11.6 / 0</td>
    <td>88.2 / 11.8 / 0</td>
    <td>58.6 / 41.4 / 0</td>
    <td>84.9 / 15.1 / 0</td>
  </tr>
  <tr>
    <td><i>Relevance</i></td>
    <td>67.1 / 31.5 / 1.4</td>
    <td>72.7 / 22.1 / 5.2</td>
    <td>75 / 22.2 / 2.8</td>
    <td>87.1 / 12.9 / 0</td>
    <td>80.7 / 10.5 / 8.8</td>
  </tr>
  <tr>
    <td>LLM-C (315)</td>
    <td>61.4 / 35.1 / 3.5</td>
    <td>69.8 / 30.2 / 0</td>
    <td>68.1 / 31.9 / 0</td>
    <td>80.3 / 19.7 / 0</td>
    <td>80 / 20 / 0</td>
  </tr>
  <tr>
    <td><i>Faithfulness</i></td>
    <td>86.9 / 11.5 / 1.6</td>
    <td>78.3 / 20.3 / 1.4</td>
    <td>69.3 / 28 / 2.7</td>
    <td>87.1 / 10 / 2.9</td>
    <td>85.9 / 9.4 / 4.7</td>
  </tr>
  <tr>
    <td>LLM-D (743)</td>
    <td>67.3 / 30.8 / 1.9</td>
    <td>61.4 / 38 / 0.6</td>
    <td>61 / 35.7 / 3.3</td>
    <td>59.8 / 40.2 / 0</td>
    <td>63 / 34.6 / 2.5</td>
  </tr>
  <tr>
    <td><i>Relevance</i></td>
    <td>70.3 / 29.7 / 0</td>
    <td>41.5 / 33 / 25.5</td>
    <td>33 / 52.3 / 14.7</td>
    <td>54.8 / 26.1 / 19.1</td>
    <td>57.1 / 24.2 / 18.7</td>
  </tr>
  <tr>
    <td>LLM-E (334)</td>
    <td>77.2 / 21.1 / 1.7</td>
    <td>73.9 / 26.1 / 0</td>
    <td>69.7 / 29 / 1.3</td>
    <td>59.2 / 39.5 / 1.3</td>
    <td>73.3 / 26.7 / 0</td>
  </tr>
  <tr>
    <td><i>Faithfulness</i></td>
    <td>80.3 / 19.7 / 0</td>
    <td>75.4 / 17.4 / 7.3</td>
    <td>63.4 / 36.6 / 0</td>
    <td>73.8 / 25 / 1.2</td>
    <td>90.6 / 9.4 / 0</td>
  </tr>
  <tr>
    <td><i>Relevance</i></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 9: Percentage of sentences labelled as Supported/Not Supported/Challenging to determine for the faithfulness, and as Directly answers the question/Add context to the answer/Unrelated to the question for the relevance, per language and model. The total number of sentences generated by the model across all languages is given in parenthesis besides the model name.

![Heatmap showing disagreement between Ground Truth and Automatic Annotator for English fine-grained labels](page_1012_1012_1012_1012.png)

Figure 5: Fine-grained faithfulness errors when automatic evaluator disagrees with ground truth.
F.2 Highlighting relevant segments

We highlight relevant statements in the passage to help annotators focus on important parts of the context. We prompt Llama 3 70B, to predict all the statements that support the passage. We use temperature equal to 0.1 in this step.

F.3 Answer generation prompt

We generate the answers using the same prompt across all models and languages. All the instructions were given in English, while the context and question were given in the testing language. For all models we set the temperature to 0.1 and the maximum number of token to 1000.

Highlighting Relevant Segments

You are an agent that verifies if sentences are supported by a given context.
You will be given a context made of several passages referred as "Passage" split into sentences, each with a numeral identifier, and each referred as "Sentence". Your task is to determine if the sentence is supported by some of the passages (using label 1) or is not supported (using label 0). Please follow this process:

(1) Explain why the sentence is supported or not supported, please write your reasoning in between the tags <rationale></rationale>. If a sentence is supported, list the number of the sentences in the passage or several passages that supported it. If a sentence is not supported, explain why is not supported.

(2) Write the final label for the sentence in between the tags <label></label>.

(3) Write the id of the supporting sentences from passages in between the tags <references></references> separated by comma. Remember to use only 0 or 1 for label.

### Context
{% for passage in passages -%}
Passage {{loop.index}}:
{% set outer_loop = loop %}
{% for sent in passage.text %}
    {{outer_loop.index}}.{{loop.index}}: {{sent|safe}}
{% endfor %}
{% endfor %}

### Sentence:
{{sentence|safe}}

Answer Generation Prompt

You are an NLP assistant whose purpose is to answer a question based on given passages. The passages may or may not help answer the question. You will need to provide the answer based only on the passages. The answer must be in language without fail. Be concise and direct without referring to passages in the answer. Avoid expressions such as "According to the passages" or "Based on the passages".

{% for passage in passages -%}
    - {{passage.text|safe}}
{% endfor %}

Question: {{question}}

Please answer directly the question above in {{language}}.

G Prompts for automated evaluation

The four prompts evaluate answer faithfulness to source passages with increasing complexity: Zero-shot (ZS) provides basic supported/not-supported classification, Chain of Thought (COT) adds explicit reasoning steps, Annotation Guidelines (AG) includes detailed evaluation criteria, and AG+COT combines detailed guidelines with reasoning steps. All prompts output their final classification in <answer> tags.

Zero-shot Prompt (ZS)

You are an automatic annotator tasked with determining whether a given answer is grounded in the list of provided evidence passages. Your role is to carefully analyze the relationship between the answer and the evidence, and then classify the answer as either "Supported" or "Not Supported". Provide your answer directly in <answer></answer> tag.

Evidence Passages:
{% for passage in context %}
    . {{loop.index}}: {{passage.text}}
{% endfor %}

Answer: {{answer segment}}

Now provided your label directly as "Supported" or "Not Supported".
Chain of thought (COT)

You are an automatic annotator tasked with determining whether a given answer is grounded in the list of provided evidence passages. Your role is to carefully analyze the relationship between the answer and the evidence, write your reasoning in between the tags <rationale></rationale> then classify the answer as either “Supported” or “Not Supported” and write answer in <answer> </answer> tag

Evidence Passages:
{% for passage in context %}
    {{loop.index}}: {{passage.text}}
{% endfor %}

Answer: {{answer segment}}

Now provided your label directly as "Supported" or "Not Supported".

Annotation Guidelines (AG)

Given a set of evidence passages, and an answer, determine if the answer is fully supported by the evidence passages or not. Analyze each sentence of the answer carefully and verify that all information it contains is explicitly stated in or can be directly inferred from the evidence passages.

Output “Not Supported” if ANY of the following are true:

The answer contains any information not explicitly stated in or directly inferable from the passages.

The answer contradicts any information in the passages.

The answer introduces any new information not found in the passages.

The answer misrepresents or inaccurately paraphrases information from the passages.

The answer draws conclusions not logically supported by the given information.

The answer changes the level of certainty, specificity, or nuance from what is expressed in the passages.

The answer does not directly address the specific aspect asked about in the question.

The answer conflates or misrepresents separate pieces of information when summarizing multiple passages.

Output Supported otherwise.
Provide this determination without any additional explanation in <answer></answer> tags. Analyze thoroughly but output only the single-word label.

Evidence Passages:
{% for passage in context %}
    {{loop.index}}: {{passage.text}}
{% endfor %}

Answer: {{answer segment}}
Now provided your label directly as "Supported" or "Not Supported".
Annotation guidelines with Chain of thought (AG + COT)

Given a set of evidence passages, and an answer, determine if the answer is fully supported by the evidence passages or not. Analyze each sentence of the answer carefully and verify that all information it contains is explicitly stated in or can be directly inferred from the evidence passages.

Output "Not Supported" if ANY of the following are true:

The answer contains any information not explicitly stated in or directly inferable from the passages.

The answer contradicts any information in the passages.

The answer introduces any new information not found in the passages.

The answer misrepresents or inaccurately paraphrases information from the passages.

The answer draws conclusions not logically supported by the given information.

The answer changes the level of certainty, specificity, or nuance from what is expressed in the passages.

The answer does not directly address the specific aspect asked about in the question.

The answer conflates or misrepresents separate pieces of information when summarizing multiple passages.

Output Supported otherwise.
write your reasoning in between the tags <rationale></rationale> and Provide your final answer in <answer></answer> tags.

Evidence Passages:
{% for passage in context %}
    {{loop.index}}: {{passage.text}}
{% endfor %}
Answer: {{answer segment}}
Now provided your label directly as "Supported" or "Not Supported".

H  More Results on Meta-Evaluators
<table>
  <tr>
    <th></th>
    <th>EN</th>
    <th>DE</th>
    <th>ES</th>
    <th>FR</th>
    <th>HI</th>
  </tr>
  <tr>
    <td>GPT4o mini</td>
    <td><b>68.37 ± 2.84</b></td>
    <td>73.58 ± 2.86</td>
    <td>69.84 ± 2.26</td>
    <td>73.74 ± 2.29</td>
    <td>74.10 ± 2.61</td>
  </tr>
  <tr>
    <td>Qwen 2.5 32B</td>
    <td>62.45 ± 3.00</td>
    <td><b>76.79 ± 2.76</b></td>
    <td><b>71.09 ± 1.98</b></td>
    <td><b>74.38 ± 2.28</b></td>
    <td><b>75.41 ± 2.72</b></td>
  </tr>
  <tr>
    <td>Llama 3.2 90B</td>
    <td>62.60 ± 2.73</td>
    <td>63.11 ± 2.70</td>
    <td>63.36 ± 2.09</td>
    <td>63.17 ± 1.97</td>
    <td>75.05 ± 2.87</td>
  </tr>
  <tr>
    <td>Llama 3.2 11B</td>
    <td>60.29 ± 2.84</td>
    <td>59.91 ± 3.13</td>
    <td>59.02 ± 2.51</td>
    <td>65.02 ± 2.46</td>
    <td>65.22 ± 2.75</td>
  </tr>
</table>

Table 10: Balanced Accuracy for five languages and four LLMs using AG+COT prompting strategy. Standard errors are calculated using 1000 Bootstrap iterations.

<table>
  <tr>
    <th>Prompt</th>
    <th>EN</th>
    <th>DE</th>
    <th>ES</th>
    <th>FR</th>
    <th>HI</th>
  </tr>
  <tr>
    <td>ZS</td>
    <td>57.22 ± 2.14</td>
    <td>61.54 ± 1.98</td>
    <td>56.94 ± 1.26</td>
    <td>60.00 ± 1.39</td>
    <td>68.96 ± 2.22</td>
  </tr>
  <tr>
    <td>COT</td>
    <td>60.71 ± 2.34</td>
    <td>63.83 ± 2.15</td>
    <td>60.61 ± 1.44</td>
    <td>63.85 ± 1.53</td>
    <td>69.68 ± 2.42</td>
  </tr>
  <tr>
    <td>AG</td>
    <td>62.72 ± 2.24</td>
    <td><b>68.87 ± 2.27</b></td>
    <td>63.61 ± 1.36</td>
    <td>68.32 ± 1.53</td>
    <td>70.20 ± 2.02</td>
  </tr>
  <tr>
    <td>AG + COT</td>
    <td><b>63.42 ± 2.44</b></td>
    <td>68.35 ± 2.37</td>
    <td><b>65.83 ± 1.64</b></td>
    <td><b>69.08 ± 1.71</b></td>
    <td><b>72.44 ± 2.07</b></td>
  </tr>
</table>

Table 11: Balanced Accuracy for five languages using four different prompting strategies, averaged across four LLMs (GPT-4o mini, Llama 3.2 90B, Llama 3.2 11B, and Qwen 2.5 32B). Standard errors are calculated using 1000 Bootstrap iterations.

<table>
  <tr>
    <th rowspan="2">Language</th>
    <th rowspan="2">Prompt</th>
    <th colspan="4">LLMs</th>
  </tr>
  <tr>
    <th>GPT4o mini</th>
    <th>Llama 3.2 90B</th>
    <th>Llama 3.2 11B</th>
    <th>Qwen 2.5 32B</th>
  </tr>
  <tr>
    <td rowspan="4">EN</td>
    <td>ZS</td>
    <td>59.8</td>
    <td>58.0</td>
    <td>51.0</td>
    <td>60.1</td>
  </tr>
  <tr>
    <td>COT</td>
    <td>61.8</td>
    <td>59.1</td>
    <td>60.0</td>
    <td>62.0</td>
  </tr>
  <tr>
    <td>AG</td>
    <td>70.0</td>
    <td>59.4</td>
    <td>53.0</td>
    <td>68.5</td>
  </tr>
  <tr>
    <td>AG + COT</td>
    <td>68.4</td>
    <td>62.6</td>
    <td>60.2</td>
    <td>62.5</td>
  </tr>
  <tr>
    <td rowspan="4">DE</td>
    <td>ZS</td>
    <td>61.1</td>
    <td>58.6</td>
    <td>54.8</td>
    <td>71.9</td>
  </tr>
  <tr>
    <td>COT</td>
    <td>63.3</td>
    <td>60.0</td>
    <td>59.6</td>
    <td>72.7</td>
  </tr>
  <tr>
    <td>AG</td>
    <td>72.0</td>
    <td>68.6</td>
    <td>60.6</td>
    <td>74.6</td>
  </tr>
  <tr>
    <td>AG + COT</td>
    <td>73.7</td>
    <td>63.3</td>
    <td>60.0</td>
    <td>76.8</td>
  </tr>
  <tr>
    <td rowspan="4">ES</td>
    <td>ZS</td>
    <td>53.8</td>
    <td>56.8</td>
    <td>53.7</td>
    <td>63.7</td>
  </tr>
  <tr>
    <td>COT</td>
    <td>56.2</td>
    <td>58.2</td>
    <td>61.9</td>
    <td>66.2</td>
  </tr>
  <tr>
    <td>AG</td>
    <td>69.7</td>
    <td>59.6</td>
    <td>54.3</td>
    <td>70.8</td>
  </tr>
  <tr>
    <td>AG + COT</td>
    <td>69.9</td>
    <td>63.4</td>
    <td>59.1</td>
    <td>71.1</td>
  </tr>
  <tr>
    <td rowspan="4">FR</td>
    <td>ZS</td>
    <td>59.5</td>
    <td>56.3</td>
    <td>56.0</td>
    <td>68.5</td>
  </tr>
  <tr>
    <td>COT</td>
    <td>60.4</td>
    <td>58.5</td>
    <td>65.1</td>
    <td>71.6</td>
  </tr>
  <tr>
    <td>AG</td>
    <td>74.5</td>
    <td>62.4</td>
    <td>60.3</td>
    <td>76.2</td>
  </tr>
  <tr>
    <td>AG + COT</td>
    <td>73.7</td>
    <td>63.2</td>
    <td>65.0</td>
    <td>74.4</td>
  </tr>
  <tr>
    <td rowspan="4">HI</td>
    <td>ZS</td>
    <td>72.2</td>
    <td>64.3</td>
    <td>66.1</td>
    <td>73.3</td>
  </tr>
  <tr>
    <td>COT</td>
    <td>72.1</td>
    <td>67.6</td>
    <td>65.4</td>
    <td>73.8</td>
  </tr>
  <tr>
    <td>AG</td>
    <td>73.8</td>
    <td>67.1</td>
    <td>65.5</td>
    <td>74.5</td>
  </tr>
  <tr>
    <td>AG + COT</td>
    <td>74.2</td>
    <td>75.1</td>
    <td>65.2</td>
    <td>75.5</td>
  </tr>
</table>

Table 12: Meta evaluation results for all combination of LLMs and prompts we have tested.
<table>
  <tr>
    <th>Fine-Grained Label</th>
    <th>#S</th>
    <th>GPT-4o mini</th>
    <th>Llama 3.2 90B</th>
  </tr>
  <tr>
    <td>Logical conclusion</td>
    <td>151</td>
    <td>80.1</td>
    <td>87.4</td>
  </tr>
  <tr>
    <td>Direct paraphrase</td>
    <td>182</td>
    <td>93.4</td>
    <td>97.3</td>
  </tr>
  <tr>
    <td>Other</td>
    <td>2</td>
    <td>50.0</td>
    <td>50.0</td>
  </tr>
  <tr>
    <td>We. Acc Sup.</td>
    <td></td>
    <td>87.2</td>
    <td>92.5</td>
  </tr>
  <tr>
    <td>Adds new information</td>
    <td>81</td>
    <td>67.9</td>
    <td>33.3</td>
  </tr>
  <tr>
    <td>Nuance shift</td>
    <td>30</td>
    <td>26.7</td>
    <td>16.7</td>
  </tr>
  <tr>
    <td>Mis-referencing</td>
    <td>20</td>
    <td>35.0</td>
    <td>20.0</td>
  </tr>
  <tr>
    <td>Contradiction</td>
    <td>32</td>
    <td>65.6</td>
    <td>50.0</td>
  </tr>
  <tr>
    <td>Opinion as fact</td>
    <td>12</td>
    <td>66.7</td>
    <td>25.0</td>
  </tr>
  <tr>
    <td>Other mistake</td>
    <td>19</td>
    <td>84.2</td>
    <td>52.6</td>
  </tr>
  <tr>
    <td>Wrong reasoning</td>
    <td>10</td>
    <td>80.0</td>
    <td>40.0</td>
  </tr>
  <tr>
    <td>We. Acc NS.</td>
    <td></td>
    <td>60.3</td>
    <td>33.8</td>
  </tr>
  <tr>
    <td>BAcc</td>
    <td></td>
    <td>73.7</td>
    <td>63.2</td>
  </tr>
</table>

Table 13: Accuracy of GPT-4o mini and Llama 3.2 90B on various fine-grained labels, when evaluating French language using AG + COT prompt

I Fine grained Analysis

To demonstrate the benchmark’s capability for fine-grained analysis, Table 13 breaks down automatic evaluation performance by error type for two models: GPT-4o mini and Llama 3.2 90B, both evaluating French answers using AG + COT prompt. The upper section of the table shows the distribution of errors when the ground truth class is Supported, categorized into logical conclusion, direct paraphrase, or other correct categories. The lower section details the types of mistakes made by each model for the Not supported category. We observe that both models show a similar pattern for the Supported category, with logical conclusions being the most common (69.8% for GPT-4o mini and 76.0% for Llama 3.2 90B), followed by direct paraphrases. For the Not supported category, both models struggle most with detecting “adding new information” (32.1% and 40.0% of mistakes, respectively) and “nuance shifts” (27.2% and 18.5%). The ranking of error types is consistent across both models, despite their different overall BAcc (73.7% for GPT-4o mini vs 63.2% for Llama 3.2 90B see Table 12 in Appendix), suggesting similar challenges in evaluation. The benchmark’s fine-grained labeling system provides a detailed view of evaluation challenges across languages. By categorizing different types of faithfulness violations, it reveals which specific errors are harder to detect for each model, offering insights into both the strengths and limitations of automated evaluation methods.

Reviewer paper summaries: This paper proposes a new benchmark for multilingual RAG systems that relies on the inputs of the native language and expert annotations, unlike the MT-based approaches. Comparison with typical MT benchmarks of the approach showed that the former had limitations and LLM performance gaps at that present.

The paper introduces MEMERAG, a multilingual meta-evaluation benchmark designed to assess retrieval-augmented generation (RAG) systems. Building on the MIRACL dataset, it evaluates both human and automated metrics for faithfulness and relevance across five languages (English, German, Spanish, French, Hindi). The study demonstrates the limitations of translation-based benchmarking and offers insights into multilingual RAG system performance while providing recommendations for evaluator development.
Summary of strengths: 1. Authors extend the popular MIRACL dataset by adding responses and human judgement annotations for the native Q in five languages.
2. The benchmark enables critical use cases, such as evaluating multilingual RAG systems, developing multilingual evaluators, and analyzing native versus translated data's effectiveness.
3. Language Diversity: While 5 language is a small list, the authors make sure to include LR languages to get a holistic overview of performance as compared to high resource languages like English.
4. Usage of Relevance (and to some extent faithfulness) like standard metrics for evolution of RAG systems makes it easy for practitioners to experiment and test.

1. Introduces a native multilingual benchmark for evaluating RAG systems
2. Provides Insights and detailed analyses for building robust automated evaluators.
4. Employs human annotation protocols with high inter-annotator agreement, ensuring reliable data quality.
5. Offers a comparison between native and machine-translated benchmarks, highlighting their discrepancies and implications.
Summary of weaknesses: 1. Annotation: Extremely low inter-annotator agreement is shown to affect faithfulness with fine-grain labels for DE, ES, and HI languages by the annotators as presented in table 2, and this raises question on the validity of the results for the three languages.
2. Experiment Concerns: Non-parallel design of questions across languages causes inconsistency that certainly impacts your findings-in cross-linguistic performance evaluation, automated evaluator rankings, and comparisons of native versus translated data.
Ex: Hindi factoid question such as "Which gas is used in cancer treatment" tends to have very short, highly relevant answers, which increases the relevance scores. However, a French question such as "What are the fastest trains in France" requires nuanced responses, which decreases faithfulness scores. Performance differences may reflect dataset inconsistencies rather than true LLM capabilities. Rankings of models and evaluators could be misleading due to uneven question complexity.
3. LLM as Judge Evaluation:  Here, Rankings are based here on their capacity to estimate faithfulness and relevance across languages. Table 5 shows variations in BAcc for LLM evaluators that could be due to inconsistency in the data rather than anything related to their performance.
4. Non-parallel questions mean the MT dataset evaluation introduces additional variability, as observed in Table 7, where rankings shift unpredictably depending on dataset setup. This makes it unclear whether the MT dataset or non-parallel design causes the ranking discrepancies.
Suggestion for 2,3 and 4): Use a subset of questions across all languages, such as "What is the capital of India?", to ensure fair comparisons and mitigate question-induced variability.

1.  Covers only few languages from multiple families, addressing both high- and low-resource scenarios.
2.  Paper lacks comprehensive strategies for addressing evaluator shortcomings in low-resource languages.
3. Annotation costs and resource requirements make it challenging to scale the benchmark beyond the covered languages to increase usability
4. Explanations on why certain languages perform better than others , for example Hindi is not sufficiently substantiated.
Comments, suggestions and typos: 1. Create a parallel questions for a subset of languages to facilitate direct comparisons and minimize language complexity bias.
2. Provide definition of terms used in paper like BAcc, CoT etc in introduction for readability.
3. Discuss findings more. For ex: Why does french have low faithfulness scores despite being high resource? and why does Hindi have high relevance?
4. Add more languages to improve the benchmark's generalization.
5. Might be a good idea to include relation between pre-training data and model performance across some of these languages.
6. Improve the annotation process with clearer guidelines to improve faithfulness scores.
7. What does "challenging" in line 484 mean? could be better defined for clarity.
8. Add 2nd level of labels in table 3 for faithfulness and relevance.

1. L17 - spelling for "Langauge"
2. Extend benchmark dataset
3. Substantiate performance trends across languages
Overall assessment scores: 3

3