INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge

Authors: Angelika Romanou, Negar Foroutan, Anna Sotnikova, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Zeming Chen, Mohamed A. Haggag, Snegha A, Alfonso Amayuelas, Azril Hafizi Amirudin, Danylo Boiko, Michael Chang, Jenny Chim, Gal Cohen, Aditya Kumar Dalmia, Abraham Diress, Sharad Duwal, Daniil Dzenhaliou, Daniel Fernando Erazo Florez, Fabian Farestam, Joseph Marvin Imperial, Shayekh Bin Islam, Perttu Isotalo, Maral Jabbarishiviari, Börje F. Karlsson, Eldar Khalilov, Christopher Klamm, Fajri Koto, Dominik Krzemiński, Gabriel Adriano de Melo, Syrielle Montariol, Yiyang Nan, Joel Niklaus, Jekaterina Novikova, Johan Samir Obando Ceron, Debjit Paul, Esther Ploeger, Jebish Purbey, Swati Rajwal, Selvan Sunitha Ravi, Sara Rydell, Roshan Santhosh, Drishti Sharma, Marjana Prifti Skenduli, Arshia Soltani Moakhar, Bardia soltani moakhar, Ayush Kumar Tarun, Azmine Toushik Wasi, Thenuka Ovin Weerasinghe, Serhan Yilmaz, Mike Zhang, Imanol Schlag, Marzieh Fadaee, Sara Hooker, Antoine Bosselut
Primary area: datasets and benchmarks
Keywords: evaluation, multilinguality, large language models

Abstract:
The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (i.e., multilingual LLMs) is bottlenecked by the lack of high-quality evaluation resources in languages other than English. Moreover, current practices in multilingual benchmark construction often translate English resources, ignoring the regional and cultural knowledge of the environments in which multilingual systems would be used. In this work, we construct an evaluation suite of 197,243 QA pairs from local exam sources to measure the capabilities of multilingual LLMs in a variety of regional contexts.
Our novel resource, INCLUDE, is a comprehensive knowledge- and reasoning-centric benchmark across 44 written languages that evaluates multilingual LLMs for performance in the actual language environments where they would be deployed.

Decision: accept (spotlight)
Reviewer scores: 8, 5, 8, 8

Reviews and discussion:
<thread>
creation date:
1730115176079

thread_type:
official_review

<thread_object>
creator: reviewer_i51g

<content>
summary:
The paper presents a new resource to evaluate the multilingualism of LLMs. The dataset, called Include, comprises multiple-choice questions from 44 languages about regional and global knowledge. The authors evaluate a battery of LLMs ranging from closed-source models (GPT-4o) to a variety of open-source models (C4AI-Aya, Mistral, Gemma, Qwen, Llama, XGLM, and Bloom). The paper presents a thorough analysis, comparing the performance of the models in different languages, domains, and regional/global knowledge. Finally, the authors discuss the challenges of evaluating these models in a setting with several languages.

strengths:
* The paper is well-written and is easy to follow.
* The resource presented in this paper addresses a very prevalent challenge: the disparity in resources involved in the development of LLMs for English and non-English languages. Moreover, not only allows for the assessment of the quality of LLMs in the given languages but also in their regional cultural knowledge, which is most of the time ignored by the large body of research.

weaknesses:
I did not find any major weaknesses in this paper. However, here are some possible improvements for this work:
* **Asses the difference in performance when the model is asked in English**: Etxaniz et al. (2024a) showed that these LLMs perform better in English although they understand the target language. They propose the " Self-translate " method, which prompts the same model to first translate the example into English before answering the question. Although I do not think you need to necessarily apply the "Self-translate" method, it would be nice to translate the examples into English (using machine translation) and compare the difference in performance. This way you can assess the knowledge encoded in the LLMs separately from their language capabilities.

questions:
I have a couple of comments and questions about your work:
* **I missed some key references**: There are a couple of papers that are very aligned with this work, for instance, in the line of generating evaluation resources for LLMs in low-resource settings (particularly Basque):
  * (Etxaniz et al., 2024b): They present (along with pre-training data and models) an evaluation suite for Basque based on C1-level certificate of proficiency exams, reading comprehension exercises, trivia questions, and  Public Service examinations.
  * (Etxaniz et al., 2024c): They analyze the effect of regional vs non-regional knowledge of the LLM using an evaluation dataset that is parallel in English and Basque.
* **Have you tried any variants of Claude?** As for my experience, Claude performs better than GPT-4o in a couple of languages other than English that I tested. I am not asking to repeat the experiments with Claude, but I just have the curiosity of whether my personal feeling is reflected in your data.
* **Remove Answer Acc. from Table 1:** This metric is not comparable among the models in the table (as far as I understand). This can lead to confusion where worse models perform better just because they generate more wrongly formatted answers. If not removed, I would at least remove the bold results to avoid comparisons.


### References:
 * [Do Multilingual Language Models Think Better in English?](https://aclanthology.org/2024.naacl-short.46/) (Etxaniz et al., 2024a)
 * [Latxa: An Open Language Model and Evaluation Suite for Basque](https://aclanthology.org/2024.acl-long.799/) (Etxaniz et al., 2024b)
 * [BERTAQA: How Much Do Language Models Know About Local Culture?](https://arxiv.org/pdf/2406.07302) (Etxaniz et al., 2024c)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730311250040

thread_type:
official_review

<thread_object>
creator: reviewer_e3pj

<content>
summary:
This work identifies a lack of high-quality multilingual data available to evaluate LLMs. Existing resources are often translated from English and focus on a specific regions or domains. To address this gap, the authors introduce a large multi-lingual evaluation benchmark from close to 2000 regional exams across 44 languages, including high-, mid and low-resource languages. A range of 15 models is then evaluated on the new benchmark. The results indicate that performance largely varies across models and languages and there is still room for improvement even among the best performing models.

[see below for my response to the author comments.]

strengths:
* The paper introduces a large and high-quality evaluation dataset that addresses previous short-comings of existing LLM benchmarks. 
* Specifically, it identifies the fact that most existing multi-lingual resources are translated and therefore do not adequately reflect regional and cultural aspects that become relevant when models are actually deployed in respective environments.
* The paper’s structure is clear, the English is good and it is easy to follow
* The main contribution is an LLM benchmark, and as such it is not conceptually novel. However, it addresses a relevant short-coming and the introduction of „regional knowledge“ makes it original.
* Results are discussed in detail and Section 5 and the Appendix

weaknesses:
* The paper argues that existing LLM benchmarks do not adequately cover multi-linguality nor regional knowledge and it lists related benchmarks in the introduction and related work. However, this argument is only on a qualitative level and would benefit from quantitative facts about, the size, number of languages, etc. of individual existing benchmarks. This could be included in a table providing an overview. [addressed]

* The paper is long on numbers, but short on more qualitative insights. For example, if a model fails to answer an exam question, is this due to a linguistic failure (the model fails to understand the question) or does it lack the regional/cultural knowledge? What types of errors do the models make? Insights of this kind would be crucial to make the results of the paper actionable for future model development, and while difficult to carry out automatically at scale, could be conducted manually on a sample for illustrative purposes. [substantially addressed]

* The evaluation also does not contain any hard evidence that the knowledge contained in the proposed benchmark cannot be tested by existing ones. It would be interesting to see correlations between performances on different benchmarks. This would be straightforward to do at least for the subsets of external benchmarks that are already included under „Rounding up the Benchmark“ in Section 3.1. [addressed]

questions:
* On the technical side, it is not clear to me how exactly the model’s answer to a question is extracted from its output. In the first paragraph of Section 5.4 (lines 440-448), the authors mention that the adopt the prompt template by Hendrycks et al. and mention that the model does follow it consistently. What does the expected output look like and how exactly is the answer retrieved in case the model does not follow it? An example would be beneficial.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730366948926

thread_type:
official_review

<thread_object>
creator: reviewer_gcel

<content>
summary:
They construct a benchmark including 197,243 QA pairs in 44 languages. They collect it from local exam sources to measure the capabilities of multilingual LLMs in a variety of regional contexts. Experimental results show that these models perform inconsistently across different languages, particularly struggling with region-specific questions.

strengths:
- The dataset is extensive, covering a broad range of domains and covering various application scenarios.

- The writing is generally clear.

weaknesses:
- Experimental analysis is not quite enough and needs more takeaways. 

Many of the experimental conclusions mentioned are known or intuitive. For example: In line 323 "Most instruction-tuned models perform slightly worse or on par with their base counterpart" which has been discovered in previous work[1][2] on English training corpus. Also for conclusions like LLMs' performance is poor on unseen language during pretraining. 

Although a lot of effort has been put into building this benchmark, I feel that there is still a lack of takeaway, especially considering that one of the purposes of the Evaluation model is to improve the model. It's better to provide more find-gained analysis and insight like Figure 4 and also the insightful display (on the Main page rather than the Appendix) for conclusions like line142 model performs
worst on cultural questions.

- Doubts about the reliability of experimental findings

As you mentioned, some questions are "Cultural" and some are "Implicitly Regional". I am curious about the instructions for evaluating these two types of questions. Have you given the country or culture context of the questions being asked like: answer the question under  XXX culture? 

- The Question Format is simply MCQ

Understanding that MCQ is more accessible and easier to deal with, however, open-ended questions can reflect the models' performance on real application scenarios.



[1] Urial: Tuning-Free Instruction Learning and Alignment for Untuned LLMs, 2023

[2] A+ B: A General Generator-Reader Framework for Optimizing LLMs to Unleash Synergy Potential, ACL, 2024

questions:
Please refer to the Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730727260359

thread_type:
official_review

<thread_object>
creator: reviewer_bvnn

<content>
summary:
This paper collects multiple-choice questions in a wide range of languages other than English and evaluates a selection of multilingual LLMs. For further analysis, the questions are categorized as cultural or region-specific (subdivided further into region-implicit and region-explicit). The evaluation shows that there is room for improvement in LLM performance and that the performance varies across different languages.

====
The score is increased after the rebuttal.

strengths:
* The paper presents a new benchmark called Include, which collects multiple-choice questions sourced from existing exams in a variety of languages and domains. The selection of exams is conducted through a community survey, which appears to be an appropriate method for reaching out to native speakers and obtaining links from them. Two down-sampled versions of Include are available, which offer an equal amount of tasks in different languages and domains.

* The paper evaluates the performance of 15 multilingual LLMs in five-shot and zero chain-of-thought fashion. 

* The paper analyses the results with respect to language, script, domain, and question category (cultural, region-implicit, region-explicit, academic) and formulates clear take-away messages.

weaknesses:
* Only two LLMs are evaluated using both 5-shot and 0-shot CoT methods. It's unclear why the 0-shot CoT evaluation is missing for most models.

* Although the paper focuses on evaluating multilingual models, which seems to be a practical scenario, an ablation study with monolingual LLMs in a subset of languages could establish an upper bound for performance.

* The paper does not provide the actual names of the collected exams, along with their publication dates and formats. This information could be important for considering potential leakage to the LLM's pre-training data.

questions:
* Why are only two LLMs evaluated using both 5-shot and 0-shot CoT methods? 

* Have you try prompting the LLM in English and the target non-English question in 5-shot evaluation? Does the prompt language affect the performance?

* What is the chance that the LLMs have already been trained on some of the Include questions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733310584687

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of discussion

comment:
We sincerely thank all the reviewers for their valuable feedback and the time they dedicated to reviewing our work, INCLUDE. Below, we provide a summary of the main points discussed during the review period.

Overall, the feedback highlighted the dataset's comprehensiveness, authenticity, and relevance, along with the clarity and impact of the analysis. Specifically:
- The reviewers praised INCLUDE for its **extensive coverage of regional knowledge across languages and domains**, rather than relying on Western-centric paradigms.
- They also highlighted the **community-driven methodology** for selecting exams.
- The writing was noted for its **clarity and strong articulation**, while the analysis of multilingual LLM performance was appreciated for its clear and **actionable insights**. 
- Furthermore, the thorough discussion of results in Section 5 and the Appendix was found to be both detailed and informative.

We also received constructive feedback from the reviewers, and we addressed it with new analysis and experiments, which we added to the paper and summarize below:

- **New Chain-of-Thought (CoT) experiments:** Based on reviewer BvNN’s recommendation to conduct CoT experiments on smaller models (<= 8B parameters), we added experiments on the most performant small variants from the model families used in our initial experiments. Our results align with findings in existing literature, confirming smaller models struggle in zero-shot CoT settings.

- **New experiments with English prompts:** Our initial prompts were in the same language as the evaluation questions. Following reviewer BVNN and i51G’s feedback, we conducted new experiments where the user prompt instructions were translated into English while keeping the question and choices in the native language. Our results suggest that English prompts can provide modest benefits for most models, but that performance is generally within 1-2% of the performance of native language prompts. 

- **New experiments on monolingual models:** Based on reviewer BvNN’s question, we evaluated seven open-source monolingual models on the relevant language-specific subsets of INCLUDE, (i.e., the languages these models were pre-trained on). Most monolingual models underperform the state-of-the-art small multilingual model Qwen-2.5 (7B). 

- **New experiments with region-aware prompts:** Based on reviewer gceL’s question, we conducted experiments that identify the region and language of each sample in the prompt instructions of the samples. The results show that providing explicit region and language information does not enhance model performance on regional questions. 

- **Analysis on data contamination:** Based on reviewer BvNN’s question, we analyze the degree of contamination on four main-stream multilingual models using the mink%++ method for training data detection in LLMs, which showed small amounts of contamination among these models. To further mitigate the impact of benchmark leakage, we will release the complete dataset, comprising 197,243 entries, incrementally over the next year.

- **Analysis of output errors:**  Following reviewer E3PJ’s feedback, we conducted a more detailed error analysis of model outputs for INCLUDE, focusing on six languages (Chinese, Turkish, Bengali, Greek, Korean, Armenian). Our analysis revealed that the model's errors were distributed as follows: 38.6% were due to a lack of regional knowledge, 32% resulted from hallucinations, 26.7% were factual mistakes, and 2.7% were computational errors. 

Finally, to further enhance the clarity and reproducibility of our work, we also provided the reviewers with additional details on the following points:
1. The methodology used for answer extraction in the evaluation process.
2. Clarifications regarding the existing benchmarks mentioned in the paper, specifically emphasizing those featuring original content (not machine-translated) and highlighting their coverage of languages and knowledge domains.
3. The limitations in the scope of existing benchmarks.

**All additional experiments, results, and analyses conducted for this rebuttal have been incorporated into the paper, either within the main sections or in the Appendix.**
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a comprehensive multilingual evaluation benchmark called INCLUDE, comprising 197,243 multiple-choice questions across 44 languages. It was sourced from local exams to evaluate LLMs' capabilities in regional contexts. Experimental results show that current multilingual LLMs perform inconsistently across languages and struggle particularly with region-specific questions.

The benchmark is large-scale and incorporates authentic regional knowledge. Experimental analysis is conducted across multiple dimensions (language proficiency, regional knowledge types, prompting strategies).

The reviewers raise the issue of the lack of detailed comparison with existing benchmarks, the design of the problem format (only MCQ type questions), and also potential confounding factors in comparing difficulty across regional question types.

Overall I think the novelty of the newly introduced datasets is actually limited, with existing similar benchmarks already available. But I do agree with the reviewers that such large-scale resources might be useful for the community. Based on these considerations, I recommend accepting this paper.