INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge
Authors: Angelika Romanou
Negar Foroutan
Anna Sotnikova
Sree Harsha Nelaturu
Shivalika Singh
Rishabh Maheshwary
Micol Altomare
Zeming Chen
Mohamed A. Haggag
Snegha A
Alfonso Amayuelas
Azril Hafizi Amirudin
Danylo Boiko
Michael Chang
Jenny Chim
Gal Cohen
Aditya Kumar Dalmia
Abraham Diress
Sharad Duwal
Daniil Dzenhaliou
Daniel Fernando Erazo Florez
Fabian Farestam
Joseph Marvin Imperial
Shayekh Bin Islam
Perttu Isotalo
Maral Jabbarishiviari
Börje F. Karlsson
Eldar Khalilov
Christopher Klamm
Fajri Koto
Dominik Krzemiński
Gabriel Adriano de Melo
Syrielle Montariol
Yiyang Nan
Joel Niklaus
Jekaterina Novikova
Johan Samir Obando Ceron
Debjit Paul
Esther Ploeger
Jebish Purbey
Swati Rajwal
Selvan Sunitha Ravi
Sara Rydell
Roshan Santhosh
Drishti Sharma
Marjana Prifti Skenduli
Arshia Soltani Moakhar
Bardia soltani moakhar
Ayush Kumar Tarun
Azmine Toushik Wasi
Thenuka Ovin Weerasinghe
Serhan Yilmaz
Mike Zhang
Imanol Schlag
Marzieh Fadaee
Sara Hooker
Antoine Bosselut
Venue: ICLR 2025 Spotlight (2025)
Keywords: evaluation
multilinguality
large language models
Primary area: datasets and benchmarks

Abstract: The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (i.e., multilingual LLMs) is bottlenecked by the lack of high-quality evaluation resources in languages other than English. Moreover, current practices in multilingual benchmark construction often translate English resources, ignoring the regional and cultural knowledge of the environments in which multilingual systems would be used. In this work, we construct an evaluation suite of 197,243 QA pairs from local exam sources to measure the capabilities of multilingual LLMs in a variety of regional contexts.
Our novel resource, INCLUDE, is a comprehensive knowledge- and reasoning-centric benchmark across 44 written languages that evaluates multilingual LLMs for performance in the actual language environments where they would be deployed.

Published as a conference paper at ICLR 2025
INCLUDE: EVALUATING MULTILINGUAL LANGUAGE UNDERSTANDING WITH REGIONAL KNOWLEDGE
Angelika Romanou's '* Negar Foroutan's Anna Sotnikova' ; Zeming Chen'. Sree Harsha Nelaturu Shivalika Singh?, Rishabh Maheshwary's Micol Altomare Mohamed A. Haggag", Imanol Schlag" Marzieh Fadace". Sara Hooker Antoine Bosselut
EPPF. "Cohere For AI, CChhere For Al Community, 'ETH Zurich, Saarland University, University of Toronto, ServicceNw, GGorgiia Institute of Technology
Dataset Contributors:
Snegha A, Alfonso Amayuelas, Azril Hafizi Amirudin, Viraat Aryabumi, Danylo Boiko, Michael Chang, Jenny Chim, Gal Cohen, Aditya Kumar Dalmia, Abraham Diress, Sharad Duwal, Daniil Dzenhaliou, Daniel Fernando Erazo Florez, Fabian Farestam, Joseph Marvin Imperial, Shayekh Bin Islam, Perttu Isotalo, Maral Jabbarishiviari, Börje F. Karlsson, Eldar Khalilov, Christopher Klamm, Fajri Koto, Dominik Krzemiński, Gabriel Adriano de Melo, Syrielle Montariol, Yiyang Nan, Joel Niklaus, Jekaterina Novikova, Johan Samir Obando Ceron, Debjit Paul, Esther Ploeger, Jebish Purbey, Swati Rajwal, Selvan Sunitha Ravi, Sara Rydell, Roshan Santhosh, Drishti Sharma, Marjana Prifti Skenduli, Arshia Soltani Moakhar, Bardia Soltani Moakhar, Ran Tamir, Ayush Kumar Tarun, Azmine Toushik Wasi, Thenuka Ovin Weerasinghe, Serhan Yilmaz, Mike Zhang
ABSTRACT
The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential eco- nomic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (i.e., multilingual LLMs) is bottlenecked by the lack of high-quality evaluation resources in languages other than English. Moreover, current practices in multilingual benchmark construction often translate English resources, ignoring the regional and cultural knowledge of the environments in which multilingual systems would be used. In this work, we construct an evaluation suite of 197,243 QA pairs from local exam sources to measure the capabilities of multilingual LLMs in variety of regional contexts. Our novel resource, INCLUDE is a comprehensive knowledge- and reasoning-centric benchmark across 44 written languages that evaluates multilingual LLMs for per- formance in the actual language environments where they would be deployed.
INTRODUCTION
The rapid advancement of AI technologies underscores the importance of developing LLMs that are proficient across diverse linguistic and cultural contexts, ensuring fair and equitable performance for stakeholders from various language groups. However, the lack of high-quality evaluation benchmarks in many languages discourages practitioners from training multilingual LLMs to meet this challenge. This evaluation gap limits the effective deployment of LLMS for many regions, exacerbates digital divides, and inhibits the economic and societal value of AI tools in many underserved communities.
The source of this gap is the multitude of challenges evaluating L.MSs for multilingual contexts. First, at a meta-level, the majority of benchmarks for LLMs are only in English (Hendrycks et al.)
2020 inter alia). While non-Englin non-English benchmarks exist for some tasks (Singh et al. 2024 Aakankshal
al. 2024 Pozzobon et al. 2024), they usually focus on single languages (Li al. 2023
Koto et al.
"Corresponding cmaii email: angel i ka. 1 Equal Supervision
https::/huugiingfaass http:///nnnaaaee///aaaaa//////ea
ude- base-44

---

Published as a conference paper at ICLR 2025
2024), specific regions (Cañete et al., 2020: Guevara-Ruie
al., 2020] Cahyawijaya al. 2022
Adelani et al.) 2024 Etxamiz et al. 2024b)) or a particular domain (Wang et al. 2024a), ignoring the
importance of joint evaluation to trace and unlock the benefits that mulnlingual capabilities could
bring to low-resource languages (Pfeiffer et al. 2022 Üstün et al. 2024 Aryabumi et al. 2024).
Technical challenges also abound due to the manner in which multilingual datasets are often collected.
Certain datasets are constructed using manually applied templates, resulting in low prompt and
completion diversity (Muennighoff et al., (022). Many more are composed of translations from
high-resource languages (e.g.. English: Lai el al. [2023] Foroutan et al. 2023 Holterann el al. [2024]
Myung et al., 2024). These datasets often contain errors (Ponti et al. [020 Plaza al. 20044) and
create translationese artifacts (Vanmassenhove et al. [222]] Hartung et al. 20233] Savoldi et al. 2021]
Ji et al., 2023). Most importantly, they do not accurately reflect the regional and cultural contexts
captured by different languages (Aakanksha et al. 2024 Awad et al. 2020 Ramezani & Xu 2023
Singh et al. 2024). As seen in Figure [ (a) (Regional Knowledge), a legal question posed in English, Russian, Greek would likely reflect user located in a different environment, where different laws may apply to respond correctly. Similarly, also seen in Figure !
(a) (Cultural Knowledge), historical or cultural perspectives the same topic may differ among the populaces of different regions.
To resolve this gap, we design a pipeline to collect a large multilingual language understanding benchmark (i.e., INCLUDE) by collecting regional resources (e.g., educational, professional, and practical tests) that are specific to countries and originally created by native speakers of each country's official languages. This collection avoids translationese (Bizzoni al. 2020) and also captures cultural nuances associated with each language, enabling rigorous evaluation of how state-of-the-arr models serve diverse language users around the world.
In our experiments, we sample INCLUDE into two subsets for different evaluation budgets and assess an array of closed and open models on these partitions. Our results demonstrate that current models achieve high variance in performance between different languages in INCLUDE, and that models often struggle with questions requiring regional knowledge. Further analysis reveals that models score particularly low on languages on which they are not intentionally trained (i.e., limiting regional knowledge acquisition), and that the possibility of transferring global (i.e., English-aligned) perspectives improves performance for less regional topics across languages.
PRELIMINARIES: LANGUAGE & KNOWLEDGE
Language availability. Languages are typically characterized as a high, medium, or low resource based on reported language availability (Joshi al. 2020), i.e., the amount of data in a language that is available online. Interestingly, the language availability of documents used for training models (Penedo et al. 2024 Xue 2020 Conneau al. 2020: Computer 2023 Ustün al. 2024 Singh et al.] (024)) differs drastically from the language distribution of non English LLM benchmarks, with the latter being more scarce. Inspired by this discrepancy, we include 44 languages in our INCLUDE we characterize the included languages based on their reported availability Xue 2020). and in Table [ show further detailed metadata for each language.
benchmark. In Figure 1
in the mC4 corpus
Language represents regional knowledge. For LLM-based systems to practically useful, they must enable interaction in the preferred languages of their users and be knowledgeable of the environments of those users. We define regional knowledge as the specifie information, culture, and practices related to a local environment that is relevant for a user's ' context. However, LLMS such as GPT-4 tend to exhibit Western bias (Tao et al.) 2024) due to the overrepresentation of Western text in training data (AlKhamissi et al. 2024). In INCLUDE, we specifically include questions encompassing the regional and cultural knowledge of diverse set of high, medium, and low-resource languages.
3 THE INCLUDE BENCHMARK
INCLUDE is dataset of 197,243 MCQA pairs from 1,926 examinations across 44 languages and 15 scripts. These examinations are collected from local sources in 52 countries, representing a rich array of cultural and regional knowledge. All questions in the dataset are presented in their native languages and scripts. In this section, we describe the data collection procedure for INCLUDE. as well as additional categorical labels we assign to each question in the dataset for later analysis.
2

---

Published as a conference paper at ICLR 2025
MOOWLEGSE
Arts Humanisso
Sodal Felenoes
Bustness & Commerce
Unsennt plea0) orb/hpo Fllonnc
ZEGOTORED spoed ponyenseupg
Humanttes
History
Socadogy
Euromm
Logic & Ptelssoplyy
Payclogy
Fruncc
Blood Kloohol Hindt 1MMDR) deiving??
Klo0ho1 1380% (BAD%)
Belggan
Cultrnlogy
Geogmphy
Managemmnt
STEM
Math
Physies
Engineering
Health oriented education
Computer Seience
Charaary
ogy
Harth
Medkire
0
0.03 4
0.06
0.08
Professional Ceriificaiin & Licenees
Occupational Licemios
CULTURAL KMOOLEDGE
Muckcal Ricese
Teuchers
Men y. Jul Hhe file uisske pastie bye Tould is
dLav) licenee
Matre Icana
righrre
fite rerseyolio 33E 20B?
Aedii
Cherew
Authaaan
Pulgerion
Armertan
Roqus
cultuse and bistocy
Faench
Cantrnn
Dutch
Cradier
hungenin
Nepal
Macadoraa
Tegalog
beceuss Docephales was killed
indemssinn
Serbeen
Alorraa
Uthaaman un
Malapdaam
Ceorg an
bydasges
Spanch
Polih
Stoman
Be
Ielagu
It txppened by cooldess
.pannse
Portuguzse
Netrew
Hindl
Nallr
Kazzch
50 cesange toe Peraies Inownice
Vietommera
Turkas
Sarean
Liraszzan
Tarrd
Uzoak
(a)
(b)
Figure I: Overview of INCLUDE. (a) Motivation: Multilingual benchmarks must reflect the cultural and regional knowledge of the language environments in which multilingual LLM systems would be used. (b) INCLUDE is multilingual benchmark compiled from academic, professional, and occupational license examinations reflecting regional and cultural knowledge in 44 languages.
3.1 DATA COLLECTION
To construct INCLUDE, we collect sources of multiple-choice exams in collaboration with native speakers and regional associations. We primarily focused on three types of exams:
Academic Exams: Exams from a variety of subjects (e.g., Humanities, STEM, etc.) at different levels (e.g., middle & high school, university), including country-specific national entrance exams.
Professional Certifications Licenses: Exams issued by industry-specins regulatory bodies for specialized fields, e.g., licensing exams for areas such as legal and medical practice.
Regional Licenses: Exams administered by regional authorities that assess specific qualifications, such as driving and marine licenses.
We design INCLUDE to assess multilingual capabilities that span beyond academic knowledge to cultural and region-specific understanding. Our data collection focuses on license and certification exams that capture regional knowledge of specific countries (in their official languages), and non- translated academic content from the humanities and social sciences capture cultural knowledge.
From the collected sources, we extract the multiple-choice questions with their corresponding options and correct answers. More specifically, this data came in different formats (e.g., PDFs, Javascript HTML forms), we use multiple pipelines to extract QA samples from these sources and curate them in machine-readable manner. The goal of this stage was to automate data extraction and then rely on human evaluation for verification and feature annotation.
Quality Control with Native Speakers. After automatic extraction, we provide native speakers (co- authors in this work) with parsed multiple-choce questions to ensure they were extracted correctly from source documents. In cases of extraction mistakes, annotators manually corrected parsed questions and answer options. Annotators also filtered out samples that referred to images or tables, and verified that samples that rely on additional context (e.g., reading comprehension) include the reference text in the question field. Finally, annotators labeled each question with additional exam metadata, such as the language of the MCQ, its topic in both English and the original language, the academic level (if relevant), and the country of origin. In total, parsed and verified 118,606 samples across 1,926 exam sources, amounting to 60.2% of the total data in INCLUDE.
Rounding out the Benchmark. To complete benchmark, we also consolidate existing datasets with extensive domain coverage in single non- CMiL (Li et al. 2023), TurkishMMI. (Yüksel et al. 2024), PersianMML.U (Ghahroodi
non-Englin English languages: ArabicMMLU (Koto et al. 2024).
3

---

Published as a conference paper at ICLR 2025
Armenian
Kanji
1657
100% original dbta
Armonian
2698
100% original dota
Appancse
Bengali
15259
oriiinal
Sengali
65346
original
Tigalog
Chinese
Micherdull
12977
6.% original data
Ch nese
599
100% orgiiaal dets
Georgion
Cyrilic
25146
nnggral
43014
seigiaal
Devanageri
Tamil
6625
107% original dats
ind
Nesai
945
107% orgginal data
Tamil
Telugu
6530
1 11563
Telugs
Hangul
Vatteluttu
1781
1076 original dlte
Screan
646
1074 criginal
Holbysbm
Humanities uс4aии
STEM scroea
2457
100%
Domain-specifcc sCx0CIQ
Professional
License
Overview of the collected data grouped by script. We depict the languages associated with each script, the total samples each script, and the percentage of the samples that were collected from new sources that have not been published by the community yet.
Figure 2:
et al. 2024) and VNHSGE (Dao et al. 2023). as well as a multilingual benchmark with limited domain coverage across multiple European languages: EXAMS (Hardalov et al., 2020p. We repurpose 78,637 samples from these published benchmarks, amounting to 39.8% of the data in INCLUDE.
In sum, INCLUDE is the largest collection multilingual exam data to-date. It is composed of 197,243 QA pairs from both novel and existing sources, and covers examinations from more than 1,926 exams in different languages, 15 scripts, and 58 knowledge domains. Figure and Appendix Table summarize the language, domain, and knowledge diversity of the INCLUDE benchmark.
3.2 CATEGORIZING KNOWLEDGE
The language breadth of INCLUDE provides the opportunity to investigate what factors drive multilin- gual performance. Consequently, annotate INCLUDE samples with category labels corresponding to factors such as the topic of question and its region-speciffcity. Given the prohibitive cost of performing sample-level annotation, we only perform a coarse annotation by labeling the exam sources of QA pairs, rather than individual samples. We describe our categorization schemes below.
Academic Domain. We manually categorized 1,926 unique exams, following the methodology in Hendrycks et al. (2020). Our categorization follows a two-level taxonomy: high-level academic area (e.g., Humanities), and particular academic field within this area (e.g., History, Philosophy, Literature) Each exam is categorized based on its title, which indicates its topic (e.g., Greek History) and associated level (e.g., high school, undergraduate, professional certification). Figure 7 provides breakdown of the number of exam samples per language, organized by this taxonomy.
Regionality. To account for regional knowledge, we categorize exam questions into two major groups: region-agnostic and region-specific knowledge. Region-agnostic questions do not require knowledge of particular regions (e.g., mathematics, physics), and their answers should remain common regardless of the language in which question is posed. In total, 34.4% of all questions collected were classified as region-agnostie particular cultural geographical context. This category is further divided into three sub-categories:
contrast, region-specific questions require knowledge that may depend on
Explicitly Regional: A question is classified as region- expricit when it pertains to legal, regulatory, or procedural knowledge of regions. Examples include questions about local laws, certifications, clinical guidelines, or licensing requirements (see Figure aa)) 18.8% of all questions were region-explicia.
Cultural: Language often serves
implicit marker of culture. For instance, in Figure [a), the answer to a question about historical figures a Greek exam may reflect a different perspective than similar question posed for a Persian exam. We categorize questions cultural when they pertain
"This taxonomy is adapted from the Outline Academic Disciplines found Wikipedia.
4

---

Published as a conference paper at ICLR 2025
region's 's cultural or historical context. This category includes questions for subjects inherently tied to region's 's language, history, or social norms. 16.4% of questions were classified as cultural.
Implicitly Regional: Finally, the region- cistontimplict category is catch-all for other questions whose answers may depend on a certain degree of regional knowledge understanding. These questions are but may require regional context answer correctly. For
not explicitly regional example, business practices may be different depending on region, even if the underlying theory is common in many places. In total, 30.4% of all questions collected were classified as region-implici..
Detailed annotation procedures for these categories are described in Appendix A.4 and general statistics about regional labels per academic area and academic field are provided in Figure
4 EXPERIMENTAL SETUP
In this section, we describe our experimental settings for evaluating models on INCLUDE.
4.1 DATA SELECTION
The breadth of INCLUDE (197,243 QA pairs in 44 languages) makes it amenable to many evaluation use cases, including monolingual evaluation in 44 languages, However, for multilingual evaluation, this same scale is prohibitively expensive for many researchers F Consequently, we curate two subsets of INCLUDE for benchmarking multilingual LLMs in different resource settings.
This subset uniformly samples 22,635 QA pairs (~12% of INCLUDE) across languages, knowledge tasks, and academic levels. The goal of this subset is to develop multilingual benchmark with broad language and task coverage. Each language has a maximum of 550 samples, with 500 drawn from domains that correspond to regional knowledge and 50 from STEM subjects.
INCLUDE-BASE,
INCLUDE LICTUDE-LITE: A lightweight subset, uniformly drawn from INCLUDE-BAAN BASE, designed for rapid assessment of multilingual LLMs with a total of 10,770 samples (~6% of INCLUDE). The upper limit per language is 250 samples and only includes region-specific domains.
For standardization (and alignment with prior benchmarks: Hendrycks et al., 2020), INCLUDE- BASE and INCLUDE-LITE contain only multiple-choice questions with four answer options. Questions from INCLUDE with fewer than four options were omitted during sampling, and questions with more than four options were pruned of options until only four remained. In the following sections, we benchmark models ($5.T) and perform analysis ($5.2 5.3) on INCLUDE-BER BASE and INCLUDE-LITE.
4.2 MODELS
We assess INCLUDE on GPT-4o (Achiam et al. 2023) as a state-of of multilingual and general- purpose model. We also investigate the role of scaling by benchmarking models that self-report parameters, comparing the larger Llama-3.1-70B-Inss. (Aryabumi al. 2024), and the Qwen2.5-14B (Yang al. 2024) 14-billion parameter model with (Dubey et al. 2024). Aya-expanse-88 8B (Aryabumi et al. 2024) and Qwen2.5-
(Dubey et al. 2024). the Aya-expanse-322
7B. Additionally, we benchmark Mistral-7B (Jiang et al. (023) and Gemma-78 7B (Team al. 2024) along with their Instruct variants. We note that some of the evaluated models neither explicitly claim to support multiple languages nor disclose the languages they were pretrained on. However, in practice, they are heavily adopted in multilingual use cases relative to explicitly multilingual models. Furthermore, even reportedly multilingual models (e.g. Aya-Expanse, which supports 23 languages) do not support all 44 languages included in our benchmark.
Prompting. Following Hendrycks et al. 2020). we report both 5-shot and zero-shot scores. For the zero-shot setting, we employ a Shain-offThoughe (CoT; Wei et al. 2022) approach by appending the translation of "let's think step by step" to the prompt (Kojima et al. 2022). We evaluate models using both (1) In- In-Language (IL) Prompts, which present the prompt instructions in the same language as the sample, and (2) English (Eng.) Prompts, which provide the prompt instructions in English. For both prompting language settings, we also test a setting where we add a Regional (Reg.) prefix
"The cost of evaluating INCLUDE using GPT-4o with 5-shot demonstrations exceeded $1000.
5

---

Published as a conference paper at ICLR 2025
INCLUDE-LITE
INCLUDE-BASE
Eng- Prompt
Reg. Prompt
Reg. Eng. Prompt
Eng- Prompt
Reg. Prompt
Reg. Eng. Prompt
Model
Langs
Prompt
Prompt
GPT-40
5-shol
77.1
76.2
76.3
76.3
77.3
76.3
76.2
76.2
Zem-shot CoT
78.2
78.4
77.7
77.8
79.0
78.9
77.6
78.5
Lumm-3.11770--Ist
shohot
70.5
70.4
70.6
70.6
70.6
70.7
70.6
70.6
Zero-sbot CoT
60.6
55.3
60.2
55.4
60.6
$6.0
60.6
55.6
Ayaeexpanse-22B
23
5-shol
52.6
57.2
49.0
60.0
52.4
$6.6
49.7
60.0
Zero-shot CoT
50.6
57.1
52.5
58.0
51.4
57.7
52,9
57.8
Qwen2.5-14B
22
5-shot
60.9
61.3
60.9
60.8
61.4
61.7
61.1
61.0
Zero-shot CoT
46.8
50.7
46.5
51.4
47.3
$1.0
47.1
51.6
Aya-expanse-8B
23
37.6
46.3
38.1
48.0
37.2
46.0
37.9
47.8
Mistral-7B (v0.3)
44.0
45.0
44.0
45.2
43.3
44.9
43.8
45.0
Alstral-7B-Inst. (v0.3)
43.5
44.6
44.2
44.7
43.6
44.5
44.2
44.7
Gemma-7B
54.4
54,9
54.3
54.9
54.5
54.9
54.2
54.7
Gemma-7B-Iust.
40.2
38.7
39.7
38.7
39.7
38.1
39.2
Qwen2.5-7B
22
53.4
54.8
53.3
54.2
54.1
55.2
$4.0
54.5
Qwen2.5-7B-Ins
22
53.4
54.2
52.8
53.7
54.6
53.2
53.9
Llama-3.1-80
50.9
52.3
50.9
51.0
51.8
51.0
51.6
Llama-3.1-8B-Inst.
53.4
54.8
52.7
53.4
53.4
54.6
53.0
54.4
Table Results on INCLUDE-LITE and INCLUDE BASE.. In-language Prompt (IL) reports model accuracy when the prompt instructions are presented in the same language as the sample. English Prompt (Eng.) reports model accuracy when the prompt instructions are provided in English. In- language Regional Prompt (Reg. + II. reports model accuracy when regional prefix is added to the In-language Prompt. English Regional Prompt (Reg. + Eng.) reports model accuracy when regional prefix is added to the English Prompt. # Langs reports the number of languages from INCLUDE publicly reported to be intentionally included in the pretraining data of each model.
to the (either in in-language or English) prompt to explicitly contextualize the region and language of the sample, asking the models to consider the cultural and regional characteristics. The maximum generation lengths for the 5-shot and zero-shot CoT configurations are set to 512 and 1024 tokens. Finally, we additionally evaluate INCLUDE using the Harness-Eoe Eval framework (Gao et al. 2024 using both In-Language Prompts and English Prompts.
5 RESULTS & ANALYSIS
5.1 GENERAL PERFORMANCE
Table [ishow the performance of all models evaluated across the 44 languages in INCLUDE-BASE BASE. For larger models, e.g., GPT-40, Llama-3.1-708 Instrruct, Aya-expanse-333 and Qwen2.5-144 provide results for both 5-shot and zero-shot CoT, and report only 5-shot accuracy for other models.
Among all models, GPT-4o achieves the highest performance, reaching accuracy of ~77. % across all domains and examples. We observe that CoT prompting moderately enhances GPT- 4o's performance, particularly in Professional and STEM-related exams (Table where the most substantial improvements were seen. In contrast, the smallest gains were observed in exams related Licenses and the Humanities. Drawing on prior studies that compare CoT and non-CoT prompting strategies across different domains (Sprague et al. 2024). we hypothesize that this observation is due reasoning skills required in professional examinations (e.g., medicine, law). and computation-hest subjects in STEM. In contrast, we observe significant performance drops by 1 1-70B- and Qwen2.5- when using CoT prompting on INCLUDE--ASS with the largest drops on subjects involving more mathematical reasoning (Ta
For smaller models (S8B parameters), Gemma-7B delivers the best overall performance, with Qwen2.5-7B and @ween.55-BB-Instee closely behind. While Gemma-7B excels in the Human- ities and Licenses categories, the Qwen models surpass others in STEM, Applied Sciences, and Professional domains (Table 2 When comparing models from the same family across different scales, we observe that the Aya-expanse-333 model outperforms the 8B model by 12% on average, while the Qwen2.5-14p model shows average ~7% improvement over its 7B counterpart

---

Published as a conference paper at ICLR 2025
+20.9 %
+11.8%
+25.2%
Sparch
lush
Vetromenn
Indorach
Fanch
Geraan
Sutch
Tussin
60.2
Pomugutte
43.2
Mayy
Marddoiin
LEhaares
Bangel
62.0
Edveles
Craation
Raky
Frach
Maccdeeta
Anrbelen
Dromian
Nepel
Freith
120hh
huggats
Keach
44.1
26,3
51.7:
Geoiion
Ceed
Maledan
tergit
'4.00
Armatión
31.5
:52.4
36.5
32.4
20
40
60
40
20
40
60
ACCURACY (M)
ACCURACT
Aya-expanse (328)
Aya ex-exxanee (8B)
Qwen 2.5 (7B)
Trained on Language
Trained on Script
Neither
Average of language & script aligned
Figure 3: Performance of models stratified by language using in-language prompting. Results are grouped by whether the language was explicitly included in the pretraining dataset of the model (Trained on Language), whether a similar language with the same script was in the pretraining corpus (Trained on Script), or whether there was linguistically similar language in the pretraining corpus (Neither). Color dotted lines represent average performance for each category for particular model. Black dotted lines represent average performance across all script-aligned languages.
INCLUDE-BAR BASE across prompting settings. As the pretraining data remained consistent across the different sizes within both the Aya-expanse and Qwen2.5 model families, we conclude that, with similar training data, increasing model size significantly enhances multilingual capabilities.
Interestingly, we see little benefit instruction-tunm tuning for improving performance on INCLUDE- DESSSE. Most instruction-tuned models perform slightly worse or par with their base counterpart, with an possible
outlier performance drop of ~ 155 across prompting settings for explanation for this gap is that instruction-tune models may have been post-trained predominantly on English data, potentially diminishing multilingual capabilities acquired during pretraining.
Overall, we observe minimal performance differences across the various prompting settings, except the CoT setting where Qwen2.5-14p and Aya-expanse benefit significantly from English prompts (over the in-language variants) and Llama-3.1 3.1--0B--BBBB- Instruct performs better with in in-language prompts. Outside of these exceptions, English prompts offer modest performance improvements for most models, but this improvement remains within 1-2% of in-language prompt performance. Apart from Aya-expanse-8BB specifying the region of the question the prompt prefix does not generally improve performance, likely because the language of the question serves as implicit proxy of the relevant region of the question for most languages in our benchmark.
Finally, we observe that performance INCLUDE-LITE is nearly equivalent INCLUDE- -BAEE across all models, with differences within 1%, demonstrating its suitability for assource-constraing evaluation settings. Likewise, when using the Harness evaluation framework, model performance remains consistent, with results also within 1% (Table 4..
5.2 LANGUAGE ANALYSIS
To better understand how LLMs perform on questions in languages seen and unseen during pretraining, we take deeper look into three open models, i.e., dya-expanse-8B. -ya-expanse-32B. and Qwen2.5- 7B, for which we have details surrounding pretraining data (and its associated language distribution). In this analysis, we specifically test three language exposure scenarios: performance languages the
7

---

Published as a conference paper at ICLR 2025
model has been intentionally trained on, performance languages the model was not reported to be trained on but for which the corresponding script was reported to be trained on, and performance on completely unseen languages and scripts during pretraining.
Figure 3 presents the language-stratifer performance of these models on INCLUDE-BASE As expected, the models demonstrate better performance languages that were reported part of their pretraining data (Trained on Language). ) All models also demonstrate some degree of knowledge transfer languages they were not trained on but which share the same script as languages in their pretraining data (Trained on Script). In this scenario, Aya-expanse-32BB achieves 44. 1% accuracy, while Qwen2.5-7B reaches 51.7% accuracy, aligning with previous research that suggests shared scripts enable cross- gross-lingual transfer between languages (Muller et al., [021] [helili et al.] (024). Other factors may also contribute to each model's performance on unseen languages, though, such as cross-lingual transfer across topologically-simin data may enhance the model's ' performance on Azerbaijani (Senel et al., (0024). Pretraining data contamination, where languages that were not intended to be in the pretraining data may still be incidentally included (Blevins & Zettlemoyer 2022), may also contribute to these transfer results.
languages. For example, the presence of Turkish
Lastly, we observe that all three models perform poorly on languages whose scripts were not represented in the pretraining corpus (Neither in Figure B} with the exception of the Georgian language for the Aya-cxpanse-322 model. Data contamination is also less of confounding factor in these cases as language identification is more robust for unique scripts (Kargaran et al.) 2023). In these cases, the model often performs worse than random, likely due to not being able to produce responses in the correct format (further details in 5.4).
5.3 REGIONAL & ACADEMIC DOMAIN KNOWLEDGE PERFORMANCE
Using the category labels outlined in Section 3.2 we conduct stratified analysis of five-shot GPT- 4o's s ability to answer different types of regional questions in INCLUDE-BASE We first note that overall performance differs strongly between languages. In some languages, the model consistently performs well across all academic domains and regional knowledge types, while in others, it struggles across the board (see Appendix Tables
10 and IT].
However, lower performance in certain languages is often linked to questions requiring regional knowledge (e.g., historical knowledge, professional certifications, medical licenses), suggesting that the model's knowledge about regions varies significantly and that its performance across languages reflects this differential. Among regional categories, in Appendix Figure we see the model performs worst on cultural questions, followed by region- expiin-- questions. Professional certification exams in different regions are particular challenge for GPT-4o (average 68. (8..%). In Persian, the model's accuracy on certification exams is notably low (43.2%), whereas it performs better on subjects such as Geography and Sociology (over 66%). Similarly, in Greek, GPT-4o achieves an average accuracy of 71.3%, but only 54. 1 % on medical license questions. Further language performance comparisons across various academic disciplines are shown in Appendix Figure
Despite these findings, we note that performance across regional question types (e.g., region-agnostic, cultural) is not deconfounded from other features such as topical difficulty and academic level. Indeed, we observe that region-agnostic questions among the most challenging as models struggle with the mathematical nature of many STEM topics (Frieder et al. 2023; Borges et al. 2024).. On average, subjects such as Mathematics and Chemistry show the lowest average accuracies (Appendix Figure Unfortunately, as the region label of any question depends on its subject, it is naturally confounded with the model's ability in that subject, regardless of whether the subject regional or not.
History is one of the few fields where we can achieve a more controlled study of regional difference as exams can be divided into two categories: those testing region-speciff historical knowledge (e.g.. "Armenian history", which we label as cultural) and general history taught in a particular region (e.g., "World History": region-implici() In Figure we observe that for all languages that have History exams with both cultural and region- iimiicii labels (with the exception of Telugu), the model
*we denote intentionally to mean that this language was reported to be in the pretraining corpus the model. 'We note "World History" as region-implici hecause the manner in which the subject is taught and evaluated may vary between regions, even if the subject material seems like it should be universal.
8

---

Published as a conference paper at ICLR 2025
Regional History
Global History
84.5
90.0
89.3
89.2
93.1
58.1
62.5
61.4
Accuracy
26.1
30.7
20
Armenian
Azerbaijani
Belarusian
Chinese
French
Kazakh
Persian
Tegulu
Russian
Figure 4: GPT-4o performance (In-language Prompt) on regional history exams (cultural) and global history exams from that region begion-implicit) based a total of 11,148 questions from INCLUDE. In each language (except Telugu), models perform better on the global history exam.
performs better on the general history exams, indicating a lack of cultural knowledge necessary to answer questions for more region region-specific topics.
Overall, the variance in performance among different regional categories in our results suggests that model performance on INCLUDE may not be rooted in across-the-boan language comprehension issues, but instead in grasping specialized regional knowledge for different languages.
5.4 CHALLENGES MULTILINGUAL EVALUATION
In our experiments, observed that models did not always follow the exact format primed by the 5-shot examples or zero-shot instructions (Format Errors in Table 3), which required generating longer output length rectify.
To empirically measure the impact of this seemingly minute evaluation design choice, we assess the five-shot performance of GPT-4o on INCLUDE- BEASE across various decoded output lengths, focusing specifically on its ability to generate a correct response within the first & follow-up tokens (k 50, 100, 200, and 512). As in our main results, we use the 5-shot prompt template from Hendrycks et al. 2020). without explicitly instructing the model how to generate correct answers. Instead, the model must induce the format from the provided demonstrations.
Table 12 presents the performance of GPT-4o across the 44 languages of the benchmark, evaluated under four different generation window settings. On average, the model shows a 3. 3.1% performance improvement when increasing the generation length window from 50 to 512 tokens. However, this effect not uniform; some languages experience significant improvements, such as Uzbek (+17.2%), Armenian (+13.1%) 1)), and Malayalam (+12.9%). Many others remain largely unaffected. A manual review and analysis of the generated outputs in languages with the largest gains reveal that the model often generates verbose responses, explaining the context before providing the final answer (i.e., ignoring the formatting in the demonstrations, but reaching the correct response). One possible explanation for these discrepancies is the model's limited ability to leverage in-context learning effectively in certain languages, potentially due to imbalances in language resources during the alignment phase (Zhang al. 2024c).
When models are prompted with instructions in English, we observe a modest performance im- provement of 1..5% across all models. Interestingly, in specific cases (e.g., experiments with the Aya-expanse family models, 1wen2.5-14B, and Llama-3. 1 Instr-ntttttttO 70B using CoT prompting), we observe significant changes in the frequency of format errors (Table 3). However, this change in frequency does not appear to impact the performance of the models regional knowledge un- derstanding (Answer Acc., Table 3). suggesting that the choice of instruction language may not significantly enhance impair the model's ability to reflect regional knowledge, but rather primarily influence the format and consistency of the outputs in certain models.
Overall, these results demonstrate that standardizing evaluation is a challenge in tasks that may lead to different output patterns (Nayab et al. 2024), which is compounded multilingual evaluations. In particular, given the incentive lower generation lengths test time (to lower inference or API costs), reliable multilingual assessment requires anticipating how models will produce outputs in different languages, and how evaluation settings might inadvertently affect measures of performance.

---

Published as a conference paper at ICLR 2025
Specifically, practitioners should reflect penalizing models for format enrors when measuring knowledge and intentionally probe for them, given they may not read the languages being evaluated.
RELATED WORK
In recent years, the creation of benchmarks has substantially improved the evaluation of LLMs.
Pioneering efforts such GLUE and SuperGLUE (Wang et al.) 2018 2019) were important in advancing language understanding tasks. Recent benchmarks, such as MMLU (Hendrycks et al. 2020), HellaSwag (Zellers et al. 2019), ARC (Clark al.. 2018), GSM8K (Cobbe et al. 2021). and BigBench (Srivastava et al. 2022). , evaluate models for more complex knowledge comprehension and reasoning. In addition to being final evaluations, they are often used to monitor LLM performance during pretraining, rather than more traditional measures such as perplexity (Penedo et al. 2024).
However, these benchmarks use only English data, limiting their utility for multilingual LLMs.
Evaluating multilingual models requires benchmarks that assess models for these same complex abilities across diverse languages. However, multilingual benchmarks initially focused on more basic linguistic abilities (Conneau et al., (2018; Ponti et al., (020)) and collections of such tasks (Liang et al., 2020, Hu et al. 2020; Ruder et al. 2021] Asai et al.] 2023 Ahuja et al.) 2023a b). Furthermore, these benchmarks generally include only few high-resource languages or are based on translations from high-resource languages, limiting the assessment of regional knowledge comprehension and reasoning. Finally, similar to English evaluations, multilingual benchmarks have trended toward saturation (Zhang et al. 2024a; Wang et al. 2024b). Although there have been efforts to create language-specific MMLU- datasets, coverage remains limited to few languages (Li et al. 2023 Koto et al. 2024 Ghahroodi et al. 2024). Most similar to our proposed effort, the Exams dataset (Hardalov et al. 2020) encompasses questions covering 16 languages collected from elementary and high school science curricula. The Aya dataset (Singh et al. 2024) also includes a sustantial release, covering 513 million data points across 101 languages, including in-language evaluation sets developed by native speakers assessing general performance and safety. However, the Aya dataset is not focused on collecting in-language exams. Our work develops a multilingual benchmark encompassing 44 languages, integrating questions from academic and professional examinations and broadening the evaluation spectrum of multilingual LLMs to include sppcciie knowledge.
Finally, a rich body of work has developed benchmarks to assess LLMs for cultural understand- ing. AAror et al.] (2024) evaluate various aspects of culture and language using questions from community forums on 15 topics. Aakanksha et al. (2024) curate safety dataset that encompasses local nuances. Myung et al. (2024) compile questions about food, sports, holidays, education, and family translated into multiple languages. Synthetic benchmarks, such as NormAd (Rao et al. 2024).
generate cu aulturally-rooterr stories to measure how well models grasp societal norms. Tools such as CultureBank source cultural descriptions from online platforms such as Tok (Shi al. 2024).
offering alternative ways to ground cultural benchmarks in dynamic, real-world knowledge. Exxaniz al. (2024a) analyze the LLM understanding of regional and non-regional knowledge using parallel evaluation dataset in English and Basque. Beyond benchmarking, Chiu et al.] (2024) proposed a tool that facilitates human human-maching -machin collaboration for co-creation of complex datasets, challenging the multicultural understanding and adaptability of LLMs. In contrast to this line of work, our study goes beyond culture as a dimension of regional knowledge, and also assesses LLMs on questions that reflect region-related factual knowledge (e.g., professional standards, law, clinical guidelines).
7 CONCLUSION
We release INCLUDE, a comprehensive multilingual evaluation suite designed to assess performance of large language models (LLMs)) across a wide range of subjects and languages for a rich array of cultural and regional knowledge. INCLUDE contains 197,243 MCQA pairs from 1,926 examinations across 44 languages and 15 scripts collected from 52 countries. Overall, our results from evaluating 15 models on INCLUDE indicate there remains considerable room for model improvement in multilingual regional knowledge understanding and that regional knowledge understanding varies significantly across languages. INCLUDE offers researchers and developers novel and valuable benchmark for evaluating and improving the regional understanding abilities of future multilingual models in the language environments where they would be used.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
The primary goal of our benchmark is to reduce disparities in regional knowledge understanding across languages, addressing the inequities in access technology and its benefits that often result from these gaps. We have designed the benchmark to reflect a diverse range of linguistic, cultural, and regional contexts, sourcing data from local and region-speciffe exam materials. Throughout the data collection process, we ensured that no private or sensitive information was included. We only collected data from exams for which there were no license issues. Our benchmark aims to capture and integrate essential cultural knowledge across many languages. We emphasize the importance of local engagement and encourage developers using this benchmark for the evaluation of monolingual models to actively consult with local stakeholders. To promote equitable access to technology and the development multilingual large language models, we release our benchmark to the community.
Linitations. In line with responsible research practices, we acknowledge several limitations in work. First, INCLUDE spans 44 languages with varying levels of resource availability, leading to different distributions of questions from various academic disciplines across languages. This disparity complicates direct comparisons between performance in disciplines across languages. Additionally, the difficulty of exams may vary not only between languages but also within the same language if exams originate from different sources. However, this limitation is also a reflection of one of the strengths of our benchmark. Questions are sourced from local examinations that reflect the regional and cultural nuances of the environments in which those exams are implemented, which was our motivation for a new evaluation benchmark. Naturally, this precludes exact correspondence between questions across languages. Another practical limitation is that regional knowledge labels were annotated at the exam topic level, rather than at the individual question level, so questions classified based their overarching topic, rather than individual content.
REPRODUCIBILITY STATEMENT
We release subsets, INCLUDE-BASH and INCLUDE-LITE alongside the associated documen- tation and code for data processing and evaluation. These resources will be made publicly available upon acceptance. To mitigate the risk of data contamination during fine-tuning, the full INCLUDE benchmark will be released in incremental stages.
Further details regarding experimental settings, including resource utilization, hyperparameters, and baseline configurations, can be found in Section 4and and Appendix A.5 which provide a comprehensive overview of our methodology.
ACKNOWLEDGEMENT
We thank Deniz Bayazit, Badr Alkhamissi, Mete Ismayilzada and Reza Banaei for reading and providing comments on drafts of this paper. We also gratefully acknowledge the support of the Swiss National Science Foundation (No. 215390), Innosuisse (PFFS-21-29). the EPFL Center for Imaging, Sony Group Corporation. and the Allen Institute for AI. This work supported part of the Swiss AI Initiative by a grant from the Swiss National Supercomputing Centre (CSCS) under project ID a06 Alps.
REFERENCES
Aakanksha, Arash Ahmadian, Beyza Ermis, Seraphina Goldfarb-Tarrann Julia Kreutzer, Marzieh Fadace, and Sara Hooker. The multilingual aligmment prism: Aligning global and local preferences to reduce harm, 2024. URL htthstps:: //tt:/aaaavvvv/.a///aaeeeeeaiava org/ abs 406.00600 18682
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.087774 2023.

---

Published as a conference paper at ICLR 2025
David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba O. Alabi, Xuanli He. Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chiamaka Chukwuneke, Happy Buzaaba. Blessing Sibanda, Godson Kalipe, Jonathan Mukiibi, Salomon Kabongo, Foutse Yuehgoh, Mmasibidi Setaka, Lolwethu Ndolela, Nkiruka Odu, Rooweither Mabuya, Shamsuddeen Hassan Muhammad, Salomey Osei, Sokhar Samb, Tadesse Kehede Guge, and Pontus Stenetorp. Irokobench: A new benchmark for african languages in the age of large language models, 2024. URL https: htttpp:://rxxivy
2406 03368
Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Maxamed Axmed, al. Mega: Multilingual evaluation of generative ai. arXiv preprint arXiv:2303. /252228.3..2288, 2023a.
Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Maxamed Axmed, Kalika Bali, et al. Megaverse: Bench- marking large language models across languages, modalities, models and tasks. arXiv preprint arXiv:2311.07463, 2023b.
Badr AlKhamissi, Muhammad EINokrashy, Mai AlKhamissi, and Mona Diab. Investigating cultural alignment of large language models. arXiv preprint arXiv: 132311 2024.
Shane Arora, Marzena Karpinska, Hung-Ting Chen, Ipsita Bhattacharjee, Mohit lyyer, and Eunsol Choi. Calmqa: Exploring culturally specific long- form question answering across 23 languages, 2024. URL httttt/s://arxx www.vio.rraabbb... 2406 17761
Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, Kelly Marchisio, Max Bartolo, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Aidan Gomez, Phil Blunsom, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. Aya 23: Open weight releases to further multilingual progress. 2024.
http:///wwwwwwwww/a.ee/eoe abs abs/a//5//25 2405 15032
Akari Asai, Sneha Kudugunta, Xinyan Velocity Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannanch Hajishirzi. Buffet: Benchmarking large language models for few-shot cross-lingual transfer. arXiv preprint arXiv:2305.1448 2023.
Edmond Awad, Sohan Dsouza, Azim Shariff. lyad Rahwan, and Jean-Françeu Bonnefon. Universals and variations in moral decisions made in 42 countries by 70,000 participants. Proceedings of the National Academy of Sciences, 17(55)2332-225 2337, 2020.
Yuri Bizzoni, Tom S Juzck, Cristina España- Bonet, Koel Dutta Chowdhury, Josef van Genabith, and Elke Teich. How human is machine translationese? comparing human and machine translations of text and speech. In Marcello Federico, Alex Waibel, Kevin Knight, Satoshi Nakamura, Her- mann Ney, Jan Niehues, Sebastian Stüker, Dekai Wu, Joseph Mariani, and Francois Yvon (eds.), Proceedings of the 17th International Conference on Spoken Language Translation, pp. 280-290, Online, July 2020. Association for Computational Linguistics. doi: 10. 11855//112020.ivvl/ 1.34. URL https: aa/aacaaaaa lathholog org/ 2020 iwslt-l...
Terra Blevins and Luke Zettlemoyer. Language contamination helps explains the cross-lingual capabilities of English pretrained models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing. pp. 3563-3574, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. (/55////2022..mllppm. 233. URL https: ptt.ttpss//aclaantohlloo www.acocnnnnoooo..oooooooooooooe org 2022.emnlp-man min... 233
Beatriz Borges, Negar Foroutan, Deniz Bayazit, Anna Sotnikova, Syrielle Montariol, Tanya Nazaret- zky, Mohammadreza Banaei, Alireza Sakhaeirad, Philippe Servant, Seyed Parsa Neshaei, Jibril Frej. Angelika Romanou, Gail Weiss, Sepideh Mamooler, Zeming Chen, Simin Fan, Silin Gao, Mete Ismayilzada, Debjit Paul, Philippe Schwaller, Sacha Friedli, Patrick Jermann, Tanja Käser, Antoine Bosselut, EPFL Grader Consortium, and EPFL Data Consortium. Could chatgpt get an engineering degree? evaluating higher education vulnerability to ai assistants. Proceedings of the National Academy of Sciences, 121(499):24444555211 2024. doi: 10. 10.11733ppas.2444555121 241955111.. URL
www. org htt.as///wwonggooooooooo//////oa
12

---

Published as a conference paper at ICLR 2025
Samuel Cahyawijaya, Holy Lovenia, Alham Fikri Aji. Genta Indra Winata, Bryan Wilie, Rahmad Mahendra, Christian Wibisono, Ade Romadhony, Karissa Vincentio, Fajri Koto, et al. Nusacrowd: Open source initiative for indonesian nip resources. arXiv preprint arXiv:2212. 09648, 2022.
José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, and Jorge Pérez. Spanish pre-trained bert model and evaluation data. In PMLADC at ICLR 2020, 2020.
Yu Ying Chiu, Liwei Jiang, Maria Antoniak, Chan Young Park, Shuyue Stella Li, Mehar Bhatia, Sahithya Ravi, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi. Culturalteaming: Ai-assisted interactive red-teaming for challenging Ilms' (lack of) multicultural knowledge, 2024. URL
http:///axxvv.ogg///o abs wttp://///////....////// 26644
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprini arXiv: 0188803.05577 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems, 2021. URL https://arxiv. org/abs/2/ /0. /4/68, 2021.
Together Computer. Redpajama: an open dataset for training large language models, October 2023.
URL http:hhtpt/:tthtie wtt.c///oooeeeeeermoottetee/////
Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. arXiv preprint arXiv:1809.05000 2018.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm an, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics. pp. 8440-8457 8451, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/2020. acl- ael-main.7747 URL https: htt::://wwwww.. acl web. org antho http:hhhooooy222..aaee
747.
Xuan-Quy Dao, Ngoc-Bich Le, The-Duy Vo, Xuan-Dung Phan, Bac-Bien Ngo, Van-Tien Nguyen, Thi-My-Thanh Nguyen, and Hong-Phuoe Nguyen. Vnhsge: Vietnamese high school graduation examination dataset for large language models. arXiv preprint 12X11:23005.2219, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Julen Etxaniz, Gorka Azkune, Aitor Sorva, Oier Lopez de Lacalle, and Mikel Artetxe. Bertaqa: How much do language models know about local culture? arXiv preprint arXiv:2406.073302 2024a.
Julen Etxaniz, Oscar Sainz, Naiara Perez, Itziar Aldabe, German Rigau, Eneko Agirre, Aitor Ormaz- abal, Mikel Artetxe, and Aitor Soroa. Latxa: An open language model and evaluation suite for basque. arXiv preprint arXivv2403. 20266, 2024b.
Negar Foroutan, Mohammadreza Banaei, Karl Aberer, and Antoine Bosselut. Breaking the language barrier: Improving cross- looss-lingual reasoning with structured self-attention. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP Singapore, December 2023. Association for Computational Linguistics.
2023, pp. doi: 10.18653/v 1tt5lv112222 1/2023.2oo findings-emnll.655 Mindings-emnile 632
URL https::/aclatthooll/.
2023
Simon Frieder, Luca Pinchetti, Alexis Chevalier, Ryan- Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, and Julius Berner. Mathematical capabilities of chatgpt, 2023.
wwg.abss.wwrgaa//a////aaaabbbass 22011 13867
Leo Gao. Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang. and Andy Zou. A framework for few- language model evaluation, 07 2024. Utt.https://eent wtt.:////reocooo////0/60000
13

---

Published as a conference paper at ICLR 2025
Omid Ghahroodi, Marzia Nouri, Mohammad Vali Sanian, Alireza Sahebi, Doratossadat Dast- gheib, Ehsaneddin Asgari, Mahdich Soleymani Baghshah, and Mohammad Hossein Rohban. Khayyam challenge (persianmmlu): Is your Ilm truly wise the persian language? arXiv preprint arXiv:2404.06644 2024.
Adriana Sitvvar-Ruuozz Isin Demirsahin, Fei He, Shan- Hui Cathy Chu. Supheakmungkol Sarin, Knot Pipatsrisawat, Alexander Gutkin, Alena Butryna, and Oddur Kjartansson. Crowdsourcing latin american spanish for low-resource -xt-tto-peech. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 6504 6513, 2020.
Momchil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, and Preslav Nakov, Exams: A multi-subject high school examinations dataset for cross-lingual and multilingual question answering. arXiv preprint arXiv:201 1,030000 2020.
Kai Hartung, Aaricia Herygers, Shubham Vijay Kurlekar, Khabbab Zakaria, Taylan Volkan, Sören Gröttrup, and Munir Georges. Measuring sentiment bias in machine translation. In International Conference on Text, Speech, and Dialogue, pp. 82-93. Springer, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009. 03300, 2020.
Carolin Holtermann, Paul Röttger, Timm Dill, and Anne Lauscher. Evaluating the elementary multilingual capabilities of large language models with multiq. arXiv preprint 44300:2403.03814, 2024.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pp. 4414441144 4411-4421. PMLR, 2020.
Meng Ji, Pierrette Bouillon, and Mark Seligman. Translation Technology in Accessible Health Communication. Cambridge University Press, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the nlp world. arXiv preprint arXiv:2004.09095 09095, 2020.
Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schuetze. GlotLID: Language identification for low-resource languages. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. gapore, December 2023. Association for Computational Linguistics. doi: 10. 10.65//11/2223. findings-emnlp4110 URL. https::/ //ac antho ogy www.g///........... f nd iesiingseeee emnn
Sin-
410
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Newral Information Processing Systems, 2022. URL
https: htppp//pp..p. www.:seetatttscccccccee arr www.o/arrrrrrrooroeeeeeeeeeeeeee Corg//orpps1:::2200 :4901717744
Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, AbdeIrahman Boda Sadallah, Aisha Alracesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, et al. Arabiemmlu: Assessing massive multitask language understanding in arabic. arXiv preprint arXiv:2402. 12840, 2024.
Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. Okapi: Instruction- large language models in multiple languages with reinforcement learning from human feedback. In Yansong Feng and Els Lefever (eds.). Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 318-327, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/
URL https: / anthol wwaechhooolyy...oo 2023. emn 1wwplppddeme 28
14

---

Published as a conference paper at ICLR 2025
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint 2rX10:2300600221, 2023.
Yaobo Liang. Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, et al. Xglue: A new benchmark dataset for cross-lingual pre-training, understanding and generation. arXiv preprint vrXiv:2004.01400 2020.
Niklas Muennighoff. Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:22 2022.
Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.). Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 448 448-462, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.186533vv1l 00233/111200 //02 w.w.cll...a.lllaemmm..a..a 2021. naacl-manc.......mmmm
URL https: S.Etps://aclanthoologg. org/
Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, et al. Blend: A benehmark for Ilms on everyday knowledge in diverse cultures and languages. arXiv preprint arXiv:2406.0900 099449988 2024.
Sania Nayab, Giulio Rossolini, Giorgio Buttazzo, Nicolamaria Manes, and Fabrizio Giacomelli, Con- cise thoughts: Impact of output length on I1m reasoning and cost. arXiv preprint arXiv:2407. 19825, 2024.
Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406. 17577, 2024.
Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. Lifting the curse of multilinguality by pre-training modular transformers. In Marine Carpuat, Marie- Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, pp. 3479-3495, Seattle, United States, July 2022, Association for Computational Linguistics. doi: 10. (0.188655 102118663vvl// http:///.2a2n......ae 2022. www.:aancl-aaam mai.mmain... 255
255. URL https: ac wew.clnnthoooyyy.. org/
Irene Plaza, Nina Melero, Cristina del Pozo, Javier Conde, Pedro Reviriego, Marina Mayor- and María Grandury. Spanish and Ilm benchmarks: is mmlu lost in translation? arXiv preprint arXiv:2406. 17789, 2024.
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Ko- rhonen. Xcopa: A multilingual dataset for causal commonsense reasoning. arXiv preprint 0030:3 2020.
Luiza Pozzobon, Patrick Lewis, Sara Hooker, and Beyza Ermis. From one many: Expanding the scope of toxicity mitigation in language models, 2024. URL httlltttt//axiiiiiie org/ abs/
2403. 038913
Aida Ramezani and Yang Xu. Knowledge of cultural moral norms in large language models. arXiv preprint NXXv:2306600857: 2023.
Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. Normad: A benchmark for measuring the cultural adaptability of large language models, 2024. URL
tattps://arxide
/brg/bb/// 2404. 12464
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, et al. Xtreme-r: Towards more challenging and nuanced 2021.
multilingual evaluation. arXiv preprint
15

---

Published as a conference paper at ICLR 2025
Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi. Gender bias in machine translation. Transactions of the Association for Computational Linguistics, 97845-874, 2021.
Lütfi Kerem Senel, Benedikt Ebing, Konul Baghirova, Hinrich Schuetze, and Goran Glavaš. Kardeş- NLU: Transfer to low-resource languages with the help of a high-resource cousin - a benchmark and evaluation for Turkic languages. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1672-1688, St. Julian's, s, Malta, March 2024, Association for Computational Linguistics. URL htt.th:tt//eet.. antho wow.hooooooo..ooooo ogy. org/ eawpg//022.a.aalll....cm long. 100.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. arXiv preprint arXiv:2310. 16789, 2023.
Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Chunhua yu, Raya Horesh, Rogério Abreu de Paula, and Diyi Yang. Culturebank: An online community-driven knowledge base towards cul- turally aware language technologies, 2024. URL https: http://aaxvvvv.. www.wawwww..aa..aaaaooaooooooett
15238
Shivalika Singh, Freddie Vargus, Daniel souza, Börje Karlsson, Abinaya Mahendiran, Wei-Yin Yin Ko, Herumb Shandilya. Jay Patel, Deividas Mataciunas, Laura O'Mahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Moura, Dominik Krzeminíski, Hakimeh Fadaei, Irem Ergun, Ifcoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Chien, Sebastian Ruder, Surya Guthikonda, Emad Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, and Sara Hooker. Aya dataset: An open-access collection for multilingual instruction tuning. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1 11521-111 1567, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 17018863/1172222.cclo URL https:://ccacccc http//claanntooloc www.hhooogyooooo/////oo 2024 2ol220011112200200011 acl acl-oon..looo......
Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To cot not cot? chain- of-thought helps mainly on math and symbolic reasoning. arXiv preprint arXh::2409.12188 2024.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alome Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022.
Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizilcee. Cultural bias and cultural alignment of large language models. PNAS nexus, 3()):pgae346, 2024.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based gemini research and technology. arXiv preprint arXiv:2403.08295, 2024.
Silma Team. Silma. 2024. URL https:://www s 1ma...m.....................11 ai
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint 2xxix:2302.13977 2023.
Ahmet Üstün, Viraat Aryabumi, Zheng Yong. Wei-Yin Ko, Daniel D' souza, Ghemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzich Fadace, Julia Kreutzer, and Sara Hooker. Aya model: An instruction finetuned open-access multilingual language model. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computationna Linguistics (Volume 1: Long Papers), pp. 15894, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1/..8666///122222..alllllooo http//alllnaooolooooooo/////////
URL https: / antho ogy. ogy.orgg/22222 2024 .acl- long. 845
16

---

Published as a conference paper at ICLR 2025
Eva Vanmassenhove, Dimitar Shterionov, and Matthew Gwilliam. Machine translationese: Effects of algorithmic bias on linguistic complexity in machine translation. arXiv preprint arXiv:2102.00287, 2021.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1rXiv:1804.07611, 2018.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems, Advances in neural information processing systems, 32, 2019.
Xidong Wang, Nuo Chen, Junying Chen, Yan Hu, Yidong Wang, Xiangbo Wu, Anningzhe Gao, Xiang Wan, Haizhou Li, and Benyou Wang. Apollo: An lightweight multilingual medical Ilm towards democratizing medical ai to 6b people. ArXiv, (oss/2403.03340 2024a. URL https org http://ooruuIDD26625333333
7/api semaniseennticccccco
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint urXiv:2406.01574 2024b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Neural Information Processing Systems, 2022. URL hhttps://apl.
Oxt//GorpusDD:24441655
Orgest Xhelili, Yihong Liu, and Hinrich Schütze. Breaking the script barrier in multilingual pre-trained language models with transliteration-based post-training alignment. arXiv preprint arXiv:2406. 19759, 2024.
Xue. mt:: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11944, 2020.
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Ly, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint
2rXIv:2309.I0305. 2023.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint 1xXiv2247710671 2024.
Arda Yüksel, Abdullatif Köksal, Lütfi Kerem Şenel, Anna Korhonen, and Hinrich Schütze. Turk- ishmmlu: Measuring massive multitask language understanding in turkish. arXiv preprint arXiv:2407. 12402, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07832 2019.
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, et al. A careful examination of large language model performance on grade school arithmetic. arXiv preprint urXiv:2405.0080 003322 2024a.
Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li. Min-k%++: Improved baseline for detecting pre-training data from large language models. arXiv preprint 2xXiv:2404.02936 2024b.
Miaoran Zhang, Vagrant Gautam, Mingyang Wang, Jesujoba O Alabi, Xiaoyu Shen, Dietrich Klakow, and Marius Mosbach. The impact of demonstrations on multilingual in-context learning: A multidimensional analysis. arXiv preprint 12776 2024c.
17

---

Published as a conference paper at ICLR 2025
Korean
Persian
Armenian
100
100
100
47
80
57
30
Acourscy
20
20
Proftssional
Economics
Profectiinal ontiltation
Litratore
Gengraphy
Socasloy
Dening Deving Utenss
Chamatry
Hatoy
Gropaeit
Dinlig Vermet
contiication
Hindi
Greek
Russian
100
103
57
55
50
30
36
63
60
41
Acnuncy
60
Accunscy (o
60
40
40
40
Dronng Aconse
Peleestnnal
Lnworter
Gengnaly
Edscorted
Iittry
Meddrse
Medical Lamner
Pelersianal
Eomenicc
Maragementt
Visull Art
Mastoel Licnns
Ecoumic
Manageneed
Law
Medtran
Qualinitty
Educsteo
Figure 5: GPT-40 performance across academic disciplines for Korean, Persian, Armenian, Hindi, Greek, and Russian. Each bar is annotated with the number of questions with correct answers.
A APPENDIX
100
100
80
80.5
80
70.5
78.7
85.7
Accurscy (86)
61.7
60
60
48,9
20
Culture
Agnustic
Explicit
Implict
Math
Chemistry
Physics
Ingineering Qualimetry
Biology
CS
Earth Science
RERIONN FABE
(b)
Figure 6: GPT-4o model performance on INCLUDE-BASE. (a) Performance across regional labels. While models typically perform better across region-explicit and regional-implicit questions, is difficult to disentangle the difficult of questions due to regionality from the subject matter itself (i.e., region-gnnann aagiin-agnootic questions may contain more STEM subjects that are traditionally harder for I.LMs). (b) Performance across academic disciplines within STEM area. We observe models perform particularly poorly on Math and Chemistry questions.
A.1 COLLECTED LANGUAGES
Table provides information about the languages in INCLUDE.
18

---

Published as a conference paper at ICLR 2025
Accuracy (1)
Model
Humanities
STEM
Domain-Specifie
Professional
Licenses
# samples
13294
2478
1964
3165
1736
GPT-4o
5-shoo
79.0
74.2
76.8
Zero-shot CoT
79.9
78.6
80.4
73.8
81.1
Llama-3.1-70B-nnste
5-shot
74.2
73.7
Zero-shot CoT
61.9
57.5
63.5
58.4
Aya-expanse-32B
5-shot
43.0
49.1
49.5
Zero-shot CoT
$2.9
47.8
55.4
44.3
52.9
Qwen2.5-14B
5-shot
66.0
65.1
Zero-shot CoT
48.6
44.4
51.6
41.6
46.9
Aya-expanse-8B
37.8
32.3
37.3
40.2
29.7
Mistral-7B (v0.3)
44.2
43.4
43.9
38.6
44.3
Mistral-7B-Instrucl (v0.3)
44.5
42.7
43.2
40.1
43.7
Gemma-7B
55.1
53.6
55.5
47.7
62.2
Genma-7B-Instrut
38.6
37.7
42.0
34.5
44.9
Qwen2.5-7B
53.4
54.2
59.1
51.3
57.8
Owwnn2.5-78-Instruct
53.5
53.3
58.1
49.5
58.6
Llmma-3-8B
51.7
49.8
52.1
43.4
51.3
Llama-3-8BInstteee
50.7
46.9
52.9
54.4
Table 2: Accuracy performance of GPT-4o (In-language prompting) on INCLUDE-BASE grouped by high-level topics. Humanities include Social Science, Humanities, and General knowledge. STEM includes Applied Science and STEM. Domain-specific covers Business & Commerce and Health oriented education. Professional includes professional certifications. Licenses cover Marine, Fishing, and Driving licenses.
In-language Prompt
English Prompt
Total Acc.
Answer Acc.
Format Errors
Total Acc.
Answer Acc.
Format Errors
Model
GPT-4o
5-ssho
77.3
79.0
2.5
76.3
Zero-shot CoT
79.0
79.2
0.2
78.9
79.1
Lama-3.1-708-Instrmcc
5-shot
70.6
70.6
0.0
70.7
Zero-shot CoT
60.6
67.9
10.9
56.3
67.8
17.0
Aya-expanse-32B
5-shot
52.4
56.2
16.9
56.6
62.7
9.7
Zero-shot CoT
51.4
57.2
10.2
$7.7
58.4
1.1
Qwen2.5-14B
5-shot
61.4
62,4
1.5
61.7
61.7
0.0
Zero-shot CoT
47.3
53.1
10.9
$1.0
52.0
1.9
Aya-expanse-8B
37.2
43.8
18.0
46.0
50.7
9.2
Mistral-7B (v0.3)
43.3
43.3
0.0
44.9
44.9
0.0
Mistral-7B-Instrune (v0.3)
43.6
43.8
0.4
44.5
44.5
0.1
Gemma-7B
54.5
54.5
0.0
54.9
54.9
0.0
-a-mma-7B-Instuel
38.7
38.7
0.0
39.7
39.7
0.1
Qwen2.5-7B
54.1
55.1
1.9
$5.2
55.2
0.0
Qwen2.5-7B-Inen
53.8
54.0
0.5
54.6
54.6
0.0
Llama-3.1-8B
51.0
51.0
0.0
51.8
51.8
0.0
1-amm-3.-8BBIInstuuc
53.4
53.4
0.0
54.6
54.6
0.0
Table 3: Results on INCLUDE-BANE for In-language and English prompting strategies. Total Accuracy represents the raw accuracy of the model for answering INCLUDE questions in each respective subset. Answer Accuracy represents the accuracy of the model when only considering samples where an answer extracted from the model's output the correct response format. Formatting Errors (%) describes the percentage of model responses that are not formatted correctly and so do not output any answer option. We mark these incorrect by default in Total Accuracy and do not include them when computing Answer Accuracy.
19

---

Published as a conference paper at ICLR 2025
INCLUDE-LITE
INCLUDE-BASE
Model
In-Languagee Prompt
English Prompt
In-languuage Prompt
English Prompt
1amma:1-70B-Instrree
70.3
70.6
70.6
70.9
Aya-expanse-32B
58,9
59.5
47.2
47.8
Qwen2.5-14B
61.8
61.9
62.3
62.6
Aya-expanse-8B
47.3
48.0
47.2
47.8
Mistral-7B
44.5
44.7
44.1
44,6
Mistral--B-IIstruuet
43.8
43.9
44.2
44.3
Gemma-7B
53.6
53.1
53.5
53.2
Demma-77-Inntrrun
39.
39.7
38.6
39.3
Qwen2.5-7B
54,4
54.9
55.0
55.5
Qwen2.5-7B-Instiee
54.5
54.6
54.8
54.8
Allam--3.1-80
52.1
51.2
51.9
54.4
53.5
54.4
Table 4: Harness evaluation results on INCLUDE-BASE.
Academic area
Academic field
Label
Logic
Agnostic Region Explicit
Law
Humanities
Language
Culture
Visual Arts, History, Philosophy, Religious studies, Performing arts, Culturology, Liter-
Region implicit/ Culture
ature
Sociology, Political sciences, Anthropology
Region implicit/Culture
Economics
Region implicit/ AaplicttAAgnnsti/RRegooo explicit Region implici//Reiion explicit Region implicit/Agnostic
Social Science
Psychology
Geography
Math, Physics, CS, Biology, Earth science,
Agnostic
STEM
Chemistry, Engineering
Qualimetry
Region explicit
Medicine Health
Agnostic/Region implicit/Region explicit Region implicit/Region explicit
Health oriented education
Accounting
Region explicit
Management, Marketing, Industrial and la- hor relations. International trade, Risk man- agement and insurance, Business adminis- tration, Business ethics, Business, Finance
Business and Commerce
Region implicit/Region explicit/Agnostic
Agriculture, Library and museum studies, Transportation
Region implicitt/Anoosiic
Military Sciences, Public Administration, Public Policy
Region implicit/Region explicit
Applied Science
Architecture and Design, Family and con- sumer science, Environmental studies and forestry, Education Journalism, media stud- ies, and communication, Social Work, Hu- man physical performance and recreation
Region implicit
Driving license, Marine license, Fishing li- cense. Medical license, Public administra- tion, Professional certification
Other
Region explicit
General knowledge
Multiple
Region implicit/Culture
Table Annotation schema for high-level Academic area and fine- ane-ggrined Academic field. The Label column lists the most likely regionality label for these exams in our dataset (e.g., region- {agnostic, implicit, explicit) or cultural), though exams from which we collect data are individually labeled with a regionality category. The first label is the most frequent one.
20

---

Published as a conference paper at ICLR 2025
Academic Domain-specifie studies
Academic Humanities
Academic studies
Language
Professional
License
Avg (%)
Albanian
95.0
88.0
83.5
89.50
Arabic
77.8
82.0
80.5
76.2
78.30
Armenian
52.7
32.0
72.2
53,60
Azerbaijani
71.3
73.6
71.4
71.90
Basque
64.8
64.80
Belarusian
51.8
42.0
50.90
Bengali
71.1
90.0
84.3
76.80
Bulgarian
93.8
60.0
90.70
Chinese
71.5
58.2
52.1
Croatian
89.0
82.0
88.40
Dutch; Flemish
86.6
87.5
80.0
86.40
Estonian
90.7
98.0
100.0
92.40
Finnish
67.0
87.0
77.8
69.90
French
83.8
50.0
81.2
68.1
80,70
Georgian
87.6
87.60
German
62.6
64.0
87.0
66.90
Greek
84.7
84.0
89.2
58.6
71.50
Hebrew
62.0
Hindi
77.7
71.9
91.5
71.8
75.10
Hungarian
66.3
80.6
75.80
Indonesian
84.0
69.1
84.8
79.50
Italian
87.7
87.2
91.7
95.5
90.00
Japanese
78.1
96.0
81.60
Kazakh
80.4
80.40
Korean
91.6
46.4
69.00
Lithuanian
92.0
97.1
82.5
81.2
90.60
Malay
84.5
80.3
Malayalam
69.6
66.0
55.0
Nepali
61.6
83.2
72.40
Macedonian
96.0
86.0
89.3
92.40
Persian
66.0
25.0
49.6
81.6
64,60
Polish
100.0
64.6
80.0
78.80
Portuguese
84.7
63.3
67.9
76,40
Serbian
92.2
86.0
91.60
Spanish
83.6
88.0
96.0
84.40
Tagalog
86.8
90.7
87.40
Tamil
70.6
54.0
69.10
Telugu
66.9
70.7
68.20
Turkish
62.0
52.0
75.9
65.30
Ukrainian
85.8
84.0
85.60
Urdu
61.7
65.3
100.0
62.50
Uzbek
63.6
84.0
73.3
69.70
Vietnamese
84,4
86.0
84,50
Russian
77.5
83.4
70.8
63.9
75.00
Table 6: Accuracy performance of GPT-4o (5-shot) on INCLUDE-BAAN BASE for each language. Hu- manities include Social Science, Humanities, and General knowledge. STEM includes Applied Science and STEM. Domain-specifi covers Business & Commerce and Health oriented education. Professional includes professional certifications. Licenses cover Marine, Fishing, and Driving licenses.
Aya-expanse-8B
XGLM-7B
Qwen-2.5-7B
L.AMA-3.11-B
Full Benchmark
0.02
0.17
0.13
0.29
Newly collected
0.01
0.14
0.11
0.25
Table 7: Data contamination rates per model on INCLUDE-BASE
21

---

Published as a conference paper at ICLR 2025
Language
Script
Family
Branch
Availability
Count
Albanian
latin
Indo-Europent
Albanian
Mid
2365
Amharic
ge
Afro-Asiatic
Semitic
Low
131
Arabic
perso-arabic
Afro-Asiatic
Semitic
High
15137
Armenian
armenian
Indo- Bud--uu
Armenian
Low
1669
Assamese
Lengali-assameee
Indo-Europy European
Indo-Iranian
Low
323
Azerbaijani
latin
Turkic
Azerhaijani North
Mid
6937
Basque
latin cyrillic
Isolate
Low
719
Belarusian
Indo-European
Slavic East
Low
687
Bengali
bengali-asee assamese
Indo-Eure European
Indo-Iranian
Mid
15259
Bulgarian
cyrillic
Indo-Europpe European
Slavic South Eastern
Mid
2937
Chinese
chinese
Chinese
High
12977
Croatian
latin
Pudo-European Indo-
Slavic South Western
Mid
2879
Czech
latin
Indo-Europent
Slavic West
High
50
Danish
latin
Indo-Eine audo-Europcan
Germanic
Mid
732
Dutch; Flemish
latin
Pndo-European
Germanic
High
2222
Estonian
latin
Uralic Uralic
Finnic
Mid
952
Finnish
latin
Finnic
Mid
1574
French
latin
Mado-European
Italic
High
2457
Georgian
mkherduli
Kartvelian
Georgian
Low
599
German
latin
mndo-European
Germanic
High
1590
Greek
greek
Indo-Furopean
Greek
Mid
6570
Hebrew
hebrew
Afro-Asiatic
Semitic
Mid
2457
Hindi
devanagari
aado-Europcan
Indo-Iranian
Mid
5167
Hungarian
latin
Uralic
Hungarian
Mid
2267
Indonesian
latin
Austronesian
High
12013
Italian
latin
Indo-European
Italic
High
3038
Japanese
kanji
Japonic
Japanese
High
2699
Kannada
kannada
Dravidian
Southern
Low
335
Kazakh
cyrillic
Turkic
Western
Low
5736
Korean
hangul
Koreanic
Korean
Mid
1781
Lithuanian
latin
Indo-European
Eastern Baltic
Mid
1397
Malay
Austronesian
Malayo-Polynesian
Mid
Malayalam
vatteluttu
Dravidian
Southern
Low
275
Marathi
devanagari
Indo-Europer
Indo-Iranian
Mid
313
Nepali
devanagari
Indo-Iranian
Mid
1470
Macedonian
cyrillic
Indo-European
Slavic South Eastern
Low
2075
Oriya
odia
Indo-Eure Sudo-Europcan
Indo-Iranian
Low
241
Panjabi; Punjabi
gurmukhi
Indo- audo-Europcan
Indo-Iranian
Low
453
Persian
perso-arabic
Indo-European
Indo-Iranian
High
23990
Polish
latin
Indo-European
Slavic West
High
2023
Portuguese
latin
Indo-European
Italic
High
1407
Russian
cyrillic
Indo-Europen
Slavic East
High
10169
Serbian
cyrillic
Indo European
Slavic South
Mid
1636
Sinhala; Sinhalese
sinhala
Auro-European
Indo-Iranian
Low
325
Slovak
Europen
Slavic West
Mid
131
Spanish
Indo-Furopean
Italic
High
2559
Swedish
latin
Indo-Furopean
Germanic
Mid
5102
Tagalog
latin
Austronesian
Low
530
Tamil
tamil
Dravidian
Southern
Mid
945
Telugu
telugu
Dravidian
South-Central
Low
11568
Turkish
latin
Turkic
Southern
High
2710
Ukrainian
cyrillic
Indo-Eure aado-Europcaa
Slavic East
Mid
1482
Urdu
aarso-
Indo Aado-Europcan
Indo-Iranian
Low
122
Uzbek
latin
Turkic
Eastern
2878
Vietnamese
latin
Austro-Asiatic
Mon-Khmer
High 8901
Table 8: Languages in INCLUDE with their associated metadata and the total count of the samples
per language.
22

---

Published as a conference paper at ICLR 2025
A.2 DETAILS ON PARSING EXAM SOURCES
Figure 8 presents the questionnaire we distributed to the community to gather diverse set of multiple-choice exams. was distributed among university student organizations and researchers at our institution 8 Participation was voluntary and not incentivized.
A.3 PERFORMANCE ACROSS ACADEMIC AREAS AND FIELDS
Distribution of academic areas and academic fields with the respective number of questions is presented in Figure 7 GPT-4o performance across languages and academic areas is in Table 9 GPT-40 performance across languages, academic fields, and related regional features is in Tables and
A.4 REGIONAL LABELS: ANNOTATION
First, we categorized the exams into one of eight broad academic areas, e.g., Humanities or Social Sciences, and then further classified each exam into a specific academic fields, egg., History or Geography. This categorization was done manually, taking into account both the exam's 's learning level and the exam's 's original topic.
Building these categories, applied one of four labels- agnosticc culture-related, region-explicit, or
based on the degree of dependence on localized knowledge required to answer the exam questions. The labels reflect the extent which specific cultural or regional knowledge is necessary. Table53lpm provides examples illustrating how different exams were typically classified under each label, show the relationship between categories and labels.
The "region implicit" label was applied when we suspected that exam content might vary across regions but could not reliably detect specific regional differences. For example, historical events, literary works, and religious interpretations may differ significantly depending on the region. Similarly, fields like marketing, management, social work, and insurance- though rooted in shared theoretical foundations-cal be practiced differently across regions. When we encountered such uncertainty, we labeled the subject as "region implicit."
Within the Humanities, fields such as Visual Arts, History, Philosophy, Religious Studies, Performing Arts, Culturology, and Literature were labeled "region implicit" when the content was not explicitly tied to particular region. However, if the exam was region-specific (e.g., Greek literature), we categorized it as "culture-related""
In the Social Sciences, Psychology was classified as "region explicit" if the exam focused on regional clinical practices; otherwise, it was "region implicit" when dealing with broader psychological theories that may vary across regions. Geography was labeled "region implicit" if the exam involved political geography and "agnostic" it focused on general geographic knowledge. Similarly, disci- plines like Sociology. Political Science, and Anthropology were classified as either "region implicit" or "culture-related; depending on whether regional specificity was required, much like History.
For Economics, exams were labeled "agnostic" when covering general economic theories, "region explicit" when addressing regional regulations, and "region implicit" when regional applications were uncertain. In STEM fields, most disciplines were categorized as "agnostic," with the exception of Qualimetry, which was labeled "region explicit" due to its specific application in post-Soviet countries for quantitative and qualitative assessment according to regional standards (
Exams related to theoretical medical subjects, such as Anatomy, were classified as "agnostic." In contrast, exams covering clinical practices and guidelines specific region were labeled as "region explicit," while others were marked as "region implicit" if regional dependence was unclear.
Accounting is generally tied to region-specific practices, it was consistently classified "region explicit." Other disciplines within the Business and Commerce category were treated similarly to Economics and labeled as mostly as "region implicit." In some cases, there were "agnostic" and "region explicit" exams.
Wee will provide institutional details upon paper publication. https://en wik pedia. aor.ipd...///iki/// imetry
23

---

Published as a conference paper at ICLR 2025
0003. Engineery
6520 Cleemitty
1549 Earth sciennee
Peotoo mantoriing an artra
Calturalog 1788
Vouus Art
348 STEM
238 Quaimmttyy
14,066 Professionall
2034 Driding Lieettte
Logg 2013
Philorophy 3822
211 Mediccll Liserse
Religous studier 8760
Las 4251
2A89 Marine Licanse
992 Fishing license
Language 163
Humantites
othe
8048 Geography
Lntrature .000
7062 Economiccs
History 20 2288
$799 Sociology
Public Policy 36
Social Sclence
1972 Political sciences
Trampprtaaiiona
Hoalth oriented
1395 Phychology
Socol Work
Applted Splenc Sclence
education
40 Aethropology
Milttry Sciencer 196
7533 Medicin
Public Adminisration 230
Genuraal knowledge
701 Health
Desigr 242
2024 Management
Sport 354
Commecoo
4022 Bioines
Ensiromennal sturtin sab
Journakerr
616 sorttcting
Accounting
Eduration1151 1157
254 Risionra ethics
Ammuttoiiii 1190
224 Finarias
Muttills 5232
Labor relatioos 204
Business admintaddtnn tooo
Internationaal traae
Figure Academic domain and academic fields with the number of examples across all languages.
Exams in Applied Science disciplines were typically categorized as "region implicit" due to the potential involvement regional variations. Similarly, exams Military Sciences, Public Adminis- tration, and Public Policy were marked as "region explicit" when tied to specific regions (e,g. Basics of National Security of the Republic Azerbaijan) and "region implicit" when regional specifics were less pronounced. For exams focused on theoretical aspects, used the "agnostic" label (e.g.. Theoretical Foundations of Food Engineering in Agriculture).
Finally, exams covering multiple topics were classified as "region implicit" unless they explicitly focused on cultural aspects a particular region, in which case they were labeled as "culture-related."
A.5 IMPLEMENTATION DETAILS
Each model was evaluated using a single A100 GPU (80GB memory), with evaluation times averaging approximately 4 hours for INCLUDE-BASE BASE.. For all models, set the decoding temperature to prioritizing deterministic outputs.
We configured response context windows based model size and task requirements. For models (v0.3), Gemma-78 7a-ua--B-Insstuct. Llama-3-8B, Mama----B-IInstrmnee XGLM-7.5BB, BL.0OM-.1BB,
such as Aya-23-8B. 8B, Mistral-7B (v0.3). Qwen2.5-7B,
24

---

Published as a conference paper at ICLR 2025
Collection of Multilingual Exams
Thank you for part ciparing this surviy!
We invite you help us build the next ere of muikiingual Mill Large Languege Models (LLMs))
We are cal collecting a pool of miittlngual axams from various languages and cultural Muitilingusl data crucial in serve diverse cultures
inc incussiity and falmess,
enabling systema understand
nitiea ecqually.
this Coogle Form, we ask you to provide a list examinations in your nativo language.
These sets question anower or pairs can fall ander the categories below.
Educational exama: National excmiiatiion of your counttry exorms roleti ve to the level educetior: High school Physics, Collego/Uhieer Universityyyyyyy Jttrature.
Professional 0.
Practical tests: e.g. Driver license test, Marine License Practice Tests.
rr esaminatiio
knowledge
Please provide the owing information ast anaddedd csamination
school)
Drecription descrppti examination
Langringe
URL of the dato The link where the dats own lasie sppllenbl))
Asswers: Deacribe rib how answeers proviced us inside cuestion file separste Six (prow the this five
Aduitio sal information, I Laaataals such
slno munnter ber rowal, short cescitation
Home scae tifiiaa anarn snarpless
Deccelption
Language
the ror titte Zheemu csnedinens meteroeedd teatt grrntt
Ansueer: in separste lie under the same USS
Size 65 acadamic tapks, for the last 20 years
Deccelption:
Language aar
URL hiwaa www.watttoaaaaeo..ooooooooo www.aaalllocmmeeeeeeaeeeoooe canoooe 7ho7aa hottoggl
Answers: At the and of each questio file
Educational examination
Natioonal sal evamni ations your country evams relative to the level education.
Pleaze provide kat esaminatioss assed format above (Ce. neme, description,
Long
Profeesioral eraminatiors
gug. Law Bur couminationn Medial Liow ne exami alkons, etc.
Please provide kot cruminations assed the fovner above (ie. name desorigsion Ainguage uniens
Practisal ecaminations
cs. Dt.Drier loance test Warine Usense Practioe Tests, ets.
Figure 8: Exam source collection form sent to the academic community.
and BLOOMZ-7.1B, set window size of 40 tokens. Larger models, including C4A1-Aya-23-35B and GPT-4, were evaluated using a 512-token context window for 5-shot tasks and 1024 tokens for zero-shot thain-of-thought (CoT) reasoning.
A.6 EXPERIMENTS ON MONOLINGUAL MODELS
To assess the performance of monolingual models on our benchmark, we evaluate seven open-source monolingual models the relevant language-specific subsets of INCLUDE, (i.e., the languages these
25

---

Published as a conference paper at ICLR 2025
1.
Altrnia 12
Tue Aprespare-122
French
Acc original data
Gomme:70 Open 23000
original
144
Acc on publishhe data
Acc on published data
(a)
Figure 9: Accuracy of different models on languages where both existing benchmark data and newly collected data are available. Each point represents the accuracy score of model for specific language. (a) Points of the same color represent the accuracy scores of single model across different languages. (b) Points of the same color represent the accuracy scores for single language across different models.
models were pre-trained on). We test Baichuan-7B (Yang et al. 2024), Calm2-7B-chat (Touvron et al. 2023), Korean-Mistral, ruGPT3.5- 11913.--3BB 13B and Sauerkrautl MM-V2- 14b-DPO. We compare their performance with the results of the most performant large-, medium-, and small-scalc models on the specific language subsets. The results of this evaluation are presented in the Table [3] The table reveals that most monolingual models underperform the state-of-the-an small multilingual model Qwen-2.5 (7B), with the exception of the German monolingual model, SauerkrautL.---214BBDDD
2023). STL (Team)
which performs on par with Qwen in the German language.
A.7 LIMITATIONS OF THE SCOPE OF EXISTING BENCHMARKS
As discussed the motivation and related work sections, there currently no multilingual benchmark that offers both high language coverage and incorporates regional knowledge. Existing benchmarks typically fall into one of two categories: either they focus extensively on a single language across various regional dimensions, or they cover certain number of languages with predominantly region-agnostic knowledge (e.g., STEM). In Table 15) we provide more details on the list of existing benchmarks that were mentioned in the paper, which feature original content (not machine-trans/e translated),, highlighting their language and knowledge coverage.
In relation these monolingual existing resources, the INCLUDE benchmark makes three significant contributions: (1) introduces original datasets from languages that are either not covered or only partially covered by existing benchmarks, (2) leverages publicly available, knowledge-intensive multiple-choice question benchmarks in various languages, (3) organizes both the existing and newly and
introduced data under unified taxonomy of knowledge, differentiating between regional knowledge
knowledge. In this context, INCLUDE incorporates existing datasets, which account for 39.8% the total collected data and 31.7% of the INCLUDE- BASE benchmark. To understand the correlations between the newly collected data and the existing benchmarks integrated into INCI.DDE, we analyzed model performance languages where both existing benchmark data and newly collected data are available. We compared performance across these datasets and examined the correlation between them. Results were stratified by language and by model type. We visualized the performances using plots (Figure 9) and calculated the R² scores to quantify the correlations (Table 14).
The analysis reveals two key conclusions: First, for given language with multiple models having published performance data, model's performance on INCLUDE can generally predicted. How- ever, for given model with published performance across different languages, its performance
26

---

Published as a conference paper at ICLR 2025
INCLUDE cannot reliably be predicted when applied to newly published language benchmark. This indicates that while INCLUDE is less impactful for languages with existing published benchmarks, is particularly valuable for assessing performance in languages with no prior resources.
A.8 ANALYSIS OUTPUT ERRORS ON INCLUDE
As outlined in Section 5.3 of the paper, model performance and consequently, the errors are heavily influenced by the model's proficiency the specific task and language. We conducted more detailed error analysis by manually investigating a sample of INCLUDE generations. We focused on six languages spanning high-resource (Chinese, Turkish), medium-resoure (Bengali, Greek, Korean), and low-res low-resource (Armenian) categories, selecting subject areas with the largest performance gaps compared to other subjects within the same language. We manually examined 150 examples, covering least two subjects per language with 10 examples per subject, analyzing the questions and answers generated by GPT-4o. We observed four main types of errors related to computational mistakes, factual mistakes, lack of regional knowledge, and format errors (Table TT). Each enror type highlights distinct limitations in the model's capabilities, from arithmetic and factual knowledge to regional understanding and prompt adherence. Our analysis revealed that the model's errors distributed as follows: 38.6% were due to lack of regional knowledge, 32% resulted from format errors, 26.7% % were factual mistakes, and 2.7% were computational errors.
A.9 DATA CONTAMINATION PREVENTION
As described in Section 3.1 INCLUDE is made up of few previously-publishee benchmarks incorporated into INCLUDE, but also newly-collected exam materials from sources contributed by multilingual community of native speakers. significant portion of the newly-collected data was derived from PDFs and textbooks, which are less likely to have been included models trained primarily web-based data.
In this section, we analyze the degree of contamination within the models using the mink%+++ (Zhang al., (024b) method for training data detection in LLMs. This method determines whether input next token forms a mode or has a relatively high probability under the conditional categorical distribution. Using this scoring mechanism, one can predict if an input sequence is part of the model's training data based on a decision threshold. This method achieves SOTA on the WikiMIA (Shi al., 2023) benchmark for training data detection. We use the decision threshold that achieves the best performance on WikiMIA as the decision threshold for our analysis. Using this method, we computed the contamination rate for each language on four main-stream multilingual models: Aya-8B, XGLM-7B, LLaMA-3.1 1-BBB, and Qwen-2.5-7B. We show the contamination rate results in Table 7
To further mitigate the risk of benchmark saturation as a result of data leakage when models are trained, we have held back the complete dataset, comprising 197,243 entries. Instead, we will release these further questions and answers incrementally over the next year. We have also reserved held-out dataset covering a wide range of the collected languages to be used for future experimental studies specifically aimed analyzing data leakage time.
27

---

Published as a conference paper at ICLR 2025
Language
Academic Area
Accuracy
Count
Humaniries
95.1
223
Language
Academic Area
Accuracy
Count
Albanian
Business Commerce
85.7
Japanese
Nocial Ncience
94.5
55
Kazakh
Humanities
80.4
500
Humanities
79.0
105
Business Commerce
79.3
Korean
Other
250
Arabic
General Knowledge
Other
86.7
76.2
105
105
Social Scienoe
Humanitiess
91.6
91.6
250
335
STEM
Business Commerce
77.5
40
Social Science
67.6
105
Lithuanian
Humanities
34,7
225
STEM
97.1
34
Armenian
Other STEM
28.0
72.2
Social Science
Humanitics
93.5
178
77
Social Science
50.5
196
Malay
Business Commerce
79.8
178.
Applied Science
75,9
Social Science
84,8
145
Humanitics
74,1
108
Humanities
56
Azerbuijani
Business & Commerce
62.5
96
General Knowlodge
73.1
78
Health- Oriented Education
80.2
96
Malayalam
Saalth--rrented Educaation
55.0
100
Science
67.6
Other
194
Basque
Other
64.8
500
STEM
66.0
47
Belarusian
Humanities
50.8
490
Nepali
Other
72.4
500
STEM
42.0
Humanities
224
Humanities
62.0
166
Macedonian
Business Commerce
89.3
224
Rengali
General Knowledge
84.3
80.1
166
Social Science
STEM
92.5
$6.0
53
50
STEM
88.0
50
Humanilics
55.3
141
Humanities
96.4
250
Persian
Other
62.4
250
Bulgarian
STEM
Social Science
74.5
141
Social Science
91.2
250
Polish
496
Applied Science
73,2
71
STEM
62.5
48
Humanitics
67.8
Applied Science
58.3
84
Chinese
Business & Commerce
53.5
Humanities
154
Health Oriented Education
60.9
87
Portugaese
Business Commerce
56.9
84
Other
68.3
142
Healthh-rieetel Edllcation
67.1
67
Social Science
76.1
169
Humanities
86.8
250
Applied Science
87.0
69
Creatian
STEM
82.0
Humanities
76.8
69
Science
90.8
Business
Dutch; Flemish
Social Science
Humanities
86.0
86.8
243
243
Health oriented education
Other
63.9
74,1
97
85
Estonian
Humanities
STEM
90.1
97.2
36
STEM
Science
94
Humanities
90.4
313
Health. Educattin
Humanities
69.5
226
Serbian
STEM
$4.0
50
Science
187
Social Science
64.6
226
Humanities
77,2
250
Humanities
68.1
86.5
266
Spanish
Health oriented edlacation
96.0
25
Social Science
74,3
74
Social Science
89.6
250
Geoggan
Humanitics
87.6
500
Tagalog
Humanities
86.8
425
Science
Humanities
83.8
37
Tamil
General knowledge
70.6
500
Greek
Business & Commeree
89.1
64
STEM
$4.0
50
Other
57.5
Applicd Science
166
Social Science
84,2
133
Telugu
Humanities
66.0
191
Hehnew
Humanities
60.0
50
Social Science
66.9
166
Humanities
166
Applied Science
Humanities
72.9
83.1
96
Turkish
Business Commerce
STEM
$2.0
75,9
166
50
Hindi
General Knowledge
83.1
Social Science
166
Health-Oriented Education
91.5
Humanities
92.4
250
Other
64.1
142
Ukrainian
STEM
81.0
50
Social Science
74,6
Social Science
79.2
250
Hungarian
Applied Science
Social Science
79.8
66,3
184
Unda
Humanities
STEM
63,3
61.7
300
49
Applied Science
71,2
125
Humanities
240
Humanities
82.4
240
Indonesian
Other
83.2
125
Uzbek
STEM
84.0
50
STEM
Science
60.0
50
Social Science
71.1
21
Humanities
250
Applied Science
85.7
35
Vietnamese
STEM
$6.0
50
Italian
Humanitics
Other
85.0
167
Social Sclence
$0.8
250
Social Science
89.8
167
Table 9: GPT-4o (5-shot, In-language prompting) performance on INCLUDE-BASE per language and academic area. Areas with less than 30 examples were excluded from the analysis.
28

---

Published as a conference paper at ICLR 2025
Language
Academii
Regional
Accurncy
Count
History
Implicit
93.1
58
Phnlosuphy
Implicit
97.6
82
Albanian
Visual Ans
Implicit
94.0
83
Implicit
Sociology
Implicit
94.5
35
History
Implicit
73.3
30
Language
Calture
Accounting
Explicit
57
Multiple exams
Implicit
86.7
108
Arabic
Driving License
Geography
Implicit
Explicit
76.2
105
Sociology
Implicit
66.7
33
History
Caltuee
26.3
95
History
Implicit
41.1
95
Armenian
Driving License
Explicit
72.2
79
Chemistry
Agnostic
20.0
30
Geography
Implicit
50.5
196
Agriculture
Implicit
85.3
34
Law
Explicit
76.2
42
Azerhuijani
Management
Implicit
66.7
36
Health
Implicil
Exonomics
Implicit
70.7
58
Basque
Professional certificttion
Explicit
64.8
500
Language
Belarasian
Literalure
Calture
67.4
13
Matl
Agnostic
40.8
49
Language Lätersture
Calture
62.5
40
Calture
61.9
Bengali
Professional certification
Mulliple exams
Explicit
Implicit
84.3
80.
166
166
Biology
History
Implicit
93.9
115
Bulgasian
Philosophy
Implicit
98.5
135
Gougraphy
Implicit
91.2
250
Medicine
Explicit
57.1
Chinese
Professional certification
Driving License
Explicit
Explicit
52.1
84.5
71
71
Political
Implicil
History
Implicit
88.2
119
Philosophy
Implicit
83.5
79
Croatian
Religious Stidies
Implicit
90.2
$1
Psychology
Implicit
Sociology
Implicit
94.8
135
History
Caltire
89.4
141
Läterature
Dutch:: Flemish
Eeunomics
Implicit
81.7
Geography
Implicit
93.9
33
Sociology
Implicit
90.1
101
Language
Law
Explicit
69.3
215
Finnish
Political
Feonomics
Implicit
Implicit
73.7
95
Sociology
Implicit
48.6
35
Culturology
Calture
94.8
77
French
Driving License
Lanpuage
Calture
124
Geography
Implicit
68.1
47
History
Implicit
93.8
161
Georgian
Language
Law
Explicit
83.6
171
Gemman
Geography
Implicit
50.0
54
Managernenx
Implicit
89.1
64
Greek
Medical License
Explicit
$4.1
133
Professional Certificttion
Explicit
60.9
133
Implicit
Hehrew
Driving License
Logie
Agnostic
Explicit
60.0
88.6
500
50
Education
History
Implicit
86.7
30
Literature
Caltuee
73.2
41
Hindi
Multiple Exams
Implicit
71
Driving License
Explicit
$7.7
71
Professional Certification
Explicit
70.4
71
Geography
Implicit
75.0
48
Table 10: GPT-4o (5-shot, In Ianlanguage prompting) performance on INCLUDE- BASE per language, academic field, and regional label. Fields with less than 30 examples were excluded from the analysis (Part 1)
29

---

Published as a conference paper at ICLR 2025
Arademic Fidd
Regional Festure
Agricsttre:
tmptten
Inveoneecnaal Staten and foroary
14.
impten
903
Geography
Brglich
48.1
Huran Pedformance
Srptish
Language
Cultmm
76.5
Indonciaa
Phofrasonal Cariiloaicn
Exceumec
Region ecplust
Region coplast
81.3
71.8
125
Sociology
Auticulture
85.7
Hinsory
trpptiii
90-4
Reglon-coplccl
Poychology
95.0
Soatsiegy
Impliin
87.7
Deving Liceus
Resicn explist
Perfroioral Coniliaaion
Peglon-ccplion
66.7
Haaayy
Culure
78.4
241
Kanah
Hatery
brecken
94.99
Kecan
Pofeeiooal Costifiction
Eccomnss
Repiss expliät
Liplich
91.6
46.0
250
Finaner
I Libussii
Phofrosseal Contilaaion
Region.eohhen ecplece
81.3
Eardi Scionce
Apaposi
97.1
Histoy Accourting
Seplich
$4.3
Maley
Region esgliin
78.8
178
Geogriphy
85.3
Maloyuaan
Malipie leam
Hoatth
impliei
Chiture
72.7
53.8
100
Marine Licerse
Exyliic
809
Napai
Profosioral Comilicaiin
Lizzne
Explish
Papaen
61.6
Haleyy
05.8
Mascdoeiia
Vazal
Buites
91.3
224
Socidloyy
92.5
53
Paniaz
Profenional Conifictiinn
Expiin
43.2
125
Geognaghy
65.0
Socialogy
14.6
Polich
Professioaal Cenification
Mut.
Agaosic
Exilih
61.7
800
Agrisstton:
7000
Implic
Postagsese
Maragenett
51.9
Heall
763
Eccoinks
Sepfic
85.7
Law
Paplic
71.3
Manegmerr
Ampicc
66.2
56
Risian
Medicia
Expic8
73.3
Licetae
Exiici
Qualinetto
75.7
Booonis
Bmpiic
63.9
Hissyy
41.5
Peychiloyy
Smptic
96.3
Socialosy
kmplici
91.1
45
Language
Cuture
05.6
Spanch
Lioontre
Expicc
91.8
Phiboophy
bepiich
90.3
Froommix
Paplica
65.6
Geogradhe
96.2
Ditoniogs
Cutore
91.6
Tipado
Larguage
Hisiry
Cuhure
Cuhore
85.3
75.2
116
ExpRch
Timil
Mateple Deam
70.6
Elacatio
23.0
Haley
Trlugs
Pioemmex
Papten
600
Geograbhy
bmpicn
71.3
Sctennes
Impize
61.3
Phikooppo
94.5
Tirkid
Puitco
75.9
166
Geography
$1.8
Implicn
Law
Expticn
92.4
250
Ukmimia
Moyicc
Apsresi
$4.0
Pyychology
79.2
Caltrologs
Hassy
hmptici
6.1
134
Lhbuk
Paypiii
60.
Medicel
Paplich
Hakery
86.3
Vistsamo
Geognaght
90.8
250
Table 11: GPT-4o (5-shot, In la-gange prompting) performance INCLUDE- BASE per language, academic field, and regional label. Fields with less than 30 examples were excluded from the analysis (Part 2)
30

---

Published as a conference paper at ICLR 2025
Language
Acc (k:50)
Acc (k:100)
Acc (k:200)
Acc (k:512)
Total gain
Uzbek
51.4
60.6
66.6
68.6
17.2
Armenian
28.0
30.7
36.0
41.1
13.1
Malayalam
57.0
57.4
61.0
69.9
12.9
Urdu
53.7
56.8
58.8
62.2
8.5
Greek
58.0
58.2
63.8
66.4
8.4
Korean
60.4
61.0
62.4
68.8
8.4
Chinese
61.8
63.5
65.5
8.3
Finnish
63.3
64.4
67.0
69.1
5,8
Basque
60.0
60.8
63.8
64.8
4.8
Polish
74.1
75.2
75,4
78.1
4.0
Azerbaijani
67.7
69.2
70.4
71.5
3.8
Dutch; Flemish
81.9
82.9
83.8
85.3
3.4
Telugu
63.9
63.9
64.8
66.6
2.7
Hindi
72.0
72.4
73.7
74.4
2.4
German
64.0
65.5
65.5
66.2
2.2
Malay
81.8
82.4
82.8
2.2
Tamil
67.3
67.3
67.8
69.5
2,2
Arabic
76.3
76.8
77.9
78.4
2.1
russian
72.6
73.6
74.1
74.6
200
Italian
88.0
88.5
89.2
89.6
1.6
Spanish
83.1
83.3
84.0
1.6
Japanese
78.6
78.6
79.4
80.0
1.4
Georgian
86.2
86.4
87.0
87.6
1.4
Vietnamese
82.4
82.5
84.9
83.8
1.4
Turkish
63.5
64.1
64.4
64.8
1.3
Kazakh
79.2
79.6
80.4
80.4
1.2
Portuguese
72.8
73.5
73.5
74.0
1.2
Bengali
75.2
75.4
76.1
76.3
1.1
Persian
60.9
61.1
61.3
61.9
1.0
Belarusian
50.0
50.0
50.2
French
80.0
80.2
80.4
80.7
0.7
Indonesian
77.8
78.2
78.4
78.5
0.7
Albanian
88.9
89.3
89.3
89.5
0.6
Lithuanian
89.7
89.7
90.1
90.3
0.6
Estonian
92.0
92.0
92.4
92.4
0.4
Croatian
87.8
88.0
88.2
88.0
0.2
Hungarian
75.3
75.3
75.5
75.5
0.2
Nepali
71.8
72.0
71.6
72.0
0.2
Bulgarian
90.7
90.7
90.7
0.0
Hebrew
86.0
86.0
86.0
86.0
0.0
Macedonian
92.4
92.4
92.4
92.4
0.0
Serbian
91.5
91.5
91.5
91.5
0.0
Tagalog
87.4
87.4
87.4
87.4
0.0
Ukrainian
85.5
85.5
85.5
85.5
0.0
Table 12: GPT-4o performance for different values of k for in-language prompting (the output generation length) per language on INCLUDE-BAN and total performance gain from 50 to 512.
Major training language
SoTA Monolingual
Monolingual Ace
GPT-4o
Qwen2.5-14B
Qwen2.5-7B
Chinese
Baichuan-7B MILMA-9B-Inste calm2-7b-ehat
38.7
68.1
82.2
Arabic
56.9
78.1
70.5
61.6
Japanese
25.0
75.0
69.2
64.7
Korean
35.3
75.0
83.2
76.8
Russian
11GPT-3,5-138 SanckkrautLM--2-44b-DDP0
53.8
69.0
68.2
59.6
German
56.8
58.3
56.1
Table 13: Accuracy of the multilingual and monolingual models for answering INCLUDE BASE questions for specific target languages.
31

---

Published as a conference paper at ICLR 2025
Language
R²
Model
R²
Albanian
0.646
GPT-4o
0.077
Chinese
0.985
Qwen2.5-14B
0.546
French
0.770
Aya-expanse-32B
0.290
German
0.495
Aya-expanse-8E
0,333
Italian
0.953
Qwen2.5-7B
0,412
Lithuanian
0.945
Mistral-7B
0.231
Persian
0.833
Gemma-7B
0.001
Polish
0.831
Llama 3.1-70B
0.020
Portuguese
0.930
Llama 3.1-8B
0,001
Table 14: R² scores between the performance different models for newly-collected data and existing benchmarks stratified by language and model.
Language coverage
Region agnostic (%)
Region related (%)
Benchmark
Knowledge Coverage
Academic knowledge
ArabicMMLU [1]
Arabic
(elementary school, high school, university). Driving License
24.8%
75.2%
Academic knowledge
CMMLJ [2]
Chinese
(elementary school, high school, university)
25.6%
74.4%
Academic knowledge
PersianMMUU [3]
Persian
(elementary school, high school.
63.1 %
36.9%
university)
Academic knowledge
TurkishMMLU [4]
Turkish
(elementary school, high school, university)
34.8%
65.2%
VNHSGE EXAMS [7]
Vietnamese
High school examinations
40.4%
59.6%
languages
High school examinations
43.7%
56.3%
Academic knowledge (elementary school, high school, university), Professional examinations (Medical exam, Bar exam, Teaching exam), Occupational Licenses (Driving license, Marine license and more)
INCLUDE-BASE (ours)
languages
7.8%
92.2%
Table 15: Existing published benchmarks descriptive and the comparison with INCLUDE-BASH
Percentage of errors (%)
Error type
Description
Errors that occur when the model fails to perform arithmetic required to answer question correctly.
Computational Errors
2.7%
Error that involve the model lacking knowledge of facts unrelated to a specific region or language. For exumple, in the question, "Which of value between
Factual Knowledge Emors
1000? Choices: [Gini coefficient, interest rate, personal eredit score, corporate economic survey index]," the model may choose incorrectly due to lack of factual knowledge.
the following
26.7%
Errors that arise when the model lacks knowledge specific particular region, even though it demonstrates proficiency the language itself.
Regional Knowledge Errors
38.6%
Errors that when the model fails to follow prompt format does provide answer, indicating challenges with language understanding instruction processing.
Format Errors
32.0%
Table 16: Breakdown of error types across 1.50 manually-annotade examples from 6 languages.
32


Reviews:
[{"review_id": "O5EP5qhgxl", "reviewer": "Reviewer_BvNN", "summary": "This paper collects multiple-choice questions in a wide range of languages other than English and evaluates a selection of multilingual LLMs. For further analysis, the questions are categorized as cultural or region-specific (subdivided further into region-implicit and region-explicit). The evaluation shows that there is room for improvement in LLM performance and that the performance varies across different languages.\n\n====\nThe score is increased after the rebuttal.", "questions": "* Why are only two LLMs evaluated using both 5-shot and 0-shot CoT methods? \n\n* Have you try prompting the LLM in English and the target non-English question in 5-shot evaluation? Does the prompt language affect the performance?\n\n* What is the chance that the LLMs have already been trained on some of the Include questions?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper presents a new benchmark called Include, which collects multiple-choice questions sourced from existing exams in a variety of languages and domains. The selection of exams is conducted through a community survey, which appears to be an appropriate method for reaching out to native speakers and obtaining links from them. Two down-sampled versions of Include are available, which offer an equal amount of tasks in different languages and domains.\n\n* The paper evaluates the performance of 15 multilingual LLMs in five-shot and zero chain-of-thought fashion. \n\n* The paper analyses the results with respect to language, script, domain, and question category (cultural, region-implicit, region-explicit, academic) and formulates clear take-away messages.", "weaknesses": "* Only two LLMs are evaluated using both 5-shot and 0-shot CoT methods. It's unclear why the 0-shot CoT evaluation is missing for most models.\n\n* Although the paper focuses on evaluating multilingual models, which seems to be a practical scenario, an ablation study with monolingual LLMs in a subset of languages could establish an upper bound for performance.\n\n* The paper does not provide the actual names of the collected exams, along with their publication dates and formats. This information could be important for considering potential leakage to the LLM's pre-training data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k6Ck5bvVZ0", "reviewer": "Reviewer_gceL", "summary": "They construct a benchmark including 197,243 QA pairs in 44 languages. They collect it from local exam sources to measure the capabilities of multilingual LLMs in a variety of regional contexts. Experimental results show that these models perform inconsistently across different languages, particularly struggling with region-specific questions.", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The dataset is extensive, covering a broad range of domains and covering various application scenarios.\n\n- The writing is generally clear.", "weaknesses": "- Experimental analysis is not quite enough and needs more takeaways. \n\nMany of the experimental conclusions mentioned are known or intuitive. For example: In line 323 \"Most instruction-tuned models perform slightly worse or on par with their base counterpart\" which has been discovered in previous work[1][2] on English training corpus. Also for conclusions like LLMs' performance is poor on unseen language during pretraining. \n\nAlthough a lot of effort has been put into building this benchmark, I feel that there is still a lack of takeaway, especially considering that one of the purposes of the Evaluation model is to improve the model. It's better to provide more find-gained analysis and insight like Figure 4 and also the insightful display (on the Main page rather than the Appendix) for conclusions like line142 model performs\nworst on cultural questions.\n\n- Doubts about the reliability of experimental findings\n\nAs you mentioned, some questions are \"Cultural\" and some are \"Implicitly Regional\". I am curious about the instructions for evaluating these two types of questions. Have you given the country or culture context of the questions being asked like: answer the question under  XXX culture? \n\n- The Question Format is simply MCQ\n\nUnderstanding that MCQ is more accessible and easier to deal with, however, open-ended questions can reflect the models' performance on real application scenarios.\n\n\n\n[1] Urial: Tuning-Free Instruction Learning and Alignment for Untuned LLMs, 2023\n\n[2] A+ B: A General Generator-Reader Framework for Optimizing LLMs to Unleash Synergy Potential, ACL, 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TKOpcrzSIl", "reviewer": "Reviewer_E3PJ", "summary": "This work identifies a lack of high-quality multilingual data available to evaluate LLMs. Existing resources are often translated from English and focus on a specific regions or domains. To address this gap, the authors introduce a large multi-lingual evaluation benchmark from close to 2000 regional exams across 44 languages, including high-, mid and low-resource languages. A range of 15 models is then evaluated on the new benchmark. The results indicate that performance largely varies across models and languages and there is still room for improvement even among the best performing models.\n\n[see below for my response to the author comments.]", "questions": "* On the technical side, it is not clear to me how exactly the model’s answer to a question is extracted from its output. In the first paragraph of Section 5.4 (lines 440-448), the authors mention that the adopt the prompt template by Hendrycks et al. and mention that the model does follow it consistently. What does the expected output look like and how exactly is the answer retrieved in case the model does not follow it? An example would be beneficial.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper introduces a large and high-quality evaluation dataset that addresses previous short-comings of existing LLM benchmarks. \n* Specifically, it identifies the fact that most existing multi-lingual resources are translated and therefore do not adequately reflect regional and cultural aspects that become relevant when models are actually deployed in respective environments.\n* The paper’s structure is clear, the English is good and it is easy to follow\n* The main contribution is an LLM benchmark, and as such it is not conceptually novel. However, it addresses a relevant short-coming and the introduction of „regional knowledge“ makes it original.\n* Results are discussed in detail and Section 5 and the Appendix", "weaknesses": "* The paper argues that existing LLM benchmarks do not adequately cover multi-linguality nor regional knowledge and it lists related benchmarks in the introduction and related work. However, this argument is only on a qualitative level and would benefit from quantitative facts about, the size, number of languages, etc. of individual existing benchmarks. This could be included in a table providing an overview. [addressed]\n\n* The paper is long on numbers, but short on more qualitative insights. For example, if a model fails to answer an exam question, is this due to a linguistic failure (the model fails to understand the question) or does it lack the regional/cultural knowledge? What types of errors do the models make? Insights of this kind would be crucial to make the results of the paper actionable for future model development, and while difficult to carry out automatically at scale, could be conducted manually on a sample for illustrative purposes. [substantially addressed]\n\n* The evaluation also does not contain any hard evidence that the knowledge contained in the proposed benchmark cannot be tested by existing ones. It would be interesting to see correlations between performances on different benchmarks. This would be straightforward to do at least for the subsets of external benchmarks that are already included under „Rounding up the Benchmark“ in Section 3.1. [addressed]", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0XLQ8nRzXc", "reviewer": "Reviewer_i51G", "summary": "The paper presents a new resource to evaluate the multilingualism of LLMs. The dataset, called Include, comprises multiple-choice questions from 44 languages about regional and global knowledge. The authors evaluate a battery of LLMs ranging from closed-source models (GPT-4o) to a variety of open-source models (C4AI-Aya, Mistral, Gemma, Qwen, Llama, XGLM, and Bloom). The paper presents a thorough analysis, comparing the performance of the models in different languages, domains, and regional/global knowledge. Finally, the authors discuss the challenges of evaluating these models in a setting with several languages.", "questions": "I have a couple of comments and questions about your work:\n* **I missed some key references**: There are a couple of papers that are very aligned with this work, for instance, in the line of generating evaluation resources for LLMs in low-resource settings (particularly Basque):\n  * (Etxaniz et al., 2024b): They present (along with pre-training data and models) an evaluation suite for Basque based on C1-level certificate of proficiency exams, reading comprehension exercises, trivia questions, and  Public Service examinations.\n  * (Etxaniz et al., 2024c): They analyze the effect of regional vs non-regional knowledge of the LLM using an evaluation dataset that is parallel in English and Basque.\n* **Have you tried any variants of Claude?** As for my experience, Claude performs better than GPT-4o in a couple of languages other than English that I tested. I am not asking to repeat the experiments with Claude, but I just have the curiosity of whether my personal feeling is reflected in your data.\n* **Remove Answer Acc. from Table 1:** This metric is not comparable among the models in the table (as far as I understand). This can lead to confusion where worse models perform better just because they generate more wrongly formatted answers. If not removed, I would at least remove the bold results to avoid comparisons.\n\n\n### References:\n * [Do Multilingual Language Models Think Better in English?](https://aclanthology.org/2024.naacl-short.46/) (Etxaniz et al., 2024a)\n * [Latxa: An Open Language Model and Evaluation Suite for Basque](https://aclanthology.org/2024.acl-long.799/) (Etxaniz et al., 2024b)\n * [BERTAQA: How Much Do Language Models Know About Local Culture?](https://arxiv.org/pdf/2406.07302) (Etxaniz et al., 2024c)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* The paper is well-written and is easy to follow.\n* The resource presented in this paper addresses a very prevalent challenge: the disparity in resources involved in the development of LLMs for English and non-English languages. Moreover, not only allows for the assessment of the quality of LLMs in the given languages but also in their regional cultural knowledge, which is most of the time ignored by the large body of research.", "weaknesses": "I did not find any major weaknesses in this paper. However, here are some possible improvements for this work:\n* **Asses the difference in performance when the model is asked in English**: Etxaniz et al. (2024a) showed that these LLMs perform better in English although they understand the target language. They propose the \" Self-translate \" method, which prompts the same model to first translate the example into English before answering the question. Although I do not think you need to necessarily apply the \"Self-translate\" method, it would be nice to translate the examples into English (using machine translation) and compare the difference in performance. This way you can assess the knowledge encoded in the LLMs separately from their language capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)