Kaleidoscope: In-language Exams for Massively  Multilingual Vision Evaluation
Authors: Israfel Salazar
Manuel Fernández Burda
Shayekh Bin Islam
Arshia Soltani Moakhar
Shivalika Singh
Fabian Farestam
Angelika Romanou
Danylo Boiko
Dipika Khullar
Mike Zhang
Dominik Krzemiński
Jekaterina Novikova
Luísa Shimabucoro
Joseph Marvin Imperial
Rishabh Maheshwary
Sharad Duwal
Alfonso Amayuelas
Swati Rajwal
Jebish Purbey
Ahmed Ruby
Nicholas Popovič
Marek Suppa
Azmine Toushik Wasi
Ram Mohan Rao Kadiyala
Olga Tsymboi
Maksim Kostritsya
Bardia soltani moakhar
Gabriel da Costa Merlin
Otávio Ferracioli Coletti
Maral Jabbarishiviari
MOHAMMADAMIN FARAHANIFARD
Silvia Andrea Fernandez
María Grandury
Dmitry Abulkhanov
Drishti Sharma
Andre Guarnier De Mitri
Leticia Bossatto Marchezi
Setayesh Heydari
Johan Obando-Ceron
Nazar Kohut
Beyza Ermis
Desmond Elliott
Enzo Ferrante
Sara Hooker
Marzieh Fadaee
Venue: ICLR 2026 Poster (2026)
Keywords: multilingual benchmarks
vision-language models
multimodal evaluation
cultural diversity
low-resource languages
machine learning evaluation
Primary area: datasets and benchmarks
TL;DR: We introduce a large-scale, culturally grounded multimodal multilingual benchmark to evaluate vision-language models across 18 languages and 14 subjects, revealing performance gaps across modalitities and low-resource settings.

Abstract: The evaluation of vision-language models (VLMs) has mainly relied on English-language benchmarks, leaving significant gaps in both multilingual and multicultural coverage. While multilingual benchmarks have expanded, both in size and language, many rely on translations of English datasets, failing to capture cultural nuances. In this work, we propose Kaleidoscope, as the most comprehensive exam benchmark to date for the multilingual evaluation of vision-language models. Kaleidoscope is a large-scale, in-language multimodal benchmark designed to evaluate VLMs across diverse languages and visual inputs. Kaleidoscope covers 18 languages and 14 different subjects, amounting to a total of 20,911 multiple-choice questions. Built through an open science collaboration with a diverse group of researchers worldwide, Kaleidoscope ensures linguistic and cultural authenticity. We evaluate top-performing multilingual vision-language models and find that they perform poorly on low-resource languages and in complex multimodal scenarios. Our results highlight the need for progress on culturally inclusive multimodal evaluation frameworks.

Published as a conference paper at ICLR 2026
KALEIDOSCOPE: IN-LANGUAGE EXAMS FOR MASSIVELY MULTILINGUAL VISION EVALUATION
Israfel Salazar* Manuel Fernández Burda" Shayekh Bin Islam 4,31
Arshia Soltani Moakhar*1
Shivalika Singh*1 Fabian Farestam" 17
Angelika Romanou' 30
Danylo Boiko Dipika Khullar Mike Zhang2:10,2
Dominik Krzemiński* Jekaterina Novikova Luísa Shimabucoro" 18
Joseph Marvin Imperial Rishabh Maheshwary" Sharad Duwal
Alfonso Amayuelas Swati Rajwal Jebish Purbey4.7 Ahmed Ruby?5
Nicholas Popovič!1.12 Marek Suppa?2.2 Azmine Toushik Wasi
Ram Mohan Rao Kadiyala? Olga T'ymboi' Maksim Kostritsya 15,16
Bardia Soltani Moakhar
Gabriel Costa Merlin Otávio Ferracioli Coletti
Maral Jabbari Shiviari MohammadAmin Farahani Fard
Silvia Fernandez 4,33
María Grandury 21 Dmitry Abulkhanov
Drishti Sharma
Andre Guarnier De Mitri
Leticia Bossatto Marchezi Setayesh Heydari
Johan Obando-Ceron' 4.2229
Nazar Kohut
Beyza Ermis"
Desmond Elliott! 2.28
Enzo Ferrante 13,32
Sara Hooker
Marzich Fadace"
Cohere Labs
University of Copenhagen
*CONICET & Universidad de Buenos Aires
*Cohere Labs Community "UC Santa Barbara EEmory University
"ZeroGrad
TTraversaal.ai "Taras Shevchenko National University of Kyiv 10 Aalborr University
"Krlsruhe Institute Technology 1> TU Dresden 13T-Tech
LLvvv Polytechnic National University HSE University
"HiveTrace
'ETH Zürich
18 "University of São Paulo
19 NaNational University Philippines 20 Federal University of São Carlos
SomosNLP Cisco
Commeniss University in Bratislava
2'iiaa Québec Al Institute
25 Uppsala University 26 Vanguard
University of Bath
Pionner Center for Al 39 AI Cirele
Université de Montréal
EPFL KAIIST
32 Annyne AI
Correspondence: salafellsslazzardd.. ku. mburda@de. uba ar, maw.ieeecchheee.ccom
ABSTRACT
The evaluation vision-language models (VLMs) has mainly relied on English- language benchmarks, leaving significant gaps in both multilingual and multicul- tural coverage. While multilingual benchmarks have expanded, both in size and language, many rely translations of English datasets, failing to capture cultural nuances. In this work, we propose K KALEIDOSCOPE, as the most comprehensive benchmark date for the multilingual evaluation of vision-language models. KALEEDOSCOPE is a large-scale, in-language multimodal benchmark designed evaluate VLMs across diverse languages and visual inputs. KALEIDOSCOPE covers 18 languages and 14 different subjects, amounting to a total of 20.91 multiple-choice questions. Built through an open science collaboration with a diverse group of researchers worldwide, KALEIDOSCOPE ensures linguistic and cultural authenticity. We evaluate top-performing multilingual langugee models and find that they perform poorly on low-resource languages and in com- plex multimodal scenarios. Our results highlight the need for progress on culturally inclusive multimodal evaluation frameworks.
INTRODUCTION
Evaluations are the backbone of measuring progress in machine learning, yet many benchmarks especially for language models remain English and Westem-centric (Joshi et al. 2020 Fan et al.)
First authors $ Senior authors

---

Published as a conference paper at ICLR 2026
HASSMIGG
What the losal number triangles the Figure bellow?
Portuguese
Serblan
Pensan
Hindi
English
Spanish
D
23
Utminiaan
Hungarian
LITERTTURE
Telugu
Mwech the
Bengaii
following
French
escerprs with
Geman
their authors
Uthuanian
Anaic
Creatian
Nepal
PJBl
Citv
Ri
Siil
Q
150
Number samples
Arts & fummmnnii
Socill Sclenoe
PMi Cii Siv
D
Qjv
Rii
Sail
Multimodal
Text
Reatoning
Iealth Science
Forttiall SU
Icl
Figure 1: Overview of the KALEIDOSCOPE Benchmark. (a) Multilingual MultilngualMMulimmdda MCQ
Samples (b) Language and Multimodal Samples Distribution. (c) Exam Category Breakdown.
2020: Dodge et al. 2021 Liu et al.
2021 Chung et al. 2022, Gehrmann et al. 2022 Lucy et al.
2024). This imbalance becomes even more striking at frontier of AI, where generative models are
rapidly expanding into multimodal territory (OpenAl et al. 2024 Google et al. 2024 Anthropic
2024] Deitke et al.) 2024 Yue et al. 2025 Qwen-Team (025)) seeking to represent a richer world
made up of different modalities such as image, text, sound. In recent years, the community has
made promising strides toward broader multilingual text evaluation (Ahuja et al. 2023 (Singh et al.
2024b Aakanksha al., 2024 Pozzobon al. 2024 [Romanou al. [024]] Singh al. 2025
Adelani et al. 2024). and multimodal benchmarks are starting to take shape (Bugliarello et al. 2022
Fu et al. 2023 Yue et al.) 2024a b [i et al., 22024a] Xu al., 2025). Yet reliable evaluation the
intersection of multilingual and multimodal tasks remains rare. This gap motivates our work.
common but imperfect solution is translating English benchmarks into other languages. While convenient, this often falls short of capturing cultural context and nuance. Translated datasets can easily reinforce et al. 2018: Singh et al.) 2025 Longpre et al. 2025) limiting their ability to assess performance across diverse settings. Moreover, automated data curation pipelines frequently amplify existing
centric knowledge and assumptions (van Miltenburg et al.) [017 Frank
quality issues (Luccioni& & Viviano 2021 Caswell et al. 2020 Kreutzer et al. 2022), with translation
artifacts such as translationese muddying evaluations (Koppel & Ordan 201 Zhang & Toral 2019
Bizzoni et al. 2020 Vanmassenhove et al. (02II) While translated data has its place, especially
for some particularly low-resource tasks (Zhou et al. 2021 Thaplival et al. 2022 Qiu et al. 2022
Ramos et al. 2024 Geigle et al. 2025 Dang al. 2024 Ustün al. 2024 Aakanksha et al.
2024
is an imperfect substitute for genuinely diverse, in-language benchmarks.
In this work, we introduce the largest benchmark of real-world, in-language exam questions blending image and text modalities. Our dataset pushes beyond simple captioning, challenging models to reason about visual content in various topics, the way humans are evaluated in exams worldwide. Through a large-scale open science effoort across 18 languages, we construct KALEIDOSCOPE (see Figure Mp. featuring diverse selection of knowledge domains across 14 subjects. With 55% of the total 20,911 questions requiring image understanding for accurate resolution, our work establish comprehensive, and inclusive evaluation framework for multimodal language models. We evaluate wide range of ofate-of-the-art models KALEIDOSCOPF, including Claude 3.5 Sonnet (Anthropic) 2024), GPT-4o (OpenAl et al. 2024), and Gemini-V V (Google et 2024), as well as smaller open- weight VLMs, such Aya- Vision model family (Cohere-For Al-cee-- (025). Molmo (Deitke et al., 2024) Pangea (Yue et al., 2025), and Qwen2.5-VL model family (Qwen-Teanl 2025). Our key contributions and findings are highlighted here:
We present the largest multilingual multimodal exam set, covering high resource (c.g., English, Spanish) to underrepresented languages (e.g., Bengali, Telugu) across diverse subjects from sociology to STEM. Most languages (10/18) include 5+ topics, with the rest focusing on multi-subtopics like mathematies or engineering. Questions emphasize vision grounded
KALEIDOSCOPE Benchmark:
2

---

Published as a conference paper at ICLR 2026
reasoning through tasks like interpreting graphs, pictures, and region-specific diagrams, supported by fine-grained metadata for model diagnostics.
- Performance Disparities: All models perform substantially better text-only questions, revealing a clear disparity across modalities. The gap widens in larger modelsl; for instance, GPT-4o shows 21.6% difference between text-only and multimodal performance, while smaller models like Molmo exhibit a much narrower gap of 3.69%. (Section (.1). Furthermore, multimodal performance varies significantly by visual data type: models are more capable of answering questions about tables (76.5%) and photographs (81.5%) compared to diagrams (62.9%).
Domain-Speeiffr Performance Disparities: We observe a significant performance gap between questions requiring knowledge of Humanities & Social Sciences and those focused on STEM subjects (Section 4.4j. On average, models present accuracy of 83.7% for humanities versus 59.2% for STEM (based on the best scores across models). Models struggle more with STEM questions, suggesting that while they can often recognize visual content and retrieve related knowledge, they lack the reasoning capabilities needed to arrive at the correct answers in STEM domains.
Crosslingual Performance Disparities: Model performance varies across languages, with better re- sults in high-resource languages and weaker performance in mid- and low-resource ones (Section 4.3) Crosslingual transfer appears to play a role, as models perform better on average in languages using Latin scripts compared to those with non-Latin scripts.
THE KALEIDOSCOPE BENCHMARK
The KALEIDOSCOPE Benchmark is global collection of multiple-choice questions sourced from real-world exams, with the goal of evaluating multimodal and multilingual understanding in VLMs. The collected exams in Multiple-choice question answering (MCQA) format which provides
structured framework for evaluation by prompting models with predefined answer choices (Hendrycks)
al. 202T Lu al. 2023 Wang al. 2024a. Yue et al. [024a] Romero al. 2024; Romanou et al. 2024), closely mimicking conventional human testing methodologies. Our work is built around
three core design principles that guide the selection, curation, processing, and addition of exams:
Images are central to KALEIDOSCOPE, as we aim to evaluate how VLMs integrate and reason about visual information to answer questions. We prioritize multimodal questions with diverse image types, complemented by a similar proportion of text-only questions for complete assessment and comparison.
Multimodality:
The benchmark contains questions in 18 languages, with a focus on under-represented mid- and low-resource languages (e.g.. Nepali, Lithuanian) alongside high-resource languages English, Spanish) for a thorough evaluation across a broad range of languages.
Multilinguality:
Diversity: Our goal is to collect exams covering wide range of topics possible ranging from Mathematics and Sociology, to Medicine and Driving Licenses, ensuring comprehensive evaluation across various domains. The final collection includes exams from 14 different domains, collected from 18 countries and with varying educational levels, allowing detailed clustering and comprehensive evaluation.
2.1 GLOBAL COLLABORATION
Our work entailed an extensive, open science process to manually collect data by working directly with native speakers of different languages (Elliott et al. 2016 Liu et al., 2021 Thapliyal al. 2022 Li et al. 2024c| Ustün et al., 2024 Singh et al. 2024b). This is acutely needed in the field of machine learning, where recent studies have highlighted that dataset creators remain predominantly Western-centric (Longpre al. 2025). The manual curation of datasets is a costly process that requires careful attention to detail in every language to ensure high-quality, contextually relevant content for evaluation. In this work, we engage in large-scale open science collection process, which brings together contributors spanning 20 nations across four continents to ensure linguistic and cultural authenticity. For related participatory research Appendix C.1
3

---

Published as a conference paper at ICLR 2026
2.2
DATA PIPELINE
Collection: We collected KALEIDOSCOPE following guidelines on type of exams and questions required, formatting, specifications, and quality control measures. Data was collected through a global call for contributions and distributed across global communities, with the majority of contributors being independent researchers in the open science community. This effort resulted in 20,911 questions from 18 countries and languages, sourced in their original languages to maintain linguistic authenticity. We prioritized original, Nomain-cxpert-writtee questions (c.g., from teachers), ensuring real-world relevance and quality. The exams were gathered from various repositories, including official government websites, question banks, and other publicly available repositories with educational materials. Throughout the process, contributors also annotated associated licenses with each dataset to allow for documentation of data provenance (Longpre et al. 2024).
Processing:
The annotation process involves two stages. First, we perform automated parsing and extraction. For directly parsable text. we use PDF or web parsers, while for non-parsable text, we employ OCR API's, such Mathpix] along with vision-language models such GPT-4o. These tools allow us to extract both text and image elements from exam source formats, which are then converted into structured outputs in LaTeX, Markdown, and JSON formats, as required. Since automated parsing can sometimes result in misaligned images and text, in the second stage we refine the extracted text. Applying heuristic rules, as well terh-performing LLMs (Claude 3.5 Sonnet and GPT-4o), we restructure the output, ensuring proper alignment of questions, text, and answer choices. Human verification follows, ensuring images are correctly linked to the corresponding questions, and checking that extracted formulas match the expected equation format.
Maintaining reliable and high quality data is essential, especially given the large-scale international collaboration in this project. To ensure integrity, we include manual validation in three stages the collection and annotation pipeline. First, at the end of the collection stage, two independent annotators validate each exam ensure conformity with the guidelines. We include a strict revision to confirm compliance with the distribution license requirements. Only exams approved by both independent annotators are included in the dataset. Next, following the annotation process, a validation script checks for JSON formatting errors, duplicates, and malformed strings that do not conform identified entry specifications (see Appendix E.3). Finally, at the last stage, two separate validators perform a final manual review of the collected files before merging them into KALEIDOSCOPE.
Quality Assessment:
Quality control also extends to the evaluation, where we analyze the most prominent failure modes. During inference, suspicious outputs, such ambiguous answers, no response, or consistent failures across models, are flagged for manual review. If an issue is identified, the entire exam containing the problematic question is reviewed for correction or removal. This process guarantees that any errors in the benchmark questions are identified and addressed, further enhancing the reliability of the dataset.
2.3 DATA STATISTICS
The final KALEIDOSCOPE benehmark contains 20,911 questions across 18 languages belonging to 8 language families. A total of 11,459 questions require an image to be answered (55%), while the remaining 9,452 (45%) text-only. The dataset covers 14 different subjects, grouped into broad domains. Figure presents an overview of the dataset; detailed statistics can be found in Appendix B.3 The majority of questions in KALEIDOSCOPE
multimodal, with the exact proportion varying across languages, ranging from 50% to 100%, with some languages always requiring images for resolution.
Each exam question contains 17 fields, including source country, language, license, educational level, category, and multimodal information. These fields are detailed in Appendix E.3 The questions are formatted in MCQA format with 4 options and single correct answer. The subject is labeled in both English and the source language. The educational level (e.g., high school, university entrance, professional licensing) is also included to ensure diverse representation. Multimodal questions additionally specify the type of image, such as graphs, tables, or diagrams. Additionally, each entry includes metadata such as source details, licensing status, and ISO 639-1 language codes. For fine- grained analysis, each question includes detailed metadata, with examples provided in Appendix E.
htttss:/mmate convert.
4

---

Published as a conference paper at ICLR 2026
The metadata allows us to evaluate how visual and textual elements interact in multimodal reasoning tasks, making the benchmark valuable for evaluating models across diverse scenarios.
KALEIDOSCOPE covers wide range of languages, including low- and mid-resource languages such as Nepali, Lithuanian, Bengali, Telugu, Persian, Ukrainian, Croatian, Serbian, and Hungarian, as well high-resource languages such as English, Spanish, Portuguese, Russian, French, German, Arabic, Hindi, and Dutch. This selection allows us to evaluate how performance is affected by the amount of resources available for given language. The dataset spans 8 different language families, providing broad linguistic range. The number of questions per language varies significantly, from 126 for Nepali to 2000 for Portuguese, Serbian, and Persian. The linguistic diversity present in KALEiDOSCOPE enables a robust evaluation of models across both widely spoken and underrepresented languages, making the dataset suitable for comprehensive multilingual assessment.
EXPERIMENTAL SETUP
3.1 MODELS
We benchmark both open-weights and closed multimodal vision-language models on KALEIDO- SCOPE, focusing on lighter open-weight models and larger closed models assess performance across a wide range of model sizes. The open-weight model include Aya- Vision-8B and 32B (Cohere- For-Al-Team, 2025), ) Molmo-7B-D (Deitke et al. 2024). Pangea-7B (uue et al.) 2025), and all
sizes of Qwen2.5-V 2.om2.55L--.. InIssrruct (Qwen- Team 2025 (3B, 7B, 32B, and 72B) to analyze the impact of model scale on KALEIDOSCOPE, All models have image and multilingual support; Aya- Vision supports 23 languages, Qwen2.5-VL supports 29 languages, and Pangea was trained on dataset spanning 39 different languages, making them strong candidates for multimodal and multilingual evaluation. For the closed models, evaluate GPT-4o (OpenAI et al. 2024) (2024/08/06), Claude 3.5 Sonnet (Anthropic 2024) (2024/10/22), and Gemini 1.5 Pro (Google et al. 2024).
3.2 EVALUATION SETUP
We designed two distinct evaluation setups to accommodate VLMs' varying reasoning and instruction- following capabilities. For closed models, we employed zero-shot prompts using the Chain-of- Thought (CoTT) method (Wei et al. 2022). instructing the model reason step-by-step before selecting the final answer within specific <ANSWER> </ANSWER> tags natural approach aligned with real-world MCQ applications. Using a common template (see Appendix D.3), we ensured equal evaluation conditions across models, with instructions translated into all evaluated languages for fully in in-lanuuage setup, following Romanou et al. (2024). For smaller open- weight models, which showed limited CoT effectiveness in preliminary experiments (Appendix D.2). we instead used direct answer generation approach, prompting models to output their choice in a JSON-structured {! choice': : ...) field. This simplified the task by reducing reasoning formatting errors, with instructions always in English regardless of question language. Further discussion on model output errors is in section 5 Due to KALEIDOSCOPE'S evaluating nature, we employ accuracy as main metric (further evaluation metrics details can be found in Appendix D.
4 RESULTS
4.1 OVERALL PERFORMANCE
We benchmark wide variety of models on KALEIDOSCOPE, with results summarized in Table I Claude 3.5 Sonnet and Gemini 1.5 Pro lead among closed models, while GPT-4o's 's performance is impacted by high format errror-pprticullle particularly in the multimodal split- thlit-though its accuracy improves significantly when considering only valid answers (see Section 5) Among open- open-weight models, Qwen2.5-VL-722 achieves the highest accuracy, followed by lightweight models like Qwen2.5-VL. 7B, which leads the 7-8B category.
AAll open- weight models are evaluated locally using 1 1XNVIDIA Ampere A100 GPU with 64GB of memory for models up to 8B, and 4xAA100 for models on the range 32B-72B. Closed models were accessed via API. To ensure consistent evaluation environment, set the temperature to 0.7, the maximum token generation to 1024, and the image size to 512x512 for all models.
5

---

Published as a conference paper at ICLR 2026
Table 1: Performance Evaluation on KALEIDOSCOPE. Results are reported as macro-averaged accuracy (%) across all languages (equal weight per language). Acc.: Accuracy over all samples; F.E.: Format Error rate (invalid responses); Valid Acc.: Accuracy excluding invalid responses. Metrics are shown for the full dataset (Overall), multimodal inputs (Multimodal), and text-only inputs (Text-only).
Overall
Multimodal
Text-only
Valid Responses
Valid Responses
Valid Responses
Model
Acc.
F.E.
Acc.
Acc.
F.E.
Acc.
Acc.
F.E.
Acc.
Claude Sonnet
62.91
63.87
55.63
57.24
73.54
0.02
73.57
Gemini 1,5 Pro
62.10
1.62
62,95
55.01
1.46
55.71
72.35
1.81
73.45
GPT--40
58.32
6.52
62.10
49.80
10,50
55.19
71.40
1.71
72.39
Bwvn2.5-VL-72B
52.94
0.02
53.00
48.40
0.03
48.41
60.00
0.02
60.01
Aya- Vision-322
39.27
1.05
39.66
35.74
11.99
36.28
44.73
0.5T
45.00
Qwen2.5-VL-322.
48.21
0.88
48.64
44.90
0.28
45.05
53.77
1.61
$4.60
Aya- vya-Vision-88
35.09
0.07
35.11
32.35
0.05
32.36
39.277
0.10
39.30
Molmo-7B-D
32.87
0.04
32.88
31.43
0.06
31.44
35.12
0.01
35.13
Pangea-78 7
31.31
7.42
34.02
27.15
13.52
31.02
37.84
0.03
37.86
Sven2.5-VL-72
39.56
0.08
39.60
36.85
0.04
36.88
43.91
0.11
43.96
Qwen2.5- VL.BBB
35.56
0.19
35.63
33.67
0.32
33.79
338.5
0.03
33.53
Multimodal vs. Text-only Performance
Non- (script)
Gemini Pro
Claude 3,5 Sonnet
Cloude 3.5 Sonnet
GPT-4o
GPT-40
Gemini Pro
60
Text Accuracy
Owen2.5 VL--788
Latn Accuracy (96
Qwen2.5 5.
Aya-vision-322
Qwen225-VL-7
Ayanvision-Ba Qwen2,5VVL-7
Aya Aia-vision-S0
Qwinn2.5:V4
Vve-2.5-VL-38
Pangea-78
Aya-issoo--B 88
Panges-7B
Molmo--7-D
20
20
Multimodal Accuracy (9a)
Non-Latin Accuracy (hi)
(a) Accuracy a function of modality.
(b) Accuracy function of textual script.
Figure Model Performance Analysis on KALEIDOSCOPE. (a) Accuracy (%) of models on multimodal and text-only questions, highlighting low performance multimodal samples. (b) Accuracy (%) by script type, revealing biases for latin scripts. Accuracy over valid responses is used to generate both figures. Identity line is added to show parity.
key trend is the performance drop in multimodal questions compared to text-only ones, with closed models showing the largest gaps (e.g., GPT-4o's steep decline). In contrast, open- ween-weight models exhibit smaller gaps, suggesting greater robustness across modalities despite lower overall scores. This gap narrows further for smaller models, with Molmo displaying the most balanced performance. This lightweight model's consistency is depicted in Figure 2a (by being closer to the identity line), reinforcing that open models while less specialized, handle multimodal tasks more uniformly.

---

Published as a conference paper at ICLR 2026
Table Model Performance Breakdown by Image Type KALEIDOSCOPE. Accuracy (%) over
valid answers across image type. Bold values indicate top-performing model.
Model
Diagram (2,182)
Figure (6,178)
Graph (733)
Map (392)
Photo (631)
Formula (487)
Table (597)
Text
(257)
Claude 3.5 Sonnet
62.9
50.5
74.2
80.1
77.8
52.1
75.0
85.2
Gemini 1.5 Pro
59.4
51.3
67.9
69.4
75.8
68.3
76.0
85.2
GPT-4o
59.6
48.2
68.4
78.8
81.5
64.4
76.5
86.2
Qwen2.5-VL-77
51.1
43.9
59.4
66.1
70.5
48.7
61.5
86.0
Aya- Vision 32B
38.6
33.4
42.0
50.0
60.2
32.4
33.1
68.8
Qwen2.5-VL-322B
46.7
41.0
53.1
58.2
65.0
47.3
58.0
82.5
Aya- Vision 8B
32.7
29.9
37.2
38.6
42.3
29.2
34.1
54.9
Molmo-7B- 7B-D
30.3
31.5
36.7
37.8
45.0
25.1
30.6
56.8
Pangea-7B
31.0
31.0
32.9
38.5
45.0
32.2
29.4
66.3
Qwen2.5-VL 5-VL-BB
38.0
34.0
44.3
48.0
53.9
34.9
40.9
76.3
Qwen2.5-VL-3B
32.8
32.33
40.2
41.2
48.2
-3.77
35.2
72.8
4.2 NOT ALL IMAGE TYPES ARE EQUAL
KALEIDOSCOPE contains eight visual information types, with accuracy varying significantly by complexity (Table Simpler inputs like text-rich images Doven2.5-VL-72. 76.3%; GPT-4o: 86.2%) and photos score higher than technical categories like Formulas and Diagrams (Qwen2.5-VL-77
38.0%; GPT-4o: 62.9%). Notably, Qwen2.5-VL-72B ranks second in text-rich images, surpassing both Gemini and Claude. Larger models show specialized strengths: Gemini 1.5 Pro dominates Formulas and Figures, GPT-4o leads in text-rich images, and Claude 3.5 Sonnet achieves the highest scores in Diagrams, Graphs, and Maps. In contrast, 20wen2.5-VL-77 consistently outperforms all lightweight models across categories, demonstrating broader capability despite lower absolute scores. The results reveal a clear hierarchy: models handle simple visuals well but struggle with structured or symbolic data, a pattern consistent across architectures but more pronounced in smaller models.
4.3
RESOURCE AND SCRIPT SENSITIVITY IN MODELS
Closed Models
100 Ganini P.5
Cloude Sornet
60
Molmo-700
5-L
Light-weight Open Models
Pargea-78
Owenzu VL
Owen2. S.V
Ays-Vision-38
-i--itoon-00
(6)
60
Anaiic
English
French
Hhot
Hungeitntithaanan
hepali
Pestan
Rusion
Sontaan
Spanish
Ueaininn
Figure 3: Multimodal Accuracy by Language in KALEIDOSCOPE, Reports performance (accuracy %) for closed models and open-weight models multimodal questions.
Performance in KALEIDOSCOPE varies widely across all 18 languages (see Figure 3). Models generally perform well in high-resource languages (e.g.. English, Spanish, German) but struggle with lower- and mid-resource ones, such Nepali and Telugu. This can be attributed the limited
7

---

Published as a conference paper at ICLR 2026
training data for these languages, complex scripts, and the exclusive use of multimodal samples for these languages (see Appendix [.3). which are inherently more challenging. Lithuanian, despite being mid-resource language, stands out the highest-performing language, with Claude 3.5 Sonnet leading in accuracy. This might be due the fact that all I Lithuanian questions belong to Col lege Gra duat i on Exams, and have major subject composition of Social Sciences and Humanities in opposition to STEM subjects, which may align well with the models' capabilities. Closed models show similar performance within each language, except for German, where Claude excels. In contrast, Qwen2.5- VL-7B consistently leads all lightweight models for almost every language, and the heavier Qwen2.5-VL-722 shows the benefits of model scale.
The results show that all models are biased towards Latin script languages. As shown in Figure 2b all models are above the parity line, exhibiting consistent higher performance for Latin scripts compared to non-Latin scripts. Full results can be found in Appendix D.3
4.4 STEM QUESTIONS EXPOSE MODEL DEFICIENCIES
KALEIDOSCOPE consists of exams covering 14 subjects and domains. We observe that models perform significantly better Humanities & Social Science questions compared to other domains. The closed models achieve high accuracy in areas like Sociology (Claude: 93.4%, GPT-4o: 93.2%), Social Sciences (GPT-4o: 88.1 %, Gemini: 85.7%) and Language (GPT-40: 85.8%, Claude: 85.5%). In contrast, performance in STEM subjects, including Mathematics, Physics, and Engineering, is notably lower, with most models scoring below 50%. This suggests that while they are generally capable of recognizing visual content and retrieving surface-level knowledge, they fall short when it comes performing the multi-step reasoning and problem-solving required in STEM subjects. An- swering these questions often demands not just factual recall but also the ability to interpret complex diagrams, apply mathematical concepts, and reason through scientific principles capabilities that current models have yet to fully master. This highlights a key gap in their ability to bridge perception and reasoning, particularly in tasks that require deeper analytical thinking. Refer Appendix [D.3] Table 9 for multimodal and complete results of model performance across subjects.
5 ANALYSIS
5.1 HOW SENSITIVE ARE VLMS TO MISSING OR INCORRECT IMAGES?
Table 3: Image Relevance Analysis for Qwen2.5-VL-7B on KALEIDOSCOPE. Model performance across the standard multimodal, Random Image, and No-Image setups to assess the impact of visual information on
accuracy.
Valid Responses
Setup
Accuracy
Format Error
Accuracy
Standard Multimodal
36.85
0.04
36.88
Random Image
32.56
3.12
33.53
No Image
33.44
0.03
33.45
To evaluate the dependency of multimodal questions on images, and the impact of incorrect image associations, we conducted an experiment using the multimodal split of KALEIDOSCOPE. Follow- Enllotiitt 2018): Thomason et al. (2019), we created two modified versions of the dataset: (1) a 'No Image" spli, where all images were removed, and (2) 'Random Image" split, where images were randomly reassigned to questions. The aim of this experiment is to assess how much the models rely on the visual information. We evaluate the performance of Qwen2.5-VI--7B on these modified splits, and the results are shown in Table 3
We observe that the model performs above the random baseline (25%) across all three splits, indicating some ability to reason from text alone. However, there is drop in performance (-3.41 % in Total Accuracy) when questions are presented without images, suggesting that the model does rely on visual information for accurate answers. The performance drop is similar for both modifications; however, we observe a significantly larger format error when the model is tested with irrelevant images. In several of these cases, the model actually acknowledges that the image does not correspond to the
8

---

Published as a conference paper at ICLR 2026
question. In contrast, in experiments with no images, the format rate is almost zero, indicating that the model attempts to answer even when visual inputs are missing
5.2
SCALING MODEL SIZE IMPROVES PERFORMANCE
60
55
Doind
40
35
5won25 VL
Oven2.5VL70
Owen2.5 VL. 328
Owen25 728
Figure 4: Model Size Analysis for Qwen2.5-VL VL Models. Performance improvement across three model sizes (3B, 7B, 32B, and 72B parameters) KALEIDOSCOPE'S ' multimodal tasks, demonstrat- ing consistent gains from increased model capacity. Note that x-axis is shown log-scale.
To analyze the impact of model size KALFIDOSCOPE performance, we evaluated all four variants of Qwen2.5-VI. We selected this model family for its well-distributed size range, as well as being the best performing model in the open weight model category. We follow the same experimental setup for all model versions.
Figure 4 shows the performance of Qwen2.5-VL variants on KALEIDOSCOPE. Model size is shown in the x-axis (log-scale), while the y-axis displays accuracy for multimodal and text-only splits, and overall score. We observe linear relationship between the logarithm the model size and accuracy. with larger models showing significant gains. The largest open model model evaluated, Qwen2.5- VI.-2BB, still underperforms the closed models, however, these results highlight the effectiveness of scaling for open models, with clear and predictable improvements at each size tier.
5.3 FORMAT ERRORS
While our experimental setup ensures majority of answers were extracted from model outputs, we observe occasional failures: models struggle to follow instructions, outputs contain formatting errors, models refuse to answer (particularly for health-related or ethical questions). Appendix Figure shows that unanswered questions concentrate in mid- to low-resource languages, and the distribution accumulates
non-latin scripts, likely due to tokenization challenges, insufficient language-specific training data, or visual-textual alignment difficulties. Pangea-7B shows the highest refusal rates, especially for Telugu (452), Hindi (130), Persian (160), and Serbian (125). While other open models show minimal unanswered counts, indicating better format adherence. Closed models (Claude 3.5 Sonnet, GPT-4o) display distinct behavior: their refusals concentrate on non- Latin, low-resource languages, but they also show high error rates for English questions, primarily health/medical queries due to policy constraints. This underscores the trade-off between content moderation and benchmark performance.
RELATED WORK
While VLMs excel in multimodal tasks, existing benchmarks (Li ct al.
2024b Vayani al. 2024
Nayak et al. 2024 Schneider et al. (2225) predominantly evaluate high high-resource
languages (e.g.. En-
glish (Zang al. 2024 Schneider et al. (025)) Chinese (Fu al. 2023
He et al. 2024)): neglecting
linguistic diversity and cultural nuances (Hengle et al. [2024] Bird 2022). Translating benchmarks via
tools like ChatGPT (Lai al. 2023), GPT-4 Yue al. (2055) or Google Translate (Li 2023).
'We observed that Qwen2.5- tends to hallucinate when no image is present. In simple experiment using the prompt "bescribe be the follow ng image", the model correctly describes the input image when provided. However, when no image is passed, the model hallucinates and generates random description.

---

Published as a conference paper at ICLR 2026
often introduces errors and cultural mismatches (Singh al. 2024a Huang et al. 2025) Recent mul- tilingual benchmarks attempt to bridge this gap: MMLU-ProX (Xuan et al. 2025) covers reasoning in 13, CVQA (Romero et al. (2024) integrates cultural visuals across 31, and PangeaBench aggregates
47 languages Yue et al. 2025). However, cultural benchmarks like MaRVI. (binary evaluation) (Liu
eet al. ) and CULTURAI
(English-only open-ended questions) (Nayak et al. 2024) remain
limited in scope or format. Moreover, the MaXM benchmark (Changpinyo et al. (022) addresses
bias and provides multilingual, multimodal assessment across 7 languages but does not focus on
cultural aspects, an area where KALEIDOSCOPE offers added value. KALEIDOSCOPE advances these
efforts by combining regionally sourced multimodal exam questions with MCQA structure, enabling granular, culturally conscious evaluation across 18 languages. Exam-style benchmarks assess VLMs under structured multilingual settings. M3Exam (Zhang et al. 2023) uses real exams in 9 languages but only 23% image-dependent questions. EX EXAMS-V (Das et al. 2024) spans 11 languages with multimodal STEM content, yet 75% of its 20,932 questions are text-only. While M5 (Schneider & Sitaram 2024 evaluates 41 languages across vision- laiguale tasks, it avoids MCQA formats. KALEIDOSCOPE surpasses these by combining 18 languages, STEM & cultural coverage, and 55% image-dependent MCQA (Table C.2. offering comprehensive multilingual exam evaluation.
7 CONCLUSION
As generative models become increasingly multimodal and multilingual, the need for robust and culturally grounded evaluation benchmarks has never been more urgent. We take step toward closing this gap by introducing the largest benchmark of real-world, in-language multimodal exam questions. By grounding evaluation in authentic exam settings from around the world, our benchmark challenges models to reason about images in ways that mirror human assessment, capturing both linguistic and cultural complexity. Our findings highlight the limitations of current models in handling this intersection of skills: multilingual understanding, visual reasoning, and culturally aware problem- solving. We hope this benchmark serves not only as a valuable tool for measuring progress but also as call to action for developing models that are truly capable of operating across languages, cultures, and modalities. Continued investment in representative, high- high-quality evaluation datasets will be essential to ensure that future AI systems equitable and globally relevant.
8 ACKNOWLEDGMENTS
We acknowledge the EuroHPC Joint Undertaking for awarding this project access to the EuroHPC supercomputer LEONARDO, hosted by CINECA (Italy) and the LEONARDO consortium through an EuroHPC Development Access call (ID:EUHPC_DD) __2___112_11 This work was supported by research grant (VIL53 122) from Villum Fonden, and by the European Union's Horizon 2020 research and innovation program under grant agreement No. 101 135671 (TrustLLM). EF gratefully acknowledges the support of the Googler Initiated Grants and the Google Award for Inclusion Research programs. MZ. is supported by the research grant (VIL57392) from Villum Fonden and also received funding from the Danish Government to Danish Foundation Models (378-00001BB. JMI supported by the National University Philippines and the UKRI Centre for Doctoral Training in Accountable, Responsible, and Transparent AI [EP/S023437/1] of the University of Bath. MS is funded by the EU NextGeneraaionEUU through the Recovery and Resilience Plan for Slovakia under the project No.
REFERENCES
Aakanksha, Arash Ahmadian, Beyza Ermis, Seraphina Coldfarb-Tarrann Julia Kreutzer, Marzieh Fadace, and Sara Hooker. The multilingual alignment prism: Aligning global and local preferences to reduce harm. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 12027-12049 Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/ 4024.emnlp-main.677 URL httpt:tt//// /ttps://aate http//alaanntoooooooooooo org./gyyy.oooooo.. 2024. emn wammlppmmiin....m 6717
David Ifeoluwa Adelani, Marek Masiak, Israel Abebe Azime, Jesujoba Alabi, Atnafu Lambebo Tonja, Christine Mwase, Odunayo Ogundepo, Bonaventure F. P. Dossou, Akintunde Oladipo, Doreen
10

---

Published as a conference paper at ICLR 2026
Nixdorf, Chris Chinenye Emezue, Sana Al-azzawi, Blessing Sibanda, Davis David, Lolwethu Ndolela, Jonathan Mukibi,, Tunde Ajayi. Tatiana Moteu, Brian Odhiambo, Abraham Owodunni, Nnaemeka Obiefuna, Muhidin Mohamed, Shamsuddeen Hassan Muhammad, Teshome Mulugeta Ababu, Saheed Abdullahi Salahudeen, Mesay Gemeda Yigezu, Tajuddeen Gwadabe, Idris Ab- dulmumin, Mahlet Taye, Oluwabusayo Awoyomi, Iyanuoluwa Shode, Tolulope Adelani, Habiba Abdulganiyu, Abdul-Hakeem Omotayo, Adetola Adecko, Abeeb Afolabi, Anuoluwapo Aremu, Olanrewaju Samuel, Clemencia Siro, Wangari Kimotho, Onyekachi Ogbu, Chinedu Mbonu, Chiamaka Chukwuneke, Samuel Fanijo, Jessica Ojo, Oyinkansola Awosan, Tadesse Kebede, Toadoum Sari Sakayo, Pamela Nyatsine, Freedmore Sidume, Oreen Yousuf, Mardiyyah Oduwole, Kanda Tshinu, Ussen Kimanuka, Thina Diko, Siyanda Nxakama, Sinodos Nigusse, Abdulmejid Johar, Shafie Mohamed, Fuad Mire Hassan, Moges Ahmed Mehamed, Evrard Ngabire, Jules Jules, Ivan Ssenkungu, and Pontus Stenetorp. MasakhaNEWS: News topic classification for African languages. In Jong C, Park, Yuki Arase, Baotian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti, and Adila Alfa Krisnadhi (eds.), Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 144-159, Nusa Dua, Bali, November 2023, Association for Computational Linguistics. doi: 10.18653/v 1 122023.jjnnnlmmie. 10. URL https://aclanthon www.aannwwwoooooooooooeeoeeee /22//222332
wtt.://0./......pppppp.p iicnlp-nnpppe mapp--maih mamai.....
107
David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Zhuang Yun Jian, Jesujoba Oluwadara Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chia- maka Chukwuneke, Happy Buzaaba, Blessing K. Sibanda, Godson Kalipe, Jonathan Mukiibi, Salomon Kabongo KABENAMUALU, Foutse Yuchgoh, Mmasibidi Setaka, Lolwethu Ndolela, Nkiruka Bridget Odu, Rooweither Mabuya, Shamsuddeen Hassan Muhammad, Salomey Osei, Sokhar Samb, Tadesse Kebede Guge, and Pontus Stenetorp. Irokobench: A new benchmark for african languages in the age of large language models. ArXiv, abs/2406.033668, 2024. URL https: / www.steeanntccieee ar. wwwp/o//roooe... ID: ID:25258332
Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. Mega: Multilingual evaluation of generative ai, 2023. URL https: pettps://arxiv.00
abs/ 2003/230... 12528
Anthropic. The Claude 3 model family: Opus, sonnet, haiku, 2024. URL https: /aapi
semant 1 cschollllaa wwt./orrpssccss TD: (T:::22229999
Steven Bird. Local languages, third spaces, and other high-resource scenarios. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7817-7829, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 1815386//vll 1/208863////222.2all 1/2022.acl-long 539. URL httpp:://ccaattaatttt
org /ttpor///0000 2022 aac- 2ttp://2222..cloonng.. 539/
Yuri Bizzoni, Tom S Juzek, Cristina España-Bonet, Koel Dutta Chowdhury, Josef van Genabith, and Elke Teich. How human is machine translationese? comparing human and machine translations of text and speech. In Marcello Federico, Alex Waibel, Kevin Knight, Satoshi Nakamura, Her- mann Ney, Jan Niehues, Sebastian Stüker, Dekai Wu, Joseph Mariani, and Francois Yvon (eds.), Proceedings of the /7th International Conference on Spoken Language Translation, pp. 280-290, Online, July 2020. Association for Computational Linguistics. doi: 10. 111118833/12222..wwaaa 1.334. www.org//20000
URL https: http:///aaaaathhoooco
Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti, and Ivan Vulié. IGLUE: A benchmark for transfer learning across modalities, tasks, and languages. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. 2370-2392. PMLR, 17-23 Jul html
2022. URL JRLhttps: httttts::/proceeiiiiiee http///oooeeeeng..eooe .mw.inngssmlm..mmmmmm.mmm....mm.
Samuel Cahyawijaya, Holy Lovenia, Alham Fikri Aji. Genta Winata, Bryan Wilie, Fajri Koto, Rahmad Mahendra, Christian Wibisono, Ade Romadhony, Karissa Vincentio, Jennifer Santoso, David Moeljadi, Cahya Wirawan, Frederikus Hudi, Muhammad Satrio Wicaksono. Ivan Parmonangan, Ika Alfina, Ilham Firdausi Putra, Samsul Rahmadani, Yulianti Oenang, Ali Septiandri, James
11

---

Published as a conference paper at ICLR 2026
Jaya, Kaustubh Dhole, Arie Suryani, Rifki Afina Putri, Dan Su, Keith Stevens, Made Nindyatama Nityasya, Muhammad Adilazuarda, Ryan Hadiwijaya, Ryandito Diandaru, Tiezheng Yu, Vito Ghifari, Wenliang Dai, Yan Xu, Dyah Damapuspita, Haryo Wibowo, Cuk Tho, Ichwanul Karo Karo, Tirana Fatyanosa, Ziwei Ji, Graham Neubig, Timothy Baldwin, Sehastian Ruder, Pascale Fung, Herry Sujaini, Sakriani Sakti, and Ayu Purwarianti. NusaCrowd: Open source initiative for Indonesian NLP resources. In Anna Rogers, Jordan Boyd-Graber. and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 13745-13818 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 00.1866311120225 122..////./2222........-n-e-ee 868. URL httpt::t///// http:e///aaaaahhhhocoeo
org/ 2023. www.fiiinnnnaeeeeem acl 868/
Samuel Cahyawijaya, Holy Lovenia, Joel Ruben Antony Moniz, Tack Hwa Wong, Mohammad Rifqi Farhansyah, Thant Thiri Maung, Frederikus Hudi, David Anugraha, Muhammad Ravi Shulthan Habibi, Muhammad Reza Qorib, et al. Crowdsource, crawl, or generate? creating sea-vl, a multicultural vision-language dataset for southeast asia. arXiv preprint 2XXiv2553.009222 2025. URL https::///// http///axivvv// wttp:///xxivi..g/aaaa.. a253 07920
Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna. Language ID in the wild: Unexpected challenges on the path to a thousand-language web text corpus. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computational Linguistics, pp. 6588 6608. Barcelona, Spain (Online), December 2020. Interna- URL
tional Committee on Computational L Linguistics. doi: 10.18653/v wttpp///222cc/ we2000..ooingmmainn... wtg. wtwpor/g/00000.. cow.:/00000...ooooooooooo 1tt.-mmain5599/77
https:
José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, and Jorge Pérez. Spanish pre-trained bert model and evaluation data. In PML4DC at ICLR 2020, 2020.
Beer Changpinyo, I Linting Xue, Michal Yarom, Ashish Thapliyal, Idan Szpektor, Julien Amelot, Xi Chen, and Radu Soricut. Maxm: Towards multilingual visual question answering. In Findings of ACL: EMNLP, 2023. URL /ttttps://axxi....m arxiv. org/ abs 2209.222222 05401
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus. Yunxuan Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros. Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling instruction-fineture language models, 2022. URL https::/arxivv org. org/
awtpg////////// abs 2210 11416
Cohere-For-Al-Tem. Aya vision: Expanding the worlds ai can see, March 2025. URL https
///oherre www.:rre.com//ooggg www.oom//log/ee att///aaavvviioooo vision]
John Dang, Arash Ahmadian, Kelly Marchisio, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. RLHF can speak many languages:
Unlocking multilingual preference optimization for LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 13134-13156 Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 1/066///1/2244mmm/ppp 729. URL https/:///caaatttooococooo org/ 2024. emnlp main 729/
Rocktim Jyoti Das, Simeon Emiloy Hristov, Haonan Li, Dimitar Ilyannvv Dimitrov, Ivan Koychev, and Preslav Nakov. EXAMS-V: A multi-disciplin multilingual multimodal exam benchmark for evaluating vision language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers). pp. 7768 7791, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1 18653/v www.:222....11o..4441 URL https: / ac anthology. org/
4207]
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Moham- madreza Salehi, Niklas Muennighoff, Kyle Lo. Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison- Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg. Michael Schmitz, Aaron Sarnat, By- ron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon
12

---

Published as a conference paper at ICLR 2026
Borchardt, Dirk Groeneveld, Crystal Nam, Sophie Lebrecht, Caitlin Wittlif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannanch Hajishirzi, Ross Gir- shick, Ali Farhadi, and Aniruddha Kembhavi. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models. arXiv preprint arXiv:2409. 17146, 2024. URL htt.:s://arxxv..rr//a//////
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In CaaiicFranncne Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1286-1305, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 0118663/v1/222.emm///s maini.9. URL https: httts:://aclantholooy.. http://oooy.rgg/21oooooo
ma in..inn... 987]
Mengnan Du, Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu. Shortcut learning of large language models in natural language understanding. Commun. ACM, 6701) December 2023, ISSN 0001-0782. doi: 10.1 10/1545935,449 URL https:: /tt.://oooo/ooo// org/ http:///ooo..r11..11/1101 11tp://.11111/333311
Desmond Elliott. Adversarial evaluation of multimodal machine translation. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun' ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2974-2978, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/v 1329. URL atttp::/acclanie log.. 1329/
Desmond Elliott, Stella Frank, Khalil Sima' an, and Lucia Specia. Multi30K: Multilingual English- German image descriptions. In Anya Belz, Erkut Erdem, Krystian Mikolajezyk, and Katerina Pastra (eds.), Proceedings of the 5th Workshop on Vision and Language, pp. 70-74, Berlin, Germany, August 2016. Association for Computational Linguisties. doi: 10.18653/ 3210. URL https://aclanthologg org. WI6 32107
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi. Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. Beyond english-centric multilingual machine translation, 2020. URL
2010. 11125
Stella Frank, Desmond Elliott, and Lucia Specia. Assessing multilingual multimodal image descrip- tion: Studies of native speaker preferences and translator choices. Natural Language Engineering, 24(3)3393-411 413, 2018. doi: 10.1017/1135132991 100000777
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulci Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Pa- pangelis, Aman Madaan, Angelina Mcmillan-major, Anna Shvets, Ashish Upadhyay, Bernd Bohnet, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hen- drik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Laura Perez Beltrachini, Leonardo F. R. Ribeiro, Lewis Tunstall, Li Zhang, Mahim Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusck, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei, Samuel Cahyawijaya, Sanja Štajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Niko- laev, Vivian Tsai, Yacine Jernite, Ying Xu, Yisi Sang. Yixin Liu, and Yufang Hou. GEMv2: Multilingual NLG benchmarking in a single line of code. In Wanxiang Che and Ekaterina Shutova (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing: System Demonstrations, pp. 266-281, Abu Dhabi, UAE December 2022.
13

---

Published as a conference paper at ICLR 2026
Association for Computational Linguistics. doi: 10.18663311/22222mmmppppppp 2tte:/2222mmmmmddeeeoss.....
URL
Gregor Geigle, Florian Schneider, Carolin Holtermann, Chris Biemann, Radu Timofte, Anne Lauscher, and Goran Glavaš. Centurio: On drivers of multilingual ability of large vision-language model, 2025. URL https:: http:///aaxvvv.....aee/ /
/ 2501 05122
Gemini Team Google, Petko Georgiev, Ving Ian Lei. Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang. et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL pettts://aaxii//
0203.05533
Adriana Guevara-Rukoz, Isin Demirsahin, Fei He, Shan-Hui Cathy Chu, Supheakmungkol Sarin, Knot Pipatsrisawat, Alexander Gutkin, Alena Butryna, and Oddur Kjartansson. Crowdsoureing Latin American Spanish for low-resource text-to-speech, In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. Marseille, France, May 2020. European Language Resources Association. ISBN 979- 17-9-10-9556- 34-4. URL
1rec- 8.001/
Zheqi He, Xinya Wu, Pengfei Zhou, Richeng Xuan, Guang Liu, Xi Yang, Qiannan Zhu, and Hua Huang. Cmmu: A benchmark for chinese multi-modal multi-type question understanding and reasoning. arXiv preprint arXiv:2401.14011 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Confer- ence on Learning Representations, 2021. URL https: http:::///peneeviewww net forumiidmmm
d7KB jmi 3GmQ.
Amey Hengle, Prasoon Bajpai, Soham Dan, and Tanmoy Chakraborty. Multilingual needle in a haystack: Investigating long-context behavior of multilingual large language models, 2024. URL
wwtp:///////////aa 1015
Carlos Daniel Hernandez Mena and Ivan Vladimir Meza Ruiz. Creating Mexican Spanish language resources through the social service program. In Chris Callison-Burch Christopher Cieri, James Fiumara, and Mark Liberman (eds.), Proceedings of the 2nd Workshop on Novel Incentives in Data Collection from People: models. implementations, challenges and results within LREC 2022, pp. 20-24, Marseille, France, June 2022. European Language Resources Association. URL https: / http////clattthooooo wwg.ooo...oooo.oooo..o.......o.. www.oorgg222..o.. n dep- .47
Kaiyu Huang. Fengran Mo, Xinyu Zhang, Hongliang Li, You Li, Yuanchi Zhang, Weijian Yi, Yulong Mao, Jinchen Liu, Yuzhuang Xu, Jinan Xu, Jian- Yun Nie, and Yang Liu. survey on large language models with multilingualism: Recent advances and new frontiers, 2025. URL
https::/ /ttp///aavvvvv... http://aawxi..oo////////ee///// wttp:br////aaaoeeooo
10936
Pratik Joshi, Christain Barnes, Sebastin Santy, Simran Khanuja, Sanket Shah, Anirudh Srinivasan, Satwik Bhattamishra, Sunayana Sitaram, Monojit Choudhury, and Kalika Bali. Unsung challenges of building and deploying language technologies for low resource language communities. In
Dipti Misra Sharma and Pushpak Bhattacharya (eds.), Proceedings of the 16th International Conference on Natural Language Processing, pp. 211-219, International Institute of Information
Technology, Hyderabad, India, December 2019. NLP Association of India. URL https://
aclanthology wrw./01gg31000 1on.///0.....oo111 1. 257
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the NLP world. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6282 6293, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v 1/0p///20022/0m2 acl- 2012200cl-maii.566 URL https: ac http///aaannttoo....oooo
202 100.aclmma 5600
14

---

Published as a conference paper at ICLR 2026
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richard Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arna Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Openassistant conversations democratizing large language model alignment. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 47669-47681 I, Curran Associates, Inc., 2023, httpss//pppoeeed.nn..eeeeppspppp paper_ /ttpep/eee...eeepppp/////////// 049f0 34000042222443133444444430033333 pdf
URL
and Band___echaarss.....
Moshe Koppel and Noam Ordan. Translationese and its dialects. In Dekang Lin, Yuji Mat- sumoto, and Rada Mihalcea (eds.), Proceedings of the 49th Annual Meeting of the Associ- ation for Computational Linguistics: Human Language Technologies, pp. 1318-1326, Port- land, Oregon, USA, June 2011. Association for Computational Linguistics. URL https //aclantho oogy.
wrg/p1l www.llogoyoggglll.ooooo
1132/
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-rrhikkh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Pa- padimitriou, Salomey Osei, Pedro Ortiz. Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Balh, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the As- sociation for Computational Linguistics, 10:50-72, 2022. doi: 10. 1..11622taclaaa00444 URL
Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. Okapi: Instruction-tuned large language models in multiple languages with reinforce- ment learning from human feedback. In Yansong Feng and Els Lefever (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demon- strations, pp. 3 318-227 Singapore, December 2023. Association for Computational Linguis- tics. doi: 18.18653/v 1/2023 1/2023.emmll-demmee. mee..--d.oo............ URL
wwwpoog..org//// org/2023
emm 1o-noo 28/
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dior Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz. Suarez, Aaron Gokaslan, Shamik Bose, David Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilic, Margaret Mitchell, Sasha Alexandra Luccioni, and Yacine Jernite. The bigscience roots corpus: A 1.6tb composite multilingual dataset. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 31809 318260 Curran Associates, Inc., 2022. URL Utt.:tpperr/ppoccecin....ee.ee..
f22p:777774422222222272223333320 and www.BBenhhhass.e.ee pdd
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13299- 13308, 2024a. doi: 10.1109/ CVPR52733.20D4401263
15

---

Published as a conference paper at ICLR 2026
Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. Bactrian-x: Mul- tilingual replicable instruction -oolloitoonfollooinng models with low-rank adaptation. arXiv preprint arXiv:2305.15011 2023.
Jian Li, Weiheng Lu, Hao Fei, Meng Luo, Ming Dai, Min Xia, Yizhang Jin, Zhenye Gan, Ding Qi. Chaoyou Fu, et al. A survey on benchmarks of multimodal large language models. arXiv preprint arXiv:2408. 08632, 2024b.
Wenyan Li, Crystina Zhang, Jiaang Li, Qiwei Peng, Raphael Tang, Li Zhou, Weijia Zhang, Guimin Hu, Yifei Yuan, Anders Søgaard, Daniel Hershcovich, and Desmond Elliott. FoodieQA: A multimodal dataset for fine-grained understanding of Chinese food culture. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. I 10077-19095, Miami, Florida, USA, November 2024c. Association for Computational Linguisties. doi: 10.18653/v 1/0.2//.........mmaaaiii 1063. URL
ht https/////aaaaahhoooogooo http emn ma in..nnn.... 1063/
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
URL https: ac Taa www.aaannnttn.ooooo org/ W04 1013/
Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. Visually grounded reasoning across languages and cultures. In Marie-Francine Moens, Xu- anjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10467-10485, Online and Punta Cana, Do- minican Republic, November 2021. Association for Computational Linguisties. doi: 10. 10 4021.emnlp-main.3 URL http:ttttcc/caatttette httphhhhwoooooo..//2.2eo2 wwtp///g....... emnlp main. 818/
Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marmu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, Xinyi Wu, Enrico Shippole, Kurt Bollacker, Tongshuang Wu, Luis Villa, Alex Pentland, and Sara Hooker. large- scale audit of dataset licensing and attribution in ai. Nature Machine Intelligence, 6:975-987, 08 2024. doi: 10. 00.1038/s42256-22 000--22255-----055-- 024-00878-8.
Shayne Longpre, Nikhil Singh, Manuel Cherep, Kushagra Tiwary, Joanna Materzynska, William Brannon, Robert Mahari, Naana Obeng-Marnu, Manan Dey, Mohammed Hamdy, Nayan Saxena, Ahmad Mustafa Anis, Emad A. Alghamdi, Vu Minh Chien, Da Yin, Kun Qian, Yizhi Li, Minnie Liang, An Dinh, Shrestha Mohanty, Deividas Mataciunas, Tobin South, Jianguo Zhang, Ariel N. Lee, Campbell S. Lund, Christopher Klamm, Damien Sileo, Diganta Misra, Enrico Shippole, Kevin Klyman, Lester JV Miranda, Niklas Muennighoff, Seonghyeon Ye, Seungone Kim, Vipul Gupta, Vivek Sharma, Xuhui Zhou, Caiming Xiong, I Luis Villa, Stella Biderman, Alex Pentland, Sara Hooker, and Jad Kabbara. Bridging the data provenance gap across text, speech and video, 2025. URL https: / arxiv...v........ wwg. abs 22122412...0 17847
Holy Lovenia, Rahmad Mahendra, Salsabil Maulana Akbar, Lester James Validad Miranda, Jennifer Santoso, Elyanah Aco, Akhdan Fadhilah, Jonibek Mansurov, Joseph Marvin Imperial, Onno P. Kampman, Joel Ruben Antony Moniz, Muhammad Ravi Shulthan Habibi, Frederikus Hudi, Jann Railey Montalan, Ryan Ignatius Hadiwijaya, Joanito Agili Lopo. William Nixon, Börje F. Karlsson, James Jaya, Ryandito Diandaru, Yuze Gao, Patrick Amadeus Irawan, Bin Wang, Jan Christian Blaise Cruz, Chenxi Whitehouse, Ivan Halim Parmonangan, Maria Khelli, Wenyu Zhang. Lucky Susanto, Reynard Adha Ryanda, Sonny Lazuardi Hermawan, Dan John Muhammad Dehan Al Kautsar, Willy Fitra Hendria, Yasmin Moslem, Noah Flynn, Muham- mad Farid Adilazuarda. Haochen Li. Johanes Lee, R. Damanhuri, Shuo Sun, Muhammad Reza Qorib, Amirbek Djanibekov. Wei Qi Leong, Quyet V. Do. Niklas Muennighoff, Tanrada Pansuwan, Ilham Firdausi Putra, Yan Xu, Tai Ngee Chia, Ayu Purwarianti, Sebastian Ruder, William Chan- dra Tjhi, Peerat Limkonchotiwat, Alham Fikri Aji, Sedrick Keh, Genta Indra Winata, Ruochen Zhang, Fajri Koto, Zheng Xin Yong, and Samuel Cahyawijaya. SEACrowd: A multilingual multimodal data hub and benchmark suite for Southeast Asian languages. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 5155-50 52033 Miami, Florida, USA. November 2024. Association for Computational Linguistics. doi: 11118665//1/222..mm// caammp-mmin.22. 296. URL https: //acl antho og..ooy......................oo. orwp.yorgg22242 emn p n. 2967
Velasco,
16

---

Published as a conference paper at ICLR 2026
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang, Song-Chum Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal rea- soning via thought chains for science question answering. hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp- 2507-2521. Curran Associates, Inc., files/pans /pt.e//pppe//022//
In S. Koyejo, S. Mo-
2022. URL
neurips http:/ssseeeirrre.eeppppppoaeeea File Lttee//332250011100aaaaaaaaaaaa3 Paaaaaaaa-aPperrree Conference pdf
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
Alexandra Luccioni and Joseph Viviano. What's s in the box? an analysis of undesirable content in the Common Crawl corpus. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 182-189, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021 2021.acl-short..0 URL bettts:://cllatthooooyy.o 2021, aclilacllleee acw-short....... 24/
Li Lucy, Suchin Gururangan, Luca Soldaini, Emma Strubell, David Bamman, Lauren Klein, and Jesse Dodge. AboutMe: Using self seff-descriptions in webpages to document the effects of English pretraining data filters. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.
Bangkok. Thailand, August 2024. Association for Computational Lin- 1/2p3///1/222.cl--11a
guistics. doi: 4007|
URL https: tettts://aclantholoooo org 2024
Muhammad Maaz, Hanoona Abdul Rasheed, Abdelrahman M. Shaker, Salman H. Khan, Hisham Cholakkal, Rao Muhammad Anwer, Timothy Baldwin, Michael Felsberg, and Fahad Shahbaz Khan. Palo: A polyglot large multimodal model for 5b people. ArXiv, abs/2402.14818, 2024. URL TD 277776778277
httpss:/epp semant i csscool ar
Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd Van Steenkiste, Lisa Anne Hendricks, Karolina Stanczak, and Aishwarya Agrawal. Benehmarking vision language models for cultural understanding. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 5769-5790, Miami, Florida, USA, November 2024. Association for Computational Linguisties. doi: 10.18653/v I/ www.emmmwllmmmnnn.... URL httt::://aclaathoooosgo org/ wwt//////////// 2024. emnlp main. 329/
Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Taiwo Fagbohunghe, Solomon Oluwole Akinola, Shamsuddeen Muhammad, Salomon Kabongo Kabenamualu, Sa- lomey Osei, Freshia Sackey, Rubungo Andre Niyongabo, Ricky Macharm, Perez Ogayo, Ore- vaoghene Ahia, Musie Meressa Berhe, Mofetoluwa Adeyemi, Masabata Mokgesi-Selingee Lawrence Okegbemi, Laura Martinus, Kolawole Tajudeen, Kevin Degila, Kelechi Ogueji, Kath- leen Siminyu, Julia Kreutzer, Jason Webster, Jamiil Toure Ali, Jade Abbott, Iroro Orife, Ig- natius Ezeani, Idris Abdulkadir Dangana, Herman Kamper, Hady Elsahar, Goodness Duru, Ghol- lah Kioko, Murhabazi Espoir, Elan van Biljon, Daniel Whitenack, Christopher Onyefuluchi, Chris Chinenye Emezue, Bonaventure F. P. Dossou, Blessing Sibanda, Blessing Basscy, Ay- odele Olabiyi, Arshath Ramkilowan, Alp Öktem, Adewale Akinfaderin, and Abdallah Bashir. Participatory research for low-resourced machine translation: A case study in African lan- guages. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2144-2160, Online, November 2020. Asso- ciation for Computational Linguistics. doi: 10.18653//120020.fmmm http:///22...iniinnemmm.m..p org/2//.rgg2000 wtw.:::....nniin.seeeme.. 195/
URL
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine. Gabriel Bermadett-Shaping Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Camphell, Andrew Cann, Brittany Carey, Chelsea
17

---

Published as a conference paper at ICLR 2026
Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu. Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Auty Bleti, Tyna Floundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, tukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjkk Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil. David Medina, Aalok Mehta, Jacob Menick, Luke Metz. Andrey Mishchenko. Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichir Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong. Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond. Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder. Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song. Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL attps::/aaxxin
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, pp. 311-318, USA, 2002. Association for Computational Linguistics, doi: 153315//073388110333333 URL httpps:///o...oo.. 1ttp:///oo..r0111..11..111111110 1073135
Wannaphong Phatthiyaphaibun, Surapon Nonesung, Patomporn Payoungkhamdee, Peerat Limkon- chotiwat, Can Udomcharoenchaikit. Jitkapat Sawatphol, Chompakorn Chaksangchaichot, Ekapol Chuangsuwanich, and Sarana Nutanong. Wangchanlion and wangchanx mrc eval, 2024. URL
https://arxiv..r org/ wwtpg///////// /t//a//s//33333 16127
Luiza Pozzobon, Patrick Lewis, Sara Hooker, and Beyza Ermis. From one to many: Expanding the scope of toxicity mitigation in language models, 2024. URL hww.hhtta.//aaii..rg//////
2403. 03893
Chen Qiu, Dan Oneatä, Emanuele Bugliarello, Stella Frank, and Desmond Elliott. Multilingual multimodal learning with machine translated text. In Yoav Goldberg, Zoritsaa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp.
18

---

Published as a conference paper at ICLR 2026
4178-4193, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: http:///202..innnnnnnnm......... URL htttp::/
htc//2022..indingss-oee 3308/]
Qwen-Team. Qwen2.5-v1, January 2025. URL
qwen2
5-v1/
Rita Ramos, Emanuele Bugliarello, Bruno Martins, and Desmond Elliott. PAELLA: Parameter- efficient lightweight language-agnostic captioning model. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 3549-3564, Mexico City, Mexico, June 2024. Association for Computational Linguistics. URL httpps:/acclatthool.g...
doi: 10. 10.18653//1/20224.1 hinere4fiinning--aaala Findings naacl.
Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shiv- alika Singh, Rishabh Maheshwary, Micol Altomare, Mohamed A. Haggag, Snegha A, Alfonso Amayuelas, Azril Hafizi Amirudin, Viraat Aryabumi, Danylo Boiko, Michael Chang, Jenny Chim, Gal Cohen, Aditya Kumar Dalmia, Abraham Diress, Sharad Duwal, Daniil Dzenhaliou, Daniel Fernando Erazo Florez, Fabian Farestam, Joseph Marvin Imperial, Shayekh Bin Islam, Perttu Isotalo. Maral Jabbarishiviari, Börje F. Karlsson, Eldar Khalilov, Christopher Klamm, Fajri Koto, Dominik Krzeminski, Gabriel Adriano de Melo, Syrielle Montariol, Yiyang Nan, Joel Niklaus, Jekaterina Novikova, Johan Samir Obando Ceron, Debjit Paul, Esther Ploeger, Jebish Purbey, Swati Rajwal, Selvan Sunitha Ravi, Sara Rydell, Roshan Santhosh, Drishti Sharma, Marjana Prifti Skenduli, Arshia Soltani Moakhar, Bardia Soltani Moakhar, Ran Tamir, Ayush Kumar Tarun, Azmine Toushik Wasi, Thenuka Ovin Weerasinghe, Serhan Yilmaz, Mike Zhang, Imanol Schlag, Marzich Fadace, Sara Hooker, and Antoine Bosselut. Include: Evaluating multilingual lan- guage understanding with regional knowledge. arXiv preprint arXiv:2411 19799, 2024. URL
httts::/arxiv..ort org/ abs http////////... 221..
David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, Bontu Fufa Balcha, Chenxi Whitehouse, Christian Salamea, Dan John Velasco, David Ifeoluwa Adelani, David Le Meur, Emilio Villa-Cueva, Fajri Koto, Fauzan Farooqui, Frederico Belcavello, Gan- zorig Batnasan, Gisela Vallejo, Grainne Caulfield, Guido Ivetta, Haiyue Song. Henok Biadglign Ademtew, Hernán Maina, Holy Lovenia, Israel Abebe Azime, Jan Christian Blaise Cruz, Jay Gala, Jiahui Geng, German Ortiz-Barajas, Jinheon Back, Jocelyn Dunstan, Laura Alonso Alemany, Kumaranage Ravindu Yasas Nagasinghe, Luciana Benotti, Luis Fernando D' D'Haro, Marcelo Virid- iano, Marcos Garittaiii,,,iiiiii.,, Maria Camila Buitrago Cabrera, Mario Rodríguez-Cantee Cantell,,,, Mélanie Jouitteau, Mihail Mihaylov, Mohamed Fazli Mohamed Imam, Muhammad Farid Adi- lazuarda, Munkhjargal Gochoo, Munkh-Erdenne Otgonbold, Naome Etori, Olivier Niyomugisha, Paula Mónica Silva, Pranjal Chitale, Raj Dabre, Rendi Chevi, Ruochen Zhang, Ryandito Dian- daru, Samuel Cahyawijaya, Santiago Góngora, Soycong Jcong, Sukannya Purkayastha, Tatsuki Kuribayashi, Thanmay Jayakumar, Tiago Timponi Torrent, Togeer Ehsan, Vladimir Araujo, Yova Kementchedjhieva, Zara Burzo, Zheng Wei Lim, Zheng Xin Yong, Oana Ignat, Joan Nwatu, Rada Mihalcea, Thamar Solorio, and Alham Fikri Aji. Cvqa: Culturally-divesee multilingual visual question answering benchmark. NeurIPS 2024, 2024.
Florian Schneider and Sunayana Sitaram. M5 - diverse benchmark to assess the performance of large multimodal models across multilingual and multicultural vision-language tasks. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 4309-4345, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/11/2224 000.6665/w/22244mmmmmmmmmml//// finding-eemnlp.2500 URL wwcpy.orgg/e. wttp////22..22.. www.ffiinnnneennee 1p 1p.250/
Florian Schneider. Carolin Holtermann, Chris Biemann, and Anne Lauscher. Gimmick - -lomilly inclusive multimodal multitask cultural knowledge benchmarking, 2025. URL BtRLhttps: //arxiv
org/ 2502. 13766)
Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwatt Kelly Marchisio. Wei Qi Leong, Yosephine Susanto, et al. Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evalua- tion. arXiv preprint arXiv:24/2.03304 033044 2024a.
19

---

Published as a conference paper at ICLR 2026
Shivalika Singh, Freddie Vargus, Daniel D' souza, Börje Karlsson, Abinaya Mahendiran, Wei-Yin Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O' O'Mahony, Mike Zhang. Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Moura, Dominik Krzeminiski, Hakimeh Fadaei, Irem Ergun, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Chien, Sehastian Ruder, Surya Guthikonda, Emad Alghamdi, Sehastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzich Fadace, and Sara Hooker. Aya dataset: An open-access collection for multilingual instruction tuning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp- 11521-11566 Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10. 18153//1/2024..clllaae URL httphttts//accaatttite
www.oo02....loooog...... 6207
Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I. Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng. Shayne Longpre, Wei-Yin Ko, Sebastian Ruder, Madeline Smith, Antoine Bosselut, Alice Oh, Andre F. T. Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee, Beyza Ermis, and Sara Hooker. Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation, 2025. URL CRLhttps.. https: /
/ttp/aaw.vv//.////a//..t/...av wtwggabs////oooooog /412 03304
Ray Smith. An overview of the Tesseract OCR engine. In Proc. Ninth Int. Conference on Document Analysis and Recognition (ICDAR), pp. 029-633, 2007.
Ashish V. Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut. Crossmodal-3600 A massively multilingual multimodal evaluation dataset. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 715-729, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.1106.11.11 18653/v 0e/55/3/1/2222.eemm h/tp:/2emm2m2mmaaaaaa 2022. ep
URL https: htat:s://aclanttollgggoo /rg/
457
Jesse Thomason, Daniel Gordon, and Yonatan Bisk. Shifting the baseline: Single modality perfor- mance on visual navigation & QA. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pp. 1977-1983, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653/v1/N19-111 URL http:::/acccccccc./ antho liroologg logy. ogg.oor//K19-1197
Ahmet Üstün, Viraat Aryabumi, Zheng Yong. Wei-Yin Ko, Daniel D' souza, Ghemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. Aya model: An instruction finetuned open-access multilingual language model. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15894-15939, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 12211666//1/22222cclloooog URL httpt::: / antho wnthaaooyooooooooooooooooo www.oorgg/2022222 2024 acc. long. 8857
Özlem Uzuner, Imre Solti, Fei Xia, and Eithon Cadag. Community annotation experiment for ground truth generation for the i2b2 medication challenge. Journal of the American Medical Informatics Association, 17(5):519-522) 09 2010. ISSN 1067-5027. doi: 00.1136/jamaa2211.00000000 URL
htttp::/doii http://///ii.0gg11.11111003
amia. a. 2010. C04200
Emiel van Miltenburg, Desmond Elliott, and Piek Vossen. Cross-linguistic differences and similarities in image descriptions. In Jose M. Alonso, Alberto Bugarín, and Ehud Reiter (eds.), Proceedings of the 10th International Conference on Natural Language Generation, pp. 21 21-30, Santiago de Compostela, Spain, September 2017. Association for Computational Linguistics. doi: 10.18653/ 35033/
v1/W17-3503. htt.attpss//aclaatoooooo.g.o
Eva Vanmassenhove, Dimitar Shterionov, and Matthew Gwilliam. Machine translationese: Effects of algorithmic bias on linguistic complexity in machine translation. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.). Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2203-2213 Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v 1/2021 Lecll-eeeeeeeeee maied.mainn..a. 188. URL https
7/ ho ogy.org//003 eacl -ma n. 1887
20

---

Published as a conference paper at ICLR 2026
Ashmal Vayani, Dinura Dissanayake, Hasindri Watawana, Noor Ahsan, Nevasini Sasikumar, Omkar Thawakar, Henok Biadglign Ademtew, Yahya Hmaiti, Amandeep Kumar, Kartik Kuckreja, Mykola Maslych, Wafa Al Ghallabi, Mihail Mihaylov, Chao Qin, Abdelrahman M. Shaker, Mike Zhang, Mahardika Krisna Ihsani, Amiel Esplana, Monil Gokani, Shachar Mirkin, Harsh Singh, Ashay Srivastava, Endre Hamerlik, Fathinah Asma Izzati, Fadillah Adamsyah Maani, Sehastian Cavada, Jenny Chim, Rohit Gupta, Sanjay Manjunath, Kamila Zhumakhanova, Feno Herini- aina Rabevohitra, Azril Hafizi Amirudin, Muhammad Ridzuan, Daniya Najha Abdul Kareem, Ketan More, Kunyang Li, Pramesh Shakya, Muhammad Saad, Amirpouya Ghasemaghaei, Amir- bek Djanibekov, Dilshod Azizov, Branislava Jankovic, Naman Bhatia, Alvaro Cabrera, Johan Obando-Ceron, Olympiah Otieno, Fabian Farestam, Muztoba Rabbani, Sanoojan Baliah, San- tosh Sanjeev, Abduragim Shtanchaev, Maheen Fatima, Thao Nguyen, Amrin Kareem, Toluwani Aremu, Nathan Xavier, Amit Bhatkal, Hawau Olamide Toyin, Aman Chadha, Hisham Cholakkal, Rao Muhammad Anwer, Michael Felsberg, Jorma Laaksonen, Thamar Solorio, Monojit Choud- hury, Ivan Laptev, Mubarak Shah, Salman Khan, and Fahad Shahbaz Khan. All languages matter: Evaluating Imms on culturally diverse 100 languages. ArXiv, 1..//2411.165066 2024. URL https: /app..pp.../// semant weschhcsshhoo sesscholar..oo wwt. Corpus/ruuuuuucc ID: 22::::::4444554
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-4575, 2015.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset, 2024a.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max KU, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 95266-95290. Curran Associates, Inc., 2024b. cc/ pctpppppp.ee..... paper_ files http://pppppeeee/eeapee
URL sttps://procceddisss.eeee
ad2 3raeeee6665eee1eeeeeeeeeeeaaaaaa Paper-aaa-ttt-----ttttee. track.pdi
Datasets and Benchmarks
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fci Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 97817138711 13871088.
Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng. Siyuan Huang, Yu Qiao, and Ping Luo. Lylm-ehub: A comprehensive evaluation benchmark for large vision-language models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(3): 1877-1893, 2025. doi: 10. 00.100/TPAM1..204.35000000
Weihao Xuan, Rui Yang. Heli Qi. Qingcheng Zeng, Yunze Xiao, Yun Xing, Junjue Wang, Huitao Li, Xin Li, Kunyu Yu, et al. Mmlu-prox: A multilingual benchmark for advanced large language model evaluation. arXiv preprint arXiv:2503. 10497, 2025.
Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun. Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9556-9567, 2024a. doi: 101109/CVVPS2733.22240009
Xiang Yue, Tianyu Zheng, Yuansheng Ni. Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. Mmmu-pm pro: A more robust multi-discipline multimodal understanding benchmark, 2024b. URL. https
//aaxiv..oo wtwpiiiv.rraabb//a/aooaooo 2409 02813
Xiang Yue, Yueqi Song. Akari Asai, Seungone Kim, Jean de Dieu Nyandwi, Simran Khanuja, Anjali Kantharuban, Lintang Sutawika, Sathyanarayanan Ramamoorthy, and Graham Neubig. Pangea: A
21

---

Published as a conference paper at ICLR 2026
fully open multilingual multimodal Ilm for 39 languages, 2025. URL httta://arxxx..orgg
abs/ 0bs/24110.25 22112.411..55
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Y. Zou. When and why vision-language models behave like bags-of-words, and what to do about it? ArXiv, abs/2210.01936, 2022. URL
semanticsche ar. Carg/Corpus.i. ID;
252734947
Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, and Chen Change Loy. Contextual object detection with multimodal large language models. International Journal of Computer Vision, pp. 1-19, 2024.
Mike Zhang and Antonio Toral. The effect of translationese in machine translation test sets. In Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Marco Turchi, and Karin Verspoor (eds.), Proceedings of the Fourth Conference on Machine Translation (Volume I: Research Papers), pp. 73-81, Florence, Italy, August 2019, Association for Computational Linguistics. doi: 10.18653/ v1/W19-5208, URL https: wttps///aaaaathooo.....oooo org 9 5208/
Wenxuan Zhang, Sharifah Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models. In Advances Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, 2023. URLhttps: //axiiv org/abs 2306. 05179
Mingyang Zhou, Luowei Zhou, Shuohang Wang, Yu Cheng, Linjie Li, Zhou Yu, and Jingjing Liu. Uc2: Universal cross-lingual cross-modal vision-and lan-uagg pre-training. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4155-4165, June 2021.
22

---

Published as a conference paper at ICLR 2026
APPENDIX
APPENDIX CONTENTS
Limitations
24
Data Collection Details
24
License
24
B.2 Difficulty Levels
24
Dataset Statisties
25
Expanded Related Work
26
Participatory Open Science Projects
26
Comparison with Other Benchmarks
27
Evaluation Metrics and the MCQA Framework
27
D
Experiment Details & Additional Experiments
28
D.I
Evaluation Metries
28
D.2
Prompt Ablation: COT Direct Approaches
................................
28
D.3
Complete Results
29
D.4
Format Errors
................................
34
D.5
Prompts
34
D.5.1
System Message
34
................................
D.5.2
Keywords
35
D.5.3
Prompt Examples
35
D.6 To What Extent Do Textual Augmentations Boost VLM Capabilities
38
D.6.1
Captioning & OCR
39
Cross-Benchmark Performance
Dataset Samples
41
Categories of Visual Elements
Selected Dataset Samples
41
Dataset Fields
23

---

Published as a conference paper at ICLR 2026
A
LIMITATIONS
While our benchmark represents an important step toward more representative multilingual multi- modal evaluations, several limitations still remain. First, the dataset is inherently imbalanced across languages. Coverage varies depending on the availability and accessibility of cxam sources, with some languages significantly underrepresented. Second, difficulty levels not uniformly controlled. Since questions are drawn directly from real-world exams across diverse educational systems, variations in exam design, curricular focus, and intended grade levels introduce potential inconsistency in task complexity across languages and modalities. Further the chosen MCQA question format, inherent to many exams, has issues, see Appendix C.3 For instance: Exploitation of biases: Models may guess correct answers by exploiting statistical patterns or poorly designed distractors, inflating performance metrics without demonstrating genuine understanding. Limited real-world applicability: Unlike open-ended queries typical in real-world applications, MCQA provides predefined options, which may not reflect natural user interactions. Choice-order sensitivity: Performance can vary based on the order of answer choices, introducing inconsistencies unrelated to model capability, Finally, while the dataset expands coverage beyond English, the overall language diversity remains limited. Many languages, especially those spoken in low-resource regions, still missing due to the scarcity of suitable exam material and annotators.
Intended Use. KALEIDOSCOPE is designed as an evaluation-only benchmark for assessing multi- lingual and multimodal reasoning under exam-style, multiple-choice conditions. Appropriate uses include: diagnosing modality gaps between image-text and text-only settings; analyzing model behavior across languages, scripts, subjects, and image types: and studying cross-lingual or culturally grounded biases within a controlled MCQA format. KALEIDOSCOPE is not intended for evaluating free-form generation, long- congecontxtt reasoning, conversational or interactive capabilities, or open- apen-ended problem solving. It is also not designed for model training or fine-tuning.
B
DATA COLLECTION DETAILS
B.1 LICENSE
To ensure ethical data usage, we prioritize sources that permit redistribution and academic use. During data collection, we filter out content from sources with restrictive licensing policies. Additionally. our dataset does not include personally identifiable information, and all collected exams are either publicly available or obtained under appropriate agreements. To further guarantee compliance, we employ a two-stage validation process in which two blinded annotators independently verify the license of each exam included in our dataset. Only items that pass both validations are included in the final dataset.
B.2 DIFFICULTY LEVELS
To better contextualize cross-language performance variability, we introduce unified four-tier difficulty taxonomy (Basic, Intermediate, Advanced, Expert) derived from the original heterogeneous level metadata. To construct the mapping, we sampled 30 examples for each "level" category and determined the typical age and educational stage at which humans would encounter similar material. This served as a proxy for aligning the original categories to our four standardized bins.
Table reports the resulting difficulty distribution for each language in KALEIDOSCOPE, Fach cell lists both the raw count of questions and the corresponding percentage of that language's 's total. While the difficulty profiles vary across languages, reflecting differences the types of publicly available exams, the overall benchmark is centered around Intermediate and Advanced material, corresponding broadly to high-school and undergraduate-leve levell content.
We provide these difficulty assignments as additional metadata while preserving the original level tags, allowing the community refine reinterpret the taxonomy according to their research needs. For full transparency, we include the final mapping used to convert the original level categories into four-tier difficulty scheme in code.
24

---

Published as a conference paper at ICLR 2026
Table 4: Difficulty distribution across languages in KALEIDOSCOPE, Each cell shows percentages with raw counts in grey.
Language
Basic
Intermediate
Advanced
Expert
Total
Arabic
0.0%
0
100.0%
382
0.0%
0.0%
0
382
Bengali
0.0%
0
72.6%
18.0%
144
9.4%
800
German
0.0%
0
0.0%
100.0%
722
0.0%
722
English
0.0%
65.4%
1065
17.0%
277
17.6%
286
1628
Spanish
0.0%
0
58.9%
873
3.8%
56
37.3%
553
1482
Persian
0.0%
0
76.3%
1526
23.7%
474
0.0%
0
2000
French
0.0%
0
100.0%
762
0.0%
0.0%
0
762
Hindi
13.1%
248
28.6%
540
21.2%
399
37.1%
699
1886
Croatian
0.0%
0
100.0%
324
0.0%
0.0%
324
Hungarian
0.0%
100.0%
0.0%
0.0%
1120
Lithuanian
0.0%
100.0%
0.0%
Nepali
0.0%
0
100.0%
126
0.0%
0.0%
126
Dutch
0.0%
0
24.2%
246
75.8%
772
0.0%
1018
Portuguese
0.0%
0.0%
100.0%
2000
0,0%
0
2000
Russian
0.0%
100.0%
1744
0.0%
0.0%
0
1744
Serbian
0.0%
100.0%
2000
0.0%
0.0%
0
2000
Telugu
0.0%
0
0.0%
100.0%
1000
0.0%
1000
Ukrainian
35.3%
437
64.7%
1237
All languages
3.3%
685
62.8%
12,869
39.5%
8,094
9.4%
1,929
20,911
B.3 DATASET STATISTICS
Table 5: Statistics of the KALEIDOSCOPE Dataset. Breakdown of subjects (Subjects), total questions (Total), multimodal questions (Visual), and text-only questions (Text) per language. Languages covered by multiple sources with single-subject cases containing specialized subdomains. Supports evaluation of both multimodal (image+text) and unimodal (text-only) capabilities. Languages are classified by resource level (high/mid/low) following Joshi et al. (2019); Singh et al. (2024b). Enables granular analysis of model performance across modalities, languages, and subject domains.
Language
Code
Subjects
Total
Visual
Text
Resources
Family
Portuguese
pt
11
2000
1000
1000
High
Italic
Serbian
2000
1000
1000
High
Balto-Slavic
Persian
Hindi fa
5
2000
1000
1000
High
Tranian
12
1886
1000
886
High
Indo-Aryan
Russian
1744
872
872
High
Balto-Slavic
English en
9
1628
814
814
High
Germanic
Spanish es
1482
741
741
High
Italic
Hungarian hu
1120
560
560
High
Uralic
Dutch nl
10
1018
509
509
High
Germanic
French
762
381
381
High
Italic
German de
722
361
361
High
Germanic
Arabic
10
382
191
191
High
Semitic
Croatian hr
324
162
162
High
Balto-Slavic
Ukrainian uk
1237
1000
237
Mid
Balto-Slavic
Bengali bn
800
400
400
Mid
Indo-Aryan
Lithuanian It
680
340
340
Mid
Balto-S -lato-SSavic
Telugu tc
1000
1000
0
Low
South Dravidian
Nepali ne
126
126
Low
Indo-Aryan
Total (18) 14
20,911
11,457
9,454
25

---

Published as a conference paper at ICLR 2026
C EXPANDED RELATED WORK
C.I PARTICIPATORY OPEN SCIENCE PROJECTS
Participatory research empowers diverse communities to actively contribute to research processes, capturing linguistic subtleties and cultural nuances directly from native speakers. Prior participatory NLP research has primarily targeted region-specific tasks such as translation, character recognition, and audio transcription. We highlight notable initiatives here which served as motivation and backbone framework for building KALEIDOSCOPE.
In Africa, the Masakhane community exemplifies impactful participatory NLP by focusing on grassroots-led data collection, annotation, and model creation for African languages, Nekoto et al.) (2020) demonstrated that communities low-resource environments significantly contribute NLP, even without formal training. Subsequent efforts by Adelani et al. (2023) have further advanced dataset curation and model development for underrepresented African languages using similar participatory frameworks. Similarly, the MaRVL dataset (Multicultural Reasoning over Vision and Language; Liu et al., 2021) employed native speakers from diverse linguistic backgrounds (Indonesian, Swahili, Tamil, Turkish, and Mandarin Chinese) to contribute culturally representative images, subsequently annotated by professional linguists. Despite its cultural richness, MaRVI.'s modest scale (under 8,000 data points) limits broader applicability beyond evaluation.
In Latin America, participatory research has also emerged and is continuously growing through the help of communities. Recent works include Hernandez Mena & Meza Ruiz (2022), which developed eight open-access linguistic resources via structured social service programs, engaging student volunteers in transcription and segmentation tasks. Concurrently, Cañete et al. (2020) and Buevvra-Rukoo et al. (2020) spearheaded crowd-sourced corpora addressing dialectal diversity and resource scarcity specific to Latin American Spanish.
Table 6: Comparison of Multimodal Benchmarks. fall but 40 questions are in English that measure machine translation capability from Chinese English.
Benchmark
Languages
Samples
Multimodal
Modalities
Human notation
An-
Answer type
MMMU
Yue et
11,550
11,264
Image-Text
Yes
MCQA
2024a
SEED Bench Li et al.
19,242
19,242
Image-Text, Video-Text
Partial
MCQA
20240
MME Fu cl
2023
11
2,194
Image-Toxt
Partial
Y/N
M3Exam (Zhang et al.
9
12,317
2,816
Image Text
Yes
MCQA
2023
EXAMS-V Das et
20,932
5,086
Image-Text
Yes
MCQA
2024
M5
Schneider& & Sitaram
41
237,094
1,422
Image-Text
Yes
Mix
2024
KALEIDOSCOPE
18
20,911
11,459
Image-Toxt
Yes
MCQA
In Southeast Asia, Project SEALD collaboration between AI Singapore and Google Research, facilitated multilingual dataset collection to support regional Large Language Models (LLMs). Outputs from SEALD underpin open-source multilingual models such as SEA- Wangchan- Lion (Phatthyyaphibun et al.) 2024). and Sahabat-Al A Related initiatives include NusaCrowe for aggregating and standardizing Indonesian NLP datasets (Cahyawijaya et al.) (023) and the SEACrowd and SEA-VL projects aimed at comprehensive evaluation and benchmarking of LLMs across Southeast Asian languages (Cahyawijaya et al.) 2025 Lovenia et al. 2024).
a global scale, the CVQA dataset (Romero et al. 2024) was created using a participatory approach, involving native speakers and cultural experts from over 30 countries. Annotators were
io/
Southeast Asian Languages in One Network Data; http://ppe//aassggpppreer///apce
Sotthest-ssii--llggggs------
https://sea-lion.a
https://sahabat-ai.c
26

---

Published as a conference paper at ICLR 2026
selected for their fluency in local languages and cultural familiarity, Many contributors were also recognized as co-authors based on their level of involvement, reinforcing a collaborative, community- driven effort. The Aya Initiative employed participatory methods, engaging over 3,000 contributors to curate instruction datasets across 14 languages, resulting in one of the largest multilingual datasets for language model training (Singh et al. 2024b; Ustün al. 2024).. Similarly, the INCLUDE benchmark (Romanou et al. 2024) leveraged participatory approaches closely aligned with our methodology. The BigScience ROOTS corpus, developed collaboratively for the BLOOM model, exemplifies large-scale participatory data collection. Approximately 62% of ROOTS data was crowd- sourced via global hackathons and open submissions, involving over 1,000 researchers from 60 countries and more than 250 institutions, resulting in 1.6 terabytes of multilingual data (Laurençon) eet al. 2022). Additionally, Uzuner et al. (20T0) underscored the viability of community-die annotation for complex, domain-specifie NIP tasks like clinical text annotation, highlighting broader applicability of participatory frameworks beyond general NLP domains.
Participatory methods have also successfully extended into reinforcement learning from human feed- back (RLHF). For instance, the OpenAssistan project, led by LAION, utilized global crowdsourcing to construct multilingual corpus comprising over 161,000 messages annotated by 13,500 volunteers. This dataset facilitated robust training of dialogue-aligned language models through extensive human feedback annotations (Köpf et al.) 2023).
C.2 COMPARISON WITH OTHER BENCHMARKS
Table 6 offers a concise comparison of key multimodal benchmarks. MMMU (Yue et al. 2024a), SEED-Bench (Li et al.) 2024a), and MME (Fu et al. (023) are single-language datasets focused mainly on image-text pairs, with SEED-Bench also incorporating video-text. MME notably smaller and only partially human-annotated using mostly true/false formats. In contrast, M3Exam (Zang et al. 2023). EXAMS-V (Das et al. 2024). and M5 (Schneider & Sitaram 2024) introduce multilingualism M5 being the most extensive with 41 languages though much of its content is not multiple-choice and lacks verified annotations.
KALEIDOSCOPE stands out by offering balanced composition of 20,911 samples across 18 lan- guages, with a strong focus on multimodal reasoning (11,459 Image-Text samples), comprehensive human annotation, and consistent multiple-choice setup. Compared existing benchmarks, K KAI.EI- DOSCOPE is more linguistically diverse than M3Exam and EXAMS- V, includes more multimodal samples than M5, and ensures higher quality through expert-verified annotations, making a robust and equitable benchmark for evaluating multilingual multimodal models.
C.3 EVALUATION METRICS AND THE MCQA FRAMEWORK
Traditional evaluation metrics for VLMs, such as exact match accuracy, BLEU (Papineni et al. 2002). ROUGE (Lin) 2004), and CIDEr (Vedantam et al. 2015), rely on surface-level n-gram comparisons that often penalize semantically equivalent answers phrased differently from reference texts. In contrast, the multiple-choice question answering (MCQA) framework (Hendrycks et al.] 2021 Romero al. 2024 Lu al. [022 Yue et al. 2024a) offers a more human-like evaluation paradigm by providing predefined answer options. This reduces ambiguity in scoring and facilitates the creation of evaluation datasets that capture both domain knowledge and linguistccculttral nuances acToss languages. Although concerns regarding oversaturation and reliance on superficial cues in MCQA exist (Du et al., 2023) Yuksekgonul et al., 2022), these can be mitigated by extending the answer option space and applying rigorous filtering strategies (Wang et al., 2024b Yue et al. (024a). Our primary challenge lies in the scarcity of questions that are both multimodal and culturally agnostic. As demonstrated by results from KALEIDOSCOPE and related studies (Maaz et al.] 2024 Nayak
et al. 2024). oversaturation is not a prevalent issue. Consequently, bridging this evaluation gap is of key importance. To ensure high data quality, source data in KALEIDOSCOPE are manually verified by qualified processors in accordance with established criteria (.2), maintaining a clear distinction between verified and unverified data.
27

---

Published as a conference paper at ICLR 2026
D EXPERIMENT DETAILS & ADDITIONAL EXPERIMENTS
D.1 EVALUATION METRICS
Given the multiple-choice nature of the task, we use accuracy as the primary evaluation metric. We report overall accuracy across all questions, as well as accuracy on the subset of questions where the model produces valid responses. A response is considered valid if the model successfully provides an answer the expected format and selects a valid option (i.e., one of the letters A, B, C, D). Invalid responses typically result from missing the selected choice, selecting invalid option, refusal to answer. To quantify these cases, we report the Format Error Rate, which measures the proportion of questions for which the model fails to generate valid answer. For grouped results, report the macro average of valid answer accuracy across languages, i.e. all languages have equal weight when computing the score.
D.2 PROMPT ABLATION: COT VS. DIRECT APPROACHES
To benchmark the models, initially designed CoT prompt instructing them think step-by-step and then provide the correct answer, marking the choice with the tags <ANSWER> </ANSWER>, However, preliminary experiments, we found this instruction complex for mid- small-sized models (32B-30) which struggled to follow it consistently.
In Table we compare results using the CoT prompt versus the direct English-language prompt adopted in our final evaluation. The error rate was considerably higher for most models under the CoT setup, even after cleaning and extracting answers with regex matching their typical output formats. Two exceptions were Pangea and Molmo, which showed lower error rates with the CoT prompt; however, this occurred because both models ignored the reasoning instruction and directly output the selected option, making extraction easier. Overall, prompt choice significantly impacted performance: the direct English prompt improved results across all models except Pangea, whose performance remained unchanged.
For closed models, we also compare the performance of CoT versus the direct prompt on GPT-4o in Table We observe interesting trade-off: while CoT improves accuracy, it also increases the format егror rate. As detailed in Section these errors are tightly related with implicit refusals, especially in questions involving medical safety-relevant content. We hypothesize that CoT gives the model more room to reason toward the correct answer but also increases the likelihood of refusals, thereby raising the format-error rate.
Based on these findings, we select the best-performing prompting strategy for each class of models: the direct prompt for open-weight models and CoT for closed-weight models.
Table 7: Comparison of CoT and direct English prompting on KALEiDSSCOPE for small models. Reported values are macro-averaged accuracy (%) across all languages.
Overall CoT
Overall In In-English
Valid Responses
Valid Responses
Model
Acc.
F.F.
Valid Acc.
Acc.
F.E.
Valid Acc.
Aya-Vision iaa-Vision-320
38.94
8.04
42.06
39.27
1.05
39.66
Aya- Vision-888
33.08
6.22
35.15
35.09
0.07
35.11
Molmo-7B-D
32.86
0.01
32.87
32.87
0.04
32.88
Pangca-7B
31.24
5.61
33.45
31.31
7.42
34.02
Qwen2.5-Ve 55-VI
35.18
6.34
37.64
39.56
0.08
39.60
Dwen2.3-VV--3.
32.90
T.40
33.33
35.56
-.19
35.63
28

---

Published as a conference paper at ICLR 2026
Table Comparison of different prompting strategies on GPT-4o. Results are shown disaggre- gated by language. Global accuracy, valid answer accuracy and format error rate are reported.
Direct Prompt
CoT Prompt
Language
Total Acc.
Valid Ace.
F.E.
Total Acc.
Valid Ace.
F.E.
Arabic
49.7
50.4
1.3
52.9
57.7
8.4
Bengali
57.4
57.5
0.2
65.6
67.3
2.5
Croatian
33.3
33.8
1.2
49.7
52.6
5.6
Dutch
57.1
58.1
1.8
58.9
62.4
5.6
English
63.9
64.0
0.1
60.8
73.4
17.1
French
37.7
37.7
0.1
61.8
64.6
4.3
German
70.8
70.8
0.0
71.6
72.6
1.4
Hindi
48.6
48.7
0.2
60.1
64.0
6.1
Hungarian
34.0
34.4
1.1
47.2
50.6
6.7
Lithuanian
83.4
83.4
0.0
86.5
88.4
2.2
Nepali
23.8
24.0
0.8
19.0
20.5
7.1
Persian
40.2
40.3
0.3
47.0
47.9
2.0
Portuguese
75.4
75.4
0.0
82.6
85.2
3.0
Russian
36.6
36.6
0.1
51.5
54.5
5.5
Serbian
33.1
33.4
1.0
47.0
52.6
10.6
Spanish
75,4
75.5
0.2
77.7
80.1
3.0
Telugu
42.1
44.3
4.9
41.6
47.9
13.2
Ukrainian
71.4
72.2
1.1
68.1
75.3
9.5
Overall
51.9
52.2
0.7
58.3
62.1
6.5
D.3 COMPLETE RESULTS
We report full results on multimodal performances grouped by subject in table Tables and full results for all questions grouped by subject, Table TO and language, Table Each table reports, for each model and category; Total Accuracy %: the accuracy over all samples, Valid Accuracy %: the accuracy over successfully extracted answers and Format Error % (FE): the proportion of unextracted answers.
Table 9: Subject- wise Performance KALEIDOSCOPE'S Multimodal Questions. Valid accuracy (%) across examination subjects for only multimodal samples, with bold highlighting top-performing models.
Closed
Open Weights
Gemini
Clande
GPT-40
Qwen2.5-38
Molmo-78
Pangen-7B
Qwen2.5-7B
Apa
Aya
Qwen2,5-72B
Humonities Social Sciences
Economics
Gengmihy
72.8
81.5
50.4
40.7
37.6
36.7
51.0
39.5
50.5
70.4
History
78.7
83.7
$6.4
48.9
42.1
42.4
52.9
45.6
61.4
77.1
Language
85.8
72.2
Social Sciences
85.7
82.9
88.1
$2.9
52.2
53.8
64.3
80.0
Sociology
92.3
93.4
93.2
64.1
61.0
57.3
73.1
57.7
70.5
87.2
Biology
62.9
63.9
37.6
35.3
33.4
42.6
35.4
40.7
$3.8
Chemistry
60.4
59.7
52.9
33.2
33.5
34.1
38.5
25.0
34.8
50.0
Engineering
57.3
64.4
56.3
28.9
24.4
24.2
32.4
30.3
34.8
48.4
Mathematics
44.4
44.0
30.4
28.8
29.0
29.
Physics
57.8
58.7
54.7
33.7
26.7
28.9
34.7
27.1
33.0
42.3
Reasoning, Healtt Selence, and Practical Skills
Reasoning
52.0
53.3
$1,0
27.4
27.5
26.6
29.5
25.1
27.6
42.3
Medicine
70.2
75.8
75.6
36.7
40.4
38.4
45.8
35.4
52.3
63.3
Driving License
64.4
64.2
73,1
39.0
44.9
39.4
44.9
41.6
47.1
54.5
29

---

Published as a conference paper at ICLR 2026
Table 10: Total Accuracy %, Valid Accuracy % and Format Error % (FE) grouped by Subject in KALEIDOSCOPE for multimodal samples.
Dehing
Yeglswring
Giepuphy
Langsage
Plydes
Reswning
liral Acc.
AIL
00.2
61.4
66.1
5711
72.8
76.2
815
40.9
69.5
51.4
51.3
39.2
V0.33
Genini 1.5
Vakd Acc.
80L3
00.4
60.6
66.1
57.5
72.8
76.2
K13
-8.6
7112
51.6
52.0
X5.7
90.3
0.3
00
0.
0.8
Inaal Ave.
$3.6
59.01
66.2
63.4
5001
81.4
$3.2
86.1
43.7
12.9
55.4
52.2
$2.0
91.0
Chande Sannet
Vali Acc.
62.9
99.7
64.2
63.6
64.4
81.5
$3.3
85.5
44.4
73.8
56.7
53.3
$2.9
95.4
0.5
0.)
1/2
Irral
66.1
45.2
16.2
1063
66.6
51.8
65.0
24.5
K7.2
GPT-4
Vakd Nox.
63.9
$2.0
73.1
06.7
56.3
80.4
$5.4
855
440
75.6
54.7
51.0
48.1
95.2
5.1
3.8
Total Ave.
37.6
39.2
50.0
50.4
28.8
40.6
43.5
72.2
30.1
36.1
33.0
27.4
52.9
$4.1
Qvent Qven25-VL-722 S-VB-22
Yali Aox.
$1.6
33.2
30.0
$2.7
28.9
40.7
48.9
72.2
30.4
36.7
35.7
20.4
$3.9
64.1
Trail Avo.
40.2
34.6
47.1
26.5
38.6
50.5
63.4
71.0
28.8
53.1
31.8
27.5
64.3
70.5
Ape 338
Vald
40.7
$4.8
29.8
$1.8
90.5
61.4
71.
20.6
50.3
33.0
27.6
64.3
70.5
Traal Ava
53.8
50.0
56.5
58.5
48.4
70.4
17.1
84.0
40.3
63.3
42.3
42.3
9000
57.2
Apw-Vadan-0n
Vald
51.5
90.
7114
15.1
-0.5
61.3
43.9
12.3
3511
87.3
Trail Aco.
15.2
33.5
66.9
27.5
24.8
37.6
43.1
00.1
28.7
00.4
26.7
27.5
51.4
50.3
Malnu- TB--n
Vold
15.3
13.5
60.9
27.5
37.6
01.
58
J0.8
76.7
27.5
53.3
61.0
Traal Arc.
2008
22.1
16.2
32.1
21.4
35.6
41.1
65.
35.0
36.7
21.9
5001
55.1
Panges-11
Vald
3141
56.7
6..0
26.5
513
573
Tmal Acc.
34.3
16.2
41.2
22.1
3033
36.7
45.3
50.6
25.0
35.4
25.1
26.3
371
593
Qwe-25-NL-77
Vald
25.1
3600
513
Irral Arc.
41A
15.5
66.9
02.7
12.4
51.0
31.8
05.7
30.1
65.8
36.7
20.5
056
15.1
Ow1125-11.318
Yaki
41.
66.9
31.4
21.0
293
7311
6.0
00
Table 11: Total Accuracy %, Valid Accuracy % and Format Error % (FE) grouped by Language in KALEIDOSCOPE for multimodal samples.
Tatks Serige
Nevvl Scipe
Fagits
Frevis
Centee
Dual
Paragis
Spundd
Teagi
Criitis
Bias
Skegrie
Neuil
Panias
Ranla
Seitian
Telagi
Red
52.6
41.5
$1.8
36.5
44.
51.3
46.5
36.1
75.0
53.3
412
45.
41.5
20.3
Gesbal 1.5
Vald
23.6
02.5
85.4
447
79.0
43.1
653
43.6
70.3
1.8
2.5
5.4
Clasdo Suumet
Vald Ace
00.2
$1.7
133
62.6
83.8
75.3
933
-96
47.5
533
36.9
814
36.3
45.6
45.4
35.6
960
75.3
09
0.2
1/2
9.8
Woll Aeo
45.2
52.1
6611
765
75.4
61.4
607
96.8
516
76.3
1443
41.
17.6
53.4
$1.5
06.5
6.1
255
100
3.4
157
0.8
909
16.1
10.
310
6.7
16.2
132
11A
Tell Amo
$3.5
65.3
143
51.
25.3
20.5
151
09.5
$1.
6633
$3.5
33.8
15.4
14
16.9
$7.0
653
Ph
100
0.5
0.0
012
0/3
6.0
09
00
612
3.5
00
0.6
Apa-Vaine-323
Veld Ace.
20.0
29.3
31.3
48.7
30.5
56.5
313
26,1
21.6
333
28.5
48.3
12.2
31.1
15.4
28.3
54.5
4111
13.3
Vexd
12.4
42.2
41A
46.5
212
22.9
2000
544
21.6
36,5
Tell Adc
25.9
33.0
18.8
41.
476
42.5
31.5
300
300
35.0
35.5
36.3
35.4
28.3
38.3
21.3
33.0
35.1
37:
0.2
056
2.0
6/
0/3
0.8
8.0
0/
0.
00
60
3.2
6m
0.0
0m
0.6
Red
35.5
11.2
55.3
46.5
46.3
33.4
27.8
11.9
33.6
21.2
33.0
Paaquu78
31.3
19.6
300
51.
40.4
31.8
21.0
33.4
38.8
35.9
16.0
13.4
Que025-VE-7B
Vald Ace
30.
35.
25.6
464
503
57.
361
35.1
26.1
339
35.6
203
15.
30.3
19.3
21.5
17.6
454
Toal Aoo
418
13.
156
34:0
11.
11.
811
Oww.3A-VL.0
16.4
05
60
07
19
5.
06
30

---

Published as a conference paper at ICLR 2026
Table 12: Valid Accuracy (%) for All Questions and All Models, by Language and General Category. I Languages are grouped by script. Empty cells indicate either no questions for that category, or only format errors in the answers from the model.
Nestathrerppl
Growd Cangers
Raghak Fooc Connas
Rasigaan
Apeers
Audl:: Repil Crattss
Hegses
Taborios
Sepd
Pesia
Race
Solaas
Ceme bo
2,502
1,00
0.551
1/B5
Heab Sciaco
8.30
Agp Maiao
0333
3.55
1/85
Paggre
1.00
1880
134
1/00
Ohend 5-10
2410
0663
545
0000
335
628
62
0.713
5352
0694
3561
133
1/00
000
1971
1021
11333
596
Millen
0.77
035
0339
0.70
540
8516
0333
DASI
Geaad 1.5k
1311
1553
083
New Vieae 139
0.8
Mileo-30-3 20-0
0.85
030
0310
E50
1350
0223
640
D455
Claal 5.5 Sueeet
5335
135
Reawiig kg
1100
App-MR60x-223
0.244
0333
Agp
Rolme-711 70.0
0399
135
Quedd 70
0.31
0.00
0.0
130
Cheake Saeee
5.55
0.57
133
0.3
0.76
0.5
3.0
0.56
2.50
85
0.5
035
071
STEM
Chicl 54726
0341
0.67
0.13
130
0027
0.50
0375
0487
6400
636
630
0427
0433
5470
0-30
155
110
5-18
0.68
0.53
0334
0330
0345
0331
6311
6382
5376
0,43
0.23
Gendei L.30v
136
Sectil Scieece
Qneed 5476
0333
1114
35%
067
075
0.01
0.133
1350
0682
835
0354
050
037
0330
5/111
Qvaed 5-10
0:34
198
(161
110
31

---

Published as a conference paper at ICLR 2026
Table 13: Valid Accuracy (%) for Multimodal Questions for All Models, by Language and General Category. Languages are grouped by script. Empty cells indicate either no questions for that category, or only format errors the answers from the model.
NestathSerppl
Greed Canggrs
Regiok Foed Coenas
Rasigatn
Apeiis
Audl:: Repii
Crates Hal
Hegses
Sepa
Pesia
Selaas
2,55
1,00
8:15
100
Heab Sciaco
8.38
Agp Maiao
580
Paggre
1.53
1880
Ohend 5-10
1250
1200
8.76
140
191
1065
135
ES2
L1K0
1750
0902
150
EBC
DA3)
1851
1171
Millen
0644
035
0779
3.067
5374
8516
DAS)
Geaad 1,5k
0.55
661
122
1013
New Vieae 139
3:214
1311
D16
Mileo-30.3 20-0
0/774
8.7
025
0463
0510
D4L3
01774
1001
0307
840
Claal 555 Seeeet
(1323
1331
0001
025
Reawiig kg
App-M660-223 223
0.244
0.24
0.178
Nolme-78. 70.0
0331
0355
Quedd 70
0.31
0233
0.55
130
"126""
Cheake
0.013
0357
0333
1.6
0.34
0.55
0.001
340
0425
245
6350
0315
0.60
0.11
6366
0.5
639
STEM
Chicl 54726
0.006
0146
0463
148
1350
3.50
0425
0330
63%
635
0350
025
E40
0196
0351
690
1156
510
020
5-78
0.313
0.56
8517
0.55
0353
6386
632
0235
5376
0.21
035
0251
137
010
Sectil Scieece
0552
1307
1114
2114
0/64
1.55
030
6190
0.133
1357
0333
035
0337
0301
0330
0134
196
0664
155
61K6
32

---

Published as a conference paper at ICLR 2026
Table 14: Valid Accuracy (%) for All Models, by Language and Multimodal Content Type. Languages are grouped by script. Empty cells indicate either no questions for that category, or only format errors in the answers from the model.
Scrtet
Nan-laata Scriet
Gearal Ganagry
Model
Freeh
Cortale
Darih
Paragaal
Swieik
Bregit
Cratle
Red:
Tkapliay
Kypal
Fesia
Raaie: Sefkie Tales
Gerad
01235
2,531
01,01
0.39
5.33
0331
D483
3.20
0:50
0,21
0.90
D:35
11:99
3,500
11/55
(031
1.815
00333
01754
1336
08483
1
11561
00354
2.48)
16415
1481
1851
8315
6541
135
0330
1331
0.53
0455
1301
11:88
1184
3111
814
1990
0118
0331
1224
DA81
0033
5370
0.33
D440
Overd 2-30
03384
0.39
0314
0231
12255
1997
125
555
1120
11983
11513
1967
0191
E114
0136
01411
01415
3.55
1350
0930
13.00
11.51
11.66
11.00
51R
Figa
Queed
1667
0376
0.335
0.583
0550
0333
0.555
5.33
8560
0.36
0.511
0.35
03343
0.53
0351
6.24
0367
3.07
0.35
0338
0333
0351
E415
5350
0.00
0287
0229
3.71
0277
0.34
0127
0.56
0270
01333
C391
0.00
1721
01514
0577
0265
0700
Diegh
Qvee 5-71
(181
1313
0111
159
11881
175
0535
Pragee 70
0.335
0460
1351
630
235
0156
630
Qvesd 2-08
0432
1250
0330
0354
0370
6310
1161
1066
1052
1.00
0131
050)
03505
1663
0339
0333
5230
0536
D455
Mi Mi.bo-7BBB D
0.000
03235
000
0133
0160
5230
6357
11.00
11544
0339
0335
5230
845
Cliat Soiee
0011
5351
0167
100
1800
Quees 5-10
1080
1875
1945
1100
1230
0.0
1985
1431
1100
0360
0396
6033
0.00
E457
Ovedd
(3N
1021
1001
1057
100
6311
0450
1607
75p
0.55
23%
01:3
1667
0336
0131
02344
0356
E335
0333
Ovend
(1406
01344
13511
5305
0136
0166
Graad Fro
0391
0/25
C55
0.35
Chual Spanet
01511
05511
E314
01211
01123
03599
14411
1447
636
3330
Millee
0335
0.315
0.33
1590
000
0356
5361
Queil 19
0435
(150)
0.35
066
0.37
Claak 55 Some
1160
1000
0507
136
610
100
Queed
1.00
1000
0344
138
6400
110)
5230
0333
Nop Miced6 68
0/00
1000
0636
106
000
0100
8530
Drages 711
(100
1960
0376
1314
1200
5230
0100
1160
0107
119
0100
110)
50W
33

---

Published as a conference paper at ICLR 2026
Table 15: Format error disaggregated analysis. For answers which were identified Format Error, we report how many correspond to answer refusal (R) vs another type of format errors (NR). Empty N-NR refusal pairs indicate that the model did not have Format Errors on that language. Note that for most models in all languages, format errors are not due to answer refusals but to other reasons like inability to follow MCQ tagging instructions. The only exception is Pangea, which seems to exhibit higher refusal rate.
Model +7
Apu Mb
Aya 32b
Clande 3.5 Somnet
Geninl 1.5 pro
GPT-4b
Molno
Pangeo
QuenZ. 76
QwenZS Sh
Qwenz.5 32b
Qwen22 73b
Langaage
Aroic
13
Crotiaa
Doic; Flcnidh
399
275
Freuch
14
33
42
10
Corman
10
Hinli
24
0
18
130
127
56
0
20
0
Husgeias
14
50
11
64
23
26
0
18
19
0
Liheunia
12
Nquaii
27
Penilan
31
38
55
116
44
Pheleguenc
70
80
24
10
0
36
13
2
84
96
07
38
Setbian
15
205
74
51
11
Spusies
31
25
Tolage
00
126
300
145
Ukrainan
73
D.4
FORMAT ERRORS
We report the distribution of format errors for all evaluated models Figure 5 Table 1. provides
finer- finer-grained breakdown, showing which format errors correspond to refusals versus missing answers.
Distribution of Unanswered Questions Across Languages and Models
Gemini Pro
Claude 3.5 Sonnet
340
3
1
0
0
2
18
2
GPT-4o
279
33
10
S7
60
45
32
20
115
132
39
96
117
18
212
75
15
400
Unarswered Questions
Owen2.5 VL--228
0
0
300
Ayo Aya-Vsion-328 328
11
23
19
29
31
3
31
13
O
14
Aya Vision-8 38
3
0
2
0
0
1
2
2
200
Qwen225-1L78
100
Molmo
2
2
0
0
0
0
0
0
0
0
Pangea-78
94
68
44
22
43
48
51
183
452
34
160
105
26
24
125
49
Qwe3255-L-36
0
fr nl
es
bn
hr
hu
Figure 5: Distribution of the number of format errors for each model/language combination. The languages are represented in their ISO 639 (set 1) code.
D.5 PROMPTS
The prompts that we used perform all experiments were designed ensure consistency across languages. Examples are shown both in English and Spanish as an overview. Below is summary of the key components.
D.5.1 SYSTEM MESSAGE
A system message sets the context for the model, instructing it act an expert in solving multiple- choice questions. For zero-shot CoT prompting, the message is provided in all the evaluation languages support language-specific evaluation.
34

---

Published as a conference paper at ICLR 2026
Zero-shot CoT:
English:
You are an expert at solving multiple-choice questions. Carefully analyze the question, think step by step, and provide your FINAL answer between the tags <ANSWER> </ANSWER>, where X is ONLY the correct choice. Do not write any additional text between the tags.
Spanish: Eres un experto resolver preguntas de opción múltiple. Analiza cuidadosamente la pregunta, piensa paso a paso preporciona tu respuesta FINAL entre las etiquetas <ANSWER> X </ANSWER>, donde X es ÜNICAMENTE la opción correcta. No escribas ningún texto adicional entre las etiquetas.
Direct answer:
You are helpful assistant who answers miltiple-choice
questions. For each question, output your final answer in JSON format with the following structure: ("choice";
"The correct option (e.g., A, B, C, or D)") ONLY output this format exactly, Do not include any additional text or explanations outside the JSON structure.
D.5.2 KEYWORDS
Language-spern speecic keywords are used to structure the prompts consistently across languages. These include terms for "Question," "Options," and "Answer" to be included when generating the prompt. For example:
English: ("question";
"Question", "options": "Options",
"answer":
"Answer")
Spanish:
"Pregunta", "options": "Opciones", "Respuesta")
( ["question";
""nswer":
D.5.3 PROMPT EXAMPLES
System messages D and Keywords D.5.2 are used to systematically craft the prompt for a model in specific language. We show examples of both closed and open model in Table 16
35

---

Published as a conference paper at ICLR 2026
Table 16: Prompt examples in KALEIDOSCOPE. Multimodal prompt samples with interleaved
image are shown for an open model and closed model.
Open Model
Closed Model
SYSTEM:
Eres un experto en resolver preguntas de opción miltiple. Analiza cuidadosanente la pregunta, piensa paso paso y proporciona respuesta PINAL entre las etiquetas <ANSWER> X </ANSWER>, donde x es ÚNICAMENTE la opción correcta. No escribas ningún texto adicional entre las etiquetas.
SYSTEM:
You are helpful assistant who answers multiple-choice questions. For each question, output your final answer in USON format with the following structure: "choice": "The correct option (e.g., A, B, C, or D) ". ONLY output this format exactly. Do not include any additional text or explanations outside the JSON structure. Output your choice the specified JSON format.
USER:
USER:
hamsii
sikk
Boader
Brnc of or qeen the gpci
wile
aabilisk mallootbaie dlvne
Question: Make CORRECT match between Group-I and Group-II, in relation to
Pregunta: Ante esta imagen en un paciente con un trastorno motor
interaction between two species.
Options:
en miembros inferiores, señale la respuesta INCORRECTA: Opciones:
A.) P-I, Q-II, R-III, S-IV
B.) P-IIT, Q-IT, R-TV, S-I
C.) P-IV, QIIII, R-II, S-I
A.) Debemos buscar una
D.). P-III, Q-IV, R-II, S-I
malformación de Chiari
Answer:
B.) En algunos casos se asocia hidrocefalia
C.) Se caracteriza por una pérdida de la sensibilidad táctil vibratoria con preservación de la sensaci ón térmica y dolorosa
D.). Puede producirse tras traunatismos infecciones
Respuesta:
36

---

Published as a conference paper at ICLR 2026
Table 17:
Caption+OCR prompt examples KALEIDOSCOPE. Prompts are shown for open and closed models in English and Spanish. Caption and OCR additions are highlighted in green.
Open Model
Closed Model
SYSTEM:
You are a helpful assistant who answers multiple-choice questions. For each question, output your final answer in USON format with the following structure: "choice"; "The correct option (e.g., A, B, C, or D) ". ONLY output this fornat exactly. Do not include any additioral text or explanations outside the JSON structuro.
SYSTEM:
Eres un experto resolver preguntas de opción miltiple. Analiza cuidadosamente pregunta, piensa paso a paso y proporciona tu respuesta FINAL entre las etiquetas <ANSWER> X </ANSWER>, donde X es UNICAMENTR la opción correcta. No escribas ningún texto adicional entre las etiquetas.
USER:
kinoludesattto.he lnt mainltos chaz (
dhat
USER:
char 1ely
Mos
Peso total
Chep 1et7
prlrnnerin to Bntnts du A3V
1
1,500 gramos
gramos
3 37000 gramos
cnooding edoorrt charneess giver belore
Caption: This table presents data total weight, measured in grams, across four months. The table consists
two columns: Mes (Month) and Peso (Ttal We Weight). Month shows weight of 500 grams, Month 2 shows 600 grams, Month shows 3, grams, and Month shows 4, 800 grams. The table is a simple grid format with plain black text white background.
Caption: The code initializes character variables 'a" to 'p' and "b' to 'x', It then calculates 'c', 'd', and fer using bitwise operations (1a, ") and character addition with ''', '-', and 't', respectively. The 'printf' function (...) Ellipses (...) within the tables indicate emitted values between the shown characters. OCR: #Fincludecstdio.hh// Annn main arge, \n \nchar - "Prr b - 'x'; Anchar (a d'nchaar d (a Anchar e (a \u 201a|n \mprintÉ (N"sc \8lnreturn 0,An An MAn Anchar *argv(]) MA inby + te; Anb 1-A83Anb + \"Hyín soVin", c, d en \nASCII encoding for relevant characters is given belowłn \n 421 43) 45 An Question: What is printed by the following ANSI program? Options:
OCR: Wes Pesototalln
[1500
gramos'n in 1_2600 grmoskn 3 3700 gemoe\na \n An \u 201c200 grams \n An
perro eachorro tenía un peso de 1.500 gramos mes de nacido. En la tabla se muestra el peso del cachorro en los primeros cuatro meses. De acuerdo con tabla, žCuál es el cambio del peso del cachorro entre un mes el mes siguiente? Opciones:
Pregunta:
A.) Disminuyó 1.500 gramos
B.) Disminuyó 2.600 gramos
C.) Aumentó 1.100 gramos
D.) Aunentó 3.300 gramos
Respuesta:
A.) K B.) 122 75 83 C.)
D. P x
Answer:
"Caption was trimmed for visualization pur-
poses.
37

---

Published as a conference paper at ICLR 2026
D.6 To WHAT EXTENT Do TEXTUAL AUGMENTATIONS BOOST VLM CAPABILITIES?
The significant performance gap between text-only and multimodal responses raises critical questions about the strengths and weaknesses of the visual processing in the tested models. this analysis, we investigate to what extent do visual processing constraints limit multimodal capabilities, and conversely, can automatically generated textual augmentation improve model performance?
To explore this direction, we generate synthetic captions (using Gemini 1.5 Pro) and Optical Character Recognition (OCR) text (Tesseract (Smith) 2007)) for all images KALEIDOSCOPE, aligning with the methodology of (Das et al. 2024). Unlike prior work that completely replaces images with text, we evaluate whether a VLM augmented with these textual inputs can boost performance.
Table 18: Accuracy on augmented multimodal inputs with image captions. Results are grouped by image type. We report Valid Accuracy (%); the highest scores are highlighted in bold for each model. Macro averaged accuracy is reported over language for both methods.
Swen2.5-VL-78
Gemini 1.5 Pro
Samples
Image
+Caption
Image
+Caption
Diagram
2,182
38.0
37.9
59.4
59.6
Figure
6,178
34.0
34.8
51.3
50.0
Graph
733
44.3
45.2
67.9
68.2
Map
392
48.0
46.7
69.4
70.9
Photo
631
53.9
54.1
75.8
74.3
Formula
487
34.9
37.3
68.3
68.7
Table
597
40.9
34.6
76.0
76.1
Text
257
76.3
79.8
85.2
83.7
Macro Avg.
11,457
36.88
36.83
55.71
54.81
Table 18 shows the results of augmenting visual inputs with synthetic captions and OCR text across diverse image types in KALEIDOSCOPE, measured by valid accuracy (%). Overall, the addition of a caption and OCR text improves the performance the selected models in 5 out image types. Both models experienced a performance boost coordinately for Graph and Formula. The experiment reveals that the utility of textual augmentation depends critically on image content type. While Gemini 1.5 Pro dominates overall performance, Qwen2.5- demonstrates selective gains when provided with captions and OCR: improvements in Graph (+0.9%), Photo (+0.2%), Formula (+2.4%), and Text (+3.5%) suggest that textual augmentation aids interpretation of content where visual elements are tightly coupled with symbolic or linguistic features (e.g., labeled axes, embedded text, or mathematical notation). Conversely, performance declines for Diagram (-0.1%), Map (-1.3%), and Table (-6.3%) with augmentation. implying that synthetic captions may introduce noise or fail capture structural relationships critical to these categories. Gemini's robustness across modalities (< 2% variation in most categories) suggests its stronger native visual understanding reduces reliance on supplementary text. The results underscore that captioning effectiveness is context-dependennt: text augmentation benefits models most when (1) visual content inherently contains extractable text (e.g., Photo with signs, Tex7 regions) or (2) symbolic patterns (e.g.. formulas, graphs) require disambiguation. However, for structurally complex text-sparse images (e.g., Map, Diagram), captioning may not compensate for deficiencies in spatial or relational reasoning. Full results, including total accuracy and format error, can be found in Table 19
38

---

Published as a conference paper at ICLR 2026
Table 19: Total Accuracy %. Valid Accuracy % and Format Error % (FE) grouped by Image
Type in KALEIDOSCOPE for captioning OCR experiment.
Qwen2.5-VL-7B
Gemini 1.5 Pro
Total Acc.
Valid Acc.
FR
Total Acc.
Valid Acc.
FR
Diagram
37.8
37.9
0.3
58.9
59.6
1.2
Figure
34,6
34.8
0.7
49.0
50.0
1.9
Graph
45.2
45.2
0.0
67.4
68.2
1.2
Map
46.7
46.7
0.0
70.9
70.9
0.0
Photo
53.9
54.1
0.5
74.2
74.3
0.2
Formula
37.0
37.3
0.8
66.7
68.7
2.9
Table
34.5
34.6
0.2
74.7
76.1
1.8
Text
79.8
79.8
0.0
83.7
83.7
0.0
D.6.1 CAPTIONING & OCR
We instantiated Gemini 1.5 Pro with the following instructions to generate synthetic captions from the images in KALEIDOSCOPE, Prompts with image augmentations are shown in Table 17 Gemini 1.5 Pro's prompt for captioning:
*xInstruction:>
You are an expert image captioner. Generate highly detailed, precise, and academically relevant textual descriptions of images sourced from exam questions, ensuring all critical visual elements are captured for accurate problem-solving.
**Guidelines:**
Exam-Specific Analysis:
Identify and describe key components (e.g., diagrams, charts, graphs, labels, symbols, exact attributes (e.g., numerical values, units, directional arrows, text annctations).
Primary Elements:
s, annotations) and their
Secondary Details: Note stylistic features (e.g., "black-and-white schematic, "color-coded bars in a graph"), , spatial relationships (e.g., "force vectors pointing northwest"), and contextual clues (e.g., axes labels, legends, scales).
Textual Elements: Explicitly transcribe all visible text (e.g., labels like "Mitochondria," numbers like "5v, " titles like "Figure 2: Velocity vs. Time").
Academic Precision:
Technical Focus: Prioritize details critical to exam questions (e.g., right triangle with hypotenuse labeled 10 em, " "a bar graph comparing GDP of 5 countries, with Japan's s bar shaded blue at 4.3 trillion").
Diagrams/ Specify type (e.g., "pie chart, " "circuit diagram") and components (e.g., "resistor symbol connected to battery").
Scientific Relevance: Highlight measurements, units, symbols (e.g., 25"c, "a pulley system with frictioniess ropes").
39

---

Published as a conference paper at ICLR 2026
Structure Clarity:
Begin with the image's purpose (e.g., "A biology diagram of plant cell") followed by a systematic breakdown (left-to-right, top-to-bettom, by functional layers).
Use neutral, objective language. Avoid assumptions unless implied by context (e.g.r" "a downward arrow labeled 9.8 m/s² likely representing gravitational acceleration").
**Output Format:**
Single paragraph (4-6 sentences).
Example:
"A physics diagram depicts two blocks frictionless inclined plane: Block A (5 kg) is connected via rope to Block B (3 kg) over a pulley. Angle theta 30% with vectors labeled F_normal and F_gravity, A scale beside the plane shows time Osto t 5s. Text at the bottom reads: 'Calculate tension in the rope.' The image monochrome, with dashed lines indicating motion direction."
Constraints:
Avoid Omissions: Ensure labels, numbers, symbols are overlcoked, even if small or peripheral.
Neutral Tone: Exclude subjective interpretations (e.g., "messy handwriting" "complex diagram") unless style is exam-relevant, (e.g., "a hand-drawn sketch with annotations").
40

---

Published as a conference paper at ICLR 2026
D.7
CROSS-BENCHMARK PERFORMANCE
Table 20 presents comparison of Kaleidoscope's overall valid accuracy with MMMU-Pro and MMMU- Vallidatin on the overlapping subset of models. The results show that Kaleidoscope occupies a difficulty range comparable to existing high-quality multimodal benchmarks: most models perform slightly above their MMMU-Pro scores but below their MMMM--Valddtionn scores. Importantly the rank ordering of models is not identical across benchmarks. Several models with similar performance on MMMU diverge noticeably on Kaleidoscope, reflecting the benchmark's unique emphasis on culturally grounded, in-language multimodal reasoning. These discrepancies reinforce that Kaleidoscope is not merely a parallel version existing datasets, but complementary evaluation that probes multilingual and cross-cultural generalization more directly.
Table 20: boossbennchark comparison on KALEIDOSCOPE, MMMU-Pro, and MMMUU Results are reported as overall accuracy (%). KALEIDOSCOPE values use valid accuracy. Asterisks (*) indicate results provided directly by the original authors.
Model
Kaleidoscope (Vliid Acc.)
MMM(-Pro (Acc.)
MMMU Validation (Acc.)
Claude 3.5 Sonnct
63.87
51.5
68.3
Gemini 1.5 Pro
62.95
46.9
65.8
GPT-4o
62.10
51.9
69.1
Qwen2.5-VI--77.
53.00
46.2
64.5
Qwen2.5-VVL-32R
48.64
N/A
N/A
Aya- Nya-Vision-32B
39.66
45.11*
N/A
Aya-Vision-8B
35.TI
39.9
N/A
39.60
38.3*
58.6*
Qwen2.5-VI.-3B
35.63
31.6*
53.1*
Molmo-7B-
32.88
N/A
N/A
Pangea-7B
34.02
N/A
N/A
DATASET SAMPLES
E. CATEGORIES OF VISUAL ELEMENTS
We group the visual elements into eight primary categories in KALEIDOSCOPE in Table If an image falls into multiple categories, we assign the most representative based on the image's content.
E.2 SELECTED DATASET SAMPLES
Table 22 presents sample from each dataset, including the question, the associated image, the provided answers, and the correct answer highlighted in green.
E.3 DATASET FIELDS
We provide description of the fields KALEIDOSCOPE in Table 23
41

---

Table 21: Types visual elements or images in the KALEIDOSCOPE benchmark. The correct answer highlighted in Bold Green. Some samples are reformatted for better presentation.
Visual Element Category
Question Image
Question Answer
Question:
Wie verhält sich die Ver- armungszone in der hier dargestellten Halbleiterdiode!
Diagram. Technical or schematic drewings illustrating processes, structures, oppl..
9
Options:
A. Sie erweitert sich.
B. Sie verengt sich.
C. Sie verändert, sich nicht.
Sie verschwindet.
0V
VV
D
Question:
Applicable
'B"-
Options:
Illustrations, draw- ings, visual representations of objects, patterns, or symbols.
Figure.
A. contains more genes
Figure-B
В. unable to replicate
C. present in the nucleus
Figure-A
D. used vector
Question: Em xicara que já contém certa quanidade de agúcar, despeja-se rafé. representa função exponencial M(t), que fornece quantidade de açticar não dissolvidn (em gramaas), minutos após o café ser despejado. Pelo gráfico, podemos concluir que.
curva abaixo
Charts. Images showing plotted such line graphs, bar charts, scatter plots, ple charts, floweharts, or- ganizational charts, and so on.
Options:
A. m(l) 2(4 2(4-1/76)
B. m(()) = (25-1/5)
C. m(t) 20- 2(5-t/50) t/50)
253
D. m(0 2 2-5-t/11e t/100)
HOCKOSCCHE
Question: Mismaticтs storo remımин
MоRHa xapaкrеprиууиmи, coupasvurcc на
nonaHy kapry?
Map. Geographical or spatial representations
Options:
A. E. XMC.MBIIBBOOO
B. BaroscsKorD
C. JL Muoro picoorpimmoo
Cavoizomeva
NOTE
PODRCNEE
Continued page
42

---

Visual Element Category
Question Image
Question Answer
Question:
Wat kun je zeggen over verzorgingagebled van deze Mc- Douald's Arnhem?
Options:
A. Het verzorgingsgebied beperkt
Photographs. Photographic images of real-world scenes, ob- jects, or people.
xich tot stad Arnhem.
Het versorgingegehieet beperkt
dasr galeec
de provincie Golderland.
C. Het verzorgingsgebied beperkt sich tot Nederland.
verzorginpspebied hoperkt
de regio Arnhem omstreken.
Question: 3tReaa ge вî Resy чy साीिकरप aी रियप di aीर, काराकाय लिरररर Kp * शाकद ć1
Formula, Mathematical equa- tions, chemical formulas, math- ematical dingrams, or related concepts.
Options:
A. 1-28g
2HI(g) - H2(g) + (2(gg)
B. 142K,
28
C. Ir2Kp
2
142RRp
Question:
دass دلمردي ب طليق Jffn كرة شرري
r
بلييي للاا
Table.
Structured data ar- ranged rows columns..
Options:
A. 0
B.
C.
D. 14
Question:
AC-G# पासरगल (BR काा EF BC
पाका
Text. Images containing pri- marily textual information.
ABC fिrgपet сमाग efees 85* कर AB=ACI
Options:
B. 5607
C. Sboc
D. Sobo
43

---

Table 22: Samples from various exams in the KALEIDOSCOPE benchmark. The correct answer highlighted Bold Green. Some samples are reformatted for better presentation.
Language
Question Image
Question and Answer
High School Exam
Plysiis
Question:
C- 1pf
للاااري الماااق اللل ul
Arabic
sube
Options:
Veff= 200V
0.2
B. 2 A
f= 200 Hz
C. 0.02
High School Exam
Geology
Question:
S ال 14 2x الnيط
الل
JMD م Ul lyyte الأرل الدراة انتع الجوول غلال من
Options:
Yo
.A
ع .B
.C
.D
BRTA Driving Test
Driving
Question:
4 foel पाशा for gиts
Bengali
Options:
A. अपरर गंेंक अभााहरतर यान
В. अारिट besser Rirew
D. eaाा काकिकगररिलक berbocers
HSC Exam
Geography
Question: कभीी 'as αहgersiलर भायातत baf नfoनr fiie εााीीि 4य बात fिरe cite
V
644 4
Bengali
ficest Gеtतfè srße
Options: A.
B. a
C. G lii
D. i, i iii
BCS Exam
Resssoiig
Question: fिया लिाप सााी fिrger oxtcee?
Bengali
Options:
A. aft
B. oft
C. bft
D. aft
Continued on next page
44

---

Language
Question Image
Question and Answer
be wi1d
Toels
Meaioo
Dutch Central Exam
Economics
Question: Bekijk bovenstaand diagram. Hoevvel ton
klein chemisch afval werd er in Nederland in 2000 ingeza-
Dutch
meld?
Options:
A. "21.000 ton"
B. *19. 519.50 ton"
C. "23 ton"
"20.500
Mathematics
Question: In the equation, and constants. Which of the following could be the value of p?
English
4 (px (pxx
Options:
C.
D. 9
UCEED Exam
Design
Four spheres start revolving clockwise in con- centric circles from their initial positions as shown below. travels at 2m/sec, green 4m/ sec, 2m/ /sec and blue Am/ssc. Which of following statement(s) is/are TRUE?
Question:
English
Options:
Yellow and green never cross (overtake) each other
B. Red and blue takes the same time complete
olution
C. Celellow takes less time than green to complete one rev-
olution
D. Blue and red will cross each other twice after the first
complete revolutions of blue
Exam
Biology
Which type food digest lebelled men- tioned the figure?
Question:
English
S
Options:
A. Potato
P
В. Pulse
Oil
D. Ghee
Continued on next page
45

---

Language
Question Image
Question and Answer
GATE
Engineering
Consider the CMOS circuil shown in the fig- ure (substrates are connected their respertive sources). The gate width to gate length L ratios W/L transistora
Question:
V
PMOSFET
shown. Both transistors have the gate oxide caparitance the threshold voltage -IV and the mobility of holes c0 /V.. the nMOSFET, the threshold voltage 1V and the mobility of electrous 300 /Va. The steady-state output vullage Vo is Options:
unit area. For the PMOSFET,
English
VO
#MOSFET
A. equal to V
than V
C. less than V
D). equal to 2V
Physician Exam
Language
de A-weging het menselijk gehoor. Deze figuur leert dat
Question:
Figuur
Options:
Flemish
A. de mens tonen rond de 1000 Hx het beste hoort.
C. mensen tonen van 10.000
meer kunnen horen.
deselfde fysische geluidasterkte altijd intens wordt gehoord.
155
1000
D. bij gelijke geluidssterkte, een mens 100 Hz zachter hoort dan 50 Hz.
Mathematical Kangaroo
Mathematics
Question: attache ensemble anneaux comme diqué ci-contre de façou former chaine de longueur. Combien d' d'anneaux sont nécessaires ? Options:
French
A.
42
C. 21
D. 85
HTa
Hifeer
Amateur Radio Exam
Engineering
100
Question: Wie großi Vaaee-Sorree-Spannne wenn sich der Schleifer
Anschlag befindet?
German
390
100.
Options:
G- 10
A. 3,5
IVV
2,77
C. 3,7 V
4-1000
D. 0,45 V
6
Continued page
46

---

Language
Question Image
Question and Answer
Science Olympiad
Biology
aाप से सिषद पेक कोयकर di 3.30 पाी सिनीाा şीस ugan ap wहकc---ारी रिनोया बोe म чहा şæ SRय-पाе साक टडअ गे ge कगाा सहा योयास अरयती steet [िv aςg uftade omg ali?
Question:
Hindi
goeft
Options:
A. पमोपर att रलडपत माररीीिाा Riकीa staf è अमतर gafeui सiğieta
होती ć1
qdiv alwrlaf
B. बगीर मंंरपीरीयाा सिरि अीती ë. रहिपर मंकरपररााा vंzef
होती अोर qalिui लाeरीt #1
C. शसीर átर रहडपर मा vigitta staft è साीिक galिru tातरc
D. शपीीय मीय aigi şheît ë. अहिप मोयपीरया सिरिय şhrlt
sit gaferi zigRa stall či
JEE (Main)
Physies
Question: रिं (a), (b), (c), (d) रिकरर Preffir कहर fil fिx Aिt @нोаाае [िााडाी Mttoreel un4
Options:
(8]
A. म.नापररर डापो,, भीम कापो,, सोहरर सेल, LDR (िाहह fि (हिारे))
Hindi
В. नोनर सायो,, साााग कसीद, LDR (साय fzd8z ((िादक), सोसर
सरा
C. सोसर सेल, LDR (िाडद fetiz अलिलटे), पीमरर डाीं, साकारल
लोमर (d)
D). लोसर aıizz, सोेार सेत, सापररा žायो,, LDR (िाह fadiz ((ीतदय)
(c)
UP-CET
Social Sciences
Question: दिx टिद fिve पहaाrरयc)
Hindi
Options:
A. नोययपर लिाा
B. xaाfरीपार fिver
C. उात fिxsll
D. अτ4 लिताl
JEE (Main) Mathematics
Question: परि sh eीी (िययy सर ससीिकरय è orgen,, A कपसर
1 (02 2x 4)3/2 dr
k
?:
Hindi
Options:
A. 1
B. 2
C.
Coutinued on next page
47

---

Language
Question Image
Question and Answer
SSC CGL Exam
Reasoning
Rिr žभंी di attalfire पाि करीदद #) # मिनालिकित # सिकीी टभीी di shatfPras पी attaa shalfrrr पोद से नररिर ??
Question:
Hindi
40
Options:
A. 4
Indaa
Pakistan
China
England
USA
C.
D. 1
33'
High School Exam
Geography
23 26
Question: Kick platumes laipsniu yra tarp Siaurès poliarinio
Piety alogražos?
Lithmanian
Options:
A. Apic 23
B. Apie 90",
23 26'
C. Apie 100".
D> Apie 132".
66 33'
PSC Exam
Reasoning
Question:
रिडरयी fिt 3,3,3,8 भहीोार şi बोन सिकçe
अापरया मिाागर gof эामतरकी Prgernn Rि aाşe अउा quf अासाररकी कनाकर रिकषकाी नसकयςह रिशपकी fिcseranz यनीी rnfotee
Nepali
A
8
A. Options:
B. 234
C. 245
D. 345
Olymplad of Informatics
Math
Question:
23 is مe pala wast golue is معر طي 232 Si lifs sursee v glast youls for th co e15 " Leigo Gawisjs As -وم ceiser classs
مolo
+NIST
Persian
nien TY Si shackiag
Options:
55 سامت جور iA
.5 بادااعت زور 1 .R
- سامت دير .C
برادد را رردييي جوواا .D
Continued page
48

---

Language
Question Image
Question and Answer
Driving Test
Driving
Question:
بست جايز بنن حوان
Options:
Persian
(++ - الوهن 4 ت A
(c 4 b و A
و
High School Exam
Engineering
Question:
&aml كام 5oll مet BC
Options:
Persian
アンク عI مماتت إشارة .B
خويي راء ككياة .c م .D
A
University Entrance Exam
Physics
Question:
Au Aip B& dis sss i ws be Es سطاي سحك كر &aal CbB Rjg ussu ,
fm
Persian
B
Options:
A. 1.5
m
B. 1.25
m
C. 1.2
D. 2
تونن
University Entrance Exam
Geology
Question:
A 144 wiOye wff ue aees for dous ung
Options:
Persian
عادي كبل دريا بدزرك A
دابك ترنن دراا بووو B الويكك دري يلخرويي +C عادي كن ببر .D
كر شيلل
Continued on next page
49

---

Language
Question Image
Question and Answer
UNESP
Social Sciences
Ohserve fachadas duas igrejas. À querda, Basilica de San Michele, construida século XII em Pavia, na Itália, direita, Catedral de Reims, erguida partir do século XIII em Reims, na França. (Georges Duby e Michel Laclotte (orgs.). História artís- tica da Europa: Idade Média 1998.)
Question:
As duas fachadas
Options:
Purtuguese
diferenciam-se pela pouca ornamentação San Michele, que expressa estilo románico, pela monumentalidade sofisticaãão de Reims. B. diferenciam-se pela solides de San Michele, sim- buliza a furga espiritual catolicismo, pela carincia de detalhes na sede papal Reims.
C. igualam-se suntuosidade quitetônico, indicando poderio econômico de Igreja católica.
rebuscamento
diferemciam-se pela discrição Michele, que rev- rigor conduta dos protestantes, pela ostentação da riqueza católica de Reims.
FAMERP Entrance Exam
Physics
Question:
Quando um gerador força eletromotriz ligado resistor de resistência
elétrica i de intensidade 2,0 A circula pelo circuito. R
Portuguese
21
A resistência interna desse gerador é igual &
Options:
A. 0, 4052
B. 0, 20Q
C. 0, 10Q.
0,3012.
Unicamp Entrance Exam
Language
Question: A imagem a seguir apresenta transcriçãn de
diálogo em video publicado no Instagram. diálogo, principal caracteristica reformulação da fala da médica é a inserção de
cund to Now node pules
componnd hostars
Options:
Portuguese
A. expressões que utilizam verbos frasais para recontextu- alizer tratamento da paciente.
B. abreviagões de substaativos, através das quais médica amplia as informações du caso.
C. girias que utilizam diversas classes de palavras para es- pecificar melhor disgnéetico paciente.
Aer oN. Thoe
D. vacábulos marcados pela oralidade, através dos quais a médica atunlisa os procedimentoss futuros.
Do beoke
Dus lloorol
Thark
Continued on next, page
50

---

Language
Question Image
Question and Answer
ENEM, Brazil
Mathematics
Question: Um segmento de rela está dividido em duas partes na proporçān âuraa quando o todo está para uma partes na mesmna razão em que cesa parte para Essa
proporcionslidede comumente solução positiva da equação xp ip Assim como a potência y², as potências superiores y podem expressa forma ap + inleiros pusilivus,
representada pela letra grega s., sou valor é dado pela
Portuguese
9+1
2011
3p-12
5g
8
que apresenlado quadro.
A potência s", escrita na forma ap + (a e são inteiros positivos), Options:
A. 2
B. 9,p +
C. 11p +
13p
Mathematical Kangaroo
Mathematics
20
Question: Kannso upogetana rospurиe 1pоyzı je
ocemveno?
Serbian
Options:
%
B. 90
C. 85 %
D. 80
Mathematical Kangaroo
Mathematics
Question: Karux reoserpireecuux qutryp pucyuue??
Options:
Russian
A. xpyron
B. Boo 7MH dourypel сстs
C. прамоугоивнниоо
D. треуromt-ииo
Medicine Exam
Pulmonology
Question: Varón de 60 años, fumaadrr activo, que presenta tos y expectoración diaria años de hemoptoica. En los últimos Presenta acropaquia en auscultación pulmonar destacan roncus sibilantes telcinspiratorros en pulmón izquierdo. La TC pulmonar de alta resolución se muestra en la imagen adjunta. [Cuál es el diagnóstico más probable?
ocasionalmente añade disnen progresia.
Spanish
Choices:
A. Carcinoma quistico.
Enfisema pulmonar.
C. Tuberculosis cavitada.
D. Bronquiectasias.
Coutinued on next page
51

---

Language
Question Image
Question and Answer
R1=?
Undergraduate Exam
Blophysics
Question: Calcule valor de la primera resistencia (RI)
Spanish
12 1 A R2= 3Q
Options:
A.
C. 12 Q
R4=1Q
R3 Q
D. 24 52
High School Exam, Colombia
Biology
Question: En un laboratorio se estudia el comporiamiento del
volumen de un gas ideal variar su lemperatura, obteniendo siguiente gräfica: Teniendo cuenta información la lemperatura aumenta de -153 CC -33 CC, zqué pasa con el volumen del gas?
Volumwnn
gráfica,
Spanish
Options:
L. Disminuye de
B. Disminuye de 10 L.
C. Aumenta de L 10 L.
D. Aumenta de 10 a L.
Undergraduate Exaan
Chermistry
Question: stys agodd Midydio dioty amenS gaigent 200?
Telugu
Options:
A. 304.9
1
304.4
C. 301.9
D. 303.4
ZNO Vision
hematiisss
Aracmmooi нулми pasiyca 6 cM siepirammm, eacynnii myxmamtii, mp sac dropay примокутиro mppaneeenineu
Question:
(amB. рисунок). ЯRoо 3 новсаснои More бути висога & mycuuam??
Ukrainian
Options:
A. 3 cM
B. CM
D. 13 cM
Driving Test
Driving
Question: To mciñ rparecopii uptuomerm pyx i.doonain
Jersesomy arrоMooLuи??
Ukrainian
Oplions:
Timau
Timas B.
C. A iG.
D. Bo Gym- miin.
52

---

Published as a conference paper at ICLR 2026
Table 23: Structured dataset fields with descriptions used in data collection protocol.
Field
Description
language
The language in which the question is written (e.g., "en" for English).
country
The country where the exam originated (e.g.. "United States")
contribitoo__contryy
The contributor's 's country of residence (e.g., "Spain").
file_name
The internal database filename for the original exam document.
source
The URL or reference the original exam document.
license
Licensing information of the exam (e.g.. "Unknown" " if not stated),
level
The educational level of the exam (e.g., "University Entrance"))
categozy_en
The exam subject category in English (c.g., "Chemistry").
categoty_source
The subject category as written in the original language (language).
lang
original_question
The original question number in the source document.
_um
question
The text of the question.
options
A list of possible answer choices. For example, ["Option A", "Option B", "Option C", "Opti on D").
answer
The index of the correct answer (e.g., for the fourth option).
question_imag
The extracted diagram, graph, table associated with the question.
image_information
A label indicating the importance of the question_image for
swering the question. Possible values include:
"useful" The image provides additional clarification.
"essential i The image is necessary to answer the question.
image_type
The category of quest on on_image (e.g., "figure", "graph",
"table e as described in Appendix H.1
53


Reviews:
[{"review_id": "EfXy4HOulO", "reviewer": "Reviewer_6PmD", "summary": "This paper addresses the problem that VLM evaluation is heavily dominated by English-language and Western-centric benchmarks. To overcome the limitations of simply translating existing datasets, the authors introduce KALEIDOSCOPE, a new benchmark composed of over 20,000 \"in-language\" multiple-choice questions from real-world exams. The benchmark spans 18 languages and 14 different subjects, with a significant portion (55%) of the questions requiring multimodal reasoning (image and text). The dataset was constructed through a large-scale open science collaboration with contributors from around the world, ensuring linguistic and cultural authenticity. The authors evaluate a wide range of state-of-the-art VLMs on KALEIDOSCOPE, including both large closed-source models and smaller open-weight models. Their findings reveal substantial performance gaps: all models struggle more with multimodal questions than text-only ones, performance is significantly weaker on low-resource languages and questions from STEM subjects, and models exhibit a clear bias towards Latin scripts.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The motivation for this work is very clear and addresses a critical gap in the field. As vision-language models become more widespread, it is essential to evaluate them beyond English. The paper makes a strong case against the \"translate-test\" paradigm and for the necessity of \"in-language\" data that preserves cultural and educational context.\n\n- The design and scale of the KALEIDOSCOPE benchmark is good. It is a comprehensive resource, covering 18 languages and a wide range of academic subjects. The use of a multiple-choice question format provides a structured and scalable evaluation framework. The inclusion of rich metadata for each question, such as image type and educational level, is also a very valuable feature that enables fine-grained analysis.\n\n- The collaborative, open-science approach to data collection is particularly commendable. By involving a diverse group of researchers worldwide, the authors have created a more authentic and representative dataset than would be possible with automated translation. This methodology directly confronts the known biases in dataset creation and should be seen as a model for future work in this area.\n\n- The paper's analysis is cmprehensive and provides several insightful findings that go beyond a simple ranking of models. The clear performance gap between text-only and multimodal questions is quantified well. Furthermore, the analysis of performance by image type (e.g., diagrams vs. photos), by subject (STEM vs. humanities), and by script (Latin vs. non-Latin) offers a deep look into the specific weaknesses of current VLMs.", "weaknesses": "- The reliance on a Multiple-Choice Question Answering (MCQA) format, while practical for evaluation, comes with inherent limitations. Models can sometimes guess the correct answer or exploit statistical cues in the options without genuine understanding. The authors acknowledge this point in the appendix, but this is a fundamental aspect of their evaluation design and should be more prominently discussed in the main paper's limitations section (Section A).\n\n- The data collection process, despite being a large-scale effort, could benefit from more transparency regarding the source material's representativeness and difficulty. The paper notes that performance in some languages (like Lithuanian) may be higher due to the subject matter of the available exams. This suggests that the benchmark measures a combination of model capability and the specific difficulty of the sourced exams, which can be hard to disentangle. A more detailed discussion of the potential imbalances in difficulty across languages and subjects would be helpful.\n\n- A potential confounding factor in the experimental results is the use of two different prompting strategies for closed and open-weight models (Chain-of-Thought for large models, direct answering for smaller ones). The authors justify this as a pragmatic choice, which is understandable. However, this means the comparison between these two groups of models is not perfectly controlled. The performance gap could be influenced by the different evaluation protocols. This should be more clearly stated and discussed as a limitation of the current experimental setup.\n\n- The analysis of format errors (Section 5.3 and Table 1) is interesting, but its presentation could be more integrated. For example, the fact that GPT-4o's accuracy improves significantly when format errors are excluded is a very important finding that highlights a specific weakness of that model. This point, and the high refusal rate of Pangea, could be woven more directly into the main results discussion to give a clearer picture of model behavior.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OooWmNWEJC", "reviewer": "Reviewer_PSVd", "summary": "Kaleidoscope is a new large-scale benchmark designed to evaluate multimodal + multilingual reasoning in vision-language models (VLMs), claiming to be the largest benchmark of its sort. It contains around 21k multiple choice questions (MCQ) across 18 languages and 14 subjects, with about half including images. The dataset is built from real-world, in-language (that is, not machine translated) educational exams contributed by native speakers from more than 20 countries. The authors evaluate both commercial API and open-source models under standardized protocols. Their analyses show large modality gaps (text-only vs. image+text), STEM-specific difficulty (compared to humanities subjects), and cross-lingual disparities (Latin script advantage over non-Latin). Despite some limitations (language balance, difficulty calibration, and reliance on MCQA format), Kaleidoscope aims to serve as a diagnostic benchmark to reveal weaknesses in a niche but increasingly important task where both multimodal and multilingual reasoning are required.", "questions": "See the above weakness sections. Besides, I have a couple of questions / suggestions.\n\n- Given the benchmark’s collaborative nature, do you plan to release a contribution protocol or submission guideline so external contributors can add new languages or exams in future versions? I think this could turn Kaleidoscope into a true community-science platform and make it a living benchmark.\n\n- The appendix is lengthy and dense. I'd like to suggest that the authors add an overview or roadmap at the beginning of the appendices. This would increase readability.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Most importantly, the core contribution is the dataset itself. It targets a niche area that is difficult to produce synthetically (e.g., LLM-generated).\n\n- The scope of this benchmark (# languages, # subjects, and a substantial proportion of image-based questions) is impressive. Having personally experienced very large, multi-national collaborations, I am genuinely impressed by the coordination and collective effort that made this possible.\n\n- Although I wish the paper provided more details about the data collection and quality control processes (including license verification), my general impression is that the authors have made a serious effort to standardize collection and validation across languages and annotator teams.\n\n- Each item includes detailed metadata (17 fields including country, subject, image type, and image importance), enabling detailed diagnostic analyses. This metadata can itself serve as a valuable resource for future research.\n\n- The paper provides interesting empirical insights that can guide the development of future model architectures or training strategies: the modality gap, STEM vs. humanities performance disparity, and Latin vs. non-Latin script biases.", "weaknesses": "- Each language has independent questions rather than parallel translations, so performance differences reflect a mixture of linguistic and content variation _(I note that the authors were fair to list this point in the limitations section)_. This complicates direct interpretation of multilingual gaps. It would be helpful if the authors could provide some indication or proxy of relative difficulty across languages.\n\n- Related to the above comment, I was curious to see a translation baseline. The authors deliberately avoid machine-translated versions, but a translated-to-English baseline would, to some extent, help isolate the effect of language from content or domain differences.\n\n- I also wish the paper included cross-benchmark context. Table 2 reports results only on Kaleidoscope, without comparisons to other multimodal or multilingual benchmarks (e.g., MMMU, SEED-Bench, M3Exam). Without this, it is difficult to assess the dataset's relative difficulty and unique diagnostic value.\n\n- Intended-use guidance (e.g., do’s and don’ts) would have been nice. Benchmarks are often misused or overinterpreted; an explicit section clarifying what Kaleidoscope can and cannot measure would help practitioners use it responsibly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L22nOt7OG1", "reviewer": "Reviewer_KzYF", "summary": "This paper presents KALEIDOSCOPE, a large-scale exam-style benchmark to evaluate vision-language models across diverse languages and visual inputs. The proposed benchmark contains 18 languages, and 14 subjects with 55% requiring image understanding. The authors collect real exams via a global open-science effort. They evaluate closed (such as GPT-4o, and Claude 3.5 Sonnet) and open models (such as Qwen2.5-VL, Aya-Vision and others). After evaluation the authors describe different findings, among them, (i) all evaluated models perform substantially better on text-only questions than in multimodal ones. (ii) The type of visual data contained in exams affect performance depending if the question contains tables, diagrams, photos, etc. (iii) Performance depends on the domain of the questions. (iV) And finally the well known problem of crosslingual disparities in high vs low resource languages.", "questions": "Please refer to weaknesses for my questions and doubts. Overall I think the paper brings good contributions and really interesting insights for current open and close vision-language models that can be useful for the community. Now I'm accepting the paper, but I look forward to see the author's responses and clarifications, and sorry if I misunderstood something, after rebuttal I will revise my decision.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Paper Strengths:\n1. The authors tackle the gap of evaluation at the intersection of multilingual and multimodal tasks. Kaleidoscope proposes in language exam questions blending image and text modalities, covering 18 languages and 8 image categories (such as diagrams, graphs, tables, etc). Compared to other benchmarks on the field, Kaleidoscope is more linguistically diverse and highly focused on multimodal evaluation.\n2. The paper provides extensive evaluations by language, subject and image type.\n3. Overall, the paper shows interesting insights and evaluations of open vs close models on multimodal QA , such as multimodal vs text only across models, resource gaps, scaling trends and domain disparities. These insights can be valuable for the community and future work in VLMs training.", "weaknesses": "Paper Weaknesses:\n1. Table 1 compares closed models that use CoT prompts, while smaller models use a direct answer prompt due to CoT instability, however this is not an apples-vs-apples comparison, it would be interesting to see the performances of closed models using the direct answer generation approach. Now Qwen2.5-VL-72B is already close in performance compared to GPT-4o, with this direct comparison maybe Qwen could surpass GPT. \n2. I think the paper needs to show some statistics related to the difficulty level of the questions, this will give readers an overall idea of the benchmark. For example Table 4, could have how many grad, undergrad questions each language contains and in this way there is some information (maybe not really precise) but gives an overall idea of how difficult the questions are.\n3. Personally, for me it was not really clear when the authors describe that Kaleidoscope allows cultural evaluation by combining regionally sourced multimodal questions. The proposed benchmark contains different categories, so I expect in fields like STEM to contain common universal topics, and in others such as Arts & Humanities and Social Sciences to contain some cultural related topics, but the question is was this quantified in some way? like how many of those cultural related questions are containing information from their specific country-language pair?, I think that exam-questions could be more global in some situations and those will not be related to the culture of each specific country-language pair.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)