EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models (2024)

Abstract: We introduce EXAMS-V, a new challenging multi-discipline multimodal multilingual exam benchmark for evaluating vision language models. It consists of 20,932 multiple-choice questions across 20 school disciplines covering natural science, social science, and other miscellaneous studies, e.g., religion, fine arts, business, etc. EXAMS-V includes a variety of multimodal features such as text, images, tables, figures, diagrams, maps, scientific symbols, and equations. The questions come in 11 languages from 7 language families. Unlike existing benchmarks, EXAMS-V is uniquely curated by gathering school exam questions from various countries, with a variety of education systems. This distinctive approach calls for intricate reasoning across diverse languages and relies on region-specific knowledge. Solving the problems in the dataset requires advanced perception and joint reasoning over the text and the visual content of the image. Our evaluation results demonstrate that this is a challenging dataset, which is difficult even for advanced vision–text models such as GPT-4V and Gemini; this underscores the inherent complexity of the dataset and its significance as a future benchmark.1

heading: 1 Introduction
text: Large Language Models (LLMs) have recently demonstrated impressive skills in understanding and generating natural languages (Brown et al., 2020; Zhang et al., 2022; Scao et al., 2022; Zeng et al., 2023; Touvron et al., 2023b). This progress has paved the way for significant advancements in LLM-based vision models (Zhu et al., 2023; Liu et al., 2023a). Notable developments like GPT-4V (OpenAI, 2023) and Gemini (Anil et al., 2023) represent a new era in image understanding, exhibiting remarkable proficiency in interpreting and analyz-
1Datasets are available at https://github.com/ mbzuai-nlp/EXAMS-V
Croatian
Hungarian
Chinese
Polish
Bulgarian
Italian
Serbian
A ra
b ic
Germ an
English
French
Physics
Biology
Hi st
or y
C he
m is
tr y G e o g ra
p h y S o c io
lo g y
P o li ti c s In fo
rm a ti c s E th
ic s R e li g io
n
P s y c h o lo
g y P h il o s o p h y F in
e A r t s
Physics
B u s in
e s s &
E c o n o m
ic s
C h e m
is tr
y
A g ric
u ltu
re
G e o g ra
p h y
To u ris m
G e o g ra
p h y
M athem
aticsPhysics
Chemistry
Biology
History
Professional
Physics
C h e m
is tr y S o c io lo g y
B io
lo g y
B io
lo g y
H is
t o r y P h y s ic
s
C h e m is t r y In fo r m a t ic s S o c io lo g y P s y c h o lo g y P o lit ic s G e o g r a p h y P h ilo s o p h y E t h ic s
C h e m
is t r y
P h y s ic s B io lo g y H is to ry P o litic s In fo rm a tic s G e o g ra p h y P s y c h o lo g y S o c io lo g y P h ilo s o p h y
S o c ia
l
C h e m
istry
S cie
n ce
Isla m
ic S tu
d ie
s
B iolog
y
Ph ysics
Physics
Business & Econom ics Geography Tourism Chemistry
Chemistry
Physics
Science
Biology
Physics
Business & Economics
Geography
Loading [a11y]/accessibility-menu.jsFi ure 1: Data distribution for our EXAMS-V dataset: languages and subjects.
ing visual data alongside textual information. However, as Vision Language Models (VLMs) grow more sophisticated, existing benchmarks are becoming outdated, and unable to accurately assess these models’ performance.
For LLM evaluation, standardized testing akin to school examinations has proven to be an effective measure of a model’s capabilities. A typical benchmark MMLU (Hendrycks et al., 2021), which contains 57 subjects across science, engineering, and humanities, has become a de facto benchmark for LLM evaluation. Several other school exam datasets have also set the standard in evaluating LLMs in different languages (Hardalov et al., 2020; Li et al., 2023b; Koto et al., 2023).
In terms of VLM, a comparable benchmarking framework is conspicuously absent. Existing benchmarks are (1) primarily monolingual, focused on English; (2) mostly not from school exams, leading to differences in methods of examining humans; (3) tend to keep images and text separate,
7768
which fails to challenge models with more complex tasks involving integrated visual elements like tables, symbols, and scientific notations.
We introduce EXAMS-V, which addresses all these issues. First, this dataset represents a significant leap forward, treating visual and text content as a cohesive unit. This forces models to engage in more sophisticated processing, including distinguishing, preprocessing, and logical reasoning over combined textual and visual information. Additionally, EXAMS-V has a multilingual reach, covering 7 language families, further enhancing its complexity and applicability.
The key contributions of our paper include:
• We introduce a novel dimension to benchmarking vision language models, requiring them to reason over a unified snapshot that includes text, images, tables, graphs, and more. For this, we propose a new multimodal multilingual dataset, EXAMS-V, comprising 20,932 questions, spanning 11 languages and 20 subjects.
• We evaluate the performance of state-of-theart large language models and vision language models on our proposed dataset.
Through EXAMS-V, we aim to set a new standard in evaluating VLMs, providing a more real-
istic and challenging benchmark that mirrors the complexity and diversity of real-world information processing.

heading: 2 Related Work
text: LLM witnessed remarkable advancements in recent years, enabling them to generate human-like text, answer complex questions, and perform a wide range of NLP tasks (Brown et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023a; Chiang et al., 2023; Liu et al., 2023c). Simultaneously to the rapid development of English-centroid LLMs, researchers have also focused on extending monolingual language models to multilingual (Scao et al., 2022; Zeng et al., 2023; Li et al., 2023a; Sengupta et al., 2023) and multimodal (Alayrac et al., 2022; Chen et al., 2022; Liu et al., 2023a; Li et al., 2023c; Bai et al., 2023). Models, such as GPT-4 (OpenAI, 2023), Gemini (Anil et al., 2023) have demonstrated exceptional performance on various benchmarks and have been widely adopted in academia and industry. However, the evaluation of these models is a critical aspect that requires careful consideration to ensure reliable and comprehensive assessments.
Several benchmarks have been proposed to assess the multimodal capabilities of LLMs (Antol et al., 2015; Hudson and Manning, 2019b; Gurari
et al., 2018; Singh et al., 2019; Lu et al., 2022; Yue et al., 2023; Lu et al., 2023). Most early-stage benchmarks consist of photos as images, and the questions ask about the objects, attributes, or relationships between objects in the image.
Recently, inspired by the use of school exams as benchmarks for LLMs, researchers have begun to collect curriculum-based questions with images for VLM benchmarking. ScienceQA (Lu et al., 2022) is one of the most popular datasets in this area. It contains 21,208 multimodal multiple-choice questions with rich domain diversity across 26 topics, collected from elementary and high school science curricula. To answer these science questions, a model needs to understand multimodal content and extract external knowledge to arrive at the correct answer. MMMU (Yue et al., 2023) is another benchmark designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate rea-
soning. It includes 11,550 questions from college exams, quizzes, and textbooks, covering six core disciplines: art, business, science, health, humanities, and technology. Similarly, MathVista (Lu et al., 2023) is a benchmark with 6,141 samples for evaluating the mathematical reasoning capabilities in a visual context. However, like all previous benchmarks, these two exam benchmarks are in English.
M3Exam (Zhang et al., 2023) is the first multilingual multimodal exam benchmark that covers 9 languages. It includes 12,317 questions, with 2,816 questions requiring information from an image to arrive at the answer. One main difference between M3Exam and our dataset is that, like all other VLM benchmarks, M3Exam separates text and images for a single question, while we embed the question in the images.
Unlike the above benchmarks, our dataset boasts a broader linguistic scope, placing a particular emphasis on low-resource languages like Croatian, Hungarian, Spanish, and French. Notably, our examination benchmark surpasses others by featuring a greater number of questions, encompassing a diverse range of types and topics. This variety includes questions with accompanying images, tables, and graphs, as well as mathematical and chemistry equations. For a detailed quantitative analysis, please refer to Table 1.

heading: 3 EXAMS-V Dataset
text: EXAMS-V is a multimodal extension of the EXAMS dataset (Hardalov et al., 2020), which is collected from official state examinations crafted by the ministries of education across different countries. These assessments, taken by high school graduates, cover diverse subjects, including core disci-
plines like Biology, Chemistry, Geography, History, and Physics, as well as specialized areas such as Economics and Informatics. The original EXAMS dataset was intended for multilingual question answering and thereby ignored the questions requiring visual information. We included additional data for English and Chinese in our EXAMS-V dataset. The subject coverage and statistics are detailed in Table 8.

heading: 3.1 Data Collection and Analysis
text: Collection and Preparation of Dataset. The dataset collection process consisted of three main steps. Initially, we retraced the original PDFs used for creating the original EXAMS dataset. For English and Chinese, we gathered high school and entrance exam questions (specifically, Gaokao and JEE Advanced Questions) from China and India, respectively.
Then, we converted each PDF document to a series of cropped images, with each image having a single question and possible answers with accompanying tables, images, graphs, etc. This required the conversion of each page in the PDF document to an image and then the use of an open-sourced labeling pipeline to place bounding boxes around each question and its answers for each page.2
The third step involved the creation of metadata for each cropped question. This metadata includes a unique ID, file path to the question snapshot, subject, grade, language, and the correct answer for the question. Each set of metadata is stored as a JSON file corresponding to a specific subject in a particular language.
Annotation Guidelines. All the bounding box annotations are done manually by the authors with the following agreed-up guidelines: Only multiplechoice questions with 3 to 5 options and exactly one correct answer are considered, as they allow for a standard automatic evaluation of the correctness of model outputs.
Along with placing the bounding boxes, we marked whether the context within the bounding box is pure text or has visual context like tables, graphs, figures, or symbols. As the result of annotation, each question sample is an image that contains the question text and candidate options, along with other vision information such as figures, tables, graphs, etc. It also includes meta-information, as
2https://github.com/Cartucho/OpenLabeling
mentioned beforehand. This rigorous process allowed us to maintain the high quality of the dataset.
Data Quality Assessment. After the completion of our annotation process, we conducted a data quality assessment on seven languages based on the availability of an annotator with language expertise. In this evaluation, we randomly selected 50 questions from each language and requested annotators to assess each image sample based on four binary criteria:
• Image Clarity: Clarity of visual elements such as images, diagrams, or tables.
• Question Clarity: Clarity of textual information in the question.
• Single Correct: The image contains a single Multiple Choice Question (MCQ) with precisely one correct option.
• Others: Identification of other issues. The other issues encompass factors that render a question invalid, such as the presence of the answer within the question snapshot.
A question is deemed completely valid only if it meets all four criteria.
Upon thorough review, all annotators unanimously deemed the samples to exhibit exceptionally high quality across all annotated languages. Specifically, all the samples in Bulgarian, Croatian, Serbian, Italian, and Arabic met the four quality assessment criteria. However, in the case of Chinese, one sample exhibited unclear image information, and another displayed a question that lacked clarity. Similarly, an English sample was deemed invalid due to the presence of the answer within the image sample. This proves the high quality of our dataset. The annotation guideline used by the annotators is provided as Figure 8 in the Appendix section C.

heading: 3.2 Data Statistics
text: The EXAMS-V dataset contains 20,932 samples in total, spanning 20 subjects from grade 4-12. It encompasses a total of 11 languages from 7 language families, while it contains parallel data in more than three languages.3 The statistics are presented in Table 3, and Table 8 per language per subject details.
3Parallel data means that questions are semantically the same but in different languages.
Language Diversity. Table 3 provides an overview of the various languages featured in the dataset, along with the number of questions and subjects available for each language. The dataset includes high-resource languages like English and Chinese and low-resource languages such as Bulgarian, Croatian, and Serbian. It offers a diverse linguistic landscape, spanning Germanic, Slavic, and Sino-Tibetan language families. We also include Arabic, which has a script directionality from right to left. Additionally, Slavic and Romance language families exhibit multiple language representations, enabling the evaluation and understanding of closely related languages. These characteristics make EXAMS-V a great fit for multimodal multilingual assessment of any LLMs and VLMs.
Parallel Questions. Examinations in Croatia and the United Arab Emirates are administered in multiple languages, facilitating the development of parallel question sets for two language groups. Specifically, for Croatian examinations, we have parallel questions available in both Serbian and Italian. Additionally, Arabic questions are paired with English counterparts for four subjects: Science, Physics, Chemistry, and Biology. This process resulted in the creation of 1,207 Serbian questions and 1,147 Italian questions in parallel with Croatian. Furthermore, for Arabic, we have developed 262 parallel questions in English.
Subject Diversity. Each education system has its own specifics, leading to some differences in curricula, topics, and even the naming of the subjects. As a result, we initially collected 83 different subjects from different countries. Since different
naming conventions for subjects in different countries, the values of the subjects were very sparse and non-uniformly populated. We performed subject aggregation to club similar subjects into one single subject and finally got 20 aggregated subjects. They were further grouped into three major categories, based on the main branches of science: Natural Sciences – the study of natural phenomena; Social Sciences – the study of human behaviour and societies; others – Applied Studies, Arts, Religion, etc. The distribution of the major categories is Natural Sciences (53.02%), Social Sciences (27.15%), and Others (19.82%).
Question Complexity. The dataset is compiled from high school examinations administered in various countries, primarily featuring questions from grades 4 to 12. Questions in natural sciences such as Physics, Chemistry, Biology, and Mathematics, demand foundational knowledge of these subjects and intricate reasoning skills. Questions related to Geography and History necessitate specific knowledge about particular regions or countries. Additionally, the Polish section comprises a compilation of 55 diverse professional exam questions across various fields, spanning from accounting to the motor vehicle service process. Answering these questions requires precise understanding of these professions.

heading: 3.3 Comparison with Existing Datasets
text: EXAMS-V differs from other datasets by mainly introducing a new way of benchmarking VLMs – passing an entire question snapshot that contains both the visual and the text components instead
of passing the parsed and processed text with the image. This leaves the model to the work of text extraction and representation. Moreover, the dataset has questions of varying complexity and diversity with most of the questions coming from high school matriculation exams. Most previous benchmarks normally require commonsense knowledge or simple physical or temporal reasoning. In contrast, the EXAMS-V benchmark requires deliberate reasoning with high school-level subject and regionspecific knowledge. Lastly, EXAMS-V aims to cover high school-level knowledge with different forms of visual features, including diagrams, tables, charts, chemical structures, paintings, geometric shapes, etc. This means that a well-performing model on EXAMS-V could be considered to surpass a human adult on general-purpose tasks. We have included a detailed comparison of EXAMS-V dataset with other benchmarks in Table 1.

heading: 4 Experimental Setup
text: As we see in Table 8, the original data appears sparse and imbalanced. To ensure a more balanced benchmark, we split EXAMS-V into training and test sets, with careful consideration for language and subject representation in the test set. We sampled 20 to 100 questions for each subject-language pair based on availability. For languages with parallel data like Croatian, Serbian, and Italian, we performed parallel splits to maintain question consistency across training and test sets. Finally, we got 16,724 training and 4,208 test instances.
We evaluate state-of-the-art LLMs and VLMs on EXAMS-V benchmark. Our evaluation is conducted under a zero-shot setting without model finetuning or in-context learning, using either APIs or NVIDIA A100 GPUs.4

heading: 4.1 Models
text: VLMs. We consider various large vision language models. We evaluated two open-source models, which have shown remarkable performance on multiple multimodal tasks: (i) LLaVA-1.5 (Liu et al., 2023a) which integrates visual embeddings with Vicuna’s linguistic space. (ii) Qwen-VL-Chat (Bai et al., 2023), a multilingual multimodal chat model trained on Chinese and English data, which possesses excellent grounding, text-reading, and text-oriented question-answering performance. We
4Our experiments were conducted in Dec-2023 and attached to the latest version of the commercial models.
also evaluated two proprietary multimodal models: GPT-4V and Gemini-Pro-Vision (denoted as Gemini-V) (Anil et al., 2023). GPT-4V is the bestperforming multimodal model by OpenAI, and Gemini-V is the mid-range model among the Gemini family of multimodal models.
Augmented LLMs. To evaluate text-only LLMs, we augment language models with two image-totext tools, namely Optical Character Recognition (OCR) and Image Captioning (IC). We employ Google Tesseract for OCR and GPT-4V for image captioning. We treat LLM augmented with OCR and IC as a vision language system. This setup was applied to GPT-3.5-Turbo, GPT-4, and Gemini Pro.

heading: 4.2 Evaluation Setup
text: Given the multiple-choice nature of the questions, accuracy served as our primary metric. Models were instructed to format answers as JSON objects {"answer": "choice"}, allowing for straightforward prediction extraction from the outputs. Based on our observations, all models under consideration can adhere to the instructions and produce the answer in a JSON format.

heading: 5 Main Results
text: We present the results across languages in Table 4. To gain a clearer understanding of model performance, we establish a random baseline by assigning an option randomly from the available choices for each question. The random baseline for all languages ranges between 19-26%.
VLM Results. Among the various VLMs, GPT4V stands out with the highest performance, achieving an overall average score of 42.78%. This score, being only 20 percentage points above the random baseline, indicates significant potential for improvement in VLM capabilities. Gemini-V, following GPT-4V in our evaluation, achieves an overall average of 31.13%.
In comparison to commercial VLMs, opensource VLMs such as LLaVA-1.5-13B and QwenVL-7B fall short in terms of language support and model performance. According to our findings, open-source VLMs are limited in language support (2 for Qwen and 1 for LLaVA) and their performance in these languages is close to the random baseline. On the other hand, commercial models exhibit broader language support, as evidenced by
their performance surpassing random outcomes in almost all languages.
LLMs Augmented with OCR and Captioning. Large language models enhanced with OCR and image captioning show superior average performance compared to standalone vision-language models. GPT-4, when augmented with OCR and captioning, demonstrates the highest overall performance among both VLMs and LLMs. This can be attributed to the precise OCR capabilities of Google Tesseract, the detailed captions produced by GPT-4V, and GPT-4’s robust textual reasoning abilities. Furthermore, unlike prompting GPT-4V to directly generate the answer, augmented GPT4 decouples the difficulties in visual information extraction and text reasoning.

heading: 5.1 Analysis from a Language Perspective
text: Comparing model performance in different languages, we find that all models show random-level results for Chinese (zh), which might be due to the inherent challenges associated with the Chinese subset. The Chinese subset, derived from the Gaokao exam, contains the highest proportion of vision features such as figures, tables, or graphs. This makes it difficult not only for single VLM but also for OCR and image captioning techniques to capture the visual information in text form fully.
Following Chinese, Arabic (ar) and English (en) emerge as the next most challenging languages. For Arabic, the low performance is associated with the image sample itself. Figure 9 shows an image sample of Arabic where we can see that, unlike other subjects, the answer choices do not have any letter associated with it. Thus, when the evaluated VLMs and LLMs are instructed to return the answer in the form of A, B, C or D, they find it very difficult to
pinpoint the correct option. For English, the top-performing models only scored about 8 % above the random baseline. The difficulty in English might be attributed to its sourcing from the Joint Entrance Exam (JEE) conducted every year for admission to Engineering Institute in India. To solve these questions, the model needs to be able to demonstrate complex multi-step reasoning along with a very good understanding of fundamental science- Physics, Chemistry, and mathematics.
Overall, the best performing VLM, i.e. GPT-4V, outperforms in four languages, whereas LLMs with OCR and captioning capabilities excel in several languages. These four languages include Bulgarian (bg), French (fr), German (de) and Hungarian (hu). If we refer to the test set distribution reported in Table 9 of the Appendix, we observe that most of the samples in these languages have very few multimodal questions. Additionally, they have very few graphical and tabular questions on which GPT-4V tends to show poor performance according to Table 6. Other languages like Croatian (hr), Serbian (sr), Italian (it), and Polish (pl) have a fair distribution of multimodal and textual questions.

heading: 5.2 Parallel Data Evaluation
text: Since Croatian, Serbian, and Italian data come from the same examination, we conducted a parallel sample experiment for these languages. The GPT4V, Gemini-V, and augmented GPT4 results are reported in Table 5. The results show dependence on the language for all the models under consideration. Augmented GPT4 has the least performance variance. This can be attributed to the accurate OCR capabilities of Google-Tesseract.
For GPT-4V, there is a significant performance gap between Croatian and Serbian with Croatian outperforming Serbian by 20.12 %. Although both languages are very similar and often mutually intelligible, their scripts differ significantly. Serbian is Cyrillic, whereas Croatian is Latin. Latin script is more widely used, and the majority of the most spoken languages in the world have Latin script. This can be attributed to the strong performance of GPT4V in languages with Latin script, like Croatian and Italian.
Even for Gemini-Vision-Pro , there is a gap in performance between Croatian and Serbian. The accuracy for Croatian is better than Serbian by 4.57%. Gemini-V exhibits a notable performance
Table 40 27.50 37.50 Text 50 62.00 48.00
Table 6: Model performance on different vision features.
Model Accuracy on English
Random 24.40 LLaVA-1.5-7B 23.60 LLaVA-1.5-13B 26.00 Moondream-1.6B 24.40
Table 7: The performance of models of different scales on the English subset of EXAMS-V.
disparity between Italian and Croatian, as well as Serbian. This discrepancy is likely because Italian, as a high-resource language, enjoys greater representation within the Gemini family of models.

heading: 5.3 Vision Feature Evaluation
text: We compared the performance of GPT-4V and Gemini-V for four different vision features: scientific symbols, figures, graphs, and tabular data. We compared it against image samples with only textual information. For the evaluation, we curated samples of different vision features from the Croatian subset. The results are reported in Table 6.
GPT-4V shows fairly good performance for questions that involve scientific symbols and figures. However, it demonstrates poor performance for questions with graphs and tabular. Surprisingly, Gemini-V can show better performance for tabular data when compared to GPT-4V. Nevertheless, its performance across the remaining three vision features — scientific symbols, figures, and graphs, was subpar.

heading: 5.4 Effect of Model Scale
text: To understand the scale effect, we compared the performance of LLaVA-1.5-7B and LLaVA-1.513B on the English subset of the EXAMS-V dataset. We also evaluated the model on Moondream (Vikhyat, 2024), a 1.6B parameter model demonstrating strong performance in visual reasoning benchmarks such as GQA (Hudson and Manning, 2019a). The results are reported in the Ta-
ble 7. The table shows that the performance of all three models is near random. However, the larger LLaVA-1.5-13b model performs better than LLaVA-1.5-7b by 2.4%.

heading: 6 Conclusion and Future Work
text: The development of EXAMS-V as a benchmark for assessing the multilingual and multimodal capabilities of VLMs marks a significant milestone in the journey towards multilingual models. Furthermore, the EXAMS-V introduces a new dimension to visual question answering where the textual information is a part of the image. Thus, EXAMS-V not only tests the multimodal reasoning capability of current VLMs but also their ability to do OCR in a multilingual context. This requires a strong perception capability to draw boundaries between textual questions and multimodal contexts like tables, figures, graphs, etc. Furthermore, the questions with in-depth knowledge of multiple disciplines or subjects and questions from physics, chemistry, and mathematics require intricate reasoning. These features collectively contribute to the considerable complexity of the EXAMS-V. We believe the evaluation of VLMs on this dataset can directly contribute to our understanding of the progress toward the expert vision language model with multilingual capability.
In future work, we plan to extend the dataset with more image samples, subjects, languages, and modalities.
Limitations
Despite its comprehensive nature, EXAMS-V, like any benchmark, is not without limitations. For ease of evaluation and analysis, we only considered and collected multiple-choice questions. We limited our multimodal analysis to four broad categories, which are scientific symbols, figures, graphs, and tabular data. But this can be further extended to finer-grained analysis. For example, scientific symbols can be further broken down into mathematical notions and chemical symbols, while figures can be broken down into maps, figures, paintings, diagrams, etc. However, this requires the collection of more data, which is difficult, particularly for low-resource languages like Croatian, Serbian, and Arabic under consideration. Furthermore, since we are collecting exam questions from different regions of the world, the difficulty of the questions varies depending on the region they originate from.
This hurts the comparability of the dataset across languages. Although we tried to include parallel questions for direct comparability, but it was feasible only for three European languages: Croatian, Serbian, and Italian.
Ethical Consideration
• Copyright and Licensing: All data in EXAMS-V are collected from public sources.
• Ethics and Data Privacy: All testing instances in EXAMS-V are carefully scrutinized to exclude any examples with ethical concerns. Since all the data are collected from exam papers there is no privacy issue.

heading: Acknowledgments
text: This research is partially funded by the EU NextGenerationEU, through the National Recovery and Resilience Plan of the Republic of Bulgaria, project SUMMIT, No BG-RRP-2.004-0008.

heading: A Dataset statistics
text: The statistics of the EXAMS-V dataset for all languages and subjects are presented in Table 8. Table 9 shows the distribution of multimodal data in the test dataset used for the evaluation of VLMs and LLMs. One point to note is that there are instances where a single image might have multiple modalities, e.g. figure and table. We count them in each of the categories in the table.

heading: B Example of OCR and GPT-4V Caption Output
text: This section shows examples of OCR and GPT-4V for different vision features.

heading: C Data Quality assessment Guideline
text: Figure 8 shows a snapshot of the annotation guideline shared with annotators for quality assessment.
The data quality assessment annotators are authors and colleagues with bachelor’s degrees and native speakers of the corresponding language.

heading: D Fine-Grained Evaluation

heading: E Sample Example from Different languages
text: Figure 9: Example from the Arabic test set with a GPT-4V output.
Prompt: The image has a multiple-choice question with four options. Provide the accurate response in JSON format as indicated: {'answer': 'xxx'}. Replace 'xxx' with the corresponding letter: 'A' for the first choice, 'B' for
the second choice, 'C' for the third choice, 'D' for the fourth choice.
GPT4V Answer: "{'answer': 'D'}"
Correct Answer: "{'answer': 'B'}"
Figure 10: A sample from Bulgarian test set with GPT-4V output

heading: Future Work
text: In future work, we plan to extend the dataset with more image samples, subjects, languages, and modalities. Limitations
Despite its comprehensive nature, EXAMS-V, like any benchmark, is not without limitations. For ease of evaluation and analysis, we only considered and collected multiple-choice questions. We limited our multimodal analysis to four broad categories, which are scientific symbols, figures, graphs, and tabular data. But this can be further extended to finer-grained analysis. For example, scientific symbols can be further broken down into mathematical notions and chemical symbols, while figures can be broken down into maps, figures, paintings, diagrams, etc. However, this requires the collection of more data, which is difficult, particularly for low-resource languages like Croatian, Serbian, and Arabic under consideration. Furthermore, since we are collecting exam questions from different regions of the world, the difficulty of the questions varies depending on the region they originate from. This hurts the comparability of the dataset across languages. Although we tried to include parallel questions for direct comparability, but it was feasible only for three European languages: Croatian, Serbian, and Italian. Ethical Consideration
• Copyright and Licensing: All data in EXAMS-V are collected from public sources