Title: EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models
Benchmark: EXAMS-V
Summary: EXAMS-V is a multi-discipline multimodal multilingual exam benchmark for evaluating vision language models.  The questions come in 11 languages across 20 school disciplines. The evaluation results demonstrate that this is a challenging dataset, which is difficult even for advanced vision–text models such as GPT-4V and Gemini.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Multilingual and multilingual model’s multitask accuracy (knowledge acquired during pretraining) across a diverse set of subjects

Phenomenon defined by authors: No
Phenomenon definition: For LLM evaluation, standardized testing akin to school examinations has proven to be an effective measure of a model’s capabilities. 
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: The task is a multiple-choice question answering task. 
Task item definition: language, subject, grade, question, choices, answers, image, image type
Task source: Human exam questions (e.g. GRE questions), Modified from another benchmark (e.g. translation into another language)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 4,800
Dataset size notes: 16,500
Dataset metadata available: Yes
Dataset metadata detail: language, grade, subject
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: subject, language
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: VQA / 
New taxonomy: General Capability / Multilinguality