Title: M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models

Benchmark: M3Exam
Summary: M3Exam evaluates LLMs on 12,317 multilingual, multimodal, multilevel multiple-choice exam questions from 9 languages and 3 educational stages.

Contribution: Fairly standard human exam benchmark with main contribution being multilinguality, multimodality, and separation into different cognitive levels (difficulties)
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: General intelligence across multilingual, multimodal, and educational dimensions

Phenomenon defined by authors: Yes
Phenomenon definition: "Evaluating the artificial general intelligence of large language models" through exam questions that are multilingual, multimodal, and multilevel (in difficulty)

Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Answering real exam MCQs in various languages and levels, some involving images.

Task item definition: Each item has context, question, options, correct label, and metadata eg. images.

Task definition detail: Prompts are language-specific; questions include textual and image-based items.

Task source: Human exam questions (e.g. GRE questions)
Task source detail: Official government graduation exams from 9 countries

Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 12317
Dataset size notes: 108
Dataset metadata available: Yes
Dataset metadata detail: language, level, subject, image requirement

Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Validation
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Descriptive accuracy only

Metric subscores: Yes
Metric subscores detail: Language, educational level, subject, image-based vs. text-only

Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Language Modelling / 
Alternate taxonomy: Multilinguality
New taxonomy: General Capability / Multilinguality