Title: Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU
Benchmark: IndoMMLU
Summary: The paper introduces IndoMMLU, a multi-task language understanding benchmark for Indonesian culture and languages, which consists of questions from primary school to university entrance exams in Indonesia created by professional teachers.
Contribution: "Approximately 25% of our data encompasses nine distinct local languages and cultures
in Indonesia, namely Lampungic (ljp), Balinese (ban), Makassarese (mak), Banjarese
(bjn), Madurese (mad), Sundanese (sun), Javanese (jav), Dayak Ngaju (nij), and Minangkabau."
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: natural language understanding for Indonesian culture and languages
Phenomenon defined by authors: No
Definition scope: Comprehensive
Definition integrity: Authors' description is unclear
Definition integrity detail: Not applicable
Phenomenon contested: Not defined
Task definition: Choose the correct option of multiple-choice questions from different educational levels and subjects.
Task item definition: Question, possible options, correct option, subject, educational level, class level, URL of the source document.
Task source: Human exam questions (e.g. GRE questions)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 14,981
Dataset metadata available: Yes
Dataset metadata detail: subject and subject area (STEM, Social Science, Humanities, Indonesian Language, Local Languages and Cultures), education level (Primary School, Junior High School, Senior High School, University Entrance Exams), class level (grades 1–12 and university entrance)
and question source (URL of source document)
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall), Distribution (perplexity, calibration, correlation)
Metric definition detail: For open-souce models model access is required, for closed-source only outputs.

"For closed-source models, we evaluate questions by comparing the first generated tokens (e.g., A, B, C) and the answer key using a regular expression.7 For open-sourced models, we benchmark two strategies. Given a question and the corresponding multiple-choice options, we calculate: (1) the probability of the full generated answer; and (2) the probability of the first token in the generated answer."
Metric aggregation: Simple Mean
Metric statistics: simple mean, correlation, calibration
Metric subscores: Yes
Metric subscores detail: The paper provides detailed subscores broken down by: subject area, subject, education level (e.g., primary, junior high, senior high, university entrance), grade level (grades 1–12), presence of negation in questions
Metric access: Model access required (e.g. logits)
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes, implicitly
Author validity detail: Face: The benchmark is composed of real exams collected by professional teachers with at least a bachelor’s degree in education.
Predictive/Concurrent: The benchmark shows fine-grained accuracy declines with education level, and GPT-3.5 only “passes” at primary school level. This supports the idea that higher scores reflect greater capabilities, and thus provides evidence of predictive validity.
Content: The benchmark spans 64 tasks across five subject areas and four education levels, targeting both Indonesian and local languages/cultures
Convergent/Discriminant: no analysis of correlation with other benchmarks
Taxonomy: NLP / Understanding
Alternate taxonomy: Multilinguality
New taxonomy: General Capability / Natural Language Understanding
Procedural notes: Available on Hugging Face: https://huggingface.co/datasets/indolem/IndoMMLU