Title: OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI
Benchmark: OlympicArena
Summary: OlympicArena is a bilingual, multimodal benchmark with 11,163 problems from 62 Olympic competitions across 7 disciplines. It evaluates LLMs’ and LMMs’ cognitive reasoning abilities using both answer-level and process-level metrics, highlighting major limitations even in state-of-the-art models like GPT-4o.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Multidisciplinary cognitive reasoning in complex scientific domains
Phenomenon defined by authors: Yes
Phenomenon definition: "The ability to reason, integrate, and apply multidisciplinary scientific knowledge across text and visual inputs to solve complex problems."
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Solve a scientific problem (text-only or multimodal) using appropriate logical and/or visual reasoning, often involving multi-step deductions.
Task item definition: Each item includes a question (with or without image) and expects a structured answer (numeric, set, expression, etc.) evaluated either by matching or model-based scoring.
Task definition detail: Problems are annotated with difficulty levels (knowledge recall, concept application, cognitive reasoning) and types of cognitive abilities required (deductive, inductive, spatial, etc.).
Task source: Human exam questions (e.g. GRE questions), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task source detail: Problems collected from public Olympic-level competitions, processed into machine-readable format with solutions annotated manually or verified by models plus human reviewers.
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: Model-based evaluation set: 548 problems (OlympicArena-ot)
Dataset size notes: Validation set: 638 problems (OlympicArena-val)
Dataset metadata available: Yes
Dataset metadata detail: Problems come from 7 subjects: Math, Physics, Chemistry, Biology, Geography, Astronomy, Computer Science.
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Validation
Response format: Multiple choice, Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: By subject, by modality, by difficulty level, by logical reasoning ability, by visual reasoning ability.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The authors perform leakage detection, manual error analysis, and process-level step scoring to validate the benchmark’s difficulty and insights.
Taxonomy: Reasoning / 
New taxonomy: General Capability / Logical Reasoning