Title: OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
Benchmark: OlympiadBench
Summary: In this work, we present OlympiadBench, an Olympiad-level bilingual multimodal scientific benchmark, featuring 8,476 problems from Olympiad-level mathematics and physics competitions, including the Chinese college entrance exam. Notably, the best-performing model, GPT-4V, attains a low average score on OlympiadBench, with an even lower school in physics, highlighting the benchmark rigor and the intricacy of physical reasoning.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: maths and physics reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: With traditional benchmarks becoming less challenging for these LLMs, new rigorous challenges are essential to gauge their advanced reasoning abilities. 
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Given a math/physics problem, provide its solution and final answer. 
Task item definition: Problem, solution, final answer. 
Task source: Human exam questions (e.g. GRE questions)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size: 8476
Dataset metadata available: Yes
Dataset metadata detail: Type of reasoning, difficulty, image inclusion. 
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Post-processing with heuristics
Metric subscores: Yes
Metric subscores detail: Difficulty, type of reasoning.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / Mathematical
New taxonomy: General Capability / Mathematical Reasoning