Challenging the Boundaries of Reasoning: An Olympiad-Level Math
  Benchmark for Large Language Models
Authors: name: Haoxiang Sun; name: Yingqian Min; name: Zhipeng Chen; name: Wayne Xin Zhao; name: Zheng Liu; name: Zhongyuan Wang; name: Lei Fang; name: Ji-Rong Wen
Published: 2025-03-27 11:20:17

Abstract: In recent years, the rapid development of large reasoning models has resulted
in the saturation of existing benchmarks for evaluating mathematical reasoning,
highlighting the urgent need for more challenging and rigorous evaluation
frameworks. To address this gap, we introduce OlymMATH, a novel Olympiad-level
mathematical benchmark, designed to rigorously test the complex reasoning
capabilities of LLMs. OlymMATH features 200 meticulously curated problems, each
manually verified and available in parallel English and Chinese versions. The
problems are systematically organized into two distinct difficulty tiers: (1)
AIME-level problems (easy) that establish a baseline for mathematical reasoning
assessment, and (2) significantly more challenging problems (hard) designed to
push the boundaries of current state-of-the-art models. In our benchmark, these
problems span four core mathematical fields, each including a verifiable
numerical solution to enable objective, rule-based evaluation. Empirical
results underscore the significant challenge presented by OlymMATH, with
state-of-the-art models including DeepSeek-R1 and OpenAI's o3-mini
demonstrating notably limited accuracy on the hard subset. Furthermore, the
benchmark facilitates comprehensive bilingual assessment of mathematical
reasoning abilities-a critical dimension that remains largely unaddressed in
mainstream mathematical reasoning benchmarks. We release the OlymMATH benchmark
at the STILL project: https://github.com/RUCAIBox/Slow_Thinking_with_LLMs.