Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large
  Language Models
Authors: name: Bofei Gao; name: Feifan Song; name: Zhe Yang; name: Zefan Cai; name: Yibo Miao; name: Qingxiu Dong; name: Lei Li; name: Chenghao Ma; name: Liang Chen; name: Runxin Xu; name: Zhengyang Tang; name: Benyou Wang; name: Daoguang Zan; name: Shanghaoran Quan; name: Ge Zhang; name: Lei Sha; name: Yichang Zhang; name: Xuancheng Ren; name: Tianyu Liu; name: Baobao Chang
Published: 2024-10-10 14:39:33

Abstract: Recent advancements in large language models (LLMs) have led to significant
breakthroughs in mathematical reasoning capabilities. However, existing
benchmarks like GSM8K or MATH are now being solved with high accuracy (e.g.,
OpenAI o1 achieves 94.8% on MATH dataset), indicating their inadequacy for
truly challenging these models. To bridge this gap, we propose a comprehensive
and challenging benchmark specifically designed to assess LLMs' mathematical
reasoning at the Olympiad level. Unlike existing Olympiad-related benchmarks,
our dataset focuses exclusively on mathematics and comprises a vast collection
of 4428 competition-level problems with rigorous human annotation. These
problems are meticulously categorized into over 33 sub-domains and span more
than 10 distinct difficulty levels, enabling a holistic assessment of model
performance in Olympiad-mathematical reasoning. Furthermore, we conducted an
in-depth analysis based on this benchmark. Our experimental results show that
even the most advanced models, OpenAI o1-mini and OpenAI o1-preview, struggle
with highly challenging Olympiad-level problems, with 60.54% and 52.55%
accuracy, highlighting significant challenges in Olympiad-level mathematical
reasoning.