Omni-MATH: A Universal Olympiad Level Mathematic Benchmark for Large Language Models
Authors: Bofei Gao
Feifan Song
Zhe Yang
Zefan Cai
Yibo Miao
Qingxiu Dong
Lei Li
Chenghao Ma
Liang Chen
Runxin Xu
Zhengyang Tang
Benyou Wang
Daoguang Zan
Shanghaoran Quan
Ge Zhang
Lei Sha
Yichang Zhang
Xuancheng Ren
Tianyu Liu
Baobao Chang
Venue: ICLR 2025 Poster (2025)
Keywords: Mathematical Benchmark
LLM Evaluation
Olympic
Primary area: datasets and benchmarks
TL;DR: We propose a comprehensive and challenging benchmark specifically designed to assess LLMs' mathematical reasoning at the Olympiad level.

Abstract: Recent advancements in large language models (LLMs) have led to significant breakthroughs in mathematical reasoning capabilities. 
However, existing benchmarks like GSM8K or MATH are now being solved with high accuracy (e.g., OpenAI o1 achieves 94.8% on MATH dataset), indicating their inadequacy for truly challenging these models. To bridge this gap, we propose a comprehensive and challenging benchmark specifically designed to assess LLMs' mathematical reasoning at the Olympiad level. Unlike existing Olympiad-related benchmarks, our dataset focuses exclusively on mathematics and comprises a vast collection of 4428 competition-level problems with rigorous human annotation. These problems are meticulously categorized into over 33 sub-domains and span more than 10 distinct difficulty levels, enabling a holistic assessment of model performance in Olympiad-mathematical reasoning. Furthermore, we conducted an in-depth analysis based on this benchmark. Our experimental results show that even the most advanced models, OpenAI o1-mini and OpenAI o1-preview, struggle with highly challenging Olympiad-level problems, with 60.54% and 52.55% accuracy, highlighting significant challenges in Olympiad-level mathematical reasoning.

Published as a conference paper at ICLR 2025
OMNI-MATH: A UNIVERSAL OLYMPIAD LEVEL MATHEMATIC BENCHMARK FOR LARGE LANGUAGE MODELS
Bofei Gaot", Feifan Song', Zhe Yang', Zefan Cai², Yibo Miao*, Qingxiu Dong', Lei Li', Chenghao Ma², Liang Chen', Runxin Xu', Zhengyang Tang Benyou Wang", Daoguang Zan', Shanghaoran Quan', Ge Zhang Lei Sha " Yichang Zhang', Xuancheng Ren",
Tianyu Liu"; Baobao Chang't
Peeing University *University of Wisconsin Madison Alibaba Group
*Shanghai Jiao Tong University 'Engineering Research Center of Information Networks
"The Chinese University of Hong Kong, Shenzhen
Institute of Software, Chinese Academy of Sciences
8 University of Waterloo
The University of Hong Kong "Zhongguancun Laboratory
ABSTRACT
Recent advancements in large language models (LLMs) have led to significant breakthroughs mathematical reasoning capabilities. However, existing bench- marks like GSM8K or MATH are now being solved with high accuracy (e.g.. OpenAl ol achieves 94.8% on MATH dataset), indicating their inadequacy for truly challenging these models. To bridge this gap, we propose a comprehensive and challenging benchmark specifically designed to assess LLMss' mathematical reasoning at the Olympiad level. Unlike existing Olympiad-related benchmarks, our dataset focuses exclusively on mathematics and comprises vast collection 4428 competition-leve problems with rigorous human annotation. These prob- lems are meticulously categorized into over 33 sub-domains and span more than 10 distinct diffieulty levels, enabling holistic assessment of model performance reasoning. Furthermore, we conducted an in-depth analysis based on this benchmark. Our experimental results show that even the most advanced models, OpenAI ol-mini and OpenAI ol -peevvee struggle with highly challenging Olympiad-level problems, with 60.54% and 52.55% accuracy, highlighting significant challenges in Olympiad-level mathematical reasoning.
in Olympiad-mathematie
Github Repo
[GitHub Page]
Rule-based Repo
[GitHub Page]
Project Page
[Project: Page Leaderboard]
Dataset
[Huggingface Dataset]
OmniJudge
[Huggingfacce Model]
INTRODUCTION
Large Language Models (LLMs) (OpenAT, 2023; Abhimanyu Dubey & Abhishek Kadian, 2024; Yang et al., 2024a) have shown remarkable capabilities in producing human-like text (Abhi- manyu Dubey & Abhishek Kadian, 2024), code generation (Hui et al., 2024; Rozière et al., 2024), and dialogue (Chiang et al., 2024). Among these capabilities, mathematical ability serves as funda- mental measure of the problem-solving and complex reasoning skills of LLMs, motivating various endeavors towards improving the mathematical reasoning of LLMs (Yang et al., 2024b; Azerbayev al., 2024; Wang et al., 2024).
Projecc Lead.
1 "orresponniig Author.

---

Published as a conference paper at ICLR 2025
95.8
KAIH
85.9
Omimi-MAAT
Performance Scores
Hathstrak-8B-13
InternMa AAATH MMHH-mxtraaa-222
Deep DeepS Codent22 2odeen/2--.-eeeeee Chleere-eeenetteteeeeeeaea Whe-inuut
HuminaaAATH-72.. 728-COT
Metallama-3.11000
Claude SONNET
DeepSteek Caden-V2
GP GPT-6 77T-0-2224-08-00 08-06*
Quenz MATH "bblnstruc
green M-MAT 72blestruct
Figure I: Comparisons among different models on GSM8K, MATH, and Omni-MATH. where the models are ranked based their performance on MATH, and those marked with "x" are closed- source models. As observed, the iterative advancements of these models show that existing bench- marks are nearing saturation. Our proposed Omni-MATH introduces a challenging benchmark to further advance mathematical intelligence in large language models.
To evaluate the mathematical capabilities of LLMs, researchers have introduced various mathemati- cal benchmarks (Zhang al., 2024), including the well-recognized GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021). However, as LLMs have evolved rapidly, these benchmarks have increasingly lost their challenge (OpenAI, 2024), resulting in a diminished capacity to differentiate between model capabilities, illustrated in Figure 1. Although some efforts have been made to establish more challenging benchmarks, most prior work has not focused specifically on evaluating mathematical capabilities (He et al., 2024). In addition, the test results are closely coupled with other LLM capabilities, such as multimodal fusion, rendering the findings to be vague for understanding the mathematical reasoning improvements (Huang al., 2024).
To bridge this gap. we introduce Omni-MATH, a universal Olympiad-Ieve benchmark specifically designed for mathematical reasoning. Our dataset exclusively focuses on text-only mathematics and encompasses an extensive collection of 4,428 competition-level problems. We curate our bench- mark based on the actual selection processes mathematical Olympiads, ensuring that our dataset includes problems from both introductory competitions and professional international contests. To further explore I.LMs' performance multiple mathematical areas, we have established a hierar- chical classification of mathematical domains, including 33 sub-domains and more than 10 distinct difficulty levels (see Figure 2), allowing for a nuanced analysis of model performance across various mathematical disciplines and levels of complexity. For better evaluation, we provide GPT-4o-based model evaluation and an open-source verifier, Omni-Judge, besides the traditional rule-based evalu- ation. Our Omni-Judge provides achieves over 91% consistency with GPT-4o and 86% consistency with human judgments, providing reliable feedback. What's more, the open-source verifier provides simple and effective way evaluate the Olympiad-level mathematical problems.
We conduct a comprehensive evaluation of the currently strongest models on Omni-MATH. As shown in Table 3, it presents significant challenge the existing models. Even the most capable models OpenAl AI ol ol-mini and op-preview, renowned for their reasoning ability, only achieve an ac- curacy of 60.54% and 52.55%. while the highest score among open-source models just 36.2%. In addition, the detailed categorization of mathematical disciplines and comprehensive evaluation provide new insights into current LLMs. For instance, LLMs show a marginally greater aptitude for solving algebra, while struggling significantly with discrete mathematics. The commonly used approach for test-time scaling, Best-of-N, has proven ineffective for Olympiad-level mathematics problems. This underscores the urgent need for further research into test-time scaling techniques. To summarize, our contribution includes:
1) We introduce Omni-Math, universal Olympiad-level mathematical benchmark with over 33 sub- domains and diverse difficulty levels, posing new challenges to the problem solving and complex reasoning capability of LLMs.
2) We comprehensively examine different evaluation models. Our experiments reveal that GPT- 4o evaluation can align well with human evaluation with accuracy of 98% and the Omni-Judge
2

---

Published as a conference paper at ICLR 2025
achieves over 90% consistency with GPT-4o, providing an efficient and reliable evaluation. 3) We conducted a comprehensive evaluation of 15 LLMs and found that even the most advanced models, OpenAl ol, struggle with highly challenging problems, with 60.54% accuracy. Addition- ally, L.LMs frequently fail to solve discrete mathematics problems and often make logical missteps, highlighting significant challenges in mathematical reasoning and problem-solving for LLMss.
From Introductory to Olympion
exomole
1WOV TW0 shortis
Compotien Unancnar
US490
10hO
Cheve
USA 1ST
GMO
Questis An vencomple munte alad cormdl of afd whene tranppow coroadi
Sinar Not www.saaiiniaaaaa Nanorall Zata wad Chappis Conpantiee
USAMMO
Yol
Exngetiicnt
HMAT Nenertorr
they Care spac wi bule maae dighl be and dhwm fui he maviral pouale oeweree andl be ber tereede wr weeee in drovn
TV. Intraductory Sonpetrter
Sett's Eosce
purtion We
Hierochical Mathematical Domains
OMNH-MATH
we Fass pare tas danrlettiii danthinn can Kassor Au
britte Koriet
Inteed if
AC ouT
Methoratie
APl
A
Lace weleeeooss bulterrr vlmand of www.weooooooe mway wo thow
Protgetre
Offferent ol Cezaki
Loge Agynttes
Gruph Therry
Scld Gaorety
beffireaeel Gremerry
Irteresdorr IreresdinAAigterr
n(s+1)
Figure 2: An overall illustration of Omni-MATH. The left-top part presents the diverse and well- structured data sources of Omni-MATH. The left-bottom part represents the hierarchical mathemat- ical domains of Omni-Math. The right part presents concrete data example of Omni-Math.
Table 1: Comparison of various mathematical reasoning benchmarks. "# Data" denotes the total number of data in the corresponding benchmark, "# T.M" denotes the number of textual English mathematical reasoning data, "# T.O.M" indicates the number of textual English Olympiad-level mathematical reasoning data, "# Domain" refers to the number of domains in Olympiad-lewe math- ematical data, "# Diff" represents the number of difficulty levels, "Leak Det." signifies whether data leakage detection is performed and "Evaluator" describes the evaluation methods employed.
Name
# Data
# T.M.
# #T...MM
Domain
# Diff.
Leak Det.
Evaluation
GSM8K
1319
1319
0
X
Rule
MATH
5000
5000
0
6
5
X
Rule
JEEBench
515
515
0
5
X
Rule
CHAMP
270
270
270
5
X
Rule
Olympiad Bench
8476
675
675
3
X
Rule
MATH Odyssey
387
387
148
12
4
X
GPT-4
Olympic Arena
11163
2036
2036
/
Rule & GPT-4
Omni-MATH
4428
4428
4428
33+
10+
/
OmniJudge (Ours) & GPT-4o
2 OMNI-MATH BENCHMARK
In this section, we explore the construction process Omni-MATH. This involves data generation, manual annotation($E.. and the classification of domains ($2.3) and difficulty levels ($2.2). Fur- thermore, we will introduce the evaluation ($2.4) of the Omni-MATH, which includes GPT-4o-based evaluation and the open-source evaluator Omni-Judge.

---

Published as a conference paper at ICLR 2025
Crawl
MathPix
Difficulty Tagging Rule-based filtering
Solation
Contest Page
Problem & Solution
Human Check
Check List
1. Are the question and answer complete and fallow LateX? question answer identical to the data source?
AoPS Cruwl
Problam
Difficully Tagsing Rule sale-bbsed Eiterrin
AOPS
Solution
Problem
Reserve
Problem:
GPT-4o Reformat
Find the all the functions satisfics
AoPS
Solution Candidane
consistentt?
Solution Candidaie
Answer:
AoPS Forum
Solatinn Candidane
xxx, because...
Discard
Figure 3: The overall data collection and annotation process of Omni-MATH.
2.1 DATA COLLECTION AND ANNOTATION
First, we conducted a comprehensive review of mathematics competitions worldwide and classified them into five levels based difficulty, scale, and prestige, referencing the website AoPS Wiki: Competition Ratings and other forums containing the discussions of mathematical contests. The de- tails presented in Table 5. It is important to note that this classification is somewhat preliminary; serves only to determine which competitions we include in our benchmark, rather than providing definitive judgments the difficulty level of each problem. After obtaining these target contests, we start our data collection and annotation process, as shown in Figure 3.
Data Collection For some target competitions like IMO and IMC, we can directly crawl the pub- licly available problems and solutions from the official website. Then we utilized MathPix to convert the PDF documents of the problems and solutions into LaTeX format. Given that the provided solu- tions guaranteed to be correct, we prioritized crawling these solutions for data collection.
For sections lacking open-source solutions, we extracted data from the famous AoPS website, which includes information from both the AoPS Wiki! and the AoPS forum'. The AoPS Wiki features rec- ognized solutions, whereas solutions in the AoPS forum are often user-uploaded and may not be correct. We crawl the problem with multiple user-uploaded solutions following methods detailed in Appendix D. A rough consistency check is conducted by extracting final answers and checking whether they identical each other, leading the exclusion of any inconsistent cases. Addition- ally, cases from which fewer than three answers could be scraped were also discarded. For each data source, we applied rules for initial filtering after obtaining the data, ensuring that both the questions and responses generally adhered to the specified input and output formats.
Data Annotation After the automated con- struction of all the data, engaged team of professional annotators, comprised of graduate and doctoral students verify the solutions and the answers of the dataset manually.
Table 2: The number of data in the construction process. "#Data" denotes the number of data, "#R.Filter" denotes the number of data after rule- based filtering, and "#H.Filter" denotes the num- ber of data after human-annotation.
For the data from Contest Page and AoPS Wiki sections, which included official solutions, the verification process is relatively simple. We fo- cus on ensuring consistency with the respec- tive data sources. Conversely, we place particu- lar emphasis on the problems sourced from the AoPS Forum. Following an initial screening.
Data Source
#Data
#R.Filter
#H.Filter
Contest Page
3750
3596
3242
AoPS Wiki
8575
298
298
AoPS Forum
17502
1100
888
htnp:///mmffooobaaaaammmmmoo//et http:///mmmmooooeeemmmnnooooooom contests
4

---

Published as a conference paper at ICLR 2025
narrow the dataset down to 1,100 problems. At this stage, we engage a team of four anno-
tators to assess whether the most frequent responses from the original forum align with the final outputs produced. Each annotator reviews 550 problems, ensuring that each problem is analyzed by two different personnel for greater reliability. We employ cross-validation to enhance the robust- ness of our findings, yielding an accuracy rate of 92.7%. After excluding inconsistent cases, we conducted manual sampling of 150 entries, resulting in an improved accuracy rate of 97.3%. The overall data generation process is shown in Table 2, and additional annotation specifics can be found in Appendix C.
After the data collection and data annotation section, we obtain the overall data for Omni-MATH.
2.2 DIFFICULTY CLASSIFICATION
Through practical research, we find out that the difficulty of Olympic-level problems varies signif- icantly. To enable a comprehensive assessment of Olympiad-leve mathematical reasoning across various levels, We categorize the overall data by different difficulty levels. Relying solely on factors such the difficulty and popularity of competitions to determine problem difficulty is insufficient. To achieve a more objective assessment of the difficulty of each problem in our dataset, we consulted the AoPS: Rating" page, which provides difficulty ratings for problems from various competitions. Specifically, the difficulty is quantified on a scale from 0 to 10, including increments such as 0.5 and 0.25. For competitions listed on this page, we directly assign the corresponding difficulty score on the problem. For those competitions not included, we utilize the existing competitions, problems, and their associated difficulties on this page as a basis for in-context learning and prompt GPT-4o to assign specific difficulty ratings. The details of this prompting procedure can be found in the Appendix E.4, Following categorization, the overall distribution of difficulty levels is presented in Figure 4.
Difficitt
parnal
cajer
Hustr. 11
Casl assi, matheralion clympiad
sur slyopied
prodies from the Avant, arrce allsata globel conteett
flom Phortest
pan MG
probitenn bomal pragzaina
207
Assrican way
topmasde, ahrmpokd
uxsime
muddle toroppaan mathermatin neemmtiial ajmmysd putvpean, zn contticll ciympudd
balkan, aprn
Unter teart, lime Anngirrs
sponusgn grouagema and
chautrtoo erection test
chiak national chympiadd
Issen
difi teaes, serection, test
one
palcam
ire sherttst
Jeatr, selectionn
Figure 4: Difficulty distribution across contests.
After conducting instance-level difficulty classification, we organized the difficulty distribution by competition. Our analysis revealed that the actual difficulty distribution rankings shown in Figure 4 closely align with the tiered classifications we previously surveyed in Table 5. This consistency further validates the reliability of our domain classification.
2.3 DOMAIN CLASSIFICATION
Unlike existing benchmarks, we argue that Olympic-level mathematics competitions encompass a wide range of complex topics. The data from different domains present a greater challenge for model generalization, and it is quite likely that a single training set will yield inconsistent improve- ments across various fields. Therefore, it is essential to categorize the existing dataset into distinct mathematical domains to better investigate the model's performance across different math areas.
http://iiniiioooooooiiinoooooeen Cikiooopttinnn ratings
5

---

Published as a conference paper at ICLR 2025
Specifically, we draw upon relevant competition guidebooks (Engel, 1998; Zeitz, 2017) organize the mathematical fields into a hierarchical tree structure. This tree-based structure not only enhances the rationality of domain classification by aligning with the table of contents of reference materials but also allows us to compute accuracy metrics at varying levels of granularity.
For the domain categorization, utilized GPT-4o to classify the problems. The prompt used for classification can be found in Appendix E.5, along with the detailed classification results.
2.4 EVALUATION
GSMESK
Omani-MATH
Toulorae many Chadesson.
Problem:
Model Answer:
The snewer ic
Problem:
nemesgative inseer SoLguaWiS equation( (tSdedaa "oslad""""SS/ollaa Pheeee
Desaia (aligaed] A (1))
many sheep
Reference Arawer:
Jusedl( 0, (1.0, M. 0J. U. 13
Reforence Answer
& (3)
0),
MATII
Resssiing
Problem:
Probiem:
Model Answer:
Compute Stnninn lefi =Ifnet]1] Sight Express your answer in radiass.
"buscd(-- (61)
Find alll paslive imagen Sarigge mads thst.
+a fxx ony i
oor m- licc power of2.
Reference
Reference Answer: boood]-daa (6)1
lhosal (f2, (2"1 (1, teet for any n'ge
Figure 5: Examples of the problem, reference answer, and model-generated answer from different datasets, where the model-generated answers are all correct. On the right side, model-generated answers have issues in reformatting or requiring additional reasoning to correctly evaluate.
We discover that conducting answer-level assessments for Olympiad-leve problems is complex task due to the diverse formats of the final answers produced by models (Figure and Section H.3). This diversity makes it challenging to evaluate model outputs using fixed rules, seen in other benchmarks, and we accordingly employ model-based evaluation for the assessment of Omni- MATH. Specifically,
leverage GPT-4o to verify whether the content generated by the tested model aligns with the standard answer. In our prompt, we provide problem, reference answer, model-generated solution, querying GPT-4o to determine whether the model solution is consistent with the reference answer. Detailed prompts and model outputs can be found in the Appendix H. Additionally, in Section 4.2, we assessed the evaluation capability of GPT-4o to ensure the accuracy of our results.
To promote public accessibility for Omni-MATH, we additionally filtered subset named Omni- MATH-Rule for reliable rule-based evaluation, as in Appendix K. What's more, for total set evalua- tion, we also developed an open-source evaluation model, named Omnii-Juge,, to assess the consis- tency between model solutions and reference answers at a low cost. Specifically, we first constructed a dataset for training (~17618), validation (~~2200), and test (>~2200) based on evaluation results from GPT-4o, which have no overlaps of questions with each other. Then we treat this task similar flow of instruction following, where the extraction of the model answer, judgment, and de- tailed justification are all required to be generated by Omni-Judge. We accordingly test multiple finally evalu- ated on the consistency between its judges and those from GPT-4o, which surprisingly exceeds 90%, demonstrating the strong feasibility of Omni-Judge. Detailed analysis can be found in Sec 4.2 and Appendix J.
instruct models while appending a case in the context to boost the performances.
OLYMPIAD-LEVEL MATH EVALUATION ON EXISTING LLMS
3.1 EXPERIMENTAL SETUP
We evaluate 15 models recognized for their strong mathematical reasoning capabilities. All prompts for these models are formatted according to the guidelines provided on their respective repositories. The detailed model information and prompt specifications can be found in Appendix F. To ensure more accurate assessment, we employ GPT-4o to evaluate whether the model outputs are consistent with the correct answers.

---

Published as a conference paper at ICLR 2025
We then analyze the accuracy rates across different domains and levels of difficulty. For clarity and simplicity in the presentation, we report the accuracy rates of the first-level subdomains of the domain tree. For difficulty, categorize it into four tiers according to the difficulty tag: T1: 1-3, T2: 3-5, T3: 5-7, and T4: 7-10. Detailed accuracy statistics for each distinct difficulty level are provided in Appendix G.
3.2 EXPERIMENTAL RESULT
Table 3: Main Result. The results with bold text represent the best performance score. Qwen2.5 (AATH-22b-Instruce and OpenAl ol-mini separately perform best in the vanilla model setting and test-time scaled model setting.
Model
Acc
Alg.
P.all
Cal
Geo.
D.M.
Num.
App.
#TI
#T2
#T3
#T4
Vanilia Models
Inteml.
14.24
18.19
12:50
10.16
8.70
8.03
10.09
12.36
42.78
8.01
10.35
6.74
DeegSSeliMATHH-7...
16.12
21.28
20:45
12.50
9.87
7.71
9.98
13.58
49.07
9.11
11.49
7.80
Mathstral-7B-vO.
19.13
23.99
25.00
13.28
12.19
10.04
14.58
16.30
53.07
10.93
15.29
11.86
19.73
24.55
23.86
13.28
13.06
8.92
15.88
16.81
55.93
13.15
12.86
9.55
Metal.lamaa3.1--00
24.16
29.15
27.59
18.75
14.76
11:74
17.03
24.66
62.66
16.82
16.95
13:71
DeepSoekk-oder--2
25.78
30.24
15.62
17.99
12.71
20:90
23.58
38
14.61
30.30
19.53
15.74
26.70
18.91
17.41
SunissMATH-778-COT
34.74
21.88
16.95
25.06
23.70
21.08
Dwen2-MATH-7o-Imstee
29.36
36.08
35.23
24.22
18.68
14.41
27.04
25.93
63.52
24.30
21.52
18.54
GPT-4o
30.49
36,
39.77
21.88
21.57
15:74
25.75
29.38
68.38
25.01
21.83
15.81
Owen2.5.MATIL 7b- Instract
33.22
39.39
37.50
31.25
26.89
16.93
28.62
30.37
66.23
29.20
24.68
20.34
Dwen2-MATH-72--lmsee
33.68
40.27
37.50
27.34
22.53
17.50
30.01
32.96
70.10
29.06
24.71
17.98
Qwen2.5
36.20
43.33
42.53
39.84
26.57
18.28
34.28
33.37
70.96
31.37
27.75
22.29
Text- tist-ttime Scaled Models
Swen2.5-MATH-76..e R
35.70
42.12
36.78
33.59
31.89
18.96
29.59
30.88
67.95
31.46
27.41
24.0
Owen2.5.MATHH. In-sttcc
42.54
39.06
19.75
33.13
30.48
27.81
23.71
Qwen2.5-MATH5. S-w.:--MAA--77----ae-- 8
43,89
34.38
18.28
34.12
32.04
23.43
Wwen225-MATH-77---mmmm RM6
35.95
43.47
47.13
35.94
19.41
32.64
34.12
68.38 38
27.66
26.28
OpenAl ol preview
52.55
57.70
57.47
53.91
43.11
31.26
49.67
53.42
80.11
50.83
42.25
37.71
OpenAT ol-mini
60.54
67.82
68.18
60.94
51.50
37.68
61.74
60.52
82.23
63.10
49.11
42.69
Olympic-level mathematics remains far from solved Our findings indicate that Omni-MATH currently presents significant challenges to all models. The strongest model evaluated is OpenAI ol-mini, which utilizes test-time enhancement to achieve an accuracy rate of just 60.54%. OpenAl ol-preview attains an accuracy of 52.55%. while the SOTA vanilla model acquires 36.2%, leaving substantial gap from the top two models. It should be noted that open-source models, like Qwen2.5- MATH, have surpassed GPT-4o the field of Olympiad mathematical reasoning.
Domain-Specific Analysis Models demonstrate a stronger proficiency in domains such as algebra, calculus, and number theory while showing significant weaknesses in discrete mathematics. We hypothesize that this phenomenon stems from the prevalence the former subjects in mathematical datasets; nearly all datasets reference algebra and calculus. Conversely, datasets related to discrete mathematics are scarcer, rendering this domain particularly challenging for models.
Limitations in Best-of-N Scaling The efficient test-time scaling techniques warrant further devel- opment. We have employed the common approach the current mathematical reasoning studies, known as Best-of-N (Wang et al., 2024; Gao et al., 2024; Yang et al., 2024b), utilizing the Qwen2.5- Math-RM-72B. However, we found that scaling the inference time did not yield consistent improve- ments in performance. We propose two potential reasons: (1) The reward model demonstrates limited ability to supervise tasks related to Olympic-level mathematics. (2) The policy model strug- gles to search for the correct solutions. Moreover, with the limitations on inference tokens for OpenAl o oo-rreviww and OpenAl ol-mini set to a maximum of 4096, which is considerably lower than the RM @ 256 costs, still observe performance that greatly surpasses that of vanilla models. Thus, the development of more efficient test- time enhancement approaches remains an open area for exploration. The further discussion about the underlying problem detailed in Appendix I.

---

Published as a conference paper at ICLR 2025
4 ANALYSIS
4.1 DATA LEAKAGE ANALYSIS
To eliminate the potential impact of data con- lamination the conclusions of the experi- ments above, it is essential to conduct data leak- age detection. Following Xu al. (2024), employed n-gram accuracy to identify any data leakage present in the existing models. Specif- ically, for each sample in the dataset, we con- catenated the problem and its corresponding so- lution, then randomly selected K positions for 5-gram extraction. Whether the given sample has been contaminated corresponds to whether the 5 grams predicted by the model are identi- cal to the ground truth grams. We report the results in Table 4.
Table 4: Results of data contamination detection. (a) Contaminated; (b) Contaminated & Correct
Model
(a)
(b)
Prop.
Avg.D.
Prop.
Avg.
DeepSeekMATTH.7..R
0.23%
4.02
0.024
1.0
DeepSeeiCCoder-22.4
0.27%
4.25
0.07%
3.2
Qwen2 Banoe-MAT--7b--nttnee
0.41%
5.3
0.09%
4.12
Mathstral-7R-v0.
0.07%
0.16%
Dwen22.5-M..
0,70%
5,39
0.27%
4.54
Quen2-MATH-- Monee-MAAH-----inmmiee
0.47%
5.27
0.166
4.5
Numina MATHH-2--A--77---00
0.56%
2.46
0.34%
MetaL.lama-3 1.mmm...--0--insnnnee
GPT-to
0.04%
4.5
0.01%
4.5
It can be seen that most models exhibit certain data leakage, as Omni-MATH based on data sourced from the internet, while with 5 grams accurately predicted in 31 samples. We also collect the samples that are contaminated and then correctly answered by the tested models, finding that the numbers of the "Contaminated" and "Contaminated & Correct" samples are both extremely low. It indicates that data leakage has little impact on the conclusions before and Omni-MATH still poses significant challenges for them.
has the highest degree of data leakage
Moreover, we focus on the distribution of problem difficulty in these contaminated samples, where for most tested models, the average difficulty of "Contaminated & Correct" samples is consistently lower than that of all contaminated samples, except GPT-4o and MetaLlama-3.11 1-1-0---instuuus that almost have no data leakage. It seems that difficulty level continues exert notable influence on the performance of models even in scenarios of data leakage.
4.2 RELIABILITY OF JUDGMENT MODELS
Despite the traditional rule-based evaluation, model-based evaluation provides flexible evaluation in an end-to-end manner without the need to extract the answers and build complex rules. There are several studies utilizing model-based evaluation like GPT-4 to evaluate the correctness of an- swers (Fang et al., 2024; Huang et al., 2024), but whether such judgment models can consistently provide reliable feedback on Olympic-level problems remains inconclusive. In this part, we conduct a meta-evaluation experiment to assess the reliability of first build a subset containing 100 samples and engage several graduate and doctoral students to annotate each sample golden judgments, i.e. whether each solution is aligned with the standard answer for the given problem, as details in Appendix C.3. Next, the predictions from judgment models are compared with the gold-standard annotations. We were surprised to find that GPT-4o acquires a 98% accuracy with human annotations, demonstrating the strong reliability of both the evaluation tool and the results in Omni-MATH. Additionally, Omni-Judge also shows considerable effectiveness with an accuracy of 86%, making it cost-effective tool for quickly evaluating and iterating models.
judgments. In detail, we
Since GPT-4o shows great consistency with human annotations, we treat it as a reliable proxy of human judgment and further analyze the performance of Omni-Judge in Appendix
4.3 EMPIRICAL ANALYSIS ON DATA SELECTION
In this section, we aim to provide practical conclusions of the data selection in pursuit of a better Olympic-level mathematical reasoner. Here we develop novel approach to estimating the impact of domain/difficull distributions on the final performance, that is, for each model, we manipulate demonstrations in the input and observe variation in performance through in-context learning.
8

---

Published as a conference paper at ICLR 2025
8:30
1:4
3:29
5:1
Inqerte Donn
13
275
1:30
Prodiction Domair
Langh
Deficaty
(1)
(0)
Figure 6: Results of data selection experiments. (a) The impact of different demonstration domains on model performance. (b) The effect of the distance within the domain tree on model performance. (c) The influence the difficulty level of in-context examples on model performance.
To balance the in-context ability of the model and its mathematical reasoning ability, choose GPT-4o as the base model for this experiment. Due to cost constraints, we randomly sample 500 data from Omni-MATH conclusions drawn from our experiments.
the evaluation set for this section. The following are the details and
Domain To investigate the mutual influence of data from different domains, we retrieve data from different domains as in-context demonstrations. To balance reasoning cost and the impact of cases, select four same-domain examples for in-context learning. We present the mutual influence among 5 main domains in Figure 6 (a). For the sake of presentation, we normalize the column according the diagonal data, i.e., putting in-domain data as the baseline. From the experimental results, we observe that the mutual influence among these domains is asymmetric. For example, Discrete Mathematics has a worse influence on Number Theory, while Number Theory has a positive effect on the prediction of Discrete Mathematics. Additionally, Applied Mathematics and Algebra show a significant positive effect on Discrete Math, with an accuracy of 124% of the in-domain data. Also, Number Theory greatly supports Geometry, with accuracy reaching 150% of in-domain accuracy. Additionally, we find that Algebra interferes with applied mathematics.
Common Length The similarity between the in-context examples and the target domain is im- portant factor that may influence the model's performance. To explore this, we conduct experiments examine the impact of the fine-grained domain similarity between demonstrations and the target domain on the model's effectiveness. The results are shown in Figure 6 (b). We define domain similarity as the length of common path on the domain tree. longer common path indicates greater domain similarity, and we find that as the domain similarity increases, the model's 's perfor- mance consistency also increases. Compared to random selection, most significant improvement is brought by the same first-level domain data, suggesting that the model has the ability to generalize the in-domain data at a broader level.
Difficulty We also discussed the impact of the difficulty of in-context examples model perfor- mance. Consistent with the settings in Sections 3.1 and 4.4, we divided the difficulty into 4 intervals. We find that the difficulty of examples in the prompt increases, the model's performance gradu- ally improves, except for a slight decline on T4. Overall, we can conclude that more difficult data would help improve the model performance.
4.4 DIFFICULTY CONSISTENCY ANALYSIS
Recent studies (Yang et al., 2024c) have shown that models may exhibit the capability to solve difficult problems, yet fail to perform well on easier ones. Specifically, the accuracy solving challenging problems is not consistently higher than that of simpler ones in all cases. In this section, we examine the consistency model performance within the problem difficulty in Omni-MATH.
Our difficulty tag is obtained differently from the pairwise data construction employed in Yang et al. (2024c), so it unsuitable to directly utilize the consistency score from their study. However, measure the trend intensity of model performance the difficulty increases for consistency. Specifically, we use the four difficulty intervals in Table 3 for the overall assessment. The model performance is defined by the variables 21, 22, F3, F4, where 0 s; 100. Then define the

---

Published as a conference paper at ICLR 2025
trend intensity A by simplifying the implementation of ARIMA Modeling (Box & Pierce, 1970) and adapting it to our situation. The implementation of A with n variables is shown below:
K (xz r1+1) min( maz(x) T.41)
if F:41 > 32 if 1i41 <=
(1)
This equation measures the predicted values and their relative comparison at each position based on the first difference. We assume that decreases as i increases. For counter-trend situations, apply a penalty coefficient that is double that of positive trends. We constrain the largest impact of each position to the maz(z) = 25. For sequences of four variables with penalty coefficient of K 2, the value range of is [-200, 75). Intuitively, the more pronounced the downward trend of z, the larger the value of A; conversely, the smaller the value of A. The experimental result is shown in Figure 7.
60-
40
50
Consitency
35
Performanca
Consiatoncy
40
30
25
20
Inammcii MATH
DaseleessAt- Al
Deso Coderid Lite Inttrut
Mathorra- 7B 1
matalamaad 708 minimt
Drepltenk- Codeev2
Cloude 3.5 s-nnee
BurinaltAT moreraaaaa -ot
Ownnz Math--p instru
C-aAN..-4441-1 insroct
GPEAe
Gweet 4188 7J0 mitestt
Deent 7PP twttrst
Openk plareeeer
Openal of mini
Figure 7: Performance and Consistency Comparison of Different Models. Consistency is measured by Equation 1. The difficulty consistency represents shows that our difficulty level is generally aligned with the performance models.
From the experimental results, we observe that the consistency of all models is positive, indicating that the difficulty increases, the overall accuracy of all models declines. This finding also vali- dates the rationality of our difficulty classification. Additionally, we note that as model capabilities improve, their consistency also increases, but exceptions also exist. This conclusion is similar to that reached in Yang et al. (2024c), even though our methods of measurement are entirely different.
5 RELATED WORK
5.1 MATHEMATICS BENCHMARKS
Mcasuring the mathematical reasoning abilities of language models has long been focal point in both academia and industry. GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) are the key benchmarks in the mathematical reasoning domain. GSM8K primarily emphasizes model's capacity to solve practical application-bass math problems, while MATH presents greater chal- lenge, encompassing university-Ievel and introductory competition-levar mathematics questions. As models become more powerful, several works have proposed more challenging problems, such as OCWCourses (Lewkowyez al., 2022), SAT (Azerbayev et al., 2024), JeeBench (Arora et al., 2023), and MATHOdyssey (Fang et al., 2024). However, models rapidly improve, the challenge posed by these benchmarks is gradually diminishing, making increasingly difficult to differenti- ate their mathematical reasoning abilities. Another line of research (Zheng et al., 2022; Azerbayev et al., 2023) employs formal languages to measure the models' theorem proving capabilities.
5.2 OLYMPIAD-LEVEL BENCHMARKS
To further explore the boundaries of large language model capabilities, many studies have at- tempted to yield Olympiad-lewe data as benchmarks. Among these, AlphaGeometry (Trinh et al., 2024) offers Olympiad-level geometry problems. CHAMP (Mao et al., 2024) provides high school competition-les math problems, distinctly annotated with concepts and hints relevant and help- ful for each problem. OlympiadBench (He et al., 2024) and OlympicArena (Huang al., 2024)
10

---

Published as a conference paper at ICLR 2025
two comprehensive benchmarks that encompass Olympiad problems from various fields. How- ever, within these benchmarks, the proportion of text-only mathematical reasoning data remains scarce. Furthermore, none of the aforemennioned benchmarks have further differentiated and ana- lyzed the problems from Olympiad mathematics competitions. Our proposed Omni-MATHH focuses exclusively text-only Olympiad-level mathematical reasoning truly explore the boundaries of current LLM's mathematical capabilities. We further categorize the problems from the Olympiad mathematics competitions into various difficulty levels, providing clear and reasonable difficulty classification setting for better diagnosing model performance.
6 CONCLUSION
In this paper, introduce Omni-MATH, comprehensive and highly challenging benchmark for Olympiad mathematical reasoning. Our experiments reveal that even the most advanced model in mathematical reasoning, OpenAl ol-mini, achieves maximum accuracy of only 60.5%, while the open-source model currently achieves only 36.2%. These results demonstrate that the high- est difficulty level in mathematics presents significant challenges for large models. Additionally, we conducted an in-depth analysis of model performance based on this dataset, aiming to provide valuable contributions to the mathematics community.
REFERENCES
Abhinav Pandey Abhimanyu Dubey, Abhinav Jauhri and et al. Abhishek Kadian. The Ilama 3 herd of models, 2024. URL WBL.nttss///rrvv..rr//aaas
Mistral AL Mathstral. httppp////aattaaaaaaaeea/aaaaaas 2024. Accessed: 2024- 7-16.
Anthropic. Claude 3.5. httph//ss//ww..ecaooocccc//////a 2024. Accessed: 2024-6-21.
Daman Arora, Himanshu Gaurav Singh, and Mausam. Have Ilms advanced enough? challenging problem solving benchmark for large language models, 2023. URL abs/2305 15074.
Zhangir Azerbayev, Bartosz Piotrowski, Hailey Schoelkopf, Edward W. Ayers, Dragomir Radev, and Jeremy Avigad. Proofnet: Autoformalizing and formally proving undergraduate-level math- ematics, 2023. URL https:://aat
org/ brg/abs/////ao 2302 12433.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Al- bert Q. Jiang, Jia Deng. Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics, 2024.
htttts://aaaiiiieeee org. hbrrg//bb/2333 10631..1...1..
George EP Box and David A Pierce. Distribution of residual autocorrelations autoregressive- integrated moving average time series models. Journal of the American statistical Association, 65(332):1509-1555 1970.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Sto- ica. Chatbot arena: An open platform for evaluating 11ms by human preference, 2024. URL https::/arxx org/ abs /2033 04132.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arx: org/abs/211 /2100. 14168.
DeepSeek-AL, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi. Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang, Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bingxuan Wang, Junxiao Song, Deli Chen, Xin Xie, Kang Guan, Yuxiang You, Aixin Liu, Qiushi Du, Wenjun Gao,
11

---

Published as a conference paper at ICLR 2025
Xuan Lu, Qinyu Chen, Yaohui Wang, Chengqi Deng, Jiashi Li, Chenggang Zhao, Chong Ruan, Fuli Luo, and Wenfeng Liang. Deepseek-coder--V. Breaking the barrier of closed-source models in code intelligennee 2024. URL htttps://axii......//////// org/ abs 2206 11931.
Arthur Engel. Problem-Solving Strategies. Springer New York, NY, 1998. doi: https///doi.or/100. 1007/697682.
Meng Fang, Xiangpeng Wan, Fei Lu, Fei Xing, and Kai Zou. Mathodyssey: Benchmarking math- ematical problem-solving skills in large language models using odyssey math data, 2024. URL https://arxii org/ abs///////////// 220. 18321.
Bofei Gao, Zefan Cai, Runxin Xu, Peiyi Wang, Ce Zheng, Runji Lin, Keming Lu, Junyang Lin, Chang Zhou, Tianyu Liu, et al. The reason behind good bad: Towards a better mathematical verifier with natural language feedback. arXiv preprint arXiv:2406. 240..:2440...222.., 2024.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-lvvel bilingual mul- timodal scientific problems, 2024. URL httpthtsss//aiii....o/a httpaaaxvvvvi....... abs/ /ta.////2.... 14008.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https::///// //wxxaivvvv..// org/ abs/ /2003. 03874.
Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Rujjie Xu, Run-Zo Ze Fan, Lyumanshan Ye, Ethan Chern, Yixin Ye, Yikai Zhang, Yuqing Yang. Ting Wu, Binjie Wang, Shichao Sun, Yang Xiao, Yiyuan Li, Fan Zhou, Steffi Chern, Yiwei Qin, Yan Ma, Jiadi Su, Yixiu Liu, Yuxiang Zheng. Shaoting Zhang, Dahua Lin, Yu Qiao, and Pengfei Liu. Olympi- carena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai, 2024. URL https: //aaavaaaavvva/aaa org/ alss 2406 12753.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, An Yang. Rui Men, Fei Huang, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-codd technical report, 2024. URL https: /arxiv.orggr////............ abs/2409 12186.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman -Solo. Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with lan- guage models, 2022. URL https: /arxiv..vvvv org/ absg//bb//// 2206 14858.
Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. [https: /bttps://hugiiggiace B///AI-MMO/KuiinMMattt CoT] (https: /ttps://///thh////// wotp://oeo/ooooottnenni.aaaa numinaa htw.:ppggssspptttaa//////a////pp
2024.
Yujun Mao, Yoon Kim, and Yilun Zhou. Champ: A competition-level dataset for fine-grained analyses of lims' mathematical reasoning capabilities, 2024. URL httpht:t::////rw
abs/2401. 06961.
OpenAl. Gpt-4 technical report. 2023. URL https://api. / http::///pp..smmaticcccco...... I
OpenAl. Learning to reason with llms, 2024. learning-to-rent reaa--tt-rrsawwiiieeeeee 1lms/.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Ev- timov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usumier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code, 2024. URL https: / aarxiv org/ abs/2308 12950.
12

---

Published as a conference paper at ICLR 2025
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL htttt:s:://axii.coo wtt.:orgggaaaaoo/aaaae 03300.
Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476 48266448822000 2024.
Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations, 2024.
URL httph:ttp//// /arxivv org/ abs/ ftbs/2311 2212 2..12.1 08935.
Yihan Wang, Si Si, Daliang Li, Michal Lukasik, Felix Yu, Cho-Jui Hsieh, Inderjit S Dhillon, and Sanjiv Kumar. Preserving in-context learning ability in large language model fine-tuning. 2022.
Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. Benchmarking benchmark leakage in large language models. arXiv preprint arXiv:2404. 18824, 2024. URL httpt: /ttps://aaxxiii org/ 2404/ 18824.
Yang, Baosong Yang, Binyuan Hui, Bo Zheng. Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang. Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Ma. Jianxin Yang, Jin Xu, Jin- gren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wen- bin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang. Xipin Wei, Xuancheng Ren, Xuejing Liu. Yang Fan, Yang Yao, Yichang Zhang. Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024a. URL https: /pas://arxivv org/ abs/ /20/24072222 10671.
Yang. Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-mate technical report: Toward mathematical ex- pert model via self eel-oimprveemett,, 2024b. URL https: /ttp///axvvv//////////e/e//a.ava org/ abs abs/2400 /4099. 12122.
Zhe Yang, Yichang Zhang, Tianyu Liu, Jian Yang, Junyang Lin, Chang Zhou, and Zhifang Sui. Can large language models always solve easy problems if they can solve harder ones?, 2024c. URL https: / arxiv.. org/ abs/ 2406 12809.
Huaiyuan Ying. Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang. Yudong Wang, Zijian Wu, Shuaibin Li, Fengzhe Zhou, Hongwei Liu, Songyang Zhang, Wenwei Zhang, Hang Yan, Xipeng Qiu, Jiayu Wang, Kai Chen, and Dahua Lin. InternIm-man math: Open math large language models toward verifiable reasoning, 2024. URL https:///// arxiv. org/ abs/ /2/2/42222 06332.
Paul Zeitz. The art and craft of problem solving. John Wiley & Sons, 2017.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kei-Wei Chang, Peng Gao, and Hongsheng Li. Mathverse: Does your multi-moda Ilm truly see the diagrams in visual math problems?, 2024. URL https: hettts://arxiv..o// http://axvvvvv./..aaa// abs/
2403. 14624.
Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. Minif2f: cross-system benchmark for formal olympiad-leve mathematics, 2022. URL hwtttttt////atte org/ ppgg/bbs/1199 00110.
13

---

Published as a conference paper at ICLR 2025
A ANALYSIS OF PROCESS-LEVEL ASSESSMENT
In this section, we utilize GPT-4o for process-level assessment. Specifically, we provide GPT-4o of the question, the corresponding standard answer, and the model-generated answer the prompt. We split both the reference answer and model-generated answer into steps and let GPT-4o evaluate the correctness of each step of the model-generated answer. If a step identified we further query an explanation for the error. Following the Gao et al. (2024), we categorize the mathematical reasoning steps as follows: Unrelated: This indicates that the step irrelevant and does not contribute towards deducing the final answer. Accumulation: This denotes that the step incorrect due mistake in the preceding step, leading to subsequent errors. Calculation: This categorization is reserved for errors arising from incorrect caleulations, which is one of the most common errors in mathematical reasoning. Logic: This applies to steps that are logically flawed in the context of solving the given problem. Other: This category encompasses steps that erroneous for reasons not covered by the aforementioned categories. Then we compiled the proportions of step-level error types and the model performances, illustrated in Figure 8.
incorrect,
Error Different
GPT-46
Distribution Types
Operal 51-mhh)
OpenAi 23-pee/ew
0wer2.5 Hors instruct
Calcuistio
Accurviation
Cakulation
Loge
4.8%
85%
Other
11.44
Accumatation
Accumusation
Logic
(a)
(b)
Figure Process-level result across differnec
Logical errors represent significant issue for the models. As shown in Figure 8 (a), among the best existing inference models, most erroneous steps can attributed to logical errors, followed by accumulation emrors and caleulation errors. This highlights the challenges posed by proposed assessment set for current models.
OpenAl O1-mini demonstrates significantly fewer errors across all categories. Figure (b) illustrates the error distribution across different categories for various models. Given that the total number of errors varies by category, applied normalization. Our findings indicate that the Ope- nAI O1-mini model outperforms other models across all error categories. Notably, GPT-4o exhibits higher incidence of calculation and logical errors compared to other math-specialized models, underscoring that specialized training in mathematics is essential for Olympiad-Leve mathemati- cal reasoning. Additionally, Qwen2.5-MATHH M-ATH-72b-Instruss shows tendency to produce unrelated content compared with other models.
14

---

Published as a conference paper at ICLR 2025
B DETAILED DATA LEAKAGE INFORMATION
We examine whether the given instance has been contaminated by testing whether the model's pre- dicted 5 grams identical to the ground truth 5 grams. The detailed experimental results illustrated in the Figure 9.
Contaminated and Correct Instances of Different Models
Average difficulty Contaminated
mintami
Average Dificulty
12
235
Desponeintat R
Chees CwweetteeCCCooo CanaennCConniiii 42-
geonzmaath 75 Instrrentt
Haltatral- 78t-tt.1
Ownen Smathh Thoiattue
Owend Smatts 72b iibinsttuus
Owentnatth instract
Moninattene 220-c0
Garáo
prenitmaatt Pouhannc
Chenzz Smatts instrree
Chwen Smetth 28h instrist
Gorntmath 2ith lnstrict
HominakktH 100-
Metoslame-
Mathstral 7B
Models
Models
(a) Number of instances of different models affected (b) Average difficulty of contaminated and by contamination and the correct instances under correct instances. contaminated conditions.
Figure 9: Data Leakage and Correct Instances for Different Models.
C ANNOTATION DETAILS
C.I ANNOTATION GUIDELINE
Task Example
lnt St for.or Sin be fris ana imagen whote
2:03100
inanection net enpry, eich con-emoo 2 C
(%), by .aaa) the interection the into I't mumne
prociale
in bead seic?
Under answer area conditional judgment, new annotation content be automatically displayed conditions are met. otherwise, it will directly the next annotation.
Figure 10: The illustration of the UI page of the annotation task.
The detailed annotation tasks and principles are listed here. Specifically, the annotators were as- signed the following three tasks:
Determine whether the current problem contains multiple questions or is proof-based. If it is a multi-part question or proof question, the evaluation of this data becomes more complex, making it unsuitable for the evaluation in our benchmark; therefore, we decided discard this part of the data.
Verify the correctness of the answers present in the existing dataset. This step is essential for assessing answer quality; different verification methods are required depending on the data source. For entries from AoPS Wiki and problems with provided solutions, it suffices
15

---

Published as a conference paper at ICLR 2025
check for consistency with the answers in the data source. For data from the AoPS
Forum, it is also necessary to refer back to the results provided in the forum.
If the original answer is found be incorrect, provide corrected answer based on the data source.
C.2 ANNOTATION DETAILS
For the 1100 questions collected from the AoPS Forum, we employed annotators to verify the consistency between the final responses and the original forum replies. Each annotator labeled 550 questions, resulting in two annotations per question. We then conducted cross-validation and sampled cases for inspection. The accuracy of the cross-validation was 12..%((122/// 100), while the post-sampling accuracy was 97.3% After removing discrepancies, we obtained a total of 888 questions from this set.
For the Contest Page questions converted from PDF to LaTeX, the quality was exceptionally high. We employed rough filtering rules eliminate some errors that occurred during the conversion process. Subsequently, a manual check was performed to ensure the consistency between answers and explanations. This part resulted a total of 3295 questions.
Regarding the questions from the AoPS Wiki, comprehensive explanations were readily available. We only needed to filter out the video-based explanations and exclude questions that did not fit within our collection scope, such AMC and AIME questions. This section yielded a total of 298 questions.
After the labeling process, the initial dataset comprised 4481 questions. Subsequently, we hired 2 annotators to inspect the entire dataset, following a set principles. After a final filtration, 53 questions were removed from the Contest Page source, resulting in final dataset of 4428 questions.
C.3 DETAILS
SET
The 100 samples of the meta- evaluattall set mentioned in Section 4.2 were derived from the infer- ence results of DeepSeek-Coue Caeese--oee---ee V2. GPT-4o, and (wwen2.5-MATH-72b InI-ruutt sampled accord- ing difficulty to ensure representation across all levels of difficulty without overlapping questions. Although the inference results come from different models, our ultimate evaluation target the hu- man alignment of judgment designed for GPT-4o or Omni-Judge to provide reliable human evaluation.
and Omni-Judge Therefore, all 100 examples
Our experiments revealed that GPT-4o achieved highly accurate results under conditions of high agreement among human annotators (97% inter-agreement) given the solution generated by 3 dif- ferent models. For annotation, we employed two PhD students and two Master's students, with each annotator labeling 50 problems. Morcover, they were instructed to determine whether the model's answers correct based mostly the correct answers rather than their judgments. Each annotator was responsible for 50 questions, allowing us to ensure that each question was annotated by two annotators, facilitating cross-validation assess inter-agreement. During individual annota- tion, three out of 100 instances of inconsistent results were observed; however, after discussion, the annotators reached consensus on the answers. The diserepancies primarily stemmed from GPT-4o misinterpreting the "Student Answer" as the "Ground Truth" under the current prompt, which we attribute to prompt formatting issues. This problem was resolved by adjusting the few-shot prompt format.
D DATA CRAWLING DETAILS
To enhance the possibility of obtaining correct answers in AoPS Wiki, we specifically targeted posts that contained hidden tags indicating "solution". as well as those that included "{.boee[["" or " "I". Following LI et al. (2024), we assume that these posts have higher probability of containing accu- rate answers. Then employed GPT-4 to reformat the user-uploaded solutions into the standardized format of solution + \boxed(Final Answer) to improve the answer format quality.
16

---

Published as a conference paper at ICLR 2025
E DETAILED DATA INFORMATION OF OMNI-MATH
E.1 DETAILED DATA SOURCE OF OMNI-MATH
Table 5: The hierarchical data source of the Omni-MATH.
Contests
Number of Problems
Highly Infiuential International Competitions
IMO
75
IMO Shortlist
190
IMO Longlist
43
Putnam
101
IMC
68
Notable International/Naiinaal Challenging Competitions
USAMO
17hO
14
China TST
CMO
38
USA TST
Intermediate Internationa//aationn Competitions
APMO
85
USAJMO
Balkan MO
26
Balkan MO Shortlist
JBMO
23
JBMO Shortlits
ToT
55
Baltic Way
Alibaba global Contest
22
Middle European Mathematical Olympiad
Other Olympiads
269
National Championships or Competitions
HMMT 2
1385
HMMT-11
896
Contest
Introductory Competitions
Pascal
249
Cayley
201
Fermat
232
E.2 DIFFICULTY
The detailed difficulty distribution of the Omni-MATH is shown in Figure 11. The difficulty distri- bution roughly follows a normal distribution
E.3 DOMAIN CLASSIFICATION TREF
The overall domain tree with the problem number is shown in Figure 12. Note that the total number is larger than the total data of 4428 because the question might belong to multiple domains.
E.4 DIFFICULTY CLASSIFICATION PROMPTS
For data not included in Aops: we need to assign difficulty score to the problem. We collect all the problems with its solution and the data source. Then GPT-4o to assign difficulty level of the problem, the prompt we use is shown in Figure 13. The relevant page information makes the prompt long show this page. Instead, upload the total prompt the additional materials.
17

---

Published as a conference paper at ICLR 2025
Distribution of Question Difficulty
850
Number Questions
200
2.5
3
1.5
3.60
8.29
Difficulty Level
Figure I1: The difficulty distribution of Omni-MATH.
E.5 DOMAIN CLASSIFICATION PROMPTS
We use GPT-4o to assign domain tag to each problem, the prompt we use is shown in Figure The domain tree makes the prompt too long to show on this page. Instead, we upload the total prompt as the additional materials.
18

---

Published as a conference paper at ICLR 2025
Mathensticcs:
Mathenatics:
Appl led Mathenaticss
Geametryi
Math Word Probl 185
Plane Geonetryt
statistica:
Polygons: 307
Mathenatical Statistica:
Anglea: 137
Pecbability:
45
Counting Methods:
Triangulations:
Permutations:
Pecimeter;
Conbirat 342
Circles: 68
Other:
Others 76
Ochez: 137
Other: 1
Shapea: 91
Volume:
Aigabra:
Sucface Area: 5
Fresigebra:
other:
Integers: 200
Geonetry:
Vractions: 60
Curvature:
necinels:
Manifolds:
Simple Equations:
Geodeafes:
Other:
Other:
Algebra:
oa-Euaiidean Gecnstry:
Algebrai: Expressions: 171
Spherical secmatry:
Equatiios inequslitiess
Eyperbolic Geanetry:
Polynonial Operationes 279
Other:
Sequences and Series:
ITherys
atterndiate Algebra:
Prine Yunbera:
Functions:
Pactorization: 201
Expoaential Functions:
coagruences: 224
hogarithnic Finstions: 21
Common Divisors (GCD)
Complex Numbers: 61
Comnon Nultiples (UCN)
Insqualities:
Other: 199
103
Precalculus:
ineam Algebra:
Fuactiona: 30
Vectors:
Limits:
Matrices: 23
Trigononetric Functions: 49
Daterninants:
Cal.cu malloulss:
Isanstocmations:
Ditferential Calculus:
Other:
Decivatives:
Abstract Algebra:
Applicstione Derivatives:
Group Theerys 65
Related Rates:
Ring Theorys
Other:
Fiaid Thacry:
Integral Calcuiua:
Other:
Applications Integrais:
Other: 108
Techniques Intecration:
sscrete Nathensticss
Single-variabier
Graph Theozy:
Molti-variable: 10
combinatoriess 706 Logie: 40
Other:
Ahgorithna: 71
Other:
Other: 7
Diffcrential Equations:
ordinacy pifferential Equations (00Ee)
Fartial diffarential Equations (FDBs)
Figure The total domain tree of Omni-MATH.
F DETAILED EXPERIMENTAL SETUP
F.1 MODELS
Here are the baseline models we evaluated:
IntemLLM2-MATT--inn *22B (Ying al., 2024); DeepSeekMATH-7b..e (Shao et al., 2024);
Mathstral-7B--v.1 (AI, 2024);
(DeepSeek-Al et al., 2024);
Metalllama-331-700-instteu (Abhimanyu Dubey & Abhishek Kadian, 2024); DeepSeek-Coder-
(DeepSeek-Al et al., 2024); Chaude-3.-SSONNETT (Anthropic, 2024); NuminaMATTH-72BBOOT (LI
al., 2024); et al., 2024a); Qwwen--MATH--7b-Insste (Yang al., 2024a); Owen225-MAATHH-7--nnnnnn (Yang (Yang et al., 2024a): Dwen2.5-MATTH-72b-
(Yang al., 2024a); GPT-4o (OpenAI, 2023); OpenAI or-preveew (OpenAI, 2024); OpenAl ol-
mini (OpenAI, 2024).
F.2 MODEL INFERENCE SETUP
For all the models under evaluation, the prompts follow the instructions provided on their respective release pages, such DeepseekMATTH-7bbRR and Qwen2.5-MATTH-7h
19

---

Published as a conference paper at ICLR 2025
CONTEXT
some nigh-lovel olympiad main protiems... evaluate citfioulty math problams.. some evailabl regarding dificuuly problems:
ed friuulyy referenceo-
Esamples for dificully levels
are problems dificully ratings of compatiions.
dificully levels
(Too long to show...
cidifiuutty relerences
QBJECTIVE
Bummariza
problem
describing concepts involved math problem.
Based on the source the given problem, as wel as the dificuily the problems referenced in these materials and the solution to the current problem, please provide en overal dificuuty score the curret problem, The coore should number between and 10, with increments 0.5, and should sign perlectly with the materials.
Data report.
Professional, acientifi.
AUDIENCE
Students. Enable them to better understand the difiicity of the math problems.
RESPONSE: MARKDOWN REPORT a
tff Summanzzlion
[Summarize the math problem a brief paragraph]
de Oificutty
(Rate the dificculyy the math problem and give the reason.]
ATTENTTINN
AAdd mem report over mmef at the end of the report.
<axample moamlle math problams
(Too long to show) olexampe pile math probleme
28 Surmar zation
The probiem recuites fincing a value that makes the equation Bitraot (B Wina hitaaa/ttteeee//aaaa///aaaaa This invottes adding two fractios and deternning the equivalent fraction.
24 Diffiulty
Rating:
Reason:
This probi em straightlowward and primarlly involves basic fraction addition, making sultable for early middle school students.
seo report over =ne
<lbxample plo math problems
(Too long to show) clacample pie math problemo
28 Surmal zation
The problem aske for the possible values Sné for regulor r-sided polygon that can be completely trannulsted into isosceles tranngles using non-intereccting diagonals. The eclution involves ansyring the propertes of the diagonals forming isosceles trangles engee and deducing that $-8 can be expressed sed in temms powers ol
26 Dificulty
Ratigg?
Reasonn The problem involves undenstording propertes isasoeles trinngles the contect of polygon triangulstin and requires orticcl reasoning establish relationshipp between the number of sides and powers of making it more complex than typeal undergr aduere level problems.
report over
-math problems
[Qusetion]:
((Quussion Herol)
(Solution::
(Sooutiin Herej)
[Souree::
(Sorce Herel)
omath problemr-
Figure 13: The detailed difficulty classification prompt.
In chat models where there are no specific instructions on how to phrase the mathematical reasoning prompts, directly input the model with the problem along with system prompt: "You experienced educator in the field MATHEMATICS."
To mitigate randomness in the responses, set the parameters follows: temperature = 0, top-p 1, and a maximum 2048 tokens. For the O1 -1-evewe and O1-mini, due to constraints of inference costs, configured the maximum completion tokens 4096. Additionally, for inference with non-API models, we employed the vllm framework.
20

---

Published as a conference paper at ICLR 2025
CONTEXT
leacher, some high otympiad problems. calogoriz
math
O6JECTIVE
brief
procdem speelli mahamatical domains. clasiifiaaion examole: Mathematies Prosabilly Combinaions.
The following class Fication framewors mathematios.
cmwt domainss
(Too show, domain free herw.)
cmmath domainie-
STYLE Data report
TONE
Professional, sclendfic.
AUDIENCE
Students them uncerstend the domain protremre.
RESPONSE MARKOOWN
Summarizaion
paragroph] [Cattegorize problem speatic mathematical indluding major domains subdomaine]
[Summarize
FATTENTION
math problem
diassfication semicolonet))
Your classitiction MUST under
atorementoned suctetes:
"Oher" the comssponcing branch
Intemmecate gebir Other Only the LAST NDDE wltwad "Ohner" the preceding must striclly conform existing frmmwook. The math domain must conforn format dassiication chain, like VApiica Nathomatics is Probability o Combinations.
Add "eew report -me" at the and of the report.
eescmmpl math prostent>
(Too long show)
</example maalle math probloms-
Summarization
The problem requires finding a value that makes the equation Stiraot (99) DX ( (Mlequarres. This ivvivvs adding two fractions and determinin the equivalent fraction.
Mgobra Prealgobro Fractions;
problem-
problem-
44 Summarizatin
The problem aska for possible values of $16 reqular n-sididd polygon that can be complestaly trin gulated into isososles triang using non inter secting diagonals. The solution involvos analyzing the proporties of the diagonais tarming isosooles trlangl les and docuctng that 9i66 can be rxpressed in torms of powers of
c report dver =o:
<mall probleme-
[Queston]::
[[Cuosioo Hanol)
Sotutonn::
(Solutio Heme))
Bource):
(Bouce Hanel)
<lmath
Figure 14: The detailed domain classification prompt.
G METRICS WITH DISTINCT DIFFICULTY LEVEL
In this section, we present the fine-grained performance of the model across varying difficulty levels in Table 6. Due to the limited number of instances at certain decimal levels, such as 4.75 and 2.25, we categorize all difficulty levels into integer buckets. For example, score of 4.75 falls under the "#D5" category, while 2.25 falls under "#D3."
The results of our experiments are depicted in the figure below. We observed consistent decline in the model's performance as difficulty increased, except for the "#D10" category, which contains very few instances (only 16), leading to significant fluctuations in results.
Additionally, we note that the accuracy for nearly all model responses the "#DI" category is exceptionally high, indicating that this difficulty level does not effectively differentiate model per- formance. However, as the difficulty increases, the differences among the model performances begin to emerge. Overall, we find that the difficulty of the GSM8K dataset mainly corresponds to the diffi- culty of "#D1," while the MATH dataset aligns with the difficulty ##D2." This further underscores that proposed benchmark significantly exceeds the existing evaluation datasets.
21

---

Published as a conference paper at ICLR 2025
Table 6: Main Result with different distinct difficulty levels.
Model
Ace
#D1
#D2
#D3
#D4
#DS
#D6
#D7
#D8
#D3
#D10
Vanilla Models
DeepSceLMATH-77b.
82.53
27.77
7.37
8.51
12.07
12.61
8.58
0.00
24.16
42.92
20.05
14.99
16.38
16.01
16.00
13.75
DeepSeck-Code--22
25.78
69.06
43.93
25.23
15.28
14.39
20.94
16.56
15.95
Clande 3.5 SONNET
26.23
88.88
69.06
46.46
26.23
14.59
15.47
19.20
17.48
19.01
6.25
NuminaMATHH-22 COT
28.45
84.12
68.26
45.45
26.02
19.58
17.64
23.16
19.93
23.92
12.5
GPT-4o
30.49
88.09
70.93
51.01
32.53
20.73
18.29
23.
21.77
16.04
12.5
33.22
86.50
66.66
52.52
36.12
25.34
23.06
25.00
23.00
20.37
18.75
Qwen225-MAATTT722-mmeeee
36.20
87.30
73.6
55.55
38.89
27.19
26.98
28.33
24.00
22.5
25.0
Test-time Scaled Models
Qwen2.5-MATH-77b- RM@
35.70
37.14
28.20
26.67
27.92
24.92
23:12
25.0
Ween2.5-MATT-776--- RM 256
35.79
85.71
73.06
58.58
37.81
26.41
27.76
28.33
24.30
20.62
18.75
Qwen2.5 MATH 726- Instruct
RM@8
36.34
84.92
73.06
59.09
40.51
27.50
24,80
27.31
26.15
24.37
12.5
Owcn2.5-MATH--722 RM@
35.95
84.92
70.4
54.04
39.83
26.80
25.89
29.15
24.92
27.5
18.75
OpenAl ol preview
52.55
88.09
79.2
76.77
58.55
46.78
43.21
41.07
39.38
40.00
12.5
OpenAl ul-mini
60.54
88.09
81.55
79.79
72.70
57.98
51.08
47.03
44.78
45.39
12.5
H DETAILED EVALUATION INFORMATION
H.1 GPT-40 EVALUATION PROMPTS
Figure 15 shows the detailed prompt we use in GPT-4o based evaluation. However, the few-shot prompt is too long show in the paper. Instead, will provide the total prompts in additional materials.
22

---

Published as a conference paper at ICLR 2025
DONTEXT
anus sadics and huree sone ligh-lo ands prebkans lua tokial sidroskaaing las conotrooiaa aapked sen.c Acdiosnally ansoeri job
cipease mociing
when inproved tillren worling firmaa.
DURECTIVE
soci you jake whake: ticktakatss usavec a concet alvan de uneed mudh snvvec
Your tadu axcakk
Monnty Materetic Nocatioaal Eepilodance Pay quciat ineetoo to any reprssioon bett awsere Craffer rrafeenrtion infatooolys, caraallss specsioos comagge spicates otadat
Profoiioall extio.
ALTIENCE
Studen: Iaable tenn tear andorand viother the snever they precuce conoet
RESEPONSE: MARKDOWN REroree
Al Stalisa Frui Asswet
"[oooud)"")
di Expisacen Jelgeent
(W Inaiticaton
unnvrr dhas the rumnin with oetermee surces (TRUB (ALSE)
(Cunchade with exploniiiioi es wieh who you iefivve the aakent sanvar cereet aecmatt"
The Diprvalence Indeeraae is only millic tall Sll The ansoer TALSE ove aree exceatt iaas anower airon sorver with inrerminliee
sawer cvntanal "aouc" sookes you vaa Sepoooolly soy compatig exnen. stakefi unevec factiaacdden fxcedook sowee nacking comále inkomsliatteee
(Th king Dhiirihie)
Sesample acation
de Stadaa Anawecr
of Eipicainnn Infgonee
leswment proklen, whith wamber eoady refireeo
to caaeefs sobason they oraide only vase wieee fig deekea dlgin Hineever problee sceuate Mberical Ml Affieee fooel
repart
leng.. thao
Sheaa auch wcitioe
Iudgeriee
FALSE
The ordents anover crse that "Saaas tue orieiin drings which pot agaee mene wit vetwene anover rediccatnng "No wibning erategs: the game bet incarfetely "The rosedn proidees adiarr plasa suaretic wis bocass ganu go indediinadl willos playee pedeausnal cat stalkss contackte aaganlii croitgy
prokirak codtiione This, ordienn
oeport over
caluroee
(Toe keng... We Bai
Stador Ansovr
Tipisainnce Jnlgerent
PALSS
The stdee's doen govide fied conclaice and find answer rriwing virch indketeso leck occeciaioon
"egetinrcc Aus
[HRoknenes Nanall
Figure 15: The evaluation prompt of GPT-4o.
23

---

Published as a conference paper at ICLR 2025
H.2 CASE STUDY OF EVALUATION ISSUE
Problem:
Find all positive integer S m$ if there exists prime number S ps such that S n'milminus m) can not be divided by pS for any integer ns.
Reference
'moq
Model-generated Solution:
solve the problem, we need determine all positive integers V W) for which theroexists prime number p 'l) such that WC does **not** divide
Answer: All positive integers at least two. Every integer m 2 satisfies the stated condition.**
GPP-to-Evaluattee Evaluation:
Equivalence Judgement: TRUE,
Instification: student's unswer indicates positive integers W m Vieg satisfy given condition, which is mathematically equivalent to the reference unswer V( m Voeq W). Both answers correctly identify that V. m ()) does not sutisfy the condition, while all other positive integers do. Thus, the meaning conveyed by the student's answer aligns with the reference
Problem:
Find all nonnegative integer solutions 5(x,y,,,W5 of the equation') wwtpl//ddoooys.t//o/aoaa.
Reference Answer:
1. 22. 0, (3, 0,
Model-generated Solution:
To find all nonnegative integer solutions (((x, y. wil!) to the equation:..... Answer
\boxed!
lbegin(aligned,
& (1,1 0,1 D),
& (3, 0,1 0,1 1),
& (1A 1,
& (2. 2. 1, D).
end (uligned]
GPT--o-Evaluation:
Equivalence Judgement: TRUE,
Justification: The student's final answer, which consists of the solutions 10((1. 0, 0, 0yt), 88(3 0, O, 1)., ((( L, 1. 0)1), and (222, 2, 1)9), matches exuctly with reference unswer. tuple denotes nonnegative integer solution given equation, <[2*x\edot3"y "ii-s"//dddo'"""eee 1.3] The correspondence of each part the solution shows that the student's solution is mathematically equivalent and correctly identifies all potential solutions.
Figure 16: The case study of GPT-4o-based evalution.
In Figure 16, we selected data from two Human Annotated Datasets that represent two scenarios where rule-based evaluation impractical. However, GPT-4o demonstrates the capability assess correctly in both situations.
The first scenario reasoning. requires nuanced understanding the problem statement, which implies that m is positive integer greater than O. The correct answer is m o 1; however, the model generates the answer as m 2. Considering the defined domain for m and the constraints of the answer, we can infer that these two expressions are equivalent. Thus, certain level of reasoning ability necessary to arrive the correct conclusion.
The second scenario involves complex formatting. In this case, the order of the answers does not align with the sequence of the tuples generated by the model LaTeX, leading to inconsistencies. This makes it challenging to employ rule-based evaluations effectively.
Table 7: Distributions of answer formats in 200 problems randomly sampled from Omni-MATH.
Number
La'Tex
Tuple
Multi-La'Tex
Function
Multi-Function
Text
95
51
2
34
24

---

Published as a conference paper at ICLR 2025
H.3 DISTRIBUTIONS ANSWER TYPES
Rule-based answer extraction and evaluation were typical methods in LLM assessment of mathe- matical capacity, utilized in prior works like MATH Hendrycks al. (2021); He al. (2024); Huang et al. (2024). Nevertheless, real-world mathematical problems vary widely in domain and corresponding answer format. To better assess the capabilities of LLMs real-world problem solvers, Omni-MATH goes beyond simple types of answer formats like multiple-choice or direct number, incorporating more problems with diverse formats.
In this part, we conduct an analysis from this perspective with randomly sampled 200 problems, as shown in Table 7, where the answer formats are split into several types:
Number A numeric answer.
LaTex A LaTeX formula.
Tuple A tuple-formatted answer.
Multi-LaTex A combination of multiple LaTeX formulas.
Function custom-defined function.
Multi-Function combination of multiple custom-defined functions.
Text: A free-form explanation.
From the statistics, problems formatted in Number, LaTex and Tuple, which can easily extracted with rules, account for only 74% of the total, while problems with more complex answers also make up a significant portion, suggesting that results across these problems is crucial. To this end, the GPT-4 evaluation applied in Omni-MATH can be useful and accurate for effectively covers different formats and achieves impressive 98% consistency with human annotation (see Sec 4.2). ensures that Comi-MATH offers a thorough and fair assessment of mathematical capacity.
I FURTHER DISCUSSION ON TEST-TIME SCALING RESULTS
Table indicates that RM@256 fails to perform better than RM@8 on Qwen2.5-MATHH- Instruct, which seems counterintuitive. We find that it can be more likely attributed to the fact that the RM involved has insufficient supervision over Olympiad-level reasoning tasks than policy mod- els. Specifically, it cannot accurately select the correct reasoning path from numerous candidates. Here we demonstrate it with the following test.
We measured the performance of the policy model: (wen2.5-MMTTH--nsst at least one reasoning trace is correct, we consider the example to assess the interference in RM se- lection, we also measured the proportion of correct COTs within the passing examples. Considering the inference cost, we sampled subset of 500 instances according to difficulty to ensure that the distribution problem difficulty similar the original dataset. The experimental result is shown below:
across N samples. If
Table 8: The performance of the policy model under Pass K and the proportion of correct COT.
Metric
@
P
Pass 16
Pass@ 32
Acc
40.8%
57.6%
63%
67.2%
Correct COT Proportion
100%
68.5%
62.5%
58.5%
The results show that the inference sampling increases, the policy model is capable of solving the majority of the problems. However, even after 32 samples, 33.8% of the problems still do not yield the correct solution. The second row of the table illustrates that as the number of samples increases, the proportion of interference items also increases. In the case of 32 samples, the correct answer proportion of the passed cases drops from 100% to 58.5%. This indicates that as the sampling count increases, it becomes increasingly challenging for the reward model to select the correct COT.
25

---

Published as a conference paper at ICLR 2025
Table 9: Omni-Judge results. We implement Omni-Judge based on three base models: LLaMA- 2-7b-Chat, SEAMA-3-8b-Instrue and LLaMA-3.1-88 1--b--nsttruc,, and evaluate their performance ac- cording to the rate successfully parsing outputs and the rate of consistency between the judge of Omni-Judge and GPT-4o for each problem.
Training LaaMA-2-7b-Chat
L.aMMA-3-Sh-
LaMMA-311Shh
Model
Homology Data
Success
Consistency
Success
Consistency
Success
Consistency
MeraLlamm--3 1amma-3,1-70----1 i-700--intttcct
98.81
73.63
99.76
77.67
99.76
82.19
DeepSeck-Cder--2
97.56
38.36
100.00
93.35
100.00
94.01
Owen2.5-MATH--DT
/
98.45
35.92
99.78
89.80
100.00
90.69
OpenAl ol-previww
/
99,33
55.03
100.00
90.83
99.78
91.28
OpeaAl el-mini
/
99.78
63.11
100.00
89.56
100.00
91.78
Mathstral-7B--4
X
99.33
31.49
99.78
95.12
100.00
95.79
NuminaMATH--22-CO0
X
100.00
31.11
100.00
88.89
100.00
90.44
Qwen2.5-MATH--22--
X
98.66
37.28
99.78
91.96
100.00
93.30
Total
98.99
45.50
99.89
89.75
99.94
91.26
Omni Judge GPT-4o
59.4760
51,6952.55
39.05 36.2
6433.22
25.4324.18
28.89
25 78
29.9528
21.391913
Madhetral-7a-00
Metaliama-J 1-708 instrudt
DeepSock CodonV2
HuminaMATH 728-COT
Quan2 5 MATH Te-instruct
Owen2.5 MATH- www.::.--AAT-------i-iie 72b-nstruu
OpanAA preview
OpenAl al-mini
Figure 17: Results of predicted accuracy on different models, conducted by Omni-Judge and GPT- 4o, respectively.
J ANALYSIS OF OMNI-JUDGE
Omni-Judge proposed as an economic but effective way to provide judgment for complicated mathematical problems accompanied by reference answers and tested solutions. In this part, we test Omni-Judge on the quality of generated judgment in two aspects: whether in a precise for- mat and whether to generate correct judgment, represented by the rates of being successfully parsed (Success) and consistency rates with golden judgments (Consistency), as in Table Note that judgments from GPT-4o are seen the golden ones here. Furthermore, tion 2.4, we experiment with various I.LaMA-serie models, and with upgraded versions, the com- prehensive capabilities of models are generally acknowledged to improve. We find the performance of Omni-Judge also shows enhancements along this trend. In detail, we collect test data from several models tested in Table 3, of which also provide data for training Omni-Judge (denoted Training with Homology Data) and the rest 3 ones do not, while they demonstrate similar conclusions.
discussed in Sec-
In terms of format accuracy, the Success rates are approximately 100% across different settings, while the Consistency rates are not. Omni-Judge model trained with LLaMA-3.1 Ins--uutt shows the best Consistency with GPT-4o in total, followed by the version of --L-AA--------ttt the other hand, Omni-Judge exhibits the lowest scores based on L.AMA-2-7b--CT 2-h-hhht.... The high Sue- cess suggests that models tend to learn formats first, aligning with findings of format learning in Wang et al. (2022). However, to provide better judgments, the base models required to effec- tively capture the optimization directions during fine-tuning or possess knowledge in this domain. Therefore, the highest Consistency of Omni- Omni-Judge LLaMA-3.1-2 I-8t-rrutttt is reasonable, and we consequently use it as the default version of Omni-Judge.
On
Another significant value of mathematical benchmarks is the provided ranking, which reflects the relative capability of each subject among all tested models. Therefore, is meaningful to compare the ranking of model capabilities from Omni-Judge and GPT-40, which suggests whether Omni-
26

---

Published as a conference paper at ICLR 2025
Judge can be a practical judgment model offering reliable feedback. As in Figure 17, Omni-Judge provides outcomes similar to GPT-4o for each tested model, as well as an identical ranking of model capabilities according to the predicted accuracy, which proves its usability.
K THE CONSTRUCTION DETAILS OF OMNI-MATH-RULE
As mentioned in Section 2.4, we additionally filtered subset named Omni-MATH- Rule for effi- ciently reliable rule-based evaluation.
We began by rewriting new version of the rule-matching code using the Qwen2.5-math rule eval- uation repository, primarily utilizing SymPy. Then we performed an initial filtering of the entire dataset of problems, which done as follows:
We first applicd the rule evaluation to the results from O1-mini, ensuring that all answers were within the "boxed" format using the system prompt. We then extracted the answers using the "last boxed" method, followed by a rule evaluation, which was fully consistent with the MATH dataset.
We then compared the rule-based and GPT-4o-bases results (given that HPT--4o-based evaluation can perform over 98% consistency with human evaluation). Based on their judgments of the policy model generation, identified the following cases:
Rule-based Correct and GPT--4-based Correct: This indicates that the cepted by both Rulc-based and GPT-4o. We consider this case to have a high probability of evaluation correctness, and we labeled it as "positive case."
Rule-based False and GPT-4o-based False: This indicates that both Rule-based and GPT- 4o rejected the answer. However, the rule-based rejection could be due to generalization issues, meaning it may not align with GPT-4o's rejection. We label this "uncertain case," and we need to use strict rules to filter out only those cases that can be reliably evaluated.
Rule-based and GPT-4o-based judgments inconsistent: This case could suggest evaluation problems with either the GPT-4o evaluation or the rule evaluation. Therefore, we marked this as "inconsistent case."
The statistics of the cases above be shown as follows:
Table 10: statisties of Rule-based and GPT-4o-basee Results
Rule-based GPT-4o-based
Count
Correct
Correct
2053
Correct
False
66
False
Correct
545
False
False
1764
We found that less than 2% of cases (fewer than 66 out of 4428) were identified by GPT-4o as in- correct answers while still matching the rules correctly. We believe these instances reliably indicate evaluation errors by GPT-4o. Upon careful examination, we discovered that some cases experi- enced parsing failures, resulting both the predicted and ground truth outputs being matched as empty strings, which accounted for 30 out of the 66 cases. After identifying this issue, removed these cases from the testable dataset. Consequently, the actual number of instances categorized as "Rule-based: Correct and GPT-4o-based: False" is reduced 36. Among these, several remain contentious; for example, GPT-4o indicated that a condition was omitted in the answer.
We then added the "positive cases" and "uncertain cases" the testable set which indicates the cases are reliable with based evaluation. The "inconsistent cases" were placed directly into the untestable set.
We applied strict rules (using regular expressions) to filter the testable set, removing cases that were more complex, such open-ended text, multi-part LaTeX, function-based problems, multiple
27

---

Published as a conference paper at ICLR 2025
functions, retaining only simpler cases that the rules could match effectively, which can filter reliable cases in "uncertain case" as much as possible.
This filtering process resulted in 2925 problems suitable for rule-based evaluation. We then em- ployed two PhD students to annotate the rule-based evaluated data. The task was to classify the type of answer based the aforementioned experiments (number tuples latex and on) and deter- mine whether could be evaluated by the rule system. We ensured that each problem was annotated 98% (2869/2925). Inconsistent cases were
by two students, and our cross-validation accuracy placed in the untestable set. This annotation process removed some tuples that were still unsuitable evaluation and some complex LaTeX cases. Ultimately. We ended up with testable set of 2821 problems and an untestable set of 1607 problems. We analyzed the difficulty distribution of the testable set, which is basically consistent with the distribution of our entire dataset.
To further validate the reliability of the testable set and the rule evaluation, we evaluated some models on the current subset. The rankings and the acc of the models were consistent with the evaluation results from GPT-4o.
Table 11: The leaderboard of total accuracy of Omni-MATH-Rule using rule-based evaluation and Omni-MATI using GPT-4o evaluation.
Model
Acc @ Rule 2821
Acc @ GPTT-4 4428
ol-mini
62.2%
60.54%
ol-preview
51.7%
52.55%
Bwen2.5-MAATH-72--nmstte
35.7%
36.20%
gwen22.5MAAA7H--b-nnstncc
32.3%
33.22%
GPT-4o
29.2%
30.49%
NuminaM/ATHH-22b-cod
27.1%
28.45%
DeppsekMMATH-7bb-R
14.9%
16.12%
28

---

Published as a conference paper at ICLR 2025
L
DATA EXAMPLES OF OMNI-MATH
Domain: Mathematics Algebra Abstract Algebra
Group Theory
Source: china national olympiad
Difficulty:
Question
Let Sp$ be prime. arrange numbers $/M(122, _p \/5 matrix SA= (ij )S. Next in it, subtract SIS from every number in We call the arrangement [i]good[[i] if we can change every number of the matrix to SOS in a finite number of such moves.
a Sp ltmmss pS select any row column and add to every number
low many good arrangements are there?
Solution
We call the arrangement 'good' if we can change every number of the matrix to in finite number moves. aim determine number good arrangements. Assume start with good matrix and let and be the numbers of operations ("numbers add one" minus 'numbers subtract onc') row and j-th column, respectively, to get zero matrix.
Then should have:
ag +
Summing up all these equations, we obtain:
Σ - +1
result, implies the of numbers that lie Auorts A.. constant Sp. By comparing these equations, we find that the first now the matrix justa translation the second row, Le, Au Agi constant for p. This true for any other well. Thus, problem boils down to finding Og. (1,2, ) and =x, x.... 1--1 e Z such that:
R, (m) 02 9p.
represents the i-th of matrix M for of generality, we can assume a; either Y; = (first column (1,2, (1,2,...,p) p]]
representing partition of (1.2)..., p²). Without loss xy-1- We try to prove that
and 0 xy < (first (1, .1, n)). If and
then
neither the first row nor any other rows contain We have two cases to consider: 1. straightforward, xy then there
1: This way,
possible position for we find = =i-1. 2. 0z =: Let R be the greatest number such that a, for all I s i k. Then where 1,2..... ........ Playing around possible positions, we
Continuing in the
x
1.....2.
at the matrix:
2
k
2k
2
3k 4k 5k+1
k+2
2k
3k+1
3k+2
M
matrix implies p. which contradiction since p prime. Therefore, cases, we suppose that aj 1. This argument works symmetrically for columns as well. Henoc, the number of good arrangements
2(p
The answer 2(p
Answer:
The answer is: 2(p!)²
Figure 18; An example from China National Mathematical Olympiad (CMO) in Omni-MATH.
29

---

Published as a conference paper at ICLR 2025
Domain: Mathematics Algebra Algebra Polynomial Operations
Souree: USAMO
Difficulty:
Question
P(x) is polynomial of degree 3n such that
P(D)
P(3n)
P(1) P())
P(3n 2)
P(2) P(5)
P(3n - 1) 0, and
P(3n+1) =30
Solution
By Lagrange Interpolation Formula f(x) - Expo ( Spitt ) Ejst [ooarttp-ste Sp-2-r)
and hence /(3n 22, [llosrapp53: Sm-ler] 2pp [Ioseess- 2536 3/-2-4] after calculations
get f(3n +1) ((00)) (()))))) (2(-1)a Given (3e+ 730
we have to find such that ((")")) (3)1) -(")) (2-11)h- =22
Lemma: even (g) (5) -.... (=(e0(()))0((((1)) and is odd ()) (5)
-2Priaa (ar= v
Using abowe Jemmas de not any solution when add, but when 3n satisfies the required condition, hence n 4
Answer:
Figure 19; An example from USA Mathematical Olympiad (USAMO) in Omni-MATH.
30


Reviews:
[{"review_id": "9HAHGjxRJX", "reviewer": "Reviewer_ssE3", "summary": "This work constructs Omni-MATH, a challenging math problem set of 4k over more than 33 domains and with more than 10 difficulty levels with difficulty information comes from either the AoPS website or GPT-4o few-shot result. 15 large language models are evaluated on Omni-MATH and it turns out be challenging even for the strongest OpenAI-o1-preview model. Further analysis shows that all models receives certain data leakage and GPT-4o is a reliable judgement model for determine whether the model-generated solution is consistent with the reference answer.", "questions": "- What are the education background of the graduate and doctoral students getting involved?\n- In Section 4.2, what the sampling strategy of the 100 subset, are all the difficulty levels covered in this subset?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- A new benchmark contribution that demonstrate the limitation of current progress of math reasoning in large language models, the detailed difficulty information would benefit the further categorization of this benchmark.\n- Careful study over the reliability of the metrics including data leakage and LLM-based judgement results.", "weaknesses": "- The explanation of the RM-version of Qwen2.5-MATH do not outperform the vanilla version and Qwen2.5-MATH RM@256 does not out perform Qwen2.5-MATH RM@8 needs further elaboration and investigation to unveil the phenomenon.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A15aOeZ598", "reviewer": "Reviewer_4CQJ", "summary": "This paper introduces OmniMath, a new Olympiad-level mathematical reasoning benchmark. It comprises 4,428 math problems, each with an instance-level difficulty classification. The authors utilize GPT-4o as a judge to evaluate the performance of LLMs. Additionally, they have developed Omni-Judge, an open-source answer evaluation LLM designed to serve as a substitute for GPT-4o. The paper conducts some analysis of experiments involving 15 different LLMs.", "questions": "- Regarding data annotation: Could you clarify what is meant by cross-validation in line 203? Moreover, in L831-834, it states, \"Subsequently, we hired 2 annotators to inspect the entire dataset, following a set of principles.\" What are these principles?\n\n- In Table 3, what do the notations \"Prop.\" and \"Avg.D.\" stand for? It seems these are not explained in the caption or main text. Additionally, how should we interpret the results for Qwen-2.5-MATH-instruct? Is the data leakage problem particularly severe here? Since some training sets for NuminaMath and the Qwen-MATH series are derived from AoPS, is there any overlap between this subset and OmniMath?\n\n- Lines 398-400 omit crucial details. For instance, did the 100 samples originate from the same LLM? What are the inter-agreement scores among the human annotators? How many student judges were involved? Are there specific examples where GPT-4o fails as a judge, and what are the reasons for these failures?\n\n- In Section 4.3, you mention investigating few-shot evaluation. Are you evaluating LLMs on OmniMath in a few-shot manner? If so, where are the prompts used for this evaluation?\n\n\nOthers \n\n- L79: should be \"shown in Table 2\". \n\n- L241-L242: we previously surveyed in Figure (where is Figure 4?)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The dataset's large size and focus on Olympiad-level problems make it a potential resource for evaluating LLMs' mathematical reasoning abilities.\n\nThe investigation into best-of-N scaling and difficulty consistency represent current areas of interest, and the analysis of these part is interesting.\n\nThe paper is well-organized and in good shape.\n\nThe experiments are comprehensive, and the analysis is thorough.", "weaknesses": "The major concern is about the evaluation. While the authors assert that GPT-4o aligns with human annotations in 98% of cases (L398-399), crucial details are omitted (e.g., whether the 100 samples originated from the same LLM? What are the inter-agreement scores among the human annotators? What is the number of student judges involved? Are there specific examples where GPT-4o fails as a judge and why?) undermining the reliability of this claim.\n\nMoreover, there is a notable contradiction with the results attributed to OmniJudge. According to L74-76, \"Our Omni-Judge achieves over 91% consistency with GPT-4o and 86% consistency with human judgments, providing reliable feedback.\" Let us consider this carefully: suppose we have 100 test examples, and GPT-4o achieves alignment with human judges on 98 of them. With OmniJudge, 91 examples align with GPT-4o, implying that at most only 2 of the 91 might be judged incorrectly by OmniJudge, which would result in at least 89 out of 100 examples aligning with human judges. This seems contradictory with the reported 86% consistency with human judgments. If the results reported for OmniJudge are indeed reliable (as they seem to be, given the detailed information provided in the appendix), then GPT-4o aligns with human judgments at most 95%. This discrepancy calls into question the reliability of the results presented in Table 2.\n\nIn addition, using GPT-4o or OmniJudge as evaluators can be time-consuming and computationally (or monetarily) expensive. A more feasible approach might involve standardizing answer formats, as seen in datasets like MATH, and designing detailed, rule-based answer-cleaning and matching protocols. Model-based judging should primarily serve as a supplementary tool (like Olympic Arena) rather than a primary evaluation method.\n\nThere is also the potential for data leakage, as some models, such as NuminaMath and Qwen2Math, use examples from AoPS as part of their supervised fine-tuning datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T52e12iVgk", "reviewer": "Reviewer_1ndo", "summary": "This paper introduces Omni-MATH, a challenging benchmark designed to assess large language models (LLMs) in mathematical reasoning at the Olympiad level. Omni-MATH contains 4,428 competition-level problems, rigorously annotated across 33 sub-domains and ten difficulty levels. This comprehensive categorization enables a nuanced evaluation of LLM capabilities in complex mathematics.\n\nThe study reveals that existing models, including advanced ones like OpenAI o1-mini, achieve only moderate success on this benchmark, with a maximum accuracy of 60.54%. The findings highlight persistent challenges in areas like discrete mathematics, while models perform relatively better in algebra and calculus. To support this benchmark, the authors introduce Omni-Judge, an open-source evaluator achieving high consistency with human judgment (86%) and GPT-4o (91%).", "questions": "Omni-judge is trained with GPT-40 evaluation results, but how to verify if the GPT-4o evaluations are reliable? \nCan the solutions for discrete math problems be written in code and executed to achieve accuracy?\nWhat are the best-of-1 results? \nWhat is the exact number used for  \"N\" in Best-of-N?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Omni-MATH targets Olympiad-level mathematics, pushing beyond traditional datasets like GSM8K and MATH, which have reached saturation. This focus on advanced problems adds a valuable, high-difficulty benchmark for assessing and improving LLMs' reasoning abilities.\n\nProblems are categorized into sub-domains and assigned difficulty ratings, allowing for fine-grained performance and error analysis. This detailed structure aids in identifying domain-specific strengths and challenges in model reasoning.\n\nThe study reveals limitations in test-time scaling methods (e.g., Best-of-N) and the impact of domain interactions on model performance. Such findings guide future research on improving LLMs’ mathematical reasoning and scaling capabilities.", "weaknesses": "More examples of the dataset should be presented in either main paper or appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fmHpnik6rV", "reviewer": "Reviewer_FqD3", "summary": "This paper introduces *Omni-MATH*, a new Olympiad-level mathematical benchmark designed to evaluate the performance of advanced models on competition-level problems. The authors assess various strong models on this benchmark and share insightful findings from their analysis. Additionally, they propose an Olympiad-level verifier to assist in confirming the accuracy of predicted and ground truth answers. Unlike simpler datasets like gsm8k or MATH, Olympiad-level problems require more sophisticated verification techniques beyond rule-based methods due to their complexity.", "questions": "See weakness", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, I find this to be a strong paper.\n\n- **Clarity**: The writing is clear and easy to follow.\n- **Comprehensive Benchmark**: This paper introduces a comprehensive and in-depth Olympiad-level mathematical benchmark, featuring questions across various domains and difficulty levels.\n- **Model Analysis**: It provides a thorough analysis of current advanced models on this benchmark, offering valuable insights into aspects such as Best-of-N and consistency.\n- **Effective Verifier**: A notable contribution of this paper is the development of an effective verifier, which improves the accuracy of validating predicted answers, addressing the unique challenges of Olympiad-level problems.", "weaknesses": "Here are some concerns about this paper:\n\n- **Verifier Training Details**: The details regarding the training of the verifier should be further clarified.\n- **Scaling Test-Time Inference**: In the test-time inference analysis, this paper primarily relies on the best-of-N strategy. While this is generally fine, for more challenging competition-level questions, advanced search algorithms—such as tree search or Monte Carlo Tree Search (MCTS)—may be required. A discussion on these more sophisticated techniques for tackling harder questions would enhance the paper's impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)