UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models
Authors: Xin Xu
Jiaxin ZHANG
Tianhao Chen
Zitong Chao
Jishan Hu
Can Yang
Venue: ICLR 2025 Poster (2025)
Keywords: Math Reasoning
Undergraduate-Level Math problems
Benchmark
Primary area: datasets and benchmarks
TL;DR: We introduce a diverse and dynamic Benchmark for undergraduate-level mathematical reasoning with large language models

Abstract: Large Language Models (LLMs) have made significant strides in mathematical reasoning, underscoring the need for a comprehensive and fair evaluation of their capabilities. However, existing benchmarks often fall short, either lacking extensive coverage of undergraduate-level mathematical problems or probably suffering from test-set contamination. To address these issues, we introduce UGMathBench, a diverse and dynamic benchmark specifically designed for evaluating undergraduate-level mathematical reasoning with LLMs. UGMathBench comprises 5,062 problems across 16 subjects and 111 topics, featuring 10 distinct answer types. Each problem includes three randomized versions, with additional versions planned for release as the leading open-source LLMs become saturated in UGMathBench. Furthermore, we propose two key metrics: effective accuracy (EAcc), which measures the percentage of correctly solved problems across all three versions, and reasoning gap ($\Delta$), which assesses reasoning robustness by calculating the difference between the average accuracy across all versions and EAcc. Our extensive evaluation of 23 leading LLMs reveals that the highest EAcc achieved is 56.3% by OpenAI-o1-mini, with large $\Delta$ values observed across different models. This highlights the need for future research aimed at developing "large reasoning models" with high EAcc and $\Delta = 0$. We anticipate that the release of UGMathBench, along with its detailed evaluation codes, will serve as a valuable resource to advance the development of LLMs in solving mathematical problems.

Published as a conference paper at ICLR 2025
UGMATHBENCH: A DIVERSE AND DYNAMIC BENCH- MARK FOR REASONING WITH LARGE LANGUAGE MODELS
MATHEMATICAL
Xin Xu", Jiaxin Zhang", Tianhao Chen", Zitong Chao, Jishan Hu, Can Yang
Department of Mathematics
The Hong Kong University of Science and Technology { [xxuca, jzhangat, tchenbb, zchaoaa @connect. ust. hk (majhu, macyang) @ust.hk
ABSTRACT
Large Language Models (LLMs) have made significant strides in mathematical reasoning, underscoring the need for a comprehensive and fair evaluation of their capabilities. However, existing benchmarks often fall short, either lacking extensive coverage of undergraduate-leved mathematical problems or probably suffering from test-set contamination. To address these issues, introduce UGMathBench, a diverse and dynamic benchmark specifically designed for evaluating undergraduate- level mathematical reasoning with LLMs.. UGMathBench comprises 5,062 prob- lems across 16 subjects and 111 topics, featuring 10 distinct answer types. Each problem includes three randomized versions, with additional versions planned for release as leading open-source LLMs become saturated in UGMathBench. Further- more, we propose two key metrics: effective accuracy (EAcc), which measures the percentage of correctly solved problems across all three versions, and reasoning gap (A), which assesses reasoning robustness by calculating the difference between the average accuracy across all versions and EAcc. Our extensive evaluation of 23 leading LLMs reveals that the highest EAcc achieved is 56.3% by OpenAI- ol-mini, with large A values observed across different models. This highlights the need for future research aimed at developing "large reasoning models" with high EAce and A 0. We anticipate that the release UGMathBench, along with its detailed evaluation codes, will serve valuable resource to advance the development of L.MMs in solving mathematical problems. Codes and data are available at https://githubb.co. waapclhuubbooom/aaaa YwggLobKUUS/UUGMMbbBaa...
INTRODUCTION
Mathematical reasoning and groblem-solvingg are critical components of human intelligence, and the ability of machines understand and address mathematical challenges is crucial for their deployment
(Ahn et al.) 2024 Liu et al. 2024 He et al. 2024a).. Solving mathematical problems with machines
has been significant research topic in natural language processing since the 1960s (Bobrow et al.
1964), initially focusing on elementary math word problems (Patel et al. 2021 Wang et al. 2017
Ling al. 2017 Welbi al. 2017 Cobbe al. 2021). With the advent of Large Language Models
(LLMs) (Brown et al.] 2020 Ouyang et al.] [022] OpenAl 20233 Team et al.) 2023] Anthropic
2024). interest in using these advanced technologies to solve math problems has continued grow.
Researchers are exploring various approaches to improve the mathematical reasoning capabilities
of LLMss, including prompting (Wei al. 2022 Wang al.) 2023b: Kojima al. 2022 Zhang
et al.
2023 Zheng et al. 2023; Xue et al. (2024), supervised tunentuning ( Yuu et al. 2023 Yu et al.
2023
Gou al. 2023
et al. [202aa] Xu et al., 2024b) Tong et al.) 22244] Yan et al.) [024]] Xu
al.
2025b Yan et al. 2025) and continued pretraining (Lewkowycz et al. 2022
Shao et al. 2024
* Equal contribution.

---

Published as a conference paper at ICLR 2025
Undergraduate--Lvee Mathematics
Evaluating True Reasoning
16 Subjects
111 Topics
583 Sub-Topics
i-th example
Types
Answer
Bttmmic Types
Diverse
Randomized Versions
more versions
M = 1,2,3
AAce
2 Compound Types
in the future
Reasoning Cap
Distribution across different subjects and levels
Data Example
UGMathBench
Subject: virgle-variable calculus
Topic: application of diff differectiaiion
Dynamic
K.P. Linear approximation and differestials Keywords Calculus, Derivatives, Linsur A cnproximation, Trignnomeric Functions
Calcults Stagle Vanable
373
Ok cultt--utttmmaaae
Algebrs
Unoor Alpelozz
Problem vt: Pind the linear approrimation at Sx-09 to
statatic
Probsbilty
Write your answer ez in the forn Sy
Fmvacial Nothmalkss
Answer_ v1: y-0.02209911 = 0,353555
Type
Difkorotal Irirommetty
Problem v2: Find the linear approcimation at Sx to
Gazmettyy
Strrac(1] (eigggss (2xx))(2())) ()s Write your answer in the forn sy=Ax+B5
Conkrateks
53
Answer y-0,1 167777 =0, 0,707107 Type: EQ
Set Theory and Logic
Comples Aralysis
Problem v3. Find the linear approsimation Sx
Nurber Theary
Lecel 5
stifra (1) [[oggr((4xx)] (4xx ]J8. Write your answer in the form Sy=Ax+B$
Abotract Nlgebra
Lanad 6
Answer 20.006255xx
Type EQ
202
603
1000
Total Guections
Difficulty Level
Figure I: Overview of UGMathBench. UGMathBench is a diverse and dynamic benchmark specif- ically designed for evaluating undergraduate-level mathematics with LLMs, covering 16 distinct subjects and featuring 10 different answer types. Each problem contains three randomized versions, with EAcc and A rigorously assessing LLMs' true reasoning skills.
Azerbayev et al. 2023). Consequently, LLMs have become increasingly capable of solving complex mathematical problems (Hendrycks et al.
2021b Ahn et al. 2024
With the rapid advancements in LLMs, evaluating their reasoning capabilities has become increasingly important (He et al.] 2024a Xu et al. 2025a| Phan et al., 2025), especially in mathematics (Liu
al., 2024 Gao et al. 2024). Although benchmarks such as GSM8K (Cobbe al.] 2021 and
MATH (Hendrycks et al. 2021b) are commonly used to assess these abilities, they are becoming insufficient due to rapid progress in model performance, as evidenced by accuracy exceeding 97%
in GSMSK (Zhou et al. 2023) and 94.8% in MATH (OpenAl 2024b). While more challenging benchmarks are being introduced (Tang et al., 2024 Liu et al. 2024). they often remain limited mathematics, which is essential due to its breadth and complexity (see Table Moreover, there are growing concerns about test set contamination in these static benchmarks (Srivastava al. 2024 Zhang et al. 2024 Qian et al. 2024 White et al. 2024). Recent efforts Qian et al. (2024); )) Srivastava el al. (2024) have introduced dynamic benchmarks by functionalizing the original problems in GSM8K and MATH to generate randomized variations through variable disturbance. However, these initiatives primarily focus on elementary and competition-level mathematies (see Table I]. These limitations highlight the pressing need for a comprehensive and dynamic benchmark specifically designed to assess undergraduate-level mathematical reasoning.
in size and scope regarding undergraduate-
In this paper, we present UGMathBench, a diverse and dynamic benchmark designed to evaluate the mathematical reasoning capabilities of LLMs across wide range of undergraduuae--eve mathemati- cal topics, as illustrated in Figure [ We meticulously collect, clean, and format undergraduate- mathematical problems from our online homework grading system (see Appendix [BI). (B.... resulting in a benchmark comprising 5,062 problems in 16 subjects, categorized into eight atomic answer types and two compound answer types. A key feature of UGMathBench is the inclusion of multiple randomized versions for each problem, which aids in assessing the true reasoning abilities of LLMs through the Effective Accuracy (EAcc) and reasoning gap (A) (see Section 3.3). EAce represents the percentage of problems correctly solved across all versions, providing insights into intrinsic reasoning skills. It operates on the premise that a model capable of solving a problem through reasoning should also be able to solve all its variants under variable disturbance (Srivastava et al.) 2024 Qian et al.) 2024). The reasoning gap. A. is defined as the difference between the average accuracy across all versions and the EAce, quantifying the robustness of reasoning when the original problems undergo slight
2

---

Published as a conference paper at ICLR 2025
modifications. These metrics help mitigate the impact of potential test set contamination (Deng et al. 2024, Dong et al. [2024] Golchin & Surdeanu] [2023] Roberts et al. 2023) and ensure a more rggorous evaluation of LLMs' mathematical reasoning abilities. These features are summarized more clearly in Figure and Table
We conducted extensive evaluation of the leading LLMs, including proprietary models such as OpenAl-ol (OpenAI (024b) and open-source models like LLaMA-3-Instruce (AI@ Meta (024). Despite their advanced capabilities, the best EAcc achieved is 56.3% by OpenAT-ol (OpenAl 2024b) and all LLMs exhibit a large reasoning gap. These results highlight the considerable challenges that UGMathBench presents current LLms terms of mathematical reasoning, underscoring the need for future research focused on developing "large reasoning models" characterized by high EAcc and a reasoning gap A
summarize our key findings:
1. Even LLMs with the most advanced reasoning ability. OpenAl-ol-mini, achieves a 56.30% EAcc on UGMathBench, much lower on other text-only mathematical benchmarks. All LLMs evaluated exhibit high reasoning gap with Robustness Efficiency (RE, the ratio between A and EAcc) ranging from 20.78% to 196.6%, pinpointing the inconsistencies of current LLMs in solving problems with variable disturbance. 3. There remains significant discrepancy among closed-source LLMs and open-source LLMs (even specialized mathematical LLMs). Among open-source LLMs, only Dven-2-Math--77 -antt-22B-nnttret and Histral-Larg' Instruct have comparable performance with GPT-4o. The average EAce varies by subject, with Arithmetic scoring 62.8%. In contrast, Abstract Algebra, Differential Equations, and Financial Mathematics have average EAces of less than 10%. 5. An error analysis of poonAl--o-ming performance reveals that calculation errors are a major concern. Even the same problem presented different randomized versions can lead varying types of errors.
Table 1: Comparison of various benchmarks. "#Types" indicates the number of answer types in the dataset. "#Subjects" specifies the number of mathematical subjects covered. "Dynamic" denotes whether the dataset is dynamic or static. "#Test" shows the number of test examples in the dataset, while "#College" refers the number of test examples at the college level.
Dataset
Level
#Types
#Subjects
Dynamic
#Test
#College
GSM8K
Elementary
1
1,319
MATH
Competition
3
5,000
MMLU-Math
All
844
116
TAL-SCQ
K12 Math
1
1,496
AGIEval-SAT-Man
High School
2
102
AGIEvvl-Math
Competition
2
938
MallegeMath
College
2,818
2,818
MathBench
All
5
1781
466
GSMIK
Elementary
1,250
FN-EVAL
Competition
3
7
2,060
VarBench- -athh
Elementary
1.319
LiveBench-Math
Competition
2
232
UGMathBench
College
10
5,062
5,062
RELATED WORK
Mathematical Benchmarks. Mathematical reasoning is increasingly vital for assessing the funda-
mental reasoning capabilities of LLMs (Ahn al.) (024). Several math-related datasets have been proposed in this area
et al. 2016 Amini et al. 2019; Hendrycks et al.) 2222 Cobbe al. 202T Hendrycks et al.) 2021b Chen et al. 2022). Among these, GSM8K (Cobbe et 2021) and MATH (Hendrycks et al. 2021b) are the most representative datasets for elementary and
high school-level math reasoning, respectively. However, as modern LLMs become increasingly powerful, these benchmarks lack sufficient challenge forlatest LLMs. Notably, ol (OpenAl b024b) achieves 94.8% accuracy on MATH, which was previously considered highly complex. To better assess the mathematical reasoning abilities of current LLMs, existing benchmarks (Shi et al.) 2023 Chen et al.) 2024 Li et al. 2024b; Xu et
researchers create variants of al. 2024b), while
others propose new, challenging math reasoning benchmarks (Chen et al. 2023 Wang et al.
3

---

Published as a conference paper at ICLR 2025
2023a: Collins et al. 2024 Tang al. 2024 Liu al. 2024 Wang et al., 2024). CollegeMath (Tang
et al. 2024) covers several college-level mathematics subjects with limited answer types. In contrast, our UGMathBench encompasses broader range of subjects, answer types, and test examples. In addition, there are also several cross-modality math-related datasets (Chen et al. 2021 Lu et al.
2023
Xu al. 2024a Yue et al. 22024a] He et al. 2024b
Huang et al. 2024 Yue et al.
2024b).
Dynamic Benchmarks for Mathematical Reasoning. Test set contamination, wherein benchmark test data appear in newer model's training set, significantly challenges fair LLM evaluation by artificially inflating performance (Deng et al. 2024; Dong et al. 2024; Golchin & Surdeanu 2023
Roberts et al., 2023). Since pretraining data often involve large corpora scraped from the Internet, any
static benchmark risks data contamination (Zhang et al. [2024] Qian et al. 2024). To mitigate this,
recent benchmarks maintain private test sets (Zhang al. 2024 Huang et al. 2024). requiring anyone who wishes to evaluate their models to submit predictions for centralized processing before publishing results on their leaderboards. However, this process can be inefficient and lacks transparency for error analysis (Qian et al. 2024). An alternative is releasing dynamic benchmarks that are periodically updated (Srivastava et al.] [2024] Qian et al. 2024 White et al.] 2024). For example, Srivastava et al. (2024) have functionalized a subset of the MATH dataset to regenerate new versions of the test set by reassigning variable values. In this vein, our UGMathBench is a dynamic benchmark featuring different sampled values for variables by setting distinct random seeds. Currently, release three snapshots for each question in JGMathBendh and plan to release new versions if leading open-source LLMs reach accuracy saturation.
3
THE UGMATHBENCH BENCHMARK
3.1 UGMATIBEECII OVERVIEW
We introduce the UGMathBench, a dynamic undergraduate- mathematical reasoning benchmark designed to thoroughly and robustly assess the mathematical reasoning ability of LLMs. UG- MathBench enables fair evaluation through randomized versions of single problems. Unlike GSMIK (Zhang et al. 2024), our test set labels are publicly available, facilitating efficient evaluation and effective error analysis. UGMathBench covers fifteen core subject areas in undergraduate-Ievel mathematics, including single-variable calculus, multivariable calculus, differential equations, probability, and more, encompassing a total of 111 specific topics (details in Appendix A.2)). UGMchhBench comprises a set of 5,062 problems in different randomized snapshots with 10 different answer types (see Appendix A.3) These answer types range from atomic types (e.g., numerical value, expression) to compound types (e.g., multiple answers in ordered or unordered lists), setting UGMathBench apart from many other math-related benchmarks that focus primarily problems to examine student performance using our grading system' records, with each problem being completed by varying numbers of students ranging from 99 to 1,537. The average accuracy on the first attempt is 56.5%, %, while the average accuracy on the final attempt increased to 96.1%.
Table 2: Benchmark Statistics
Statistic
Number
Total Problems Number of Versions
5562
x
Total Subjects/opiis Total Answer Types
16/111
Total Difficulty Level
6
Average Problem Tokens Average Number of Answers
122.63
2.77
single answer with an atomic type. We randomly select 100
3.2 UGMATHBENCH CREATION
Our UGMathBench creation process has three distinct phases; data collection, data cleaning & deduplication, and answer type annotation.
The dataset for UGMathBench is carefully compiled from the online grading system of our institute's 's undergraduate courses (see Appendix B.T). All problems our system are generated by programs that specify particular variable values to ensure correctness and maintain the same solution (see Appendix A.I We gather all mathematics--elated problems, resulting in 16 subjects and I11 topics in total. To prevent student cheating, our grading system offers randomized versions of most problems (see Figure [.. similar to the variable disturbance approach in Qian et al. (2024). To create dynamic benchmark, exclude static problems without randomized versions, as well as those containing images, ensuring a text-only reasoning benchmark. The collected problems originally in HTML format.
Data Collection.
4

---

Published as a conference paper at ICLR 2025
Data Cleaning and Deduplication. After collecting problems in the HTML format, utilize the bs 4" aand Python packages to convert them into Latex. Since no conversion process is flawless, manually verify the converted LaTeX files against the original HTML files. The latex files are then further organized into the format shown in Figure]] [
After converting and cleaning all the problems, we perform deduplication within each subject based on embeddings generated by text eemeeddngggnng -daa-dddnn-------------- 002 to remove duplicated problems (see Appendix B.2).
The thresholds
and the number of questions that are filtered out are given in Table TO
Answer Type Annotation. Meta-information (e.g. subject, topic, subtopic, difficulty level as shown in Figure [] is stored in our grad- ing system and is easily extracted along with the LaTeX files. The primary task determining the answer types. Problems requiring definitive responses are largely categorized into two main classes: atomic and compound. Questions with a single required answer fall into the atomic type, while questions with multiple answers are classified as compound, represented by a list of atomic answers separated by commas. The atomic type can be further classified into eight types, and the compound answer lists can be either ordered or unordered, with each atomic answer fitting one of the aforemen- tioned eight types. Simple examples of each type are provided in Table 3 and detailed definitions are available Appendix A..
Table 3: Examples of eight atomic answer types.
Type
Example
Numerical Value
Expression
Equation
Interval
Trae/Talse
Yes
MC with single answer
A
multiple answers
Open-Ended Ended
b(l-x)
3.3 EVALUATION METRICS
We denote the set of test examples in UGMathBench by with a specific test example denoted as e; where represents the index of the example. Each example consists of questions presented in different randomized versions: .' "..
where V is the total number of version 3 The q." ...."" corresponding ground trouud-truth answers for these versions q: ......, q. are denoted by a! a a The answer generated by an LLM M for specific version of the question in the i-th test example is denoted by M(r)). Inspired by Srivastava et al. 2024), we define the following metrics to evaluate the true mathematical reasoning ability of LTL M in UGMathBench.
Accuracy Version Acc, is defined as the average accuracy of model M on the set of questions with version v in D;
Acc, Σ Pl E IlM())) ail
where II is an indicator function and |D] denotes the number of examples in UGMathBench. assesses the performance of LLM on the specific version e from Benhhh
Average Accuracy AAce defined as the mean of all Acc.:
AAce EV Acc,
V
This metric evaluates the performance across all versions of the questions.
Effective Accuracy EAce defined as the accuracy in solving a test example e, across all its V versions:
EPP I[M())) = ai Ve E (1,2,..,,J]]
EAcc
[D]
If a model is able to solve test case using proper reasoning, it should correctly solve this problem for all randomized versions. Thus, effective accuracy measures the fraction of test cases correctly solved across all versions V. measures true reasoning of test cases in UGMathBench.
Reasoning Gap A is defined as the percentage decrease between AAcc and EAcc. It provides measure of the robustness of reasoning, with A being true reasoning with high robustness.
https://dcss. httpss//docs.ythonn.oo wywpthonoor///libraaaa librarr/reehtm
Currently, V=3 and we plan release more versions the future.
5

---

Published as a conference paper at ICLR 2025
Robustness Efficiency (RE) is defined as the ratio of the Reasoning Gap (A) to the EAcc, expressed as RE = A /EAcc. This metric evaluates the extent of the reasoning gap relative to the model's s effective reasoning ability (i.e.. EAcc). RE captures robustness by taking the effectiveness of mathematical reasoning into account, with lower values indicating superior performance in adapting variations across different versions of problems in UGMath Achieving higher EAcc and a lower A results in a more favorable (lower) RE, reflecting improved robustness relative to "true" reasoning ablility of LLMs.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Evaluated LLMs. Our evaluation covers 23 leading LLMs, including closed-source commercial LLMs and open-source LLMs. Based on our UGMathBench, we provide thorough evaluation of the mathematical reasoning capabilities of current L.MMs. The evaluated L.I.ss are listed below:
For proprictary LLMs, we select OpenAI-ol-min (OpenAl 2024b). GPT4o (OpenA
2024a), GPT4o-mini (OpenAl 2024a), and Claude-3-Opus (Anthropic 2024).
For open-source general-purpose LLMs, we evaluated the
series (8B,
70B) @ Meta 2024), Qwen2-Instruct (7B, 72B) Yang et al,, 2024a). Yi-1.5-Chat (6B, 9B,
34B) Al et al. 2024), Mistral-7B-Instruee (Jiang et al. 2023).
Mistral-Nemoo-nssiee-240000
(Mistral 2024c).
(Mistral 2024d), Mistral-Large-Instruut-220
Mistral 2024b), DeepSeek-MOE--16B-CNG (Dai et al.
2024). and DeepSeek-V2-Lite-Chal
DeepSeek-A1 2024).
We also include some specialized math LLMs: DeepSeckMath-7B (-RL, -Instruct) (Shao et al. 20024), Qwen2-Math (7B, 72B) Yang al. 2024b). Mathstral-7B (Mistral 2023). and
NuminaMath-7B-Col) (Beeching et al. 2024).
Details of these LLMs are provided in Appendix C.1
Evaluation Settings. We employ Acc, to evaluate the average performance of version AAcc to measure the average performance across all versions, EAce to quantify true reasoning, and reasoning gap A to assess the robustness of reasoning (see Section 3.3 To remove the effect of sensitivity of few-shot prompts (Lu et al. 2022 Ma et al.] (023). all our experiments use zero-shot prompts, tailored to different answer types for better answer extraction and rule-based matching. Detailed prompts are given in Appendix C.2 We use vL.L to speed up the evaluation process. To maintain consistency in evaluations and facilitate reproduction, set the maximum output length to 2,048 tokens and employ a greedy decoding strategy with temperature 0.
4.2 MAIN RESULTS
The overall experiment results are shown in table 4 We have the following key observations:
UGMaht is a challenging benchmark for evaluating the mathematical reasoning ca- pabilities of LLMs. Even LLMS with the most advanced reasoning abilities, OpenAl-ol (mini version), achieve only 56.3% EAcc on UGMathBench, while most open-source LLMs, including most specialized mathematical models, struggle to reach a 30% EAcc. Compared to commonly used mathematics benchmarks like MATH (Hendrycks et al., 20216), UGMathBench proves to be more challenging. For instance, OpenAl-ol- -mini achieves 90% on MATH (v.s. 56.3% on UGMathBench).
Even leading LLMs still have inconsistencies when solving problems with multiple versions. LLMs with an AAcc greater than 20% display a reasoning gap A exceeding (or near) 10%. All LLMs demonstrate extremely high RE on UGMathBench. with values ranging from 20.78% to 196.6%. Among the five models with the lowest RE, three of them are from OpenAl (OpenAl-ol-mi) 20.78%; GPT-4o: 20.89%; Mistral-Large-Inss Instruttt 24.36%; Qwen2-Math 72B Instruct:: 24.39%; GPT-4o-mini: 27.87%) These results pinpoint the limitaions of current LLMs and urge us develop ture "large reasoning models" with high EAce and A = 0.

---

Published as a conference paper at ICLR 2025
Table 4: Main Results on UGMathBench (all figures are in %). Models are classified into three categories according to their purpose and origin. The best results within each column are bolded and the best results of open-source Chat LLMs within a similar parameter size group are underlined.
Models
Ace1
Ace2
Aces
AAcc
EAсc
A
RE
Closed-source LLMs
OpeniAl-o1--iii-222444735
68,02
67.95
68.04
68.00
56.30
T1.70
20.78
2PT--40-022400--
59.92
60.79
60.41
60.37
49.94
10.43
20.89
GPTT-4--mini-222-
202-----22----11
51.58
53.14
52.61
52.44
41.01
11.43
27.87
48.62
50.32
49.47
49.47
37.00
12.47
33.69
Open-source
Chat LLMss
V1-13-6B-Chii
25.33
26.33
26.39"
26.25
IS723
T1702
72.33
Mistral-7B-Instruu
10.19
11.16
10.33
10.56
4.44
6.12
137.6
Qwen2-7B-Instee
35.60
37.30
36.23
36.38
25.15
11.23
44.65
LLaMA3-8B-Inttnn
16.00
17.01
16.63
16.55
8.91
7.64
85.74
Yi-1,5-9B-Cha
33.72
34.29
34,85
34.29
21.12
13.17
62.36
Eistral-Nemo-Instruee
24.53
25.62
25.09
25.08
15.43
9.65
62.57
PrepSeck-MOE- 16B-Chat
5.59
5.85
5.97
5.80
1.96
3.85
196.6
DeepSeek-
SeS-SSck-V2-Lite V2-Lite-Chaa
12.82
13.67
12.76
13.08
5.69
7.39
130.0
Mistral-Small -aae---mmaal----eetttte 2409
40.10
40.52
40.04
40.22
28.84
11.38
39.45
Yi-1.5-34B-Chat
37.08
38.11
37.65
37.61
24.34
13.28
54.55
LLaMA3-70B-BB
33.25
34.35
33.27
33.62
23.27
10.35
44.48
Qwen2-724 (aan2-72B-lae Instructt
47.49
48.56
47.23
47.76
35.78
11.98
33,50
Mistral- Large-
Large-Instruct Instruct 2407
55.91
56.16
55.97
56.01
45.04
10.97
24.36
Specialized Mathematiccal LLMs
DeepSeek Math BB-h-7B-Jah 7B-Instruct
23.61
24.87
23.19
23.89
13.61
10.28
75.52
DeepSeek DeepSeck-Mathh17 Math-7B-RL
28.66
29.97
28.94
29.19
19.24
9.95
51.71
NuminaMath-7B-Ca.
29.32
30.07
30.01
29.80
18.81
10.99
58.44
Mathstral-7B-vl
28.57
28.47
28.51
28.51
17.94
10.58
58.96
Qwen2- Math-7B-Instruct
43.01
44.13
44.05
43.73
32.46
11.27
34.73
Qwen2-Malt 1atn2-Math-255 Math-72BBInstruet
56.95
57.05
57.09
57.03
45.85
11.18
24.39
There remains significant discrepancy between closed-source models and open-source LLMs. OpenAI-ol-min -mini achieves the best results across Ace;, i = 1, as well as in AAcc and EAcc. However, the most powerful open-source LLM evaluated. Swen2-Math-72B Instrucc, still falls short: it shows 10.97% lower AAcc and 10.45% lower EAcc compared to OpenAl-ol-min best-performing open-source chat model Mistral-Large-Insstee In07ruct-27 which ranks second among all open-source L.Mss. Only 2 out of 19 open-source L.MSs exceed GPT-4o-mini in terms of FAcc, and only 3 out of 19 have an EAce comparable to Claude-3-Opus Additionally, more than half the open-source LLMs (10 out of 19) have EAcc smaller than 20%.
The
ANALYSIS
In this section, we conduct an in-depth analysis the performance the 23 LLMs evaluated on UGMathBench by investigating the following research questions: 1. What is the relationship between EAce and Acc,? (Section [5. How do model size and model series influence performance on
UGMathBench? (Section 5.2 3. How do L.MSs perform across different subjects, difficulty levels, and, different topics on UGMathBench? (Section 5.3) 4. What are the typical response errors made by the best- best-performing LLM (OpenAl-ol-min)) and how they distributed? (Section 5.4)
5.1 RELATIONSHIP BETWEEN EACC AND ACC,
To investigate the relationship between EAcc and Acc,, scatter plots of EAcc against each Acc, From Figure we have the following conclusions:
shown in Figure 2
All LLMs fall below the diagonal lines. Each LLM evaluated is represented as point in the subfigures Figure 2 plotted on the axes of (Acc,, EAcc). Although LLMs exhibit small variations in accuracy across different versions, they consistently demonstrate lower EAcc than Acc,, which suggests that the accuracy of individual versions is insufficient for assessing the reasoning capabilities of L.MMs. By considering EAcc alongside accuracy, can gain better understanding of how L.LMs
7

---

Published as a conference paper at ICLR 2025
Y--15 1-1.5-34B-Chal
Qwen2-72BB 72B-lattodd
Mistral 175 instruct
Owend Math-7B4 instruct
DeepSieok Math: 7B-Instrrott
Mistral
Qwenz Instruct
DecpSeck
LLaMAS- MaMMMA--B-Instruut BB-nsttuuct
NamimwMath-7.cCom
Y--1. 5-1-Chhtt
Mistral Mintml--aarpe Large Instruct-2607
LLaMA3-70tt-metrun
Mathetral-7
Qwea2
DeegSeek
57
Edoce
Edoce
03-
35
10
Figure 2: EAcc v.s. Acc, on UGMathBench.
Op 202---222--9-112
GPT-40-2024-00-000
GFT-4o mini 2024-07-18
Claude 3 3-0e---3pppppppp 20260229
Yi
Owen: Chat
DeepSeck Chat
Numina
Qwen Math
LLaMA
DeepSock-Matt
1,75
1.25
EAco
2.28
1.00
10
0.25
Deep SenSS-tt
Dop 1e3-Ch
0.00
22
70
123
pommeters)
parameters)
Figure 3: Left: EAce Model Size. Right: 2-RE v.S. Model Size. The comparison chart of performance versus performance (EAcc and RE) on UGMathBench for all LLMs evaluated, with models from the same series connected by lines of the color. The horizontal dotted lines represent the score of close-source LLMs.
perform when solving problems that have different randomized versions. The discrepancy between EAcc and Acc, highlights new inconsistency mode (Ahn et al. 2024) of current LLMs: they may become inconsistent in their answers when the problem is slightly altered.
There is apparent trend that high EAcc consistently leads to high Acc., A model with high EAcc is more effective in handling variable disturbances, resulting high accuracy for each version. However, EAce becomes increasingly large, the difference between Ace, and EAAc tends to increase until it stabilizes around 10%.
5.2 THE EFFECT OF MODEL SIZE AND MODEL SERIES
Figure has shown how EAce and 2-RE changes with the parameter size. We can observe that:
LLMs within the same series have shown steady improvement the parameter size increases. When the model size increases from 7B to around 100B, EAcc substantially improve and RE steadily decrease for Qwen-Chat, Qwen-Math, Mistral, Deepseek-Chns Yi-Chat series, indicating a steady improvement in performance in effectiveness and robustness of mathematical reasoning.
and
8

---

Published as a conference paper at ICLR 2025
Specialized mathematical LLMs typically outperform their general-purpose counterparts. For example, the Qwen2-Mate series achieves significantly higher EAcc and lower RE than its general- purpose chat LLMs with the same model size. Among all 7B specialized mathematical LLMs, Qwen2-Math-77B-nnttuee ranks first surpassing DeepSeek-Math (second best) by a large margin.
5.3 PERFORMANCE ACROSS DIFFERENT SUBJECTS AND DIFFICULTY LEVELS
Figure 4a
shows the average EAcc of different models in each subject. Figure 4b shows the averaged EAce all subjects with respect to different levels of difficulty. The detailed performances across different subjects and topics for each model can be found in Appendix and
and
The average EAce varies across different subjects. LLMs are effective at solving Arithmetic problems, achieving 62.8% EAcc. In addition to Arithmetic, LLMs are also adept at Algebra, Combinatorics, and Complex Analysis (over 30% average EAcc). The three least effective areas are Abstract Algebra, Differential Equations, and Financial Mathematics, which typically require challenging domain knowledge (less than 10% average EAcc).
In general, the averaged EAcc decreases as the level increases. OpenAl's ol-mini is the strongest among all models and wins by a larger margin as the level increases. Mistral-LarreeSmmll--nnnss Instrutt and Qwen2 series are the most competitive open-source models on benchmark, with EAcc comparable to leading commercial models such as GPT-4o. However, as the level of difficulty increases, they still lag behind GPT-4o, suggesting the gap between the leading open-source LLMs and proprietary LLMs in solving difficult math problems.
Arttmmeti
Alpars
c1-mini-2028-00-1
Complex Analynt
Geometry
27.00
11.9%
2p1-44--0224008-000
Caloate Singl Varable
20.6
Mistral
Number Therry
Owen2-Me Mati--Mttt------
inntruut
Liver Alyehra
Iriponmettry
34.3%
4o
Set theory und Lagie
Frobablitt
2.7h
Lovdl
Ovon2-77.f--nttrun
Blaulee3-ouua2224222
Statleiie
Lovel
Love
Math
Lovel
Quwnn-77BBuutteee
neferentiol Tiguation
Level
Misttri-Smill-nttrndd
Thrnnda Mtttwmatice
0.
02
(a) EAcc across subjects
(b) EAce across levels
Figure 4: Relationship between EAcc, subject, and level of difficulty. (a) EAcc of different subjects, averaged across all models. Each bar consists of several segments with colors indicating their
corresponding difficulty level. Notice that the length of each color segment only indicates its proportion within all problems of all levels within that subject, and is not comparable between levels or subjects. (b) EAce of different levels, averaged across subjects. Only models with top-10 EAce are included for brevity. I Levels 5 and 6 are combined since level 6 has few samples.
5.4 ERROR ANALYSIS
We perform a comprehensive error analysis on OpenAl-ol --iii by randomly selecting 100 problems, each having least one incorrectly solved version (yielding total of 300 versions). As shown in Figure 5a, there are 231 incorrect versions, and openAll-l-ming failed to solve 56% of the problems across all versions. We then categorize these errors into six types, as illustrated in Figure 5b. Calculation errors, including both numerical and expression errors, represent the largest category, with several examples provided Appendix(E E We find that OpenAl-ol- tends to streamline its outputs to avoid generating too long responses, sometimes leading to erroneous results. Additionally, we encounter some "bad questions" that primarily arise due to overly complex structures (e.g. containing long tables) or inadequately described (e.g. undefined variables in previous problems). This is because our homework grading system (see Appendix B.I) h) is designed for students with a user- friendly interface, and some problems may not be suitable for LLM to solve. In our sample of 300 versions, 19 were identified as "bad problems." " giving us an estimated occurrence of approximately 2.7% in JGMathBench These problems do not impact our main claims, L.LMs are able

---

Published as a conference paper at ICLR 2025
to solve these "bad questions." We will refine these types of problems to make them more suitable for LLM evaluation in the future. No answer cleaning process is perfect, and improved evaluation codes continue to be released in MATH (Hendrycks al. 2021b). We will also actively update evaluation repository to improve its quality. Inspired by recent model-based evaluation efforts (Gao al. 2024 Xu et al. (025a), we have updated code repository to integrate the model-basee evaluation introduced by Xu et al. (2025a). Updated results will be made available there.
Notably, we have found that even when OpenAl-ol 1-mii solves a problem incorrectly among all its randomized versions, the error types can be different. As shown in Figure 5a there are around 16.1% % such inconsistent errors among the 100 problems sampled.
Total Count Consistent Lrror Inconsistent
expression calcluation ermr
40-
numeric calculation error
Count
27.
flawed resoning
25.01
bad questions
instruction following
misunderstanding
Number of Wrong Versions
(a) Count Number of Wrong Versions
(b) Error Type Distribution
Figure 5: Error Analysis of OpenAl-ol-mint on UGMathBench.
5.5 FURTHER ANALYSIS
About Gelf-Improvemen.. To examine how LLMs perform with refinement UGMathBencch we conducted experiments using Progressive-Himt Prompting (PHP) (Zheng et al. 2023 (see Ap- pendix Gp. Detailed results can be found in Table 32 Appendix G Although PHP improves EAcc and AAcc for most LLMs, the enhancements are not significant, indicating considerable room for future development. The fine-grained results in Table 333 suggest that the impact of refinement for GPT-4o varies across different subjects. For instance, PHP improves GPT-4o's performance in abstract algebra by 7. 1,1,, yet reduces performance in probability by 2.08% in terms of EAcc. Our UGMathBench serves an excellent testing ground for future research into refinement methods for solving undergraduate- mathematics with LLMs.
Reasoning Gap and Test Set Contamination. To explore how models specifically overfitting to particular variation affect the reasoning gap, we mixed portion of the test set from one version with MetaMathQA (Yu et al. 2023) and then conducted supervised fine-tuning (SFT) Llama-3-8B on this data. Details of the SFT process are provided in Appendix H. and the results are presented in Table [] As the proportion of the test set included in the training data increases, the reasoning gap (A) also becomes more pronounced. This study serves as an initial investigation into test set contamination during the SFT stage. It's worth noting that contamination at the pre-training stage is also significant area of interest (Razeghi et al.) 2022: Jiang et al. 2024).
Table 5: Effects of Overfitting.
Propection
A
3.43
3.80
4.50
4.64
6 CONCLUSION
Current mathematical benchmarks are often inadequate, lacking comprehensive coverage of undergraduate-level math problems or being susceptible to test-set contamination. To fill these gaps, we propose UGMathBench, a diverse and dynamic benchmark for undergraduate--eve math- ematical reasoning. Our fine- ane-grained analysis has pointed out the potential inconsistencies when LLMs encounter problems with slightly different versions. We hope that our UGMathBench can contribute future development of "true" reasoning L.Mss.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was partially supported by grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project Reference Number: AoE/E-601/24-N), We would like to thank all reviewers for their helpful suggestions in improving this paper.
REFERENCES
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. In Neele Falk, Sara Papi, and Mike Zhang (eds.). Proceedings of the 18th Conference of the European Chupter of the Association for Computational Linguistics: Student Research Workshop, pp. 225 237, St. Julian's s. Malta, 2024. Association for Computational Linguistics. URL attps://aclarcee
eacl -1ww-3rw.17
01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang. Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang. Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang. Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by 01.ai, 2024.
AI@Metaa Llama model card. 2024. URL
11aieaaalmaaa/aa// lama3/
3lob/main/MM00 Mina/Mi/MMOLLL.. HARD.md.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Towards interpretable math word problem solving with operation- based formalisms. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2357-2367, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10. 1653/ URL
Hajishirzi. MathQA:
atttss:/accaanthol... org/N19 1245
Anthropic. Claude
https::/www, 2024. Accessed: 2024-09-23.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. ArXiv preprint, abs/2310.10631,, 2023. URL httpp://arzivv.r/////a
2310.10633
Edward Beeching, Shengyi Costa Huang, Albert Jiang. Jia Li, Benjamin Lipkin, Zihan Qina, Kashif Rasul, Ziju Shen, Roman Soletskyi, and Lewis Tunstall. Numinamath 7b cot. https://
huggingface co/AI AA--- MO/ Www.NNniianaaaaa 7B 7B-CoT] 2024.
Daniel Bobrow et Natural language input for a computer problem solving system. 1964.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL sttpps:/ppooceeseee cb4 9674 41 1117741865... wte.ffaac142ffff26 w2w444644422200 Abst ract html
www.ipsuppp.. 0C pctpaper//020.0.
Jiagi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing, and Liang Lin. GeoQA: A geometric question answering benchmark towards multimodal numerical reason- ing. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the
11

---

Published as a conference paper at ICLR 2025
Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 513-523, Online, 2021. Association for Computational Linguistics. doi: 10.18653/1120211 fiindingniiinnnin acl.46. URL
https::/aclantholLe wwt.or/////. 2021..011.... findings-aci.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. ArXiv preprint,
abs/2211.12588, 2022. URL https: //axxi aawivv..vvvv..///ii org/ abs /tlp:/221..2222 12221.11181111
Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang. and Tony Xia. TheoremQA: A theorem-driven question answering dataset. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7889-799899 1989-7901, Singapore, 2023. Association for Computational Linguistics, doi: 2023. emnlp main. memiin.888
URL htttps:/aclaattholggg. org/
Xinyun Chen, Ryan Chi, Xuezhi Wang, and Denny Zhou. Premise order matters in reasoning with large language models. ArXiv preprint, abs/2402.08939, 2024. URL httpt/tss///xxii..r.....
abs/ 2aa/2400 2402. 08939
Karl Cobbe. Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. ArXiv preprint, abs/2110.14168 2021. URL https::/arxiv..or
abs abs/2110, 2110011111.. 14168
Katherine M Collins, Albert Q Jiang. Simon Frieder, Lionel Wong, Miri Zilka, Umang Bhatt, Thomas Lukasiewicz, Yuhuai Wu, Joshua B Tenenbaum, William Hart, et al. Evaluating language models for mathematics through interactions. Proceedings of the National Academy of Sciences, 121(24):
e2318124121, 2024.
Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization mixture-of- experts language models. ArXiv preprint, abs/2401 2024. URL htttt://///axx org/
abs/ abs/2401, 2201. 06066
DeepSeek-AL. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024.
Chunyuan Deng, Yilun Zhao, Xiangru Tang. Mark Gerstein, and Arman Cohan. Investigating data contamination in modern benchmarks for large language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). pp. 7770--799, Mexico City, Mexico, 2024. Association for Computational Linguistics. URL httph: http:://aaaaattan
org/ 2024 naacl long. 482]
Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, and Ge Li. Generalization or memorization: Data con- tamination and trustworthy evaluation for large language models, ArXiv preprint, 105./2022.55938, 2024.
wwwpiiiiorrgaaaaoo www/aagg/a.......ao.... 2402 01.4422.1999 15938
Bofei Gao, Feifan Song, Zhe Yang. Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma. Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. ArXiv preprint, abs/2410.07985, 2024. URL
abs/ / 2410. 07985
Shahriar Golchin and Mihai Surdeanu. Time travel in 1lms: Tracing data contamination large language models. ArXiv preprint, abs/2308.00 084493, 2023. URL httphht./aaxii..or////////////// 2308. 08493
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. Tora: A tool-integrated reasoning agent for mathematical problem solving. ArXiv preprint, abs/2309. 17452, 2023. URL htt.::ttts://aatt... www./oog//////aaeeoo / 17452
12

---

Published as a conference paper at ICLR 2025
Chaoqun He, Renjie Luo. Yuzhuo Bai, Shengding Hu. Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. ArXiv preprint, abs/2402. 14008, 2024a.
http///xxvvvvvoaaaaaaa/aaaaaa//a 14008
Zheqi He, Xinya Wu, Pengfei Zhou, Richeng Xuan, Guang Liu, Xi Yang, Qiannan Zhu, and Hua Huang. Cmmu: A benchmark for chinese multi-modal multi-type question understanding and reasoning, ArXiv preprint, abs/2401.14011 2024b. URL https: /ttpps://ariv, hogg/bbs/4400 14011
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. 9th International Conference on Learning Representattinns ICLR 2021, Viriual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a.
wtw.:e/iwwwweeeeouuueeemme wtt..nneeeoooonnnde........ dd mm
new wwrumoorummmmm
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardr. Measuring mathematical problem solving with the math dataset. ArXiv preprint, abs/2103.0///1111 2021b. URL
abs wwwp///3...... 03874)
Zhen Huang, Zengzhi Wang. Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyu- manshan Ye, Ethan Chern, Yixin Ye, et al. Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai. ArXiv preprint, abs/2406. 12753, 2024. URL
httpss//axxxvv.or///abbs. www.oogg//bbo.o..
12753
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. ArXiv preprint,
2023. URL https:://txxtiiit http://aaaxvvv..aaabaaa abs ab3///10
06825
Minhao Jiang, Ken Ziyu Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo. Investigating data contamination for pre-training language models. ArXiv preprint, abs/2401.06059, 2024. IRLLhhtps://aaxiiii wwt./o/gg/gaaaaaeoooooooooeeee/e hbtp://bb/224111.. 06059
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwa- sawa. Large language models are zero-shot reasoners. In Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Process- ing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 2022, 2022. URL http:://pppers.niss..coom ctt.pps.ccppppeee wttp://aaaseaaap/2222222 / 00t.000000000000000000000000--00
htm.
Rik Kedzio-kedii Subhro Roy, Aida Amini, Nate Kushman, and Hannanch Hajishirzi. MAWPS: A math word problem repository. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1152-1157, San Diego, Cal- iforiaa 2016. Association for Computational Linguistics. doi: 10.18653/v1/N16- 1136. URL /ttts:://cclantholoog 1ww.m/hooooo...../111111111 1136
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solk Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December nips.
9, 2022, 2022. URL http: Bab.beefeeeeoa00033cceeeeaeaarre
httpseesapppppeeeeee /202 1022/haan 2022/hash/ / html
Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7b language models already possess strong math capabilities. ArXiv preprint, has//2033.04706 2024a. URL httpps://aaatt htwparrwwwooggbbb///00a0
13

---

Published as a conference paper at ICLR 2025
Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong. and Wei Bi. Gsm-plus: A comprehensive benchmark for evaluating the robustness of Ilms as mathematical problem solvers. ArXiv preprint, abs/2402.19255,, 2024b. URL/attpa://arx: //axiiii///iiiiee/aa org/ abs 22224222122222 19255
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 358-167, Vancouver, Canada, 2017. Association for Computational Linguisties. doi: 10. 10.18653/v1/117-11 1015. Utthttps:://
Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhou, Wenwei Zhang, Songyang Zhang, Dahua Lin, and Kai Chen. Mathbench: Evaluating the theory and application proficiency of Ilms with a hierarchical mathematics benchmark. ArXiv preprint, abs/2405. 12209, 2024. URL
arxiv orgg abs / 12209
Pan Lu. Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li. Hannaneh Hajishirzi, Hao Cheng, Kai-We Chang. Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. ArXiv preprint, abs/2310.02255,, 2023. URL https
// //axxiv.co org. abs atth//bss2211111 2310..0000000. 02255
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Lingwistics (Volume I: Long Papers), pp. 8086-8098, Dublin, Ireland, 2022. Association for Computational Linguistics. doi: 10. (0055//1/2022.ac-looo
URL
org/ 2022 acl long 556
Huan Ma. Changqing Zhang, Yatao Bian, Lemao Liu, Zhirui Zhang, Peilin Zhao, Shu Zhang, Huazhu Fu, Qinghua Hu, and Bingzhe Wu. Fairness- guided few-shot prompting for large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10- 2023. /tt.3/////t..//aaaaaaaaaa...../.
httpppppprrrraaaaaaaaaaaaeeeeele http://ecccpppppeeeeeeeee/////// 8678 455725500.4833300000003333444403
16,
Mistral. Mathstral. http:://mssttaaaaa//////////tata
2023. Accessed: 2024-09-
23.
Mistral. Mistral-7b-insttuuttv...3
hetess//bagghnnfaac.cco//tttttt
2024a.
Mistral. Mistral large 2.
2024b.
Mistral. The future of ai: Trends and predictions.
https://misttral.aii/e mistral-rrmo// 2024c. Accessed: September 29, 2024.
Mistral. mistala/miistallsaallnnmssetttts hctess://ongiinfaae..o/oossssss |/tps///ewoo//oooo///ooooooooooa
2024d.
OpenAl. Gpt-4 technical report. ArXiv preprint, abtpmr30.aab/2000.78774.2222 2023. URL htttt::///aat
ozg/abs/2303.08772
OpenAl. Hello gpt-4o. https://openai. htt.peaai..cmm/ndee/eeeeeeaaeeee oogea//aaoooooooooo///ooooogggag 2024a.
OpenAI. Learning to reason with 1lms. https://openal wwwpeenna...oeeeeeeeeooommooeeee index/ 2024b. Accessed: 2024-09-23.
Tearming-tt--ressso--ttt--------
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances Neural Information Processing Systems 35: Annual Conference Neural Information
14

---

Published as a conference paper at ICLR 2025
Processing Systems 2022. NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022. 2022. URL
/ http::/pppppppeeeeeeeaaeaeaaeeea files/paper/ 2022////ee22/////////2 20e20022aaaaae 22tp//2a22/2///////// /
htmm
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve sim- ple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tuue Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, pp. 2080-2094, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v 12222...aaccl mann. URL maacc-maa in. 168
Long Phan, Alice Gatti, Ziwen Han, Nathaniel L.,, Josephina Hu, Hugh Zhang, Sean Shi, Michael Choi, Anish Agrawal, Arna Chopra, al. Humanity's last exam. ArXiv preprint, abs/2501. 14249, 2025. URL http::
org /ass//ab///aa 20///2501.100 14249
Kun Qian, Shunji Wan, Claudia Tang, Youzhi Wang. Xuanming Zhang, Maximillian Chen, and
Zhou Yu. Varbench: Robust language model benchmarking through dynamic variable perturbation.
ArXiv preprint, abs/2406. 17681, 2024. URL htRL:tt:::/// http:::tttatttttt.e //axas:////vvvv.aa org/ aas////////// 2 2406.444 17681
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining
term frequencies few-shot reasoning. ArXiv preprint, abs/2202.07206, 2022. URL https
abs 7/arxiv. wttpa///rrraa///aeeooooo.o httpbb//222..22220
Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley. To the cutoff... and beyond? a longitudinal perspective on Ilm data contamination. In The Twelfth International Conference Learning Representations, 2023.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Mingchuan Zhang, YK Li, Y Wu. and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv preprint, abs/2402.00 0030000, 2024. URL 03300
otg./ abs 2402
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honoludu, Hawaii, USA, volume 202 of Proceedings of Machine Learming Research, pp. 31210- 31227. PMLR, 2023. URL httt::://occceedeeeeeeoooo
mlz wwwpspesssspppppp /t055//2200 w21///22222//2222222 shi 1n12 23a html
Saurabh Srivastava, Anto PV, Shashank Menon, Ajay Sukumar, Alan Philipose, Stevin Prince, Sooraj Thomas, et al. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. ArXiv preprint, 19450 2024. URL httpp::/arxiv..rr// http://xxiv.or///b/a.///// 2402. 9450
Zhengyang Tang, Xingxing Zhang, Benyou Wan, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. ArXiv preprint, abs/2403.00284, 2024. URL https://
arxiv.org/abs 2403..222243... 02884
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean Cean-Baptiste Alayrac Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. ArXiv preprint, abs/2312. 11805, 2023. URL (tlthttps:// htttts:///xxiii
oorg/abs/ 2312. 11805
Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving. ArXiv preprini, abs/2407. 13690, 2024. URL
https://arxix: //tpiv///xvvv.//. org/ abs 2407 13690
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solvine abilities of large language models. ArXiv preprint, abs/2307.106355 2023a. |t //arxi//aiii www.orrg/aale otgg/bb//////..// 10635
15

---

Published as a conference paper at ICLR 2025
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023b. URL httph:t////// http:///peeeevvveeee ww..nreeeeeeweee.e.......... net/ pdf 1df7idd1PLINTMMI
Yan Wang, Xiaojiang Liu, and Shuming Shi. Deep neural solver for math word problems. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods Natural Language Processing, pp. 845-854, Copenhagen, Denmark, 2017. Association for Computational Linguistics. doi: 10. 10.18653//1/DI7-11000 URL https
//aclanthology. wtg/D/7-108000001
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. ArXiv preprint, abs/2406.01574, 2024. URL
https::/aaxiis org/ httrg//a//// www./40/.... 01574
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference Neural Information Processing Systems 2022, NewrlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL http: http:://paaper..ee wttp://.....sss..pppppeepeeee oc fiepepppsseeepppp..elleeeeeeeeee fttpp//prelssspeee//eepepa hwt.pppppp000. 2022 2tt////22//aa/// / html
Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsoureing multiple choice science questions. In Leon Derczynski, Wei Xu, Alan Ritter, and Tim Baldwin (eds.). Proceedings of the 3rd Workshop Noisy User-genent "ssr-generated Text, pp. 94- 106, Copenhagen, Denmark, 2017. Association for Computational Linguistics. doi: 10.116633/1/W17-444
org/w11-444 7-1333
Colin White, Samuel Dooley, Manley Roberts. Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. Livebench: A challenging, contamination- free Ilm benchmark. ArXiv preprint, abs/2406. 19314, 2024. http:tttss///xxi..rrrr 2406. 9
Xin Xu, Shizhe Diao, Can Yang, and Yang Wang. Can verify step by step for incorrect answer detection? ArXiv preprint, abs/2402. 10528, 2024a. URL 10528
Xin Xu, Tong Xiao, Zitong Chao, Zhenya Huang, Can Yang. and Yang Wang. Can llms solve longer math word problems better? ArXiv preprint, a88/2405.14804, 2024b. URL https
Xin Xu, Qiyun Xu, Tong Xiao, Tianhao Chen, Yuchen Yan, Jiaxin Zhang, Shizhe Diao, Can Yang, and Yang Wang. Ugphysics: A comprehensive benchmark for undergraduate physics reasoning with large language models. ArXiv preprint, abs/2502.00334, 2025a. URL https
7/arxiv. wwwp://wwrrr.bbb.....cooom www.a///...... 00334
Xin Xu, Yan Xu, Tianhao Chen, Yuchen Yan, Chengwu Liu, Zaoyu Chen, Yufei Wang, Yichun Yin, Yasheng Wang, Lifeng Shang, et al. Teaching Ilms according to their aptitude: Adaptive reasoning for mathematical problem solving. ArXiv preprint, abs/2502. 12022, 2025b. URL
/btg/bb//////eo https://arx: wwg. abs 2502. 12022
Shangzi Xue, Zhenya Huang, Jiayu Liu, Xin Lin, Yuting Ning, Binbin Jin, Xin Li, and Qi Liu. Decompose, analyze and rethink: Solving intricate problems with human-like reasoning cycle. In The ehihth- Annual Conference on Neural Information Processing Systems, 2024. URL https: http:p:://ppereeee ew net htt///ormm//// i d= 1DD j7
Yuchen Yan, Jin Jiang, Yang Liu, Yixin Cao, Xin Xu, Xunliang Cai, Jian Shao, et al. S c-math: Spontaneous step-level self-correction makes large language models better mathematical reasoners. ArXiv preprint, abs/2409.01 5244, 2024. URL https: //tax//axiiii wwwpwwwwwwwwwwwaa../..//a./a..ra 2409 01524
16

---

Published as a conference paper at ICLR 2025
Yuchen Yan, Yongliang Shen, Yang Liu, Jin Jiang, Xin Xu, Mengdi Zhang, Jian Shao, and Yueting Zhuang. Mathfimer: Enhancing mathematical reasoning by expanding reasoning steps through fillile task. ArXiv preprint, fos/2502.116844, 2025. URL htttpa://arxiv..o
abs/2502. 11.00.116600 11684]
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. ArXiv preprint. abs/2407.10671, 2024a. httpttts///xxi....../ www. wwwgg/bb//// 0aabs//447.10555
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward math- ematical expert model via self- improveett...ent.. ArXiv preprint, abs/2409. 2024b. URL
httpss:/arxiv. org/ abs 2409. 12122
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. ArXiv preprint, abs/2309.12284. 2023. URL httpts
//arxiv. org/ abs 2t0..///30..... 12284
Xiang Yue, Xingwei Qu, Ge Zhang. Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. ArXiv preprint, abs/2309.05653, 2023. URL httph::
org/ abs/abass.3./ 006.333.9.666333
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang. Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-discipline multimodal under- standing and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556 9556,9567, 2024a.
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Ming Yin, Botao Yu, Ge Zhang, et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. ArXiv preprint, abs/2409.0281 2024b. URL
orrg /ataaaab244555 02813
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, et al. A careful examination of large language model performance on grade school arithmetic. ArXiv preprint, abs/2405.000 0000320,,,,, 2024. URL htt.h:ts///// pettps://arxiv.org
abs/2403 2405. 00332
Yifan Zhang, Jingqin Yang. Yang Yuan, and Andrew Chi-Chih Yao. Cumulative r with large language models. ArXiv preprint, (ob//2308.043/// 2023. URLAttts://arxivv.rr
04371
Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. Progressive-hint prompting improves reasoning in large language models. ArXiv preprint, abs/2304.09797, 2023. URL
httts:://rxiv... /rg/ abs / 2204 09797
Aojun Zhou, Ke Wang. Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin. Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, et al. Solving challenging math word problems using gpt-4 code interpreter with based velf-veriiication. ArXiv preprint, abs/2308.07921. 2023, URL
https://aas //rxxvv org/ abs 2308 07921
17

---

Published as a conference paper at ICLR 2025
LIMITATIONS
This work has several limitations. First, UGMathBench focuses text-only reasoning, whereas some undergraduate-Ievel math problems require images for their solutions. Developing multimodal benchmark for undergraduate--evel mathematics will be future work. Second, UGMathBench is designed as an English-language benchmark. Extending UGMathBench to support multiple languages could be interesting avenue for future research. Third, the number of problems in certain subjects limited. Expanding these subjects would be valuable.
A DETAILED STATISTICS OF UGMATHBENCH
A.1 INTRINSIC MECHANISM DYNAMIC PROBLEMS
A key feature of our UGMathBench is its dynamic nature. In this appendix, detail how this is achieved. All problems in our homework grading system (see Appendix [B.]) are stored as programs written in Program Generation, established programming language for mathematics. These programs strictly specify conditions to ensure that the generated variations of cach problem do not fundamentally alter their nature, required solution approach. difficulty level, or underlying knowledge points. One such program is shown in Listing and two different versions of this problem it generates are given in Figure 7 The relationship between the first and second variables is defined by Serpmt = -1 + 2 which maintains the consistency of the concepts, techniques, and solutions involved different versions of each problem.
A.2 DISTRIBUTION OF PROBLEMS
Our UGMathBench covers various subjects in undergraduate-lever mathematics. The detailed topics and the number of subtopics of each topic across different subjects are listed in Table There
are 111 topics and 583 subtopics across 16 subjects in total. Furthermore, the distribution information of our benchmark on different subjects and difficulty level is presented in Table Note that there are problems with missing difficulty level in our online homework grading system (see Appendix B.1) and we remain as is for consistency, The keywords of our UGMathBench are shown in Figure 6 The detailed results across different subjects and topics are discussed in Appendix D and
A.3 ANSWER TYPES
By carefully reviewing large collection of problems and referring to various past benchmarks (He et al. 22224a] Huang al. 2024), we classify all answers to be two main categories: atomic and compound. There are 8 atomic types and compound types. Each compound type is composed of list of atomic ones. These types designed to encompass wide range of problems. Detailed definitions for each answer type can be found in Table
B
UGMATHBENCH CREATION
B.1
DATA SOURCE
UGMathBench originates from questions in the online homework grading system of our institute. utilizing WebWork an open-source online platform licensed under GNU Widely employed for assigning mathematics and science homework in educational settings, WebWork benefits from collaborative contributions by educators across various institutions.
Each question in WebWork is tagged with keywords related concepts and difficulty level based on 06 which helps simplify statistical analysis and cognitive assessment. To prevent
Bloom's 's taxonomy cheating from each other, WebWork is able generate tailored problem sets with different random seeds, making it popular among educational institutions.
hhtps://wwww.ccbwrrk.oom
18

---

Applications of integration Published as a conference paper at ICLR 2025
Table Topics of each subject and corresponding number of subtopies included in UGMathBench.
Subject
Topics
Sub-Topics
Arithmetic
Integers
Fractionsrrational numbers
Decimals
Percents
Irrational numbers
Other
Units
Algebra
Algebra real numbers and simpliiying expressions
Ahsolute value expressions and functions
Properties of exponents, rational exponents and rudicals
2
Cartesian coondinate system
Factoring
Functions
Transformations of functions and graphs
Linear equations and liunctions
Qundratic equations and functions
Operations polymonial and rational expressions
Polynomial equations and
Variation and power functions
Systems of equations inequalities
Functions with fructional exponents and radical func- tions
Rational equations and functions
Inverse functions
Exponential logarithmic expressions functions
Finite sequences and scries
Conic sections
theory logic
Operations
Relations between sets
Functions
Propositional Ingic
First order logic
Pattern matching
Tigonometty
Geometric and algebraic loundations for tigonomeity
Trigonometric functions
Triangle trigonometry
Analytic trigonometry
Polar coordinates & vectors
Combinatorics
Counting
8
Recurrence relations
Geometry
Shapes
Circle geumetry
Vector geometry
Calculus single variable
Calculus of vector valued functions
Concepts multivariahle fincctions
Differeetiaaion of multivariable functions
Integration of multivariable functions
Vector fields
Vector calculus
Fundamental theorems
Calculus multivariable
Limits and continuity
14
Differeatiaton
13
Applications of differentiation Integrals
18
Techniques integration
16
Inlinite sequences
Parametric
Polar
Linear Algebra
lincar equations
Matrices
8
factorizations
Buclidean spaces
Abstract vector spaces
Eigenvalues and eigenvectors
Inner products
transformations
Determinants
Number Theory
Divisibility
Congruences
Diophantine equations
Pinancial Mathematics
Annuttes
Bonds
Equations
Interest
Options
Expected and contingent payments
Equities
19

---

Published as a conference paper at ICLR 2025
Table 7: Topics of each subject and corresponding number of subtopics included in UGMathBench (Con't).
Subject
Toples
Sub-Topics
Probability
Sample Space
Random variables
Discrete distributions
Continoous distributions
Laws, theory
Several variables
Statisties
Experimental design
Sample survey methods
Exploratory data analysisddescrppiiee statistics
Sampling distributions
Confidence intervals
Hypothesis tests
13
Simple regression
Time series
Point estimation
Multiple regression
Generalized lincar methods
Complex Analysis
Arithmetic
Complex equations
Complex plane
Complex functions
Analytic functions
Scries residues
Differential Equations
Introductory concepts
First order differential equations
Higher order differential equations
Laplace transforms Systems of diffcrential equations
Numerical methods
solutions
Partial differential equations Groups
Abstract Algebra
Rings
Fields and polynomials
Total Count
I11
583
statistic
vector
permutation miningo
linear
equations
Compound
interest
Buston
decimal
linear
approximation
taylor series
Inerendals Derivatives triangle
Calculus
combinatorics
surface Integral
eiperandd
algebra
line
directional derivative
descriptive
algebre
statistics
becidd
Vector
systems
field
generaate
ard
atore
union
oiffrren
right
parsel
counting
conversion
PDE
unit
systa
identity
stem
tangent
differential
hypothers
meall
dis
order operations
testing
logic
prepe
divergence
proof
Grdde
economics
rhersl
tangent line
division
Inagral
anale
measure
rate
Derivative
conditional
continujt
matrix
Laplace
differencia
transform
Series
vector
space
Statistics
separable
sine
:calculus
Mod
cosine
algebraic
expression
differential
prediaate
equaa
atton's
double integral
subspaces
definite
integrals
inference
oar
continuous
line
integral
integral
statistic
negative
eersectio
tradient.
work
Complex
prrpootion
disonyttion
mathematics
for business
first order
regression
complex numbers
Figure 6: Word Cloud of Keywords in UGMathBench.
20

---

Published as a conference paper at ICLR 2025
Table Statisties of UGMathBench across different subjects and difficulty levels.
Level
Level 2
Level 3
Level 4
Level 5
Level 6
All
Arithmetic
22
179
47
53
322
Algebra
25
176
80
73
583
Set theory and logic
0
35
11
12
9
0
69
Trigonometry
5
86
44
17
25
0
178
Combinatorics
3
28
47
4
5
88
Geometry
5
48
86
13
6
161
Calculus single-variable
4
297
317
175
176
982
Calculus multivariable
13
244
37
Linear Algebra
2
232
Number Theory
2
7
Financial Mathematics
7
105
89
346
Probability
11
167
139
2
336
Statistics
44
151
158
46
0
401
Complex Analysis
0
40
7
2
1
51
Differential Equations
2
47
115
56
84
305
Abstract Algebra
0
16
7
42
Grand Total
145
1,855
1,751
657
588
12
5,062
Table 9; Answer Types and Definitions
Answer Type
Definition
Atomic Types
Numerical Value (NV)
Problems where answer numerical value, including special values such e, v7, sin r/8, etc., represented LaTeX.
Expression (EX)
Problems requiring expression containing variables, e.g., 8n.2 xr represented in LaTeX.
Equation (EQ)
Problems requiring an equation containing variables, e.g., y = 2x + represented in LaTeX.
Interval (INT)
Problems where the answer is a range values, e.g., (-o0, 2) U 00) represented as an interval in LaTeX.
True/False (TF)
Problems where the answer is either True or False, Yes or No, T or F, Y or N, etc.
MC with Single answer (MCS)
Multiple-Choice (MC) problems with only one correct option (e.g., one out of four, out of five, etc.). The options can be captical letters (ABCD) any other string according to the problems (independent dependent, etc.).
MC with multiple answers (MCM)
Multiple-Che Choice (MC) problems with multiple correct options (e.g., two out of four, two out five, two out of six, etc.). The options can captical letters (ABCD) or any other string according to the problems (independent or dependent, etc.).
Open-Ended (OE)
Problems whose answers can be term, name, any other string that satisfies the description of the problem, for example, the name of the variable or function that occurs the problem (which should be treated diffcrently with EX).
Compound Types
Ordered List (OL)
Problems where answer an ordered list, c.g. a coordinate (1, 2,, 3), (2t, t²), etc.).
Unordered List (UOL)
Problems where answer is an unordered list, e.g., set or multiple solutions for an equation.
As mentioned in Appendix A.I our problem generation programs will remain the main problem structure and knowledge points. One such example is illustrated in Figure / and the corresponding program is given in Listing [ The dashed red box highlights the differences between randomized
21

---

Published as a conference paper at ICLR 2025
First make substitution then integration by parts ovaluate the intagral.
First make subsiittiion and then integration by parts evaluate integral.
f
f &o (2)
d cos (a=) dx
Answer:
+
Answer:
Preview Input
C Prpvipw Inpust
Figure 7: An Example of Our Online Homework Grading System Based on WebWork. The dashed red box highlights the differences between randomized versions of the problem.
versions of the problem. For this specific example, the second random variable is initialized through random (2, 10). The relationship between first and second variables defined by expnt -1 2+a. This meticulous setup maintains the consistency of the concepts, techniques, and solutions involved different versions of each problem. Another example is demonstrated in Figure where 'n' is initialized to randomly chosen exponent from 2 to 10 in increments of 0.1.
All problems within WebWork are stored in the Problem Generation language and are presented in HTML format with JavaScript and external resources. This poses challenges for human interpretation and LLMs analysis, urging us to clean and re-format them in Latex (see Section 3.2
Sa=random (2, 10); 4 all-inclusive integers between and 10
Sexpnt Sa;
Sans = "1/Sa=x*aassin (ASAS #Right answer
TEXT beginproblem ());
BEGIN TEXT
First make a substitution and then use integration by parts to evaluate the integral.
$BR
M[ Vint x^ Sexpnt) (cos(x dx A]
SBR
Answer:
A[ {annsrull (40)1) I(+)) (CC)
END_TEXT
#Compare the right answer to the input ANS(fun__cml $ans, mode=> antider
Listing I: Problem Generator Code for the Problem in Figure
B.2 DEDUPLICATION
After converting and cleaning all the problems, perform deduplication within each subject. More specifically, we adhere to the following steps: First, we transform each question into a vector with dimension 1536 using the embedding model text text-embedd: ng-ada-002, which is the most capable 2nd generation embedding model of OpenA We then calculate pairwise cosine similarities using the embeddings in the previous step. Finally, threshold is selected based on manual inspection within each subject, and problems that have cosine similarity higher than that threshold with existing problems are excluded. The thresholds and the number of questions filtered out for different subjects are presented in Table 10
We use subject-agnostic thresholds and filter out 9,382 questions in total,
"The information be found at wwwp://www.nnnnicccccococmnnnnne
22

---

Published as a conference paper at ICLR 2025
Determine whether the series absolutely convergent, conditionally convergent, or divergent:
Determine whether the series absolutely convergent, conditionally convergent, divergent:
8
sin 2Tz)
sin 2Tz
The series is
The series is
A. absolutely convergent
O abxolufely convergent
conditionally convergent
conditionally convergent
C. divergent
divergent
Inpul
Preview Inpui
Figure 8: Another Example of Our Online Homework Grading System Based WebWork
Table 10: Thresholds of deduplication and the number of questions that filtered out in UGMathBench.
Threshold
Before
After
# Filter Out
Arithmetic
0.92
845
322
523
Algehra
0.86
4663
583
4080
Set theory and logic
0.96
69
Trigonometry
0.91
399
178
221
Combinatorics
0.89
140
88
52
Geometry
0.92
267
161
106
Calculus single-variable
0.90
3234
982
2252
Calculus multivariable
0.94
866
654
212
Linear Algebra
0.93
1235
498
737
Number Theory
0.92
68
46
22
Financial Mathematics
0.92
653
346
307
Probability
0.91
453
336
Statistics
0.92
599
401
198
Complex Analysis
0.95
129
51
78
Differential Equations
0.92
741
305
436
Abstract Algehra
0.94
58
42
16
Grand Total
14444
5,062
9382
C DETAILED EXPERIMENTAL SETUP
C.I EVALUATED LLMS
variety of LLMS covered in evaluation, including closed-source commercial models and open-source models, general-purpose models and models dedicated for math problem solving. Closed-source LLMs follows:
ol-preview (OpenAl 2024b): An early preview of OpenAl's ol model, designed to reason about hard problems using broad general knowledge about the world. We used oo-preview- 2024-09-12 for our evaluation.
GPTT4o AOpenAl 2024a): GPT-4o multimodal, and has the same high intelligence as GPT-4 Turbo but is much more efficient.
Claude-3-Opus (Anthropic 2024):
Anthropic's most intelligent model, claimed to outper- form its peers most of the common evaluation benchmarks for Al systems.
We evaluated the following open-source general-purpose L.LMs on our benchmark:
-Llama33- (Al@ Meta
2024 LLaMA 3 Community License
23

---

Published as a conference paper at ICLR 2025
Mistrrl-7B-lnsttruct--333
(Mistral
2024a):
Apache 2.0
MistralANemo--nssruue---007 MMistral
2024c):
Apache 2.0
Nistral-imall--nttruu---999
(Mistral
2024d):
MRL License
Mistral-Large-Insrrre-2200
Mistral
2024b
MRL License
Qwen2-Instruct (Yang et 2024a): Qwen2 series are developed with dedication to math and coding. We used 7B and 72B models. 7B models are licensed under Apache 2.0, while 72B models are under Tongyi Qianwen License.
Yi-1.5-Chat (Al et 2024): Yi-1.5 delivers stronger performance coding, math, reason- ing, and instruction-followee capability compared its predecessor. We used 6B, 9B, 34B variants. Yi-1.5 series are licensed under Apache 2.0
DeepSeek-VV--Lite--Ce V2-Lite-Chat
(DeepSeek-A)
2024): model under Model License code under
MIT License
deepseck-moe-16b--hm (Dai et License
2024 model under Model License, code under MIT
The following specialized math LLMs are evaluated in our study:
DeepSeekkatth775 (Shao et al. 2024h: DeepSeekMath initialized with DeepSeck-Codee Coder- v1.5 7B and continues pre- pre-training math-related tokens. We tested both DeepSeekMath- 7B-RI. and MeepSeek.Math777 BeepSeeetMath-77B-nnttu variants. Models are under Model License while code is under MIT License
Qwen2-Math (Yang et al. 2024b): Qwen2-Math is a series of specialized math language models built upon the Qwen2 LLMs. We evaluated 7B and 72B variants. They are under the same license as Qwen2-Instruct series.
Mathstral-7B (Mistral 2023): Mathstral stands on the shoulders of Mistral 7B and special- izes in STEM subjects. This model is published under Apache 2.0
(Beeching et al. 2024 This model is finetuned from DeepSeekMath-77 7B-bae with two stages supervised fine-tuning to solve math problems using chain of thought (CoT). is licensed under Apache 2.0.
Numinamath-7B-Con
C.2 EVALUATION PROMPTS
The evaluation prompts in experiments are given in Table 11 where detailed answer type deseriptions are given in Table T2 Following He et al. (2024a); Huang et al. (2024), these prompts are specially designed for different subjects and answer types for better evaluation. Note that, for chat models, we will apply chat template for better evaluation.
D RESULTS ACROSS DIFFERENT SUBJECTS
The detailed results across different subjects are given in Table 13
14 15
16
17
18
19
20
21
22
23
25
26
27
and From the results, we have the following observations:
OpenAI-ol- -ini achieves the best results across nearly all subjects, although GPT-4o sometimes excels in terms of RF.
For open-source LLMs, Diwen-2-Math772B-insmee achieves the best results in almost all subjects. However, Mistral-Large-iisstnet--2 outperforms Qwwnn22Math-72-iinttee in Algebra.
Some LLMs even achieve zero EAcc in certain subjects. For instance, Biktral-B--Instruee get zero EAce in Set Theory and Logic, and seven LLMs exhibit zero EAcc in Abstract Algebra.
The variation in the reasoning gap differs significantly across subjects, providing more fine- eme-inrieed information on how different LLMs perform across various domains.
24

---

Published as a conference paper at ICLR 2025
Table 11: Evaluation prompts for problems with single answer or multiple answers. (problem) the specific problem to evaluate. {subject) denotes the subject this problem belongs to and all subjects are given Table 6 fanswer_type type_ddscrippione are specified in Table T2] (number of of_anserrs) stands for the number of answers in the problem evaluated.
Evaluation Prompt for Single Answer
The following an undergraduaat-lever mathematical problem in (subject). You need solve the problem by completing all placeholders [ANS].
This problem involves only one placeholders [ANS] be completed. The answer type uunserr_ypee_dsscrpiion
Problem: [problem]
All mathematical formulas and symbols you output should be represented with LaTeX. Please end your response with: "The final answer is ANSWER where ANSWER should be your final answer.
Evaluation Prompt for Multiple Answers
The following is an undergraduate-level matheratical problem in (subject). You need to solve the problem by completing all placcholders [ANS].
This problem involves (number_of [uuberr_o_ansserr] types are, order, [answer_type sepe_ddecrpption.
annswrs)] placeholders [ANS] to be completed. Their answer
Problem: [problem)
All mathematical formulas and symbols you output should be represented with LaTeX. Please end your response with: "The final answer is ANSWER where ANSWER should be the sequence of your final answers, separated by commas.
Table 12:
Descriptions of answer types included evaluation prompts, where [options) ) the specific options from the multiple choice question evaluated.
Answer Type
Answer Type Description
NV
a numerical value without units
EX
an expression an equation a range interval
EQ
INT
TF
either True or False
MCS
one option for a multiple choice question with options [options]
MCM
than one option concatenated without space ple choice question with options [options), for example: BD a word, phrase, term or string satisfies the requirements of the problem
of multi-
OE
OL
an ordered of answers surrounded by parentheses with any answer types, for example (1, 2 Truc). where "ordered list" means changing the order of elements results in different answers
UOL
an unordered list of answers surrounded by parentheses with any answer types, for example, (1, 22, True). where "unordered list" means chang- ing the order of elements results the same answer
25

---

Published as a conference paper at ICLR 2025
Table 13: Results on Abstract Algebra (all figures are in %). The best results within each column are bolded and the best results of open-source Chat LLMs within similar parameter size group are underlined.
Models
Acc;
Ace:
Aces
AAcc
EAcc
A
RE
Closed-souree LLMy
OpeniAll-ii-minni222---0000000-
76.19
73.8r
11,42
73.8
57.14
T6.67
29.17
GPT-44-2022-08--0
42.86
50.00
52.38
48.41
28.57
19.84
69.44
GPT-40-mini-2024-077
26.19
35.71
38.09
33.33
14.29
19.04
133.2
Slade=330Opus22224022
23.81
38.10
26.19
29.37
11.90
17.47
146.8
Open-source
Chat LLMs
YF-13-6B--0 Chat
4.76
2.38
9.52
3,56
0.00
3,36
00
Mistral-7B-Instruue
2.38
2.38
0.00
1.59
0.00
1.59
Owen2-7B-Instruct
4.76
19.05
13.49
4.76
8.73
183.4
3LaMM33-B-Insttrn
0.00
7.14
9.52
5.56
0.00
5.56
00
Yi-1,5-9B-CM
7.14
14.29
11.90
11.11
0.00
11.11
Seatral-Nemo-nnstuet-22222
0.00
9.52
4.76
4.76
0.00
4.76
0.00
2.38
2.38
1.59
0.00
1.59
DeepSeck- V2-Lite-Chat
0.00
2.38
2.38
1.59
0.00
1.59
00
Mistral-Small-Inne -aaa---mmlaa-----t-------
9,52
19.05
21.43
16.67
7.14
9.53
133.5
11-1.5-34B-Chat
19.05
9.52
9.52
12.70
0.00
12.70
-LaMA3-70B-Instee
19.05
16.67
19.04
18.25
4.76
13.49
283.4
Qwen2 7-B--22BBBnc
28.57
47.62
30.95
35.71
14.29
21.42
149,9
Mistral-Large Instruct 2407
2407
35.71
35.71
45.24
38.89
21.43
17.46
81.47
Specialized Mathematicall
LLMS
DeepSeek Mathh-B-Insseie
7.14
9.52
4.76
7.14
0.00
7.14
DeepSeek- DeepSeek-MatthP25 Math-7B-RR 7B-RL RL
4.76
19.05
11.90
11.90
2.38
9.52
400.0
NuminaMath-7B-CCC
7.14
16.67
7.14
10.32
0.00
10.32
00
Mathstral-7B-vhl
9.52
4.76
7.14
7.14
0.00
7.14
Qwen2-Math-
23.81
33.33
28.57
28.57
14.29
14.28
99.93
Owen2-Math- Matt-MMat-hhmeee 72B-IIstuctt
45.24
40.48
42.86
42.86
26.19
16.67
63.65
E
ERROR ANALYSIS
We perform error analysis Section
5.4 and here we showcase several examples of various error
types in Table 29
30
and 31
The distribution of the relative error for OpenAl-ol -mii numerical values is illustrated in Figure We excluded numerical answers identical to the ground truth, as their logarithmic relative error would be negative infinity.
F RESULTS ACROSS DIFFERENT TOPICS
The performances of differerent LLMs 20 topics shown in Figure I0 12 and [3] We observe that different LLMs exhibit varying performance patterns across these topics, and even models within the same family show differences in their rankings.
G REFINEMENT RESULTS
Progressive-Hie Hint Prompting (PHP) (Zheng et al. 2023) is technique designed to enhance automatic, iterative interactions with LLMs. PHP uses previously generated answers as hints to progressively guide users toward the correct solutions. In our experiments, we employ the zero-shot manner to ensure fair comparison with the primary experiments in Table which helps to assess the impact of refinement
the performance of LLMs in solving undergraduate-level mathematical problems. We have set the maximum number of interaction rounds to five. To save cost, we only experiment with GPT-4o for closed-son source LLMs. The results presented in Table 32 While PHP can improve AAce and EAce in most cases, the improvements are not substantial. There remains considerable potential for enhancing the mathematical reasoning abilities of LLMMs in solving undergraduate-level
26

---

Published as a conference paper at ICLR 2025
Table 14: Results Algebra (all figures are in %). The best results within each column bolded and the best results of open-source Chat LLMs within similar parameter size group are underlined.
Models
Ace;
Ace2
Aces
AAcc
EAce
A
RE
Closed-source LLMs
OpenAl-oTFmini 2024009-I
T6.16
15,77
74.44
75.36
66.72
3.64
12,95
GPT-44-2024-00
70.50
71.53
72.04
71.36
64.67
6.69
10.34
N0T-44--ini-2024--00
67.41
67.24
66.38
67.01
59.52
7.49
12.58
Clande-3-0pus-20244022
63.29
66.38
63.81
64.49
54.20
10.29
18.99
Open-source
Chat LLMs
Yi-1.3-6B-hha
38.77
37.22
40.48
38.82
25.21
13.61
53.99
Mistral-7B-Instnur
19.73
21.96
19.55
20.41
11.32
9.09
80.30
Qwcn2-7B-Instmue
51.63
53.00
54.03
52.89
41.85
11.04
26.38
LLaMA3-8B- I-Stuutntt
30.53
33.28
30.19
31.33
18.35
12.98
70.74
Yi-1.5-9B-Chat
46.83
48.89
48.37
48.03
33.79
14.24
42.14
Mistral-Nemo-Instreee 2407
39.79
43.05
39.79
40.88
28.82
12.06
41.85
DeepScck-MON Baeeee-----------C
12.01
12.35
10.98
11.78
4.97
6.81
137.0
DeepSeek- Sar-ecck---------eeee Chttt
23.33
24.01
24.53
23.96
11.32
12.64
111.7
Distral-Small-IIstrue-----9
58.32
57.46
56.95
57.58
47.86
9.72
20.31
Yi-1.5-34B-Chat
53.34
53.00
52.66
53.00
40.31
12.69
31.48
LLaMA3-708 I0B-Instruct
50.09
51,80
51.11
51.00
40.48
10.52
25,99
Qwen2-72B-Instruee ListrallLarge--nssruut-----
63.81
63.46
64.49
63.92
54.72
9.20
16.81
68.44
68.10
67.92
68.15
59.86
8.29
13.85
Specialized Mathematical LLMs
DeepSeek PeepSeck-Math-222 Math-7B Math--B-Innsruut
40.31
38.77
38.25
39.11
26.07
13.04
50.02
DeepSeck-Math-7Pe
44.43
47.51
44.94
45.63
34.31
11.32
32.99
NuminaMath-77-CCo
45.80
45.45
46.14
45.80
34.65
11.15
32.18
Mathstral-7B-
45.97
43.91
43.57
44.48
32.25
12.23
37.92
Qwen2-Male
11122-MMth----B-1 -B-Instruct
56.09
57.46
57.63
57.06
47.86
9.20
19.22
1aan2-Math-701
1aail772B--Isnstt Instruct
67.58
67.75
67.24
67.52
58.83
8.69
14.77
35%
Threshold: 0.001
30%
25%
Percentage
20%
15%
10%-
5%
0%
-15
-10
-5
0
Log Relative Error
Figure 9: Relative Error of Numeric Answers excluding ones that are identical to the ground truth. Red dotted line indicates the tolerance threshold when we evaluate model answers.
mathematics. The results for PHP across different subjects for GPT-4o (see Table 33 indicate that the impact of PHP is subject-agnostic.
27

---

Published as a conference paper at ICLR 2025
Table 15: Main Results Arithmetic (all figures are in %). The best results within each column are bolded and the best results of open-source Chat LLMs within similar parameter size group are underlined.
Models
Acc;
Ace:
Ace,
AAcc
EAcc
A
RE
Closed-source LLMy
OpeniAl--ii-minni222---0000000-
93.48
92.86
94.10
93.48
90.37
3.11
3.44
GPT-40-2024-08-00
91.93
91.30
92.24
91.82
87.27
4.55
5.21
GPT-4o-mini-2022-077
89.44
87.27
90.06
88.92
83.23
5.69
6.84
2ouude3--0uus22224222
84.78
85.71
86.96
85.82
78.26
7.56
Open-source
Chat L.MS
YF-13-6B--0 Chat
63.98
64.9r
67.70
63.33
50700
IS.33
31706
Mistral-7B-Instnuue
30.43
34.78
33.54
32.92
15.22
17.70
116.3
Owen2-7B-Instruct
77.64
78.26
78.57
78.16
68.94
9.22
13.37
LLaMAA388-Insstne
51.24
53.42
54.97
53.21
33.54
19.67
58.65
Yi-1,5-9B-CM
73.60
73.60
70.81
72.67
59.32
13.35
22.51
Seatral-eemo-nnstuet-22222
69.88
70.19
72.67
70.91
55.28
15.63
28.27
DeepScek-MOE-165 16B-Chh
26.09
30.43
31.68
29.40
12.42
16.98
136.7
DeepSeck- V2-Lite-Char
50.31
53.42
49.07
50.93
32.92
18.01
54.71
82.61
84.78
81.68
83.02
73.29
9.73
13.28
11-1.5-34B-Chat
75.47
74.84
75.47
75.26
62.42
12.84
20.57
3LaMA3-70B-Instone
79.81
82.61
83.85
82.09
72.36
9.73
13.45
88.51
88.20
89.44
88.72
82.61
6.11
7.40
Mistral-Large Instruct 2407
2407
90.37
89.13
90.06
89.86
82.61
7.25
8.78
Specialized Mathematical LLMs
DeepSeek Baath-BBInstrrn
62.73
68.63
65.22
65.53
49.69
15.84
31.88
DeepSeek- DeepSeek-MatthP25 Math-7B-RR 7B-RL RL
74.53
76.71
76.71
75.98
67.08
8.90
13.27
NuminaMath-7B-CC
74.22
75.78
77.33
75.78
62.42
13.36
21.40
Mathstral-7B-vhh
71.74
72.67
72.36
72.26
59.63
12.63
21.18
Aren2-Math-7B--
88.51
84.78
86.65
86.65
78.88
7.77
9.85
90.68
90.68
90.99
90.79
86.96
3.83
4.40
H REASONING GAP AND TEST SET CONTAMINATION
Given the limited number of examples in the test set for one version (in terms of SFT), it is necessary mix the set with general mathematical SFT data. For our experiments, we adopt MetaMathQA (Yu ct al. 2023), high-quality SFT dataset for math word problems, which includes 395,000 training examples. We use Lama-3-8B as our base model and set the maximum output token length 4096. Following Tong et al. (2024), we set the learning rate to 5e-5, use warmup ratio of 0.03, adopt cosine decay, and train the model for one epoch. Training one model takes approximately three hours on four A100 GPUs. The results are presented in Table We set the proportion of the test set (from one version) for SFT to 5%, 10%, 15% and, 20% and see how the reasoning gap varies. As the proportion of the test set included in the training data increases, the reasoning gap (aloo becomes more pronounced. This study serves as an initial investigation into test set contamination during the SFT stage. is important note that contamination the pre-training stage is also significant of interest (Razeghi et al. 2022 Jiang et al. 2024).
28

---

Published as a conference paper at ICLR 2025
Table 16: Main Results Calculus multivariable (all figures in %). The best results within each column are bolded and the best results of open-source Chat LLMs within a similar parameter size group are underlined.
Models
Ace;
Ace3
Aces
AAcc
EAcc
A
RF
Closed-sone source LLMy
OpeniAl--1-miin-222240000
59.33
60.09
59.94
59.79
48.32
T1.47
23.74
GPT--4-20044-08066
50.00
49.69
49.24
49.64
38.23
11.41
29.85
GPT-4o-mini-2022400777
42.81
44.80
43.27
43.63
32.11
11.52
35.88
Claude-3-Opus-22240229
35.93
39.14
35.93
37.00
24.01
12.99
54.10
Open-source
Chat LLMs
Yi-1.3-6B-Chal
20.34
22,48
20.03
20.95
10.40
10.55
101.4
Mistral-7B-Instmuet
5.05
3.98
4.59
4.54
1.22
3.32
272.1
Owen2-7B-Instruct
25.38
25.23
26.15
25.59
16.06
9.53
59.34
LLaMA3-8B-inttruc
6.27
7.80
7.03
2.45
4.58
186.9
Yi-1.5-9B -.-aat
28.75
28.75
29.51
29.00
16.67
12.33
73.97
DistrllNermo-1mstuee240777
13.61
12.39
10.55
12.18
5.81
6.37
109.6
DeepScek-MOE-166--HG
1.68
0.46
1.22
1.12
0.00
1.12
DeepSeek-V2-Lite-Char V2-Lite-Chht
5.05
3.52
4.49
1.38
3.11
225.4
Siiiral-Small-Insttte
27.98
29.51
27.68
28.39
17.28
11.11
64.29
Yi-1.5- Yi-1.5-34B 1-h--3t-
26.45
27.98
26.15
26.86
14.22
12.64
88.89
LT
20.79
23.39
20.95
21.71
12.23
9.48
77.51
Qwen2-72B-Innn
35.02
37.92
35.32
36.09
23.55
12.54
53.25
Mistral-Large-Istteee Instruct-471
46.94
49.08
47.71
47.91
36.70
11.21
30.54
Specialized Mathematical
I.LMs
DeepScek-Mathh-7B Matth-BB--nnstruce
11.16
12.34
12.23
11.98
3.35
6.63
T23.9
DeepSeek-Math-T15
18.81
17.58
16.06
17.48
8.56
8.92
104.2
NuminaMath-77--C
20.49
20.80
22.02
21.10
11.47
9.63
83.96
Buthstral-7B-vO.
16.36
17.89
17.89
17.38
9.63
7.75
80.48
Qwen2-Matt Math--ath--.... IBIIntrucc
35.78
36.09
33.33
35.07
24.77
10.30
41.58
Qwen2- 1ren2-Math-Ao Math-72B-Insttnn
48.47
49.39
51.22
49.69
39.45
10.24
25.96
Table 17: Main Results Calculus single variable (all figures are in %). The best results within each column are bolded and the best results of open-source Chat LLMs within a similar parameter size group underlined.
Models
Ace1
Acc;
Aces
AAcc
EAce
A
RE
Closed source LLMs
OpenAllaol-minn-2224----111
72.81
73.32
72.61
72.91
59.37
13.54
GPT-40-20024-08-66
63.65
65.48
65.48
64.87
53.05
11.82
22.28
GPT-4o-mini-2024-07--1
54.68
57.43
55.91
56.01
42.26
13.75
32.53
Claude3--Ouss-22440229
47.96
51.32
50.61
49.97
34.73
15.24
43.88
Open-source Chat
Yi-1.3-BB-Chat
27.39
"3.57
"30.66"
29.97
16.80
13.17
78.39
Mistral-7--Instuun
8.86
10.29
8.96
9.37
4.38
4.99
113.9
Qwen2-7B-Instruct
39.51
41.96
39.41
40.29
27.70
12.59
45.45
LLaMA3-8B-Instrrue
13.14
13.95
13.44
13.51
7.03
92.18
Yi-1.5-9B-Chal
37.27
37.88
38.19
37.78
23.42
14.36
61.32
Mistral-Nemoo--nstuue2440
21.28
23.22
22.20
22.23
12.32
9.91
80.44
DepSSed-MMEE-IG--Cha.
4.58
4.48
4.48
4.51
1.22
3.29
269.67
PeepSeek-V2-Lit---
11.30
12.53
12.02
11.95
4.38
172.83
39.41
40.73
39.51
39.88
26.37
13.51
51.23
Yi-1.5-34B-CI
39.41
41.34
39.21
39,99
24.95
15.04
60.28
LLaMA3-70B-Instrice
32.89
36.46
34.21
34.52
22.81
11.71
51.34
Qwen2-72B Innstructt
47.86
50.20
48.88
48.98
33.50
15.48
46.21
Mistral-Large-Ineet Instruct-
57.33
59.17
59.57
58.69
45.93
12.76
27.78
Specialized Mathematical
L.MSs
23.883
27.339
23.833
23.002
13.75
1T.27
$1.36
DeepSeek- Math-7B-RL
30.04
30.65
30.65
30.45
20.06
10.39
51.79
NuninaMath-7 TB
32.08
32.89
32.79
32.59
20.16
12.43
61.66
Mathstral-7B-vol
27.70
27.19
29.53
28.14
16.29
11.85
72.74
Qwen2-Math-7BB
-a-Mtthh---nntteett Instruct
46.03
49.49
49.49
48.34
35.74
12.60
35.25
Owen2-Math-728. Instruct
61.71
61.81
60.08
61.20
49.29
11.91
24.16
29

---

Published as a conference paper at ICLR 2025
Table 18: Main Results Combinatorics (all figures in %). Models are classified into three categories according to their purpose and origin. The best results within each column are bolded and the best results of open-source Chat LLMs within a similar parameter size group are underlined.
Models
Ace1
Acc
Aces
AAce
EAce
A
RF
Closed source LLMs
OpeniAl--1-minnn2222--00
90.91
92.05
99..45
92.80
88.64
4.16
4.69
GPT--4--022400-066
77.27
77.27
78.41
77.65
61.36
16.29
26.55
GPT-44o-min-22224007-11
70.45
69.32
73.86
71.21
56.82
14.39
25.33
Claude3-Opus-22240029
56.82
65.91
68.18
63.64
50.00
13.64
27.28
Open-source Chat LLMs
Y1-1.3-6B-Chh
25.00
34,09
29.55
29.55
18.18
11.37
62.34
Mistral-7B-Instrus
10.23
12.50
11.36
11.36
2.27
9.09
400.4
Qwen2-7B-Instruct
39.77
45.45
44.32
43.18
28.41
14.77
51.99
LLaMA3-SB-Insiruct
22.73
23.86
23.86
23.48
14.77
58.97
Yi-1.5-9B Chat
32.95
43.18
44.32
40.15
25.00
15.15
60.60
MistralNeeme-nnstuc-2200
26.14
30.68
32.95
29.92
18.18
11.74
64.58
DeepSeck-MOEE-166-
6.82
11.36
7.95
8.71
2.27
6.44
283.7
DeepSeek- V2-aite--hht
17.05
22.73
12.50
17.42
6.82
10.60
155.43
Sattral-SmallIInsttee
46.59
59.09
50.00
51.89
38.64
13.25
34.29
Yi-1.5-34B-CHD
37.50
48.86
52.27
46.21
27.27
18.94
69.45
47.73
52.27
45.45
48.48
37.50
10.98
29.28
Qwen2-72B-Instee
60.23
63.64
60.23
61.36
47.73
13.63
28.56
Mistral-Large-Insse
2407
69.32
72.73
71.59
71.21
57.95
13.26
22.88
Specialized Mathematical
LLMs
DeepSeck- Maircc--MMahh-7--nntteeee
26.14
37.50
27.27
30.50
17.005
13.25
77.71
30.68
42.05
39.77
37.50
26.14
11.36
43.46
NuminaMath-7B-Cot
30.68
35.23
37.50
34.47
23.86
10.61
44.47
Mathstral-7B-ve Dowen-MMah--7e Instruct
34.09
38.64
35.23
35.98
26.14
9.84
37.64
62.50
62.50
64.77
63.26
47.73
15.53
32.54
Qwen2-Math Instruct
76.14
76.14
75.00
75.76
62.50
13.26
21.22
Table 19: Main Results Complex analysis (all figures are in %). The best results within each
column are bolded and the best results of open-source Chat LLMS within a similar parameter size group are underlined.
Models
Acc:
Acc;
Aces
AAcc
EAcc
A
RE
Closed-son source LLMs
OpenALcal-miini2222409-1122
74.51
74.51
76.47
75.16
64.71
10.45
16.15
GPT-40-2024-08-06
66.67
70.59
70.59
69.28
58.82
10.46
17.78
GPT-49-mini-22224007-1
62.75
62.75
62.75
62.75
50.98
11.77
23.09
Claude33-Opus:2224022
47.06
49.02
50.98
49.02
37.25
11.77
31.60
Open-source
Chat L.MMs
YF113-6B-Char
33.29
31.25
31.37
34.64
23.33
T1.IT
47.22
-istral-7B-Instmuett
11.76
11.76
9.80
11.11
5.88
5.23
88.95
Qwen2-7B-Instruet
45.10
50.98
43.14
46.41
35.29
11.12
31.51
LLaMA3-8B-Instru
19.61
17.65
17.65
18.30
13.73
4.57
33.28
Yi-1.5-9B-Chat
39.22
39.22
39.22
39.22
27.45
11.77
42,88
Sestaal-eemoo-nstuut-444000
29.41
25.49
25.49
26.80
19.61
7.19
36.66
DeepSeek-MOE-166-
1.96
3.92
1.96
2.61
0.00
2.61
DeepSeek- DeepSeek-V2-LLte-Che. V2-Lite-Chat
17.65
13.73
15.69
15.69
7.84
7.85
47.06
49.02
43.14
46.41
37.25
9.16
24.59
Yi-1 11.5-34B 5-34B-Chat
47.06
45.10
39.22
43.79
25.49
18.30
71.79
LLaMA3-70B-inntruut
43.14
33.33
37.25
37.91
25.49
12.42
48.72
Qwen2-7255 2a--n---2BIInttuee
52.94
62.75
62.75
59.48
47.06
12.42
26.39
Mistral-Large-IInne Instruct-
64.71
64.71
60.78
63.40
52.94
10.46
19.76
Specialized Mathematical L.LMs
35.29
1
21A55
30.72
23.53
7.T9
30.56
DeepSeek-Maii Math-7B-RL
35.29
41.18
47.06
41.18
27.45
13.73
50.02
NuminaMath- TB-CoT
33.33
35.29
37.25
35.29
21.57
13.72
63.61
Mathstral-7B-vo
37.25
29.41
31.37
32.68
15.69
16.99
108.3
Maren--Math77B-Innse
56.86
49.02
52.94
52.94
41.18
11.76
28.56
Dwen2-Math-72B.
70.59
72.55
72.55
71.90
64.71
7.19
11.11
30

---

Published as a conference paper at ICLR 2025
Table 20: Main Results on Differential equations (all figures are %). Models are classified into three categories according to their purpose and origin. The best results within each column are bolded and the best results open-source Chat LLMs within a similar parameter size group are underlined.
Models
Acc
Acc2
Aces
AAcc
EAcc
A
RE
Closed-sourg LLMs
29.84
29.18
29.51
29.51
22.62
6.89
30.46
2P7-44-20244-0-06
23.93
24.59
25.57
24.70
19.34
5.36
27.71
GPT-4o-mini-2224-07-BB
20.00
22.30
19.67
20.66
16.07
4.59
28.56
Claud=3--ppus-2224029
18.69
18.03
20.33
19.02
13.11
5.91
45.08
Open-source
Chat LLMs
Yi-1.5-6B-Chr
6.23
9.51
4.59
6.78
2.95
3.83
129.8
Mistral-7B-Instruct
3.28
3.93
4.59
3.93
1.64
2.29
139.6
Qwen2-7B-Instruct
11.80
12.13
11.48
11.80
7.54
4.26
56.50
L.LaMA3-S--Inttuut
4.26
5.90
5.57
5.25
2.62
2.63
100.4
Yi-1.5-9B-Chat
11.80
13.44
12.13
12.46
6.23
100.0
Mistral-Nemo-Insteee
8.52
7.87
9.18
8.52
4.26
4.26
100.0
DeepSec-MOEE-66B--0
0.66
1.31
0.00
0.66
0.00
0.66
DeepSeek-
2.62
2.62
1.97
2.40
0.33
2.07
627.3
Histral-Smmal--nnstuee
15.74
15.74
14.43
15.30
5.46
55.49
Yi-1,5-34B-Chal
12.13
15.08
12.13
13.11
7.21
5.90
81.83
L Instruct
12.79
13.44
12.13
12.79
7.54
5.25
69.63
Owen2-72B-Instrren
19.34
20.00
18.36
19.23
13.77
5.46
39.65
Mistral-Large-Insttutt---4
22.95
25.90
24.92
24.59
18.69
5.90
31.57
Specialized Mathematical LLMs
D
6.89
T.54
5.90
6.78
1.97
448T
249.2
DeepSeck-Math-7B-B.
8.52
10.82
9.84
9.73
5.25
4.48
85.33
NuminaMath-7B-CoT
10.82
10.16
9.62
5.36
125.8
Mathstral-7BB 7B-v...
10.49
10.49
9.51
10.16
4.92
5.24
106.5
Qwen2 MBt Instruct
15.41
18.69
18.03
17.38
12.13
5.25
43.28
Qwen2 Math 72B Instruct
22.95
22.95
22.95
22.95
17.38
5.57
32.05
31

---

Published as a conference paper at ICLR 2025
Table 21: Main Results on Financial mathematies (all figures are in %). Models are classified into three categories according to their purpose and origin. The best results within each column are bolded and the best results open-source Chat LLMs within a similar parameter size group are underlined.
Models
Acei
Acc2
Aces
AAcc
EAcc
RE
Closed-source LLMs
OpenAAlol-minn-222440000000
46.53
49.13
51.16
48.94
28.32
20.62
72.81
GPT-40-2022-08-06
28.61
29.77
29.77
29.38
18.21
11.17
61.34
GPT-46-minn-22024-07711
16.19
17.63
19.65
17.82
9.54
8.28
86,79
Clade=3--Ouus-22240229
19.65
21.39
20.52
20.52
10.98
9.54
86.89
Open-source Chat LLMs
Yi-1.5-6B-Cha
5.49
6,94
6.94
6.45
3.18
3.27
102.8
Mistral-7B-Instmnet
1.16
3.47
2.02
2.22
0.87
1.35
155.2
Qwen2-7B-Instructt
8.67
12.43
8.96
10.02
3.76
6.26
166.5
LLaMA3-SB-Instruct
2.60
2.02
3.18
2.60
1.16
1.44
124.1
Yi-1.5-9B-CC
9.25
14.45
12.43
12.04
4.05
197.2
Mistral-Namo-Inssrut22224
4.62
5.49
6.07
5.39
1.73
3.66
211.6
MeepScck--MF--66B---
0.58
0.87
1.16
0.87
0.00
0.87
DeepSeek-V2-Lite- V2-Lite-Chaa
3.76
3.76
3.76
3.76
1.45
2.31
159.3
Mistral-Small Small-Instruet-24000
12.43
11.85
13.01
12.43
5.78
115.1
Yi-1.5-34B-Chai
11.85
14.45
13.58
13.29
5.20
8.09
155.6
LLaMA3-70B- Instrruut
8.09
10.12
9.25
9.15
4.62
4.53
98.05
Qwen2-72B-Instruu
15.03
18.50
16.47
16.67
8.96
7.71
86,05
Mistral-Large-Instruutt
22.54
26.59
21.97
23.70
15.32
54.70
Specialized Mathematical
LLMs
4.91
6.07
4.62
5.20
2.31
2.89
125.
DeepSeck-Math-7B-..
5.49
7.51
5.78
6.26
2.02
4.24
209.9
NuminaMath-7B--Cn
8.67
9.25
9.25
3.76
141.0
Mathstral-7B. 7B--str 10.1
4.91
9.83
7.23
7.32
3.18
4.14
130,2
Qwen2 Math- 7B Instruct
10.69
17.05
15.32
14.35
5,49
8.86
161.38
Qwen2 Math 72B Instruct
28.90
26.88
29.19
28.32
16.47
11.85
71.95
32

---

Published as a conference paper at ICLR 2025
Table 22: Main Results Geometry (all figures in %). Models are classified into three categories according to their purpose and origin. The best results within each column are bolded and the best results of open-source Chat LLMs within a similar parameter size group are underlined.
Models
Ace1
Acc2
Aces
AAec
EAcc
A
RE
Closed-source LLMs
71.43
67.08
71.43
69.98
58.39
T1.39
T9.35
GPT--4--0044008066
64,60
63.98
63.98
64.18
51.55
12.63
24.50
GPT--4o-mimi-22044-07-1-
60.87
55.90
59.63
58.80
44.10
14.70
33.33
Claude-3-0puu-20224022
S7.76
49.69
49.07
$2.17
37.89
14.28
37.69
Open-source
Chat LLMs
Yi-1.3-6B-Cha.
32.92
27.33
30,43
30.23
18.01
T2.22
67.85
Mistral-7B-Insse
8.70
8.70
6.83
8.07
3.11
4.96
159.5
Qwen2-7B-Instruct
50.31
39.75
42.86
44.31
29.19
15.12
$1.80
LLaMA338B-Instmu
24.84
19.88
19.88
21.53
9.94
11.59
Yi-1,5-9B-Che
38.51
37.89
36.02
37.47
22.36
15.11
67.58
Mistrl-Nemmo-Imsrrot-4000
31.68
27.95
28.57
29.40
16.77
12.63
75.31
DeepSeck-MOE-166B-
4.97
3.73
6.21
4.97
1.86
3.11
167.2
DeepSeek-Ve BaaSSeek-VV-Lite--hhee
11.18
14.29
13.46
4.97
8.49
170.8
Mistral-Small-- S0mmamllsstuutt-24000
44.72
47.21
46.58
46.17
32.92
13,25
40.25
Yi-1,5-34B-CH
43.48
37.89
44.72
42.03
27.33
14.70
53.79
39.13
38.51
39.75
39.13
25.47
13.66
53.63
Twen2-72B-Bo Instructt
59.01
49.69
53.42
54.04
13.05
31.84
Mistral-Large-Isttueee 2407
61.49
57.76
60,25
59.83
50.31
9.52
18.92
Specialized Mathemaricall LLMs
Deep Tarnseck-Math-7RRE Math-7BB-nnstruee
29.19
24.22
24.22
25.88
T4.29
T1.59
81.1T
DeepSeek DeepSeek-Math-7D5 Math--B-RL
34.78
29.19
29.81
31.26
19.25
12.01
62.39
SaminnaMtth-BB-Ca
33.54
31.68
28.57
31.26
19.25
12.01
62.39
Mathstral-7B 1
39.13
32.92
37.89
36.65
24.22
12.43
51.32
Owen2-Math- Math-7B-aa -B-htreeInnttrut
52.80
42.86
45.96
47.20
34.16
13.04
38.17
Qwen2- Math-
67.08
61.49
64.60
64.39
53.42
10.97
20.54
Table 23: Main Results Linear algebra (all figures in %).. Models are classified into three categories according to their purpose and origin. The best results within each column are bolded and the best results of open-source Chat LLMs within similar parameter size group are underlined.
Models
Acc:
Acc
Aces
AAcc
EAcc
A
RE
Closed source LLMs
OpenALoal-miin-2222409-1122
74.90
74.90
75.90
75.23
65.86
9.37
GPT-40-2024-08-066
63.86
61.65
61.65
62.38
51.61
10.77
20.87
GPT-46-minn-22224007-1
57.63
57.03
56.22
56.96
43.37
13.59
31.34
Claude-3-Opus22224022
51.20
48.80
49.00
49.67
37.15
12.52
33.70
Open-source Chat L..Ms
YF113-6B-Chrr
23.59
23.69
21.49
22.96
T2.45
TO.ST
84,42
-istral-7B-Insttuett
8.03
7.43
7.23
7.56
2.61
4.95
189.7
Qwen2-7B-Instruc
36.35
34.54
32.73
34.54
22.29
12.25
54.96
LLaMA3-8B-Instru
13.25
12.45
12.65
12.78
6.43
6.35
Yi-1.5-9B-Chat
34.14
31.93
31.93
32.66
18.47
14.19
76.83
Seataal-eeno--nstuut-444000
23.69
20.48
23.90
22.69
11.24
11.45
101.9
DeepSeek-MOE-166-
2.01
2.41
3.61
2.68
0.80
1.88
235.0
DeepSeek- DeepSeek-V2-Lite-hhe V2-aite-Chat
9.24
7.43
8.30
1.81
6.49
358.6
43.17
38.35
41.57
41.03
28.51
12.52
43.91
Yi-1 11-1.5-344 1.5-34B 5-34B-Chat
35.34
31.93
34.94
34.07
20.08
13.99
69.67
LLaMA3-70B-inntruuc
34.34
30.72
29.32
31.46
19.28
12.18
63.17
Qwen2-7255 2
52.81
51.00
49.80
51.20
38.15
13.05
34,21
Listral-Large- Instruct-240
62.25
59.04
58.43
59.91
48.59
11.32
23.30
Specialized Mathematical
L.LMs
20.08
21.78
1B.00
19.75
9.44
TO.3T
TO9.E
DeepSeek-Matie Math-7B-RL
26.91
26.51
23.09
25.50
15.46
10.04
64.94
NuminaMath- TB-CoT
25.70
25.50
27.11
26.10
14.66
11.44
78.04
Mathstral-7B-v0
29.92
23.90
24.70
26.17
13.86
12.31
88.82
46.39
43.17
43.57
44.38
32.73
11.65
35.59
Dwen2-Math-72B.
61.45
59.24
62.85
61.18
47.59
13.59
28.56
33

---

Published as a conference paper at ICLR 2025
Table 24: Main Results Number theory (all figures are in %). Models classified into three categories according to their purpose and origin. The best results within each column are bolded and the best results of open-source Chat LLMs within a similar parameter size group are underlined.
Models
Ace;
Ace3
Aces
AAcc
EAcc
A
RF
Closed source LLMy
TheniAl--l--mii-222--50
80.43
76.09
73.91
76.3f
58.70
T8.1
30.85
GPT--4-2024-08-066
60.87
63.04
67.39
63.77
56.52
7.25
12.83
GPT-4o-mini-2022447777
60.87
58.70
58.70
59.42
52.17
7.25
13.90
Claude-3-Opus-22240229
47.83
54.35
56.52
$2.90
39.13
13.77
35.19
Open-source
Chat LLMs
Yi-1.5-6B-Chal
26.09
26.09
19.37
23.91
13.04
10.87
83,36
Mistral-7B-Instuuet
6.52
6.52
8.70
7.25
2.17
5.08
234.1
Qwen2-7B-Instruct
34.78
39.13
41.30
38.41
23.91
14.50
60.64
LLaMA3-8B-Instrue
19.57
21.74
15.22
18.84
8.70
10.14
116.6
Vi-1.5-9B-hhr
34.78
32.61
34.78
34.06
23.91
10.15
42.45
Destral-eemo-Issuuee240777
17.39
28.26
28.26
24.64
8.70
15.94
183.2
DeepScek-MOE-166--CC
6.52
6.52
2.17
5.07
2.17
2.90
133.6
DeepSeek-V2-Lite-Char V2-Lite-Chat
15.22
13.04
15.22
14.49
4.35
10.14
45.65
34.78
41.30
40.58
30.43
10.15
33.36
Yi-1.5- 1i-1.5-34B 1-5-33t-
36.96
34.78
47.83
39.86
26.09
13.77
52.78
LT L
30.43
23.91
36.96
30.43
15.22
15.21
99.93
Qwen2-72B-Innn
47.83
58.70
43.48
50.00
36.96
13.04
35.28
Mistral-Large-Istteee 1nstruct-47777
58.70
58.70
58.70
58.70
45.65
13.05
28.59
Specialized Mathematical
L.LMs
DeepSček- DeepScck-Mathh-TB- Matth-BB-Bnnnttucc
23.91
28.26
30.43
27.34
T5.22
T2.32
80.95
DeepSeck-Mathh715
21.74
26.09
26.09
24.64
13.04
11.60
NuminaMath-77--C
17.39
32.61
32.61
27.54
13.04
14.50
111.2
Mathstral-7B-vO Qwen2-Matt Matt--MthhB.... IBIIntrucct
28.26
30.43
30.43
29.71
17.39
12.32
70.85
39.13
43.48
43.48
42.03
30.43
11.60
38.12
Qwen2- 1ron2-Math-To Math-72B-Innstn
60.87
54.35
63.04
59.42
50.00
9.42
18.84
Table 25: Main Results on Probability (all figures are in %)..
Models classified into three categories according to their purpose and origin. The best results within each column are bolded and the best results of open-source Chat LLMs within similar parameter size group are underlined.
Models
Ace:
Acc
Aces
AAcc
EAcc
A
RE
Closed-souree LLMs
Openallallmmini-2224409--
66.07
68.45
68.15
67.56
51.49
16.07
31.21
GPT-40-2024-08-06
66.07
66.07
65.77
65.97
54.76
11.21
20.47
GPT-46-minn-20224007-1
50.89
56.85
52.38
53.37
39.88
13.49
33.83
Claude-3-Opus:2224022
56.55
59.23
59.82
58.53
44.94
13.59
30.24
Open-source Chat L.MMs
YF113-6B-Chr
20.24
22,62
21.33
21.43
T0.42
T1O0T
T03.7
-Bstral-7B-Insttuett
10.12
11.01
10.42
10.52
3.57
6.95
194.7
Qwen2-7B-Instruct
31.85
39.58
33.63
35.02
20.54
14.48
70.50
LLaMA3-8B-Instru
16.37
15.48
16.27
9.23
7.04
76.27
Yi-1.5-9B-Chat
28.27
30.06
31.85
30.06
13.69
16.37
119.6
Sestaal-eemoo-nstuut-444000
23.81
26.79
25.00
25.20
15.18
10.02
66.01
DeepSeek-MOE-166-
5.65
4.76
4.17
4.86
1.19
3.67
308.4
DeepSeek-V22 DeepSeek-V2-Liie-Che V2-Lite-Chat
9.82
10.42
3.27
7.35
224.8
39.58
43.15
43.45
42.06
27.68
14.38
51.95
Yi-1 11-1.5-344 1.5-34B 5-34B-Chat
37.20
41.67
38.39
39.09
23.51
15.58
66.27
LLaMA3-70B-inntruuc
35.12
36.31
34.23
35.22
22.02
13.20
59,95
Qwen2-7255 20--2-2-BBInntrue
48.51
48.51
44.05
47.02
33.04
13.98
42.31
Listral-Large-Innne Instruct-
59.82
60.12
58.33
59.42
45,54
13.88
30.48
Specialized Mathematicall L.LMs
21.73
12.52
21.4.
12.02
TO.7T
TI.3T
T03.6
DeepSeek-Matie Math--B-hh----
24.40
25.89
26.19
25.50
14.88
10.62
71.37
NuminaMath- TB-CoT
25.60
26.49
25.30
25.79
13.39
12.40
92.61
Mathstral-7B-vo
27.38
29.46
25.30
27.38
14.88
12.50
84,01
Brren--Math77B-nnese
36.61
44.35
39.88
40.28
25.89
14.39
55.58
Dwen2-Math-72B.
58.33
62.20
59.52
60.02
44.05
15.97
36.25
34

---

Published as a conference paper at ICLR 2025
Table 26: Main Results Set theory and logic (all figures in %). Models classified into three categories according to their purpose and origin. The best results within each column are bolded and the best results of open-source Chat LLMs within similar parameter size group are underlined.
Models
Ace;
Ace3
Aces
AAcc
EAcc
A
RE
Closed source LLMy
OpeniAl--1--iii-2002-0000
78.26
13.91
79.71
77.29
66.67
T0.62
15.93
GPT-44-20044008066
75.36
76.81
68.12
73.43
59.42
14.01
23.58
GPT-4o-mini-20224-0771
59.42
57.97
59.42
58.94
40.58
18.36
45.24
Claude-3-Opus-22240229
56.52
60.87
62.32
59.90
43.48
16.42
37.76
Open-source
Chat LLMs
Yi-1.3-6B-Chal
14.49
14,49
18.84
15.94
7.25
8.69
119.9
Mistral-7B-Instuuet
2.90
5.80
5.80
4.83
0.00
4.83
Owen2-7B-Instruct
20.29
36.23
24.64
27.05
13.04
14.01
107.4
LLaMA3-8B-instrue
8.70
8.70
11.11
5.80
5.31
91.55
Yi-1.5-9B -.--at
24.64
18.84
26.09
23.19
5.80
17.39
299.8
Distrll-ermoommstuee240777
24.64
33.33
26.09
28.02
18.84
9.18
48.73
DeepScek-MOE-160--HG
2.90
1.45
1.45
1.93
0.00
1.93
DeepSeek-V2-Lite-Char V2-Lite-Chht
4.35
7.25
5.80
5.80
1.45
4.35
300.0
47.83
44.93
40.58
44.44
30.43
14.01
46.04
Yi-1.5- 1i-1.5-34B 1-h--3t-
30.43
40.58
43.48
38.16
21.74
16.42
75.53
LT L
28.99
26.09
24.64
26.57
15.94
10.63
66.69
52.17
55.07
49.28
52.17
36.23
15.94
44.00
Mistral-Large-Istteee Instructt41
59.42
60.87
57.97
59.42
44.93
14.49
32.25
Specialized Mathematical
I.LMs
DeepSček- DeepScek-Mathh-BB Mathh-BB--nnstruce
21.74
18.84
13.04
17387
3.70
9.T7
TO5.4
DeepSeek-Mathh
23.19
23.19
24.64
23.67
10.14
13.53
133.4
NuminaMath-77--C Mathstral-7B-vO 1ren2-Math-775 IB-Inttucc
17.39
26.09
18.84
20.77
8.70
12.07
138.7
23.19
26.09
20.29
23.19
14.49
8.70
60.04
31.88
39.13
27.54
32.85
18.84
14.01
74.36
1ao2-MMth-7225
60.87
56.52
55.07
57.49
40.58
16.91
41.67
Table 27: Main Results on Statistics (all figures in %). Models classified into three categories according to their purpose and origin. The best results within each column are bolded and the best results of open-source Chat LLMs within similar parameter size group are underlined.
Models
Ace:
Acc
Aces
AAcc
EAcc
A
RE
fooed-souree LLMs
Openallallmmini-2224409--
64.09
62.34
61.85
62.76
49.88
12.88
25.82
GPT-40-2024-08-06
59.10
62,09
60.10
60.43
49.38
11.05
22.38
GPT-49-minn-22224007-1
43.14
46.38
48.63
46.05
32.92
13.13
39.88
Claude-3-Opus:2224022
52.12
53.12
50.37
51.87
37.16
14.71
39.59
Open-souree
Chat L.LMs
YF113-6B-Char
13.96
T6.96
21.45
18.12
6.23
T1389
T90.9
-istral-7B-Instmuett
11.97
11.72
11.47
11.72
2.99
8.73
291.0
Qwen2-7B-Instruct
26.68
27.93
29.18
27.93
13.97
13.96
99.93
LLaMA3-8B-Instru
10.72
13.47
16.21
13.47
5.99
7.48
124.9
Yi-1.5-9B-Chat
26.43
23.94
31.67
27.35
13.97
13.38
95.78
Distral-Nemmo-Instut---2A
22.94
27.18
24.94
25.02
14.71
10.31
70.09
DeepSeek-MOE-166-
3.24
3.49
4.74
3.82
1.00
2.82
282.0
DeepSeek- DeepSeek-V2-Liie-Che. V2-Lite-Cha
8.98
9.23
8.89
2.74
6.15
224.5
Mistral Ssmall-Intrrut-22222
37.66
35.91
35.66
36.41
23.69
12.72
53.69
Yi-1 F1-1.5-344 5-34B-Chat
32.92
37.41
37.91
36.08
21.45
14.63
68.21
LLaMAA-70B-inntruuc
22.44
20.70
21.45
21.53
11.97
9.56
79.87
Twen2-72B-Innee
43.14
44.39
44.89
44.14
31.42
12.72
40.48
Mistral-Large Instruct-240
54.36
50.12
53.62
$2.70
39.90
12.80
32,08
Specialized Mathematicall L.LMs
18.95
T6.36
18.95
T8.25
6.73
T1:56
T77.8
DeepSeek-Maie Math-7B-RL
21.70
24.44
22.69
22.94
9.23
13.71
148.5
NuminaMath- TB-CooTT
17.71
18.20
18.95
18.29
6.48
11.81
182.3
Mathstral-7B-vo
19.70
22.69
21.70
21.36
9.73
11.63
119.5
31.42
31.17
37.66
33.42
17.71
15.71
88.71
Dwen2-Math-72B.
47.38
50.87
47.63
48.63
33.92
14.71
43.37
35

---

Published as a conference paper at ICLR 2025
Table 28: Main Results Trigonometry (all figures are in %). Models classified into three categories according to their purpose and origin. The best results within each column are bolded and the best results of open-source Chat LLMs within similar parameter size group are underlined.
Models
Acc1
Acc;
Aces
AAcc
EAcc
A
RE
Closed- source LLMs
OpenAl-ol--mii-22244000
75.84
74.16
69.10
73.03
56.74
16.29
28.71
GPT-44-2024-08-06
67.98
74.16
66.29
69.48
51.69
17.79
34.42
GPT-44--min-22024007-18
52.25
58.99
55.06
55.43
37.64
17.79
47.26
Claude33-Opus-22240029
53.93
52.81
53.37
53.37
37.64
15.73
41.79
Open-source
Chat LLMs
Yr-13--6B-hha
24.16
24.72
20.79
23.22
T1.24
T1,98
TO6.5
Mistral-7B-Innttuet
6.74
7.87
6.18
6.93
1.69
5.24
310.0
Owen2-7B-Instruct
36.52
41.01
42.13
39.89
25.84
14.05
54.37
LLaMA3-8B-Inss I-s-ruuttt
14.61
13.48
13.67
4.49
9.18
204.5
Yi-1.5-9B-Chal
32.02
30.34
32.02
31.46
15.73
15.73
100.0
Instruct-2407
23.03
23.60
24.72
23.78
11.80
11.98
101.5
3.93
3.93
4.49
4.12
0.00
4.12
DeepSeek- DeepSeek-V2-LLte--e Chat
10.67
15.73
11.80
12.73
3.37
9.36
277.7
38.76
41.01
45.51
41.76
27.53
14.23
51.69
Yi-1.5-347 Chat
44.38
39.33
37.64
40.45
25.28
15.17
60.01
LLaMA3-70B-Inse
33.71
36.52
34.83
35.02
23.03
11.99
52.06
Qwen2-72B-Innn
52.25
50.56
50.00
50.94
35.39
15.55
43.94
Mistral-Large-Insttue--000
65.73
64.61
66.85
65.73
50.00
15.73
31.46
Specialized Mathematical
LLMs
DeepSée--Malii-t---nntt
26.40
28.09
29.78
28.09
T1.24
I6.83
I49.9
DeepSeek-Math: Manpseee--Mtth-BB-A
32.02
33.71
36.52
34.08
20.22
13.86
NuminaMath CoT
33.71
34.83
30.34
32.96
17.42
15.54
89.21
Mathstral-7B 7B-v0..
30.34
31.46
34.27
32.02
18.54
13.48
72.71
Qwen2-Math --10--Mtth-BB In-Instrcc
53.93
50.00
51.69
51.87
38.76
13.11
33,82
Owen2-Math-72B-inse
62.92
65.17
63.48
63.86
50.00
13.86
27.72
36

---

22 Fs 8.5 Published as a conference paper at ICLR 2025
Table 29: An example of Linear algebra with expression calculation error.
Problem:
You have boxes of five types. They are characterized by their length L, their width W, their height H, their age A, and their price P, as given in the following table:
type
L
W
H
A
P
1
1
1
1
1
II
1
1
2
3
2
III
1
2
3
2
2
IV
4
1
2
V
1
2
5
1
3
So for example, boxes of type IV measure cost $2 each. Suppose the sum the lengths your boxes 25, sum of their widths is 40, the sum of their heights is 75, the sum their ages is 43, and the total price of those boxes is 51. You have [ANS] boxes of type [ANS] boxes of type II, [ANS] boxes of type III. [ANS] boxes of type IV. [ANS] boxes of type
by feet, year old, and
ol-mini Completion: To solve the system of equations based the given constraints, we define the variables as follows:
21 Number of Type I boxes
22 Number of Type boxes
23 Number of Type III boxes
24 Number of Type IV boxes
275 Number of Type boxes
Using the provided data, we set up the following equations:
1.
25
æ1 + 224 27,
50, 75 3. x1 + 2x2 323
4. #1 + 32g + 27g + 24 Jg 43
5. 21 + 2mg + 22g + 2xg 3xs 51
By systematically solving these equations, find the values of the variables:
25 2.5
Ø1 25 1.5
Z3 25 + = 3.5
24 14 2z,
[It skipped detailed steps and obtained incorrect results in equation manipulation]
**The final answers 1.5, 8.5, 3.5, 2.5
Ground- Trouhd-Truth Answer: 4, 6, 6, 4,
Model Output: 1.5, 8.5, 3.5, 9, 2.5
37

---

Published as a conference paper at ICLR 2025
Table 30: An example of Financial mathematics with numeric calculation error.
Problem: How long will it take for $3100 compounded semiannually annual rate 1% to amount $7000? [ANS] years
ol-mini Completion: To determine how long will take for $3100 grow to $7000 with semiannual compounding at an annual rate 1%, we can use compound interest formula:
A= P (1+
Where: 7000 (the final amount) = 3100 (the principal amount)
= 0.01 (the annual interest rate) n = (the number of compounding periods year) f is the time in years Plugging in the known values:
0.01
7000 3100
+ 2
7000
(1.005)
3100
2.25806/1516 = (1.005)
Taking the natural logarithm of both sides:
(2 ((202555)))1))) 21 lm(10055)
In ln2.2258064516)
(n.000)))
Calculating the values:
lm22.258064516) 0.8154244
[should be 0.8 0.8145080376)
ln(1.005) 0.00498752
[should be 0.00498754141]
0.8154144
A 81.56
0.00498752
0.8154244
With the wrong parameters, should be
81.746479921
2 0.00498752
With the right parameters, should be
0.8145000036
81.6542615
0.00498754141
The final answer 81.56
Ground -ruth Answer: 81. 499649991
Model Output: 81.56
38

---

Published as a conference paper at ICLR 2025
Table 31: An example of Financial mathematics with flawed reasoning.
If you make a deposit into bank account, what interest rate (com- pounded weekly) should you invest if you would like to double your investment in 78 weeks? [ANS] % (Note: Your answer should be accurate to two decimal places)
Problem:
ol-mini Completion: To determine the interest rate r (compounded weekly) required double your investment in 78 weeks, can use the compound interest formula:
FV PV [1
100
here should be (1+ r ) for 52 weeks year
Where: FV is the future value IPV is the present value r the weekly interest rate (in percentage) is the number of weeks Given that you want to double your investment:
TS
=
100
To solve for r, take the natural logarithm of both sides:
In(2) = 78 x In
1 +
100
ln(
In
+ 100
78
In/31
+
78
100
ln(2)
100
In(3)
e
x 100
Calculating the above:
691147 78
1) x 100 ~ 0.89%
The final answer 0.89
Ground-Trufe -ruth Answer: 46.4157
Model Output: 0.89
39

---

Published as a conference paper at ICLR 2025
al mini 2024-09 09-12-992
gpt 4p mini 2024-07-15
Integers
95.0%
Integers
92.5%
Limits and continulty
Limits contisuity
64.7%
Systems of linerr equations
75.7%
Linear equations and functions
61. 11%
Linear equations functions-
67.0%
Difierentiation
57.5%
retiiten
64, 7%
Bystems of line equations
55 69%
Integration of multivariable functions-
63, 8%
Sample Space
50.330
Sample Space-
60.1%
Vector genmetry
42. 42.5%
A Applications of differentiatio
58.9%
Apglications of diff diffeeeitiation
4.0 8966
Infinite sequinces sertes-
57.2%
sequences and series
29.
Vactor geomeity
54.8%%
Defferentiation of muliiariahls functions
35.2%
Applications of integration
54.7%
Continuous distributions
34.7%
Vector calculus-
51.9%
Vector calculus
32.1%
Confilence intervals
49.0%
Calculus of vactor valued functions
32.0%
Continuous distributions
14.
Confidence Intervals
31.3%
Differesttation of multivartable functions-
44.7%
Level
Integration of multvartable functions
30.0%
Level
39.3%
Level 2
Applications integration
28.4%
Level
Hypothesis tests
5%
Level
Hypothesis tests
19.8%
Level 3
Calculus of vector valued functions-
28. 9%
Level
First order differentall equations
15.1% 1%
Lavel 4
First order difforential equations
19.9%%
Level
Internat
13.5%
Annulties-
10.8%
Level
Annulties 0.0
Level 6
0.0
0.2
0.6
0.8
FAce
gpt-4o-2024
Mastral- Lesstrl-LLrge-Beee Sastral-Largg-mmstruu--20000 Instruct-2to077
Integers
41.7%
Integers
80.2%
Limits and continity
82.44%
Limits and continuity
67.3%
Sample Space
74.1%%
Linear equations and functions
66.15%
Linear equations functions
67.0%
Systems line equations Di flmmentttiionn
61.1%
Systema of lineer equations.
66.0%
60.8%%
Ditforentitiio
64.7%
Sample Space
57.3%
Applications of difforentistion
52.8%
Vector geametry
47.3%%
Vector geometry
48.0%
Applications of differentiation
43.00%
Infinite sequences series
47.1%
Incinite secuences and sertes
Confodence intervals
47.0%
Integration of muktvariable functions
30. 99
Applications of integration
45.3%
Applications integration
37.2%
Continusus distributions
42.9%
Vettor calculus
36.8%
Integration of multivartbble functions
41.3%
Confelence intervals
34.9%
Hypothosis
29.7%
Continuous distriuttons
24.7%
Diffrestiation of multivarfable functions
36.39
Level
Detirentiation of multivrrabll functions
33.5%
Level
Vector calealus
35.8%
Level
Hypothesis
31.9%
Level
Calcalus of vector valued functions
32.0%
Level 3
Calculus of vactor valued functions Inferest
20,
Level 3
Interest
28.8%
Level
23.3%
Level
Pirst order differential equations
10.5%
Level
First order differentaal equations
10.5%
Annulties
0.8%
Level
Annulties 0. 0%
Level 6
0.0
0.2
0.8
EAce
Owen2- Miten2-Math-2aa -nstrrtt
Qweb2-72B B2s-tstt
Integers
87.5%
Integers
87.5%
Dificrentsstice
65.4%
Linear equations and functions
58.3%
Limits and continuity
64.7%
Dxfferentatiin
54.9%%
Linear equations and functions
Systems linear equitions
51.4%
Sample Space-
61.5%
Limits and continuity
48.2%
Systems lineer equations.
57.6%
Sample Space
46.2%
Vactor geometry
51.4%
Vector geometry
39.
Infinite sequinces and series-
48.6%
Applications of differentittinn
34.0%
Applications of differentia tice
$0.3%
Infinite sequenoes and series
20.3%
Vector calculus
44.3%
Difirentiation of multivrrable functions
26.8%
Integration of multivartable functions
41.3%
Calculus of vector valued functions
25.8%
Differentlation of multivarfbble functions
38.00 33..0%
Hypotheais tests
23.3%
Applications of integration.
37.22 .9%
Vector calrulus
21.7%
Confidence intervals
32.5%%
Confidence intervals
20.5%
Continuous distributions
28.6%
Level
Continuous distrinutions
19.4%
Level 1
Calculus vector valued functions-
27.8%
Level
Integration multvariable functions
Interest
27.6%
Level
Applications of integration
15.5%
Level 3
Hypathesis
25.0%
Level
Intereat
14.7%
Level
First order differential equations Annultios
17.1%
Lavel
First order ditffrential equations
11.0%
Lavel
1.7%
Level
Annuitios
0.0%
Level
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Edce
BAcc
Figure 10: Model accuracy across topics
40

---

Published as a conference paper at ICLR 2025
claude 3 opus 20240229
Qwen2 Mith 7B II-stuutttt
Integers
85.0%
Integers
80.8%
Sample Space
Limits and contiouity
60 60.0%
Linear equations and functions
56.55
Linear equations and functions
51 .1%
and continulty
56.5%
Defferentiation
49.7%
Systems of lnner equations
47. 2%
Systems of Linear ar equations
46. 59.55%
Diférrentazicon
46.4%
Sample Space-
377 8%%
Applications of differentistio
37.00
Applications of éifferenicctonn
35.0%
Vactor geometry
2.%
Integration of mullivariable functions
51.9%
Continueus distributions
Vector geometry
30.11
Infinito se sequinoss and saries.
29.0%
Infinite sequenoes and series.
28. 3%
Differsstiation of muitivarfable functions
26.3%
Vector calculus
26.4%
Calculus of vector valued functions-
24.7%
Applications of integration
26.4%
Hypothoois tests
24.1%
Calesluss of vector valued functions
22.7%
Confidence intervais
24.1%
Differentiation of multivariable fun.ctions
Integration of multivartable functions-
21.0%%
Level
Continueus distributions
19.4%
Level
Applications of integration
20.99%
Level
Confedence intervals
10.1%
Level
Vector calculus-
17.9%
Level 3
First order differential equations 12. 3%
Level
First order difforentidl equations
Interest
10.3%
17.8%
Lavel
Hypothesis
Interest
9.8%
Lavel
Annulties
0.8%
Level 6
Annuities
0.0%
Level
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Meta- 1ama-33708- Instruct
Mistral-Sman Sww.llllnsttuctt22e555
Integers
80.0%
Integers
76.7%
Diffemntistion
30.5%
Linear equations and fun cunntions-
65.7%
Linear equations and functionn.
39.1%
Limits and continuity
45.9%
Sample Space-
33.6%%
Sample Space
41.3%
Limits and contimuity
32.9%
Diff-rentictin
39.255
Applications of difforentiaiion
22.6%
Systems linear equations
37.5%
Systems of linser equations-
22. 22.2%%
Applications of difiorentistion
30.6%
Vector geumetry
23.99%
Victor geametry
30.1%
Infinite sequences series-
Calculus vector valued fun:tiions
Calculus of vector valued functions-
14. 4%
Differentaation of multivarablee functions-
22.99%
Diffecectiation of multivariable functions-
13.4%
Continuous distributions
19444%
Continuous distributions-
11.2%
Infinite sequences and series-
18.8%
Integration of multivarabbl functions-
10.9%
Hypothesis
15.5%
Application tions of integration
9.5%
Confolence intervais
14.5%
Hypotheais tests
8.6%
Lavel
Integration of mulivariabll functions
12. 3%
Lavil
Vector caleslus.
8.5%
Level
Vector calculus.
13.3%
Level
First order differential equations
0.2%%
Level 3
Applications of intogration
10.0%
Level
Interest
6. 6.1%
Level
Interest
9.8%
Level
Confidence intervals
4.0%
Pirst order differentaal equations
9.6%
Annultics
0.0%
Level 6
Annulties
0.0%
Level
0.6
0.8
0.0
0.4
0.G
0.8
deepseek math-7h-rl
Qwen2: 7B Instruct
Integers
73.3%
Integers
70.8%
Limits and continuity
41.2%
Limits and continuity
66.7%
Differentaziion
41 41.2%
Differentiztion
63.1%
Linear equations and functions
33.9%%
Linear equations functiion
42.6% 69%
Systems of linear equations-
31.0%
Systems of linear equations-
38.2%
Sample Space
21. .0%%
Applications differentia tion
32.1%
Applications of differentiatio
1...1% 1%
Sample Space
28.
Vector goometry
16.44%
Vector gonmetry
25.3%
Calculus vactor valued functions
12.0%
Applications of intognation
20.3%
Infiaite sequences and series 12.33
Inflaite sequences and setfes
18.1%
Continuous distributions
10.2%
Differentatton of multivarablee functions
17.9%
Differestiation of muitivarabble functions
8.9%%
Calculus of vector valued functions
17.5%
Applications of integraticn.
8.8%
Integration of multivartable functions
16.7%
Integration of multivartable functions-
0.7%
Continuous distribttonn
14.31
First order differentinl equations
6. 1.2%
Level
Confolence intervals
13.3%
Level:
Vector calculus-
5.
Vector calculus-
11.3%
Hypothesis tests-
4.3%
Level 3
Hypothesis tasts.
9.5%
Level
Intersst
3.7%
Level
First order differential equations
8.9%
Level
Confidence intervais
3.6%
Lavel
Interest
7.4%
Lovel
Annultivs
0.0%
Level 6
Anmultios,
0.0%
Level 6
0.0
0.2
0.4
0.6
0.0
0.2
044
0.6
0.8
EAce
Figure 11: Model accuracy across topics
41

---

Published as a conference paper at ICLR 2025
NuminaMath 7B-CoT
Yi-1 5.5 BBat
Integers
69.2%
Integers
69.2%
and continulty
nitatentiation
Linear equations and functions-
37.4%
Linear equations and functions
36.5%
Diffirentitiion
33.3%
Limits and continuity
35.3%
Systems of liner equations
27.1%
Systems of llnear equations
25.7%
Applications of diféorentiatinn
1.855%
Applications of differentiation
24.5%
Sample Space
Vector geometry
19.59%
Vector geometry-
16.4%
Applccations of lntegration
18.2%
Applications of integration
16.2%
Integration multivariable functions-
18.1%
Integration of mul tivariable functions.
13
Diffevvstaaiion of multivariable functions Coufidencs intervals
17.3%
Differestiation of muitivarfable functions
12.8%
16.9%
Infinito sequines series-
12.3%
Sample Space
16.8%
Calculus of vector valued functions.
10.31
Vactor caleuluu
16.0%
Continuous distributions
9.2%
Calculus vector valued functions
15.5%
Vector calculus-
8.5%
Level 1
lnfinite sequences and series-
12.3%
Level
Interest
6.7%
Level
Continucus distributions
12.2%
Level
First order differential equations-
4.1%
Lecel
pyyotteesi tests-
9.5%
Level
Hypotheeis
3.4%
Tovel 4
First order differential equations 0.21%
Lavel
Confodencs intervais
Level
Intervat
Annuities-
0.0%
Lovel 6
Annutties
0.0%
Level
0.8
Mathstrul- 7B-v0.1
Yi-1.5-34B-- 3-bttbb
Integers
66.7%
Integers
66.7%
Linear equations and functions-
Linear equations and functions Differeniaaiin
45.2%
Limits and continuity
29.4%
42.5%
Differentiarion
28.1%
Limits coatinuity
42.4%
Vactor goometry
22.66%
mppl Sqace
31.5%
Sample Space
22.4%
Systems linea equations
30.6%
Applications of differentistion
18.1%
Appliatioon of dittereniiatinn
24.2%
Systema of linear equations
1.600%
Vector goometry
23.3%
Calculus vector
functions
Confiden.c intervals
22.
Infinate sequinces and sertes-
9.1%
Appiccations of integration
17.015
Inlegration of multivariblle functions-
9.4%
Continueus distributions
17.3%
Diffcreattation of multivartable functions-
8.0%
Infintte sequences and sertes-
15.9%
Continueus distributions
8.2%
Integration multivamable functions
15.9%
Hypothosis
Difeevetiation mul tivariabibb functions
15.6%.
Vector caleslus
6.6%
Locel 1
Hypolbesis tests
13.8%
Lewi:
Intersst
6.1%
Level
Calcalus valued functions.
13.4%
Level
Applications tions of integration
6.1%
Level 3
Vactor caleulus
12.3%
Level
First order differential equations
4.1%
Level 4
First order differential equations
9.6%
Level
Confidence intervals
2.4%
Level
Interest
Annulties
0.0%
Level 6
Annulties
0.0%
Level
0.0
0.2
0.4
0.6
Mistral-Nemo- Nemo- Instrurt- 2407
derpsonk 7h Ph-instruct
Integers
63.3%
Integers
57.5%
Linear equations and functions-
33.9%
Linear equations and functions-
21.3%
Limits and continuity
29.01%
Diffeentiation
24.2%
Difécrentiation
22.9%
Limits cootinulty Sample Space
23.5%
Sample Space-
20.39
16.8%
Vacter geometry
15.8%
Applications of diffecentaation
15.8%
Systema of linear equations
15.3%
Systema of linear equations
15.3%
Applications of differentiatin
33.69%
Vector geometry
11.
13.65%
Hypothesis
9.5%
Difervatiation tivariaalle functions
7.3%
Calculus of vector valued functions
9.35
lnflaite se sequences and series
7.2%%
Continuous distributions
Applcattonn integration
6.8%
Infinite sequences and series 8.0%
Calcolus of vector valued functions
5.2%
Differectiation of muitivartable functions-
6.1%
Eypotteesis
4.3%
Confidence intervals
4.8%
Vector calculus
First order differentinl equations
4.8%
Level 1
Interwst 3.7%
Level:
Integration of multivartable functions-
4.3%
Level
Integration multivariable functions
Applications of integration
4.1%
Level 3
Confidence intervals
3.6%
Level
Vector caleslus
3.8%
Level 4
Continuuus distributions
3.1%
Level
Interest
3.7%
Locel
First order differentia ecuations
2.1%
Annultive
0.0%
Level G
Anmiltio
0.0%
Level
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
EAcc
EAcc
Figure 12: Model accuracy across topics
42

---

Published as a conference paper at ICLR 2025
Vi-1.5 6B Chat
Meta Liama 3 Instruct
Integers
54.2%
Integers
41.7%
and continulty
Linear equations
Differentiztio
30.1%
Difterentattin
17.095
equatioos functions
25.21
Sample Space
14.0%
Systems of liner equations
22.9%
Limits and cootinulty
11 ..%0
Applications of diféorentiation
17.4%
Vector geometry
8.2%
Vector geometry
16.4%%
Applications of differenttaion
Sample Space-
14.0%
Systems of linear equations
6.9%
Applications of integrationn
12.2%%
Diffecectiaiion multivariable functions-
Differvetiation of multivarabll functions.
11
Hypotheois tasts
3.3
Calculus valued functions
10.39
First order differential equations
4.1%
Confolence intervais
9.6%
Continueus distribution
Integration of muitivarfable functions
9.1%
Coefidence intervals
2.4%
Infinitee sequences series
sequences series 2.2%
Continueus distributions-
0.225
Level 1
Vector calculus- 1.9%
Level
Vector calculus
7.5%
Level
Interest 1.8%
Level
Interest-
Lecel
Integration multivariable functions- 1.%%
Level
Hypothesis
3.4%
Lovel 4
Applications of integration 1.4%
Level
First order differential equations
Calcalus vector valued functions
Annuities-
0.0%
Local 6
Annuities- 0.0%
Level
0.8
DeepSeek Lite Chat
Mistral-778 Baaal///---Intttttttt.
Integers
40.8%
Integers
24.2%
Linear equations and functions-
Linear equations and functions
13:
Differentiation
11.8%
Differntiaaion
10.55 5%.5%
continuity
Limits coatinuity
Applications of diffamntiation
5.3%
mample Space
6.3%
Sample Space
4.0%
linear equations 3.5%
Systems of linear equations
Vecter geometry 2.7%
Vector geometry
3.4%
Applications of differentaation 2.6%
Applications of integration
Diferenttation multivariable functions
Interest 2.5%
Calculus of vector valued functions 2.11%
Confidence intervals 2.4%
Appleations integration 2.0%
Differstttatinn of muittvartable functions 2.2%
Interest 1.8%
Hlyotthees 1.7%
Infinite sequinces and series 1.4%
Calculus vactor valued functions
order differential eguations
Continuous distributions 1.0%
Lovel 1
Continucus distributions 1.0%
Lewil
caleslus
Level
Hypathesis
0.9%
Infinite suquennes and saries 0. 7%
Level 3
Confidence intervals 0.0%
Level
Integration of muitivariable functions
0.7%
Level 4
Integration muitivariable functions 0.0%
Level
Annutties
0.0%
Level
Vector calculus
First order differential equations
0.0%
Level 6
Annulties 0.0%
Level
0.4
dorpseek mno- 1 16b-chat.
Integers
17.5%
Linear equations and functions
Limits and continuit
3.5%
Difécrentizzioe
Sample Space
1.4%
Vector geometry
1.4%
Applications of differentictioe
1.1%
Continuous distributions
1.0%
Infinite sequeeoes saries
Systems of liner equations 0.7%
Calculus Confidence intervals 0.0%%
functions 0.0%
Integration of multivartable functions 0.0%
Annuities 0.0%
Hypotheeis teats 0.0%
Level 1
calculus
0.0%
First order differential equations 0.0%
Level 3
Application of integration
0.0%
Level 4
Interest 0.0%
Lovel
Differestiaiin of mui tivriablee functions.
0.0%
Level
0.0
0.2
0.4
0.6
Figure 13: Model accuracy across topics
43

---

Published as a conference paper at ICLR 2025
Table 32: Results of PHP on UGMathBench (all figures are in %). "diff." refers to the improvement over results Table The best results within column with "diff." are bolded and the worst results are underlined.
Models
AAce
diff. AAcc
EAce
diff. EAce
diff. A
Closed-source LLMs
GPT--4-22224088-6
60.73
+0.36
30.44
+0.50
10.29
0.14
Open-source
Chat LLMs
Yi-1.3-6B-Chai
26.11
-0.14
15.57
+0.24
10.54
-0.88
Mistral-7B-Insteen
10.76
+0.20
4.41
-0.03
6.35
+0.23
Qwen2-7B-Instruee
36.38
25.29
0.14
11.09
-0.14
LLaMA3-8B-Instie
16.41
-0.14
8.61
-0.3
7.8
+0.16
Yi-1.5-9B-Chat
34.50
+0.21
21.75
+0.63
12.75
-0.42
Mistral-Nemo Instruct-2407
Nostral-ommo-nnstuut-2222
25,46
+0.38
15.19
-0.24
10.27
+0.62
DeepSeek-MOE Chat
6.34
+0.54
2.07
+0.11
4.27
+0.44
DeepSeek-V2-Litee
13.98
6.44
+0.75
+0.25
Saarral-mmall--nstnee--44000
39.66
-0.56
27.90
-0.94
11.76
+0.38
Yi-1,5-34B--C -35--atB-
37.57
+0.04
23.83
-0.51
13.74
+0.47
LLaMA3-70B-Instuu
33,93
+0.31
23.24
-0.03
10.69
+0.34
Qwen2-
47.89
+0.13
36.08
+0.30
11.81
-0.17
Mistral-I anaeell-arge 2407
55.97
+0.04
45.17
+0.13
10.8
+0.09
Specialized Mathematical LLMs
24.57
+0.68
14.90
+T.29
9.67
-0.61
SrepSeck-Matth-15
29.42
-0.23
19.58
+0.24
9.84
-0.47
NuminaMath-7B-Ca.
27.81
-1.99
16.62
-2.19
11.19
+0.20
Muthstral-7B-vl
29.75
+1.24
19.26
+1.32
10.49
-0.08
Owen2-Math-77-Insse
42.65
-1.08
31.18
-1.08
11.47
0.0
Qwen2- Matt-MMtthh-aaaa Math-77BBInntrueet
57.35
0.32
46.04
+0.19
+0.13
Table 33: Results across different subjects of PHP for GPT-4o. (all figures are in %). "diff." refers the improvement over results in Table The best results within column with "diff." are bolded and the worst results are underlined.
Subject
AAce
diff. AAec
EAcc
diff. EAce
A
diff. A
Arithmetic
92.11
+0,29
88.47
+1.20
3.64
-0.91
Algebra
72.33
+0.97
65.35
+0.68
6,98
+.29
Set theory and logic
71.98
-1.45
57.97
-1.45
14.01
0.0
Trigonometry
67.79
-1.69
53.37
+1.68
14.42
-3.37
Combinatorics
78.79
+1.14
65.91
+4.55
12.88
-3.41
Geometry
67.49
+3.31
54.04
+2.49
13.45
+1.82
Calculus single-variabl
65.11
+0.24
53.26
+0.21
11.85
+0.03
Calculus multivariable
49.85
+0.21
38.53
+0.30
11.32
-0.09
Linear Algehra
62.85
+0.47
52.81
+1.20
10.04
40.73
Number Theory
+1.45
54.35
+2.18
10.87
-0.73
Financial Mathematics
29.87
+0.49
20.81
+2.60
9,06
-2.11
Probability
65.38
-0.59
52.68
-2.08
12.7
+1.49
Statistics
60.27
-0.16
48.88
-0.50
11.39
+0.34
Complex Analysis
70.59
+1.31
58.82
11.77
+1.31
Differential Equations
25.03
0.33
18.03
-1.31
7.00
+1.64
Abstract Algebra
53.17
+4.76
35.71
+7.14
7.46
-2.88
44


Reviews:
[{"review_id": "nJg8T6zt4T", "reviewer": "Reviewer_Ywbr", "summary": "The paper introduces UGMathBench, an undergraduate-level math reasoning benchmark for LLMs. THe benchmark includes 5,062 problems across 16 subjects, with randomized variations to prevent test-set contamination and evaluate reasoning consistency. All problems are taken from the authors’ school curricula. The authors also propose a variety of metrics that could theoretically be applied to other benchmarks (with additional work) to enable more robust benchmarking of LLM abilities. Testing 23 LLMs, the highest EAcc achieved was 56.3%, highlighting challenges for current models.", "questions": "* Same as weaknesses — I think there are possibly more interesting analyses that could be conducted given how rich the data in UGMathBench is (as opposed to the usual open vs. small, regular vs. specialized, etc.)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The authors create a novel undergraduate-level benchmark for math reasoning.\n* The proposal of the Effective Accuracy, Average Accuracy, Reasoning Gap, and Robustness Efficiency metrics are interesting, and are hopefully adopted by math benchmarks as a whole to promote more robust benchmarking.\n* The results on each of these metrics are described at length, with potential reasons explored for any interesting/unexpected results.\n* UGBench is fine grained, with each question being labeled with a main topic and subtopic, enabling a more granular study of various LLMs’ strengths and weaknesses.\n* The appendix is robust and includes detailed information about the creation and distribution of the benchmark.", "weaknesses": "* The overall discussion of results feels slightly generic (besides what is driven by the metrics proposed in the paper). It demonstrates a lot of what has already been established (namely, undergraduate math being a challenging task, closed source models performing better than open source, bigger model size = better performance, specialized LLMs’ improved performance over unspecialized ones, etc.) UGBench is fairly uniquely positioned in how granular it is (in terms of each question being associated with a topic/subtopic). I believe this could lead to interesting comparisons between models and which skill sets certain families of models succeed/fail on.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T55AY0bFSd", "reviewer": "Reviewer_ZpsQ", "summary": "This work underscores the limitations of current mathematical benchmarks due to the rapid advancements in mathematical reasoning capabilities of LLMs and issues of test-set contamination. To address these challenges, the authors introduce UGMathBench, a comprehensive benchmark designed to effectively evaluate these capabilities. UGMathBench consists of over 5,000 problems across various subjects, aiming to provide a fair assessment of LLM performance. It addresses common issues in existing benchmarks by offering a dynamic dataset that can generate different question variations by setting random seeds. Additionally, the authors propose new metrics to evaluate accuracy and reasoning robustness, facilitating a deeper analysis of the challenges LLMs face in mathematical reasoning.", "questions": "1. How do you ensure that changes to specific values within a problem do not fundamentally alter its nature or required solution approach? Given that small modifications can impact mathematical properties such as convergence and factorization, are there any safeguards or verification steps to confirm that randomized problem variants retain similar problem structures, difficulty levels, and solution approaches as the original?\n2. Would it be feasible to incorporate a tolerance level for minor discrepancies in numerical answers, especially those influenced by floating-point precision errors? For instance, a defined rounding tolerance (e.g., within 0.01%) could help accommodate acceptable computational variations and more accurately reflect the model’s reasoning accuracy.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. **Dynamic Dataset:** UGMathBench provides multiple randomized versions of each problem, allowing for a robust assessment of LLM reasoning abilities. This approach enables a more rigorous evaluation by examining how well models handle variations of the same problem.\n   \n2. **Diversity:** The benchmark covers a wide range of mathematical topics, ensuring comprehensive subject matter coverage.\n\n3. **Difficulty Level:** UGMathBench includes undergraduate-level problems, challenging LLMs with more advanced content.\n\n4. **Innovative Evaluation Metrics:** The paper introduces metrics like Effective Accuracy (EAcc) and Reasoning Gap (∆) to measure LLM performance. EAcc captures the percentage of problems correctly solved across all problem variations, while the Reasoning Gap quantifies the robustness of reasoning when problems are modified slightly.\n\n5. **High-Quality Problem Collection:** The problems are carefully curated, cleaned, and formatted from an online undergraduate-level homework grading system, ensuring quality and relevance.\n\n6. **Fully Public Benchmark:** UGMathBench is publicly available, enabling LLM developers and users to conduct thorough error analysis.\n\n7. **Detailed Error Analysis:** The benchmark provides insights into the current limitations of LLMs in mathematical reasoning, guiding future directions for improvement.", "weaknesses": "1. **Premise of Problem Variants:** The paper assumes that \"a model capable of solving a problem through reasoning should also be able to solve all its variants under variable disturbance.\" The benchmark achieves this by randomizing numbers in each question to create multiple versions while theoretically maintaining the problem’s difficulty and intrinsic properties. However, simply changing certain values can fundamentally alter the problem. For example, modifying the denominator in $\\int_{a}^{b} \\frac{x^3}{x^4 - 1} \\, dx$ to $\\int_{a}^{b} \\frac{x^3}{x^4 + 1} \\, dx$ requires a completely different solution approach. Similarly, changing the series $\\sum_{n=1}^{\\infty} \\frac{1}{n^2}$, which converges to $\\frac{\\pi^2}{6}$, to $\\sum_{n=1}^{\\infty} \\frac{1}{n}$ transforms it into the divergent harmonic series. These examples show that certain modifications in mathematical problems lead to essientially different problems, which may not align with the benchmark’s intended assessment of consistent reasoning across equivalent variants.\n\n2. **Precision in Numerical Calculations:** Classifying the example in **Table 12** as a \"numerical calculation error\" might be overly strict, given that the minor difference between 12305.72 and 12305.670999 likely falls within acceptable rounding tolerance. Unless the benchmark is meant to test extreme precision, such small discrepancies shouldn’t necessarily indicate an error. Accuracy assessments could allow for minor rounding differences or specify a standard precision level (e.g., significant figures or decimal places) for intermediate calculations to ensure fair evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OoZ0r3fUA7", "reviewer": "Reviewer_2zWK", "summary": "Authors propose a new benchmark namely UGMathBench comprising of undergraduate level math problems for testing mathematical reasoning abilities of LLMs. Authors also propose two metrics namely effective accuracy and rasoning gap and perform extensive experiments with multiple open and closed source LLMs showing that there is still a big scope for improving mathematical reasoning abilities of LLMs.", "questions": "what are the limitations of the current work? Pls include a description of limitiations of current work too as it is important", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) Dataset is comprehensive consisting of 5062 problems including problems from different areas of mathematics.\n\n2) I acknowledge the efforts made by the authors in cleaning the data by performing deduplication ensuring that no problems are repeated and including multiple versions of the problems for thorough evaluation of mathematical reasoning abilities of LLMs.\n\n3) Authors have promised to make the benchmark dynamic as in to include new and more tougher problems in the future\n\n4) The proposed metrics namely effective accuracy and reasoning gap provide an effective way to evaluate the reasoning abilities of LLMs.", "weaknesses": "1) Authors should experiment with more advanced prompting techniques like progressive hint prompting [1]. Progressive hint prompting allows LLMs to use previously generated answer as a hint which allows LLMs to correct/ refine their mistakes. I believe that this experiment would provide crucial insights if LLMs can solve tough mathematical problems with refinement. I will increase my score if authors perform this experiment.\n\n2) Authors mention in line 477 that there are some number of \"bad questions\" present in the benchmark which may not be suitable for LLMs to solve. Pls mention the number and percentage of bad questions present in the benchmark as it directly effects the quality of the benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sO7vhp4Raj", "reviewer": "Reviewer_LM6m", "summary": "This paper introduces UGMathBench, a new benchmark for evaluating mathematical reasoning capabilities in LLMs at the undergraduate level. The benchmark addresses key limitations of existing evaluation methods by incorporating 5,062 diverse problems spanning 16 subjects, with each problem featuring three randomized versions to help detect potential test set contamination. The authors contribute two new metrics: effective accuracy (EAcc), which measures consistent problem-solving ability across variations, and reasoning gap ($\\Delta$), which quantifies reasoning robustness by comparing average accuracy to EAcc. Detailed error analysis showing calculation errors as the primary failure mode and revealing inconsistencies in how models handle different versions of the same problem.", "questions": "- What was the threshold used for deduplication and how many questions were filtered out?\n- How well do students perform on the benchmark? (Assumming this data is availble/not private)\n- Is it possible to derive the expected reasoning gap as a function of sampling temperature?\n- How do models specifically overfit to one variation effect the reasoning gap?\n- Were the \"bad questions\" found in the error analysis filtered out of the benchmark?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The functional variation approach represents a significant advancement over traditional math benchmarks. By systematically generating variations, it provides a robust mechanism to detect and account for potential test set contamination and overfitting - a critical concern given the widespread training of models on public mathematics content.\n\nAdditionally, the benchmark's difficulty is carefully calibrated to the current capabilities of LLMs while maintaining sufficient headroom for future improvements. This thoughtful positioning ensures the benchmark will remain relevant and discriminative as model capabilities advance, providing a stable metric for tracking progress in mathematical reasoning.\n\nThe authors offer valuable insights through: granular performance breakdowns across difficulty levels, subject-specific analysis that illuminates relative strengths and weaknesses, detailed examination of error modes that reveals systematic failure patterns (about half of the errors were due to calculation errors). This analysis not only establishes current baseline performance but also provides insights for model improvement and understanding of LLM mathematical capabilities.", "weaknesses": "The observed consistency in the reasoning gap metric (converging to approximately 10 across increasing ACC values) raises important methodological concerns. This finding contradicts established results from previous papers on functional variations and test set contamination, which demonstrated significant performance disparities among models with comparable ACC levels (a pattern typically indicative of overfitting). Two issues warrant discussion:\n\n1. The apparent \"reasoning gap\" may primarily reflect sampling temperature effects rather than true reasoning capabilities. By definition, expected accuracy (EAcc) will be lower than actual accuracy (AAcc) due to the stochastic nature of the sampling process. This suggests the metric might be predominantly measuring a model's robustness to temperature variation rather than its reasoning capabilities.\n2. Models previously demonstrated to overfit on the test set would be expected to exhibit substantially larger reasoning gaps. The absence of this pattern in the current results suggests that either: a) the metric is not effectively capturing reasoning differences, or b) the relationship between overfitting and reasoning capabilities requires re-examination. The authors claim that \"These metrics help mitigate the impact of potential test set contamination,\" but their is no provided evidence that the proposed metrics detect contamination.\n\nThese observations suggest the need for additional validation of the metric's ability to meaningfully differentiate between genuine reasoning gaps and artifacts of the sampling process.\n\nMinor:\nFigure 1's layout appears overcrowded with difficult-to-read text sizes and fonts. Additionally, I would strongly reccomend against using pie charts as humans are bad at comparing areas.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)