Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist
Authors: Zihao Zhou
Shudong Liu
Maizhen Ning
Wei Liu
Jindong Wang
Derek F. Wong
Xiaowei Huang
Qiufeng Wang
Kaizhu Huang
Venue: ICLR 2025 Poster (2025)
Keywords: Mathmatical Reasoning
Geometry reasoning
Evaluation
(Multi-Modal) LLMs
Primary area: foundation or frontier models, including LLMs

Abstract: Exceptional mathematical reasoning ability is one of the key features that demonstrate the power of large language models (LLMs). How to comprehensively define and evaluate the mathematical abilities of LLMs, and even reflect the user experience in real-world scenarios, has emerged as a critical issue. Current benchmarks predominantly concentrate on problem-solving capabilities, presenting a substantial risk of model overfitting and fails to accurately measure the genuine mathematical reasoning abilities. In this paper, we argue that if a model really understands a problem, it should be robustly and readily applied across a diverse array of tasks. To this end, we introduce MathCheck, a well-designed checklist for testing task generalization and reasoning robustness, as well as an automatic tool to generate checklists efficiently. MathCheck includes multiple mathematical reasoning tasks and robustness tests to facilitate a comprehensive evaluation of both mathematical reasoning ability and behavior testing. Utilizing MathCheck, we develop MathCheck-GSM and MathCheck-GEO to assess mathematical textual reasoning and multi-modal reasoning capabilities, respectively, serving as upgraded versions of benchmarks including GSM8k, GeoQA, UniGeo, and Geometry3K. We adopt MathCheck-GSM and MathCheck-GEO to evaluate over 26 LLMs and 17 multi-modal LLMs, assessing their comprehensive mathematical reasoning abilities. Our results demonstrate that while frontier LLMs like GPT-4o continue to excel in various abilities on the checklist, many other model families exhibit a significant decline. Further experiments indicate that, compared to traditional math benchmarks, MathCheck better reflects true mathematical abilities and represents mathematical intelligence more linearly, thereby supporting our design. Using MathCheck, we can also efficiently conduct informative behavior analysis to deeply investigate models. Finally, we show that our proposed checklist paradigm can easily extend to other reasoning tasks for their comprehensive evaluation.

Published as a conference paper at ICLR 2025
Is YOUR MODEL REALLY A GOOD MATH REASONER? EVALUATING MATHEMATICAL REASONING WITH
CHECKLIST
Zihao Zhou Shudong Liu
Maizhen Ning Wei Liu Jindong Wang
Derek F. Wong" Xiaowei Huang
Qiufeng Wang Kaizhu Huang
Xi an Tiaotong-liverpon University
Universityy of Liverpool University of Macau
'HKUST 'William & Mary
Duke Kunshan University
https://mathcheck. wwt.su.itttuiii. .io/
ABSTRACT
Exceptional mathematical reasoning ability is one of the key features that demon- strate the power of large language models (LLMs). How to comprehensively define reflect the experi-
and evaluate the mathematical abilities of LLMs, and ence in real-world scenarios, has emerged as a critical issue. Current benchmarks predominantly concentrate on problem-solvine capabilities, presenting a substantial risk of model overfitting and fails to accurately measure the genuine mathematical reasoning abilities. In this paper, we argue that if model really understands a problem, it should be robustly and readily applied across diverse array of tasks. To this end, we introduce MATHCHECK, a well-designed checklist for testing task generalization and reasoning robustness, as well as an automatic tool to generate checklists efficiently. МАтHCHECK includes multiple mathemattcal reasoning tasks and robustness tests to facilitate a comprehensive evaluation of both mathe- matical reasoning ability and behavior testing. Utilizing MATHCHECK, we develop BIEEHCCHCKK-SSS and
to assess mathematical textual reasoning and multi-modal reasoning capabilities, respectively, serving as upgraded versions of benchmarks including GSM8k, GeoQA, UniGeo, and Geometry3K. We adopt MATHCHECK H-MMHEKKK-MMMM and MATHCHECK GEO to evaluate over LLMs and 17 multi-modal LLMs, assessing their comprehensive mathematical reasoning abilities. Our results demonstrate that while frontier LLMs like GPT-4o continue excel in various abilities on the checklist, many other model families exhibit a significant decline. Further experiments indicate that, compared to traditional math benchmarks, MATHCHECC better reflects true mathematical abilities and repre- sents mathematical intelligence more linearly, thereby supporting our design. Using MATHCHECK, we can also efficiently conduct informative behavior analysis to deeply investigate models. Finally, we show that our proposed checklist paradigm can easily extend to other reasoning tasks for their comprehensive evaluation.
INTRODUCTION
The Al community has been placing significant emphasis on mathematical reasoning as a means
to explore the upper limits of intelligence in large language models (LLMs) (Achiam al. 2023
Team et al. 2023; Meta 2024 Jiang et al.] 2024 Wei et al. 2022 Trinh et al., 2024 Romera- - Paredes
al. 2024 and multi-modal large language models (MLLMM)) (OpenAl) 2024c Lu al. 2023).
large number of efforts have been made on how to enhance (M)LLMS' mathematical reasoning
abilities. In pre-training, Wang al. (2023d); Shao et al. (2024); Lin
(0224): Zhang al.
(2024c) studied the impact of the quality of mathematical corpus; in post-training, Yue et al. (2023); (0224)) augmented a huge number of synthetic data, and then developed
Yu et al. (2023):
"Equal contribution. Email: zibao. @hou@_iverpool. ac. strieccssssoonnggggee.a.. com
1 Corresponning author. Email: ufnn. waw.gngwwtnttttue..e..

---

Published as a conference paper at ICLR 2025
Problem Solving
Answerable Judging
Outcome Judglng
Process Judging
Original Problem
rete takes zolts blue taber Nov eany tolts in cotal lees take?
A. robs takes 2 botts of much white Fiber, How
robe takes bolite of blue and that balts in total: coes it talle?
robe tases 2 belts of buu faber Bow rany take?
Hor
does
nent enteett Step 1: Identiity amount Stop
bolta in totak coes it
Step
Matesply the colle bote and white Fibee sopether to Fied total sueter bolts, The 3 "enswert Step 3
bite fiber The
"answer": 3yB
"answer"'s Unanoverable
i5 4 bolts in total.
Seed Date
Problem Undersfonding
To wake robe, you noed ngtf saly bolts of
make robe, need botts of atue fiber and halr compared the cotal, number of bolts required
Bake you need belts Khat the
make robe, you need 2 balts
the number of bolts required for the robe?
total number of bolts
whitee Fiber compared to nuaber of solts requliod fer the rate?? "answer"ss 3.0
"solutian"t Step cotculate
"sclution": : Step 22 Step 2: Detemnie the
Reasoning Robustness
2 Ialue)e Ivhitel 3.
as
blue
"answer": Unarowe nable
anover 35
bolts. The onswer 15 4. Step
tasttr craftiing Lucerious rose. The design reqsires half blue
lusurizus craftiin
A twilor is cratting requtrmss rebe- The design fiter
tadior is cratting
Irrelesont Disturbance
of white laber, To and considered tastor bolts golden thread from the rays, tvertuslly decided ld quady for the cervenny. How
requires
luxur ious rober The design
fiter and helt that awort How
whate riser,
requires botts of blue
and half t928 aoount of white bolts fiber- needed for dis repordinng the goldon throad!
of white filer, total
boits in total are needed for
disnegerding
needed for the rase,
go poloon thread?
Stop 21 Cal calculate the aecunt of requires, whicch couble the
"sotuttions Step
disrejording the goldon
"ag lutionn: Stes is coloulate
roaded the rebe, sisreganding the golden
are
*wrxwerf:: fosuerable
ceston onsver iot 388 yards.
amount, ss 2 bolts 2 w botts. step 3: ... unswerr
fiber
Step
Scamario Understanding
2. rabe takes bolts
mibe takes of half
Blue fibbr and half that each white fiberr
robe takes botrs
blue fiter and fever white Fiber- st takes the value of unlnown variablle x?
in
of uniooon variable
that What to the value
"colution" Step 1: Let's Step To not hav nany fibor should be
that What is the value of untrmon vartable
solts teeat,
that is the value of
"solucion"" Stop Let's say The
"anover"s Unanswerabte
the value of wnser in Correct
total, the esuation
The ansver "artwer": Step
couald
Task Generalization
Figure 1: Overview of MATHCHECK design. The horizontal axis examines the task generalization of four math tasks while the vertical axis examines the reasoning robustness through four problem varieties. All data are generated from seed data, which is also from mainstream benchmark dataset.
supervised fine-tuning (SFT) for math problem-solvingg Recently, Luong et al. (0024) and Sun et al. (2024b) explored variants of reinforcement learning (RL) for further improvements.
To guarantee the high mathematical reasoning ability has been reached, it is crucial to fairly evaluate models' performance. Current mainstream methods rely on the performance across math problem- solving tasks of varying difficulty levels, such as GSM8k (Cobbe et al. 2021 of elementary level, MATH (Hendrycks et al.) (02T) of high school level, and TheromQA (Chen et al. 2023a] of university level. Recently, some mathematical datasets that are more challenging, diverse, and multi-modal have been proposed to enhance the mathematical evaluation (He et al. 2024 Liu et al. 2024c] Lu et al., 2023 Zhang et al. 2024b). However, these current evaluation methods focus on individual tasks (most of which are problem-solving) and robustness tests for each problem. In other words, they do not provide comprehensive guidance whether L.LMs really achieve mathematical reasoning ability. In this paper, we argue that: if a model really understands a problem, it should work robustly across varions tasks about this problem. Therefore, is necessary to evaluate models by multi- tasks with diverse robustness test. Through such investigation, the real reasoning ability of a model can be comprehensively evaluated. As result, we can also perform detailed behavior tests on models (Ribeiro et al. 2020).
Drawing motivations from this insight, we introduce MATHCHECK, a well-designed checklist for testing task generalization and reasoning robustness. MATиCHECK includes general mathematical reasoning tasks and diverse robustness testing types to facilitate a comprehensive evaluation of mathematical reasoning ability and reasoning behavior testing. As shown in Figure horizontally, we examine the task generalization including problem solving, answerable judging, outcome judging. and process judging. Vertically, we test the reasoning robustness through the original problem and its three robustness variants consisting of problem understanding. irrelevant disturbance, and scenario understanding. The data of each cell in the checklist corresponds to a specific type of robustness test and task form. To facilitate the construction of checklist, we propose an (M)LLMs-drive generation framework to automatically generate this data. Figure 2 illlstrates the MATHCHECK data collection
2

---

Published as a conference paper at ICLR 2025
process, where the seed solving problem is firstly rewritten to its robustness problems, next all generated solving data are utilized to construct other task forms.
Utilizing MATHCHECK, propose MATпCHECK- GSG, MATHCHECK dataset generated from GSM8k (Cobbe et al, 202T). It contains a total of 3,096 high-quality samples consisting of 129 groups checklist matrix, which be used evaluate mathematical textual reasoning ability com- prehensively. Besides, acknow ledging the community's 's focus on multi-modal reasoning capabilities, we further propose MATHCHECK-OO GEO evaluate the multi-modal geometry reasoning ability, Generated from GeoQA (Chen et al. 202T), UniGeo (Chen et al. 2022), and Geometry3K (Lu et al., 2021), it contains total of 1,440 samples with a checklist matrix of 60 groups. noteworthy that the construction pipeline of MATHCHECK can be applied to most mathematical datasets to dynamically establish a comprehensive and flexible evaluation benchmark, thereby mitigating data contamination (Zhou et al. 2023a Zhu et al. 2024a
We conduct extensive experiments on 26 LLMs and MILMs including different scales, API- base and open source, generalist and mathematical models. We find that frontier LLMs like GPT-4o continue to achieve superior performance in our MATUCHECK, but many other model families exhibit significant decline. Further experiments indicate that compared to solving original problems which is the paradigm of mainstream benchmark, our MATHCHECK evaluation aligns more accurately with the genuine mathematical reasoning ability of the model. Utilizing MATHCHECK, extensively analyze the models' behaviors including training on massive solving data, reasoning consistency, performance on different complexity problems and applying different prompting technologies. Finally, we show the potential of applying MATUCИECK paradigm to other reasoning tasks such as commonsense reasoning and code generation, promoting more comprehensive evaluation of reasoning ability.
MATHCHECK
МАТICСEЕK is well-designed checklist that includes general mathematical reasoning tasks and diverse robustness testing types for comprehensive evaluation, as well as a tool to automatically generate large number of test cases in the manner of checklist. In our checklist, various mathematical tasks are arranged in rows to assess task generalization, whereas diverse variants of mathematical problems are placed in columns to evaluate reasoning robustness. We will elaborate on the task types in Section 2.1 problem variants in Section 2.2, and how we construct checklist data in Section 2.3
2.1 TASK GENERALIZATION
Testing models across different tasks on the same domain not only offers a comprehensive and profound evaluation of their capabilities (Frank 2023) but also caters the practical demands and complexities of real-world applications (Ji et al. 2023). In MATHCHECK, we incorporate four math
tasks including Problem Solving, Answerable Judging, Outcome Judging, and Process Judging.
Problem Solving. In this task, we ask the model to solve a given math problem. As the most widely used method test mathematical reasoning ability in contemporary research (Cobbe al. 202T Hendrycks et al. 202I), it necessitates the model to analyze the problem, recall and apply appropriate math knowledge. and finally conclude reasoning results.
Answerable Judging. Given a math problem, models need to determine whether the problem provides sufficient information to answer the question. This task requires the model analyze the question, then identify the essential conditions required for solving this question, subsequently verify whether these conditions are provided within the problem statement. Previous works utilized it to examine whether the model is a reasoner with critical thinking instead of a random parrot (Li et al. 2024b Sun et al.) 2024a
Ma et al., 2024).
Outcome Judging. Given math problem and one of its solutions. let the model determine whether the final answer of the given solution correct. Outcome-Judging is a noarse-grained judgment of solutions since the model only focuses the correctness of the final answer. Researchers often apply the outcome-ja outcome-judging ability of models to verify the correctness augmented data (Tang et al. 2024) and provide outcome rewards in reinforcement learning (Luong al. 2024).
Process Judging. Given a math problem along with its wrong solution, the model is required to identify the step where the errors begin. Compared with the outgone-judging, the process-judging
3

---

Published as a conference paper at ICLR 2025
Robustness Problem
Task Data Construction
Construction
Problem Set
MathCheck
fransform
Organization
Rewrihe
Seed Data
Figure 2: MATПICПЕCK generation pipeline.
task is a more fine-grained judgment the solution, which demands the model judge step by step until the wrong step is located. It can help to debug the given wrong solution.
2.2 REASONING ROBUSTNESS
model that truly understands the inherent mathematical logic of a problem will exhibit reasoning robustness diverse variations of this problem (Stolfo al. 2023). Motivated by this, utilize four problem forms including the original problem and its three rewritten variants to examine the reasoning robustness of models.
Original Problem. It is the seed problem of other reasoning robustness variants. At a minimum functionality test, it can check whether the model has the basic mathematical capabilities when modifications have been made.
Problem Understanding. It refers to transforming the original problem into new one that uses different wording or different sentence structures but does not change the mathematical logic of its original version (Patel et al. 2021; Zhou al., 2024, Li et al. 2024b). It pays more attention to semantic robustness, and aims to examine whether models can correctly reason when dealing with different descriptions of the same mathematical logic.
Irrelevant Disturbance. It refers to inserting irrelevant conditions that are related to the topic of the original question, but have no impact on the final answer. Previous studies have disclosed that large language models are easily distracted by such perturbations (Shi et al. (023). It needs the model to distinguish which conditions are necessary and which are irrelevant to the problem.
Scenario Understanding. When models comprehend the scenario of math problem and its underlying logic. they should be able to solve other questions within that scenario (Liu et al. 2021 Yu et al.) 2023 Zhou et al.) (2023b). Therefore, we alter the original question to evaluate whether model has a comprehensive understanding of the scenario. For example, as shown in Figure] [ we ask the question "the number of blue bolts" instead of "the number of total bolts".
2.3 CHECKLIST CONSTRUCTION
Creating MATHCHECK data is labor-intensive and time-consuming process. The advent of LLMss has introduced new level of flexibility and quality to generate mathematical content (Norberg et al.) 2023 Li al. 2024b). Therefore, we employ (M)LLMs (e.g., GPT-4-Turbo in our experiments) as engines to automatically generate our МATICCECCK data. The data construction pipeline is shown in Figure Users first assemble collection of math problems with labels seed data. Second, (M)LLMs mitially rewrite these problems into their robustness varieties to make up the robustness problem set. Third, each problem this set will be extended to construct multiple mathematical tasks about this problem. Finally, all data are manually checked to form MATHCHECK dataset correctly.
Based on the seed data, we automatically generate another three robustness problems as shown in the first column of Figure [] Problem Understanding and Irrelevant Disturbance are the tasks of rewriting problems without altering the final answer. Hence, we prompt the model to rewrite our math problems while maintaining the original answer. For Scenario Understanding, we first extract variable from the problem as a new answer, then prompt the model to change the question based on the extracted variable. Once we obtain the four robustness reasoning problems of the solving task, we rewrite them respectively to construct multiple tasks, including Answerable Judging, Outcome Judging and Process Judging as shown in the corresponding row of Figure 1 For the Answerable
4

---

Published as a conference paper at ICLR 2025
Judging task, we prompt the model to eliminate condition from the original problem which is crucial for solving it to obtain an unanswerable problem. For Outcome Judging task, we ask the model to solve the problem and acquire candidate solutions, then these solutions are labeled (Correct or Incorrect) according to the final answer. For Proeess Judging task, we apply the solution rewritten data. Specifically, given problem along with its
ability of (M)LLMS construct proeess-judgit correct solution, we prompt the model to make mistakes from the given steps and results in a wrong answer. In such way, we can get wrong solution while its mistake steps remain simultaneously. All of our prompts are listed in Appendix F.2
3 EXPERIMENTS
3.1 DATASETS
We use МАTIССEЕK comprehensively measure the mathematical reasoning ability across textual and multi modal settings. Consequently, two benchmarks MATHCHECK-GSM and MATHCHECK- GEO are introduced.
MATHCHECK GSM is a MATHCHECK dataset generated from GSM8k (Cobbe et al. (021). We choose GSM8k as the seed benchmark since (1) it is most widely used for evaluating mathematical textual reasoning capability. (2) we aim to determine whether advanced models are genuinely capable of reasoning at the grade school level. We first collect a test-mini set of GSM8K, which includes 129 problems sampled evenly according to the difficulty] Subsequently, we generate 129 МAТПСЦΙCK style groups, totaling 3,096 high-quality samples by MATHCHECK, can be used to evaluate the real mathematical reasoning ability of LLMs GSM8k-level problems. A group of MATHCHECK GSM case problems are listed in Appendix G.I
MATHCHECK-GEE GEO is dataset for geometry problems, which is the representative task for evaluat- ing multi-modis reasoning capability. First, we collect seed geometry problems from GeoQA (Chen ett aal..222 202T). UniGeo (Chen et al. (0022), and Geometry3K (Lu et al. (202T), containing 60 problems in both English and Chinese. Subsequently, we generate 60 MATUCCECK style groups, totaling 1,440 high-quality samples. Notably, this is the first geometry problem dataset involving answerable, outcome, and process judgment tasks. MATHCHECK GEGO gives research community a harder and multi-modal MATHCHECK style dataset, as well as showing the extensibility of MATHCHECK. A group of MATHCHECK -GOO case problems are shown in Appendix G.2
All datasets are checked with meticulous manual validation to ensure high quality and reliability. To this end, we recruited three graduate students who underwent training tailored to the requirements of our research. This rigorous verification process not only enhances the quality of our data but also reinforces the validity of our findings. Finally, our automatic data generation pipeline can achieve an average pass rate of 84.61 % (Appendix our checklist reported in Appendix
The detailed data statisties and quality discussion of
3.2 EXPERIMENTAL SETUP
To systematically benchmark the mathematical reasoning capabilities of existing LLMs, we include a comprehensive evaluation of 43 models, comprising 26 LLMs and 17 MLLMs. These models principally divided into two categories: generalist models encompassing both API-based commercial LLMs and open-sourced LLMs (large and small scale), and specialized mathematical models. We use the Fl metric for Outcome Judging and Answerable Judging tasks, and the Acc metric for the other two tasks. The list of selected models and details of evaluation setup can be found in Appendix [
3.3 MAIN RESULTS
Tables I and 2 illustrate the performance of various models the MATHCHECK-GSM and MATHCHECK-CO HECCKGEO, respectively. The leftmost column represents the average performance across all tasks and all question variants. The middle four columns detail the performance various mathe- matical reasoning tasks, while the right four columns display performance across different question variants. Consequently, each model is represented by a 4 4x4 checklist table, which showcases the
We define the difficulty according the number of reasoning steps of its answers (2 steps 8 steps)
5

---

Published as a conference paper at ICLR 2025
Table 1: Model performance on MATHCHECK-GSSE PS: Problem Solving, AJ: Answerable Judging, OJ: Outcome Judging, PJ: Process Judging, OP: Origianl Problem, PU: Problem Understanding, ID: Irrelevant Disturbance, SU: Scenario Understanding. Each score is the average score of related units. For example, 'll'' means all units, 'PS' includes solving units on four problem types, 'OP' includes original problems on four tasks units.
Models
All
PS
AJ
OJ
PJ
OP
PU
ID
SU
Generalist Models
01 01-preview
93.2
91.3
94.0
93.2
94,1
95.6
93.4
90.5
93.1
O1-mini
92.7
93.6
95.0
88.9
93.6
95.5
94.2
91.0
90.5
GPT-4o
92.0
95.0
95.0
90.1
87.8
94.6
91.6
92.0
89.6
GPT-4o-mini
87.2
90.1
89.6
88.6
80.4
88.9
89.4
85.6
85.1
111144Tuubb-20240040
90.9
93.8
95.9
87.8
86.0
93.8
90.4
90.8
88.6
GPT-3.5-Turbo
61.4
73.5
64.3
48.3
59.5
65.4
64.6
60.1
55.4
Gemini-1.5-Pro
86.3
88.6
89.5
87.6
75.0
88.0
90.2
85.0
82.0
Claud-335-somme-202440000
90.2
94.8
95.3
90.9
79.9
92.5
92.1
89.9
86.3
Chaud=33-opux-222442229
83.5
81.6
92.0
78.7
81.8
86.3
85.6
81.9
80.3
Claud=3-somnet2222440299
75.0
77.9
88.9
65.1
68.0
76.5
77.8
73.7
71.9
Claude-3-huikk-20224022
57.5
79.7
49.9
44.3
56.0
61.9
62.4
55.9
49.6
Llama-3. 1immn-3.--7B-Insttiie
82.2
93.3
91.2
89.8
Llama-3-70B 3ee0-----BB--s-ss
84.7
84.6
76.7
87.7
86.7
84.7
DeepSeek V2
76.9
85.1
84.4
83.5
Mixtral 8 x 7B-Instruct
59.9
63.9
61.6
62.8
61.5
58.8
Mixtral 7B-Base
44.7
35.3
50.6
47.8
41.2
Sweel.5-72B-CCC
64.2
31.9
35.1
$7.0
51.1
43.6
72.0
89.7
70.8
63.2
64.1
77.6
78.7
71.1
60.4
Phi-3-Mini-4K -IIstrut-Isttruct
64.1
71.3
64.5
62.9
57.6
68.5
66.6
61.2
60.0
Llama-3, 3,1---111811 8B-BBBB 1---B--nsttt
71.0
76.9
65.8
77.2
64.0
74.6
73.6
66.0
69.6
Llama-3-8B -ana---8B-Instruct
64.2
68.6
61.4
64.9
61.8
67.8
68.8
62.9
57.1
ChatGLM3-6B
36.5
32.6
41.7
50.1
21.7
39.7
35.9
31.3
39.1
Mathematical Models
SeopSeck-Math-7BB
50.7
79.5
50.0
45.1
28.
53.3
51.2
47.5
50.6
SeepSeck-MMth-7B...
50.2
70.0
64.8
40.4
25.8
51.6
54.4
45.8
49.2
DeepSeek-Maie
Maateeek--M------- Math-7B-Bas
44.0
49.8
51.5
44.0
30.8
49.0
46.0
37.0
44.1
MetaMath-LLLmm22-00 LLama2-70BB
45.7
70.0
35.7
45.3
31.6
49.9
51.5
43.4
37.8
Table 2: Model performance MATHCHECK-GRRH
Models
All
PS
AJ
OJ
PJ
OP
PU
ID
SU
Generalist Models
GPT-4o
65.3
37.5
75.5
69.5
58.8
65.2
67.0
64.3
64.8
GPT-4o-mini
59.0
50.8
69.8
61.4
53.8
61.9
62.0
54.1
57.8
GPT-44-Tubo-20244000
61.7
51.3
72.3
64.0
59.2
63.2
62.9
61.7
58.9
GPT----VisoonPreeiiee
60.0
46.7
71.1
63.6
58.8
59.3
62.8
57.8
60.2
Gemini-l.5-PP
58.7
47.5
67.4
55.0
64.6
62.3
58.6
57.1
$6.9
Gemini-I.5-Flash
56.8
45.0
75.1
50.6
56.7
56.8
59.7
53.8
$7.1
Claubb:35--ommee222220002
58.7
54.2
71.0
53.0
56.7
59.9
63.8
54.3
56.8
Clandee3-oquus-222244299
47.2
34.2
60.6
46.7
47.5
47.2
49.1
42.4
50.2
Clumbe33-somee2222240022
49.9
35.8
59.0
51.6
52.9
51.2
53.0
44.7
50.4
Clande-3hhilku-222440000
36.7
27.9
41.3
41.7
35.8
39.2
38.8
33.3
35.4
OWen2-Vi--22B-Instrnce
61.4
60.0
53.1
61.3
71.3
69.0
62.4
58.0
56.4
QWen2-VL-7B-BB
42.1
35.8
49.4
46.4
36.7
40.9
45.6
41.7
40.0
InternVL-115--Cin
37.6
22.1
54.9
46.8
26.7
42.9
34.8
37.3
35.5
MiniCPM-Llamal V-2.5
37.3
37.5
38.1
45.0
28.8
37.4
45.0
35.2
31.6
L.aVA--.6-Mistral--- Instruct
31.8
10.0
38.8
51.2
27.1
33.8
35.5
28.4
29.2
Phi-3-Vision -2iiiinn------nnneeeee- Instruct
29.6
12.9
35.0
48.6
22.9
32.6
31.8
28.2
26.0
Cog CogVI.M2-Llanna-Chae Chaa--hhh9r19
24.6
7.9
26.4
46.3
17.9
27.2
28.0
22.4
20.9
model's performance various dimensions. in Appendix A and
The details of all checklist tables are further elaborated

---

Published as a conference paper at ICLR 2025
On MATHCHECK-G (Table D.. O1- -1-preview and O1-mini exhibit outstanding performance with impressive overall score of 93.2 and 92.7, demonstrates strong effect of extending reasoning thought exploration. GPT-4o is closely followed with score of 92.0 and demonstrates top performance on the problem solving task and irrelevant disturbance variants. These results indicate that strong foundational models still possess formidable and robust performance across variety of mathematical achieves the highest score
reasoning tasks. Among the open-source LLMs, D0MMaa-,--77B-In--e of 90.5 and performs excellently across a range of tasks and problem variants. Its performance has significantly improved compared to LLaMA-3 version and surpasses that of GPT--4o-mini. Besides, wwenl.5-72B-
underperforms in tasks other than problem solving, which we suspect due to its special optimization of the solving task. This phenomenon is also observed across all math- customized models, which tend to be trained on similar mathematical problems and problem-solving processes, resulting in relatively narrow scope of reasoning capabilities.
On MATHCHECK GEO (Table 2) GPT-4o demonstrates the best performance, achieving top score of 65.3 in the All category. The performance of GPT4-turbo-202440409 and GPT4-Vision- Preview is similar, reaching scores of 61.7 and 60.0, respectively. In particular, the performance of Claude-3-sonnet is slightly superior in visual contexts compared to that of its larger counterpart, Claude 3-opus. Among the open-source MLLMs, the large-size MLLMs demonstrate surprisingly strong performance, with the most of small-size MILMMs exhibited poor performance especially in probelm solving, which suggests that the multi-modal reasoning capabilities of open-source small-size open-source MILMs still have significant room for improvement.
attaining 60.4 over the UPT-4-Vision-PPeviiey However,
MathCheck
1.0
Accursiy OF MatrChock GSM
Jams
Accuracy CSMBk
0.8
0.8
0.6
Notral BcTo
Plara Cale Cad
0.6
Votre Left Som
0.4
0.880,
0.097
0.912,
0.091
0.4
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
Accuracy of GSM1K
Accurscy of GSMLK
Figure 3: Correlation with GSMIk Zhang et al. 2024a), dataset that reflects real mathematical
reasoning ability. p and represent the Pearson Correlation Coefficient, and Root Mean Square Error.
3.4 MATHCHECK REPRESENTS MATHEMATICAL INTELLIGENCE MORE LINEARLY
One desiderata of a good mathematical benchmark is to reflect real mathematical intelligence perfectly. We follow previous works (Zhang et al. 2024a Huang et al. 2024a) to assess "intelligence" from practical standpoints and use performance on private data (Zhang et al. 2024a) and compression efficiency (Du al., 2024 HHang al. 2024a) surrogates
the genuine mathematical abilities of models. By examining the correlation between MATHCHECK and these surrogates, we can verify whether our design effectively reflects mathematical intelligence, and how it compares to traditional benchmarks.
Correlation with Private Data. Unlike traditional open-sourced benchmarks, private data is less likely to be contaminated or overfitted, making it an appropriate proxy of genuine mathematical intelligence. We adopt GSMIk (Zhang et al. (0044)). new private GSM8k-level dataset, to measure the real mathematical reasoning of models. We compare the correlation of model performance between GSMlk and MATHCHECK- MOMBEECKKGMMGGSMB.. As shown in Figure 3 the left part illustrates the correlation between GSM8k and GSM1k. It reveals that most LLMs achieve scores up to 80% on GSMSk, with scores concentrated in the top half of the graph. However, on GSMIK, the scores are evenly distributed, indicating that some LLMs such as deepseek-matt-7BBRRR have inflated scores on GSM8k. This suggests that the GSM8k score is not reliable benchmark for assessing the true mathematical reasoning ability of the models. In the right sub- sub-figure, WATHCHECK-GSM and GSMIk display good positive correlation, and some models that do not perform well on GSMIk can be detected by MATHCHECK -GSM. By comparing the Pearson correlation coefficient and the
7

---

Published as a conference paper at ICLR 2025
GSMOK vs BPC-Loss
MathCheck- GSM vs BPC-Loss
0.8
p: -0.822.
Accuracy MattCheckk L-CSM
0.6
pt
GSMBU
0.6-
0.5
Syand 5:72e
0.42
0.36
BPC-Loss
BPC-Loss
Figure 4: Performance correlation with BPC-loss, which reflects compression efficiency (Huang
et al. 2024a). The lower BPC-loss represents the higher compression efficiency.
DeapSeck Math-78-8aa0
LL
70-
70.0
75,0
MetaMeih-LLLMM
Accumeyy (1%)
DeepSeek- MaypSeee-Mal-77--RL
80
60
40.8
40.4
41.5
S5.7
304
25.8
28.1
31.8
Prodfan Boleng
Outeure Julling
Prucess July
Proskos Selving Arseorabie Julging Outicee Judginn Plososs .udging
Figure 5: Behavior of mathematical models trained massive solving data.
root mean square error, shows that MATHCHECK has higher correlation coefficient with GSMIK; mitigating bias evaluation caused by overfitting and data contamination.
Correlation with Compression Efficiency. Compression efficiency has been empirically proven that represent intelligence well (Du et al.) 2024) even linearly (Huang et al. 2024a). well aligned with
the belief that compression is closely connected to intelligence (Deletang et al. 2024). Following compression
Huang et al. (2024a), we BPC-Loss in Arxiv papers tagged with "Math" to efficiency as surrogate. Figure sssows the correlation between BPC-Loss and GSMBK/MattCheckk GSM. The left sub-figure reveals that single traditional benchmark like GSM8K cannot adequately reflect genuine mathematical ability, as indicated by the low Pearson correlation coefficient (p -0.822). Many models, such the Qwen series, deviate significantly from the regression line. In contrast, the right sub-figure displays the correlation with our LATHCHECK-GST demonstrating that MATHCHECK-GST exhibits significantly better correlation with genuine intelligence, with Pearson correlation coefficient of p = 00.955. Our method shows that many models, such as the Qwen series, have scores on our benchmark that align more accurately with their true mathematical abilities. It shows that our design can represent mathematical intelligence more linearly.
4 BEHAVIOR ANALYSIS
MATHCHECK contains multi-dimensionn information for evaluation, therefore we can observe the behaviors of the models on it to help analyze the models.
Behavior of Math Models. Recently, some works claim that math reasoning ability is greatly
improved by training on massive amounts of math solving data. To validate whether their mathemati- cal reasoning ability really improves, we examine the behaviors of the math models and their base models on МATиCHЕСK. As shown in Figure compared with the base model, the performance of D
on solving units is greatly improved. However, the performance improvement on other units is limited, or even downward. The same phenomenon can be observed MetaMath. It implies that training solely massive solving data (Yue et al.] 2023 Li et al.) 2024a; Tang et al.] 2024) is not the right direction to improve mathematical reasoning ability. Instead, training models with diverse mathematical data, beyond just solving, should be considered.
8

---

Published as a conference paper at ICLR 2025
100
Original Problem Solving
CoT (zero-shos)
Teweshot
(fewssht))
All
90
Problem
Solving
Understanding
Accuracy
80
Answerake Judg ng
Inreevant Disturbance
GPT44
60
LLMMa-3-708
Phi-3-medium
Outcame Judg ng
Probiem moarstanding
50
Deepseek-math-7b-l
>=
3
Process Judging
Origianl
Reasoning Steps
Problem
Figure 6: Performance on different complexity levels (i.e., Figure 7: Different prompting technolo- gies on MATHCCECK-GGM.
reasoning steps) MATHCHECK-GGM.
Reasoning Consistency. We analyze the reasoning consistency of generalist models across each unit MATUCHECK, and the detailed results are shown in popendix[A]ann and We can see most of them show good reasoning consistency since they achieve similar scores each unit, such GPT series, Llama-3 series and Mixtral series on MATHCHECK -GM and GPT series on MATHCHECK- GEO. This is interesting finding it substantiates a problem can robustly work well on multiple related tasks. Meanwhile, we also find that some models perform reasoning inconsistently. For example, wwenl..-72B-chatt Claude-3- 3-Hiku and Phi-3-Medium show excellent performance on the solving task but much worse in other units of MATHCHECK-GSII On МATHCHECK- GGMM, Internet-VL achieves high score of 40.0 on the original problem solving but decreases considerably when the problem switches to other robustness variants. These abnormal inconsistency behaviors of generalist models are highly similar to those mathematical models, revealing that they may conduct excessive decoration original benchmarks.
assertion: model that really understands
Behavior on Different Complexity Levels. We categorize the complexity of problems based on the number of reasoning steps of the original problems, and select representative models of varying sizes for evaluation, as depicted in Figure 6. We can observe that the models' accuracy on the original problem solving fluctuates and does not show obvious downward trend the problems are more difficult. While the score "ALL"" shows a steady downward trend, it implies that MATHCHECK better demonstrates the reasoning skills and capabilities required when problems become difficult.
Behavior on Different Prompting Technologies. We evaluate five prompting techniques including Zero-shot, Few-shot (Brown et al.) (2020), CoT (Wei et al., (202), Least to Most prompting (Zhou
al. 2022), and Plan-and-Solve prompting (Wang et al. 2023b). The results GPT-3.5-Tuubb on WATHCHECK-GSH are illustrated in Figure 7 Overall, Chain of Thought (CoT) and Plan-and-Solve (PS) in the zero-shot setting demonstrate superior performance, though this is not consistently the case across all tasks and settings. In contrast, the Few-shot prompt generally yields worse results than the Zero-shot prompt. Through detailed analyses, we find that the math reasoning generalization of LLMs is sensitive to Few-shot samples, which inspires us that Zero-shot with advanced prompt techniques (e.g., CoT or PS) may be better choice in mathematical reasoning tasks.
5 MATHCHECK APPLIED TO OTHER REASONING TASKS
MATHCHECK can be adapted to other reasoning tasks beyond mathematical problems. We attempt the migration of the MATHCHECK paradigm in both commonsense reasoning and code generation.
Commonsense Reasoning. It requires LLMs to apply parametric knowledge to reason and solve problems. We choose the date understanding task Big-bench (bench authors (023) as testbed since is wildly used to measure commonsense reasoning ability (Wei et al. 2022). Appendix E. shows the case of applying MATHCHECK to date understanding. Similar to math reasoning. date understanding is numerical reasoning task, therefore it can easily utilize variants of each unit in MATHCHECK. With MATHCHECK, raw data of date understanding have various test variants to examine the reasoning robustness and task generalization, helping us comprehensively evaluate models' date understanding.

---

Published as a conference paper at ICLR 2025
Code Generation. We would like to show the possibility of transforming MATHCHECK in some real-world reasoning tasks such as code generation. Appendix (.2 demonstrates a case of applying MATHCHECK to code generation. Unlike numerical reasoning, the adaptation of code generation should consider task relevance. For real-world tasks such as agents and robotics application, multiple variants reflects the diversity of environment and user requirements.
RELATED WORK
Benchmarks of Textual Mathematical Reasoning. Numerous benchmarks have been proposed
to evaluate the mathematical reasoning capabilities including (Amini et al.. 2019 Cobbe et al.
2021 Frieder et al.] 2024). Some datasets, such as the elementary-level GSMSK Cobbe al.
2021
Consequently, more challenging datasets have been introduced, including those at the high-
school level (Hendrycks et al.] (202]). university level (Sawada al.) 2023] Zheng et al., (022)) and olympic level (Huang et al. 2024b). Additionally, to provide a more comprehensive evaluation of mathematical reasoning abilities, numerous benchmarks have been developed that measure the robustness of mathematical reasoning (Li al. (2244b), including semantic perturbations (Wang et al.,
2023a Zhou et al.) 2024). reverse problem-sol solving (Yu et al. 2023; Berglund et al. 2023). , irrelevant distractions (Shi et al. 2023 Li et al. 2023) and functional variation questions (Srivastava et al.
2024| Gulati et al. (024). Above benchmarks paradigm can not comprehensively reflect reasoning
ability a given level. Therefore, MATHCHECK tries to go for better reasoning benchmark paradigm.
Benchmarks of Visual Mathematical Reasoning, Recently, multi-modal large language models
have demonstrated outstanding capabilities in visual-language reasoning tasks (Allaway al. 2022
Chen et al. 2023b; Yang et al.] 20233 Team et al.] 2023). Several benchmarks (Lin et al. 2014
Antol al. 2015 Hudson & Manning
et al.] 2019 Mobasher al. 2022) have been introduced to assess the visual reasoning capabilities of multi-modal large language models across
various modalities including abstract scenes, geometric diagrams, graphics, and charts (Lu al. 202T
Chen et al. | 2022, Masry et al., 2022 Kazemi et al. 2023] Lu et al., 2023). MATHCHECK- GEO offers comprehensive evaluation and testing platform for the research on visual math reasoning.
Benchmarks of Reasoning Consistency. Prior studies have identified limitations in reasoning
consistency. Wu al. (2023) designed counterfactual tasks to demonstrate that L.MS often rely on memorization to address general reasoning tasks. [Berglund et al.) (2023) found that LLMS struggle to answer inverse questions such as "B is A" after training on "A is B". In code reasoning, Gu eet al. (2024) and Liu et al. (2024a) observed that LLMs successfully generate solution but fail to correct the wrong one. Similarly, Oh et al. (2024) found the gap between generation and evaluation in TriviaQA (Joshi al. 2017). These findings inspire the design of MATHCHECK.
Strategies of Improving Mathematical Reasoning. Community has made significant efforts to enhance mathematical reasoning. In pre-training stage, previous works focus collecting (Wang
et al. 2023d: Paster et af.] 2024 Shao et al., 2024) and synthesizing (Akter et al.) 2024) math
documents. In addition, Lin et al. (2024) selected key tokens in math data during pre- pre-training In
post-training, numerous works generated massive problem-solving data for SFT (Yue et al.) 2023
al. 2024a Tang al. al.] 2024). Besides, reinforcement learning such as GRPO (Shao et 2024)
PRM (Lightman et al.) 2024) can further improve reasoning ability. In inference, prompt and search
strategies make LLMs reasoning better (Zhou et al. 2022 Wang al. 20236 Yao et 2024a).
CONCLUSION
In this paper, we argue that if model really understands a problem, it should be able to successfully solve various tasks and variations of that problem. Based on this. we introduce MATHCHECK. checklist for testing task generalization and reasoning robustness, along with an automatic tool for efficient checklist generation. MATHCHECK provides clear view of model performance across dimensions, enabling more comprehensive evaluation. Using it, we develop MATHCHECK-GSM for textual reasoning and MATHCHECK-GGE
for multi-modal reasoning. We evaluate massive (M)LLMs and conduct detailed analysis model behaviors on MATHCHECK. Subsequently, we reveal that MATHCHECK better reflects reasoning abilities than prior benchmark paradigm. Finally, we show the potential of applying MATHCHECK paradigm to other reasoning tasks. We hope our practice can constitute significant stride towards better reasoning benchmark paradigm.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
The work was partially supported by the following; National Natural Science Foundation of China under No. 92370119 62436009, 62276258 and 62376113, XJTLU Funding REF-22-01-002. the Science and Technology Development Fund of Macau SAR (Grant Nos. 0007/2024/AKP, FD- CT/0070/202/AMMI FDCT/060/202/AF)) the UM and UMDF (Grant Nos. MYRG-GRG2022 00006-FST-UMDE NYTGGGGRG222-00015-SSTTT- EF2002-00000-FFT),, and the Commonwealth Cyber Initiative (CCI).
EF2024-00185 EF2023-00151-FST,
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. /4219, 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2333.08774 2023.
Syeda Nahida Akter, Shrimai Prabhumoye, John Kamalu, Sanjeev Satheesh, Eric Nyberg, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Mind: Math informed synthetic dialogues for pretraining llms. arXiv preprint 0rXiv:2410./2885 2024.
Emily Allaway, Jena Hwang, Chandra Bhagavatula, Kathleen McKeown, Doug Downey, and Yejin Choi. Penguins don't t fly: Reasoning about generies through instantiations and exceptions. arXiv preprint arXiv:2205. 1658, 2022.
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziossi, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv:1905.13311 2019.
Anthropic. Claude 3, 2024a. URL http://ppw..//.aaatrrrroo//..///
Anthropic. Claude 3.5 sonnet, 2024b. claude 3- 3--snettt
URL
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 2023.
BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL
net forum? poorumiides dim-miiLs-uuusssss
Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Lims trained on" is b" fail to learn" b is a". arXiv preprint arXiv:2309.1220 12288, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P Xing, and Liang Lin. Geoqa: geometric question answering benchmark towards multimodal numerical reasoning. arXiv preprint arXiv:2 105. 14517, 2021.
Jigqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression. arXiv preprint 0222072 2022.
11

---

Published as a conference paper at ICLR 2025
Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. Theoremqa: A ducorem-drives question answering dataset. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023a.
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Car- los Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up a multilingual vision and language model. arXiv preprini arXiv:2305. 102005.88500 18565, 2023b.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.142388 2023c.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 10.:22100.1466. 2021.
Gregoire Deletang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter, and Joel Veness. Language modeling is compression. The Twelfth International Conference on Learning Representations, 2024. URL BRLhhtp::: httttss://openevvee..o normm jd=jjzngiynnus
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu. Zhilin Yang. and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 320-335, 2022.
Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective, 2024.
Michael C Frank. Baby steps in evaluating the capacities of large language models. Nature Reviews Psychology, 2(8):451-44 4522, 2023.
Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz. Philipp Petersen, and Julius Berner. Mathematical capabilities of chatgpt. Advances in Neural Information Processing Systems, 36, 2024.
Alex Gu, Wen-Ding Li, Naman Jain, Theo X Olausson, Celine Lee, Koushik Sen, and Armando Solar-Lezama. The counterfeit conundrum: Can code language models grasp the nuances of their incorrect generations? arXiv preprint arXiv:2402. 19475, 2024.
Aryan Gulati, Brando Miranda, Eric Chen, Emily Xia, Kai Fronsdal, Bruno de Moraes Dumont, and Sanmi Koyejo. Putnam-axiom: A functional and static benchmark for measuring higher level mathematical reasoning. In The 4th Workshop on Mathematical Reasoning and Al at NeurIPS'24 24, 2024.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-leve bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 103.03874, 2021.
Yuzhen Huang, Jinghan Zhang, Zifei Shan, and Junxian He. Compression represents intelligence linearly. arXiv preprint arXiv:2404.099 099377 2024a.
Zhen Huang, Zengzhi Wang, Shijic Xia, and Pengfei Liu. Olympicarena medal ranks: Who is the most intelligent ai so far? arXiv preprint arXiv:2406. 2024b.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
12

---

Published as a conference paper at ICLR 2025
Hyangeun Ji, Insook Han, and Yujung Ko. A systematic review of conversational ai in language education: Focusing on the collaboration with human teachers. Journal of Research on Technology in Education, 55(1):48 63, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint TrXXv:2401.040850 2024.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zeytlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprchension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1601- 1611, 2017.
Mehran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut. Geomverse: systematic evaluation of large models for geometric reasoning. arXiv preprint ar2142
2023.
Chen Li, Weiqi Wang, Jingeheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7b language models already possess strong math capabilities. arXiv preprint arXiv:2403.. 040.2-4033.44006, 2024a.
Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A comprehensive benchmark for evaluating the robustness of lims as mathematical problem solvers. arXiv preprint arXiv:2402. 19255, 2024b.
Tianle Li. Wei-Lin Chiang. Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsoureed data high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. /939, 2024c.
Zekun Li, Baolin Peng. Pengeheng Hc, and Xifeng Yan. Do you really follow me? adversarial instruc- tions for evaluating the robustness of large language models. arXiv preprint arXiv:2308. 10819, 2023.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 2014, Proceedings. Part V 13, pp. 740-755. Springer, 2014.
Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang. Jian Jiao, Nan Duan, et al. Rho-1: Not all tokens are what you need. arXiv preprint Stt1:2400407965. 2024.
Changshu Liu, Shizhuo Dylan Zhang, Ali Reza Ibrahimzada, and Reyhaneh Jabbarvand. Code- mind: A framework to challenge large language models for code reasoning. arXiv preprint arXiv:2402.0968 09664, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhou, Wenwei Zhang, Songyang Zhang, Dahua Lin, and Kai Chen. Mathbench: Evaluating the theory and application proficiency of Ilms with hierarchical mathematics benchmark. arXiv preprint arXiv:2405. 12209, 2024c.
Qianying Liu, Wenyu Guan, Sujian Li, Fei Cheng, Daisuke Kawahara, and Sadao Kurohashi. Roda: Reverse operation based data augmentation for solving math word problems. IEEE/ACM Transac- fions Audio, Speech, and Language Processing, 30:1-11, 2021.
13

---

Published as a conference paper at ICLR 2025
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song- Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint arXhv:2005.04165, 2021.
Jong-Chun Zhu.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. The Twelfth International Conference on Learning Representations, 2023.
Trung Quoe Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li. Reft: Reasoning with reinforced fine-tuning, arXiv preprint arXiv:2401.08967, 2024.
Jingyuan Ma, Damai Dai, and Zhifang Sui. Large language models are unconscious of unreasonability in math problems. arXiv preprint S...:::2003.194455 2024.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEF/evf conference on computer vision and pattern recognition, pp. 3195-3204, 2019.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
Meta. Introducing meta llama 3: The most capable openly available Ilm to date. htlps://ai.mela com/blog/meta-llane 2024.
Shaghayegh Mobasher, Ghazal Zamaninejad, Maryam Hashemi, Melika Nobakhtian, and Sauleh Eetemadi. Parsvqa-caps: A benchmark for visual question answering and image captioning in persian. people, 101:404, 2022.
Kole Norberg, Husni Almoubayyed, Stephen E Fancsali, Logan De Ley, Kyle Weldon, April Murphy, and Steven Ritter. Rewriting math word problems with large language models. In A/Ed23: artificial intelligence education, empowering education with LLMs workshop. 2023.
Juhyun Oh, Eunsu Kim, Inha Cha, and Alice Oh. The generative ai paradox on evaluation: What can solve, it may evaluate. arXiv preprint 1iXi:22422.06204 2024.
OpenAl. Gpt-3.5-turbo. 2022.
OpenAl. Gpt-4o, 2024a. URL
OpenAl.
bttp:://gpenai.commmmnee gpt 40 mini mewlllddvancinng costt-cost-ooo cowt-ooffiiieeee ittee-iineelleinne
Gpt-4o mini, 2024b.
URL
OpenAL
Gpt-4v, 2024c.
URL
https::/openat.ccm http:///peeaal.com/reeeerrhh
gpt 4v- 4w-44v--yyysee www.ssssttemmmm card
OpenAI.
http:://openaa.com//nebe -aptaai-o-mmiii -hwpaaoo----a-i--daaaaann------- www.ostefeffiittttcoeoeo-o---eee -eanttti-esnnnnnn
O1-mini,
2024d.
URL
OpenAI. O1-preview, 2024e.
URL.
https://openai.cm hemm/indexx
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. In The Twelfth International Conference on Learning Representations, 2024.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2080-2094, 2021.
Marco Tulio Ribeiro, Tongshuang Wu. Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of nlp models with checklist. arXiv preprint arXiv:2005.041 18, 2020.
14

---

Published as a conference paper at ICLR 2025
Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg. Pengming Wang. Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625(7995):468 475, 2024.
Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John Nay, Kshitij Gupta, and Aran Komatsuzaki, Arb: Advanced reasoning benchmark for large language models. arXiv preprint urXiv:2307.33692, 2023.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Y Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanacl Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In Proceedings of the 40th International Conference on Machine Learning, pp. 31210-31227, 2023,
Saurabh Srivastava, Anto PV, Shashank Menon, Ajay Sukumar, Alan Philipose, Stevin Prince, Sooraj Thomas, et al. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. arXiv preprint arXiv:2402. 19450, 2024.
Alessandro Stolfo. Zhijing Jin, Kumar Shridhar, Bernhard Schoelkopf, and Mrinmaya Sachan. A causal framework to quantify the robustness of mathematical reasoning with language models. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023.
YuHong Sun, Zhangyue Yin, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Hui Zhao. Benchmarking hallucination in large language models based unanswerable math word problem. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation CAREC-COLING 2024), pp. 2178-2188, 2024a.
Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. Easy-to-hard generalization: Scalable alignment beyond human supervision. arXiv preprint arXiv:2403.09472. 2024b.
Zhengyang Tang, Xingxing Zhang, Benyou Wan, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. arXiv preprini 2XXi:24433002844 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 67577995):476-48 2024.
Haoyu Wang, Guozheng Ma, Cong Yu, Ning Gui, Linrui Zhang, Zhiqi Huang, Suwei Ma, Yongzhe Chang, Sen Zhang, Li Shen, et al. Are large language models really robust to word-level perturba- tions? arXiv preprint 2309v // 166, 2023a.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy KeiWei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. arXiv preprint arXiv:2305.04 004999 2023b.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision- -inion-language model's perception of the world at any resolution. arXiv preprint ArXiv: /2/9/, 2024.
Weihan Wang. Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang. Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvim: Visual expert for pretrained language models, 2023c.
Zengzhi Wang, Rui Xia, and Pengfei Liu. Generative ai for math: Part matthiiee: A billion-tokern scale pretraining corpus for math. arXiv preprint 17X1v223/2.17/20, 2023d.
15

---

Published as a conference paper at ICLR 2025
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. of prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24873 2022.
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arXiv preprini M.Xiv:230.0024777 2023.
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang. Chung-Ching Lin, Zicheng Liu, and Li- juan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprins S34Y::230..14222 9(1):1, 2023.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024a.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint 040000180,, 2024b.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12288 2023.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653 2023.
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, et al. A careful examination of large language model performance on grade school arithmetic. arXiv preprint arXiv:2405.00332. 2024a.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403. 14624, 2024b.
Yifan Zhang, Yifan Luo, Yang Yuan, and Andrew C Yao. Autonomous data selection with language models for mathematical texts. In ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2024c.
Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. Minif2f: cross-system benchmark for formal olympiad-level mathematics. arXiv preprint arXiv: 109.001 10, 2021.
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205. 10625, 2022.
Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. Don't make your IIm evaluation benchmark cheater. arXiv preprint arXiv:2311.01964 2023a.
Zihao Zhou, Maizhen Ning. Qiufeng Wang. Jie Yao, Wei Wang, Xiaowei Huang, and Kaizhu Huang. Learning by analogy: Diverse questions generation in math word problem. In Findings of the Associ- ation for Computational Linguistics: ACL 2023, pp. 11091-11104, Association for Computational ogy. wwrg//r/0002 f ndi ngs-acl. 705
Linguistics, 2023b. URL
Zihao Zhou, Qiufeng Wang. Mingyu Jin, Jie Yao, Jianan Ye, Wei Liu, Wei Wang, Xiaowei Huang, and Kaizhu Huang. Mathattack: Attacking large language models towards math solving ability. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19750-1977 2024.
16

---

Published as a conference paper at ICLR 2025
Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. Dyval: Dynamic evaluation of large language models for reasoning tasks. In The Twelfth International Conference on Learning Representations, 2024a.
Kaijie Zhu, Jindong Wang, Qinlin Zhao, Ruochen Xu, and Xing Xie. Dyval 2: Dynamic evaluation of large language models by meta probing agents. arXiv preprint w4X. 2024b.
17

---

Published as a conference paper at ICLR 2025
APPENDIX
Heatmap of MATHCHECK-GSM
19
Heatmap of MATHCHECK-GEO
Data Statistics and Quality
22
C.I Overview Data
22
C.2 Effectiveness of GPT-4-turbo Rewriting
23
C,3 Discussion of data bias generated by GPT
23
Evaluation Setup
24
E MATHCHECK Applied to Other Reasoning Tasks
25
E.1 Date Understanding
25
E.2 Code Generation
26
Prompt List
27
F.1
Evaluation Prompt
27
F.2 Data Generation Prompt
G Case Problems
32
G.1 Case Problems in MATHCHECK-GSM Problem Group ID: GSM-54
32
G.2 Case Problems in MATHCHECK-GEO. Problem Group ID: GEO-15
38
18

---

Published as a conference paper at ICLR 2025
A HEATMAP OF MATHCHECK-GSM
OL Privicw
DLL Mini
4p
mobiim
95
0
95.7
95.5
98.4
9/.6
96
91.1
99
2
wcBla
96.9
96.9
91.8
93.0
Lnkkllaning
91.5
93.6
94.2
94.6
93.8
Unbic
Tebber 96.1
96
5
88
96.8
93.2
92.2
90.7
veleoand
92.2
9J.8
96
Uodentandin
94.6
33.8
90.9
93.0
03.0
93.8
87.6
87.6
Scari
02.2
Pobler
Asueesale pxg
Guisee pagging
Poblim Salwing
Aneverable
Poblin
Powess
Selinng
79 1xgea 1
20240409
93.0
90.2
BG.4
86.0
Origmal
97.1
90.7
Orighes 81.4
50.6
Underanniine
93.0
80.4
90.8
84.4
Undamaard dig
93.0
97.1
06.2
85.3
Undertanding
72.9
65.6
51.0
65.1
Indeuss
84.5
Dirurranee
mun
05.9
45.0
62.0
86.8 8
91.0
92.0
68.9
Secte
91.5
92.2
92.5
78.3
73.5
43.4
Pubker Anuenattc
Outomm
4 Outooe
Procem
habler
Inavenble
Outsve
Prowas
33.0
80.8
Ohiginal
97.7
96.9
84.5
Undentar
91.5
93.4
89.3
82.9
Undentarding
96
1
96.5
90.5
85.3
84.5
93.8
80
83.7
86.0
09.8
85
76.0
91.5
95.0
90.2
82.9
Serurtinns
74.4
93.8
75.6
83.7
81.3
88.9
92.3
Suma 1V
88.4
69.3
feswerati
Outrone
Frozem
Poblen Aneverable
Dulcome
Problen
Asseerabe
Outtome
frocm
Claude-2 S00eee--12/2110
No.den
79.1
89.3
70.5
Original
50.3
60.5
vol
96.3
96.3
99.4
89.9
Undwwarding
76.0
90.7
67.2
77.5
Undentesding
86.4 8
48.4
47.9
66.7
98.3
96
86.1
86.8
trckwai
86.2
40.1
94.5
94.2
88.3
82.2
80.6
89.1
42.2
Suctars
96.1
08.2
91.7
60.8
Frbban Aesnerst
Oatcoree
Prozes
Poblen Anpverable
Potlens Ansienbll
Gusame
Proves
89.5
$3.4
OHoinal
65.8
90.7
90.7
84
$1.4
Undentardiing
88
1
84.3
79.5
04.5
56.6
55
61.7
69.0
86.0
86.5
81-1
Dit Donie
83.7
84.3
82.2
03.7
Dularbence
55.8
58.0
62.4
58.9
83.5
Sce seb
79.6
60
62.5
Roben deseerate
Outoree
Proces
Poblkr Nusweralle
Oukoma
Pooess
Mixtrak-BB -er-all--77--Beee
Owen]
55.0
53.4
28.2
35.7
72.3
30.1
(Viik
69.2
Ardde 45.0
48.8
56.3
39.5
69.8
68.3
26.1
39.5
Uode chanding
91.5
78.6
69.8
75.2
Inehoal
Catatennce
02.2
86.3
60.7
62.5
65.1
33.3
46.4
8
53.3
Scerat
51.3
Prbian Aaswersk Oations
Prore
Potlm Jesnerab Outcome
Prowa
Prinss
Jangng
Figure& 8: Visualized heatmap of MATHCHECK-GSM Part
19

---

Published as a conference paper at ICLR 2025
85.1
78.7
Undarardryy ding
76.0
64.4
62.9
MedenStrring
02.2
64
75.2
72.9
75.1
63.0
66.6
66.7
62.8
63.9
55.3
62.0
56.
63.9
63.8
65.9
Sore1o
75
Scesere
56.4
Sahin
Redinn deave le
Cartove
Precess
Hoteen Nnoiooll
mades
ChatGLMB
DecoSeck
DcenSeek Nath
Orgin
Otyiral
45.7
76.7
64.2
Undemsrrd
32.0
41.9
49./
19.4
Undentanding
84
40./
40.3
24.8
Pobbin
76.7
63.3
32.6
Mesubencc
17.8
J0.4
44.3
24.0
Oits Danne
02.6
47.0
44.6
34.1
57.4
59.5
40.7
25.6
Underttanding
58.6
60.0
72.3
36.0
10.4
Pob Aeswarak
kissmo
Solving
DeepSiekc Nath Base
Hetahlath-1
lama2-708
Orgnnl
58.9
19.9
38.0
Pullen
77.5
38.8
46.8
36.4
Poblia
32.6
49.9
36.8
28.7
Oata
38.1
30.6
48.6
90.2
Undemardiing
48.8
58.
42.5
26.4
Undertandingg
59.0
24.8
36.6
20.9
133910
50m
1A38F6
Figure 9: Visualized heatmap of MATHCHECK-GSM Part 2.
20

---

Published as a conference paper at ICLR 2025
B HEATMAP OF MATHCHECK-GED
GPT-4c
GPT 407-40-m mini
GPT-4 Turbo 20240409
Orgna
13.3
77.7
74.5
35.0
$7.0
70.7
61.9
58.3
weBlar
53.3
75.8
63.7
50.0
Untostenday
48.3
74.9
64.8
60.0
55.0
66
3
65.0
53.3
30.3
12.5
00.3
00.0
48.3
48.3
71.6
73.4
58.3
76.0
06.3
38.3
50.0
62.4
48.3
51.7
73.3
53.8
Process
Poblrn Nnoveikle
Prwess
Sowingg
junging
Julpes
71.4
Origin
5G.7
66.5
59.4
66.7
78.3
47.2
Undenveriing
45.0
70
09.1
06.7
Undeslanding
60.0
71.0
56.9
66.7
Undeslendigg
41
76.5
53.9
06.
45.0
Desurtince
HOUT
41./
66.4
03.3
68
54,0
300019
$1.7
72.5
60.3
51.7
63.7
61.7
76.7
Rnbinn Aesnerst Oatcone
Frozes
Outicem
Process
Pobla Anevenble
Outora
Pooes
S-Opus
Orgea
60.0
53.3
Origin
31.7
53.4
50.0
51.0
56.7
Undosue mm
56.7
13.1 1
60.3
65.0
UnJoslendiny
38.3
67.7
41.9
48.3
40.0
62
55.0
$5.0
45.0
70:0
52.2
50.0
Docatannc
maat
25.0
59.6
46.7
36.3
31.7
52.1
55.1
40.0
Senaro
Scnearto
nobee en deceerikk Outooeo
Procss
Naken Aicerrate
Outicee
Pacess
Pokion anowesbllc
Powes
Elaude--44 10:00-21244000
OWiir 2-VL-728--nae 28-broiuct
35.0
39.0
45.8
36.7
68.3
60.6
73.6
73.3
43.6
50.0
indwworrding
30.0
45.5
38.1
41.7
Undaslanding
53.3
55.7
63.8
76.7
33.3
59.0
48.4
41.7
45.0
43.1
Decurtance
56.7
51.0
57.4
66.7
Ineewart 36.7
46.7
43.1
38.3
28.3
35.2
39.9
Outtone
Pcoro
Nezee Arseraale
Outcoms
Prctss
Publi
Srowerati
Pooes
51.3
Origin
34.8
40.9
21.7
53.0
15.0
57.3
46.7
20.0
43.3
48.9
51.2
36.7
11.7
39.5
$5.0
35.0
UNdosoe 4mg
Usit Staning
15.0
52.4
45.0
36.7
Dountann
26.7
41.2
44,7
26.3
8.3
41.7
40.3
23.3
Sonaro
55.4
Mole en deceerik Outoneo
Aicerrtte
outicme
Moces
Hokion irovesbble
Powess
11.7
35.1
52.0
32.7
Und
20.0
Undemardingg
13.3
33.8
53.4
26.7
11.7
25.0
47.1
28.3
51.1
10.0
24.0
49.0
6.7
Seraro
31.0
Scesarto
16.7
Pebien deswrrkee Octores
Promm
Maces
Figure 10: The visualized heatmap of MATHCHECK-GEO.
21

---

Published as a conference paper at ICLR 2025
C DATA STATISTICS AND QUALITY
C.1 OVERVIEW OF DATA
Table and Table show the data statisties of MATHCHECK-GSM and MATHCHECK-GEO, Table shows the data statistics of each group in MATHCHECK-GIS and MATHCHECK-GEO In each
group, since answerable judging and outcome judging are binary-classfficttion tasks, we try our best to include two different labels in these units for fair evaluation.
Table 3: Data statistics of MATHCCEECK-GSM
Problem Solving
Answerable Judging
Outcome Judging
Process Judging
Original Problem
129
258
258
129
Problem Understanding
129
258
258
129
Irrelevant Disturbance
129
258
258
129
Scenario Understanding
129
258
258
129
Table 4: Data statistics of MATHCHECK-GEO
Problem Solving
Answerable Judging
Outcome Judging
Process Judging
Original Problem
120
120
60
Problem Understanding
60
120
120
60
Irrelevant Disturbance
60
120
120
60
Scenario Understanding
60
120
120
60
Table Data statistics of each group in MATHCHECK -GM and MATHCHECK-GEO
Problem Solving
Answerable Judging
Outcome Judging
Process Judging
Original Problem
2
2
Problem Understanding Irrelevant Disturbance
2
2
Scenario Understanding
2
22

---

Published as a conference paper at ICLR 2025
C.2 EFFECTIVENESS OF GPT-4-TURBO REWRITING
In the process of human evaluation, we selected three graduate students as human annotators, all of them possess the mathematical skills required for evaluating the generated data. Our human evaluation principle is that the generated mathematical problems should maintain the correctness of mathematical logic. For example, in the "Problem Understanding", the generated question should not alter the logical structure of original question, which ensures the consistency between rewritten question and answer. The generated data will be marked as a failure any of annotators determines that the generation failed. Furthermore, annotators corrected each failed data instead of discarding them. This approach ensures our dataset is entirely accurate and the evaluation results are reliable.
We conduct statistics the pass rate of MATHCHECK- GSM rewritten by GPT4-turbo, shown in T It can be seen that the rewriting pass rate is high, which reflects the effectiveness of our generation method. The success rate of Problem Understanding and Scenario Understanding higher than 90%. There is a pass rate of 86.82% in the Irrelevant Disturbance and 81.40% in Wrong Step Rewriting. It provides references when we use MATHCHECK generation.
Table 6: Pass rate (%) checked by human annotators for the data generated by GPT4-turbo.
Rewriting Type
Problem
Irrelevant Disturbance
Scenario Understanding
Unanswerable Question Rewriting
Wrong Step Rewriting
Understanding
Human Pass Rate
93.02
86.82
91.47
85.38
81.40
C.3 DISCUSSION OF DATA BIAS GENERATED BY GPT
While we acknowledge there are possible self-bias in LLM-rewritten questions, we assert that this bias is acceptable and does not undermine the conclusions or rationality of MATHCHECK. This is supported by considerations across several dimensions.
Motivations. The motivation behind MATHCHECK is establish a paradigm that mitigates bench- mark hacking in the evaluation of mathematical reasoning, thereby revealing the genuine mathematical reasoning abilities of language models more comprehensively. Rewriting is an integral part of the MATHCHECK pipeline, which can naturally be performed by either humans or LLMs. While we acknowledge that involving experts in the rewriting process might be the fairest approach, the scala- bility of this method is significant concern, as noted in several of today's LLM benchmarks, such as Arena Hard (Li et al. 2024c) and MT-Bench (Zheng et al. 2023), due to the high associated costs. To enhance scalability and practicality, we opted to use LLMs the rewriters. Given that GPT-4 is widely recognized as the most advanced model accessible to the public, we believe that choosing GPT-4 as the rewriter is the closest approximation to the quality of expert human rewriting.
Hlmann-Checeee Questions. In fact, for the data construction which the LLM participates in, we mainly utilize the powerful rewriting ability of LLMs to edit the seed math problem instead of generating a new one from scratch. Moreover, we manually check the generated text to avoid some unnatural generated text.
Experimental Results and Analysis. On one hand, although the data are generated by GPT-4-Turbo our experiments, they do not bring extra benefits GPT-Family models make them obviously outperform others. As shown in Table the performance of Claude-3.5 -onnett is similar with GPT- Turbo, and even much better than GPT-4o-mini, which follows the commonsense on these L.Mss. On the other hand, we compare the experimental results on -on-Gie-Reenitten and GPT-Rewritten Questions. In some data constructions where the LLM is not involved, GPT4-family exhibits the same performance ranking as the score "All". Specifically, the samples in Original Problem&Outcomm Judging (OP-OJ) belong to Non- GPT-Reewiiiter Questions, which are generated based on the rules. Table shows that the performance ranking on non-LLMM-eneratee generated data is close to the score "All", where GPT-series continues to perform better than other advanced models. All these results verify that the possible bias to GPT models is acceptable in our MATHCHECK.
23

---

Published as a conference paper at ICLR 2025
Table Model performance on Non-GPT-Rewiritenn Questions of MATHCHECK-GSM
Models
All
OР-OJ
GPT-4o
92.0
91.8
MPPT44-urbb-22240444
90.9
88.9
Gemini-1.55-
86.3
84.6
Claude-3-Opus-202222
83.5
82.5
84.7
85.4
D EVALUATION SETUP
We conduct evaluations of multiple representative generalist and mathematical models on our MATH-
CHECK benchmark. For MATHCHECK-GGG -SMM, the evaluation models encompass: (a) Generalist
models, including proprietary models such as O1-Preview OpenAA 2024e), O1-Mini (OpenAT
2024d). GPT-4o (OpenAl 2024a), GPT-4o-mini (OpenAl 2024b). 4 Turbo (Achiam al.
2023). GPT-3.5-
[ (222)). Gemini-1.3--1 (Team et al. 2023). Claude-3 (Anthropic
2024 Claude-3.5-Sonme Anthropic (2024b), Llama-3 Llama-3. 3. DeepSeek V2 (Shao al.
2024). Mixtral x 7B (Jiang et al.] 2024), Qwenl (Bai et. 2023), Phi-3 (Abdin et al.] (0024) and
ChatGLM3 (Du al., 2022]; (b) Mathematical models, including DeepSeek-Math (Shao al. 2024)
and MetaMath (Yu et al. 2023). For MATHCHECK-GEOO we conduct evaluations on generalist models: (a) proprietary models such as GPT-4o (OpenAI 2024a), GPT--4omini (OpenAl, 2024b),
GPT-4-Turbo (Achiam et al.) (203)) GPT-4-vision (OpenAT]2024c), Gemini-1.55 1-5-Pi TTemm et al.)
2023), Claude-3.5-Sonnee Anthropic (2024b) and Claude-3 (Anthropic) 2024a); (b) open-source mod-
els including Qwen2-VL (Wang et al. 2024). InternVL-1.5 (Chen al. 2023c), Phi-3- -ision (Abdin)
al., 2024), A.LaVA--.6-MMisral-775 Instruct (Liu al. 2024b), MiniCPM-Llama3-V V-2.5 (Yao et al.
b024b) and CogVLM2-Llama3
(Wang et al. 2023c).
For Problem Solving and Process Judging tasks, we employ accuracy as the evaluation measure. For Outcome Judging and Answerable Judging tasks, utilize Macro-FI as the metric. We employ a zero-shot setting for generalist models and a few-shot setting (two-shot) for base models and mathematical models to enhance their ability to follow specific instructions and tasks. All the prompts used for evaluating (M)LLMs are provided in Appendix F.]
For all the close-resourced models, utilize the default hyper-parameters, setting the temperature O and the max tokens to 1,024. Similarly, for all open-source models, the parameters are uniformly configured as follows: do do__ddssnsso sample is set False, max gen gen_len len to 512, and the temperature is set to 0.
hhtpps://ai http:////ii.meeaa..cmmm/////////
https://ai .wt.a.a.m.mm. wwt.am.commllg//mee....... 3-
24

---

Published as a conference paper at ICLR 2025
E MATHCHECK APPLIED TO OTHER REASONING TASKS
Solving
Answerable Judging
Outcome Judging
Process Judging
Original Problem
Yesterdar date date
Yesterday's 's date was 4/38/2021. Mhat the date tosor row in WA/00/M/Y?
Yesterday's 's date was 4/38/2821, what is the date tororrow in
Yesterday's date was 4/30. what the date comDrrow NN/ W0/00//777
4/38/2021, Mhat tonorrow in HV/00/MYY?
"solution": Step 1:
Step 2: Determine the next day-
/01/00/MY??
"answer": Unanswerable
"solut ton": If yesterday was
4/38/2821, then Tonorrow
Since the current date i5
"answer": 5/2/2021
would be 5/02/2021.
"answer": Correct
The answer is 05/01/2021.
the last day of Aprit
Seed Data
"answer": ; Step
Problem Understunding
Yesterday April 38, 2021- Mhat the date tonorrow M1/00/YYY7
Yesterday April 30, 2023- Mhat the tomorrow H9/00/Y99?
Yesterday wes April 30, 2821. What date
Yesterday was April, 2821. Mhat is the date tomorrow in WK/DD/YP99
in 1M/00/YY7
"solution"s ... tono would ld 2021, 1M/00/1TY format
"solution"s Step If yesterday 4/38/2021, then Tоmorrow May
"answer": 5/2/2021
"answer": Inanswerable
Reasoning Robustness
2871. Step 2t... The answer
e5/01/2021.
is 05/01/3021.
"answer"; Incorrect
"answer": Step
Yesterday 2021. 4/23/2021. What
Scanario Understunding Irrelevant Disturbence
Yesterday was April Ll 30, 2021. A week ago it was 4/23/2021, What is the
Yesterday was April 30, 2021. week ago it was 4/23/2025. Mhat is the
Yes Yesterday was April, 30, 2821. week ago it was 4/23/2821. that the
it
date MK/00///Y2
date tonorrow in WM/00/W097
date tasorrow row in reN/00/YW??
date Lusorruw in 1R//00/YY??
"answer": 5/2/2021
"answer": Answerable
"solut foe": Tonorrow 05/81/2871.
The
... Stop ... Stop Since are soving forward by ane day from April 38th,
"satution":
"answer": Incorrect
one
day date, The answer is 05/01/2021. "answer" ; Step
Yesterday was April 2821. The date Tosorrow is 5/2/2821. what's the value of
Yesterday was April
Yesterday uas April X, 2021. The date tonorrov is 5/2/2021. What's 's the va lue of x?
Yesterday was April X, 2021. The date is 5/2/2021. What's 's the
5/2/2921.
value of x?
"satution": If tesorrow is May 2, 2821, then So, valuee of is 30 "answer":: Correct
"sotution" ton Stop1s Step Therofor, yesterday would May 1st, 2021. The valuee x i5 1. "answer's Steo 2
"answer": rinanseerable
2:
"ansver": 30
Task Generalization
Figure 11: Case of MATHCHECK in Date Understanding.
E. DATE UNDERSTANDING
To show that our proposed benchmark paradigm MATHCHECK can be adapted to other reasoning tasks beyond mathematical problems, we try to transform some representative reasoning task into MATHCHECK paradigm. We firstly apply it in commonsense reasoning, which requires L.LMs to apply world knowledge to reason and solve problems. Specifically, we choose the date understanding task in Big-bench (bench authors (023) since it is wildly used task measure commonsense reasoning ability (Wei et 2022).
Figure IT shows the case of applying MATIICHECK date understanding. Similar to mathematical reasoning, date understanding is a numerical reasoning task, therefore it can easily utilize variants of each unit in MATHCHECK. For example, in Irrelevant Disturbance, we can add some irrelevant date conditions to cause disturbance. In scenario understanding, we can ask for other variables in order to examine whether models have comprehensive understanding of this date knowledge. This case demonstrates the high adaptability of MATHCHECK commonsense reasoning task especially numerical reasoning.
25

---

Published as a conference paper at ICLR 2025
Solving
Answerable Judging
Outcome Judging
Process Judging
Problem Understonding Original Problem
Write Furction pether that takes strang returns without numerr.
Veste functior pythen
takes returns
that taces etring and returns string without numbrrs,
Krite Function
tates string and "oluation's
strarg vithout some specstic chors.
"solacion""
remoonn renfon bet, withoen, nume- thar text:
"onsver": Un
tex, ae toll surss
2
3
defremove, pumpppddt
telt wothout jume for cher intost
ouar test:
3
iioot char Hilgitti
text withou sunce ther return attour pums Step
bad wit
"nnwer"": Correct
Weite pytton tunction that vithout namber
Write pytton function that takcs String returns
takes string and returrs 2t
fonction
takes a string and returns it
furction that
nisber.
Nathout number.
aithout sumsers
"answer":
for in bedt
foor- thar Hiight
numbent)
"answer": Aesuerable
solation""
detremoove runitrett:
todl wwhhol runs
chartt test:
tax witheut
pisst chor ldight
bett:
Reasoning Robustness
tent Jum tech mors
whthort
test without jumn chcher
retur tedt
"answer"
Incorrect
"answer": Step
Irrelevant Disturbance
hrate pythor function that takes string containing etc.
takes containing
letters, netbers, symssis, etc.
pytton
Write sythan function that
letters,
string containing
etc.
Write pytton function that containing
letters, numbers,
and returns the string without Ehe numbers.
string the surters.
and retures the strang without
and returns the string without
numbers.
nuebers.
"answor":
"answer"; Answerable
*colution"
without
refremove serramoennumjeee mtamtttt
forcherin in bad
ihe dhe hoight
If the height
test weroout nurt thor witneutttumm num
withort nume char return withos Lyars
tnlight
withort nume ctar return test, wwt.wtttttttrr
"ansver": Incorrect
"answer"s " Step 3
Scenario Understending
Meite cuncttonn in Javs that vithout rusters.
string
krtte fanchion in jais that without
takes stringg and returrs strlig lutionn
furction Java
takes string and retures strting Mithout numbers.
pubkk crrt ttring renoenlammttrinng Insut)
"anover": Unanswerable
public
Minsrringgringg input)
"solutionn=
public easi der ng mecelaaattii Inpurt
String teet loper replowaat "1
retur toot:
return tust:
String text a itstt /WM7%
"ariwerr: Correct
"sosuere": Step
Task Generalization
Figure 12: Case of MATHCHECK in Code Generation.
E.2 CODE GENERATION
In addition to commonsense reasoning task, we would like to show the possibility of transforming MATHCHECK some real-world reasoning tasks. Specifically, we choose the code generation task due to its high relevance to Text2Sql, agents and robotics. Figure [2 demonstrates case of applying МATUCHECK to code generation. Unlike numerical reasoning tasks, the adaptation of code generation needs to consider task relevance. For example, in Scenario Understanding, we can ask models to write the same function in other program languages (Python to Java in our case) in order to examine whether models have a comprehensive understanding of this function requirements. It shows that MATHCHECK have potential for world tasks such as agents and robotics application. While encourage researchers to design more specific variants towards their reasoning task on MATHCHECK framework to test reasoning robustness and task generalization.
26

---

Published as a conference paper at ICLR 2025
F PROMPT LIST
F.1 EVALUATION PROMPT
You are an AI assistant that determines whether math problems are solved correctly. Answer the question. Finally give the answer in the format: The is:
Question: [QUESTION]
Answer:
1: Zero-shot Prompt of Problem Solving
You are an Al assistant that determines nes whether math probl ems are sol ved correctly. will first give you math problem and its solution, help judge whether the final answer is correct or incorrect. Give your judgment between Correct or Incorrect. Finally summarize your answer in the format:
The answer 1s:
Question: [QUESTION]
Solution: [SOLUTION]
Judgement:
2: Zero-shot Prompt of Outcome Judging
You are an AI assistant that identify which step begins the error in solution. I will give you math problem along with a wrong solution. Please help me identify fy the step where the errors begin. Finally give the wrong step in the format:
The answer is: Step i
Question: [QUESTION]
Solution: [SOLUTION]
Judgement:
Zero-shot Prompt of Process Judging
You are AI assistant that determines whether math problems answerable or unanswerable. Please analyze whether the question provides sufficient information to obtain an answer. Give your judgment between Answerable or Unanswerable. Finally summarize your answer in the format: The answer is:
Question: [QUESTION] Judgement:
4: Zero-shot Prompt of Answerable Judging
You are an AI assistant to help me solve math problems. Answer the question. Finally give the answer in the format: The answer is: Follow the given examples and answer the question.
Question: Leah had 32 chocolates and her sister had 42. If they ate 35, how many pieces do they have left in total? Answer: Step 1: Originally, Leah had 32 chocolates. Step Her sister had 42. So in tctal they had 32 42 74.
Step After cating 35, they had The answer is 39.
35 39.
Question: Jason had 20 1o11ipops. He gave Denny some lollipops. Now Jason has 12 lcllipops. How many 1611 pppps did Jason gi ve Denny?
27

---

Published as a conference paper at ICLR 2025
Answer: Step Jason started with 20 lollipops. step Then he had 12 after giving some Denny. Step 3: so he gave Denny 20 - 12 8. The answer is 8.
Question:
[QUESTION]
Answer:
5: Few-shot Prompt of Problem Solving
You are an Al assistant that determines whether math problems are solved correctly. will first give you math problem and its solution, help judge whether the final answer is correct or incorrect.
Give your judgment between Correct or Incorrect. Finally summarize your answer in the format: The answer is: Follow the given examples and give your judgment.
Question: Leah had 32 chocolates and her sister had 42. If they ate 35, how many pieces do they have left in total?
Solution: Step 1: Originally, Leah had 32 chocolates. Step Her sister had 42. So tctal they had 32 42 74.
Step 3: After eating 35, they had 74 35 39.
The answer is 39.
Judgment: Step and Step accurately calculate the total number of chocolates they both had originally. step 3 correctly calculates many they have left after eating 35 chocolates.
The answer is: Correct.
Question: Jason had 20 lollipops. gave Denny some lollipops. Now Jason has 12 lellipops. How many lollipeps did Jason give to Denny? Solution: Step 1: Jason started with 20 lollipops. Step2: Then he had 12 after giving some to Denny. Step3: So he gave Denny 20 12 8. The answer 32.
Judgment: Jason ended up with 12 lcllipops after giving some to Denny, having started with 20. Therefore, the calculation to find out how many lollipops Jason gave to Denny should be:20 - 12 8.
The answer is: Incorrect.
Question: [QUESTION]
Solution: [SOLUTION]
Judgement:
Few-shot Prompt of Outcome Judging
You are an AI assistant that identify which step begins the error in solution. I will give you math problem along with a wrong solution. Please help me identify the step where the errors begin.
Finally give wrong step in the format: The answer is: Step Follow the given examples and give your judgment.
Question: Leah had 32 chocolates and her sister had 42. If they ate 35, how many pieces do they have left in total?
Solution: step 1: Originally, Leah had 32 chocolates.
Step 2: Her sister had 42. So in total they had 32 + 42 84.
Step 3: After cating 35, they had 84 35 49. \nThe answer is 49. Judgment: The judgment of the vev steps is follows,
Step 1: Correctly states leah's initial amount of chocolates.
28

---

Published as a conference paper at ICLR 2025
Step Incorrectly calculates the total number chocolates both Leah and her sister had originally. The answer is: Step 2.
Question: Jasen had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lcllipops. How many lollipeps did Jason give Denny? Solution: Step 1: Jason started with lollipops.
Step 2: Then he had 12 after giving some to Denny.
Step 3: So he gave Denny 20 + 12 8.
The answer is 32.
Judgment: The correct method find out how many lollipops Jason gave Denny would be to subtract the amcunt he had left from the amount he started with: 20 - 12 8. Thus, The reasoning error begins at Step 3.
The answer is: Step 3.
Question: [QUESTION] Solution: [SOLUTION]
Judgement:
Few-shot Prompt of Process Judging
You are AI assistant that determines whether math problems answerable or unanswerable. Please analyze whether the question provides sufficient information to obtain an answer.
Give your judgment between Answerable Unanswerable. Finally summarize your answer in the format: The Follow the given examples and give your judgment.
is:
Question: Leah had 32 chocolatcs and her sister had 42. If they ate 35, how many pleces do they have left in total? Judgment: The question provides all necessary information to perforn the cal caluuaaiion
The answer is: Answerable.
Question: Jason had 20 lollipops. He gave Denny lollipops. How many lollipops did Jason give to Denny?
Judgment: The question is not answerable as given. The reason is that there is insufficient information te determine the exact number of lollipops Jason gave to Denny. The answer is: Unanswerable.
Question: [QUESTION] Judgement:
8: Few-shot Prompt of Answerable Judging
F.2
DATA GENERATION PROMPT
Your objective is to rewrite a given math question using the following perturbation strategy. The rewritten question should be reasonable, understandable, and able be responded by humans.
Perturbation strategy: Problem Understanding: It refers to transforming the original problem into new problem that uses different wording or different sentence structures but does not change the solution of the original problem.
The given question: (QUESTION)
Answer of the given question: (ANSWFR)
29

---

Published as a conference paper at ICLR 2025
Please rewrite the question using the specified perturbation strategy while minimizing edits to avoid sigmificant deviation in the question content.
It is important to ensure that the rewritten question has only one required numerical answer. You just need to print the rewritten question without answer.
The rewritten question:
Question: [QUESTION]
Answer: (ANSWER)
Given step: (STEP)
The rewritten answer:
9: Prompt of Problem Understanding Rewriting
Your objective is to rewrite a given math question using the following perturbation strategy, The rewritten question should be reasonable, understandable, and able be responded by humans.
Perturbaticn strategy: Irrelevant Disturbance: It involves introducing distracting conditions that have impact the final answer. These introcuced conditions should relevant the topic of the original question and preferably include numerical values. However, the rewritten problem must maintain an identical solution to that of the original problem.
The given question: (QUESTION)
Answer of the given question: (ANSWER)
Please rewrite the question using the specified perturbation strategy while minimizing edits to avoid significant deviation in the question content.
It is important to ensure that the rewritten question has only one required numerical answer. You just need to print the rewritten question without answer.
The rewritten question:
Question: (QUESTION)
Answer: (ANSWER)
Given step: (STEP) The rewritten
10: Prompt of Inrelevant Disturbance Rewriting
Your objective is to rewrite a given math question using the folloxing perturbaticn strategy. The rewritten question should be reasonable, understandable, and able to be respended to by humans.
Perturbation strategy: Jnanswerable question: It refers to eliminating condition from the original question that is crucial for solving it while keeping the rest of the content unchanged. The rewritten problem should longer have valid answer, it Jacks the constraint that removea.
The given question: (QUESTION)
Answer of the given question: (ANSWER)
Please rewrite the question using the specified perturbation strategy while minimizing edits to avoid significant deviation in the question content.
that the rewritten question has only required numerical answer. You just need print the rewritten question without answer,
It is important
The rewritten question: Question: (QUESTION) Answer: (ANSKER)
30

---

Published as a conference paper at ICLR 2025
Given step: (STEP)
The rewritten answer:
11: Prompt of Unanswerable Question Rewriting
You are assistant help rewrite question into declarative statement when its answer provided. Follow the given examples and rewrite the question.
Question: How many
in the parking lot? The answer is 5. in the parking lot.
Result: There
Question: How many trees did the greve workers plant today? The answer is 6.
Result: The grove workers planted trees today.
Question: If they ate 35, how many pieces do they have left in total? The answer is 39.
Result: They have 39 pieces left in total if they ate 35.
Question: Hew many lollipops Jason give to Denny? The answer is 8.
Result: Jasen gave 8 lollipops te Denny.
Question: How many toys does he have now? The answer is 9.
Result: He now has toys.
Question: How many computers are new in the server room? The answer is 29.
Result: There 29 computers new in the
rocm.
Question: How many golf balls did he have at the end of wednesday? The answer is 33.
Result: He had 33 golf balls the end of Wednesday.
Question: Hew much money does she have left? The answer Result: She has 8 money left.
Question: (QUESTION) The answer is (ANSWER)..
Result:
12: Prompt to Rewrite Question and Answer into Declarative Statement
Pollowing a question and its correct solution. Rewrite the solution on according following requirements: (1) Do not change the format (2) Keep those steps before the given step unchanged (3) Make minor changes to the given step that the reascning of this step and subsequent steps are incorrect, resulting in inccrrect answer.
Quostion: (QUESTION)
Answer: (ANSWER)
Given step: (STEP)
The rewritten answer:
13: Prompt Generate the Wrong Step
31

---

Published as a conference paper at ICLR 2025
G CASE PROBLEMS
G.1 CASE PROBLEMS IN MATHCHECK-GSM PROBLEM GROUP ID: GSM-54
[Question]: Mike plays ping peng for 40 minutes. In the first 20 minutes, he scores 4 paints. In the second 20 minutes, he scores 258 more points. How many total points did he score? [Answer] 9.0
14: Problem Solving Original Problem
[Question] During 40-minute ping pong session, Mike scores points in the initial half, the latter half, he manages increase his score by 25 compared to the first half. what is the total score Mike achieved in this session?
[Answer] 9.0
15: Problem Solving Problem Understanding
[Question]: Mike plays ping pong in a local tournament and decides to practice for 40 minutes before the first match. During his practice session, in the first 20 minutes, while intermittently checking his phone and hydrating, he manages to seere 4 points. In the following minutes feeling more warmed up and despite short break to adjust his paddle's grip tape, he scores 258 more peints than in the first session.
Considering these di distractions, how many total points did Mike score in his 40-minute practice session? [Answer] : 9.0
16: Problem Solving Irrelevant Disturbance
[Question] Mike plays ping pong minutes. In the first 20 minutes scores x points. In the second minutes, scores 25 more points. He scored total points. what is the value of unknown variable x
[Answer]: 4.0
17: Problem Solving Scenario Understanding
[Question]: Mike plays ping pong for 40 minutes. In the first 20 minutes, he scores points. In the secend 20 minutes, he scores 25% more points. How many total points did he score? [Answer] Answerable
18: Answerable Judging (Answerable) Original Problem
[Question]: Mike plays ping pong for minutes. In the first 20 minutes, he scores points. the second 20 minutes, his performance increases by 258. How many total points did he score? [Answer] Unanswerable
19: Answerable Judging (Unanswerable) Original Problem
[Question]: During a 40-minute ping pong session, Mike scores 4 points the initial half. In the latter half, he manages to increase his score by 25% compared to the first half. What is the total score Mike achieved in this session?
[Answer]: Answerable
20: Answerable Judging (Answerable) Problem Understanding
32

---

Published as a conference paper at ICLR 2025
[Question|: During a 40-minute ping pong session, Mike scores points in the initial half. In the latter half, be manages to increase his score by 25$ compared to the first half. what is the total score Mike achieved in this session?
[Answer]: Unanswerable
21: Answerable Judging (Unanswerable) Problem Understanding
[Question]: Mike plays ping pong in a local tournament and decides to practice for 40 minutes before the first match. During his practice session, in the first 20 minutes, while intermittently checking his phone and hyérating, he manages to score points. In the following 20 minutes feeling more warmed up and despite short break to adjust his paddte's grip tape, he scores 25% more peints than in the first session.
Considering these distractiens, how many total points did Mike score in his 40-minute practice session? [Answer] Answerable
22: Answerable Judging (Answerable) Irrelevant Disturbance
[Question]: Mike plays ping pong in a local tournament and decides to practice for 40 minutes before the first match. During his practice session, in the first minutes, while intermittently checking his phone and hycrating, he manages to score points. In the following 20 minutes, feeling more warmed up and despite a short break to adjust his paddLe's grip tape, he scores 25% more peints than in the tirst session. Considering these distractions, how many total points did Mike score in his 40-minute practice session? [Answer] Unanswerable
Answerable Judging (Unanswerable) Irrelevant Disturbance
Mike plays ping pong for 40 minutes. In the first 20 minutes he scores points. In the secend 20 minutes, he scores 25% more points. He scored total points. What is the value unknown variable
[Question]:
[Answer] Answerable
24: Answerable Judging (Answeruble) Scenario Understanding
Mike plays ping pong for minutes. the first 20 minutes, he scores points. In the second 20 minutes, he scores 25% more points. What is the value of unknown variable x? [Answer] Unanswcrable
[Question]
25: Answerable Judging (Unanswerable) Scenario Understanding
[Question] Mike plays ping pong for minutes. the first 20 minutes, he scores points. In the secend 20 minutes, he scores 25% more points. How many total points did he sccre? [Solution]:
Step 1: Find the number of points Mike scored in the second 20 minutes. To this, take which is 4 points, and increase this by 253.
number of pcints he scored in the first 20 minutes,
Step 2: To calculate 258 of 4, divide by 4 which gives (since 253 is the same as 1/4th).
Step 3: Add this 25% (which is peint) the original points find out how many points he scored the second minutes. So, points point = points.
Step 4: Now, to find the total number of points Mike scored in the 40 minutes, add the points scored in the first 22 minates to the points scored the second 20 mi minutes.
33

---

Published as a conference paper at ICLR 2025
So, points (first 20 minutes)
points (second 20 minutes) points
The answer is 9 points.
[Answer]: Correct
26: Outcome Judging (Correct) Original Problem
[Question] Mike plays ping pong for minutes. In the first nutes he scores paints. the secend 20 minutes, he scores 25% more points How many tctal points did he sccre?
[Solution]
Step 1: Mike scores points in the first 20 minutes.
Step Mike scores 25$ more points the second 20 minutes.
Step 3: Mike's 8 total points are the sum of his points in the first and second 20 minutes.
The answer is 28 points. [Answer] Incorrect
27: Outcome Judging (Incorrect) Original Problem
[Question]: During 40-minute ping pong session, Mike scores 4 points in the initial half. In the latter half, he manages to increase his score by 25% compared to the first half, what is the total score Mike in this session?
[Solution]:
Step 1: Calculate Mike's score in the first half, which is given as points.
Step Calculate the 25 increase in score the second half. | (25) ( of } 4 (1frac(25)100)) Atimes - 1W).
Step 3: Add this increase to the initial score to find the score for the second half. \M44
5\)).
Step Combine the scores from both halves get the total score. (1 \ (first half) } ( (second half) 9\1)
The answer is 9. [Answer] Correct
28: Outcome Judging (Correct) Problem Understanding
[Question] During a 40-minute ping pong session, Mike sceres 4 points the initial half. In the latter half, he manages to increase his score by 25 compared to the first half. what is the total score Mike achieved in this session?
[Solution]:
Step 1: Mike scores 4 points in the initial half, so his score after 20 minutes is 20 minutes \/x points/20 minutes points.
Step 2: In the latter half, Mike's score increases by 25$ compared to the first half, 25% increase) his points after 20 minutes is 8 points + points W points.
points Step 3: Therefore, Mike's s total score after 40 minutes is 10 points points (frem the initial half) = 14 points. The answer is 14 points. [Answer]: Incorrect
29: Outcome Judging (Incorrect) Problem Understanding
[Question]: Mike plays ping pong in a local tournament and decides to practice for 40 minutes before the first match. During his practice session, in the first minutes, while intermittently checking his phone and hycrating, he manages to scere 4 points. In the following minutes feeling more varmed up and despite a short break to adjust his paddle's grip tape, he scores 25% more points than in the first session.
Considering these di distraction,, how many total points did Mike score in his 40-minin nute practice session?
34

---

Published as a conference paper at ICLR 2025
[Solution]:
step Identify the points scored by Mike in the first minutes, which is points.
step 2: Calculate the number of pcints scored in the next 20 minutes by finding 253 more than the points sccred in the first session. 258 of 4 points is (which is 0.25 4)
Step 3: Add the additional points to the first session's a points to find the total points scored in the second session. So, 4 points point = points in the second session.
Step To find the total points sccred in the 40-minute practice session add the points from the first and second sessions together.
So, points + points points. AnAnThe answer is points. [Answer] Correct
30: Outcome Judging (Correct) Irrelevant Disturbance
[Question]: Mike plays ping pong a local tournament and decides to practice for 40 minutes before the first match. During his practice session, in the first 20 minutes, while intermittently checking his phone and hycrating, he manages to sccre 4 points. In the following 20 minutes feeling more warmed up and despite short break to adjust his paddle's grip tape, he scores 25% more pcints than in the first session. Considering these distractions, hew many total points did Mike score in his 40-minute practice session? [Solution]:
Step 1: Calculate the points sccred in the first 20 minutes. \nd points 20 minutes = points per minute step 2: Calculate the points scored in the second 20 minutes, An25# more points than in the first session 1.25 2 points/minute 2.5 points/ minute
Step 3: Calculate the total points scored in the 40-minute practice session. \n2 points/minute x 20 minutes - 40 points\n2.5 points/mi nuite x 20 minutes 50 points Step 4: Add the points scored in the first and second sessions. An40 points 50 points 90 points The answer is 90 points. [Answer] Incorrect
31: Outcome Judging (Incorrect) Irrelevant Disturbance
[Question]: Mike plays ping pong for minutes. the first 20 minutes, he scores points. in the second 20 minutes, he scores 25% more poi nts. He scored total points. What is the value of unknown variable x? [Solution]:
Step 1: Define the given informatien: Mike plays ping pong for 40 minutes scores x points in the first 20 minutes, and scores 25% more points in the second 20 minutes.
peints in the second 20 minutes, can express the points scored the second 20 minutes \\x 0.25x!1). Step 3: This can be simplified as \1(1.25x\ representing the points scored in the second 20 minutes.
step 2: Since he scores 25
Step 4: since he scored 9 points in total, we can write the equation: In
Mix + 1.25x = 9XA
Step 5: By simplifying the equaticn, we combine like terms: \n\ 12.25x 9\1]
Step 6: To find the value of x, we divide both sides of the equation by (2).
2.25:\n\\(x
Step 7: Simplify the division: \n \xx 4\M The answer is x 4. [Answer] Correct
32: Outcome Judging (Correct) Scenario Understanding
35

---

Published as a conference paper at ICLR 2025
[Question]: Mike plays ping pong for minutes. In the first 20 minutes, he scores x points. In the second 20 minutes, he scores 25 more points. He scored total points. what is the value of unknown variable x?
[Solution]:
Step 1: Let's first find out many points Mike scored the first 20 minutes.
Step 2: Since Mike scored x points in the first 20 minutes, we know that x/2 9 (since he scored 9 points in total). step 3: New, know that ( 18. The answer 18. [Answer] Incorrect
33: Outcome Judging (Incorrect) Scenario Understanding
[Question]: Mike plays ping pong for 40 minutes. In the first 20 minutes, he scores points. In the secend 20 minutes, he 258 more points. How many total points did he sccre? [Solution]:
step 1: Find the number of points Mike scored in the second 20 minutes. To do this, take the number of peints he scored in the first 20 mi minutes, which is 4 points, and increase this by 253.
Step 2: To calculate 25 of 4, divide by which gives (since 253 is the same as 1/4th).
Step 3: Add this 25% (which is 2 peints) to the original 4 points to find out how many points he scored in the second 20 minutes. So, points - points 6 points.
Step 4: Now, to find the total number of points Mike scored in the 40 minutes, add the points scored in the first 20 minutes to the points scored in the second 20 minutes. Sc, points (first 20 minutes) + points (secend 20 minutes) 10 points. The answer is points. [Answer]: : step
34: Process Judging Original Problem
[Question]: During 40-minute ping pong session, Mike scores 4 points in the initial half. the latter half, he manages increase his by 258 compared to the first half. What is the total score Mike achieved in this session?
[Solution]:
Step 1: Calculate Mike's score in the first halt, waich is given points.
Step 2: Calculate the 25% increase in score for the second half. (1 W
div 25 0.16\
Step 3: Add this increase to the initial score to find the score for the second half. (M 0.16 4.16\\.
Step 4: Combine the scores from both halves to get the total score. M4 \ttext (first half) } 4.16 \ttext (second half) } = 8.16\1).
The answer is 8.16. [Answer] step
35: Process Judging Problem Understanding
[Question]: Mike plays ping pong in à local tournament and decides to practice for 40 minutes before the first match. During his practice session, in the first 20 minutes, while intermittently checking his phone and hydrating, he manages to score points. In the following 20 minutes feeling more warmed up
despite short break to adjust his paddle's grip tape, he scores 25% more points than in the first session.
Considering these distractions, how many total points did Mike score in his n0-minute practice session? [Solution]
36

---

Published as a conference paper at ICLR 2025
Step Identify the points scored by Mike in the first 20 minutes, which is points.
step 2: Calculate the number of points scored in the next 20 minutes by finding 25% more than the points scered in the first session. 25 of 4 points is 1 (which is 0.25 4)
Step 3: Add the additional points the first session's a points find the total points scored in the secend session. So, 4 points 1 point points in the second session.
Step 4: To find the total points scored in the 40-minute practice session add the points from the first and second sessions together. So, 4 points 6 points 10 points.
The answer is 10 points. [Answer]: Step
36: Process Judging Irrelevant Disturbance
[Question]: Mike plays ping pong for 40 minutes. In the first 20 minutes, he scores points. In the second 20 minutes, he scores 25$ more points. He scored total points. what is the value of unknown variable x? [Solution]:
Step 1: Define the given informaticn: Mike plays ping pong for 40 minutes scores x points in the first 20 minutes, and scores 258 more points in the second 20 minutes.
peints in the second 20 minutes, express the points scored the second minutes as \Mx 0.25x(1). Step 3: This can be simplified as \1(1.25x) representing the points
step 2: Since scores 25%
scored in the second 20 minutes.
Step 4: Since he scored 9 points in total, can write the equation:\n
AMx + 1.25x = 9%%1
Step 5: By simplifying the equation, we combine like terms: \n [ 9\\]
Step 6: To find the value of x, we divide both sides of the equation by 2.25:\n\M[x (2)
Step Simplify division: \n\ 511J answer is 5. [Answer]: Step
37: Process Judging Scenario Understanding
37

---

Published as a conference paper at ICLR 2025
G.2 CASE PROBLEMS IN MATHCHECK-GEO. PROBLEM GROUP ID: GEO-15
G
E
D
Figure 13: Geometry diagram for geometry problems in group 15.
[Question]] As shown in the figure, the diameter CD of Modot. O crosses the mi miépoint G of chord EP, Vangle DCF 20.0, 0, then Wangle ROD is equal to 0) V Naggree [Answer] 40.0
38: Problem Solving Original Problem
[Question]: In the circle with center O, diameter CD intersects the midpoint G of the chord EF, and the measure of angle DCF is 20 degrees. Determine the measurement of angle ECD in degrees. [Answer] : 40.0
39: Problem Solving Problem Understanding
[Question]: In the figure of circle 0, the diameter CD intersects the midpoint G of the chord EF. The length of the chord B5 is 7.5 em, which angle measurements. The angle Vangle DCF is given to be 20.0 degrees. We need to calculate the angle Mangle BCD, What is the measure of this angle in degrees? [Answer]: 40.0
is irrelevant
40: Problem Solving Irrelevant Disturbance
[Question] As shown in the figure, the diameter CD Modot. o crosses the miapoint G of chord EF, Mangle DCF Vangle ECD is equal 40\ 40V\degree. What is the value of unknown variable x? [Answer]: 20.0
41: Problem Solving Scenario Understanding
[Question]: shown in the figure, the diameter Modot. o crosses the miapoint G of chord EF, Mangle DCF 20.0, then Mangle EOD is equal to () (oVoreee
[Answer] Answerable
42: Answerable Judging (Answerable) Original Problem
[Question]: As shown in the figure, the diameter CD Vodot 0 chord EF, Wangle DCF 20.0, then Mangle EOD is equal () Videggee [Answer] Unanswerable
43: Answerable Judging (Unanswerable) Original Problem
[Question]: In the circle with center o, diameter CD intersects the midpoint G of the chord EF, and the measure of angle DCF is 20 degrees. Determine the measurement [Answor]: Answerable
angle ECD in degrees.
44: Answerable Judging (Answerable) Problem Understanding
38

---

Published as a conference paper at ICLR 2025
[Question] In the eirele with center o, diameter CD intersects the midpoint G of the chord EF. Determine the measurement of angle EOD in degrees.
[Answer]: Unanswerable
45: Answerable Judging (Unanswerable) Problem Understanding
[Question]: In the figure of circle G, the diameter CD intersects the mi midpoint 6 of the chord EF. The length of the chord BF is 7.5 cm, which is irrelevant to our angle measurements. The angle Vangle DCF is given to 20.0 degrees. We need to calculate the angle Vangle ECD. What is the measure of this angle degrees? [Answer]: : Answerable
46: Answerable Judging (Answerable) Irrelevant Disturbance
[Question]: In the figure of circle the diameter intersects the midpoint G of the chord EF. The length of the chord is 7.5 cm, which is irrelevant to our angle measurements, We need to calculate the angic Vangle EOD. What is the measure of this angle in degrees? [Answer] Unanswerable
47: Answerable Judging (Ununswerable) Irrelevant Disturbance
[Question]: As shown in the figure, the diameter CD Wedot o crosses the midpoint G of chord EF, Vangle DCF = Mangle ROD is equal to 40 40V\degree. What is the value of unknown variable x? [Answer] Answerable
48: Answerable Judging (Answerable) Scenario Understanding
[Question] As shown in the figure, the diameter CD of Modot o crosses the miapeint G of chord EF, Mangle DCF = x. What is the value of unknown variable x? [Answer] Unanswerable
49: Answerable Judging (Unanswerable) Scenario Understanding
[Question]] As shown in the figure, the diameter CD of Modot O crosses the mi miépoint G of chord EP, Mangle DCF 20.0, 0, then Wangle ROD is equal to (VVeegee
[Solution]:
Step 1: Identify that the angle DCF is an angle formed by a chord (CF) and tangent (CD) at the point c. Step 2: Recognize that the angle between chord and tangent the point contact is equal angle ECF in this case.
the angle in the alternate segment, which
Step 3: Thus, the measure of angle ECF is also 20 degrees.
Step 4: Observe that the diameter CD passes through the center 0 of the circle and cuts the chord EF its midpoint G.
Step 5: Recall that a line drawn from the center of the circle to the midpoint of a chord is perpendicular to the chord.
Step 6: Therefore, triangle EDG is a right-angled triangle at G.
Step 7: Understand that angle EOD is double the angle ECF because angle
FOD is the angle at the center whereas angle ECF is the circunference based on the Circle Theorem.
Step 8: Calculate angle EOD as 2 times angle ECF, which is 2 20 degrees
Step Perform the multiplication:
20 degrees degrees.
The answer is: Angle BOD measures degrees. [Answor]: Correct
50: Outcome Judging (Correct) Original Problem
39

---

Published as a conference paper at ICLR 2025
[Question] As shown the tigure, the diameter CD Vedot o crosses the miapoint G of chord EF, Vangle DCF = 20.0, D, then Mangle EOD is equal to () ldegree
[Solution]:
bisected by diameter CD point G. Step 2: Recegnize that Vangle DCF is an angle formed by the diameter CD and chore EF, so it is inscribed in the semicircle of Vodot c.
Step 1: Netice that chord
Step 3: Recall that an angle inscribed in semicircle is a right angle; hence Wangle DCF 90V\degree (nct the given 201 \degree, which seens to be mistake).
step 4: Realize that Vangle EOD is the angle at the center of the circle and it stands on the are EF
Step 5: Understand that the angle at the center of circle is twice the angle the circumference when both stand on the same Step 6: Since Mangle DCF a right angle (901 (dVereere)) and Vangle BOD is twice that angle at the center, Mangle EOD is 2 * 90) Step 7: Calculate 2 90% \degree to get 180 \degree. The answer is Vangle EOD is equal to 180% Adegree. [Answer] Incorrect
51: Outcome Judging (Incorrect) Original Problem
[Question] In the circle with center o, diameter CD intersects the midpoint G of the chord EF, and the measure of angle DCF is 20 degrees. Determine the measurement of angle EOD in degrees. [Solution]:
Step 1: Identify that the angle DCF is an angle formed by chord (CF) and a tangent (CD) at the point C.
Step 2: Recognize that the angle between a chord and a tangent at the point of contact is equal to the angle in the alternate segment, which is angle ECF in this case.
step 3: Thus, the measure
angle ECF is also 20 degrees.
Step 4: Observe that the diameter CD passes through the center 0 of the circle and cuts the chord EF at its midpoint G.
Step 5: Recall that a 1ine drawn frem the center of the circle to the midpoint of chord is perpendicular
the chord.
Step 6: Therefore, triangle EOG is a right-angled triangle at G.
Stop 7: Understand that angle EOD is double the angle ECF because angle
EOD is the angle at the center whereas angle ECF is at the circunference based the Circle Theorem.
Step 8: Calculate angle EOD as times angle ECF, which
20 degrees
Step 9: Perform the multiplication: 2 20 degrees = 40 degrees. The answer is: Angle BOD measures 40 degrees. [Answer] Correct
52: Outcome Judging (Correct) Problem Understanding
[Question]: In the circle with center 0, diameter CD intersects the midpoint G of the chord EF, and the measure angle DCF is degrees. Determine the measurement of angle ECD in degrees. [Solution]:
Step 1: Identify that chord EF is intersected at its midpoint G by the diameter CD; this implies that triangle
isosceles with CG GF. Step 2: Recognize that angle DCF is given as 20 degrees and that angle DGF is alse 20 degrees because CGF is an isosceles triangle and angle CGS equals angle DGF.
Step 3: Realize that angle EOF is twice angle DGF since angle EOF is the angle subtended at the center by cherd EF, and angle DGF is the angle subtended at the circumference by the same chord (the Angle at the Center Theorem).
Step 4: Calculate angle ROF as 2 angle D&F which is 2 20 degrees 40 degrees.
40

---

Published as a conference paper at ICLR 2025
Step 5: Note that angle EOD is twice angle EOF because EC is diameter of the circle, and therefore angle EOD subtends semicircle, making it straight angle or 180 degrees.
step 6: Divide 180 degrees by the angle EOF to find angle BCD. Since BOF is 40 degrees, angle BOD is 180 degrees / 40 degrees 4.5 times ROF.
Step 7: Determine that there is mistake in the previous step: angle EOF is part of angle BOD and not the other way around. Therefore, angle EOD consists of two EOF angles plus the straight angle DOE, which is 180 degrees.
Step 8: Sum up the angles find angle BOD: angle ZOF (40 degrees) angle DOF (180 degrees) angle (40 degrees) which is 40 180 40. Step 9: Add the angles to get the measurement of angle BOD: 40 + 180
- 260 degrees.
The answer is 260 degrees. [Answer] Incorrect
53: Outcome Judging (Incorrect) Problem Understanding
[Question]: In the figure of circle o, the diameter CD intersects the midpoint G of the chord EF. The length of the chord BF is 7.5 em, which is irrelevant to be 20.0 degrees, We need to calculate the angle Mangle BOD, What is the measure of this angle in degrees? [Solution]:
angle measurements. The angle Vangle DCF is given
Step 1: Recognize that Mangle DCF is angle formed by intersecting chord (EF) and diameter (CD) of the circle.
Step 2: Acknowledge that an angle formed by a chord and a tangent (or in this case a dianeter) at the point of contact is equal to the angle in the alternate segment, which means \langle DCF is equal to angle Mangle EOF.
step Consider that triangle with CO = OD (both are radii of the circle).
a diameter, so triangle coD an isosceles
Step 4: Recall that the base angles in an isosceles triangle are equal, which means Mangle CDO - Vlagll ECo. AnStep 5: Since Vlangle CDF is given 20.0 degrees and Vlangle CDO Vangle DCO, each of these angles is also 20.0 degrees.
Step 6: Understand that Vlangle FOD is the sum of Wangle ECF and Mangle FOD (since Mangle EOF and Vlaglle FOD are adjacent angles). step 7: Since Mangle EOF is equal to Vangll DCF, which is 20.0 degrees, then Vangle EOF also 20.0 degrees.
Step 8: Calculate the measure of Mangle BOD by adding the measures of W
angle EOF and Vangle FOD, which are both 20.0 degrees.
Step 9: Perform the calculation: Wangle BOD Vlangle ECE + Vlanlle FOD 20.0 degrees 20.0 degrees.
Step 10: Conclude that Mangle EOD measures 40.0 degrees. The answer is 40.0 degrees. [Answer] Correct
54: Outcome Judging (Correct) Irrelevant Disturbance
[Question] In the figure of circle C, the diameter intersects the midpoint G of the chord EF. The length of the chord EF is 7.5 cm, which is irrelevant to our angle measurements. The angle Wangle DCF is given to be 20.0 0 degrees. We need to calculate the angle Wangle BOD. what is the measure of this angle degrees? [Solution];
Step 1: Calculate the points scored in the first 20 minutes. And points in 20 minutes = points per minute
Step 2: Calculate the points scored in the second 20 minutes. An258 nore points than in the first session 1.25 x points/minute 2.5 points/ minute
step 3: Calculate the total points scored in the 40-minute practice
session. \n2 points/minute 20 minutes 40 points\n2 \22.5 points/minute x 20 minutes 50 points
41

---

Published as a conference paper at ICLR 2025
Step Add the points scored in the first and second sessions, \n40 points 50 pcints points The answer is 90 points. [Answer]: Incorrect
55: Outcome Judging (Incorrect) Irrelevant Disturbance
[Question]: As shown in the tigure, the diameter CD of Vodot o crosses the midpoint G of chord EF, Vangle DCF - x Vangle BOD is equal to 40% /degree. what is the value of unknown variable x?
[Solution]:
Step 1: Recognize that diameter CD of circle passes through the center of the circle (point о) and bisects chord EF at point G. Step 2: Since CD is a diameter, it creates a semicircle from to F passing through C.
Step 3: Recegnize that Vaaglle EOD is the angle subtended by the dianeter and thus is a right angle \degree)) because it is an angle in a semicircle.
Step 4: Understand that triangle ROD is right triangle with Vangle ROD being 90) \degree and Vangle BOF being half of Vangle the center of the circle BOF half of the eircle (an are 1801) degree).
because is
Step 5: Since Mangle EOD is 40V\degree half of that, Step Netice that Mannle DCF and \ \angle EOF are vertical angles and hence are equal. Therefore, = Vangle DCF Vangle EOF. Step 7: Substitute the value from Step 5 into the equation for x, therefore 20 (ddereee
given, Vanglle ECE must be Vangle BOF 40\\degree / 2 - 20\ Wddgreee
The is 20 \degree [Answer] : Correct
56: Outcome Judging (Correct) Scenario Understanding
[Question] As shown in the figure, the diameter CD Vodot o crosses the micpoint G of chord EF, Mangle DCF x Mangle ECD is equal to 40V\degree. What is the value of unknown variable x?
[Solution]:
step 1: Recognize that angle Vangle EOD lies the dianeter of the cirele, which means it's central angle and the are ED intercepts is 2x\langle EOD
80Videgree.
Step 2: Since CD is a diameter, any angle formed on the circumference by diameter CD is right angle (90) (degree)))) Hence, angles Vlanlle CFR and 90 degrees each.
Maaglle CED
Step 3: The are EG is half the circumference of the circle intercepted by angle Wangle CFE, so the measure of are EG is 180) \degree since Vangge CFE is right angle.
EG because is the midpoint of EF, hence are is also 1801 idegree
Step 4: Arc GF is the same
Step 5: The entire are EF, which is the sum of ares 26 and GF, is 180 degree 180) \degree = 1d0gree - 80\\degree = 280\ 330V\eegree since the circle is 360 \degree and we must subtract are ED (80) \degree) which is not part of are EF.
Step The angle Vangle DCF an inseribed angle that intercepts are EF. The measure of an inseribed angle is half the measure of the are it intercepts.
Step 7: Calculate the measure of angle Mangle DCF which is half of the of are EF; 1/2 280 Adegree 140) idegree The answer is 140\\degree. [Answer]: Incorrect
57: Outcome Judging (Incorrect) Scenario Understanding
42

---

Published as a conference paper at ICLR 2025
[Question]: As shown in the figure, the diameter CD Vodot 0 the midpoint G of chord EF, VangleDCF 20.0, then VangleBBD is equal to ()) (VVeegree
[Solution]:
Step 1: OE and OF are the radius of the circle, 08-OF.
Step
is the midpoint of EF, EG-FG.
Step 3: Because triangle OGE and triangle OGF are congruent triangles.
step 4: so the angle BOD is equal to the angle FOG,
Step 5: because the angle FOG is the outer angle of the triangle COP, the angle FOG=angle DCF+angle CFO.
Step 6: because OC-Or (they are CFO.
radius of circles), angle Dcr=angle
Step 7: therefore the angle FOG-2* angle DCF.
Step 8: se the angle EOD = 3 angle DCF.
Step 9: Given the angle DCF=20.0 the angle E0D=60.0.
The answer is: Angle EOD measures 60.0 degrees. [Answer]; : step
58: Process Judging Original Problem
[Question] In the eircle with center c, diameter CD intersects the midpoint G of the chord EF, and the measure of angle DCF is 20 degrees. Determine the measurement of angle ECD in degrees. [Solution]:
step OE and OF are the radius
circle, so 08-OF.
Step 2: G is the midpoint of EF, sc EG-FG.
Step 3: Because triangle OGE and triangle OGF are congruent triangles.
Step 4: se the angle BOD is equal te the angle FOG.
Step 5: because the angle FOG is the outer angle of the triangle COF, the
angle roG=angle DCF+angle CFO.
Step 6: because CC-OF (they are all radius of circles), angle DCF=2+ CFO.
Step 7: therefore the angle FOG-1 angle DCF.
1.5 angle DCF. Step 9: Given the angle DCr-20.0 the angle K00-30.0. The answer is: Angle EOD measures 30.0 degrees. [Answer] : Step
Step sc the angle
59: Process Judging Problem Understanding
[Question]: In the figure of circle 0, the diameter CD intersects the mi midpoint G of the chord EF. The length of the chord RF is 7.5 cm, which is irrelevant
angle measurements. The angle VangleDDC is given to be 20.0 degrees. We need to calculate the angle MangleeED. What is the measure of this angle in degrees?
[Solution]:
Step 1: OE and OF are the radius of the circle, oz-OF.
the midpoint Step 3: Because triangle OGE and triangle OGF are congruent triangles.
step
EF, EG-FG.
Step 4: se the angle EOD is equal to the angle FOG.
Step 5: because the angle FOG is the outer angle of the triangle COF, the
angle FOG=angle DCF+angle CFO.
Step because OC-OF (they are all radius of cireles), angle DCF=angle CFO.
Step 7: therefore the angle fOG=angle DCF.
Step 8: the angle BOD angle DCF. Step Given the angle DCF=20.0 the angle K00-20.0. The answer is: Angle ROD measures 20.0 degrees. [Answer]: Step
60: Process Judging
Irrelevant Disturbance
43

---

Published as a conference paper at ICLR 2025
[Question] As shown in the figure, the diameter CD Vodot 0 crosses the midpoint
chord EF, WangleDCF degree, What is the value of unknown variable x?
Vlangle20D equal to 4011
[Solution];
Step 1: OE and OF are the radius of the circle, 08-OF.
Step is the midpoint of EF, EG-FG.
Step 3: Because triangle OGE and triangle OGF are congruent triangles.
step 4: so the angle BOD is equal to the angle FOG,
Step 5: because the angle FOG is the outer angle of the triangle COF, the angle FoG=angle DCF+angle CFO.
Step 6: because OC-Or (they CFO.
radius of circles), angle Dcr-angle
Step 7: therefore the angle F0G-4* angle DCF.
Step 8: se the angle EOD = 4 angle DCF.
Step 9: Given the angle E0D=40.0 the angle DCF 10.0. The answer is x degrees. [Answor]; : step 7
61: Process Judging Scenario Understanding
44


Reviews:
[{"review_id": "wNNlsdw6IJ", "reviewer": "Reviewer_eZPq", "summary": "This paper presents MathCheck, a benchmark/checklist for testing (M)LLMs' task generalization and reasoning robustness in mathematical problems. It utilizes GSM8k and GeoQA to create new math problems that are of different styles (e.g., irrelevant disturbance) and different types (e.g., process judging). By evaluating 26 LLMs and 17 MLLMs on this benchmark, the authors find that some models show generally robust performance (e.g., strong close models like GPT-4o), while other models (e.g., Qwen1.5-72B-Chat) show performance drop in alternative problem settings (e.g., OJ and ID). Overall for MLLMs there is large room for improvement on the multi-modal subset (GEO) of the benchmark. They include several analyses to support the reliability of the benchmark and show how the main ideas in the paper can be applied to other evaluation domains such as commonsense reasoning.", "questions": "Who are the human annotators that evaluated the quality of GPT-4T rewriting? Are they (a subset of) the authors? (That is implied from C.3, but I think it should be explicitly stated.) How did one determine whether a problem \"passed\"? I just encourage the authors to add a few sentences to better clarify the human evaluation process.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is clearly written. The examples and figures are illustrative. For example, I can easily understand what the benchmark is about by looking at Figure 1. The sections are structured well.\n\n- The task setup is innovative (i.e., the 4 x 4 matrix). It is valuable to have these different styles and formats of problems in one place for evaluation and comparison. The argument that the 4 x 4 setup is relevant to measuring generalization and robustness (at least to some degree) is sound.\n\n- The benchmark has text only and multi-modal parts, which are of interest to multiple sub-communities in the field.\n\n- The authors studied a comprehensive set of LLMs, covering models that are open and closed, big and small, generalist and specialized. And they studied several different prompting strategies, which we know matter for mathematical reasoning.\n\n- The authors do a good job attempting to ensure data quality and establish benchmark reliability. For example, they show human pass rates w.r.t. LLM-based problem generation, and they show that the accuracy of MathCheck-GSM is highly correlated to that of GSM1k, a putative high-quality private dataset. \n\n- The authors not only discuss how the evaluation method introduced in this paper can be general (applied to other domains), but they also have already done work in that direction by creating variants of \"data understanding\" and code generation tasks. This would prompt other researchers to consider how this method may be relevant to their own domains of interest.\n\nOverall, I believe that this paper and dataset are a valuable contribution to the AI for math research community.", "weaknesses": "- One weakness of the benchmark is about the difficulty of the problems. MathCheck-GSM is based on GSM8k, which is a grade-school-level benchmark that the SoTA models perform very well on. And correspondingly they tend to perform well on MathCheck-GSM. It would have been a stronger benchmark if MathCheck also included high-school and beyond problems, so that we could have a better understanding of where and how models like GPT-4o and O1-preview fail. But this is not a major weakness and can be left for future work. MathCheck already has promise for measuring progress in smaller/weaker models and multi-modal models.\n\n- A similar weakness to the above is about the diversity of problems. The seed data only contain two types of problems: simple word problems and geometry problems.\n\n- The related work section needs to be improved/expanded. Only talking about other math benchmarks is not enough. I encourage the authors to add at least two paragraphs. One can be on what people have done to improve mathematical reasoning with LLMs (training, inference, prompting, LLMs + formal tools, etc.)—the broad landscape of AI for math. Another can be similarly spirited benchmarks in other domains. For example, the use of counterfactual tasks in [1] is also for evaluating how robust and general LLMs' problem understanding and reason abilities are. \n\n[1] Wu et al. (2024) \"Reasoning or reciting? Exploring the capabilities and limitations of language models through counterfactual tasks\" NAACL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZuIjtXxUUY", "reviewer": "Reviewer_WuK6", "summary": "This paper introduces MathCheck, a new mathematical reasoning benchmark for large language models (LLMs). It aims to address the limitations of existing benchmarks, which often focus on evaluating individual tasks rather than LLM models' understanding of the problems. MathCheck incorporates carefully augmented math problems in different dimensions, including generalization and reasoning robustness.", "questions": "In Figure 6, could the accuracy drop be attributed to the increasing difficulty LLMs faced in generating or reconstructing task generation entries for more complex problems?\n\nAppendix C2 presents pass rates for different rewriting types. Would it be practical to also show the pass rates for different complexity levels? (This may not be necessary if the such questions are omitted in evaluation.)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper provides a novel approach to evaluating math problems by considering whether the models understand the problems, not just solve them. The dimension of task generalization is innovative, offering various methods to judge problem answers and providing insights into evaluating reasoning ability. The linearity and performance on different complexity levels demonstrate the potential of the new benchmark for comprehensive evaluation of models' math reasoning abilities.", "weaknesses": "The data generation pipeline with an 86% pass rate, although good, doesn’t sound ideal given the scale of samples. The paper could benefit from briefly discussing in the main text when showing the pass rate how low-quality generated questions are handled when detected.\n\nNitpicks: Line 431: It would be helpful to include references to the alleged works. Line 463: quote left mark is incorrect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sfb3LjFyXM", "reviewer": "Reviewer_GGUq", "summary": "This paper introduces MATHCHECK, a new evaluation framework for assessing mathematical reasoning capabilities in LLMs. The framework's key innovation is its comprehensive evaluation approach that tests both task generalization and reasoning robustness. The authors develop two datasets using this framework: MATHCHECK-GSM for text based mathematical reasoning and MATHCHECK-GEO for multimodal geometric reasoning. They demonstrate that while some frontier models like GPT-4o maintain strong performance across their checklist, many other models show significant performance degradation compared to their standard problem solving abilties. The authors also provide evidence that MATHCHECK correlates better with true mathematical reasoning abilities compared to other benchmarks. Finally, the authors show that the current paradiem of finetuning models for math probalem solving capabilities leads to stagnation or even degredation in other areas of math reasoning.", "questions": "- What is the pass rate for GPT-4o rewritting for the MATHCHECK-GEO benchmark?\n- The data generation pipeline has a pass rate of ~85%. It is not explicit in the paper whether the released banchmark has filtered out the questions that don't pass or left them in.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In terms of originality and quality, it introduces a novel evaluation paradigm that goes beyond traditional problem-solving assessments to comprehensively evaluate mathematical reasoning through multiple tasks and robustness tests. While there have been some previous work on creating function variations, this paper goes beyond simply rewritting the question.\nThe methodology is rigorous, with extensive empirical validation involving 43 models and a careful data generation process that achieves an 84.61% pass rate with manual validation. The authors validate their framework's effectiveness through correlation with private data and compression efficiency metrics, imlying performance on this benchmark is more robustly capturing llm math abilities.\nThe paper is reasonably clear in its presentation, with well-structured explanations of the framework and comprehensive documentation of methodology.\nThe paper is moderately siginificant. The paper provides a more reliable way to evaluate true mathematical reasoning capabilities of LLMs and is exstensible to other domains as well. In particular, the ability to evauate llms on areas other than problem solving is very useful to MCTS and LLM-as-a-judge methods.", "weaknesses": "Would recommend citing 1+ papers on functional variations benchmarks which programatically generate question variations.\n\nMATHCHECK-GSM seems to be saturated by frontier models, making it hard to get an accurate measure of their relative abilities. Constructing a MATHCHECK benchmark from MATH or another harder benchmark would be ideal.\n\nIn Appendix C, it is unclear what is meant by \"data statistic\". I assume in tables 3 and 4 it is the total count of questions for that variation type, but I'm not sure what table 5 is trying to show.\n\nRecommended:\n- Evaluate models on MATHCHECK adapted to other reasoning tasks (i.e. commonsense reasonin and code)\n\nMinor:\n- Use (M)LLM and MLLM inconsistently/unclear what the term means\n- It would be nice to have tables 1 and 2 in a plot somehow so we can better see how performance compared across evaluation categories.\n- The text in Figure 5 is a bit small and in general hard to interpret\n   - I would recommend using a bar chart or similar where it is easier to see the performance deltas (this might require aggregating over the rows)\n- Figure 6 should have an x-axis label\n- In figure 7, the pink and red are hard to differenciate (perhaps remove the color fills and only include the outlines)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kX4i7QFe1A", "reviewer": "Reviewer_n8tf", "summary": "This paper explores the question that how to comprehensively evaluate LLMs’ mathematical abilities. Considering that if a model understands a problem, it should be applied across different tasks and problem varieties, they introduce MATHCHECK. MATHCHECK is a checklist for testing task generalization and reasoning robustness. For textual and multimodal language models, they develop MATHCHECK-GSM and MATHCHECK-GEO separately utilizing MATHCHECK.", "questions": "1.\tA seed problem is expanded to include 4 math tasks and 4 problem varieties (as shown in Fig.1), resulting in a total of 4\\*4=16 problems. Therefore, for the 129 style groups in MATHCHECK-GSM, the total number of samples should be 129\\*16=2064. However, in Sec. 3.1, the number of samples in MATHCHECK-GSM is reported as 3096. Could you clarify this discrepancy?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1.\tThe paper studies an important research question from a fresh perspective, evaluating the models’ reasoning abilities across various tasks and problem varieties.\n2.\tAbundant experiments: A total of 43 models are evaluated on the proposed benchmarks, which serve as comprehensive baselines for further studies.\n3.\tThe paper is well-written and easy to follow.", "weaknesses": "1.\tAlthough the paper attempts to analyze the mathematical reasoning capabilities of existing models from multiple aspects (including PS: Problem Solving, AJ: Answerable Judging, OJ: Outcome Judging, PJ: Process Judging, OP: Original Problem, PU: Problem Understanding, ID: Irrelevant Disturbance, SU: Scenario Understanding), there are the following shortcomings. On one hand, some models (such as O1-preview) have already achieved very good performance (>90%) on these datasets and have shown extremely strong robustness, indicating that the evaluation methods proposed in the paper may not be challenging enough for the most advanced models currently available. Moreover, the reviewer did not find any new insights proposed by the evaluation method for the improvement of existing models. On the other hand, the goal of the paper is to explore whether models truly possess mathematical reasoning abilities, and the multiple aspects proposed seem insufficient: if one model performs well in all the proposed aspects, does it truly have reasoning abilities?\n2.\tGiven that these benchmarks are generated by LLMs and only verified by humans, how to ensure the diversity across different question groups? For example, with different seed questions, LLMs may tend to include similar irrelevant information when generating the “irrelevant disturbance” variety, leading to a less comprehensive evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)