MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
Authors: Kaixuan Huang
Jiacheng Guo
Zihao Li
Xiang Ji
Jiawei Ge
Wenzhe Li
Yingqing Guo
Tianle Cai
Hui Yuan
Runzhe Wang
Yue Wu
Ming Yin
Shange Tang
Yangsibo Huang
Chi Jin
Xinyun Chen
Chiyuan Zhang
Mengdi Wang
Venue: ICML 2025 poster (2025)
Keywords: mathematical reasoning
benchmark
robustness
Primary area: deep_learning->large_language_models
TL;DR: We construct MATH-P-Simple and MATH-P-Hard to benchmark LLM's math reasoning against simple and hard perturbations, and examine memorization issues.

Abstract: Large language models have demonstrated impressive performance on challenging mathematical reasoning tasks, which has triggered the discussion of whether the performance is achieved by true reasoning capability or memorization. To investigate this question, prior work has constructed mathematical benchmarks when questions undergo simple perturbations -- modifications that still preserve the underlying reasoning patterns of the solutions. However, no work has explored hard perturbations, which fundamentally change the nature of the problem so that the original solution steps do not apply. To bridge the gap, we construct MATH-P-Simple and MATH-P-Hard via simple perturbation and hard perturbation, respectively. Each consists of 279 perturbed math problems derived from level-5 (hardest) problems in the MATH dataset (Hendrycks et al., 2021). We observe significant performance drops on  MATH-P-Hard across various models, including o1-mini (-16.49%) and gemini-2.0-flash-thinking (-12.9%). We also raise concerns about a novel form of memorization where models blindly apply learned problem-solving skills without assessing their applicability to modified contexts. This issue is amplified when using original problems for in-context learning. We call for research efforts to address this challenge, which is critical for developing more robust and reliable reasoning models. The project is available at https://math-perturb.github.io/.

MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
Kaixuan Huang' Jiacheng Guo +l Zihao Lit
Xiang Ji Jiawei Ge Wenzhe Lif!
Yingqing Guo f!
Tianle Cai Hui Yuan
Runzhe Wang 11 Yue Wa 1I Ming Yin Chi Jin'
Shange Tang
Yangsibo Huang?
Xinyun Chen? Chiyuan Zhang² Mengdi Wang"
Abstract
demonstrating signs of general intelligence (Bubeck et al., 2023). As LLMS become more intelligent, the research com- munity responds by developing and adopting new bench- marks to guide the development of better models (Wang et al., 2024; Zhou et al., 2023; Liu al., 2024; Rein et al., 2023; Yan et al., 2024).
Large language models have demonstrated im- pressive performance on challenging mathemati- cal reasoning tasks, which has triggered the dis- cussion of whether the performance is achieved by true reasoning capability or memorization. To investigate this question, prior work has con- structed mathematical benchmarks when ques- tions undergo simple perturbations modifica- tions that still preserve the underlying reasoning patterns the solutions. However, no work has explored hard perturbations, which fundamen- tally change the nature of the problem so that the original solution steps do not apply. To bridge the gap, construct MATH-P-Simple and MATH- P-Hard via simple perturbation and hard pertur- bation, respectively. Each consists of 279 per- turbed math problems derived from level-5 (hard- est) problems in the MATH dataset (Hendrycks et al., 2021). We observe significant performance drops on MATH-P-Hard across various models, including ol-mini (- (-16.49% and gemini-2.0- flash-thnnking (- 12.9%) We also raise concerns about novel form of memorization where mod- els blindly apply learned problem-solving skills without assessing their applicability to modified contexts. This issue is amplified when using orig- inal problems for in-context learning. We call for research efforts to address this challenge, which is critical for developing more robust and reliable reasoning models. The project is available here.
In mathematical reasoning, the field has progressed from simpler datasets like SVAMP (Patel et al., 2021) and GSM8K (Cobbe et al., 2021a) to more challenging bench- marks such as MATH (Hendrycks et al., 2021), Olympiad- Bench (He et al., 2024), and AIME problems. Models continue strike higher performance these advanced benchmarks through stronger architectures, novel training approaches, and better training data (OpenAI, 2024; Yang et al., 2024; Shao et al., 2024; DeepSeek-Ar et al., 2025).
Nevertheless, concerns about data contamination and out-of- distribution generalization remain. Model performance can be artificially high if variants of the evaluation set leak into the training datasets if its distribution is over-represented. In these cases, the model could be merely doing pattern recognition and memorizing the solution steps without un- derstanding the underlying rationale, making it vulnerable to perturbations of the problem formulation (Zhang et al., 2024; Srivastava et al., 2024).
Several works have been proposed to quantify the robust- of reasoning models against such perturbations (Shi et al., 2023a; Mirzadeh al., 2024; Zhang al., 2024; Sri- vastava et al., 2024; Gulati et al., 2024; Zou et al., 2024), Notably, Srivastava et al. (2024) created Functional-MATHH by manually rewriting the original problems in the MATH benchmark (Hendrycks et al., 2021) into problem templates, where the numerical values in the problem statements and the corresponding answers can be varied automatically to generate infinitely-many versions that use the same math mroblem-solving skills. They observed performance drops between the modified benchmark and the original bench- mark for several state-of-thh-an language models, indicating that those models are indeed biased towards the original configurations of numerical values due some form of data contamination. However, most existing work focuses
1. Introduction
Large language models (LLMs) have achieved remarkable progress in solving many previously challenging tasks and
*Core contribution 'Princeton University "Google. Correspon- dence to: Kaixuan Huang <kaixuanh @ princeton.edu>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

MATH- Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
Overview of MATH-Perturb Benchmark
Question: Given the formola, find out the ringe of the finction (figure for illustmsiin only, not given).
ORSGINAL
SIMPLE PERTURBATION
HARD PERtURIATION
+ Sr
+ +
Presigedirs
Namber Theory
Amseer (-00, (1,(
Anewer (-x0, -1J0 (-1,00
Answer: (-xe,3 3 2, U11+ 2/7 (000
MATH-Perturb
Model Responses:
Precallculus
Counting & Protabuing
1. Simplity the function
1. Simplity the finetion
1. Simplity the function
Ini intormddie difor
+x+ 1,6++
z(x + 3) +
Gesieeyy
Algabraa
x
2. Behavier infinity
Behavior infinity
Behavier infinity
Tim
lim y F-
lim
lim y
-00
Tim
3/ -000
Split
Type
Size
I cannot be making the denominatoe in the original lonoula.
lavalid valuess
z cannor be making the denominator O the sriginal Recaala.
Invalid values
a cantot be making the denominator in the oeiginal Becnals.
Invalid values
MATH---Simple
Simple Perturhation
279
Answer: (-00, 1) (1,)o)
Answer: (-x, -1 (-1,cc))
Answer: (-00, oc)
MATH-P-Hard
Hard Porturbation
279
Carrertnoe
Currrctuea::
Carrectnan:
Figure/ 1. Left: The overview of MATH- Perturb Benchmark. Right: An example of the original problem, its simple perturbation, its
hard perturbation, and the corresponding model responses that overfit the short-cut solution. The simple perturbation to the problem non-essential, so the modified prohlem can he solved using the same method as the original problem. The hard perturbation changes the problem fundamentally and it requires more difficult prohlem-solving skills. The shortcut solution can solve the original problem and its simple perturbation but fails the hard perturbation.
on perturbing non-critical parameters (e.g., numerical val- ues) that do not alter the fundamental reasoning patterns required to solve the problem. We refer to such changes as
We summarize our contributions and key findings below:
We design and construct MATH-P-Simple (simple per- turbation) and MATH-P-Hard (hard perturbation), each consisting of 279 perturbed math problems that origi- nate from the level-5 (hardest) problems of the MATH dataset (Hendrycks et al., 2021). The datasets are curated by 12 graduate-level experts with rigorous rubrics and cross-checking workflow for quality control (Section 2).
simple perturbations. While prior studies have shown that LLMs can generalize across a range of problem variants by relying on bag-of- bugrof-hhuristics reasoning (Nikankin et al., 2024; jylin04 et al.), this form of generalization does not necessarily reflect a true understanding of the underlying principles. As result, models may still fail when faced with substantial shift in reasoning patterns.
We benchmark the math reasoning abilities of 18 LLMs (Section 3.1), and show that all the models, including ol -mii and gemini-2.0-flasse -a.---las-tthiikinng suffer significant performance drops (10%-25%) on MATH-P-Hard. This indicates these models are biased towards the original distribution of reasoning patterns and suffer from out- of-distribution effect when facing problems with hard perturbations.
In this work, we take one step forward beyond simple pertur- bations. We consider hard perturbations: while at lexical level (e.g. edit distance) the modification is similar to simple perturbations, we ensure to change the problem formula- tions fundamentally so that the original solution paths are no longer applicable to the perturbed settings; see Figure for a comparison between the two types of perturbations. A genuinely robust reasoning model that understands the underlying rationales should not only solve the modified problems under simple perturbations but also be able to judge whether the problem formulations change in a way that fundamentally alters the problems and respond accord- ingly, instead of applying the learned skills indiscriminately.
We conduct in-depth failure mode analysis (Section 3.2) and identify a new form of memorization, where the model memorizes the problem-solving techniques from the training set and blindly applies them without judging whether the modified settings are still suitable.
We investigate the influences of in-context learning (ICL) with the corresponding original unmodified problem and solution (Section 3.4), and demonstrate that ICL with original example may hurt the model MATH-P-Hard, as the models may fail to recognize the subtle differences and get misled by the demonstration.
As the capabilities of large language models continue to advance and the average-case performance continue to im- prove, the generalization abilities against hard perturbations may soon become the primary bottleneck in their real-world usages. Addressing this challenge will be critical for ad- vancing the robustness and reliability of future LLMs.
2

---

MATHI-Prrturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
Original
80
MATH-P-Simple
Accuracy (%)
MATH-P-Hard
60
40
20
aa thimking
Gemin1-2.0 0
a1 spreetaww
o1-minl minl
Gamini-2. 2.0 stash
Gamink
GPT-4o
-turbo
Claud b- 3.5
Sonnet
cleude -Opus
dB-3n Instrcct
Llama-3 1
-9b-ik
Gemma-2
mini- innt-instruct
Phica,3
Deepsnnke math
Ovren S. Math
Ri-testruct
Mathstral -Th-w
NuminaMan
-78
-130
MataMath
MAmmoTHO T2mmoN22-0 88
it
Figure Performance on MATH-P-Siman Simple, MATH-P-Hard,, and the corresponding Original problems. We observe performance degrada- tions across all models on MArd-P-Hard.
2. Dataset Curation
ing models and elicit potential memorization behaviors, we ask the annotators to make as minimal modifications as possible. Therefore, the modified problems stay close to the original problems in the text form.
Origin of the Dataset. We choose the popular MATH benchmark (Hendrycks et al., 2021), which contains chal- lenging mathematical reasoning problems sourced from American high school mathematics competitions such as the AMC 10, AMC 12, and AIME. Each problem belongs to one of the 7 subjects: Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus. Besides, each problem is labeled with a difficulty level of 1 (easiest) to (hardest). The problems may contain LaTeX and Asymptote graphics language for describing mathematical concepts and geometric figures.
Changed Answers: For both of the modifications, we guarantee that the answers to the modified problems are different from the original answer. Therefore, models cannot cheat by pattern recognition and outputting memo- rized solutions.
Quality Control. We recruited 12 annotators (PhD stu- dents) with strong mathematical backgrounds for the anno- tation task. All the annotators hold bachelor's degree in mathematics, have done researches in theoretical machine learning, and/or competed in mathematical competitions during high school.
As the state-of-the- the-t-t reasoning models can already solve MATH problems with overall accuracies higher than 90% (OpenAI, 2024; Team et al., 2024a; DeepSeek-AI et al., 2025), we opt to focus only on the hardest level-5 problems in our work, and create new benchmarks from these level-5 problems. We use level-5 problems from both the train split and the test split as the seed problems, so we are able to investigate whether language models behave differently on the two splits.
To ensure the quality of the benchmark, all the annotators are required to double-check their annotations. Each modified problem is also cols-validated by an independent annotator to make sure the answer is correct.
Additionally, we manually went through all the problems where the ol-mini's answer and the annotated answer differ and confirmed that the annotated answers are correct.
Annotation Criterion. For each problem, we modify the problem to create two variations:
(1) for MATH-P-Simple, we make simple perturbations, i... non-essential modifications to the problem, ensuring that the modified problem can be solved using the same method as the original problem.
Benchmark Overview and Statistics.
After removing several annotations that failed the quality checks, we obtained 279 pairs of modifications, where 164 examples are from train split and 115 examples are from test split. The numbers of problems in each of the 7 sub- jects are listed in Table 3. Figure shows one example of our benchmark.
(2) for MATH-P-Hard, we make hard perturbations, i.e., small but fundamental modifications to the problem so that the modified problem cannot be solved using the same method as the original problem. Instead, requires deeper math understanding and harder mroblem-solving skills.
To quantify how similar the original problem and the modi- fied problem are, first, we calculate the edit distance be- tween the modified problem and the original problem, normalized by the length of the original problem. Be- sides, we compute the cosine similarities between the em- beddings of the two problems, where we use OpenAl's
Besides, we ensure the following additional require- ments:
Minimal Edits: To test the generalization of the reason-
3

---

MATtt-Prrturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
MATH-P-Simple
Simple Perturbation
Original Problem
Hard Perturbation
MATH-P-Hard
What is the range the function
What the range of the function
What the range of the function
12
Minimal Edits
y
Minimal Edits
Solution
Original Solution
Solution
can factor numerator to [++3)e++
We can factor the numerator to
Different Problem-solving Methods
The function simpliiie to y= 3 + a 2.
e+2 The function equivalent to zr when -2 However, because
get y
Same Problem- solving Method
get (e+3)e++))
Cauchy-Sahwar Inequality,
The function is equivalent to z
2 2 2v2, (a 0).
when o -1. However, because
cannol 2, y cannot
x equal -1, y cannot
22 (z 0).
equal Therefore, the range is
equai 1. Therefore, the range is
Therefore, the range is
(-00, (-
(-o0, 1) o1)
(-00, 2v +v
Answer
Changed Answers
Original Answer
Changed Answers
Answer
(~xe, 1)1
(-o0, U (1)
(-00, 2v U 22242222200
Simple Perturbation Strategies
Hard Perturbation Strategies
Numerical modifications;
Increasing the complexity of the mathematical objects:
Asking for different relatad quantity;
Altering key numbers to large values (disabling brute-force solution);
Addinglremoving non-essential constraints;
Relaxing constraints to cover more general cases;
Modifyng essential conditions (removing simpiifynng properties);
Figure 3. Illustration of the annotation process for MATH-P-Simple and MATH-P-Hard,
text-embedding-3 3-rrge embedding model. The distribu- tions of the normalized edit distance and the cosine similari- ties are shown in Figure 4.
semantic-based retrieval methods.
Common Strategies for Perturbations.
For MATH-P-Simple, most of the problems are modified numerically without making the problems fundamentally different. Our annotators have checked these numerical mod- ifications are non-essential to the problems, so the modified problems can be solved with the same reasoning patterns. Besides, our annotators also adopt other types of changes. For example, asking for different but related quantity, adding/removing non-essential constraints, and changing a mathematical concept to its contrasting counterpart.
250
320
200
200
100
50
5.0
02
04
31
1.
80
04
28
1.
20
23
23
For MATH-P-Harde the modification strategies are more di- verse and problem-specific. general strategy is to increase the complexity of the mathematical objects involved. For example, raising the degrees of polynomials will make them harder to solve factorize. Altering key numbers to large values can make brute-force solutions infeasible. Instead, solving the problem requires deriving general formulas or applying deeper theorems rather than relying on computa- tional shorteuts. Other common strategies include relaxing constraints cover more general cases, changing essen- tial conditions so the original simplifying properties (e.g. symmetry, reducibility, linearity) no longer hold.
80 Oosine
Embodding
05
66
1.0 3.
02
Benkariss Brocdd
66
03
10
Figure 4. The distributions of edit distances and cosine similarities of embeddings between the perturbed problems and the origi- nal problems. The edit distances normalized by the lengths of the original problems. The embedding model is OpenAl's next-embedding-3--Bee
We also calculate the Mean Reciprocal Ranks (MRRs) when using the perturbed problem as the query retrieve the cor- responding original problem from the set of all 279 original problems, with the cosine similarities of embeddings being the ranking method. The MRRs of the MATH-P-Simple problems and MArH-P-Hard problems are 0.995 and 0.986, respectively, indicating that the corresponding original prob- lem and solution are likely to be retrieved using typical
3. Experimental Results
Evaluation Setting. We adopt zero-shot chain-of-thought (CoT) (Wei et al., 2022: Kojima et al., 2022) as the standard evaluation method on our benchmarks. For comparison, we also evaluate the models the set of the original 279 prob-
4

---

MATtt-Prrturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
Table Zero-shot CoT performance of the L.MSs (accuracy, %). Original refers the set of 279 unmodified problems. For the train and test columns, we report the accuracies for problems that originate from the train split and test split. respectively.
Original
MATH-P-Simple
MATH-P-Hard
Model
All
train
test
All
train
test
All
train
test
Gemmin--00-ffas--hhinnn----gpppp
92.47
92.68
92.17
91.04
87.80
95.65
78.14
77.44
79.13
ol-preview
87.81
88.41
86.96
87.81
87.80
87.83
72.40
73.78
70.43
al-mini
94.27
93.90
94.78
94.98
93.29
97.39
78.49
79.27
77.39
Geemini-200-fash-eepp
88.17
87.20
89.57
82.80
81.71
84.35
67.03
68.29
65.22
1e5pin--1.5ppr
77.78
77.44
78.26
77.42
76.83
78.26
56.63
56.10
57.39
GPT-4o
67.03
68,90
64.35
62.01
60.98
63.48
39.43
37.80
41.74
GPT-4-turbo
56,99
55.49
59.13
55,20
56.71
53.04
34.41
36,59
31.30
Claude-3.5-Sonnet
64.52
62.80
66.96
58.42
57.32
60.00
38.71
38.41
39.13
Claude-3-Opus
41.94
39.02
46.09
41.94
39.63
45.22
26.52
25.00
28.70
36,56
45.12
24.35
31.54
35.37
26.09
10.04
10,98
8.70
Gemma-2-9b-i
27.60
28.05
26.96
60
30.49
23.48
11.83
12.80
10.43
Phi-3.5-mini minn-insstrcc
26.16
27.44
24.35
28.67
26.83
31.30
14.34
15.24
13.04
Deepscck-mathh77-11
37.28
42.68
29.57
33.33
35.37
30.43
13.62
15.85
10.43
Qwen2.5 2--5-Mat--77 7
58.78
59.15
58.26
51.61
50.00
53.91
27.24
29.88
23.48
Mathstral- 7b-
36.56
43,29
26.96
36,20
42.07
27.83
14.70
16.46
12.17
NuminaMath-7B--CC
43.73
51.22
33.04
40.14
44.51
33.91
17.20
18.90
14.78
MctaMath-133B 138-VV10
21.15
32.32
5.22
7.53
7.32
7.83
5.73
4.88
6.96
MAmmoTH2 TH2-8B
12,90
11.59
14.78
17.92
17.07
19.13
7.53
10.37
3.48
lems, referred to Original in the following subsections. We do not allow any tool usage including access to code interpreter, as we find that many problems can be trivially solved by writing a brute-force search program.
For problems that originate from the test split, idcally, both the original problem and its MATH-P-Simple modification should be equally "unseen" to the model. We observe mixed results empirically from Table 1: for gemini-2.0-faa f-l.shh-eeexppp... GPT-4-turbo, Seaude-3.5-soneee the performance drops are larger than 5%, while surprisingly the performance of Phi- 3.5-mini-instruct increases. For most of the models we evaluated, the accuracies MATH-P-Simple test split are close to the accuracies on the original test split. We commend that while Srivastava et al. (2024) found a rela- rively 58% to 80% performance drop between their modified benchmark and the original MATH benchmark among a dif- ferent set of the models (the best model they tested was GPT-4), we did not observe such huge gaps for the models evaluate, which is sign of the progress in the robustness of the newly developed models against simple perturbations.
To check whether the generated answer matches the ground- truth answer, we adopt an equivalence checker follow- ing Hendrycks al. (2021); Shao et al. (2024), which first performs string normalization and then uses sympy package to check the equivalence of two mathematical objects.
3.1. Benchmarking the Performance of LLMs
We consider wide range of language models includ- ing long-CoT models, closed-sourced large models, open- sourced small models, and math-specific models. The ver- sion information of the models is deferred to Appendix A.
In Table 1, we report the overall accuracies of the LLMs on Original, MATH-P-Simple, and MATH-P- and also separately calculate the accuracies for problems that origi- nate from the train split and test split. As expected, for all the models we evaluate, we find that the performance on -ard is significantly lower than the original problems, which indicates HArd is more difficult.
Inference-time Scaling, Scaling inference-ting -ime computes has been shown to be able to boost the performance of LLMs (Wang et al., 2022; Brown et al., 2024; Wu et al., 2024; Cobbe et al., 2021b; Lightman et al., 2023). We defer the study of inference-time scaling on our benchmarks to Appendix C.5.
3.2. Failure Mode Analysis
In the meantime, most models also suffer slight perfor- mance drop on MATH-P-Simple compared to the original problems. We note that the performance drops mainly come from the train split. Generalization errors still exist for the state-of-the-arr models even when the test examples follow the exact same reasoning patterns as the training problems.
To study the generalization abilities of models against hard perturbations. we focus on the set of problems where the models fail on the MATH-P-Hard modification but correctly solve either the original problem or the MATH-P-Simple modification, which accounts for 20%-47% of the total
5

---

MATTH-Perturb:
Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
Original
MATH-P-
Ten people are sitting around round Three of them are chosen at random to give a presentation. What is the probability that the three chosen people were sitting in consecutive seats?
Problem:
Ten people are sitting around a round table. Three of them are chosen at random to give a presentation in specific order. What is the probability that the first and second presenter sitting in consecutive seats and the same time the second and third presenter were sitting consecutive seats?
Problem:
table.
Solution:
(laude3,5-Sonnee
Let's s count as our outcomes the ways to select 3 people without regard to order. There are (5) 120 ways to select any 3 people. The number of successful outcomes is the number of ways to select 3 consecutive people. There are only 10 ways to do this think of first selecting the middle person, then we take his or her neighbors.
1) First, let's understand what we're looking for:
We need 3 people in specific order
The 1st and 2nd people must he sitting next to each other
The 2nd and 3rd people must be sitting next to each other
This means all three people must be sitting consecutively around the table
<-Omitting several lines->
1
Therefore, the probability is
10
Therefore, the probability is
120
12
12
Figure 5. An example of memorization coupled with incorrect reasoning: The model incorrectly reduces the modified condition to the original condition, and then follows the original reasoning pattem. The correct answer is 1/36 We manually performed 20 repeated trials and found that Blaude-3.5-Somae has pass rate of 50%. Among the mistakes, 30% are due to the memorization issue above.
problems. For these problems, one can use the correct solutions to the easier problems as reference to better determine the failure modes on the hard problems. We defer the discussion on the other cases Appendix C.I.
for Claude-3.5-Somee spections of 20 error cases. The general failure modes due to insufficient capa- bilities are largely reduced for stronger models, making the memorization issues more prominent. As the capabilities of language models continue to ad advance, we expect the memorization issues will be the next bottleneck of reasoning mod- els, and we urge more studies on inves- tigating the generalization abilities of reasoning models against hard perturba- tions.
via manual in-
ol-mini
mem
others
First, we observe general failure modes when models are exposed to harder problems, including making mistakes in basic numerical computations and algebraic operations, making unjustified claims, missing several cases, and lack- ing certain math knowledge. These types of errors are more prominent in weaker models.
Claude-3.5 3.5-Sonnet
mem.
others
Besides general failure modes, when compare the wrong solution to the MATH-P-Hard modification with the solu- tions to the easier versions, we are able to recognize an adequate number of memorization issues. Specifically, we found that models may ignore the modified assumptions and presume that the original assumptions still hold; see Figure 5 for an example. In other cases, the models may blindly apply the techniques for the original problems without first determining whether these techniques are still suitable in the modified setting (the responses in Figure are such an example generated by GPT-4o). Interestingly, the models may even output the desired outcome of the original problem (not provided in the context) instead of the modified problem. c.g. Figure This kind of memo- rization behavior is difficult to capture with most existing type of perturbations in the literature (similar to our MATH- P-Simple) that does not require different solving strategies.
3.3. Is Mode Collapse a Problem?
We investigate whether the model makes errors due mode collapse, which means the model fails to identify the dif- ference between the perturbed problem and the original problem (seen during its training time) and the model's 's re- sponse collapses to the response to the original problem with the identical answer.
For each model, we report Namee the number of problems where the model's final answer coincides with the ground- truth answer of the corresponding original problem. For those responses, we also compute the edit distance between the full response to the modified problem and the full re- sponse to the original problem. The full result is deferred to Table in the appendix.
These issues are often coupled with other types of er- rors and pervasive among the models we evaluated. For large models, we estimate the percentages of errors caused by memorization to be 40% for ol-mini and 25%
We see that this type of failure mode accounts for less than 10% of the total errors except for three models (gemini-2.0- mash-thinkingg-xp ol ol-mini, and gemini-2.0 2emiiii-20-ffae 0-fllsh--exx) on
6

---

MATtt-Prrturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
Original
MATH-P-Hard
Problem: If [ In/2|² = 2, then find all integer values
Problem:
If [n2/9] |n/3]² 2. then find the smallest integer value of
of n.
Solution:
ol-mini:
If n is even, then we can write n = 2m for some integer m.
Let's solve the equation step-by-step:
Substituting,
<-Omitting several lines->
5. Anallyee for each value r:***
/4] - [(2m)/2J² m² m = O.
<-Omitting several lines->
#*For = 1:*8
Hence, n must be odd; can write 2m + for some integer m. Substituting,
2 2
<3
(2m + 1)2/44 [22m 1)/2]² <-Omitting several lines->
Solving gives k = 3 and & 4. Therefore, n = 10 and 13. <-Omitting several lines-> **Final *nswer:
= m.
Thus, we find m 2 and n 5 the unique integer solution.
10 and 13
Figure 6. An example of memorizing the desired outcome. The model outputs all integer values instead of the smallest integer value. The correct answer is 10 We manually performed 20 repeated trials and found that ol ol-mini has pass rate of 75%. All the 25% errors are due this specific memorization issue above.
MATH-P-Hard. After manual inspection, we find that ex- cept for only 1 problem pair where gemma-2-9b-it generates the identical answer for the original problem and the modi- fied problem, we do not see collapses of the outputs the superficial text form. Therefore, we conclude that naive recitation of the training material is not the major reason for producing the same answers. Instead, the model's responses to the modified problems often collapse to the responses to the original problems in more subtle manners, e.g. ignor- ing or failing to understand the modified assumptions; see Figure for an example.
are subtle differences between the original problems and the MATH-P-Hard modifications, the models may fail to recognize such differences and be misled by the demon- strated solutions. Accordingly, in Table 7 and Figure 7, and we calculate and visualize (I) T the num- ber of problems that initially the model fails on without the in-context demonstrations but answers correctly with the in-context demonstrations, and (2) MMgorret -wornnt- the num- ber of problems that initially the model answers correctly without demonstrations but fails on with demonstrations.
We observe that many MATH-P-Hard problems become solvable with the original problems and solutions as demon- strations. The percentages the number of total errors with- out demonstrations are larger for closed-sourced large mod- els (24%-40%) and smaller for open-sourced small models 15%), due to their differences in mathematical capa- bilities and in-context learning capabilities. However, we also observe many MATH-P-Hard problems become incor- rect with demonstrations, and the percentages are higher for large models (1 (18--40%) than small models (4%-15%). The mislcading effect counteracts the effect of in-context learning, leaving only marginal improvements (less than 5%) on the MATH-P-Hard for most models.
3.4. Does In-context Learning Help or Hurt?
In this subsection, we investigate whether using the corre- sponding original unmodified problem and solution as the one-shot in-context learning (ICI.) example will help with the modified problems in MATH-P-Sime Simple and MATH-P- Hard. We visualize the influences of ICL for three models in Figure 7 and defer the full result to Table 6.
As expected, using the original (problem, solution) pair as a one-shot in-context demonstration boosts the performance of nearly all the models on MATH-P-Simple, which should be solvable by simply applying the original solution steps to the modified setting.
As in-context learning can be viewed as a form of (test- time) training, we hypothesize that any naive fine-tuning technique with a limited distribution of problem settings will hurt the generalization of the language models against hard perturbations.
As for the MATH-P-Hard modifications, there are two fac- tors that need to be considered: (1) ICL effect: the original solutions may supply the model with desired mathemati- cal knowledge that is also helpful for solving the modified problems; (2) misleading effect: on the other hand, as there

---

MATtt-Prrturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
Original
70
100
Simple (D (0-shat)
60
60.57
15.9566999
89.
89.25
Error Rote (%)
20
Simple (ICL)
Hard (0-shot)
Hard (ICL)
21.86
+6,0920.77
50
40
37.99
-4.70
80
60
68
68:46
63.44
15
30
10
8.96
20
94
40
5,38
20
10
MATI--PSimple
MATI--P-Hard Saalmll-..-ffash--hhhhee
MATT--P-Simple
HArd
NATH-P-Simple
MATr--P-Hard Llama-3.1-0B-Inee
Figure 7. The emorr rates (%) of the models without and with the original problem and solution the in-context learning (ICL) example. For --THd--Hard, we decompose the influences of in-context learning into ICL effect (the down arrow J). which reduces the error rates, and misleading effect (the up arrow 1), which increases the error rates.
4. Related Work
the mathematical reasoning context. Our methodology falls into the category of counterfactual tests (Zhang et al., 2023; Wu et al., 2023; Zheng et al., 2023; Xie et al., 2024), where we construct perturbed problems different from the exist- ing ones to test the generalization of LLMs and examine memorization effects. Through extensive case studies, we find that LLMs can exhibit subtle forms of memorization beyond naive verbatim memorization.
Perturbations to Existing Mathematical Benchmarks. There is a considerable amount of work focusing on per- forming perturbations to existing mathematical benchmarks. Shi et al. (2023a) built GSM-IC from GSM8K (Cobbe et al., 2021b) by adding irrelevant context to the problem. GSM- Plus (Li et al.,, 2024b) creates types of variations to each of the GSM8K problem and ensure that the perturbed prob- lem is of the same difficulty. Mirzadeh et al. (2024) built GSM-Symbolic that alters the numerical values and entity names via symbolic templates of both the problems and the solution steps. Similarly, Functional MATH (Srivastava et al., 2024) is created from the MATH dataset (Hendryeks al., 2021), and Putnam-AXIOM (Gulati et al., 2024) from the Putnam Mathematical Competition.
Comparison with MATH (Shah et al., 2024). Shah et al. (2024) created MATH by combining random pairs of skills extracted from MATH (Hendrycks et al., 2021) generate harder problems that require both skills to solve. Their benchmark is mathematically harder, but there are no natural "original problems" as references. Therefore, MATH is not directly suitable for investigating the memorization effects of language models. In comparison, our MArH-P-Hard are modified directly from the problems in MATH so that the modified problems require harder skills solve. MATH- P-Hard can serve as both harder math benchmark and a testbed to investigate memorizations of LLMs.
This line of work performed simple perturbations to exist- ing mathematical benchmarks and the perturbed problems can be solved with the same solution steps and the same reasoning pattern as the original ones. In contrast, we per- formed hard perturbations to curate MATH-P-Hard, where the original reasoning pattern does not apply.
5. Conclusion
Memorization. Memorization is well-studied phe- nomenon in machine learning (Feldman & Zhang, 2020: Zhang et al., 2021; Feldman, 2020) and has become increas- ingly prevalent in large language models, due to the growing of the pretraining corpora and the scaling of the model sizes. Verbatim memorization, i.e., recitation of the training ma- terial, has significant potential consequences ranging from privacy violations (Carlini et al., 2022; Brown et al., 2022; Huang et al., 2023) and copyright infringement (Shi et al., 2023b; Karamolegkou et al., 2023; Wei et al., 2024; Chen et al., 2024) to training data security risks (Carlini et al., 2021; Nasr et al., 2023). Prior work has investigated vari- ous factors influeneing verbatim memorization, including sequence duplicates (Lee et al., 2021; Hernandez et al., 2022), model size (Tirumala et al., 2022), and sequence position (Biderman et al., 2023).
In this work, we study the generalization of large language models' math reasoning abilities against hard perturbations of the problems. We modified 279 problems from the level- 5 problems of the MATH dataset (Hendrycks et al., 2021) into MATH-P-Simple (used for control experiments) and MATH-P-Hard, via simple perturbations and hard perturba- tions, respectively. We found performance degradations of all models on MATH-P-Hard, and many of the errors can be traced new form of memorization, where the model memorizes the problem-solving techniques from the train- ing set and blindly applies them without judging whether the modified settings are still suitable. Using the original unmodified problem and solution for in-context learning can deteriorate this issue. We expect the generalization against hard perturbations to be the next major bottleneck of L.LMs' reasoning abilities and urge future work in this direction.
In contrast, we investigate the effect of memorization within
8

---

MATtt-Prrturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
Acknowledgements
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
We acknowledge Professor Jonathan Cohen (Princeton) and Andrew Tomkins (Google) for the helpful feedback and discussion. Kaixuan Huang acknowledges the support of Google PhD Fellowship. Chi Jin acknowledges the support from the National Science Foundation NSF-OAC-2411229 and NSF- 2233297...7 Mengdi Wang acknowledges sup- port by NSF grants DMS-1953686, UIS-2107044, and ONR grant 1006977. The research is also supported by Princeton Language and Intelligence (PI.)) Compute Cluster.
Anthropic. Claude-3-5-sonee 2024. URL https://www.
anthropic /ttp:///eeeeeeclaaaaa--------0-- =5tt:///oooooooooo////o///////ao son-ee.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O' Brien, K., Hallahan, E.,, Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for ana- lyzing large language models across training and scaling. In International Conference on Machine Learning, pp. PMLR, 2023.
Impact Statement
Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling, arXiv preprint arXiv:2407.2178) 2024.
This paper introduces new mathematical benchmarks for evaluating the robustness of mathematical reasoning capa- bilities of large language models (LLMs). Our benchmark can help researchers identify weaknesses and guide future improvements of reasoning models, which could drive im- provements in their applications such automated theorem proving, scientific discovery, and AI-driven tutoring.
Brown, H., Lee, K., Mireshghallah, F., Shokri, R., and Tramèr, F. What does it mean for a language model to preserve privacy? In Proceedings of the 2022 ACM conference on fairness, accountability, and transparency, pp. 2280-2280999 2022.
References
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. Sparks of artificial general intel- ligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. 127/2, 2023.
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan,
A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl,
H., Benhaim, A., Bilenko, M., Bjorck, J., Bubeck, S., Cai, M., Cai, Q. Chaudhary, V., Chen, D., Chen, D., Chen, W., Chen, Y.-C., Chen, Y.-L., Cheng, H., Chopra, P., Dai, X., Dixon, M., Eldan, R., Fragoso, V., Gao, J., Gao, M., Gao, M., Garg. A., Giorno, A. D., Goswami, A., Gunasekar, S., Haider, E., Hao, J., Hewett, R. J., Hu, W., Huynh, J., Iter, D., Jacobs, S. A. Javaheripi, M., Jin, X., Karampatziakis, N., Kaufffman,, P., Khademi, M., Kim, D., Kim, Y. J., Kurilenko, L., Lee, J. R., Lee, Y. T., Li, Y., Li, Y., Liang, C., Liden, L., Lin, X., Lin, Z., Liu, C., Liu, L., Liu, M., Liu, W., Liu, X., Luo, C., Madan, P., Mahmoudzadeh, A., Majercak, D., Mazzola, M., Mendes, C. C. T., Mitra, A., M,Mi, H., Nguyen, A., Norick, B., Patra, B., Perez- Becker, D., Portet, T.. Pryzant, R., Qin, H., Radmilac, M., Ren, I.., de Rosa, G., Rosset, C., Roy. S., Ruwase, O., Saarikivi, O., Saied, A., Salim, A., Santacroce, M., Shah, S., Shang, N., Sharma, H., Shen, Y., Shukla, S., Song, X., Tanaka, M., Tupini, A., Vaddamanu, P., Wang, C., Wang, G., Wang, L., Wang, S., Wang, X., Wang. Y., Ward, R., Wen, W., Witte, P., Wu, H., Wu, X., Wyatt, M., Xiao, B., Xu, C., Xu, J., Xu, W., Xue, J., Yadav, S., Yang, F. Yang, J., Yang, Y., Yang, Z., Yu, D., Yuan, L., Zhang, C., Zhang, C., Zhang, J., Zhang, L. L., Zhang, Y., Zhang, Y., Zhang, Y., and Zhou, x. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL httttp::/arxxv.orgggt
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert- Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Frlingsson, U., al. Extracting training data from large language models. In 30th USENIX Security Symposium 2021.
(USENIX Security 21), pp.
Carlini, N., Jagielski, M.. Zhang, C., Papernot, N., Terzis, A., and Tramer, F. The privacy onion effect: Memoriza- tion is relative. Advances in Neural Information Process- ing Systems, 35:13263 13276, 2022.
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A.. Kaiser, L., Bavar- ian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert- -osss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V. Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code. 2021.
Balaji, S., Jain, S., Saunders, W.,
14219.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,

---

MATTH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
Chen, T., Asai, A., Mireshghallah, N., Min, S., Grimmel- mann, J., Choi, Y., Hajishirzi, H., Zettlemoyer, L.,, and Koh, P. W. Copybench: Measuring literal and non-literal reproduction of sppyrigh--rotectee text in language model generation. arXiv preprint urXiv2247..70087, 2024.
A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., al. The llama 3 herd of models. arXiv preprint wrXiv:2407.217888 2024.
Feldman, V. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954-959, 2020.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint vrXiv22110.1416. 2021a.
Feldman, V. and Zhang, C. WWhat neural networks mem- orize and why: Discovering the long tail via influence estimation. Advances in Newral Information Processing Systems, 33:2881-2891, 2020.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L.,, Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021b.
Gulati, A., Miranda, B., Chen, E., Xia, E., Fronsdal, K., de Moraes Dumont, B., and Koyejo, S. Putnam- AXIOM: A functional and static benchmark for mea- suring higher level mathematical reasoning. In The 4th Workshop on Mathematical Reasoning and Al at NeurlPS' 24, 2024. URL https://openrevii net/ Forum?id-VVmlllee0ff
DeepSeek-AI, AI, Guo, D., Yang. D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang. P. Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D. Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G. Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang.
He, C., Luo, R., Bai, Y., Hu, S., Thai, Z. L.., Shen, J., Hu, J., Han, X., Huang, Y., Zhang, Y., Liu, J., Qi, L., Liu, Z., and Sun, M. OlympiadBench: A challenging benchmark for promoting agi with olympiad-leve bilingual multimodal scientific problems, 2024.
H. Ding, H., Xin, H., Gao, H., Qu, H., Li. H., Guo, J., Li, J., Wang, J., Chen, J., Yuan, J., Qiu, J., Li, J., Cai, J. L., Ni, J., Liang, J., Chen, J., Dong, K., Hu, K., Gao, K. Guan, K. Huang, K., Yu, K., Wang, L.. Zhang, L., Zhao, L., Wang. L., Zhang. L., Xu, L., Xia, L., Zhang, M., Zhang, M., Tang, M., Li, M., Wang, M., Li, M., Tian, N., Huang, P.. Zhang, P., Wang, Q., Chen, Q., Du, Q., Ge, R., Thang R., Pan, R., Wang, R., Chen, R. J., Jin, R. I.., Chen, R., Lu, S., Zhou, S., Chen, S., Ye, S., Wang. S., Yu, S., Zhou, S., Pan, S., Li, S. S., Zhou, S., Wu, S., Ye, S., Yun, T., Pei, T., Sun, T., Wang. T., Zeng, W., Zhao, W., Liu, W., Liang, W., Gao, W., Yu, W., Zhang, W., Xiao, W. L., An, W. Liu, X., Wang. X., Chen, X., Nie, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yang X., Li, X. Su, X., Lin, X., Li, X. Q., Jin, X., Shen, X., Chen, X., Sun, X., Wang, X., Song, X., Zhou, X., Wang. X., Shan, X., Li, Y. K., Wang, Y. Q., Wei, Y. X., Zhang, Y., Xu, Y., Li, Y., Zhao, Y., Sun, Y., Wang, Y., Yu, Y., Zhang, Y., Shi, Y., Xiong, Y., He, Y., Piao, Y., Wang, Y., Tan, Y., Ma,
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song. D., and Steinhardt, Measuring math- ematical problem solving with the math dataset. NeurIPS, 2021.
Hernandez, D., Brown, T., Conerly, T., DasSarma, N., Drain, D., El-Showk, S., Elhage, N., Hatfield-Dooe Z., Henighan, T., Hume, T., al. Scaling laws and inter- pretability of learning from repeated data. arXiv preprint arXiv:2205. 10488 2022.
Huang, Y., Gupta, S.. Zhong, Z., Li, K., and Chen, D. Privacy implications of retrieval-based language models. arXiv preprint arXiv:2305. 14888, 2023.
iylin04, JackS, Karvonen, A., and Can. Oth- ellogpt learned bag of heuristies. https: //www. lew.wrrnso.....o Mahelllogp--eernnee -aweeeeenddaaaa-----o-aa-eeeeeee -hea---sssurrssscss-1 Accessed on Date (2025-01-28).
Y. Liu, Y., Guo, Y., Ou, Y., Wang, Y., Gong, Y., Zou,
Y. He, Y. Xiong, Y. Luo, Y., You, Y., Liu, Y., Zhou, Y.,
Zhu, Y. X., Xu, Y., Huang, Y., Li, Y., Zheng, Y., Zhu, Y. Ma, Y., Tang, Y., Zha, Y., Yan, Y., Ren, Z. Z., Ren,
Z., Sha, Z., Fu, Z., Xu, 7., Xie, Z.,, Zhang, 7., Hao, Z.,, Ma. Z., Yan, Z. Wu, Z., Gu, Z., Zhu, Z., Liu, Z., Li, Z., Xie, Z., Song, Z., Pan, Z., Huang, Xu, Z., Zhang, Z., and Zhang. Z. DeepSeek-RI: Incentivizing reasoning capability in lims via reinforcement learning, 2025. URL https://arxins www./bgg///b//ee////oo 25011 12948.
Karamolegkou, A., Li, J., Zhou, L., and Søgaard, A. Copy- right violations and large language models. arXiv preprint arXiv:2310.13771 2023.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Ad- vances in neural information processing systems, 35: 22199-22213, 2022.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle,
10

---

MATHH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N. Deduplicating train- ing data makes language models better. arXiv preprint arXiv:2107.064999 2021.
Human Language Technologies, pp. 2080-2094, On- line, June 2021. Association for Computational Lin- guistics. doi: 10. 10.18653/w1/2221.nnee mai.:/000....aammmaaaaaaaaa 168. URL https://acllntthol.gg wwg.:g....rg/0g00 org/2221 naa.c 168/.
Li, J., Beeching. E., Tunstall, L., Lipkin, B., So- letskyi, R., Huang, S. C., Rasul, K., Yu, L., Jiang, A., Shen, Z., Qin, Z., Dong. B., Zhou, L., Fleureau, Y. Lample, G., and Polu, S. Numina- math. Chttps://github..cm hwtp://hwccccmooooottttmmooooo aimo- hiinoppoorressppieelttppppp/gggp com/ projeett-uuttta//imnppprreespppp report/numina_daaasst..oo 2024a.
Rein, D., Hou, B. L., Stickland, A. C., Petty. J., Pang, Y., Dirani, J., Michael, J., and Bowman, R. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022 2023.
Shah, V., Yu, D., Lyu, K., Park, S., Yu, J., He. Y., Ke, N. R., Mozer, M., Bengio, Y., Arora, S., et al. Ai-assisted generation of difficult math questions. arXiv preprint arXiv:2407.21009, 2024.
Li, Q. Cui, L., Zhao, X., Kong. L., and Bi, W. Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers. arXiv preprint arXiv:2402. 19255, 2024b.
Shao, Z., Wang. P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. arXiv preprint arXiv:2305.20050 2023.
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E. H., Schärli, N., and Zhou, Large language models can be easily distracted by irrelevant context. In Inter- national Conference Machine Learning, pp. 31210 31227. PMLR, 2023a.
Liu, J., Xia, C. S., Wang, Y., and Zhang. L. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024.
Shi, W. Ajith, A., Xia, M., Huang. Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, L. Detecting pretrain- ing data from large language models. arXiv preprint arXiv:2310.1677 1678 2023b.
Mirzadeh, I., Alizadeh, K., Shahrokhhh,, H., Tuzel, O., Bengio, S., and Farajtabar, M. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229 2024.
Srivastava, S., PV, A., Menon, S., Sukumar, A., Philipose, A., Prince, S., Thomas, S., et al. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. arXiv preprint arXiv:2402. 19450, 2024.
Nasr, M., Carlini, N., Hayase, J., Jagielski, M., Cooper, A. F., Ippolito, D., Choquette-Choo, C. A., Wallace, E., Tramèr, F., and Lee, K. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311.17035. 2023.
Team, G., Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang. S., et al. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.088 2024a.
Nikankin, Y., Reusch, A., Mueller, A., and Belinkov, Y. Arithmetic without algorithms: Language models solve math with a bag of heuristics. arXiv preprint arXiv:2410.21272, 2024.
Team, G., Riviere, M.. Pathak. S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models a practical size. arXiv preprint arXiv:2408.0011 2024b.
OpenAI. OpenAl ol. 2024. URL https://openai.com index/openai-ol-syeeee -wsaaii-----sssts-----lall-el
Team, Q. QwQ: Reflect deeply on the boundaries of the un- known, November 2024. URL https:///wwenll wwapoml.....iiiii io/blog/ pptp://weeppp-ppeeeeee/.ee-
Patel, A., Bhattamishra, S. and Goyal, N. Are NLP models really able to solve simple math word prob- lems? In Toutanova, K., Rumshisky, A., Zettlemoyer, L.. Hakkani-Tuu Tur. D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., and Zhou, Y. (eds.), Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics:
Tirumala, K., Markosyan, A., Zettlemoyer, L.,, and Agha- janyan, A. Memorization without overfitting: Analyzing the training dynamics of large language models. Ad- vances in Neural Information Processing Systems, 35: 38274-38290, 2022.

---

Zhang, H., Da, J., Lee, D., Robinson, V., Wu, C., Song, W., MATTH-Prrturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
Wang. X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency im- proves chain of thought reasoning in language models. arXiv preprint arXiv: 11171, 2022.
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Aruiraj. A., He, X., Jiang, Z., et al. Mmlu pro: A more robust and challenging multi-task language under- standing benchmark. arXiv preprint arXiv:2406.01574, 2024.
Zhang, C., Ippolito. D., Lee, K., Jagielski, M., Tramèr, F,, and Carlini, N. Counterfactual memorization in neural language models. Advances in Neural Information Pro-
cessing Systems, 36:39321-39366 2023.
Wei, B., Shi, W., Huang, Y., Smith, N. A., Zhang, C., Zettle- moyer, L., Li, K., and Henderson, P. Evaluating copyright takedown methods for language models. arXiv preprint arXiv:2406. 18664, 2024.
Zhao, T., Raja, P., Slack, D., Lyu, Q. et al. A careful ex- amination of large language model performance on grade school arithmetic. arXiv preprint arXiv:2405.00332 2024.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of tfthh-of- prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24831 2022.
Zheng, C.,, Zhou, H., Meng, F., Zhou, J., and Huang, M. On large language models' selection bias in multi-choice questions. arXiv preprint arXiv:2309.03R8
2023.
Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. Instruction-folloinng evaluation for large language models. arXiv preprint arXiv:2311.0777 2023.
Wu, Y., Sun, Z., Li, S., Welleck, S., and Yang, Y. An empir- ical analysis of compute-optimal inference for problem- solving with language models. 2024.
Zhou, Z.. Liu. S., Ning, M., Liu, W., Wang. J., Wong, D. F. Huang, X., Wang, Q., and Huang, K. Is your model really good math reasoner? evaluating mathematical reasoning with checklist. In The Thirteenth International Conference on Learning Representations, 2025. URL https:/ wtw.ppperrvvieeeeeeeoroiiiiiiiie nett/forum/idd-noyyHIR....
Wu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang, B., Kim, N., Andreas, J., and Kim, Y. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arXiv preprint ar XwX.:2330700277777 2023.
Xie, C., Huang, Y., Zhang, C., Yu, D., Chen, X., Lin, B. Y., Li, B., Ghazi, B., and Kumar, R. On memorization of large language models in logical reasoning. arXiv 2024.
Zou, C.,, Guo, X., Yang, R., Zhang, J., Hu, B., and Zhang, H. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. arXiv preprint arXiv:24/ 20.00383, 2024.
preprint arXiv:2410.2312
Yan, F., Mao, H., Ji, C. C.-J., Zhang, T., Patil, S. G., Stoica, I. and Gonzalez, J. E. Berkeley function calling leader- board. / bewkeley..ey...... hatpee..duuubbllooe edu/blogs/ 8
leaderbbo html,
2024.
Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., et al. Qwen2. 5-math techni- cal report: Toward mathematical expert model via self- improvement. arXiv preprint arXiv:2409. 12122, 2024.
Yu, L., Jiang, W., Shi, H., YU, J., Liu, Z., Zhang, Y., Kwok, J., Li, Z., Weller, A., and Liu, W. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https:
//openreview.o wwwp::eet/fooooe mett/ForummxdddNNNNgNNo.
Yue, X., Zheng, T., Zhang, G., and Chen, W. Mammoth2:
Scaling instructions from the web. Advances in Neural Information Processing Systems, 2024.
12

---

MATTI-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
A. Version Information of the Models
consider the following models in the paper.
long-CoT models: ol-preview, ol-mini (OpenAI, 2024), Gemini 2.0 flash thinking
closed-source models: GPT-4o, GPT-4 Turbo (Achiam et al., 2023), Gemini 1,5 Pro, Gemini 2.0 flash (Team al., 2024a). Claude 3.5 Sonnet, Claude 3 Opus (Anthropic, 2024);
open-sourced general-purpost models: Llama 3.1 (Dubey et al., 2024), Gemma (Team et al., 2024b), Phi-3.5 (Abdin al., 2024);
math-specific models: MetaMath (Yu et al., 2024), MAmmoTH2 (Yue et al., 2024), Deepseek- (Shao et al., 2024),
Qwen2.5-Math (Yang et al., 2024), NuminaMath (Li et al., 2024a), Mathtral'.
Table 2. Version information of the models
Model
Provider
Version/L.ink
Gemmin-20-ffash--tiininng
Google DeepMind
2024-12-19
-prreview
OpenAl
2024-09-12
ol-mini
OpenAl
2024-09-12
2.mfni-2.0-fassh-xpp
Google DeepMind
2024-12-11
Gemini-1.5-pro
Google DeepMind
semini-1.5-pro-000
GPT-4o
OpenAl
2024-08-06
GPT-4-turbo
OpenAl
2024-04-09
Claude-3.5-sonnet
Anthropic
2024-10-22
Claude-3-opus
Anthropic
2024-02-29
Llama-3 3.1-88. 1-1-8B-Innttree Instructt
Open- Open-Sourced
Gemma-2-9b-it
Open- Opene-ourced
Phi-3.5-mini-instrutt
Open-Sourced
Deepseek-math-7bbbl
Open-Sourced Open-Sourced Open-Sourced Open-Sourced
Qwenn.55-Math-7-Innst Instruct
Mathstral-7b-vO.1
NuminaMath-7BBCol
MetaMath- -3B-V1.0
Open-Soue Sourced
MAmmoTH2-8B
httpps//wweeiiiiaeccccoooott/eea Open -ourred
B. Benchmark Statisties
Table 3. Number problems corresponding to different subjects.
Subject
Number (Percentage)
Prealgebra
35 (12.54 %)
Algebra
79 (28.32 %)
Numher Theory
36 (12.90 %)
Counting Probability
(13.62 %)
Geometry
21 (7.53 %)
Intermediate Algebra
48 (17.20 %))
Precalculus
22 (7.89 %)
Total
279
Mathtral https://mistral.ce htt.s//rra....e//eeesssstsstaal
13

---

MATTI-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
Table 4. Number and percentage of the models' responses that belong cach of the four categories.
Model
Case
Case II
Case III
Case IV
212 (75.99 %)
5 (1.79 %)
6 (2.15 %)
56 (20.07 %)
-p-eevwww
194 (69.53 %)
10 (3.58 %)
8 (2.87 %)
67 (24.01 %)
ol-mini
218 (78.14 %)
4 (1.43 %6)
(0.36 %)
56 (20.07 %)
Gemini-2.0-ffash-exp
176 (63.08 %)
11 (3.94 %)
11 (3.94 %)
81 (29.03 %)
Gemini-1.5-pro
145 (51.97 %)
28 (10.04 %)
13 (4.66 %)
93 (33.33 %)
GPT-4o
94 (33.69 %)
56 (20.07 %)
16 (5.73 %)
113 (40.50 %)
GPT-4-turbo
81 (29.03 %)
72 (25.81 %)
15 (5.38 %)
111 (39.78 %)
Claude-3.5-Sonnet
88 (31.54 %)
56 (20.07 %)
20 (7.17 %)
115 (41.22 %)
Claude-3-Opus
49 (17.56 %)
99 (35.48 %))
25 (8.96 %)
106 (37.99 %)
Llama-3, 3,1-88B-nnttrur
21 (7.53 %)
137 (49.10 %)
(2.51 %)
114 (40.86 %)
Gemma-2-9b-it
22 (7.89 %)
164 (58.78 %)
11 (3,94 %)
82 (29.39 %)
Phi-3.5-mini-instruce
22 (7.89 %))
161 (57.71 %)
18 (6.45 %)
78 (27.96 %)
Decpscck-math-7b44
25 (8.96 %)
138 (49.46 %)
13 (4.66 %)
103 (36.92 %)
Qwen2.5-Math-7B3 Instructt
(21.86 %)
70 (25.09 %)
15 (5.38 %)
133 (47.67 %)
Mathstral-7b-v0.1
28 (10.04 %)
136 (48.75 %)
13 (4.66 %)
102 (36.56 %)
NuminaMath-7B-Co.
39 (13.98 %)
118 (42.29 %)
9 (3.23 %)
113 (40.50 %)
SetaMath-13B-VV
(2.15 %)
199 (71.33 %)
10 (3.58 %)
64 (22.94 %)
MAmmoTTH2-BB
(3.23 %)
201 (72.04 %)
12 (4.30 %)
57 (20.43 %)
C. A Additional Experimental Results
C.1. Categorizing Model Responses Across Problem Variations
Recall that for each problem, we have a MATH-P-Simple modification which can be solved using the same method as the original problem, and a MATH-P-Hard modification which requires more difficult problem-solving skills. Therefore, there 8 possible cases regarding the correctness of the model's 's responses to the three problems. Modulo the fluctuations the model's correctness among the MATH-P-Sime Simmple variations, we summarize the model's responses into the following cases:
Case I: at least of the original problem and the MATH-P-Simple modification is solved correctly, and the MATH-P-Hard modification is also solved correctly.
Case II: both the original problem and the MATH-P-Simple modification are solved incorrectly, and the MATH-P-Hard modification is also solved incorrectly.
Case III: both the original problem and the MATH-P-Simpi modification are solved incorrectly, but the MArH-P-Hard modification is solved correctly.
Case IV: at least one the original problem and the MATH-P-Simple modification is solved correctly, but the MATH-P- Hard modification is solved incorrectly.
For each of the models, we calculate the percentage of the responses in Table 4. As expected, stronger models have a higher percentage of Case responses and lower percentage of Case II responses. Interestingly, the percentages of Case III responses are small (less than 10%) but non-zero, where the models cannot solve the easier variants but can solve the hard variant correctly. After manual inspection, we found that this is due to the misalignment between the models' capabilities and the annotators' perception of the difficulties math problems.
C.2. Is Mode Collapse a Problem?
We provide Table 5 to support Section 3.3.
14

---

PATturrerturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
Table 5. The number of errors with answers that match the corresponding original answers. The edit distances are normalized by the length responses to original problems.
MATH-P-Simie Simple
MATH-P-Hard
Model
Num. Errors
Normalized Edit Distance
Num. Errors
Normalized Edit Distance
Ylsame
Thoral
percentage
min.
avg.
Tlsame
Thooal
percentage
min.
avg.
Gemini-2.0- Gemini-20-laaattiiinnn---ag
2
25
8.00
0.553
0.611
0.668
16.39
0.508
0.679
0.976
ol-preview
1
34
2,94
0.652
0.652
0.652
77
6.49
0.729
1.07
1.89
ol-mini
0
14
0
N/A
N/A
N/A
60
15.00
0.559
14.7
126.0
Gemini-2.0-flsbb-ee
4
48
8.33
0.644
0.82
1.09
14.13
0.546
1.1
1.76
Gemini-L.5-pro
5
63
7.94
0.472
0.751
121
9.09
0.257
0.866
1.58
GPT-4o
3.77
0.709
0.773
0.937
0.489
0.777
1.2
GPT-4-turbo
5
125
4.00
0.621
0.74
9.29
0.636
0,932
Clande-3.5-Somnc
6
5.17
0.509
0.729
7.60
0,461
0.741
1.92
Claude-3-Opus
3
162
1,85
0.355
0.485
0.614
15
205
7.32
0.463
0.841
1.54
Llama-3.1-88 --nt-stt-BB-
13
191
6.81
0.595
0.901
1.99
18
251
7.17
0.618
0.946
2.7
Gemma-2-9han
3
202
1.49
0.361
0.506
0.716
246
2.85
0.662
1.08
Phi-3.5-mini-instuue
8
199
4.02
0.427
0.61
0.832
239
5.02
0.289
0.754
1.69
Deepseck-math-7h--
4.84
0.189
0.423
0.676
4.56
0.121
4.24
Twwon.5-MMath-BB-maee
6
135
4.44
0.376
0.591
0.813
10
203
4.93
0.273
1.01
4.91
Mathstral-7h-vkh
II
178
6.18
0.0989
0.645
0.964
13
238
5.46
0.105
0.586
0.984
NuminaMath-7B-CCo
12
167
7.19
0.241
0.743
1.62
14
231
6.06
0.204
1.04
2.22
MetaMath-
13.-
13
258
5.04
0.27
0.55
0.748
14
263
5.32
0.509
0.982
2.83
MAmmoTH2-8B
5
229
2.18
0.00214
0.666
1.25
258
3.49
0.708
0.822
1.04
15

---

MATHH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
C.3. The Effect of In-Context Learning
In Table 6, we report the performance of in-context learning (ICL) with the corresponding original (unmodified) problem and solution the in-context learning example. Furthermore, decompose the influences MATH-P-Hard into the ICL effect and the misleading effect in Table 7 and visualize the influences for representative models in Figure 8, Please refer to Section 3.4 for the full discussion.
Table 6. Performance comparisons without and with the original problem and solution the in-context learning example.
MATH-P-Simple
MATH-P-Hard
Model
Original (0-shot)
zero-shot
ICL w. original
zero-shot
ICL w. original
Geminn--.0ffushhttiinin----
92.47
91.04
94.62
78.14
79.21
ol-preview
87.81
87.81
91.40
72.40
74.19
ol-mini
94.27
94.98
94.98
78.49
78.49
Gemini-2.0-fash--xxp
88.17
82.80
89.96
67.03
67.38
Gemini-1.5-pro
77.78
77.42
88.17
56.63
60.57
GPT-40
67.03
62.01
77.06
39.43
43.01
GPT-4-turbo
56.99
55.20
69.89
34.41
39.07
Claude-3.5-Sonnet
64.52
58.42
83.15
38.71
49.46
Claude-3-Opus
41.94
41.94
68.10
26.52
33.33
Llama-3
3.1-8BBInsttruc
36.56
31.54
36.56
10.04
10.75
Gemma-2-9b-it
27.60
27.60
42.65
11.83
14.34
Phi-3.5-mini-instruct
26.16
28.67
36.92
14.34
14.34
Deepseek-math-77-rl
37.28
33.33
45.52
13.62
15.41
Qwen2.55-Mtt-77-Innse
58.78
51.61
56.99
27.24
26.88
Mathstral-7b-v0.11
36.56
36.20
48.39
14.70
16.49
NuminaMath-7BBCoT
43.73
40.14
47.31
17.20
17.20
MctaMath-I3B-V1.0
21.15
7.53
11.11
5.73
3.58
MAmmoTH2-8B
12.90
17.92
31.18
7.53
5.73
16

---

MATtt-Prrturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
Table 7. Effects of in-context learning (ICL) with original example on MATH-P-Hard The percentages of nncorrect - wrong) are normalized by the number of errors with ICI. while the percentages of v (wrong - correct) are by the number of errors without ICL. Model
num errors (zero-shot) num. errors (ICL w. original)
nncorrect -> wrong) ro(wrong -> correct)
Demmini-2.0fllsst- -hink---iiik
61 (21.86 %)
58 (20.79 %)
(29.31 %)
20 (32.79 %)
ul- -l-preview
77 (27.60 %)
72 (25.81 %)
21 (29.17 %)
26 (33.77 %)
ol-mini
60 (21.51 %)
60 (21.51 %6
24 (40.00 %)
24 (40.00 %)
Gemini-2.0 f.pp
92 (32.97 %)
91 (32.62 %6)
30 (32.97 %)
31 (33.70 %)
Gemini-L.S-pro
121 (43.37 %)
110 (39.43 %)
27 (24.55 %)
38 (31.40 %)
GPT-4o
169 (60.57 %)
159 (56.99 %)
31 (19.50 %)
41 (24.26 %)
GPT-4-turbo
183 (65,59 %)
170 (60.93 %)
33 (19.41 %)
46 (25.14 %)
Claude-3.5-Someet
171 (61.29 %)
141 (50.54 %)
27 (19.15 %)
57 (33.33 %)
Claude-3-Opus
205 (73.48 %)
186 (66.67 %)
35 (18.82 %)
54 (26.34 %)
251 (89.96 %)
249 (89.25 %)
18 (7.23 %)
20 (7.97 %)
Gemma-2-9b-it
246 (88. 17 %)
239 (85.66 66.66 %)
14 (5.86 %)
21 (8,54 %)
Phi-3.5-mimii-nnstuuctt
239 (85.66 %))
239 (85.66 %)
17 (7.11 %)
17 (7.11 %)
Deepseck-math-7b-l
241 (86.38 %)
236 (84.59 %)
19 (8.05 %)
24 (9.96 %)
Qwen2.5-Math-7B-Inste
203 (72.76 %)
204 (73.12 %)
32 (15.69 %)
31 (15.27 %)
7o-hsttral-b--O..
238 (85.30 %)
233 (83,51 %)
19 (8.15 %)
24 (10.08 %)
NuminaMath-7B-CoT
231 (82.80 %)
231 (82.80 %)
23 (9.96 %)
23 (9.96 %)
MetaMath- 13B-V1.0 V1.0
263 (94.27 %)
269 (96.42 %)
11 (4.09 %)
5 (1.90 %)
MAmmoTH2-88
258 (92.47 %)
263 (94.27 %)
12 (4.56 %)
(2.71 %)
30
Original
25
0.0%
5.9%
Simple (Q-shot)
21.51
+8.50 21.51
60.57
+11:11 58.00
25
Simple (ICL)
4.9%
Eiror Rale (%)
20
Hard (ICL)
Hard (0-shat)
21.95
+6.09 20.79
60
37.00
39.6% &
-1470
15
-5.00
30
10
40.0%
1.96
0.
0.0%
20
22.94
3.38
6
5.02 3.02
P.rple
MATH Hard
0
MATH-F-Smpp
MATH-PHard
MATH PP Simple
MATH P Hard
Gamiin-20--laah-hiiiii
o1-mini
GPT-4o
70
17.5% 1
100
0,8%
1
100
2.8%
61.20
89.96
+6.45
80.17
02.0
85 66
20,95 10.88%.o
Error Rate (5%
500
59.5%
+8.69
50.54
80
7.3%
72.40
53.44
40
22.43
57.35
30
20
16,85
10
MATH-R.S
MATH--PHaadd
0
MATH-P-Simple
MATH.P-Had
0
MATH-P mmll
Maad-P-Haadd
C.suda-3..SSomll
Liema-3,1188 1-st-na
Gemma-2-9b-it
2.1%
80
0.5% f
86.55
86.38
+6.01
04.59
72.76
+11.47 73.12
80
18.3%
Error Rate (%)
66.67
60
11.1%
60
48.39
43.01
40
20
20
MATH.F-Simple
MATH-P-Haard
MATH---
mmple
MATH+P P-ard
Deepseak-matt-7b-il
Gwwen2.-Mathh---innttii
Figure 8. The error rates (%) of the models without and with the original problem and solution as the in-context learning (ICL) example. For MATH-P-Hard, we decompose the influences of in- co-tont learning into ICL effect (the down arrow J), which reduces the error rates, and misleading effect (the up arrow 1), which increases the emrrr rates.
17

---

MATHI-Perturb:
Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
C.4. Ablation Study: Example
In-Context Learning with the Original Example In-Context Learning with Random
In Table 8, compare (1) the performance of one-shot in-context learning with the corresponding original unmodified (problem, solution) with (2) the performance of ICL with random problem and solution chosen from the same category the query problem. We find that ICL with the original problem and solution consistently outperforms ICL with random example except for only one case.
Table Performance comparisons without and with the original problem and solution as the in-context learning example.
MATH-P WATH-P-Simple
MATH- P-Hard
Model
ICL original
ICL (random)
ICL original
ICL (random)
ol-mini
94.98
92.83
78.49
75.99
Gemini-1.5-pro
88.17
75.99
60.57
51.97
GPT-4o
77.06
63.08
43.01
37.28
GPT-4-turbo
69.89
57.71
39.07
32.62
Claude-3.5-Somee
83.15
62.37
49.46
40.86
Claude-3-O0 O-uus
68.10
45.52
33.33
23.66
-lama-3.1-88-Insttee
36.56
28.32
10.75
6.45
Gemma-2-9b-ir
42.65
27.60
14.34
12.90
Thi-3.5-mini-insteee
36.92
20.07
14.34
10.39
Deepseek-math-7lb-.
45.52
34.41
15,41
13.26
Qwen2.55Maut-77B-nnstte
56.99
55.20
26.88
26.16
Mathstral-7b-v0l
48.39
24.37
16.49
8.96
NuminaMath- 7B- 7B-CoT CoT
47.31
24.73
17.20
10.04
MetaMath-13B-VV
11.11
8.60
3.58
5,38
MAmmoTH2-8B
31.18
3.94
5.73
2.15
C.5. Inference-time Scaling Behaviors
In this subsection, we investigate the inference-time scaling behaviors of LLMS on our benchmarks. We compute the pass@k metric following Chen al. (2021). Specifically, for each problem, generate N solutions independently, and compute the @ass@k metric via the following formula for each 1 < k N:
(VE9)
pass@k-
E Erroblem
where c is the number of correct answers of the n runs.
We also compute the performance of self-consistency (Wang et al., 2022), a.k.a., majority voting, where for each k we randomly sample & responses from the runs and get the majority- voted answer. We report the average and standard deviation among 5 random draws. We only evaluate three models: ol-mini. -Jamm-331--BB-mme Instruct. For Llama-3. 1-88--BBon I-8trunttt and Owwen.55Maahh- Instrucct,, choose N 64, while for ol ol-mini set N 8. The results are plotted in Figure 9.
and Qwen2.5-Math-7TB
Owenn-Qenn2 S-att Bi-ssscc
aimini
20. Original spaenaad)
MATTH land 19C)
46.
Criginal
90.5
50-
833
20
16x
00.34
Figure 9. The effect of scaling up inference-time compute. We report pass @asss@ k and self-consistency (SC) accuracies for different numbers of solutions k.
18


Reviews:
[{"review_id": "B1SJ5GU98p", "reviewer": "Reviewer_Bq4D", "summary": "The paper constructs MATH-Perturb to evaluate the math reasoning generalization of LLMs under simple and especially hard perturbations. The authors create MATH-P-Simple (279 problems) and MATH-P-Hard (279 problems) datasets from level-5 problems in the MATH dataset. Experiment results on 18 LLMs show significant performance drops on MATH-P-Hard, indicating they struggle with hard perturbations and are biased toward the original reasoning patterns. Failure mode analysis reveals that many of the errors can\nbe traced to a new form of memorization, where LLMs memorize the problem-solving techniques from the training set and blindly apply them without judging whether the modified settings are still suitable.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback on our work! **We evaluated 12 new long-CoT models that appeared near or after the ICML submission deadline.** The results [here](https://anonymous.4open.science/r/icml2025_13579_math_perturb_rebuttal-C0F8/) show no sign of saturation on MATH-P-Hard. We would like to provide detailed responses below:\n\n---\n\n> **Q1**: … This kind of memorization might be normal, simply because the model lacks the ability to solve more difficult problems. For students, they may master easy questions but be unable to solve difficult ones.\n\n**A1**: \n**We believe hard perturbation is a valid and important setting, especially when reasoning models are deployed for end users or agentic uses**. It is common for end users or agent systems to make slight changes to the inputs that fundamentally alter the questions. If the model fails to identify the changes and applies the memorized solutions, it may have bad consequences, even though one can argue this kind of memorization is normal. We hope our benchmark can inspire future work in this direction.   \n\n---\n> **Q2**: If it is possible to design problems that are equivalent in question type and difficulty to MATH-P-Hard but very different from the original questions (e.g. with a large edit distance). And if the models perform better on these problems than on MATH-P-Hard, it may indicate that they habitually use the solution approach of the original questions when solving MATH-P-Hard.\n\n**A2**: Thank you for the insightful suggestion! **We designed MATH-P-Hard to mimic the original problem formulations to deliberately elicit memorization behaviors of the models.** Designing problems that are equivalent in question type and difficulty to MATH-P-Hard but with large edit distances will lead to a good subset for *isolating* the memorization effect. We agree with you that if a model solves this type of problem correctly but fails on MATH-P-Hard, we can claim that the model possesses the skills to solve the harder problem but habitually uses the memorized approach due to the superficial similarity of the problem formulation to the original one. This is an interesting **follow-up direction**.\n\n---\n> **Q3**: Perhaps it would be beneficial to discuss or analyze some benchmarks[1] that focus on perturbations at the level of mathematical problem-solving tasks. [1] Zhou et al. Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist. In ICLR 2025.\n\n\n**A3**: Thank you for the pointer! **We will cite and discuss the paper in our next revision.**\n\n- The paper establishes a pipeline to generate perturbations of problems in 4*4 = 16 different variants, featuring 4 task generations (problem-solving, answerable judging, outcome judging, and process judging) and 4 reasoning robustness modifications (original problem, problem understanding, irrelevant disturbance, and scenario understanding). They focused on simpler GSM8K dataset and multimodal geometry datasets. \n\n- Our paper focuses on dissecting “perturbation” into simple perturbations and hard perturbations, and investigates the proportion of the failures that are due to memorization. We selected MATH level-5, which is the harder high-school competition level. \n\n---\n> **Q4**: When the model fails to solve difficult problems, is it simply due to the model's insufficient capabilities, or should it be attributed to the model's excessive memorization?\n\n\n**A4**:  **We have discussed the failure modes in Section 3.2.** In short, the performance drops in MATH-P-Hard can be attributed to both insufficient capabilities to handle harder problems and memorization issues. The two failure modes often couple with each other. For stronger models, the general failure modes due to insufficient capabilities are largely reduced, making memorization issues more prominent.\n\n---\n> **Q5** I think there may be a lack of some more in-depth analysis, based on the existing benchmarks in the current community. For example, could it provide ideas or clues on how to achieve easy-to-hard generalization?\n\n\n**A5**: We agree with you that the community currently lacks a more in-depth analysis of the existing benchmarks, which **motivates** our work. \n\nThere are some works focusing on easy-to-hard generation using MATH dataset (e.g. train on level 1-3 problems and test on level 4-5 problems) [1]. However, for this setting, there aren’t paired data with similar problem statements but different solutions and difficulty levels. **We believe our benchmark can serve as a testbed for future studies on easy-to-hard generalization and scalable oversights.** \n\n- [1] Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision. Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, Chuang Gan\n----\n\n**We sincerely hope that our responses can address your concerns, and we would greatly appreciate it if you would consider raising your score of our work to a *clear accept* given the responses.**", "claims_and_evidence": "I think the generalization of mathematical reasoning abilities, and even general reasoning abilities, which this paper focuses on, is worthy of exploration. The authors summarize the situation where LLMs can answer the original questions correctly and also handle simple variations of the original questions (such as variable substitution), but fail to solve the hard variations, as a **new form of memorization**. \n\nI have some reservations about this claim:\n- If it is considered a new form of memorization, it might be categorized as memorization of problem-solving abilities. It has learned the core abilities for such problems and can solve various variations of them, but is at a loss when facing more difficult problems or may still apply previous habitual assumptions.\n- This kind of memorization might be normal, simply because the model lacks the ability to solve more difficult problems. For students, they may master easy questions but be unable to solve difficult ones.\n- If it is possible to design problems that are **equivalent in question type and difficulty to MATH-P-Hard** but very different from the original questions (e.g. with a large edit distance). And if the models perform better on these problems than on MATH-P-Hard, it may indicate that they habitually use the solution approach of the original questions when solving MATH-P-Hard.", "methods_and_evaluation_criteria": "Yes, I think this benchmark is valuable.", "theoretical_claims": "Yes, please refer to \"Claims And Evidence\" for details.", "experimental_designs_or_analyses": "Yes, the experimental design is generally reasonable.", "relation_to_broader_scientific_literature": "Perhaps it would be beneficial to discuss or analyze some benchmarks[1] that focus on perturbations at the level of mathematical problem-solving tasks.\n[1] Zhou et al. Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist. In ICLR 2025.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "** Strengths**:\n- This article is well-organized and easy to understand.\n- The ideas discussed are of great value.\n- The constructed dataset is also valuable, which is conducive for the community to compare different difficulty variants of the same or similar problems.\n\n** Weaknesses and Questions**:\n- When the model fails to solve difficult problems, is it simply due to the model's insufficient capabilities, or should it be attributed to the model's excessive memorization?\n- I think there may be a lack of some more in-depth analysis, based on the existing benchmarks in the current community. For example, could it provide ideas or clues on how to achieve easy-to-hard generalization?", "other_comments_or_suggestions": "None", "questions_for_authors": "** Weaknesses and Questions**:\n- When the model fails to solve difficult problems, is it simply due to the model's insufficient capabilities, or should it be attributed to the model's excessive memorization?\n- I think there may be a lack of some more in-depth analysis, based on the existing benchmarks in the current community. For example, could it provide ideas or clues on how to achieve easy-to-hard generalization?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qmhMd0cauK", "reviewer": "Reviewer_aCm6", "summary": "This paper investigates the robustness of mathematical reasoning models when faced with out-of-distribution problem modifications. The authors introduce MATH-P-Simple and MATH-P-Hard, two benchmark datasets that test models under simple and hard perturbations, respectively. Their evaluation reveals significant performance drops on MATH-P-Hard, highlighting that models tend to blindly apply memorized problem-solving skills without assessing their applicability to modified contexts.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1** The authors evaluate instruction-tuned MLLMs …\n\n**A1**. We would like to first clarify that our dataset only contains textual input, and we evaluated on text-only LLMs, not Multimodal LLMs. \n\n---\n> **Q2**: A key limitation of this study is that the authors do not evaluate RL-based models...\n\n**A2**: **We did evaluate RL-based models.**\n\n- (1) We evaluated 3 long-CoT models: Gemini 2.0 flash thinking, o1-preview, and o1-mini. These models are believed to be tuned with RL, using similar techniques as Deepseek-R1. Please note that R1 (released on 2025/01/20) is considered concurrent work.\n- (2) We additionally provided the evaluation results on the 12 new long-CoT models [here](https://anonymous.4open.science/r/icml2025_13579_math_perturb_rebuttal-C0F8), including R1. \n- (3) Besides the long-CoT models, please note that Deepseek-Math-7B-RL and Qwen2.5-Math-7B-Instruct both underwent an RL tuning stage.\n---\n> **Q3**: Essential References Not Discussed: Perturbation benchmark has been explored by DYNAMATH [2].\n\n**A3**: **We have already discussed and properly cited the DynaMath in our submission. The contributions of the two benchmarks do not conflict with each other:** \n - DynaMath proposes 7 different perturbation types but most of them fall into the category of simple perturbations. In contrast, our paper studies hard perturbations.\n- DynaMath focuses on multimodal mathematical reasoning settings and evaluates MLLMs, while our paper focuses on text-only settings. \n\n---\n> **Q4**: it is unclear what distinguishes their findings from prior observations, such as those in [1], which already highlight that models memorize solution steps without truly understanding the underlying reasoning.\n\n**A4**: **Our contributions are orthogonal to [1], and our findings are different from GSM1K [1].** Specifically:\n\n- (1) First of all, **the GSM1K benchmark [1] is already saturated**, with models achieving over 95% accuracy (see the online leaderboard in https://scale.com/leaderboard/math). **The leaderboard was officially deprecated in January 2025** by Scale AI, and the benchmark does not contain newly released RL-tuned models, such as Deepseep-R1. \n\n- (2) The mechanisms of the memorizations are different: the authors of [1] stated their contribution as *“To measure the existing benchmark contamination on GSM8k, we created GSM1k, a held-out benchmark designed to match the difficulty and structure of GSM8k.”* In contrast, in our evaluation results, we showed that **naive memorization of the contaminated data is not a significant issue for the newly developed models**, and these models are already capable of generalizing to simply-perturbed problems. Instead, the memorization effects on MATH-P-Hard are caused by failing to recognize the essential differences between the perturbed problems and the original ones. \n\n---\n> **Q5**: The authors should explicitly define what they mean by \"problem-solving techniques\" in line 432.\n\n**A5**: **We believe the term is already clear from the context.**\n\n Cautions should be taken when giving a definition to such an abstract concept. By problem-solving techniques one can mean “the procedure of applying mathematical knowledge and mathematical operations” for solving a problem. This roughly corresponds to the steps of the chain-of-thought solution. Similar concepts are utilized in [[1]](https://arxiv.org/abs/2405.12205) and [[2]](https://arxiv.org/abs/2407.21009). \n\n---\n> **Q6** ... further research is needed to determine whether alternative training paradigms, such as RL or perturbation-based fine-tuning, can overcome these issues. ... an ablation study comparing models trained on perturbed vs. non-perturbed datasets would be necessary.\n\n**A6**: \nWe agree that thorough studies on the effects of RL and perturbation-based fine-tuning datasets are necessary. This is an important follow-up but is **outside the scope of this work**. As a benchmark paper, our goal is to curate a high-quality dataset and identify new  memorization issues as a current limitation of reasoning models, and encourage future studies.\n\n---\n> **Q7** ... Does exposure to perturbed training data actually improve generalization, or does it reinforce memorization biases?\n\n**A7** Adopting the same technique to curate a training dataset with hard perturbation is a promising future direction. However, **to ensure high quality, our benchmark was curated by expert-level annotators,** which is *too costly* for constructing a large-scale training dataset. We encourage the community to explore hybrid methods to synthesize training datasets with both state-of-the-art LLMs and expert-level annotators. **Again, this is outside the scope of this work.**\n\n---\n\n**Given your review, we believe there were major misunderstandings on our work. We sincerely hope that our responses can resolve the misunderstandings and address your concerns, and we would greatly appreciate it if you would like to re-evaluate our work given the responses.**", "claims_and_evidence": "The MATH-P-Hard introduces hard perturbations that alter the reasoning path, thereby increasing problem-solving difficulty. The authors evaluate instruction-tuned MLLMs and demonstrate that these models memorize problem-solving techniques from the training set rather than genuinely adapting to problem modifications. However, prior work, such as DeepSeek-R1 [1], suggests that reinforcement learning (RL) techniques can help models reduce memorization and explore reasoning paths more effectively. A key limitation of this study is that the authors do not evaluate RL-based models, leaving an open question regarding their effectiveness in addressing memorization biases in mathematical reasoning tasks.\n\nPerturbation benchmark has been explored by the work [2].\n\n[1]DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.\n[2] Chengke Zou et. al., DYNAMATH: A DYNAMIC VISUAL BENCHMARK FOR EVALUATING MATHEMATICAL REASONING ROBUSTNESS OF VISION LANGUAGE MODELS, ICLR2025.", "methods_and_evaluation_criteria": "The authors claim to have discovered a novel form of memorization, but it is unclear what distinguishes their findings from prior observations, such as those in [1], which already highlight that models memorize solution steps without truly understanding the underlying reasoning.\n\nThe authors should explicitly define what they mean by \"problem-solving techniques\" in line 432.\n\n[1] Zhang, H., et al.,  A careful examination of large language model performance on grade school arithmetic. arXiv 2024.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "If the authors use the same technique and data engine employed for MATH-P-Hard generation to create a training dataset, and then fine-tune MLLMs on this dataset, the issue of memorizing problem-solving techniques may likely remain a bottleneck of MLLMs.\n\nDoes exposure to perturbed training data actually improve generalization, or does it reinforce memorization biases?\n\nTo truly assess whether fine-tuning on such data mitigates memorization, an ablation study comparing models trained on perturbed vs. non-perturbed datasets would be necessary.", "relation_to_broader_scientific_literature": "This paper provides a valuable contribution by identifying new limitations in reasoning adaptability, but further research is needed to determine whether alternative training paradigms, such as RL or perturbation-based fine-tuning, can overcome these issues.", "essential_references_not_discussed": "Chengke Zou et. al., DYNAMATH: A DYNAMIC VISUAL BENCHMARK FOR EVALUATING MATHEMATICAL REASONING ROBUSTNESS OF VISION LANGUAGE MODELS, ICLR2025.", "other_strengths_and_weaknesses": "The paper is well-structured and the experiments are comprehensive and easy to follow.\n\nThe study provides valuable insights into model generalization and out-of-distribution reasoning.", "other_comments_or_suggestions": "I have no further comments.", "questions_for_authors": "I have no further questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JcWdINgKu7", "reviewer": "Reviewer_qtvW", "summary": "This paper proposes a new benchmark by modifying 279 MATH hard problems and evaluates the popular model on these questions. They also provide various analyses of the performance on these questions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback on our work! **We evaluated 12 new long-CoT models that appeared near or after the ICML submission deadline.** The results [here](https://anonymous.4open.science/r/icml2025_13579_math_perturb_rebuttal-C0F8/) show no sign of saturation on MATH-P-Hard. We would like to provide detailed responses below:\n\n---\n>**Q1** The performance drop of SOTA models like Gemini and O1 is acceptable, indicating that the math-solving ability of SOTA LLMs is truly strong. This raises concerns that the new dataset may not be sufficiently difficult and could become outdated quickly, given the rapid advancements in math-focused LLMs.\n\n**A1**: \nThank you for raising the concern! We would like to discuss with you our thoughts and emphasize our contributions as well:\n\n- (1) **There are still around 20% of the problems (55 problems) that the SOTA long CoT models fail to solve.** So, one way to address the issue is to artificially **split the benchmark into two subsets**, for example, an “easy” set and a “difficult” set. The “difficult” subset can be used to evaluate SOTA long-CoT models while the “easy” subset can be used to evaluate small or short-CoT models. In that case, the SOTA performance on the “difficult” subset will be low, which leaves room for improvements and also saves evaluation cost. \n\n-  (2). One may still be concerned by the size of the “difficult” subset. For reference, the functional variants subset of Putnam-AXIOM [1] only contains 52 problems, and GSM-Symbolic  [2] only contains 100 problems. AIME 2024/2025 contains 30 problems. **So we believe ~55 problems are an adequate number to claim sufficient contribution. These problems can serve as the seed to curate more problems.** \n\nReference: \n- [1] Putnam-AXIOM: A Functional and Static Benchmark for Measuring Higher Level Mathematical Reasoning\n- [2] GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models\n\n---\n\n> **Q2** How do you view the concept of synthesized math problems? You may consider adding a discussion on this topic in the paper.\n\n\n**A2** Thank you for the suggestion! **In our next revision, we will add a short discussion of this future direction in the conclusion section.** We believe using synthesized math problems in training is a promising approach for improving the robustness against hard perturbations. For example, one can synthesize a training dataset with paired examples of (original problem, its hard perturbation) via hybrid methods that involve both state-of-the-art LLMs and expert-level human annotators.\n\n---\n> **Q3** I find the boundary between ‘Perturbations’ and a new question to be somewhat vague. Your example, ‘From a line to a hyperbola,’ seems more like a completely new question rather than a perturbation. In my view, ‘Perturbations’ should focus more on adding disturbances to the problem, such as introducing irrelevant or misleading information. Defining perturbations in mathematics appears to be challenging.\n\n**A3**: We agree with you that giving a precise definition of “perturbation” may be challenging. In our paper, we use **“simple perturbation”** to refer to the cases where the reasoning patterns of the modified problem remain the same, which should be closer to the “perturbation” in your view. \n\nIn contrast, for **hard perturbation**, we agree that the modified problem is essentially a new problem in the sense that the two problems have different reasoning patterns. We still call the modified problem a perturbation of the original one because they look similar superficially. **We designed MATH-P-Hard in this way to deliberately elicit memorization behaviors of the models.**\n\nSetting aside the debate of definitions, **we believe that hard perturbation is a valid and important setting, especially when reasoning models are deployed for end users or agentic uses**. It is common for end users or agent systems to make slight changes to the inputs that fundamentally alter the questions. If the model fails to identify the changes and applies the memorized solutions, it may have bad consequences. We hope our benchmark can inspire future work in this direction. \n\n---\n**We sincerely hope that our responses can address your concerns, and we would greatly appreciate it if you would consider raising your score of our work to a *clear accept* given the responses.**", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "Provide the new evaluation benchmark.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. The new math evaluation benchmark is valuable and needed by the community.\n2. Creating and verifying problems manually is valuable and more reliable.\n3. The motivation for modifying widely used MATH hard problems is well-founded.\n4. The modification method of MATH-P-Hard is convincing.\n5. The analysis of various phenomena is helpful.\n\nWeakness:\n1. The performance drop of SOTA models like Gemini and O1 is acceptable, indicating that the math-solving ability of SOTA LLMs is truly strong. This raises concerns that the new dataset may not be sufficiently difficult and could become outdated quickly, given the rapid advancements in math-focused LLMs.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. How do you view the concept of synthesized math problems? You may consider adding a discussion on this topic in the paper.\n2. I find the boundary between ‘Perturbations’ and a new question to be somewhat vague. Your example, ‘From a line to a hyperbola,’ seems more like a completely new question rather than a perturbation. In my view, ‘Perturbations’ should focus more on adding disturbances to the problem, such as introducing irrelevant or misleading information. Defining perturbations in mathematics appears to be challenging.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DWJT0klaV5", "reviewer": "Reviewer_tTX2", "summary": "This paper constructs a new dataset by applying simple an hard perturbations to the hard problems in the original MATH dataset. Experimental results show drop in performance for almost all models.\n\n## Update after rebuttal\n\nI remain positive about the paper after reading author rebuttal.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback on our work! **We evaluated 12 new long-CoT models that appeared near or after the ICML submission deadline.** The results [here](https://anonymous.4open.science/r/icml2025_13579_math_perturb_rebuttal-C0F8/) show no sign of saturation on MATH-P-Hard. We would like to provide detailed responses below:\n\n---\n> **Q1**: Both train and test splits seem relatively small. Did you observe significant variance in the performance of the same model when running it multiple times?\n\n**A1**: **We did not observe significant variances in the performance.** \n\n- Our Fig. 9 contains the error bars of the performance of multiple runs, which shows the standard deviation is less than 1% (note for Fig. 9: the Self-Consistency with $k=1$ corresponds to the standard evaluation.) \n- For our new results on 12 long-CoT models, the averaged standard deviation of the performance of 3 independent runs is 0.91%. \n- One may still be concerned by the size of our benchmark. For reference, we would like to point out that the functional variants subset of Putnam-AXIOM only contains 52 problems, GSM-Symbolic only contains 100 problems, and AIME 2024/2025 contains 30 problems.  \n\n---\n> **Q2** I am curious whether the issue in Figure 5 is repeatable or it just a one-off because of the specifics of the problem statement. Could you provide more details about your manual inspection of 20 error cases?\n\n**A2**: **The memorization issue is a frequently observed phenomenon in our experiments. We did not cherry-pick the example in Figure 5.** We understand that readers may have concerns about the proportions of memorization issues, so we have quantified them manually. We plan to open-source the benchmark so our claims can be publicly scrutinized. \n\nWe attach the raw logs of our manual inspections in the [anonymous github link](https://anonymous.4open.science/r/icml2025_13579_math_perturb_rebuttal-C0F8/). As the generated solutions are long and unformatted in Markdown, we omit them altogether with the problem statements and only include the problem ID, error type, and comment. \n\n\n\n---\n> **Q3** Generally a lot of analysis seems anecdotal. Would it be possible to provide statistical evidence for the phenomena described?\n\n**A3**: **We have already taken extra caution before drawing any conclusion in the experiment section. To support each claim, we have provided quantitative numbers of different metrics as well as qualitative studies that require extensive human labor.** We hope our response to your Q2 can mitigate your concern. Besides this, could you specifically point out the claim that you think lacking statistical evidence? We are happy to address any further concerns.\n\n---\n> **Q4**: it might still be good to evaluate with code interpreter, and then remove those problems that are trivially solvable with code as there reasoning is likely less necessary.\n\n**A4**: **We don’t think reasoning is less necessary for problems that can be trivially solvable with code.** Many number theory problems and counting problems can be solved via brute-force code solutions. However, for example, a counting problem may require the mathematical knowledge of **inclusion-exclusion principle** to solve, and a number theory problem may require knowledge of **finite group theory** to solve. The answers to these problems can often be produced by trivial brute-force codes, but they do require mathematical knowledge and reasoning ability.  \n\n---\n> **Q5** Somehow I am not fully sure to what extent would I even consider these Hard perturbations as actual perturbations of the original dataset. .... So it's not that unexpected to me that LLMs perform worse there.\n\n\n**A5**: **Unexpectedness of experimental results shouldn't undermine the contribution of the actual experiments.**\nOur results may be well-expected from your high-level conceptual argument. Nevertheless, if our experiments suggested that “LLMs are strong enough to distinguish between different perturbation types and solve these problems perfectly”, this finding could also be well-expected from an opposite but compelling argument, e.g., LLM developers have intensively built training data to cover all the perturbation cases. **Therefore, curating the benchmark out and empirically verifying the hypothesis should be a valid and important contribution.** \n\n**we believe hard perturbation is an important setting, especially when reasoning models are deployed for end users or agentic uses.** It is common for end users or agent systems to make slight changes to the inputs that fundamentally alter the questions. If the model fails to identify the changes and applies the memorized solutions, it may have bad consequences. We hope our benchmark can inspire future work in this direction. \n\n---\n**We sincerely hope that our responses can address your concerns, and we would greatly appreciate it if you would consider raising your score of our work to a clear accept given the responses.**", "claims_and_evidence": "The claims are generally supported by evidence and convincing, though some interpretations of experimental results seem anecdotal.", "methods_and_evaluation_criteria": "Evaluation criteria make sense.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "- Both train and test splits seem relatively small. Did you observe significant variance in the performance of the same model when running it multiple times?\n- I am curious whether the issue in Figure 5 is repeatable or it just a one-off because of the specifics of the problem statement. Could you provide more details about your manual inspection of 20 error cases?\n- Generally a lot of analysis seems anecdotal. Would it be possible to provide statistical evidence for the phenomena described?\n- \"We do not allow any tool usage including access to a code interpreter, as we find that many problems can be trivially solved by writing a brute-force search program.\" - it might still be good to evaluate with code interpreter, and then remove those problems that are trivially solvable with code as there reasoning is likely less necessary.", "relation_to_broader_scientific_literature": "The contributions are well positioned as they directly improve over prior work that considered only simple symbolic perturbations (e.g. GSM8K-Symbolic).", "essential_references_not_discussed": "--", "other_strengths_and_weaknesses": "I think the paper has a nice contribution as most of the existing work around this problem (generating perturbations of a dataset) falls short of creating interesting/hard perturbations.\n\nSomehow I am not fully sure to what extent would I even consider these Hard perturbations as actual perturbations of the original dataset. Sure at the syntax level the modification is very small. But for example, in Figure 3, hard perturbation leads to a completely different solution so I would not consider these problems similar. So it's not that unexpected to me that LLMs perform worse there.", "other_comments_or_suggestions": "Generally it would be good to have more details on how were the experiments on failure modes performed, and have more convincing argument that it's not just anecdotal evidence based on few examples.", "questions_for_authors": "Please see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose a benchmark consisting of "hard" perturbations of the MATH benchmark. They argue that while models can do well on easy perturbations, the performance decrease is significant on the harder perturbations. The construction of the benchmark and the results are sound. Reviewers appreciated the clear description of the dataset and its utility for evaluating reasoning. At the same time, I suggest authors to put their benchmark in perspective with other recent perturbation-based datasets mentioned by reviewers and consider if they really want to call it a new kind of memorization, or just a facet of memorization that is highlighted by the current benchmark.
Decision: Accept (poster)