From Abstract to Contextual: What LLMs Still Cannot Do in Mathematics
Authors: Bowen Cao
Dongdong Zhang
Yixia Li
Junpeng Liu
Shijue Huang
Chufan Shi
Hongyuan Lu
Yaokang Wu
Guanhua Chen
Wai Lam
Furu Wei
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Mathematical Reasoning
Evaluation
Primary area: datasets and benchmarks

Abstract: Large language models now solve many benchmark math problems at near‑expert levels, yet this progress has not fully translated into reliable performance in real‑world applications. We study this gap through contextual mathematical reasoning, where the mathematical core must be formulated from descriptive scenarios.We introduce CORE-MATH, a benchmark that repurposes AIME and MATH-500 problems into two contextual settings: Scenario Grounding (SG), which embeds abstract problems into realistic narratives without increasing reasoning complexity, and Complexity Scaling (CS), which transforms explicit conditions into sub‑problems to capture how constraints often appear in practice. Evaluating 61 proprietary and open‑source models, we observe sharp drops: on average, open‑source models decline by 13 and 34 points on SG and CS, while proprietary models drop by 13 and 20.  Error analysis shows that errors are dominated by incorrect problem formulation, with formulation accuracy declining as original problem difficulty increases. Correct formulation emerges as a prerequisite for success, and its sufficiency improves with model scale, indicating that larger models advance in both understanding and reasoning. Nevertheless, formulation and reasoning remain two complementary bottlenecks that limit contextual mathematical problem solving. Finally, we find that fine‑tuning with scenario data improves performance, whereas formulation‑only training is ineffective. However, performance gaps are only partially alleviated, highlighting contextual mathematical reasoning as a central unsolved challenge for LLMs.

Published as a conference paper at ICLR 2026
FROM ABSTRACT TO CONTEXTUAL: WHAT LLMs
STILL CANNOT Do IN MATHEMATICS
Bowen Cao Dongdong Zhang
Yixia LI
Jumpeng Liu'
Shijue Huang
Chufan Shi
Hongyuan Lu
Yaokang Wu?
Guanhua Chen²
Wai Lam'
Furu Wei?
'The Chinese University of Hong Kong
Miicoooftt
Soothhem University of Science and Technology
'The Hong Kong University of Science and Technology
University of Southern Califormia
'FaceMind "Carnegie Mellon University
bwcaod 1 ink. cuhk. edu.hu. hk
dozhang@m.lcrosoftt.om
ABSTRACT
Large language models now solve many benchmark math problems near-expert levels, yet this progress has not fully translated into reliable performance in real-world upplications. We study this gap through contextual mathematical rea- soning, where the mathematical core must be formulated from descriptive sce- narios. We introduce ConfextMATH, benchmark that repurposes AIME and MATH-500 problems into two contextual settings: Scenario Grounding (SG), which embeds abstract problems into realistic narratives without increasing rea- soning complexity, and Complexity Scaling (CS), which transforms explicit con- ditions into sub-problems to capture how constraints often appear in practice. Evaluating 61 proprietary and open-source models, average, open-source models decline by 13 and 34 points on SG and CS, while proprietary models drop by 13 and 20. Error analysis shows that errors are dom- inated by incorrect problem formulation, with formulation accuracy declining as original problem difficulty increases. Correct formulation emerges prereq- uisite for success, and its sufficiency improves with model scale, indicating that larger models advance in both understanding and reasoning. Nevertheless, formu- lation and reasoning remain two complementary bottlenecks that limit contextual mathematical problem solving. Finally, we find that fine-tuning with scenario data improves performance, whereas formulation-only training is ineffective. How- ever, performance gaps are only partially alleviated, highlighting contextual math- ematical reasoning as central unsolved challenge for LLMss.
observe sharp drops:
INTRODUCTION
Large language models (LLMs)) now dominate mathematical benchmarks, scoring nearly perfectly on AIME OPenAl 2024, Guo et al. 2025 OpenAl 2025) and even reaching IMO gold] Yet these successes remain contined well-defined benchmark problems, with little sign of comparable progress on the broader reasoning skills required for real-world impact (Qian et al. 2025).
This gap reflects fundamental divide in mathematics. On one side are well-defined abstract prob- lems, such as algebra analytic geometry, that can be solved through established strategies and symbolic manipulation (Polya 1945 Schoenfeld] 2014). On the other side are scenario-based prob- lems, ranging from financial analysis to scientific research and engineering design, where the math- ematical core is conveyed through concrete narrative detail. Existing benchmarks overwhelmingly target the former (Cobbe al.] 2021a| Lightman al., 2023 He al.) (024). leaving the latter largely unexamined. In this work, we term this underexplored domain contextual mathematical reasoning: the ability to formulate and solve the mathematical problem when it is embedded in narrative scenarios with indirect or layered conditions.
"Tii work done during internship at Microsoft Research Asia. "Corrsponning author
https: /xx com allaaaelwwweee / http://ttuu///1777723333333

---

Published as a conference paper at ICLR 2026
(1) Original Problem:
Let N denote the number ordered triples of positive integers (a.b.e) such b, is 3 and x c is multiple of 3?, Find the remuinder when N divided by 1000.
B+4x2
(2) Scenario Grounding:
th.three distinct components within a sophisticated smart energy system...
VX
...with positive values that not exceed 729 units...
CORE HIGTH
...iiisng their output metrics the third power and summing these values, must yield a total that perfectly divisible by 2,187...
5kmm
...find the remainder when this number divided by 1,000 for reporting purposes...
(3) Complexity Scaling:
In simulation lah, three autonomous drones are sent out along separate axes one moves north-south, one east-west, and one vertically. Each drone travels forward a positive number of whole steps. 'Teese steps remain below certain threshold, which is known
power 3. known that step counts as high as 2000 are not allowed, while allowed range. As part synchronization test, the system calculates combined energy score by summing the cubes of the step counts of all three drones. Syachronization only succoeds if this total energy is also a multiple of a certain power of 3, From prior data, energy total of 1458 fails to trigger the syne, but 8748 does trigger it. How many such ordered drone configurations satisfy both conditions? Report your answer remainder when this number divided by 1000.
within
Figure I: Example from ContextMATH, based on AIME 2025 Problem 15. In Scenario Grounding (SG), mathematical components are mapped to a narrative. In Complexity Scaling (CS), explicit conditions are concealed in sub-problems requiring an extra inference step. Consistent color-coding highlights correspondence between mathematical components across the three versions. LLMs re- main strong on abstract benchmarks but drop accuracy on SG, with the gap widening further on CS.
To investigate this capability, we introduce ContextMATH (Contextual Reasoning Evaluation for Math), a benchmark designed to probe contextual mathematical reasoning systematically. Rather than seeking massive collections of real-world tasks, ConteXIMATH builds on standard sources LiiieesAAMM AIME 2024, AIME 2025, and MATH-500 (Lightman et al. 2023 and instantiates each problem in two controlled narrative variants: Scenario Grounding (SG) embeds abstract mathemat- ical structures into concrete narratives with real-world entities and interactions. 'he reasoning core remains unchanged, but the problem is situated in deseriptions that naturally introduce contextual detail. Complexity Scaling (CS) conceals explicit conditions within sub- uub-problems in the scenario. For example, an absolute position may be given relative to a reference point, or a numerical bound may appear as the result of a short calculation. These sub-problems yield exactly the information originally explicit, while reflecting how constraints are often expressed in real-world settings. This format also reduces reliance surface pattern matching, since models must first interpret the sce- nario to recover the original conditions. A Although demonstrated here for mathematics, the same approach applies other domains and datasets, enabling systematic evaluation of contextual rea- soning beyond math. Figure [ original problem correspond to system components SG and drone movements in CS.
iillsstrates such mappings: for example, the variables (a, b, c) in the
Our results show that both leading open-source and proprietary LLMs experience a substantial de- cline in contextual mathematical reasoning compared their strong performance on original bench- marks. On average, open- models drop 13% on SG and 34% on CS, while proprietary models fall 13% and 20%, respectively. For example, Owen3-322 falls from 81.25% on AIME 2024 prob- lems to 67.92% on SG and 57.08% on CS. A similar trend holds for proprietary systems: even DeepSeek-BB R1 drops from 86.67% to 73.33% (SG) and 53.33% (CS) on AIME 2025.
To better understand these limitations, we analyze models' ability to formulate the mathematical core from narrative scenarios. We find that performance declines as problems become harder, and although larger models perform better, even GPT-5 averages only 81.4% formulation accuracy. Furthermore, correct formulation proves to be a prerequisite: correctly solved problems show much higher formulation accuracy than average, establishing formulation as the first bottleneck. We fur- ther observe that as models scale and formulation accuracy improves, correct formulation increas- ingly becomes sufficient for solution correctness, indicating progress in both problem understanding
2

---

Published as a conference paper at ICLR 2026
and reasoning. Yet even GPT-5 reaches only 82.7% sufficiency, showing that reasoning remains
second major bottleneck for real-world mathematical problem solving.
We further explore whether the performance drops can be alleviated via training. Our results show that fine-tuning with scenario data effectively improves models' performance on our benchmark, yet sizable drop contextual variants still remains, indicating that contextual mathematical reasoning is far from solved. In contrast, directly training a dedicated formulation model proves ineffective, suggesting that formulation ability is difficult to acquire from paired supervision alone.
Our contributions can be summarized as follows:
New task. We frame contextual mathematical reasoning, the ability to extract and solve the mathematical core from narrative descriptions, as a critical but underexplored capacity for LLMs.
New benchmark. We propose ContextMATH. which instantiates problems from AIME and MATH-500 into Scenario Grounding (SG) and Complexity Scaling (CS) variants to probe mathe- matical formulation and reasoning under contextual challenges. Evaluations show large accuracy drops on SG and CS relative to abstract benchmarks, revealing clear and persistent gap between abstract and contextual mathematical reasoning in current LLMs.
New insights. Through quantitative and qualitative analysis, we identify formulation and reason- ing as complementary bottlenecks, and provide evidence that training with scenario data improves performance, whereas training a dedicated formulation model remains ineffective. These findings suggest that contextual mathematical reasoning is learnable, but cannot be reduced to isolated skills, highlighting the need for integrated approaches to advance formulation and reasoning.
2 RELATED WORK
Benchmarks Abstract Mathematical Problems. Much of the progress in evaluating LLMs' mathematical ability has been driven by benchmarks centered on abstract, well-specified prob- lems. GSM8K (Cobbe et al., 2021b) covers grade school arithmetic word problems, while MATH (Hendrycks et al., (022b) spans high-school curricula, and AIME, AMC23, and Olympiad- emphasize challenging competition-style questions. There are also bench- marks that target formal theorem proving, such as PutnamBench (Tsoukalas et al. 2024), Lean- Dojo (Yang al. 2023). and MiniF2F (Zheng al. (02T). While some existing problems contain simple narratives such "Jack had 8 pens and Mary had 5 pens.... (Patel et al., 2021) these con- texts are shallow and limited scope. As such, current benchmarks primarily capture performance on clearly formulated abstract problems, leaving open the question of how well models can reason about mathematics when it is embedded in richer contexts.
Bench (He
From Abstract Problems Scenarios. Across domains, growing line of work has begun to examine model reasoning in realistic settings. For example. WebArena (Zhou et al. 2024)) evaluates agents in interactive web environments, while SWE-bench (Jimenez et al.) (0024) and BaxBench (Vero et al., 2025) focus on real-world software engineering tasks. Our work falls within this broader scope, but focuses specifically on mathematical reasoning in scenarios.
3 PROBING LLMs' ABILITY TO SOLVE MATH IN SCENARIOS
3.1 OVERVIEW
central question of this work is whether LLMs possess contextual mathematical reasoning the ability to formulate and solve the mathematical core when problems are embedded in descriptive sce- narios. This capacity is crucial for applying mathematics practical domains, where tasks rarely ap- pear as ready-made equations. Collecting large numbers of real-world instances, however, is costly and difficult to scale. We therefore leverage established benchmarks with guaranteed correctness and systematically transform them into contextual variants. Concretely, we build ContextMATH. which instantiates each benchmark problem two forms: Scenario Grounding, preserving the orig- inal structure but embedding it in narrative context; and Complexity Scaling, encoding explicit con- ditions into simple sub-problems. We then evaluate a diverse set of 46 soures models and 15 proprietary models, including GPT-5 and DeepSeek-R1.
3

---

Published as a conference paper at ICLR 2026
Model
AIME 2024 (%)
AIME 2025 (%)
Ori
SG
SG Avg@3
C8
Ori
SG
CS
Copilot
30.0
30.0
-0%
28.9
-4%
23.3
-22%
33.3
20.0 ( -10%
16.7
-50%
spt-to-mini
6.7
3.3
-50%
6.7(
-0%
3.3
-50%
10.0
6.7 -33%
0.01
100%
ol-mini
60.0
53.3
-11%
53.5
-11%
10.0
-33%
40.0
33.3
-17%
23.3
-42%
spt-4o
16.7
13.3
-20%
11.1(
-33%
3.3
-80%
10.0
6.7( -33%
0.0
100%
spt-4.1-mini
46.7
26.7
( -33%)
314( ( 266%
30.0
-96%
53.3
30.01 ( -44%
16.7
-69%
gpt-4.1-nano
26.7
23.3
-13%)
18.9
-29%
6.7
-75%
33.3
20.0
-40%
13.3
-60%
RI
93.3
70.0
-25%
70.0
25%
66.7
.9%%
86.7
73.31
-15%%
53.31 (
-38%
Doubso-15 1.5
90.0
70.0 ( -22%)
66.7
26%
56.7
-37%
76.7
53.3
-30%
43.3
-43%
Qwen-max
23.3
16.7
-9%
14.4 38%
6.7
-71%
13.3
10.0
-25%
0.01
100%%
QwQ-plus
86.7
56.7 ( -35%
60.0
-31%
46.7
.46%
73.3
53.3
-27%
43.3
-41%
Grok3
43.3
23.3(
-46%
22.2(
49%
23.3
-66%
26.7
13.31
-50%
20.0
-25%
Gemini 2.5 Flash
70.0
63.3 ( -10%)
61.1 ( (13%%)
53.3
-24%
70.0
43.3
-38%
30.0
-57%
Gemini 2.5 Pm
83.3
73.3 ( (-12%)
68.9
(-17%)
76.7 -8%
83.3
56.71
-32%
50.0
-40%
o3
83.3
70.0 ( 116%)
73.3 ( -12%)
66.7 ( 200%
76.7
70.0
.9%
60.0
-22%)
gpt-5
90.0
83.3
-7%%
82.2( .9%
80.0( -11%
90.0
80.01
-11%
66.7(
.26%%
Table I: Accuracy of proprictary models on ContextMATH. Best and second-best results per column are in bold and underlined, respectively. Parentheses indicate the relative performance change from Ori, with larger drops highlighted in a deeper shade of red. To verify that our findings are consistent and not artifact of specific scenarios, we additionally generated and annotated two AIME2024- SG sets, reporting the average accuracy across all three in the SG Avg0 3 column.
3.2 BENCHMARK CONSTRUCTION
Our benchmark is constructed from AIME 2024, AIME 2025, and MATH-500 Our design follows two principles: preserving mathematical equivalence and embedding problems in realistic narrative contexts. To realize this, we employ a series of structured prompts that guide an LLM (o1 -mi ni) through iterative scenario generation, selffverification, and revision. Human experts then review and refine each item to guarantee mathematical equivalence, clarity and concisenes
Scenario Grounding (SG): Evaluating the Application of Math in Context. The primary goal of scenario grounding is to assess whether a model can apply its mathematical knowledge when problems presented in a narrative context, without increasing the core reasoning difficulty. To achieve this, our generation process is guided by a multi-step prompt that instructs the model to first explicitly map all abstract mathematical elements specific real-world objects (e.g., mapping "a variable x" to "the initial number of barrels of oil"). The prompt then directs the model to define the rules of interaction between these objects based on the attributes and relationship in the problem. This structured approach ensures that all mathematical components of the original problem are preserved in the final scenario (see Appendix for the full prompt).
Complexity Scaling (CS): Encoding Implicit Constraints through Sub-Problems. In many real-world settings, quantitative conditions must inferred from indirect descriptions, such as sim- ple calculations, relative positions, or everyday facts. To reflect this natural form of complexity, CS problems transform some direct conditions into the outputs of simple, self-contained sub-problems. For example, instead of stating there are "25 indicator lights", the problem may specify that "the total number of unique pairs of indicator lights is exactly 300". Our generation prompt provides principled strategies for such transformations, including encoding values as the solutions to num- ber theory or combinatorics problems, replacing explicit functions or constants with variables to be determined from given data points, and rephrasing geometric relationships in terms of physical or structural descriptions. This design also provides a more faithful and robust test of a model's rea-
We select only problems with difficulty level 2 3 from MATH-500 embedding into meaningful scenarios. For simplicity. we continue to refer to this filtered subset as MATH-500 throughout the paper. We do not construct the CS set for MATH-500 because some remaining items are trivial to transform further, e.g., "Evaluate (1 + (6 - 3 'hhe average lengths of SG and CS problems are 133 and 176 words, respectively, both well within the processing capabilities of current I.IMs.
-00,, since easier items are less suitable for
4

---

Published as a conference paper at ICLR 2026
Model
AIME 2024 (%)
AIME 2025 (%)
Math-500 (%)
Ori
SG
CS
Ori
SG
CS
Ori
SG
Qwen3-0.68
7.9
6.2
-21%
-95%
15.4
-65%
-84%
62.4
42.6
(3-2%
Qwen2.5-Math-1.58
11.5
2.5
-7%%
0.4(
966%
4.6
0.8
-82%
0.0
100%
32.5
25.8
-21%
- --Disill-wee--ll
28.3
20.0
-29%
7.1
-75%
19.6
9.6(
-51%
5.8
700%
74.0
57.6
-22%
- --DeepScallR-115 1.----11..---Prrieee Preview
41.5
23.3
-44%
7.5
82%
30.4
15.4
49%
7.5
-75%
80.1
64.2(
-20%
Opeen 1.5B
62.5
34.2
14.6
50.4
21.2(
11:2(
87.3
70.1
- --DeepMatt-Onm--
62.9
(33.8
-16%
13.8
-88%
55.8
20.8
-63%
12.5
-78%
87.1
73.0
-16%
Qwen3-1.5B
29.6
-36%
24.6
67.4
-20%
Qwen3-4B
70.4
52.5(
-25%
134.6
-51%
64.2
39.6
-38%
33.8
-47%
90.9
78.8
-13%
7R/8B
0w 7athhBBB
10.8 6.7 -38%
1.5
83%
5.0
3.3
-33%
0.8
83%
44.8
36.7
(118
+ IpeeMMall-Neemorron 7B
72.9 52.1
29%
(30.0
59%
60.0
40.4(
-3%
29.2
51%
90.5
78.5
-13%
* RI-Distil--e 1-Disill-Qwen-78
48.8 40.0 -18%
23.3
-2%%
41.5
22.5
46%
15.0
64%
87.4
(-55% -15%
AaceMath-Rl-Neeetteee
69.2 48.8 ( -30%
.2%
54.2
26.7
-51%
22.5
-58%
89.9
79.0
-12%
Qwen3-8B
73.8 61.5( -16%
42.9
42%
64.6
48.3(
-25%
35.8.
.45%
91.0
81.0
(-15%
+R ---R--0528 2-1-028-Qwee3--88
75.0
55.0
.27%
133.6
47%
65.8
48.3(
-27%
32.9
-50%
90.7
77.2 (15%%
-+RRaa -RReell-boha22.5 2-88
74.2 58.3
-21%
41.5
44%
67.9
47.9
-29%
)37.1
45%
91.5
82.0(-115)
148
Qwen2.5-148
6.2
3.8
3.3
0.0
48.5
34.9 -28%
--R---isril-Qe 14B
67.5
47.
.30%
35.4
48%
50.8
26.2
48%
25.8
.4%
89.5
76.9
(-146
--OppeMatt 14B
73.8
4
63.8
42.5
29.2
54%
90.8
80.8
(-15%
Qwen3-14B
80.0
61.6
-19%
10.8
-36%
72.9
49.2
-33%
142.1(
-42%
92.6
81.9 -12%
+ -RReaL-bobaa--14B
82.9
65.8
-21%
153.8
-35%
73.3
52.1
-29%
33.22
-47%
91.9
82.9 -10%
Phi-i-rrasoning-pluss
80.4
60.
-25%
52.91
-34%
71.5
55.4(
-22%
39.6
-45%
92.6
83.1
-10%
Qwen2.5-32B
11.2
3.8
-16%
+ -OpeeaMattt-eeoooooee
57.1
42.5 ( -66%
1
-11%
52.1
34.6
-34%
27.5
-47%
75.8
61.8(
-18%
---l--Diull-Qwem3328
69.6
-25%
33.0
78.9
-12%
Qwen3-328
81.2
67.91
-16%
15.11
-30%
70.0
54.1
-22%
14.00
-36%
92.1
82.7
-10%
ABRaaL-bobb-2-328
-20%
-29%
77.1
-29%
13.8
-10%
QwQ.32B
80.4
58.3
.27%
13.31
344
66.2
53.3
20%
39.2
41%
92.5
82.9 -10%
RI-Distill 70B
65.4
38.8
29.2
77.3 -14%
Table 2: Accuracy of open-source models on ContexIMATH. Models are grouped by parameter scale, and best and second-best results within each group shown in bold and underlined, respec- tively. Parentheses indicate the relative performance change from Ori, with larger drops highlighted by a deeper shade of red, Arrows (-+ denote variants obtained from the model listed directly above. Full results are provided in Appendix A.3
soning ability by reducing reliance on superficial pattern matching, a behavior widely documented in prior studies showing the limited generalization of LLMs (Cao et al. 2024 Sun et al. 2025).
Quality Control. We place high importance on data quality. Our approach involves two key mea- sures: first, have optimized the instructions and generation pipeline through extensive pilot stud- ies; second, our procedure involves three experts, all with advanced degrees in Computer Science and backgrounds in competitive mathematies, to fundamentally ensure accuracy. They conduct in- dependent reviews based on the following criteria:
Assessing the scenario for narrative plausibility and clarity, ensuring it introduces no unnecessary complexity or ambiguity.
Independently formulating an abstract math problem from the scenario to verify its solvability and mathematical equivalence to the original.
Testing the scenario on Gemini and GPT-5; if these models fail, the reviewer diagnoses whether the failure originates from ambiguity in the problem description.
5

---

Published as a conference paper at ICLR 2026
A scenario is accepted only after passing all checks. Any identified issue flags the scenario for revision. In cases disagreement, the reviewer with the strongest mathematical background leads discussion to reach consensus, which may trigger regeneration revision cycle.
3.3 EVALUATION SETUP
Models. We evaluate 46 advanced open-source models (including base. SFT, and RL-tuned) and 15 frontier proprietary model Details are provided in Appendix A.2
Metrics. We use accuracy as the performance metric. For open-source models, we sample 16 solutions per problem using their recommended generation configurations and report the average accuracy. For proprietary models, which are accessed via APIs or client interfaces that often have rate limits higher costs, report the accuracy of a single-pass evaluation.
3.4 RESULTS
Our main experimental results, presented in Table I and reveal three clear insights:
Contextual Complexity as Universal Bottleneck. Both open-source and proprictary mod- els show consistent and severe performance drops on contextual scenarios. For instance, - falls from 75.0% on AIME 2024 to 39.6% on its CS vari- ant, and QwQ wwQ-plus from 86.7% to 46.7%. Even GPT-5, with an estimated 1,8T parameters, drops 26% AIME 2025-CS. These findings make clear that contextual mathematical reasoning remains a fundamental and unresolved challenge, underscoring that progress on abstract math does not fully translate into robust performance in contextual settings.
PooSSek-R1-1 -eseeek--1-022---0
Scale Mitigates but Does Not Solve the Problem. In open-source evaluations, larger models retain more robustness, but scaling does not eliminate contextual failure. In the OpenMath-Nemott ron series on AIME 2024-CS, the 1.5B model drops 77%, compared to 43% and 51% for 14B and 32B. Similar trends hold for B1-Distil-Owend 5 and Owen families. Larger models are more resilient, but interpreting complex narratives remains core bottleneck.
Initial SFT Improves Robustness, but Further Specialization Does Not. first stage of su- pervised fine-tuning consistently improves both original and contextual accuracy. For example, Qwen2. 5 contextual drop. However, further SFT or RL for advanced reasoning does not improve textual tasks. Models such as R1-Dist: -ee--wenn-- 7 --RL-Nemot ron and Qwen 1ee33-888--eeee -B---eeeSSeee----e-eeee--e-e CS, often with even larger drops. This suggests current post-training can over-specialize to canoni- cal formats, reinforcing pattern recognition rather than contextual reasoning.
not only boosts AIME accuracy but also reduces
0528 10550-wwen3 3w 8B improve AIME 2024/25 but not SG or
ANALYSIS OF FAILURE MODES: THE FORMULATION BOTTLENECK
The performance degradation observed in Section 3 motivates an analysis of the underlying failure modes. Our central hypothesis is that these failures do not primarily stem from flawed computational reasoning, but from the incorrect interpretation of the problem's mathematical structure from its narrative context. This section validates this hypothesis through combination of qualitative analysis and quantitative experiments that isolate contextual mathematical problem formulation skill.
4.1 QUALITATIVE ANALYSIS: INCORRECT MATHEMATICAL INTERPRETATION
To examine how such failures occur. we ask GPT-5 to categorize the errors made by DeepSeek R1, Gemini ni 2.5 Pro, and Qwen 3-33B on the SG and CS sets of AIME 2024 and 2025, re- stricted to cases where the same models solve the original version correctly.
'Since AIME 2024 and AIME 2025 results already established performance trend for proprietary models, and given the high API costs of Math-500, we excluded proprietary models from this part of the evaluation.

---

Published as a conference paper at ICLR 2026
Model
Formulation Accuracy (%)
Formulation Necessity (%)
Formulation Sufficiency (%)
MATH ATME24 ATME25 Avg.
MATH
ATME24
ATME25 Avg.
MATH AIME24
ATME25 Avg.
RI-Disull-Qwea--le
62.3
48.8
40.4
48.1
70.6
52.6
58.3
58.5
65.0
13.8
12.5
23.5
11-Distil-Qweee-7
75.7
57.9
47.1
57.1
82.9
67.0
60.8
67.7
81.1
38.7
24.5
41.5
RI-Distll-Qwea--B
85.7
68.8
51.7
65.3
90.1
76.2
79.8
80.4
80.8
47.9
37.0
50.1
Di-DitillQQee--2BB
87.3
71.7
59.6
70.0
91.8
78
73.9
79.3
42.7
53.3
Qwen3 0.6B
57.4
37.9
40.4
42.8
69.2
76.4
29.2
56.1
13.5
Qwen3-1.7B
82.5
57.1
56.7
62.0
90.1
79.1
75.8
80.0
733.8
28.6
20.8
34.5
Qwen3-4D
84.8
64.6
47.1
61.6
90.3
85.55
67.4
79.2 22
83.6
57.1
54.2
61.3
Qwen3-8B
86.3
77.9
63..3
73.88
91.3
86.7
77.2
83.8
86.3
57.7
51.0
60.7
Qwen3-14B
64.6
72.4
90.4
78.5
88.4
84. 8
85.2
61.0
62.2
66.3
Qwen3-32B
89.3
77.
65.8
75.0
92.0
77.2
8.1.4
81 9
85.0
63.0
56.6
64.9
gpt5
90.5
85.0
73.3
81.4
94.5
87.6
79.2
84.2
79.2
82.7
Table 3: Problem formulation performance across different models. Formulation Accuracy denotes how often model correctly translates scenario into its formulation. Formulation Necessity quan- tifies the extent to which correct formulations are required for correct reasoning, while Formulation Sufficiency evaluates whether correct formulations reliably lead to correct reasoning. The AIME24 and AIME25 columns show the average of their SG and CS sets, while the 'Avg.' column is the mean over all sets. Darker cell colors indicate higher values.
Errors are grouped into four categories: for- mulation (incorrect mapping from narrative to math), calculation, logic, and other (e.g., trun- cation, repetition). Figure that formu- lation errors dominate, accounting for roughly 80% of failures across all three models and far exceeding any other type. A typical case given in Appendix [A.4] where DeepSeek R1 fails to recognize that "the time for gear complete one rolation is adjustable, but it musi not exceed six rotations per minute" im- plies the inequality 2 10, with denoting seconds per rotation. This consistent pattern confirms that formulation is the primary weak- ness in contextual reasoning. This presents an interesting contrast with Cheng et al. (2025): while their findings suggest that abstract for- mulation is not the primary bottleneck on simpler benchmarks like GSMSK (Cobbe et al., 2021a) la), work demonstrates that formulation becomes critical bottleneck once the mathematical core embedded in complex, contextual narratives.
Formulation Error
Calculation Error
Logical
Other Errors
Error Ratio
03
02
DeepSeek R1
Gemini 2.5 Pro
Qwen3-328
Figure 2: Distribution of error types in failure cases AIME 2024/2025 SG and CS problems, where ratios indicate the proportion of cases exhibiting each error type.
4.2 QUANTITATIVE VALIDATION: EVALUATING PROBLEM FORMULATION FROM CONTEXT
In this section, evaluate model's ability abstract mathematical formulation from given scenario. For each problem in the SG and CS test sets, the model prompted to output only the core formulation, stripped of all narrative details. In addition to reporting formulation accuracy, analyze the necessity and sufficiency of correct formulations for correct reasoning, highlighting the fundamental relationship between them.
Evaluation Setups.
We carefully design the instructions and examples to prompt ol-mini as an automated judger, which determines whether the model's s output is mathematically equivalent the original problem [l For each problem, we evaluate samples using the same seed set as in our benchmark evaluation (Section and report three complementary metrics. The first is
Wee manually annotated the outputs of Qwen 1 48 and 3-wen33324 and found that the judger's assess- ments align with human judgments 90% of cases.

---

Published as a conference paper at ICLR 2026
Formulation Accuracy, defined as the mean of the judger's assessments across samples. The sec- ond is Formulation Necessity, which measures how often a correct problem formulation is required for a correct solution. Formally, let F denote whether the formulation is correct and R whether the reasoning produces the correct answer. Necessity is defined as the conditional probability
Formulation Necessity P( True R True).
This quantifies the consistency of the relation R F. The third is Formulation Sufficiency, which evaluates whether a correct formulation reliably leads to a correct solution. given by
Formulation Sufficiency P(R True TTue).
This corresponds the coverage of the relation F = R. Together, these three metrics offer coherent perspective: accuracy captures overall performance, while necessity and sufficiency jointly characterize the directional dependency between formulation and reasoning.
Results. To present an overview of model
behavior, we report aggregated formulation metries Table 3 where the AIME24 and AIME25 scores are averaged over their SG and CS sets for simplicity, while the full break- down is provided in Appendix A.5 To further analyze the relationship between formulation and overall task performance, bine these metrics with the reasoning accura- cies from Section 3 and visualize the correla- tions in Figure 3 Our findings can be summa- rized into three key insights:
90
Accuracy
Necessity
80-
Formulation Metries
Suflidency
70
60-
50
also
30
20
10
Reasoning Accuracy
(i) Formulation accuracy declines with problem difficulty. Table 3 shows that for- mulation accuracy (orange) decreases steadily from MATH- AIME24 AIME25 (e.g., Qwen3-4B: 84.8% + 64.6% - 47.1%), con- sistent with the increasing difficulty of the underlying problems. This trend shows that problems with a harder mathematical core re- main more difficult for models to formulate correctly. Scaling improves performance (e.g., Qwen3-0. 6B averages 42.8% vs. 75.0% for 3-3B3) but even GPT-5 remains below 75% AIME25, underscoring that the formula- tion challenge grows with problem difficulty and cannot be resolved by scale alone.
Figure 3: Relationship between reasoning accuracy and formulation metrics. Each point represents a model, with the x-axis showing its average reason- ing accuracy across all subsets and the y-axis show- ing the corresponding values of formulation accu- racy (orange). necessity (green), and sufficiency (blue). The fitted lines indicate the overall trends.
(ii) Correct formulation is highly necessary for correct reasoning. On average, necessity (green) consistently above accuracy across all models. For example, Qwen3-4R achieves 61.6% accuracy 79.2% necessity, and GPT-5 81.4% vs. 85.6%. Figure 3 confirms this pattern: necessity closely tracks reasoning accuracy but is consistently higher, indicating that correct reasoning greatly depends on having a correct formulation.
(iii) Beyond formulation, reasoning ability still limits success. Sufficiency (blue) improves with scale but lags behind both accuracy and necessity. In Table Owen3-8B records 73.8% accuracy and 83. 33.8% necessity but only 60.7% sufficiency, while GPT-5 achieves 82.7% sufficiency. Figure further highlights this gap: sufficiency rises with reasoning performance but consistently trails ne- cessity, indicating that while correct formulation is indispensable, reliably completing the reasoning process remains challenging even for the strongest models, thus forming a second bottleneck.
5 IMPROVING CONTEXTUAL MATHEMATICAL REASONING
Our analysis in Section 4 nario, and (ii) carrying out the subsequent reasoning. In this section, we explore training strategies addressing these challenges, through both end-to-end fine-tuning and dedicated formulation models.
reveals two bottlenecks: (i) formulating the mathematical from sce-
8

---

Published as a conference paper at ICLR 2026
Model
AIME 2024 (%)
AIME 2025 (%)
Math (%)
Math-P (%)
AMC23
Average
Ori
SG
CS
Ori
SG
CS
Ori
Simple
Hard
Qwen3 Buse
53.5
SPTos.
32,5
21.3
14.2
27.9
18.5
11.5
80.5
65.5
81.2
66.3
75.6
45.6 (+22.7%%
SFTssn
32.1
17.3
31.7
20.2
11.9
78.8
80.0
65.1
74.7
47.0 (423.7%)
36.9
31.7
19.2
30,2
22.1
12.7
81.4
72.3
82.7
69.0
78.1
48.8(+25.5%)
Qwen3-8B-Base
13.3
7.5
2.9
10.2
6.0
1.2
58.5
46.2
61.1
38.9
54.4
27.3 0.0%)
SFTon.
44.4
35.4
20.0
32.7
21.2
15.0
83.7
74.0
86.9
73.6
83.9
52. (+24.8%)
SFTsyn
47-7
44.8
30.6
37-5
25.8
20.6
85.6
72.8
83.3
55.4 (427.996)
SPTMi
46.2
42.4
29.5
35,9
26.8
20.5
85.9
76.7
86.9
74.4
86.6
55.6 55.6(+)8.3%
Qwen3-14B-Base
14.8
11.0
4.8
10.2
7.1
1.2
60.8
50.2
63.5
43.9
55.9
29.4 0.0%)
SFTon.
50.4
39.0
25.2
41.7
25.2
20.4
85.9
73.4
85.5
75.0
89.1
55.5 (+26. 15%
30.3
88.7
70.3
(+3109) 61.3 (1.3 (+31..9))
SITTMs
56.5
62.5
38.8
47.2
34.6
26.5
86.7
77.8
88.2
76.1
89.8
Table 4: Performance of Qwen3 Base models after supervised fine-tuning. Each base model is compared with variants trained on original problems (SFTon): synthetic scenario problems (SFT syn), their mixture Mix) The best result within each block is highlighted in bold.
5.1 END-TO-END TRAINING
Data. (i) Original Data: We use DeepMath-103k (He et al. 2025), large-scale dataset of chal- lenging mathematical problems spanning Algebra, Calculus, and Geometry. Each problem paired with three solutions generated by DeepSeek R1; randomly sample one per problem as the ref- erence solution. (ii) Synthetic Scenarios: We generate contextual scenarios for DeepMath problems following the procedure in Section
To enable scalable data generation without manual verifica- tion, we use Owen3-322 3-23 to solve the generated scenarios and retain only those whose final answers match the original problems, ensuring high likelihood of equivalence. This yields 50k validated contextual problems with model-provided reference solutions. For fairness, we also restrict the original dataset the corresponding 50k subset when training.
Experimental Setup. We fine-tune the Qwen3-Base series under three training regimes for trolled comparison: (1) original data only (+SFTon- 50k), (2) synthetic scenario data only (+SFT syn- 50k), and (3) balanced mixture of both (4SFT 100k). All settings trained with the same number of steps for fair comparison. In addition to our own benchmark, we further evaluate models on AMC23 and Math-Perturb (Huang et al., 2025) assess whether the improvements generalize beyond our benchmark. AMC23 measures abstract mathematical reasoning independent of con- textualization. Math-Perturb tests generalization under data shifts, with two variants derived from level-5 MATH problems (Hendrycks et al. (221aa): Simple, which alters only non-critical surface parameters (e.g., numbers), and Hard, which modifies the underlying mathematical core. Imple- mentation details are provided in Appendix A.6
Results Table 4 shows that SFT markedly boosts contextual reasoning, e.g., Owen3-14B-Bas solves only 11.0% of AIME 2024-SG problems. but rises 52.5% with SFT Comparing regimes, scenario supervision syn)'syn. is more effective than original problems (SFToa.) on con- textual problems, showing that models must see narratives explicitly to acquire this skill. The mix- ture regime (SFTMix) provides the best overall balance-achieviie sizes-suggesting complementarity between abstract and scenario data. Beyond our benchmark, models also improve on AMC23 and Math-Pert Perturb, indicating that targeted scenario training does not degrade abstract reasoning and even enhances robustness to distribution shifts. Larger mod- els benefit more from scenario supervision, but even with SFT they solve under 40% of AIME 2024/25-CS problems, highlighting both clear progress and substantial remaining headroom.
the strongest averages across all
5.2 TRAINING A DEDICATED FORMULATION MODEL
With formulation identified as bottleneck, natural question is whether a model trained solely for formulation, combined with an existing solver, can improve contextual reasoning. This experiment offers preliminary test of that idea.

---

Published as a conference paper at ICLR 2026
Data. We use scenario oorigini pairs from Section 5.1 where each training instance maps textual scenario to its equivalent abstract problem.
3B and Qwen3 105---455 143 as formulation models, and also employ them reasoning models. We compare three settings: (i) w/o formulation: the reasoning model solves the scenario directly; (ii) formulation, training-free: the formulation model is used without fine-tuning, and its output is solved by the reasoning model; (iii) w/ formulation, trained: the formulation model is fine-tuned on scenario-original pairs before handing outputs to the reasoning model. At evaluation, sample one formulation per scenario and report the average accuracy of 16 reasoning outputs. Implementation details are provided in Appendix A.6 A.6
Experimental Setup. We fine-tune Qwer:
Results. As shown in Table 3 the best performance comes from solving scenarios directly (w/o formula- tion). Adding an untuned formulation stage leads to a slight drop, indicating that the pipeline introduces extra errors that propagate to reasoning. With a tuned for- mulation model, performance collapses further. Since training is stable and shows no signs of overfitting, we conclude that formulation is difficult to learn effectively from scenario- scenario-original pairs alone, and we leave the ex- ploration of other approaches to future work.
Reasoning Model
Formulation Model (Qwen3-8/14B)
Untuned
Tuned
8B
148
8B
14B
Qwen3-8B
48.9
53.4
22.3
Qwenn-14B
57.7
51.8
86.2
21.8
24.6
Table 5:
Average accuracy on Con- texIMATH. "w/o" denotes directly solv- ing the scenario with the reasoning model.
CONCLUSION
In this paper, we investigate LLMs' ability to perform contextual mathematical reasoning, where the mathematical core must be extracted from narrative context before solving. Through the Con- texIMATH benchmark, we shows that models which perform strongly on abstract math exhibit large drops on contextual variants, with errors dominated by problem formulation. Our analyses identify formulation and reasoning two complementary bottlenecks that jointly constrain performance. Although these processes may intertwine complex problems, our controlled framework provides a first step toward disentangling them and demonstrates that formulation constitutes a key obstacle. We further show that end-to-end training with contextual data improves robustness, while directly training dedicated formulation model is ineffecttive. Overall, our study establishes contextual math- ematical reasoning as a central unsolved capability of LLMs and highlights it as a key frontier for progress toward reliable real-world applications.
ACKNOWLEDGEMEEN
The work described in this paper is substantially supported by grant from the Direct Grant of Faculty of Engineering, The Chinese University of Hong Kong (Project Code: 4055275).
REFERENCES
Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi. Neel Joshi, Piero Kauffmann, Yash Lara, Caio César Teodoro Mendes, Arindam Mitra, Besmira Nushi, Dimitris Papailiopoulos, Olli Saarikivi, Shital Shah, Vaishnavi Shrivastava, Vibhav Vineet, Yue Wu, Safoora Yousefi, and Guo- qing Zheng. Phi-4-reasoning technical report, 2025a. URL
2504 21318
Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, et al. Phi-4-reasoning technical report. arXiv preprint arXiv:250..21318 2025b.
Bowen Cao, Deng Cai. Zhisong Zhang. Yuexian Zou, and Wai Lam. On the worst prompt per- formance of large language models. Advances in Neural Information Processing Systems, 37: 69022-69042, 2024.
10

---

Published as a conference paper at ICLR 2026
Ziling Cheng, Meng Cao, Leila Pishdad, Yanshuai Cao, and Jackie CK Cheung. Can LLMs rea- son abstractly over math word problems without CoT? disentangling abstract formulation from arithmetic computation. In Christos Christodoulopoulls, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 14306-1433 Suzhou, China, Novemher 2025. Association for Com- putational Linguistics. ISBN 977-8-9997998800 166-3332-6
doi: 10. 11263///1/222..mmmllll main. 723, URL wwtp////.2222. emnlp ma meain.... 723/
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint wrXiv:2110.41166 2021a.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021b. URL hhttps://arxix org/ 105//bs/220 /as./2100... 14168
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang Hc, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinforcement through implicit rewards, 2025. URL
01466
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, Tongkai Yang, Binhang Yuan, and Yi Wu. Areal: A large-scale asynchronous reinforcement learning system for language reasoning, 2025. URL https::/arxiv...
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint vrXiv:2501.1294. 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang. Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 3828-3850, 2024.
Zhiwei He, Tian Liang. Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Deepmath- 103k: large-scale, challenging, decontaminated, and verifiable mathemat- ical dataset for advancing reasoning, 2025. URL arw.v.iiiivv///ii///aaaaaaaaavva org/ abs 2504. 11456
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. arXiv preprint 2r 103.03874, 2021a.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021b. URL htt.t:ttp///tcc....tt /arxiv hbtgg//b///11 22033 03874
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Shum. Open-reasoner- zere:: An open source approach to scaling up reinforcement learning on the base model, 2025. htt/https: //ar:ivv/////vv/a//aa org/ abs/ 2503.5503..20 24290
Kaixuan Huang, Jiacheng Guo. Zihao Li, Xiang Ji. Jiawei Ge, Wenzhe Li, Yingqing Guo, Tianle Cai, Hui Yuan, Runzhe Wang, et al. Math-perturb: Benchmarking Ilms' math reasoning abilities against hard perturbations. arXiv preprint arXiv:2502.0644 2025.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https:::: //pppe/evi......
net/forurm iiddiidd== 6yNQM66
11

---

Published as a conference paper at ICLR 2026
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20788 2025.
Zihan Liu, Yang Chen, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acemath: Advancing frontier math reasoning with post-training and reward modeling. arXiv preprint, 2024.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qing- wei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09588 2023.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing 01 o1-preview with a 1.5b model by scaling rl, 2025. Notion Blog.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances in Neural Information Processing Systems, 37:124198-124235, 2024.
Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, and Igor Gitman. Aimo-2 winning solution: Building state-of-the-art math- ematical reasoning models with openmathreasoning dataset. arXiv preprint arXiv:2504.1666 16891, 2025.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025. URL http::/:///.xxx..o///////o 25301
OpenAI. Learning reason with Ilms, 2024.
OpenAl. Introducing gpt-5, 2025.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 2080-2099 2094, 2021.
George Polya. How to solve it: A new aspect of mathematical method. Princeton university press, 1945.
Cheng Qian, Hongyi Du, Hongru Wang, Xiusi Chen, Yuji Zhang, Avirup Sil, Chengxiang Zhai, Kathleen McKeown, and Heng Ji. Modelingagent: Bridging Ilms and mathematical modeling for real-world challenges. arXiv preprint arXiv:2505. 15068, 2025.
Alan H Schoenfeld. Mathematical problem solving. Elsevier, 2014.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL /ttpp://////axxi http:://wwwwiw..o.a///////ss wtt.:org//bbb///////
03300
Chufan Shi, Yixuan Su, Cheng Yang, Yujiu Yang, and Deng Cai. Specialist or generalist? instruction tuning for specific nlp tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 15336-15348, 2023.
Chufan Shi. Haoran Yang, Deng Cai, Zhisong Zhang, Yifan Wang, Yujiu Yang, and Wai Lam. A thorough examination of decoding methods in the era of 1lms. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 8601-8629, 2024.
12

---

Published as a conference paper at ICLR 2026
Yiyou Sun, Shawn Hu, Georgia Zhou, Ken Zheng, Hannaneh Hajishirzi, Nouha Dziri, and Dawn Song. Omega: Can Ilms reason outside the box in math? evaluating exploratory, compositional, and transformative generalization, arXiv preprint arXiv:2506. 2025.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URLhttps: qwwnnwwe lm.
github. ww/bloi.llogggo qwen2.57
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025a. URL httpps:/qwenlm. github.....t........ hatpa///oooooooogggggggg// 32b/
RUCAIBox STILL Team. Still-3-1.5-preeveen Enhancing slow thinking abilities of small models
through reinforcement learning. 2025b. URL httts:://gtthhii com/ http://hhwbcccc////C//////////./ RUCATBox/Slow
Thinking_wine LIMS
George Tsoukalas, Jasper Lee, John Jennings, Jimmy Xin, Michelle Ding, Michael Jennings, Ami- tayush Thakur, and Swarat Chaudhuri. Putnambench: Evaluating neural theorem-provers on the putnam mathematical competition. Advances in Neural Information Processing Systems, 37: 11545-1156$ 2024.
Mark Vero, Niels Mündler, Victor Chibotaru, Veselin Raychev, Maximilian Baader, Nikola Jo- vanović, Jingxuan He, and Martin Vechev. Baxbench: Can Ilms generate correct and secure backends? arXiv preprint arXiv:2502. 1144, 2025.
Yifan Wang, Yafei Liu, Chufan Shi, Haoling Li, Chen Chen, Haonan Lu, and Yujiu Yang. Insel: A data-efficient continual learning paradigm for fine-tuning large language models with instructions. arXiv preprint arXiv:2403.1 11435, 2024.
An Yang, Beichen Zhang, Binyuan Hui, Bofci Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvemmet. arXiv preprint arXiv:2409. 12122, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 009808850 2025a.
Cheng Yang, Chufan Shi, Siheng Li, Bo Shui, Yujiu Yang, and Wai Lam. Llm2:
Let large lan- guage models hamness system 2 reasoning. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pp. 168-177, 2025b.
Kaiyu Yang, Aidan Swope, Alex Gu, Rahul Chalamala, Peiyang Song. Shixing Yu, Saad Godil, Ryan Prenger, and Animashree Anandkumar. Leandojo: Theorem proving with retrieval- augmented language models. Advances in Neural Information Processing Systems, 36:21573- 21612, 2023.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025. URL https:: /ar/aaiivvvv....... org/ abs/2503 18892
Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. Minif2f: cross-system benchmark for formal olympiad-level mathematics. arXiv preprint arXiv:2109.00) 10, 2021.
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: Á realistic web environment for building autonomous agents, 2024. httth:t/p:/aaat org/ abs/ 2307 1354
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
ETHICS STATEMENT
This work studies LLMs through the lens of contextual mathematical reasoning. All data used are derived from publicly available math benchmarks (AIME and MATH-500), with additional scenario variants generated automatically and verified for mathematical equivalence. No private or sensi- tive data were used. We evaluate only open-source and commercially available models, and all experiments are conducted under non-interactive settings without user data collection. While our results reveal limitations of current L.Mss, these findings are intended to encourage more rigorous evaluation and robust model design, not to promote unsafe deployment. We will release our bench- mark openly to facilitate reproducibility and further research, subject to the same usage terms as the underlying datasets.
REPRODUCIBILITY STATEMENT
We take reproducibility seriously this work. The construction pipeline of ContextMATH, in- cluding all generation, verification, and revision prompts, detailed in Section Appendix(A.l Evaluation setups are described in Appendix [A.2] with complete accuracy results for open-source models reported in Appendix AA.3] and results for proprictary models Table IT The analysis of Implementation details of all training experiments are provided in Appendix [A.6] We will release our benchmark publicly, ensuring that independent researchers can fully reproduce our benchmark, analyses, and training results.
formulation metrics is presented in Section 4 with full breakdowns in Appendix 7
THE USE OF LARGE LANGUAGE MODELS (LLMs)
In preparing this paper, made limited use of LLMs to support the writing process. Specifically, LLMs were applied to check grammar, improve clarity, and polish the style of drafts. All technical content, experimental design, analysis, and conclusions was restricted to language refinement.
work, and the role of LLMs
A.1 PROMPTS FOR DATA CONSTRUCTION
This section provides the generation, verification, and revision prompts used to transform orig- inal AIME and MATH problems into Scenario Grounding (SG; see sections A.I.T to A.1.3) and Complexity Scaling (CS; see sections A...1) versions.
A.1.1 SCENARIO GROUNDING: GENERATION PROMPT
SG Scenario Generation Prompt
Your Goal: Convert an abstract math problem into a concrete, real-world story. The new story must be mathematically identical the original. You can draft the story first, then use the following steps to ensure accuracy and format your final output correctly.
[Step 1] Map All Mathematical Components
List every single mathematical element from the original problem: numbers, variables, shapes, and operations. For each, assign specific real-world analogue. Do not omit any numbers or symbols.
Bad Mapping (Vague): x -> initial quantity
Good Mapping (Specific): x The initial number of barrels of oil
Bad Mapping: sqrt(2) - a multiplier
Good Mapping: sqrt(2) - special efficiency boost calculated the square root of
14

---

Published as a conference paper at ICLR 2026
SG Scenario Generation Prompt (continued)
Output Format for Step
[Step 1J [Mathematical Element] + [Specific Real-World Analogue]
[Step 2] Define the Real- World Rules of Interaction
Combine the analogues from Step 1 into coherent narrative context (e.g., engineering, physics, logistics). Explicitly define how the mathematical operations translate into actions rules within your story. This step bridges the gap between individual components and the final narrative.
Example Math: f(x) = 2x +
Example Rule: "The output of machine for any given input found by doubling the input value and then adding fixed calibration offset of 3."
Output Format for Step 2: [Step 2] [Mathematical Operations] > [Specific
Rule]
[Step 3] Write the Final Problem
Combine the elements from the previous steps into concise story. Strict Requirements:
Perfectly Equivalent: All numbers, relationships (like perpendicularity), and constraints from the original problem must be present and unambiguous in your story.
No Math Language: Avoid mathematical terms (triangle, variable, x, cos) and symbols. Use descriptive names (a triangular field, the initial investment, cosine-wave).
Clear Question: The final question must ask for the same value as the original (with the same output format).
Output Format for Step 3: [Step 3] Question: [The complete, final real-world problem in plain text.]
Now, apply this process to the following problem: {original probbeem placeholder)
A.1.2 SCENARIO GROUNDING: VERIFICATION PROMPT
SG Scenario Verification Prompt
You are a quality checker. Your task is to verify a real-world story is a correct and clear representation of original math problem.
Your Primary Goal:
Ensure perfect mathematical equivalence in clear and concise narrative. The story does not need be a hyper-realistic scientific model; only needs to be coherent and unambiguous scenario.
Please review the provided materials by considering these key questions:
Mathematical Integrity: Does the story perfectly preserve all numbers, relationships (e.g.. perpendicularity, equality), and operations from the original problem? Is anything missing or changed?
Clarity & Language: Is the story easy understand? Does it successfully avoid math- ematical jargon and symbols, using clear descriptions instead? the final question unam- biguous?
2.
3. Conciseness: Is the story direct and to the point, without unnecessary details that over- complicate the problem?
15

---

Published as a conference paper at ICLR 2026
SG Scenario Verification Prompt (continued)
Your Output:
Please provide your review brief summary of your findings. After your summary, conclude with the mandatory [Overall Assessment] line.
Here the output Format:
[Your findings] [Overall Assessment] [Pass/Fail]
Here is the input for your review:
1. Original Math Problem porginal.oobleen.paccloddd
Conversion Mappings:
2.1 Concept Mappings: {concept mappings list lissltsssscceeee placcchcchlldr
2.2 Relationship and Attribute Mappings: (relationship mappings list placeholder)
3. Real-World Problem: (real_wer world felllddprobbmeeeeee ttxt_eeetpeee_ee placeholder)
A.1.3 SCENARIO GROUNDING: REVISION PROMPT
SG Scenario Revision Prompt
Carefully analyze the following feedback to revise your previous response to address all problems.
Here the feedback: (feedback placeholder)
Output your revised version in plain text in the following format:
[Step 1]
[Mathematical Element] - [Specific Real-World Analogue]
[Step 2] [Mathematical Operations] + [Specific Real-Worke World Rule]
[Step 3] Question: [The complete, final real-world problem in plain text.]
A.1.4 COMPLEXITY SCALING: GENERATION PROMPT
CS Scenario Generation Prompt
Task:
Enhancing Math Problem Difficulty through Contextual Embedding
Your goal is to increase the difficulty of a given mathematical problem's real-world version, while preserving its core mathematical structure and the integrity of its real-worle mapping. Employ the following strategies:
Maintain Core Alignment
Ensure all mathematical quantities, variables, and relationships directly correspond to elements within the real-world context. The underlying mathematical problem must remain solvable and equivalent to the original.
16

---

Published as a conference paper at ICLR 2026
CS Scenario Generation Prompt (continued)
Embed Conditions through Layered Obfuscation Replace direct mathematical statements and values with more natural, layered, and context- rich descriptions that require deductive reasoning from the solver.
A. Contextualized Numerical Properties (Disguising Specific Values):
This technique involves disguising specific numerical values by describing them through a property they possess within a realistic, tangible scenario. The goal is for the solver derive the original number based solely the provided context, using common knowledge or fundamental mathematical principles. Avoid creating custom or overly ob- scure scenarios that would require information beyond general understanding. Apply this method only when it naturally fits the number in question; don't "on't force its application.
Example (for the value '4' hours): Rather than stating "lasts hours," " describe as "lasts a number of hours equivalent to the unique combinations on the drone's 's two two-digit pre-flight security keypad, where each digit can only be '1' or 22"." (This naturally leads to
2 x 2 4 combinationsshours)
* Example (for the value ' "The unique positive whole number that, you square it, then subtract time itself, and finally subtract two, the result is zero."
& Example (for the value "6""): "smallest integer such that C! is divisible by 255
B. Relational & Functional Reverse Engineering (Disguising VariablessFFrmulas))
If the original problem involves analyzing a function with specific constants (e.g..
fa) (ar (z B)...,, make these constants unknown (e.g., describe them as "labels are "lurred,",""caa "'alibration is missing", etc.).
Provide a minimal set of realistic data points or observed behaviors sufficient for the solver to uniquely determine these unknown constants or the full function's 's form.
Example: Instead of giving f(x) (==18)(=-72)... describe it as system's 's perfor- 2
mance, providing P(22) and P(33) values, requiring deduction the hidden constants 18 and 72.
C. Structural & Conceptual Reinterpretation (Disguising Geometric/Abstrac Rela-
tions):
For geometric problems or abstract relationships, rephrase mathematical properties in terms of tangible real-world structures or processes.
Example (Symmetry): Describe figure's 's symmetry as "a balanced design around central axis" or "perfectly mirrored components." Then, we only describe half of the plot. Example (Transformations)) If there scaling rotation relationships, we can only describe a smaller prototype that a scaled replica of other graphics.
Example (Equivalent Constraints): Reinterpret mathematical constraints using -cenario-specific terms. For example, "a movable point lies along the perpendicular bisector of AC" instead of AL CLL. Another example is translating 2g + 3%
to 2 cos 6g, if we let
Ik = i0k
3. Introduce Plausible, Irrelevant Information
Incorporate details that are contextually relevant to the scenario but mathematically extra- neous to the problem's 's solution. This increases cognitive load and realism. Example: For drone problem, mention its battery capacity, wing span, payload weight, or the color of the packages being delivered, provided these details do not influence the distance-speed-time calculations.
Language Refinement and Simplicity:
Make sure all descriptions are concise and clear. Avoid redundancy and unnecessary repeti- tion.
17

---

Published as a conference paper at ICLR 2026
CS Scenario Generation Prompt (continued)
Avoid using mathematical variables (such as x, k, A, B, etc.) directly in the situation de- scription, but give them specific real-world meanings (such as temperature, size, threshold, etc.).
Output Format:
First think about how apply these strategies the math problem. Then output the enhanced problem in plain text after "Enhanced Contextual Math Problem:".
Here's ' the input:
Original Math Problem: foriginal problem pla.eemmllcchhddrr
Real-World Version: Treal_world, www.llpooooooeee- wttptteeetppppppe_eeete text_placeholder
A.1.5 COMPLEXITY SCALING: VERIFICATION PROMPT
CS Scenario Verification Prompt
You are tasked with verifying the quality and accuracy a real-world version an abstract math problem.
Issue Types Defined:
EQUIVALENCE, ERROR: The enhanced problem's mathematical core equivalent to the original, or it's unsolvable. Crucially, you must solve any nested or abstract sub- problems to confirm the derived mathematical information uniquely matches the original problem's data.
Example Suggestion: "The definition for 'standard mission duration' should lead to value of 4. Consider linking it to a 2 x matrix operation if the context allows."
REALISM.ERROR: The context embedded condition feels contrived, unnatural, or like a "brain teaser" and isn't plausible real-world scenario.
Example Suggestion: "Instead of overly abstract obscure derivation for value (e.g., a specific dimension or time), consider linking to more common measurement, a simple calculation based widely known facts, basic geometric property that naturally fits the scenario."
CONCISENESS ERROR: The problem statement is unnecessarily long, verbose, or unclear.
Example Suggestion: "Condense the description of the speed increment's derivation by directly stating the polynomial's property more concisely."
The output doesn't follow specified formatting (e.g., uses math variables, incorrect header, ask for different value). Example Suggestion: "Avoid using variable m in the context."
FORMAT ERROR:
OTHER: Any other issues.
Review Output Format:
Start your output with an 'ASSESSMENT' status. If the 'ASSESSMENT' 'FAIL', you must provide 'FEEDBACK in the specified parseable format.
[Overall Assessment]
[Pass/Fail]
18

---

Published as a conference paper at ICLR 2026
CS Scenario Verification Prompt (continued)
ASSESSEENT is FAIl., provide FEEDBACK like this:
[FEEDAACK]
LYSUE
-DESCRIPTION: [Concise deseription of the specific issue.]
-LOCATION: [Optional: Specific phrase section from the problem.]
ISSUE_TYPE: [Another ISSUE_TYPE applicable]
-DESCRPTTINN: [Another concise description.]
Here the input:
Original Math Problem (original_problem_ placeholder)
Real-World Problem Scenario: (real_ pworlee.robemmmeem pww..pccecccoodrrrrrrr
A.1.6 COMPLEXITY SCALING: REVISION PROMPT
CS Scenario Revision Prompt
Carefully analyze the following feedback to revise your previous response to address all problems.
Here the feedback: {feedback plachoold..a
Output Format:
Output the enhanced problem plain text after "Enhanced Contextual Math Problem:".
A.2 MODEL. EVALUATION DETAILS
All open-source models tested were sourced from the Hugging Face Hub. We used the optimal generation parameters (e.g., temperature, top-p, top_k) for each model, following the configurations recommended in their respective papers official GitHub repositories. The maximum response length was set to 32,768, or capped at the model's 's maximum supported length if shorter.
All proprietary models were evaluated using their official APIs publicly available client interfaces. Unless specified otherwise, inference was conducted using greedy decoding (temperature = 0) for reproducibility (Shi et al. 2024). with maximum response length of 32,768 tokens.
Some proprietary models were accessed via their web or client interfaces using default parameters during specific time windows. These include Copilot, Grok3, and Gemini (2.5 flash and 2.5 pro), which were evaluated for AIME 2024 between June 4th and June Ith, 2025, and for AIME 2025 between June 18th and June 24th, 2025.
For models accessed via API, we used their specified versions. These include Doubao-1.5-thinking pro (version 2025-04-15 16k max tokens), Qwen-max (version qwen max- 1eag-laaestt on pplus-latest
2025-04 -9,, 8k max tokens), QwQ-plus (version qwq-qwqqqooeooe cauus-ue latest on 2025-03-00 8k max tokens), DeepSeek-R1 (version 250528, 16k max tokens), gpt-4o-mini (2024-07-16), ol-mini (2024-09-12), gpt-4o (2024-08-06) gpt-4.1-mini (2025-04-14), gpt-4.1-nano 2202500-1444 4), and (2025-20 04-16).
o1 mi and GPT-5 were accessed via shared endpoints that only support temperature of 1.0.
A.3 FULL RESULTS FOR OPEN-SOURCE MODELS
Table reports the complete ContextMATH results for all evaluated open-source models, includ- ing: http://........7/////////B/BBB/ (Yang al. 2025a), Owen2.5-Math-115/7 Qwen2.5-
19

---

Published as a conference paper at ICLR 2026
Model
AIME2024 (%)
ATME2025 (%)
Math-500 (%)
Ori
CS
CS
Ori
<4B
Qwen3-0.6B
6.2
-21%
95%%
5.4(
-84%
-32%
Qwen2.5 M.---h---.BBB
11.7
2.5
79%
0.41
96%
4.6
0.8
.2%%
0.0
100%
32.5
26.8
-11%
R Distil Swelil--eeell.
28.3
20.00
-9%
7.10
75%
19.6
9.6(
70%
74.0
57.6
-22%
DeepSealeR-I.558 Prr-rriiw
41.7
23.3
-44%
7.5
-8%%
30.4
16.4
-49%
7.5
-75%
80.1
64.2
-20%
DeepMathhI..8
38.3
24.0(
-36%
10.81
-72%
28.3
13.8
-51%
5.0
-82%
81.9
65.4(
-20%
Still-3-1..55 15B-Preview
38.3
18.8
-51%
7.91
-79%
24.6
11.2(
-54%
2.5
-90%
76.8
09.7
-21%
62.5
34.2(
-5%%
14.61
-77%
50.4
21.2(
-58%
11.2
-78%
87.3
70.1
-20%
- -DeepMath-Omo-1155
02.9
33.8
-46%
13.8
-78%
56.8
20.8
-63%
12.5
-78%
87
73.0
(16%
Qwen3-1.7B
46.2
29.61
-36%
12.51
-73%
34.6
24.6
11:2
-67%
84.1
67.4( -20%
Qwen3-4B
70.4
52.5
-25%
34.6
-51%
61.2
39.6
-38%
33.8
-47%
90.9
78.8 -13%
7R/SB
Qwen2.5-7B
100%
87%
43.9
-31%
+Open Zeto-ner-Zeto-88
15.8
12.5 -21%
6.2(
61%
14.6
7.1
-51%
2.5
83%
69.3
53.0
-33
DDDepMatb-Zer--78
87%
74.2
59.6
-20%
Qwen2.5-Matb-78
10.8
6.7
-38%
1.7(
-55%
5.0
8.3(
-33%
0.8
-33%
44.8
36.7(
-18%
OpenMaul-Nemotrrrt
52.1
-29%
30.00
55%%
40.40
90.5
+RI RI-Distl-Qwen--7
18.8
40.0
-18%
23.3(
-52%
41.7
22.5
-46%
15.0
-64%%
87
73.9
-15%
AceMath Ne.ootmmm 7B
69.2
48.8
32.91
-52%
54.2
26.7(
22.5
58%
89.9
79.0
Qwen2.5 Swen2.5-Math-766 Math-BB Zero
16.2
6.7
-59%
0.8
955%
4.6
3.3(
27%
0.0
100%
49.6
33.4
-33%
SRL
20.8
9.21
17.5
10.4(
2.9
83%
70.7
58.3
DeepMath Zero Math 7B
30.4
18.8
-38%
9.2(
.70%
22.1
15.8
28%
7.1
68%
77.4
63.8
188 (18%%
QwenZ.5.nne M-aahh Out Zeru
-42%
11.2
96%
66.2
-33%
- Eurus-2-BB PRIME 2-7--PRMME- Zero
20.0
13.3
-33%
6.7(
66%%
6.7
10.00
-50%
0.0
100%
59.9
39.7
-34%
Eurus-2-BB-SPT
3.33
-10%%
0.0
50.0
38.5
-23%
- --rrrs---7B-PRIM
20.0
13.3 (
-33%
6.7(
-67%
10.0
3.31
67%
3.3
-67%
68.3
57 ( -16%
RI-Distil-Tlama--B
43.3
29.2
-33%
10.81
-75%
30.4
17.5
11:2
-63%
81.6
65.6
-20%
Qwen3-8R
73.8
61.7
.16%
42.91
.42%
64.6
48.3
-25%
35.8
45%
91.0
81.0
-11% (11%)
DDeepSeck-RR Ba--sekk-R1-5228--8 -10.--R11--28-20033-11 Qwen3-8B
75.0
55.0
39.61
65.8
48.3
32.9
-50%
90.7
77.2
-15%
--ARRal.-bohut 2-8B
74.2
58.3
-21%
41.7(
.4%
67.9
47.9
-2%
37.1
45%
91.8
82.0
(-1%
-11%
Qwen2.5 Math
6.2
2.51
778%
9.2
5.0
0.0
72.0
54.0
225%
14B
Qwen2,5-14R
6.2
3.8
.0%
1.71
-73%
3.3
1.7(
-50%
0.0
100%
48.5
34.9
-28%
< -RI--issil-Qwen
67.5
47.1
30%
35.41
.48%
50.8
26.2(
-8%
25.8
49%
89.5
76.9(-14%
-14%
OopenMath-Nemarrnn
73.8
51.7
.30%
42.1
.3%
63.8
42.5
-33%
29.2
.4%%
90.8
80.8
-11%
Qwen3. 14B
80.0
64.6
-19%
50.81
.36%
72.9
49.2(
-33%
42.1
42%
92.6
81.9
112%
-+ A+Rcall A.ccaL-bbba-2---8
82.9
65.8
-21%
53.8
-35%
73.3
52.1
-29%
39 (
-17%
91.9
(
-10%
thi----cssonnng-pluus
80.4
60.4
(-25%
-25%
-34%
71.7
55.4(
-22%
39.6
-45%
83.1
-10%
2.32
Qwen2.5-32B
11.2 6.7
-11%
3.3(
-70%
3.88
2.9(
-22%
0.0
100%
45.2
37 8
-16%
57.1
42.5
266
27.9(
-51%
52.1
31.6
-34%
27.3
-7%
75.8
61.8(
118%
+ Ri--Disil-QQwaa-32B
69.6 52.5 -5%%
39.2(
-44%
56.2
39.6(
-30%
30.0
47%
89.4
78.9
-12%
Opann-Rasonnor-2oo--28
13.8 30.1 ( (30%
25.4(
-42%
32.9
25.4(
-23%
15.00
-54%
81.3
75.1
(
Qwen3-32B
812 67.9( -16%)
37.1
-30%
70.0
54.4
-22%
45.0
-36%
92.1
82.7(
-10%
ARcal--boba-2-328
81.7 65.4 -20%
58.3
-20%
77.1
55.0
-29%
43.8
-43%
92.3
82.9
-10%
QwQ-32B
80.4 58.3 ( -27%
53.3
-34%
66.2
53.3
200%
39.2
-11
92.5
82.9
-10%
Qwen2.5-32B-Insimen
15.4 14.2
-8%
5.4(
-65%
15.8
9.2(
-42%
1.7
-89%
70.3
54.5
-3%
-+11.1-32B
58.8 47.91 ( 118%
38.3
( -35%
46.7
36.2(
-22%
322.1
-31%
88.8
77.9
-12%
DeepSeek-He RI-Distill Liama--0
65.4
48.8
-25%
388.8
-41%
50.0
38.81
-22%
29.2
-42%
89.8
77.3( -14%
Qwen2.5-Mail Maanh..--att-728- -20--tt2--tttt
19.2
13.8
-28%
6.
-65%
14.6
7.3
-99%
3.33
-77%
74.7
60.5 -19%
Table 6: Full results of open-source models on ContextMATH. Models are grouped by parameter scale. Parentheses indicate the relative change from Ori, with larger drops highlighted by deeper red. Arrows (->) denote variants obtained from the model listed directly above.
20

---

Published as a conference paper at ICLR 2026
Math-7BB72B-Instunn at. DeepSeck-RI-Distee / R1-Distil
wwt.:/8//.B.D.peeeece.ee DeepSeck-RI-0525 0aap--00528--033----3 (Guo et al. 2023 DeepSealeR 1.5---reviiPww (Luo
et al.
2025).
Still33-1.B--reviine
(Team 2025b), Phi- -ha---raasn plus (Abdin et al. 2025b),
Qwen2.5-32BBIInstrre
(Team 2024), Open-Reasoner-TZnno 7B332B (Hu
al.] 2025).
DeepMath-1155h
DeepMath-Zero-7hh DeepMath-Omnn
(He et al.) 2025)
(Moshkov et al., 2025), AceMath-RL-Nemoree
(Liu
al. 2024), Qwen2.5-Math-7B-SRL- SRL--eero, (wenn.5-MMahh7B-SRR.
(Zeng al. 2025). Qwen2.5-
Math-7B-OOtt-erool (Liu et al. 2025), Murus-2-7B-PRIMAA
Burus-2-7B-SFI
Eurus-2-7B-
PRIME (Cui et al. 2025), AReaL- 2aaal-ooba---BB1444 2-8/1/BBBB (Fu al. (025). thi-4-reasoning-llue (Ab-
din et al., 2025a). QwQ332B (Team. 2025a), and s1.1-32B (Muennighoff et al., 2025). Across
architectures and training variants, accuracy drops sharply on both SG and CS tasks compared the original problems, with the CS setting consistently causing the largest degradation. Larger models generally achieve higher accuracy, yet significant performance gaps remain. We also observe that while an initial stage of supervised fine-tuning tends to improve robustness, additional specialized tuning (e.g., RL or further SFT) often yields little benefit for contextual tasks (Luo et al., 2023]
Shi et al. 2023 Shao et al.. 2024 Wang al.) [2024] Meng al. 2024 Yang al.) [222bb] Guo et al. 2025) Overall, these expanded results provide a comprehensive view of model behavior that underlies the key findings discussed in Section 3.4
A.4 REASONING ERROR EXAMPLE OF DEEPSEEK RI
We provide an example in A.4.1 to illustrate a typical failure mode DeepSeek R1. The model's algebraic reasoning was largely correct, but it misinterpreted how geometric constraints in the see- nario map back to the original problem. leading to an incorrect answer. This case highlights how formulation errors can propagate when subsequent reasoning valid.
A.4.1
CASE STUDY: MISINTERPRETING GEOMETRIC CONSTRAINTS
An example of R1 reasoning error
Original question:
Rectangles ABCD and EFGH are drawn such that D, E, C, are collinear. Also, A, D. H. G all lie on circle. If BC 16,AB 107,FG 17, and EF 184, what is the length CE?
Generated Scenario:
At large construction site, rectangular platforms are being assembled for robotics demonstration. The first platform, called the Alpha Pad, is 107 meters long and 16 meters wide. The second platform, the Echo Deck, 184 meters long and 17 meters wide.
The platforms are being arranged in very specific layout:
The back-right corner of the Alpha Pad, the front-left corner of the Echo Deck, the front-right corner of the Alpha Pad, and the front-right corner of the Echo Deck all lie along this straight track.
Meanwhile, the back-left and the back-right corners of both platforms are all positioned along the edge a circular fountain that marks the center of the plaza. Given this setup, how far apart are the front-right corner of the Alpha Pad and the front-left corner of the Echo Deck?
R1 response:
The platforms are arranged such that the back-right corner of the Alpha Pad (BR.A), the front-left corner of the Echo Deck (FL.E), the front-right comerr of the Alpha Pad (FR_A), and the front-right corner the Echo Deck (FR_E) lie on straight track. Additionally, the back-left and rack-right corners of both platforms lie on circular fountain centered at the plaza's center.
21

---

Published as a conference paper at ICLR 2026
example of reasoning error (continued)
The Alpha Pad 107 meters long and 16 meters wide, and the Echo Deck is 184 meters long and 17 meters wide. The straight track is treated the x-axis in coordinate system, with the fountain center at (h, k) and radius r. The back corners on the fountain and the points the track are used to determine the positions. For the Alpha Pad:
-BR_A is at (0, 0).
FR is at (107,0) 0). as the distance along the length is 107 meters.
BL. A is (0, 16), assuming the platform is above the track (y 0). This gives k = from the circle conditions, and 1 64.
For the Echo Deck:
-
-FL. is at (b, 0).
-FR_E is at (b 17, 0), the width is 17 meters, Incorrect mathematical interpretation
BL (b, 184), assuming the platform is above the track (y > 0).
The subsequent reasoning process is correct, but based on wrong conditions,
GPT-5 analysis:
Primary error type: Incorrect mathematical interpretation (with a secondary logical/rounding
issue at the end).
What went wrong:
1) Misassigning which side of the Echo Deck lies along the straight track.
2) Misassigning which Echo Deck side forms the chord on the circle.
3) Additional logical issue (unjustified "nice fraction".)
How this led to the wrong answer:
By putting 17 along the track instead of 184 and using 17-length chord for the circle instead of 184, the computed b (the x-position of FL E) shifted incorrectly, producing [107-
bl z 60.5, rather than the correct CE 104. The mistake is structural (wrong mapping of sides to the line and circle), not just arithmetic.
Model
Formulation Accuracy (%)
Formulation Necessity (%)
Formulation Sufficiency (%)
AIME24
AIME25
MATH
AIME24
AIME25
MATH
AIME24
AIME25
MATH
SG
CS
SG
CS
SG
SG
CS
SG
CS
SG
SG
CS
SG
CS
SG
RI-Distil-Qwea-LLBB
57.5
40.0
49.2
31.7
62.3
61.4
43.8
41.7
75.0
70.6
17.5
10.1
9.1
15.9
65.0
11-Distil-Qwen--BB
60.0
55.8
53.3
40.8
75.7
67.8
66.2
63.2
82.9
46.3
31.1
26.7
RI RI-Distil-Qwen-145
74.2
63.3
53.3
50.0
85.7
80.8
71.7
78.3
81.3
90.1
53.7
42.2
31.7
42.3
80.8
RI C1-Distii-O0 DistiilQwenn-3 Qwen-2BB
77.5
65.8
59.2
60.0
87.3
80.1
77.0
69.8
78.0
91.8
51.7
45.9
46.1
39.4
83.5
Qwen3-0.6B
46.7
29.2
45.8
35.0
$7.4
52.8
100.0
33.3
25.0
69.2
8.4
3.6
1.5
1.9
52.2
Owen3-1.7B
66.7
47.5
60.0
53.3
82.5
7.3.7
84.5
68.4
83.3
90.1
32.4
24.8
27.6
14.1
73.8
Qwen3-4B
74.2
55.0
51.7
42.5
84.8
91.1
79.9
63.7
71.0
90.3
65.5
48.8
50.7
57.8
83.6
Qwen3-8B
81.7
74.2
66.7
60.0
86.3
89.8
83.6
77.3
77.2
91.3
66.0
49.3
54.2
47.7
86.3
Qwen3-14B
77.5
68.3
66.7
62.5
87.1
82.1
74.8
81.0
95.8
90.4
64.6
57.4
62.4
62.0
85.2
Qwen3-32B
84.2
70.0
72.5
59.2
89.3
80.5
74.0
86.2
76.6
92.0
66.4
59.6
$7.5
55.7
85.0
gpt-5
93.3
76.7
80.0
66.7
90.5
96.0
79.2
83.3
75.0
94.5
85.7
82.6
83.3
75.0
86.9
Table Problem formulation performance comparison across different models. Formulation Aceu- racy denotes the proportion of cases in which model correctly translates scenario into its mathe- matical formulation. Formulation Necessity quantifies the extent to which correct formulations are required for correct reasoning. Formulation Sufficiency evaluates whether correct formulations reli- ably lead to correct reasoning. Darker cell colors correspond to higher values.
A.S FULL RESULTS FOR PROBLEM FORMULATION ANALYSIS
Table reports the complete formulation performance results. The finer granularity confirms the same overall trends: formulation accuracy decreases with problem difficulty, necessity consistently exceeds accuracy, and sufficiency improves with scale but remains lower overall. These detailed
22

---

Published as a conference paper at ICLR 2026
results provide a comprehensive view of model behavior underlying the averaged metrics discussed in Section
A.6 IMPLEMENTATION DETAILS.
We fine-tuned the model in the full-parameter SFT setting on 4xA 100 80GB GPUs. We adopted per-device batch size of I, accumulated over 32 steps, yielding an effective global batch size of 128. The model was optimized for 1200 steps with cosine scheduler, 10% warmup, and peak learning rate of 5e-5. Mixed-precision training with 6 was enabled.
23


Reviews:
[{"review_id": "XmBkMOj0S5", "reviewer": "Reviewer_MmP5", "summary": "The paper investigates LLMs’ mathematical reasoning in contextual scenarios, where the underlying math must first be formulated from narrative descriptions before being solved. It introduces CORE-MATH, a benchmark that repurposes AIME and MATH-500 problems into two controlled variants: Scenario Grounding (SG), which embeds problems in realistic narratives without altering the core math, and Complexity Scaling (CS), which hides explicit conditions behind simple sub-problems to mimic how constraints appear in practice.\n\nAcross 61 models, the authors observe substantial accuracy drops from the original abstract problems to SG—and even larger drops on CS—implicating problem formulation as a primary failure mode. Training experiments indicate that fine-tuning on scenario data improves robustness, whereas a formulation-only pipeline is ineffective.", "questions": "Since AIME and MATH datasets may have already been heavily used or augmented during LLM training, how would the results differ if the authors included other benchmarks, such as MathOdyssey?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The problem is interesting, as it focuses on realistic contextualization of mathematical reasoning.\n\nThe paper introduces a new benchmark, CORE-MATH, which is built upon AIME 2024, AIME 2025, and MATH-500 datasets.\n\nThe proposed Scenario Grounding (SG) and Complexity Scaling (CS) strategies are effective for constructing contextual mathematical problems that test both problem formulation and reasoning capabilities.", "weaknesses": "Is the mapping from the original problem to the narrative automatic or human-assisted?\n\nHow do the authors ensure that the mapping is accurate? The generated narrative may not exactly match the original mathematical problem, which could alter its meaning.\n\nThe Scenario Grounding (SG) and Complexity Scaling (CS) strategies are used for data construction. How do the authors guarantee that these transformations do not change the underlying problem semantics?\n\nWould it be possible to train models on similar data types so that their performance can be better preserved across contextualized settings?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "crq0B5eh5R", "reviewer": "Reviewer_YVi6", "summary": "The authors introduce CORE-MATH, a new benchmark that repurposes problems from established sources like AIME and MATH-500 into more realistic, narrative-driven scenarios. This is done through two variations: \"Scenario Grounding,\" which embeds problems in a narrative, and \"Complexity Scaling,\" which conceals conditions within sub-problems. Through an extensive evaluation of 61 models, the paper demonstrates a significant drop in accuracy on these contextual tasks. The primary cause of failure is identified as \"problem formulation\"—the inability to correctly extract the core mathematical structure from the narrative. The authors find that while model scale helps, it doesn't solve the issue, and that end-to-end fine-tuning on scenario-based data improves performance, whereas training a model solely for formulation is ineffective.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work tackles the highly relevant and important gap between benchmark success and practical, real-world capability in LLMs, pushing research beyond abstract problem-solving.\n2. The CORE-MATH benchmark is well-designed. By building upon trusted sources (AIME, MATH-500) and systematically creating two distinct types of contextual challenges (SG and CS), the authors provide a controlled framework for analyzing this problem.\n3. The study is thorough, evaluating a wide array of 46 open-source and 15 proprietary models. The analysis goes beyond simple accuracy metrics, effectively identifying problem formulation as the key bottleneck through both qualitative and quantitative evidence.", "weaknesses": "The primary weakness of the work lies in the lack of detail regarding the benchmark construction process, which is critical for ensuring the benchmark's validity and reproducibility. The paper states that an LLM (01-mini) was guided by structured prompts to generate the contextual variants, which were then reviewed by human experts. However, several key aspects unclear:\n1. Why choose o1-mini but not stronger models? Did the authors compare its performance with other frointer LLMs?\n2. What was the protocol for the human expert review to guarantee mathematical equivalence? How many experts reviewed each problem, and what was the procedure for resolving disagreements to ensure the final scenarios were valid?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vHPSiR3MIH", "reviewer": "Reviewer_qQ6S", "summary": "This paper introduces CORE-MATH, a benchmark for evaluating LLMs on contextual math reasoning. The benchmark converts problems from AIME and MATH-500 into scenario-based narratives and more complex narratives with embedded sub-problems, Complexity Scaling. The authors evaluate 61 and proprietary open-source LLMs and find that performance drops sharply when moving from abstract to contextual settings. Through error analysis and targeted experiments, the main limitation is identified as incorrect problem formulation rather than failures in step-by-step reasoning. The paper also examines how fine-tuning and formulation-specific training can reduce these errors and shows gains, but a large gap remains. The work demonstrates that contextual math reasoning is still an open challenge for current LLMs.", "questions": "- The proposed SFT setup helps but does not remove the drop under CS. The paper positions this as a first step, but the current training recipe and analysis do not yet show a clear path to solve the remaining gap. What suggestions do the authors have to further address this research gap, or how would the authors justify the limited improvement through their method?\n\n- Typo: the caption reference \"Figure 4\" should be \"Table 5.\"", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The research question is well motivated and the main finding is relevant. The paper provides a useful observation: the authors identify contextual complexity as a general bottleneck that limits current LLMs' reliability on multi-step reasoning.\n\n- The paper provide insightful error analysis. The detailed breakdown of error types (Figure 2) demonstrates that formulation errors predominate across architectures in contextual settings, which is often overlooked in prior work.", "weaknesses": "- The paper builds several automatic annotation and categorization steps and relies on an LLM judge with only light human checks. The authors state that they use o1-mini to decide whether a model output is mathematically equivalent to the reference solution, and that manual checks on Qwen3-14B and Qwen3-32B show more than 90% agreement with human judgments. However, the paper does not report the exact sample sizes, selection protocol, prompts, or agreement statistics, e.g., Cohen's kappa, Pearson/Spearman correlation of scores, or confidence intervals. The heavy use of an LLM-as-judge without full validation details becomes a concern.\n\n- The error analysis uses GPT-5 to assign error categories to outputs from other models, but the paper does not provide the prompts or templates that define each category. Also, there is no study of judge bias or stability, e.g., prompt ablations, temperature sweeps, and no human expert evaluation. A small human audit with inter-rater agreement would be useful. \n\n- CORE-MATH draws from AIME-2024, AIME-2025, and a filtered subset of MATH-500, with each original item converted into two variants, SG and CS. This means the total pool starts from a relatively small number of sources. The paper probes robustness by adding two extra SG versions for AIME-2024 and reporting an averaged score, SG Avg@3, but it does not report standard deviation or standard error, and it does not repeat this check for CS or the other subsets. It is not clear whether that the results could be sensitive to specific paraphrases; Another concern is the limited scale in CORE-MATH, which makes results less convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J5LKgQHx7F", "reviewer": "Reviewer_xYoD", "summary": "This paper tackles a very relevant gap, i.e. models that perform well on abstract math often struggle when the same problems are presented in a realistic narrative. The authors build a benchmark by transforming existing items from AIME and a filtered MATH-500 subset into two contextualized variants. The first, Scenario Grounding (SG), wraps the same core math in a short story while keeping the solution unchanged. The second, Complexity Scaling (CS), conceals some explicit conditions behind small sub-problems, requiring the model to first identify what is needed before solving. They evaluate a large set of models and observe a clear drop from the original to SG and a further drop to CS. They also analyze where errors originate and introduce three formulation metrics (formulation accuracy, necessity, and sufficiency) to distinguish between “setting up the right math” and “doing the math.”. The authors test training strategies and find that simple SFT on scenario-style data gives measurable improvements without harming abstract performance. \n\nI see the contribution as timely and practically important. The benchmark design, although not perfect, addresses a real need: testing whether models can extract the correct structure from a story. The formulation-vs-reasoning framing is helpful for analysis, and the training results give a simple recipe that others can try. The novelty is, however, moderate (a new benchmark plus analysis), but the empirical scope and the focus on formulation make it valuable to the community, provided the evaluation is strengthened.", "questions": "1) Could you report inter-annotator agreement and a small difficulty-parity audit for SG, and a clearer rubric for CS to show it does not change the task class? \n2) Would you consider re-scoring a stratified subset with a different judge family, some exact/symbolic checks where possible, and a larger blinded human pool?\n3) It would also help to provide confidence intervals or bootstraps, and to balance sampling across model groups (or clearly separate the summaries). \n4) Please comment on step-level near-duplicate analysis and on the sensitivity of the SFT gains to the choice of solver used for filtering.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The work isolates a phenomenon that has been widely reported, ie, strong models on abstract math often fail when conditions are embedded in text. The two-variant construction is simple and scalable, and the error analysis is quite illuminating, showing that setting up the math is frequently the key difficulty. I also appreciate that the authors do not stop at measurement, but also include initial training interventions that enhance contextual performance without compromising abstract skills. The paper is mostly clear, and the benchmark seems practical to adopt.", "weaknesses": "My main concerns are about measurement validity and fairness. The SG/CS transformations would benefit from multiple independent annotators and a quantitative audit to ensure equivalence and controlled change. The formulation metrics rely on an LLM judge. The human validation sample is eventually too small, and there is no check with a different judge family or a symbolic/exact solver, where feasible. The open-source vs proprietary comparison lacks confidence intervals to assess robustness. Contamination or near-duplicate checks are not presented, which matters for items derived from widely circulated math sets. The SFT pipeline filters scenarios using one solver, which can introduce selection bias and may not generalize.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)