Atomic Thinking of LLMs: Decoupling and Exploring Mathematical Reasoning Abilities
Authors: Jiayi Kuang
Haojing Huang
Yinghui Li
Xinnian Liang
Zhikun Xu
Yangning Li
Xiaoyu Tan
Chao Qu
Meishan Zhang
Ying Shen
Philip S. Yu
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Mathematical Reasoning
Atomic Thinking
Primary area: evaluation
TL;DR: We have decoupled the math atomic capabilities of large language models and explored their interaction relationships in mathematical reasoning tasks.

Abstract: Large Language Models (LLMs) have demonstrated outstanding performance in mathematical reasoning capabilities. However, we argue that current large-scale reasoning models primarily rely on scaling up training datasets with diverse mathematical problems and long thinking chains, which raises questions about whether LLMs genuinely acquire mathematical concepts and reasoning principles or merely remember the training data. In contrast, humans tend to break down complex problems into multiple fundamental atomic capabilities. Inspired by this, we propose a new paradigm for evaluating mathematical atomic capabilities. Our work categorizes atomic abilities into two dimensions: (1) field-specific abilities across four major mathematical fields, algebra, geometry, analysis, and topology, and (2) logical abilities at different levels, including conceptual understanding, forward multi-step reasoning with formal math language, and counterexample-driven backward reasoning. We propose corresponding training and evaluation datasets for each atomic capability unit, and conduct extensive experiments about how different atomic capabilities influence others, to explore the strategies to elicit the required specific atomic capability. Evaluation and experimental results on advanced models show many interesting discoveries and inspirations about the different performances of models on various atomic capabilities and the interactions between atomic capabilities. Our findings highlight the importance of decoupling mathematical intelligence into atomic components, providing new insights into model cognition and guiding the development of training strategies toward a more efficient, transferable, and cognitively grounded paradigm of "atomic thinking".

Atomic Thinking of LLMs: Decoupling and Exploring Mathematical Reasoning Abilities
Jiayi Kuang', Haojing Huang", Yinghui Li Xinnian Liang', Zhikun Xu
Yangning Li Xiaoyu Tan Chao Qu", Meishan Zhang?, Ying Shen 18j Philip S. Yu
Sun Yat-sen University, TTiinghua University, ByttDancc Inc.
AArizona State University, Tencent Youtu Lab, FFudan University
Harbin Institute of Technology (Shenzhen), *Pengcheng Laboratory University of Illinois Chicago
Abstract
Large Language Models (LLMs) have demonstrated outstanding performance in mathematical reasoning capabilities. However, we argue that current large- scale reasoning models primarily rely on scaling up training datasets with diverse mathematical problems and long thinking chains, which raises questions about whether LLMs genuinely acquire mathematical concepts and reasoning principles or merely remember the training data. In contrast, humans tend to break down complex problems into multiple fundamental atomic capabilities. Inspired by this, we propose a new paradigm for evaluating mathematical atomic capabilities. Our work categorizes atomic abilities into two dimensions: (1) field-specific abilities across four major mathematical fields, algebra, geometry, analysis, and topology, and (2) logical abilities at different levels, including conceptual understanding, forward multi-step reasoning with formal math language, and counterexample- driven backward reasoning. We propose corresponding training and evaluation datasets for each atomic capability unit, and conduct extensive experiments about how different atomic capabilities influence others, explore the strategies elicit the required specific atomic capability. Evaluation and experimental results on advanced models show many interesting discoveries and inspirations about the different performances of models on various atomic capabilities and the interactions between atomic capabilities. Our findings highlight the importance of decoupling mathematical intelligence into atomic components, providing new insights into model cognition and guiding the development of training strategies toward more efficient, transferable, and cognitively grounded paradigm of "atomic thinking".
Introduction
In recent years, Large Language Models (LLMs) have achieved remarkable performance in lan- guage understanding [+S] visual perception gH complex reasoning [1/22) agentic intelligence 26) and honesty (27-30). mathematical reasoning has emerged key focus for LLM cognitive abilities (1]. Mathematics, as a fundamental reasoning task, offers both verifiable answers and a wide range of difficulty levels, drawing increasing research attention (7) Recent studies, particu- larly those involving reasoning models such as OpenAl's ol and DeepSeek-R1, have demonstrated strong mathematical performance, achieving impressive results on many challenging benchmarks 38
YYinhhui Li is project leader.
CCorrespond to Yinghui Li (liyinghuihhhe @ gmail.com), Ying Shen muil..ys.l.l.....duuu..... Ying Shen
the corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

What are the advantages of atomic thinking compared with chain of thought?
2. What atomic capabitities are worth exploring?
3. How do advanced models perform at each atomic capability?
Prove that if two angles of triangle are
Question:
Field Atomic capability
Algabra
equal, then the triangle is issceless.
Algebra
Reverse
93
Geametry
Chain Thought Thinking
Geometry
Topology
Step Definition of Isosceles
triangle.....
Analysis
Forward
Step 2: Equal angles correspond to Step
Reason
Analysis
VS.
Logic Atomic copability
Atomic Thinking
Atom 1 1(Geometry isosceles
Definition
Topology
triangle....
Atom Forward Reasoning LA=
Forward Ressoning
Attribute
2B = side BC side AC,
OpensW of
Dcopsedk R1
o.-.
Geemiz Freview
dluude Aucmmet
lower length and clearer structure!
sonceptual Understanding
Intenn MinMMMM-MMttttt Puy-208
Figure 1: This figure illustrates an overview of our atomic thinking. It compares thought chains and atomic thinking, highlighting the efficiency of atomic thinking. Next, it shows the atomic capabilities we focus on. Finally, it provides the performance of advanced models in every atomic capability.
Current approaches enhance mathematical performance by scaling up training data, incorporating diverse mathematical problems, and complex reasoning paradigms [41 42 32]. WizardMath [3
synthesizes complex multi-field data via varied math instructions, while models like OpenAl ol train extended chains of thought to perform higher-level reasoning. As the complexity of mathe- matical problems increases, some studies, such as Lean4 [43 and Lean-STaR 4. leverage formal mathematical languages to mitigate ambiguity in natural language multi-step reasoning, as well as enhancing the reward model for problem-solving process [45) However, as model size and data scale approach saturation, these training paradigms are facing challenges:
Do models truly grasp mathematical concepts and inference patterns, or are they merely memory problems by chain of thought reasoning training?
Is there more fundamental cognitive atom that can break through the current paradigm. and what are the advantages of this atomic thinking compared with the chain of thought?
Figure compares the differences between the two paradigms. Existing math reasoning strategies, such chain-of-thought and tree-of-thought, tend to rely heavily on sequential context 355 48
50) This not only leads to inefficient use of computational resources but also introduces noise through excessive self-correction in long reasoning chains. contrast, human reasoning typically decomposes complex problems into atomic problems, solving them incrementally and integrating only the essential information for subsequent steps, which is referred to as atomic thinking (51). This atomic thinking paradigm promotes flexible and structural reasoning, enabling more efficient Thus, decoupling the mathematical atomic abilities of LLMs is not only essential for assessing their cognitive depth but also key to transitioning from the current "question drilling" paradigm to "atomic thinking" framework. So we wonder what atomic capabilities are worth exploring?
problem solvin-oovinggg like data probe [52 54
Current mathematical benchmarks mostly assess models' accuracy in end-to-end problem solving 575 offering little insight into a systematic assessment of atomic capabilities. To address this, we propose novel framework for exploring mathematical atomic abilities, encompassing both field and logical reasoning capabilities, with an emphasis on ensuring minimal overlap between different atomic units while maintaining broad coverage of mathematical tasks, For field atomic abilities, we draw inspiration from modern mathematics and construct four foundational fields: algebra, geometry, analysis, and topology. For logical reasoning abilities, we reference cognitive psychology in math reasoning to define three core capabilities:
(1) Conceptual Understanding, which grasps math definitions and axioms; (2) Forward Reasoning with Formal Math Language, which conducts rigorous, multi-step reasoning using symbolic systems; (3) Counterexample-drive Backward Reasoning, requiring constructing counter- counter-examples and leveraging backward reasoning. For each atomic ability, we construct training and testing data, ensuring interpretability and isolating cross-

---

ability. We conduct the evaluation experiments as shown in Figure to explore: how do advanced models perform each atomic capability?
Beyond decoupled atomic capability evaluation, we also conduct composite experiments across reasoning levels and varied atomic abilities investigate how atomic abilities influence each other. Our findings on all the results reveal several key insights:
Field-level performance: LLMs perform better in algehra and analysis, while struggling in geometry and topology. Interestingly, models exhibit atypical behavior in topology, performing worse on casier tasks yet better on harder ones.
Logical reasoning abilities: Larger LLMs exhibit stronger conceptual understanding. likely due to superior pretraining memory. However, even advanced commercial models struggle with constructing counterexamples, indicating a gap in backward reasoning skills.
Cross-field interaction: Training on low-difficllty data can hinder high-level skill expres- sion in some fields. Notably, activating algehraic abilities significantly improves performance in other fields, which is often more than direct training the target field.
Cross-Logic interaction: Conceptual understanding enhances other reasoning abilities and field atomic abilities. Surprisingly, training solely on definition completion tasks suffices to stimulate high-level ability, outperforming models trained on more complex data. This reveals the supporting value of conceptual understanding in mathematical training.
2
Related work
Mathematics-enhaned large language models Mathematical reasoning has become key focus in exploring the upper bounds of LLMs cognitive capabilities (58, [5]. Unlike general-purpose models such as GPT-4 (38) and Gemini B1. math-enhanced LLMS emphasize field-specific strategies, including data augmentation, pretraining, fine-tuning [60], and reinforcement learning on large-scale mathematical corpora [37]. WizardMath (33) synthesizes diverse math data through instruction
generation and leverages RLHF and process supervision. NuminaMath [6]l adopts Tool-Integrated Reasoning (TIR) generate math data, including questions with fine-graine solutions. Qwen2.5- Math 32 fine-tunes Qwen2.5 on proprietary high-quality math data. InternLM2-M enhances logical rigor by integrating formal mathematical language, code interpreters, and theorem proving in Lean4. Deepseek-Math- (35 focuses on data engineering and efficient RL training.
[36]
Mathematical benchmarking Due to their objective correctness and structured difficulty 62 math tasks are ideal for evaluating LLMs [55] 64 655 [6, (57.. Benchmarks like MATH [66 and
GSM8K (67] test high school and clementary- reasoning and have become standard. To meet the demands of advanced models, more challenging datasets such as OlympiadBench (68] target competition-level exams, Formal theorem proving benchmarks like [69] Putnam Bench, CoqGym [64), and MiniF2F (65) further assess logical reasoning with tools like Coq and Lean. However, most benchmarks assess end-to-end question-sol solving performance without decomposing tasks into atomic reasoning abilities. Our work aims to decouple and analyze these atomic abilities and their interactions support finer-grained reasoning evaluation and lightweight but effective training.
3 Atomic capability decoupling and interaction
Inspired by the atomization of human cognition, we propose decoupling the mathematical capabilities of LLMs into atomic abilities. We categorize atomic capabilities into two major types, ensuring that each capability disentangled from the others while jointly covering a wide range of mathematical tasks. We construct corresponding training and test sets to evaluate the model performance of different atomic abilities. Beyond evaluating individual capabilities, we further investigate their interactions. Specifically, we explore how stimulating one atomic capability may affect others, offering insights into how such interactions can be leveraged to enhance targeted atomic abilities and promote compositional problem-solving strategies.
3

---

Field Atomic Ability
GPT-4o
Template
Filter Field
Filter Difficulty
sampling
Atomic Capability
Raw Math Datasets
Logic Atomic Abllity
1. Conceptual understanding
Reverse
in the blank
formal language? couter example?
Figure 2: This figure illustrates our data construction procedure.
3.1 Atomic capability design
Field atomic capabilities We refer to the core field division of modern mathematics. It is worth noting that if we divide the capabilities too finely (e.g.. dividing them into more than ten types), the atomic units will be too loose and the interaction between atomic abilities will be too complex, which may not be particularly significant correlation. Therefore, finally divide the field's atomic abilities into Algebra, Geometry, Analysis, and Topology. To analyze the influence of difficulty levels in more fine-grained aspect, we further divide each field into low difficulty (level and high difficulty (level 2). Similarly, we divide it into two difficulty levels to avoid complex interactions.
Logical reasoning atomic capabilities For logic reasoning atomic ability, we refer to the human cognitive mode, deconstruct the general process of complex reasoning, and identify core abilities:
1. Conceptual understanding is fundamental, which includes definition identification and attribute description. The definition identification task requires the LLM to complete the name of the corresponding definition in statement. The property description task requires completing the detailed description a mathematical definition, including the premises, the conditions, the key words, and parameters.
2. Forward reasoning with formal language: With reference human cognitive processes, we designate forward multi-step reasoning as a higher-level logical atomic ability. Since natural language reasoning is often confronted with vague proof's and uncritical assumptions, we emphasize the forward reasoning using formal mathematical language.
3. Backward reasoning with counterexamples: In addition to step-by-step forward multi-step reasoning, backward reasoning with counterexamples is also a very important ability in mathematical reasoning. By skillfully constructing appropriate counterexamples, proofs can be effectively accomplished that are difficult to perform directly with forward reasoning.
worth noting that we do consider computational abilities. This is because we mainly concereed with the logical reasoning ability of mathematical reasoning. Computational capability is difficult to decouple from the reasoning process. addition, when computations are needed, LLM invoking relevant mature computational tools would be more efficient and accurate.
3.2 Data construction for capability evaluation
To decouple each atomic capability, we construct the training and testing data, which contributes to evaluation and further exploration of the interaction. The overview of our data construction is shown Figure More data statistics and examples can be found in Appendix A
For field capabilities, we collect data from current benchmarks such as MATH (66]. GSM8K [67].
Gaokao-Bench [70]. OlympiadBench (68. AIME, MMLU (7]), and DeepMath (72). Using combination of template matching and LLM-assitted annotation, we reclassify problems into four fields. When there are original field labels in the raw data, LLM takes more account of the primitive labels to better align with human thinking. Difficulty annotation is done by template matching. We take into account the data sources of the original questions (e.g., primary and secondary school questions or Olympiad questions), the original difficulty labeling information, and we emphasize aligning the difficulty classifications of different fields. After that, we randomly sample the data to ensure that the number of each field is relatively balanced. Finally, we randomly divide the training set and test set with ratio of 3:1. For conceptual understanding, we extract math definitions and axioms from NrturalProofs (73) and generate ill-in-theeblanke questions. For forward reasoning,

---

collect questions and proofs with formal language such as SeenWorkbook., and filter the questions with definite answers. For backward reasoning, we use counterexample-dnveer reasoning statement from CounterMath (74.. Given data scarcity, we maintain a 1:1 train-test split for logic atomic capability to ensure evaluation robustness.
3.3 Atomic capability interaction
In addition to decoupling individual atomic capabilities, also try to explore the correlation and interaction between atomic capabilities. One of the most basic strategies is trying to stimulate a certain atomic capability and observe whether it has an impact on other atomic capabilities. This will provide insight for subsequent research on how to specifically stimulate the required atomic capabilities. Therefore, we explore the following atomic capability interactions:
Cross-diffcculy Interaction: We investigate how training on Level 1 or Level 2 tasks within field affects performance at the other level.
Cross-field Interaction: We train on tasks from one field and evaluate transfer effects to others, particularly among related or complementary fields.
Logical Capability Interactions: We examine whether conceptual understanding can support high-level ability, improving forward backward reasoning, and whether forward and backward reasoning mutually reinforce each other. Additionally, we assess whether high-level reasoning enhances foundational understanding.
Reasoning- -o-ffiiin-t--filld Interaction: Given the abstract nature of some fields (e.g., topology and analysis), we test whether improving logical reasoning capabilities can enhance performance in field-specific tasks, especially those requiring conceptual abstraction.
Experimental settings
4.1 Decoupled atomic capability evaluation
Baselines We evaluate diverse set of LLMs with focus on mathematical reasoning. Open-
source models include Deepseek-Man Math-Mt-tBR-- (35), Eurus-2-7E-PRIME. (75), Qwen2.5-Math-
7B-Instruct (32], NuminaMath-77BTTT --R [6]], I
[36) Abel-7B/13B (3),
WitardMath-7B [33). Mathstral-7E MetaMath-MMistall-77 [41]. Xwin-Math-7B/13B [42],
QwQ-32B For proprietary models, we use GPT-40, OpenAI o1 Deepseek-R1" Claude3.7-
sonne and Gemini2.5-prad" This selection spans varied training paradigms, data sources, architec- tures, and origins (academie industrial). Open-source evaluations conducted 4x 48GB GPUs, while proprietary models are accessed via official APIs.
Prompts and metries We adopt default Chain-of-Thought (CoT) prompting, instructing models to enclose answers in \boxed{[) for extraction. Accuracy is computed via exact match with reference answers. For Conceptual Understanding, models complete missing definitions. Forward Rea- soning requires multi-step derivations using formal mathematical language. Accuracy is used for both. For Backward Reasoning, we refer to previous work CounterMATH [74]: (1) F-1 score on the statement judgement, and (2) Example, Strict, Loose metric to evaluate whether the model constructs valid counterexamples. Detailed description our prompts and metrics can be found in Appendix B
4.2 Training for atomic capability interaction
To examine interactions among atomic abilities, we fine-tune Qwen2.5-Mathl www.-aa--I-nnttutttt.. Ins--nnnttrrtt using supervised LoRA training [6 on 4xL20 48GB GPUs, with learning rate of 1.0e-5. Training about
32beeviiee pdf
5

---

Table 1: Model performance on field atomic capabilities. We bold the optimal and underline the suboptimal of models. The low and high difficulty levels correspond to level and level 2, respectively.
Field
Algebra
Geometry
Analysis
Topology
Difficulty Level
Low
High
Low
High
Low
High
Low
High
Open-source models
IntemLMM--matthpluus-77
49.2
35.9
33.0
31.4
41.9
41.5
27.2
37.0
7e
52.0
33.8
35.5
39.3
44.6
37.6
22.6
33.7
Eurus-2-BB-PRIME
50.7
33.9
36.5
32.3
45.4
32.0
36.5
NuminaMath- 7B-TTR
52.4
28.2
39.9
32.1
46.8
33.6
26.4
25.1
59.6
41.4
42.4
37.7
45.2
37.9
20.6
25.8
Mathstral-7B-v
42.3
33.3
36.4
28.1
32.9
29.8
24.1
37.1
Modelg7B
Abel-7B-002
46.7
32.9
36.1
35.2
32.9
26.3
25.6
31.5
Xwin-Math-77
52.6
37.3
38.3
35.1
45.3
32.9
19.7
Qwen2.5-math- mat22---tt--Inttucc--6
80.5
65.2
57.3
51.9
67.7
66.5
52.1
53.4
Abel-13B
63.4
47.3
42.7
41.8
40.4
30.2
24.6
Swin-Math-1a
78.3
51.6
49.5
46.7
55.7
54.8
46.7
38.3
Model 7B
67.8
49.6
49.3
46.1
54.3
47.8
45.4
38.9
QwQ-32B-Prent Preereew
85.1
66.3
59.6
53.8
72.3
67.2
54.9
46.1
Commercial models
OpenAl ol
93.6
87.1
66.3
62.3
59.3
46.9
52.9
49.4
GPT-4o
69.3
50.5
48.5
36.3
55.1
53.2
53.0
49.4
Deepseek-rl
83.6
80.9
76.8
78.0
70.3
77.7
Claude3.7-somee
83.5
72.0
58.0
39.0
80.3
68.1
56.0
58.6
Gemini2.5-pro
80.5
79.6
70.2
73.1
67.5
63.8
68.1
66.2
different interactions adheres to unified hardware setup and consistent hyperparameters. After training, evaluation follows Section 4.1 to ensure comparability across settings.
Analysis and discussion
5.1 Experimental analysis of field atomic capabilities
We evaluate several advanced models by assessing their performance across decoupled atomic abilities in distinct mathematical fields. The detailed results are presented in Table We observe that:
Larger models exhibit stronger atomic capabilities Model performance varies significantly across scales. general, larger models perform better, benefiting from greater capacity and more extensive training data. Among 7B-scale models, Qwven2.5-math-Ime Instruct achieves notably superior results across all evaluated fields, even outperforming some larger models such as InternL.M2-Math Plus-20B. Analysis of its outputs shows that it generates longer reasoning chains, facilitating deeper logical inference and enhancing its problem-solvie sollinn capabilities.
Algebra and analysis perform better Models tend to exhibit stronger mathematical atomic abil- ities in Algebra and Analysis, while performance in Geometry and especially Topology remains weaker. Since we analyze the training data of the open-sourced models we have evaluated, the results reveal that Geometry and Topology are significantly underrepresented in the training data. Moreover, although geometric problems are textually presented, they often require spatial or visual reason- ing an area where LLMs typically struggle. Consequently, current models demonstrate limited atomic capabilities in these fields. Improving performance in underrepresented areas, particularly Topology and Geometry, a pressing research challenge. For example, the low performance on topology-related tasks may reflect a lack of understanding of abstract mathematical structures. A promising direction to integrate core mathematical concepts during training to stimulate relevant atomic skills. We explore such cross-ability interactions in Section 5.3

---

Table 2: Model performance on logic atomic capabilities, where Attr. and Def. are short names the Attribute description and definition task. We bold the optimal and underline the suboptimal results.
Concept
Forward Rea.
Backward Rea.
Attr. (Acc.)
Def. (Acc.)
Acc.
F-1
Example(%% Strict(%)
Loose(S)
Open-source models
IncenLLM2-mathpplus-
43.2
46.2
23.7
33.9
36.6
9.0
Drepseck-mutt---- 1-
39.4
46.5
27.6
32.2
65.9
18.9
20.6
Burus-2-7B-PRMM--
23.1
27.9
41.4
37.5
64.8
28.5
32.0
NumimuMath-7B5
22.8
27.3
30.2
30.4
54.1
13.0
13,7
Mo
MetaMatl-Mistral-75
19.6
25.6
28.6
31.0
26.5
0.4
Malbstral-7B-- (Babstrall7B-v1.00
21.7
29.8
32.9
28.2
38.9
7.5
Abcl-7B-002
20.9
31.0
33.7
34.4
66.1
16.0
Xwin-Math--T
18.8
26.3
26.4
28.1
31.3
1.2
Qwen2.5 math Instruct-70 7b
34.4
50.3
34.4
38.3
74,2
30.2
33,2
Abel-13B
31.2
37.2
22.4
24.4
0.8
Xwin-Math-1338
29.8
45.9
33.1
30.2
31.3
1.2
1.7
Model 7B
InternL.22
37.8
QwQ-32B
62.7
74.6
42.6
39,9
70.0
38.6
43.8
Commercial models
OpenAl al
68.9
78.6
58.7
60.1
55.8
39.8
40.9
GPT-4o
38.1
48.3
37.7
59.0
44.7
19.7
21.3
Deepseek-rl
76.4
84.5
55.6
80.7
86.8
54.2
65.3
Claudee,,7somn Gemini2.55 5-roo
45.8
60.7
56.8
64.8
78.0
45.0
52.5
48.6
60.4
56.7
23.0
90.8
65.1
75.7
Internal difficulty-Ievel analysis We further examine model performance across different difficulty levels within each field. In Algebra, models often experience sharp drops in accuracy on high-difficulty problems, indicating a gap in advanced atomic capabilities. In contrast, performance degradation in Geometry and Analysis is less severe. This suggests the need for training paradigms that better balance basic and advanced skill acquisition to ensure robust generalization. An interesting anomaly arises Topology, where models sometimes perform better harder problems than on easier ones. We hypothesize that this is due to mismatches between the models' training distributions and our evaluation data: some high-difficulty problems may incidentally align with abstract patterns the models have implicitly learned. This counterintuitive result shows that what humans find hard may not be hard for LLMs, which encourages deeper exploration into the field atomic ability decomposition.
Further results in Appendix C.2 show that training data difficulty significantly affects model perfor- mance. Notably, excessively low-difficulty training data may degrade accuracy across difficulty levels. Thus, balancing training data difficulty is essential for fostering generalizable atomic capabilities.
5.2 Experimental analysis of mathematical logical reasoning atomic capabilities
We evaluate several ootte-of-thearr mathematical models across multiple dimensions of logical reasoning. Detailed results are shown in Table 2] From these results, there are some insights that:
Models recognize definitions but struggle with deeper conceptual understanding All models perform better recognizing definitions than completing missing properties, indicating surface- level grasp of mathematical rigor. This suggests that while models can identify known concepts, they often lack precise internal representations. Larger commercial models significantly outperform smaller open-source ones; for example, deepseek-rl scores 84.5 definition recognition and 76.4 in property completion. This disparity reflects the importance of pretraining, where larger models benefit from superior long-range memory and MoE (mixture-of-experrs mechanisms that mitigate knowledge forgetting. Current math models emphasize problem-solving conceptual understanding, contributing to the gap between basic concept recognition and advanced reasoning or proof tasks. Section 5.4 provides further case studies.
Structured reasoning with formal language remains challenge for smaller models Since large-scale commercial models show better performance with the best accuracy of 58.7, the models with smaller parameters struggle with both understanding the questions with formal math language and applying formal mathematical language to reason, despite performing reasonably well in natural- language-bars based reasoning. This suggests that intensive prrbllemddrillngg may promote pattern
7

---

Table 3: Performance comparison after stimulating various atomic capabilities. We color the
positive?
negative influence as green / red
Field
Algehra
Analysis
Geametry
Topology
Difticllty Level
Low
High
Low
High
Low
High
Low
High
Qwen-base
80.5
65.2
67.7
66.5
52.1
53.4
52.1
53.4
Qwen-train-A Algebra
80.2 (10.3)
(14.5
(18.1)
(15.0
66.7 (+1366
(14.1)
56.0 (13.9)
62.3 (18.9)
Owea-train-Analyii
81.3 (10.8)
66.4 (11.2)
(14.)
(11.))
(12.4) (16.))).))
46.1 (16.0)
54.1 (10.7)
Qwen-rainn-eomeeeee
79.6 (10.9)
68.1 (12.9)
(f2.1
59.5 (17.0)
(15.2)
(1..6
52.8 (10.7)
60.9 (17.5)
Qwen-train Topolooy
79.7 (J0.8)
64.4 (10.8)
(13.8
(17.3)
61.7 (19.))
(1..2)
53.7 (11.6)
59.1 (15.7)
Table 4: Performance comparison after stimulating various field atomic capabilities that are trained on -aaeem..--mmah--luu----
Field
Algehra
Analysis
Gevanerry
Topology
Difiiullyy Level
Low
High
Low
High
Low
High
Low
High
Internl M2--MMM--mmmtttue
49.2
35.9
33.0
31.4
41.9
41.5
27.2
37.0
InternL.M-ttrain-- Al-erra
48.1 (11.1)
37.8 (11.9)
(12.8)
33.6 (*2.2)
(311()
42.1 (10.6)
(10.6)
37,9 (10.9)
48.6 (10.6)
36.3 (10.4)
32.2 (10.8)
30.5 (_0.8)
44.3 (12.4)
44.0 (12.5)
28.7 (T1.5)
38.8 (11.8)
memorization over structured formal reasoning. Some models attempt to bridge this gap by generating code-like representations aid multi-step deduction. Notably, reasoning-oriented models such as o1 and DeepSeek- -abeee R1, equipped with stronger long-range inference and self-reflection, achieve outstanding results in this category.
Limited counteresample abilities reveal the limits of problem-solving training For the ability to judge the truth value of mathematical statements, open-source models average around 30 F-1 points, and even advanced models like QWQ-32B reach only 39.9, while Deepseek-RR optimized for mathematical reasoning, scores 80.7. In generating counterexamples, the Qwen series performs partic- ularly well, sometimes surpassing commercial models like o1. Conversely, models like MetaMath succeed only 26.5% of such tasks. Although the best model Gemini2.5-pre demonstrates superior performance across various metrics when it constructs counter-example other models on example consistency is still relatively low, with almost none exceeding 50%. This reflects the limitations of training paradigms overly focused on direct problem-solving, which hinders higher-level abstraction and conceptual reasoning.
the performance of the
5.3 Experimental analysis interactions between atomic abilities
5.3.1 Influence between atomic abilities across different fields
We investigate the influence between fields among field atomic abilities, as summarized Table 3 In particular, atomic abilities in algebra consistently exhibit positive effects across all other fields, with particularly strong gains observed in Analysis and Geometry. In some cases, Algebra training even yields greater improvements than in-field training. For instance, when evaluating Geometry atomic abilities, in-field training improved performance by 6.1 and 2.4 points at Levels and 2, respectively, whereas activating Algebraic abilities led to larger gains of 13.6 and 4.2 points. This suggests a complementary relationship among atomic abilities, likely because algebraic problem- solving emphasizes fundamental reasoning skills that underpin more abstract fields, and our case study in Section 5.4 demonstrates that. These findings highlight the potential of leveraging diverse, field-specific atomic abilities enhance target capabilities more effectively. However, we also observe negative transfer effects. Strengthening atomic abilities in Topology led to performance declines in Algebra and Analysis, with the largest drops reaching 0.8 and 7.3 points on Level This may be due to substantial data distribution divergence. We have provided a visualization result in Maeemmm-2--ttppppp
Appendix F further validate our conclusions, we have now included as an additional baseline for training and evaluation, aiming to further strengthen our claims. The results, shown Table are consistent with our observations on 11t.2-5-mah-inttteee and provide strong complement to previous findings. These findings underscore the need to consider that,
8

---

Table 5: Performance comparison after stimulating various logic atom capabilities.
Atom Capability
Concept
Formal language
Counter example
Attrihure
Definition
Acc.
F-1
Qwen-basc
34.4
50.3
34.4
30.2
Owen-train- Concept
34.8 (10.4)
53.7 (13.4)
53.5 (f19.1)
40.1 (19.9)
Qwen-train Backward
30.3 (J4.1)
46.3 (14.0)
50.3 (+15.9)
(110.9)
Owen-train-Forvee
25.1 (19.3)
44.4 (15.9)
53.7 (119.3)
40.2 (110.0)
Table 6: Performance comparison after stimulating conceptual understanding of atomic capabilities.
Field
Algebra
Analysis
Geometry
Topology
Difficulty Level
Low
High
Low
High
Low
High
Low
High
Qwen base
80.5
65.2
67.7
66.5
52.1
53.4
52.1
53.4
Qwen train Concept
(10.8)
66.1 (10.9)
(12.0)
(15.8)
(110.2
(16.4)
55.6 (13.5
57.2 (13.8
when trying stimulate one, interactions between fields can significantly impact performance in unintended ways.
5.3.2 Interactions among logical reasoning atomic abilities
We also explore the interaction among logical reasoning atomic abilities as shown in Table 5] Our findings indicate that conceptual understanding plays fundamental role in supporting two higher-level reasoning abilities. Training solely on fill-in-the-bland tasks that activate conceptual comprehension atomic ability results in substantial improvements of 19.1 and 9.9 in forward and backward reasoning. This aligns with our earlier analysis: current mathematical models, especially those with limited parameter capacity, exhibit significant deficiencies in conceptual understanding, which constrain the model's capacity to develop more advanced reasoning skills, Furthermore, forward reasoning and counterexample construction appear to mutually reinforce each other, indi- cating the potential for bidirectional enhancement between forward and backward reasoning atomic abilities. However, enhancing high-level atomic abilities alone can lead to a decline in conceptual understanding, revealing that overemphasis on solving complex problems may lead to forgetting of basic mathematical concepts.
5.3.3 Influence of conceptual understanding on field-specific atomic abilities
We also study the impact of logical atomic abilities different fields. Particularly, investigated how conceptual understanding supports field atomic abilities. The results in Table]]sshow signif- icant performance improvements, particularly in fields that rely abstract reasoning, such as Analysis and Geometry. It is worth noting that activating conceptual understanding ability produces better outcomes than direct training on field-specific tasks. These results further confirm the crucial role of conceptual understanding as a foundational atomic ability, which highlights the need for fu- ture research focus more on fostering deep mathematical reasoning and concept-based learning, rather than relying on question repetition or difficulty escalation.
5.4 Case study
To better understand the deficiencies of the model in some atomic capabilities and the impact of stimulating one atomic capability on another atomic capability, we conduct a case study. Specifically. we compare model predictions before and after stimulating algebra ability on a geometry task. As shown in Figure 3 before training, the model produces an incorrect result due to a flawed geometric assumption. After training, the model correctly applies knowledge of central angles and isosceles triangles, demonstrating the correlation from algebraic reasoning to geometric analysis.

---

Error Case Topology
Error Case in Conceptual Understanding Quettion: SAS be Noetherian We SAS is
Determine the number of spanning the complete graph V.K. 9 such that vertex has degree
Questian:
"If all of its local rings are complete intersections.
Answer: 143360
Answer: complete intersections
Model prediction: 8750 X
Model prediction: complete intersection ring X Model output: -... Noetherian ring W. complete intersection ring we say W A is complete intersection ring.
outpul: ....4 neighbors computed [944 7011, and then assumed the
(from vertices),
MK_ St), giving (SMB) 125),
8750.
Comparison of the difference in geometric ability before and after stimulating algebraic ability
Question: The following diagonal drawn in regular decagon, creating and quadrilateral. What the measure of Sx?
import markersAnfor(int i<:10; (ndrawi http::/000.....0000010///0000000 d:/000000000000000000000
Answen: 36
Model prediction before: 45 X
Model prediction after:
Model output: .....The internal angle of regular decagon 144 14444rc144 the vertox half of that, s0 x=72/2-45x 72 \boved(45)e=72/2245
Model output: Th...Te central angle between A and
216 triangle AFG isoscales with angles and
equal... so angle Fis (180-72//2-36- (180-72)/2 72//2 36
Figure 3: Case study about the cases in Topology and Conceptual Atomic ability and the comparison of stimulating one atomic capability on another atomic capability.
6 Conclusion, limitation, and future directions
In our work, we focus on decoupling the mathematical reasoning ability of LLM into atomic abilities and exploring the interactions between atomic abilities. We design field atomic capabilities and logical reasoning atomic capabilities with data. The evaluation results and analysis of decoupled atomic abilities on advanced models highlight the limited performance of some abilities. We further explore the interactions, for which observe whether other atomic capabilities are affected when one atomic capability stimulated. This process provides very interesting inspirations, including facilitating role of algebraic abilities and the supporting of conceptual comprehension ability, both on logical and field atomic abilities. However, we have not explored more advanced strategies to stimulate specific atomic capability, such curriculum learning reinforcement learning. Our findings will encourage exploring how to better stimulate the required atomic abilitics and utilize multiple atomic abilities to solve complex mathematical tasks.
Acknowledgement
This research is supported by Key-Area Research and Development Program of Guangdong Province, Granted No. 2024B1111060000 This research is also supported by Basic Research Fund of Shenzhen City (JCYJ202408 13 12009011,, This work is supported in part by the New Generation Artificial mnelligence-Nattinall Science and Technology Major Project (2025DD0123033).
References
[1] L. Zheng, W.-L. Chiang, Y. Sheng, T. Li, S. Zhuang, Z. Wu, Y. Zhuang, Li, Z. Lin, E. Xing, et al., "Lmsys-chat- Im: A large-scale real-world Ilm conversation dataset," in The Twelfth International Conference on Learning Representations.
[2] Y. Li, Q. Zhou, Y. Li, Li, R. Liu, R. Sun, Z. Wang, C. Y. Cao, and H. Zheng, "The past mistake is the future wisdom: Error-driven contrastive probability optimization for chinese spell checking," " Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022 (S. Muresan, P. Nakov, and A. Villavicencio, eds.), pp. 3202 3213, Association for Computational Linguistics, 2022.
[3] X. WU, Y.L.. Li, J. Sun, and C. Lu, "Symbol-llm: leverage language models for symbolic system in visual human activity reasoning." Advances neurul information processing systems, vol. 36, pp. 19680-29691 2023.
10

---

Z. Wang. Y. Su, R. S. Shah, R. Guo, J. Gu, H. Li, K. Wei, Z. Wang, L. Cheng, S. Ranathunga, M. Fang, J. Fu, F. Liu, R. Huang, E. Blanco, Y. Cao, R. Zhang, S. Yu, and W. Yin, "Lims assist NLP researchers: Critique paper (meta- "reiewing, in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November Association for [4] Y. Li, Q. Zhou, Y. Luo, S. Ma. Li, H. Zheng. X. Hu, and P. S. Yu, "When lims meet cunning texts: fallacy understanding benchmark for large language models," in Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NewrlPS 2024, Vancouver, BC, Canada, December 10 15, 2024 (A. Globersons, L.. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang, eds.), 2024.
[5] Du, Y. Wang, W. Zhao, Z. Deng. S. Liu, R. Lou, H. P. Zou, P. N. Venkit, N. Zhang, M. Srinath, H. Zhang, V. Gupta, Y. Li, T. Li, F. Wang. Q. Liu, T. Liu, P. Gao, C. Xia, C. Xing, C. Jiayang,
12-16, 2024 (Y. Al-Onaizan, M. Bansal, and Y. Chen, eds.), pp. Computational Linguistics, 2024.
[6] Y. Li, S. Huang, X. Zhang, Q Zhou, Y. Li, R. Liu, Y. Cao, H. Zheng, and Y. Shen, "Automatic context pattern generation for entity set expansion," IEEE Trans. Knowl. Data Eng., vol. 35, no. 12. pp. 12458-12465 2023.
[7] C. Dong, Y. Li, H. Gong, M. Chen, J. Li. Y. Shen, and M. Yang, "A survey of natural language generation." ACM Comput. Surv., vol. 55, no. 8, pp. 173:1-173:33 2023.
[8] Lu, Qin, L. Qiao, Y. Li, X. Dai, B. Ke, J. He, R. Qiao, D. Yin, X. Sun, et al., "Youtu-llm Unlocking the native agentic potential for lightweight large language models," arXiv preprint arXiv:2512.246111 2025.
[9] H. Wang. Li, Y. Li, H. Zheng, W. Jiang, and H. Kim, "Exploring the implicit semantic ability of multimodal large language models: A pilot study on entity set expansion," " in 2025 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2025, Hyderabad, India, April 6-11, 2025, pp. 1-5, IEEE, 2025.
[10] J. Kuang, Y. Shen, J. Xie, H. Luo, Z. Xu, R. Li, Y. Li, X. Cheng, X. Lin, and Y. Han, "Natural language understanding and inference with MLLM in visual question answering: A survey," ACM Comput. Surv., vol. 57, no. 8, pp. 199:1-1,, 2025.
[11] W. Hu, Y. Xu, Y. Li, W. Li, Z. Chen, and Z. Tu, "Bliva: A simple multimodal I1m for better handling of text-rich visual questions." in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 2256-2264, 2024.
[12] Kuang, Y. Li, C. Wang, H. Luo, Y. Shen, and W. Jiang, "Express what you see: Can multimodal Iims decode visual ciphers with intuitive semiosis comprehension?"" in Findings of the A Association.fe for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 August 1, 2025 (W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, eds.), pp. 12743-12774, Association for Computational Linguistics, 2025.
[13] T. Zhou, D. Chen, Q. Jiao, B. Ding, Y. Li, and Y. Shen, "Humanvbench: Exploring human- centrie video understanding capabilities of milms with synthetic benchmark data," 2025.
[14] Z. Yue, L. Zhang, and Q. Jin, "Less is more: Mitigating multimodal hallucination from an eos decision perspective," in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 11766-1 11711, 2024.
[15] Q Jiao, D. Chen, Y. Huang. B. Ding, Y. Li, and Y. Shen, "Img-diff: Contrastive data synthesis for multimodal large language models," in Proceedings of the Computer Vision and Puttern 2025.
Recognition Conference, pp.
[16] Y. Li, Z. Xu, Chen, H. Huang, Y. Li, S. Ma, Jiang, Z. Li, Q. Zhou, H. Zheng, and Shen, "Towards real-world writing assistance: A chinese character checking benchmark with faked the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024 (I.. Ku, A. Martins, and V. Srikumar, eds.), pp. 6656-8668, Association for Computational Linguistics, 2024.
and misspelled characters," " in Proceedings the
11

---

[17] Y. Li. W. Zhang, Y. Yang. W. Huang. Y. Wu, J. Luo, Y. Bei, H. P. Zou, X. Luo, Y. Zhao, C. Chan, Y. Chen. Z. Deng. Y. Li, H. Zheng, D. Li. Jiang, M. Zhang, Y. Song, and P. S. Yu, "Towards agentic RAG with deep reasoning: A survey of rag-reasoning systems in llms," CoRR, vol. abs/a500.099455 07.9077.. 2025.
[18] A. Havrilla, S. Raparthy, C. Nalmpantis, J. Dwivedi- Yu, M. Zhuravynski, E. Hambro, and R. Raileanu, "Glore: when, where, and how to improve Ilm reasoning via global and lo- cal refinements." in Proceedings of the s1st International Conference on Machine Learning, pp. 17719-17733, 2024.
[19] X. Tan, H. Wang. X. Qiu, L. Cheng, Y. Cheng, W. Chu, Y. Xu, and Y. Qi. "Struct-x: Enhancing the reasoning capabilities of large language models in structured data scenarios," " in Proceedings of the 31 ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. pp. 2584- 2595, 2025.
[20] L. Yuan, G. Cui, H. Wang. N. Ding, X. Wang, B. Shan, Z. Liu, J. Deng, H. Chen, R. Xie, et al., "Advancing Ilm reasoning generalists with preference trees," " in The Thirteenth International Conference Learning Representations.
[21] X. Hu, X. Li, J. Chen, Y. Li, Y. Li, X. Li, Y. Wang, Q. Liu, L. Wen, P. S. Yu, and Z. Guo, "Evaluating robustness of generative search engine on adversarial factual questions," CoRR, vol. abs/2403. 12077, 2024.
[22] Z. Yi, D. Zeng, Z. Ling, H. Luo, Z. Xu, W. Liu, J. Luan, W. Cao, and Y. Shen, "Atten- tion Basin: Why Contextual Position Matters in Large Language Models," arXiv e-prints, p. arXiv:2508.00 05 05828, Aug. 2025.
[23] H. Luo, J. Kuang, W. Liu, Y. Shen, J. Luan, and Y. Deng, "Browsing like human: A multimodal web agent with experiential fast-and-slow thinking," in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), ACL 2025, Vienna, Austria, July 27- August 1, 2025 (W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar, eds.), pp. 14232 14251, Association for Computational Linguistics, 2025.
[24] M. A. Islam, M. E. Ali, and M. R. Parvez, "Mapcoder: Multi-agent code generation for com- petitive problem solving," in Annual Meeting of the Association of Computational Linguistics 2024, pp. 4912-4944, Association for Computational Linguistics (ACL), 2024.
[25] J. Wang, M. Zerun, Li S. Zhang, C. Chen, K. Chen, and X. Le, "Gta: a benchmark for general tool agents," Advances in Neural Information Processing Systems, vol. 37, pp. 75749-75790, 2024.
[26] W. Zhang. Y. Bei, L. Yang, H. P. Zou, P. Zhou, A. Liu, Li, H. Chen, J. Wang, Y. Wang, Huang, S. Zhou, J. Bu, A. Lin, Caverlee, F. Karray, I. King, and S. Yu, "Cold-start recommendation towards the era of large language models (Ilms): A comprehensive survey and roadmap." " CORR, vol. abs/2501.01 01945, 2025.
[27] X. Tan, S. Shi, X. Qiu, C. Qu, 7. Qi. Y. Xu, and Y. Qi, "Self-criticism: Aligning large language models with their understanding of helpfulness, honesty, and harmlessness," in Proceedings of the 2023 conference on empirical methods in natural language processing: industry track, pp. 650-662, 2023.
(28) G. Chujie, S. Wu, Y. Huang, D. Chen, Q Zhang, 7. Fu, Y. Wan, Sun, and X. Zhang, "Hon- estllm: Toward an honest and helpful large language model," Advances in Neural Information Processing Systems, vol. 37, pp. 7213-7255, 2024.
[29] W. Hua, X. Yang, M. Jin, Z. Li, W. Cheng, R. Tang, and Y. Zhang, "Trustagent: Towards safe and trustworthy I1m based agents through agent constitution," in Trustworthy Multi-modal Foundation Models and AI Agents (TiFA).
[30] Y. Li, H. Huang, J. Kuang, Y. Li, S. Guo, C. Qu, X. Tan, H. Zheng, Y. Shen, and S. Yu, "Refine knowledge of large language models via adaptive contrastive learning." in The Thirleenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, OpenReview.net, 2025.
12

---

[31] G. Comanici, E. Bieber, M. Schaekermann. I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang. E. Rosen, et al., "Gemini 2.5: Pushing the frontier with advanced rea- soning, multimodality, long context, and next generation agentic capabilities," arXiv preprint arXiv:2507.0626), 2025.
[32] A. Yang. B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, Huang, et al., "Qwen2 technical report," arXiv preprint arXiv:2407. 10671, 2024.
[33] H. Luo, Q. Sun, C. Xu, P. Zhao, J. Lou, C. Tao, X. Geng, Q. Lin, S. Chen, and D. Zhang, "Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct," arXiv preprini arXiv:2308..m 09588, 2023.
[34] L. Yu, W. Jiang, H. Shi, Y. Jincheng, Z. Liu, Y. Zhang. J. Kwok, 7. Li, A. Weller, and W. Liu, "Metamath: Bootstrap your own mathematical questions for large language models," in The Twelfth International Conference Learning Representations, 2024.
[35] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, al., "Deepseckmath: Pushing the limits of mathematical reasoning in open language models," arXiv preprint arXiv:2402. 24020000000 2024.
[36] H. Ying, S. Zhang, L. Li, Z. Zhou, Y. Shao, Z. Fei, Y. Ma, J. Hong, K. Liu, Z. Wang, al., "Internlm-man Open math large language models toward verifiable reasoning," arXiv preprint arXiv:2402.06332, 2024.
[37] E. Chern, H. Zou, X. Li, J. Hu, K. Feng, J. Li, and P. Liu, "Generative ai for math: Abel." https://gt thub. com/ CAA//GAIRR-IA CAIR-NLLY NLP/abel, 2023.
[38] OpenAI, "Git-4 technical report," 2023.
[39] Y. Li, Y. Li, X. Wang, Y. Jiang. Z. Zhang. X. Zheng. H. Wang, H. Zheng, F. Huang. J. Zhou, and P. S. Yu, "Benchmarking multimodal retrieval augmented generation with dynamic VQA dataset and self-adaptive planning agent," in The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, OpenReview.net, 2025.
[40] Z. Xu, Li, R. Ding, X. Wang, B. Chen, Y. Jiang. H. Zheng, W. Lu, P. Xie, and F. Huang, "Let Ilss take on the latest challenges! A chinese dynamic question answering benchmark," in Proceedings of the 31st International Conference on Compulational Linguistics, COLING 2025, Abu Dhabi, UAE, January 19-24, 2025 (O. Rambow, L.. Wanner, M. Apidianaki, H. AI-Khalifa, B. D. Eugenio, and S. Schockaert, eds.), pp. 10435-10448, Association for Computational Linguistics, 2025.
[41] L. Yu, W. Jiang, H. Shi, J. Yu, Z. Liu, Y. Zhang, J. T. Kwok, Z. Li, A. Weller, and W. Liu, "Metamath: Bootstrap your own mathematical questions for large language models," arXiv preprint ArXiv:2309. 12284, 2023.
[42] C. Li, W. Wang, J. Hu, Y. Wei, N. Zheng, H. Hu, Z. Zhang, and H. Peng, "Common 7b language models already possess strong math capabilities," arXiv preprint arXiv:2403.04700 403.040046 2024.
[43] L. d. Moura and S. Ullrich, "The lean 4 theorem prover and programming language." in Automated Deduction CADE 28 (A. Platzer and G. Sutcliffe, eds.), (Cham), pp. 625-635, Springer International Publishing, 2021.
[44] H. Lin, Z. Sun, Y. Yang, and S. Welleck, "Lean-star: Learning to interleave thinking and proving," arXiv preprint a40i:: 10040, 2024.
[45] W. Sun, Q. Du, F. Cui, and J. Zhang, "An efficient and precise training data construction framework for process-supervised reward model in mathematical reasoning," in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 August I, 2025 (W. Che, J. Nabende, Shutova, and M. T. Pilehvar, eds.), pp. 4292 4305, Association for Computational Linguistics, 2025.
[46] X. Tan, T. Yao, C. Qu, B. Li, M. Yang, D. Lu, H. Wang, X. Qiu, W. Chu, Y. Xu, et al., "Aurora: Automated training framework of universal process reward models via ensemble prompting and reverse verification," arXiv preprint 2rXiv:2502.11533 2025.
13

---

[47] A. Didolkar, A. Goyal, N. R. Ke, S. Guo, M. Valko, T. P. Lillicrap. D. J. Rezende, Y. Bengio, M. C. Mozer, and S. Arora, "Metacognitive capabilities of llms: An exploration in mathematical problem solving," in Advances in Neural Information Processing Systems 38: Annual Confer- ence on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024 (A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomczak, and C. Zhang, eds.), 2024.
[48] I. Mirzadeh. K. Alizadeh, H. Shahrokhi, O. Tuzel, S. Bengio, and M. Farajtabar, "Gsm-symbolie: Understanding the limitations of mathematical reasoning in large language models," arXiv preprint arXiv: arXiv:2410.05223 2024.
[49] X. Yu, B. Zhou, H. Cheng, and D. Roth. "Reasonagain: Using extractable symbolic programs to evaluate mathematical reasoning," arXiv preprint arXiv:2410.19056 2024.
[50] H. Luo, Y. Deng. Y. Shen, S. K. Ng, and T.-S. Chua, "Chain-of-exemplar Enhancing distractor generation for multimodal educational question generation," in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 7978- 7993, 2024.
[51] F. Teng, Z. Yu, Q. Shi, J. Zhang, C. Wu, and Y. Luo, "Atom of thoughts for markov IIm test-time scaling," " arXiv preprint arXiv:2502. /2018, 2025.
[52] Z. Ling, D. Chen, L. Yao, Q. Shen, Y. Li, and Y. Shen, "Diversity reward: Fine-tuning Ilms on a mixture of uamain-undetermined data," arXiv preprint arXiv:2502.04380, 2025.
[53] S. Huang, S. Ma. Y. Li, M. Huang, W. Zou, W. Zhang, and H. Zheng. "Lattevll: An interactive Ilms evaluation benchmark with incomplete information from lateral thinking puzzles," in Proceedings of the 2024 Joint International Conference on Computational Linguisties, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy (N. Calzolari, M. Kan, Hoste, A. Lenci, S. Sakti, and N. Xue, eds.), pp. 10186-10197, ELRA and ICCL, 2024.
[54] T. Yu, C. Jiang, C. Lou, S. Huang, X. Wang, W. Liu, J. Cai, Y. Li, Tu, H. Zheng, N. Zhang, P. Xie, F. Huang, and Y. Jiang. "Seqgpt: An out-of-the-box large language model for open domain sequence understanding." in Thirty- Eighth AAAI Conference on Artificial Intelli- gence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada (M. Wooldridge, J. G. Dy, and Natarajan, eds.), pp. 19458-19467, AAAI Press, 2024.
[55] A. Amini, S. Gabriel, P. Lin, R. Koncel- Kedzzorsk,,, Y. Choi, and H. Hajishirzi, "Mathqa: Towards interpretable math word problem solving with operation-basel formalisms," arXiv preprint arXiv: 1905. 13319, 2019.
[56] H. Liu, Z. Zheng, Y. Qiao, H. Duan, Z. Fei, F. Zhou, W. Zhang, S. Zhang, D. Lin, and K. Chen, "Mathbench: Evaluating the theory and upplication proficiency of 1mm with a hierarchical mathematics benchmark," 2024.
[57] Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao, "Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts," in International Conference on Learning Representations (ICLR), 2024.
[58] L. Qin, Q. Chen, X. Feng, Y. Wu, Y. Zhang, Y. Li, M. Li, W. Che, and P. S. Yu, "Large language models meet NLP: survey," CoRR, vol. abs/2405. 12819, 2024.
[59] L. Qin, Q. Chen, Y. Zhou, Z. Chen, Y. Li, L. Liao, M. Li, W. Che, and P. S. Yu, "Mul- tilingual large language model: A survey of resources, taxonomy and frontiers." CoRR, vol. abs/2404.04925. 2024.
[60] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. L. D. Zhou, et al., "Chain-of- thought prompting elicits reasoning in large language models," Advances in neural information 2022.
processing systems, vol. 35, pp.
14

---

[61] J. Li, E. Beeching, L. Tunstall, B. Lipkin, R. Soletskyi, S. Huang, K. Rasul, L. Yu, A. Q. Jiang, Z. Shen, et al., "Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions," Hugging Face repository, vol. 13, p. 9, 2024.
[62] Y. Li, H. Huang. S. Ma, Y. Jiang, Y. Li, F. Zhou, H. Zheng, and Q. Zhou, "On the (in)effectiveness of large language models for chinese text correction." CoRR, vol. abs/2307.0900 .09007.0 2023.
[63] Y. Li, S. Qin, J. Ye, S. Ma. Y. Li, L. Qin, X. Hu, W. Jiang, H. Zheng, and P. S. Yu, "Re- thinking the roles of large language models in chinese grammatical error correction," CoRR, vol. abs/2402.1 1420, 2024.
[64] K. Yang and J. Deng, "Learning prove theorems via interacting with proof assistants," in International Conference on Machine Learning, pp. 6984-6994, PMLR, 2019.
[65] K. Zheng, J. M. Han, and S. Polu, "minif2f: cross-system benchmark for formal olympiad-level mathematics," " in International Conference on Learming Representations, 2022.
[66] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt, "Measuring mathematical problem solving with the math dataset," NeurIPS, 2021.
[67] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, "Training verifiers solve math word problems," arXiv preprint arXiv:21 10. 14168, 2021.
[68] C. He, R. Luo, Y. Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, J. Liu, L. Qi, Z. Liu, and M. Sun, "Olympiadbench: A challenging benchmark for promoting agi with olympiad- levell bilingual multimodal scientific problems," 2024
[69] G. Tsoukalas, J. Lee, Jennings, J. Xin, M. Ding, M. Jennings, A. Thakur, and S. Chaudhuri, "Putnambench: Evaluating neural theorem-provers the putnam mathematical competition," in The Thirty-eight Conference on Neural Information Processing Systems Datasets and Bench- marky Track.
[70] X. Zhang, C. Li, Y. Zong, Ying, L. He, and X. Qiu, "Evaluating the performance of large language models on gaokao benchmark," arXiv preprint 1244,4 2023.
[71] Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Aruiraj, X. He, Z. Jiang, el al., "Mmlu-pro: A more robust and challenging multi-task language understanding bench- mark," in The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[72] Z. He, T. Liang, J. Xu, Q. Liu, X. Chen, Y. Wang, L. Song, D. Yu, Z. Liang, W. Wang, al., "Deepmath- 103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning," arXiv preprint arXiv:2504. /456, 2025.
[73] Welleck, Liu, R. L. Bras, H. Hajishirzi, Y. Choi, and K. Cho, "Naturalproofs: Mathematical theorem proving in natural language." in Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1. NeurIPS Datasets and Benchmarks 2021, December 2021, virtual (J. Vanschoren and S. Yeung, eds.), 2021.
(74) Y. Li, Kuang, H. Huang, Z. Xu, X. Liang, Yu, W. Lu, Y. Li, X. Tan, C. Qu, et al., "One example shown, many concepts known! counterexample-driver conceptual reasoning in mathematical Ilms," arXiv preprint arXiv:2502. 10454, 2025.
[75] G. Cui, L. Yuan, Z. Wang, H. Wang, W. Li, B. He. Y. Fan, T. Yu, Q. Xu, W. Chen, J. Yuan, H. Chen, K. Zhang, X. Lv. S. Wang, Y. Yao, H. Peng, Cheng, Z. Liu, M. Sun, B. Zhou, and N. Ding, "Process reinforcement through implicit rewards," 2025.
[76] E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li. S. Wang, L. Wang, and W. Chen, "LoRA: Low-rank adaptation of large language models," in International Conference on Learning Representations, 2022.
15

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claims well reflect the motivation and contribution of this paper and provides inspiration for subsequent research.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations and future directions in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Our work mainly explores the atomic capabilities of LLM mathematics reasoning from an experimental perspective as an empirical study, and does not conduct theoretical research.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the models, hardware information, prompt design, etc. used for training and evaluation in Section 4and the Appendix [B] and provide data and code in the supplementary materials for review. We will further disclose the relevant data and code facilitate subsequent researchers to reproduce our results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
17

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide our data and code in the supplementary materials for review. We will further disclose the relevant data and code in GitHub and Huggingface to facilitate subsequent researchers reproduce results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https
/
CommeSumeeesa Suode///ggiideeeoeeuuuubiii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the models, hardware information, prompt design, etc. used for training and evaluation in Section 4 and the Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due the limitation of experimental resources (especially for expensive commercial models) and considering that these studies cannot support the main claims, we did not study the error bars, confidence intervals and other statistical data in our experiments. In addition, we have provided sufficient experimental details, data, and codes to ensure the reproducibility and credibility of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification; Details of the compute resources used to conduct the experiments are provided in Section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We promise that strictly follow the code of ethics of NeurlPS 2025.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We describe our impact in the Section and 6 and declare that our research has conflicts of interest and will not cause negative impact.
19

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We have not released models and data with high risk of abuse. All the models and data used our research have been strictly cist-controlled and open-source.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-us, should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The models and data are all official open source, call the official APL. All use complies with their license.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
20

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the datasets.papersstt paperswi thcode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All the data and codes processed are organized and documented, and submitted together with the supplementary materials for review. We will further open source these data and codes allow more researchers to participate in the exploration.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
21

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in our research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM for what should or should not be described.
22

---

A
Additional details about data
After our data construction process, collected the data corresponding each decoupled atomic capability and divided into training set and test set. The specific statistical results are as following Table 7 and T
Table 7: Data statistics of FIELD atomic capabilities.
Algebra
Geumnety
Analysis
Topology
Field Cap.
Level
Level 2
Level
Level 2
Level 1
Level 2
Level
Level
Train/Test
3813/1277
4517/1505
3351/1117
3391/1331
3276/1092
4077/1358
3336/1112
3176/1058
Table 8: Data statistics of LOGICAL atomic capabilities.
Conceptual Understanding
Forward Reasoning
Backward Reasoning
Logic Cap.
Attribute Description
Definition
Formal Language
Counter-example
Train/Test
1225/1217
1683/1661
1061/1032
1225/1217
In addition, we provide data examples of our logical atomic capabilities as shown in Figure to help readers better understand the different logical reasoning atomic tasks.
Attribute Description Task in Conceptual Understanding Atmoic Abilities
"title": fitedddddeiiittoo---eeed-------d
"question": "Consider diagram fields as equation-inside-rmee We say that SK$ and are
the map
\lotimes_K Bllongrightarrw KL, Viqua Vlsum Vottmes y_ Wongmappto Viuu x_ Y_ Injective."
"Final answer": Tinearly disjoint over $k$ in 510mgaa5""
Definition Task in Conceptual Understanding Atmoic Abilities
"title:: "DefiittionReecirocall""
"question" "Let 5x WAA real number such $x SWddrrac 1 called
"final_answer" rreciprocal of Sx5"]
Formal Language Multi-step Reasoning in Forward Reasoning Atmoic Abilities
"question": [a +R
a a 1= (a_rec: V 2) (n- aa 3 f:lll B M,an=fn = by
sorry".
'final_answer". answer": "The closed form is $a_n
l....uus alnintras n/nsimp ja_res, a0, at]", "sse almsimp [o0, a_ree]", fun nnnintro n'ninduction' with ItAnsimp [n0)\nsimp la_ IH]", "refine'( fun => 'ninduction n <;> simp la rec, a0, a1, *I" "refine { a.?.) \nintro r'ninduction 42 simp la_rec, *J", "use aknintro r'vinduction n <p simp la_rec, *I", "use n n'yinntr nininduction <p> simp a_res, "refine' a, 'nintto nyninduction' with IHAnsimp [a0])nsimpp Ja1, a_res, IH]", "use fun n** n)nintra ninrfl". "let 1:00 + ls fun nivrrffine' f, fun nu rf))
"solution":
Counter-example-rrien Reverse Reasoning Atmoic Abilities
"question" : (Sa"(m] necessarily the smallest positive power square (Wlattttssss ofa.".
'final answer": "True",
"solution" "For example, the E-element cyclic group SG-(a)S. have Stal-WWeette
Figure 4:
The figure presents data examples in logical atomic capability, including property de- scriptions, definition recognition, formal mathematical language-driven forward reasoning, and counterxxamlee-drienn backward reasoning.
23

---

B Additional details about the experimental prompts and metrics
Prompt used for experiments The used prompts are summarized as follows. For field atomic capability, we use the same default chain of thought prompt:
Prompt for Field Atomic Capability
Please think step by step to solve the following question, and put your final answer within [boxed[]. {question]
For conceptual understanding atomic capability, wo prompt the LLMs to fill in the blank:
Prompt for conceptual understanding Capability
Please think step by step to fill in the blank in of following statement, and put your final answer within (bboee[[].
{question]
For forward multi-step reasoning, we prompt the model to use formal math language to solve the question:
Prompi for forward reasoning capability
Please think step by step to solve the following question by formal math language, and put your final answer within boxed ).
{question]
For counter-example-diee driven backward reasoning, we prompt the model to judge the statement true or false, which is the same setting of counter-matis
Counter-example backward reasoning prompt
[statement] Please think step by step about whether the above statement is True or False, and put your final answer within (]boxed[].
Evaluation metrics of counter-example based backward reasoning In the evaluation of counterexample-drin backward reasoning, we go beyond computing the score for judging the truthfulness of given statement. Inspired by the CounterMATH framework, we also assess GPT-4o's capability to generate examples within its reasoning output. This achieved through Example Extraction, which detects and retrieves instances where the model explicitly introduces or references countercxamples to support its claims, Alignment Assessment then determines whether each extracted example is consistent with predefined Reference Example in terms of logical rea- soning pattern, problem dccomposition strategy, and goal relevance. Notably, since a proposition may have multiple valid counterexamples, exact replication of the reference is not mandatory for determining consistency. Instead, the reference serves as a guiding benchmark for GPT-4o, mitigating the risk of fully autonomous evaluations that may diverge from numan-aligned reasoning standards.
Specifically, the Examples metric indicates the percentage of problems where the model incorporates examples in its solution process. Striet Align represents the fraction of model-generated examples that fully match the reference reasoning alignment, while Loose Align captures the proportion of cases where at least one example aligns with the reference example.

---

C Additional experiment and analysis of impact of atomic abilities across difficulty levels within the same field
C.1 Additional Results in Field Interaction
We run each experiment multiple times with different seeds and report means and standard deviations shown in Table 9
Table 9: Performance comparison across various mathematical fields.
Field
Algebra
Analysis
Geometry
Topology
Level
Low
High
Low
High
High
Low
High
Qwen-hase
80.5
65.2
67.7
66.5
52.1
53.4
52.1
53.4
Qwen-train-Alehhe
80.2 (#0.4)
69.7 (#0.5)
75.8 (=0.6)
71.5 (#0.7)
65.7 (#0.3)
$7.5 (=0.6)
(=0.5)
62.3 (2:0.)
Quen-tain-Geomert
79.6 (+0.5)
68.1 (+0.4)
69.8 (+0.6)
59.5 (+0.8)
(40.6)
56.0 (+0.5)
(2084)
(+0.3)
C.2 Difficulty Interaction in Field Atomic Ability
We conduct comparative experiments within each mathematical field to investigate how training on datasets of varying difficulty levels affects the development of atomic abilities. The results in Table TT demonstrate that models trained on high-difficulty data exhibit performance improvements on both easy and hard test, with particularly notable gains the easier tasks. This indicates that mastering complex knowledge not only activates high-level atomic abilities but also facilitates the co-activation of lower-level atomic abilities. However, training solcly on low-diffieulty data can negatively affect performance on harder tasks. This phenomenon is especially evident fields such as Algebra, Analysis, and Topology, where models trained on easier data show a performance decline when evaluated on more difficult tasks. These findings suggest the need for careful balancing of difficulty distribution in training datasets to stimulate atomic abilities across the full spectrum and prevent knowledge forgetting.
Table 10: Performance comparison of different training levels across various mathematical fields. We color the positivet negativel, influence green red.
Field
Algebra
Analysis
Geometry
Topology
Level
Low
High
Low
High
Low
High
Low
High
Qwen-buse
80.5
65.2
67.7
66.5
52.1
53.4
52.1
53.4
Qwen-trane trnn-trai-12 12
83.2 (f2.7)
68.3 (f3.1)
76.8 (19.1)
66.4 (J0.1)
$9.6 (17.5)
57.7 (f4.3)
53.1 (+1.0)
53.8 (10.4)
Qwen-train-ll
79.5 (L1.0)
64.9 (10.3)
70.5 (f2.8)
62.2 (14.3)
58.9 (16.8
57 (?3.6)
52.2 (10.1)
(21.))
Qwen-train-all
80 (J0.5)
69.7 (14.5)
71.8 (14.1)
64.9 (L1.6)
57.4 (15.3)
56.1 (12.7)
531.6)
59.1 (15.7)
Table 11: Performance comparison of different training levels across various mathematical fields. We color the positivet negativel influence green red.
Field
Algebra
Anulysis
Geometry
Topology
Level
Low
High
Low
High
Low
High
Low
High
Qwen-hase
80.5
65.2
67.7
66.5
$2.1
53.4
52.1
53.4
Owen-train-12
83.2 (f2.7)
68.3 (f3.1)
76.8 (19.1)
66.4 (10.1)
59.6 (17.5)
$7.7 (f4.3)
53.1 (f1.0)
53.8 (10.4)
Qwen-train-ll
79.5 (L1.0)
64.9 (10.3)
70.5 (f2.8)
62,2 (14.3)
58.9 (168)
57 (13.6)
52.2
(21.)
Qwen-train-al
80 (J0.5)
69.7 (14.5)
(14.1)
64.9 (L1.6)
57.4 (15.3)
(12.7)
(31.6
59.1 (15.7)
D Human Evaluation
We conduct a human evaluation on a sample of the data during the initial phase of our experiments in order to verify the accuracy of our automatic evaluation metrics. To do this, selected two models,
25

---

Gemini and Qwen, and sampled 20 problems from each dataset for manual review. Furthermore, since the majority the problems are calculation-basee the final answers have a relatively fixed format. This, combined with our constraint that the answer must be in the boxed format, greatly simplified the validation process and made easier to check for accuracy. The table below shows the number of cases (out of 20) where the automatic evaluation aligned with the human evaluation for each model and dataset.
Table 12: The human evaluation category-wise scores.
Algebra
Geometry
Analysis
Topology
Model
Forward Rea.
12
11
12
11
12
11
12
Qwen2.5 Math Instract- 7B
20/20
20/20
20/2
20/20
20/0
20/20
19/20
19/20
18/20
20/20
20/20 20/20
Gemini2.5-pro
20/20
20/20
20/0
20/20
19/20
20/20
20/20
2020
19/20
20/20
E
Visualization results of field atomic ability interaction
We have provided a heat map in Figure 5 that represents the correlation between atomic capabilities the field, providing a more intuitive result.
infuenedd Feid Ability
Figure 5: This figure shows the impact of stimulating one ability on the remaining abilities, where deeper colors indicate greater positive facilitation effect, while lighter colors indicate greater negative impact.
26


Reviews:
[{"review_id": "VDgJqIWvQt", "reviewer": "Reviewer_Hb5o", "summary": "This paper introduces a novel paradigm termed \"Atomic Thinking\" to evaluate and understand the mathematical reasoning abilities of Large Language Models (LLMs) at a more granular level. The authors argue that current end-to-end evaluation methods fall short of revealing whether LLMs genuinely master mathematical concepts or merely memorize problem-solving patterns. Inspired by human cognition, which decomposes complex problems into fundamental components, this work decouples mathematical intelligence into two primary dimensions: (1) field-specific atomic abilities, covering algebra, geometry, analysis, and topology, and (2) logical atomic abilities, which include conceptual understanding, formal forward reasoning, and counterexample-driven backward reasoning.\n\nTo support this framework, the paper contributes specialized training and evaluation datasets for each atomic capability. Through extensive experiments on a wide array of state-of-the-art open-source and proprietary LLMs, the study provides a detailed-performance landscape. A key contribution lies in the interaction experiments, where the authors fine-tune models on one atomic ability to measure its impact on others.\n\nThe main findings are insightful: LLMs generally exhibit stronger performance in algebra and analysis but struggle with geometry and topology. Crucially, the paper reveals that even top-tier models have significant deficiencies in backward reasoning (i.e., constructing counterexamples). It also establishes that foundational skills, particularly conceptual understanding and algebraic abilities, play a pivotal role in boosting performance across other, more abstract domains. These contributions offer a new lens for analyzing LLM cognition and provide actionable insights for developing more efficient and generalizable training strategies for mathematical reasoning.", "questions": "1. The \"decoupling\" of skills is central to your work, but it's hard to do perfectly. How did you control for skill overlap when creating your datasets? For example, how did you isolate \"forward reasoning\" from \"conceptual understanding\"?\n2. Your interaction experiments (Table 3, etc.) are based on single training runs. Can you run these experiments multiple times with different seeds and report the mean and standard deviation? Without this, it's difficult to know if the results are just due to chance. This is my main concern.\n3. The result where models are better at hard topology problems than easy ones is interesting. Can you provide some examples of these problems? Or, even better, show some evidence from training data that might explain this strange result? Right now, it feels too speculative.", "limitations": "Yes, the authors have included a \"Conclusion, limitation, and future directions\" section (Section 6) that acknowledges some limitations of their work, such as not exploring more advanced training strategies like curriculum or reinforcement learning. This is a good start.\nHowever, the discussion could be improved by also explicitly addressing the key limitations raised in my review, namely:\n\n1. The inherent difficulty of achieving \"pure\" decoupling of atomic abilities and its potential impact on the experimental conclusions.\n2. The lack of statistical validation for the interaction experiments due to single-run evaluations, which affects the robustness of the transfer learning claims.\n\nAdding these points would provide a more complete picture of the work's current scope and strengthen the paper by demonstrating a thorough self-assessment.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths:\nThe \"Atomic Thinking\" framework is a clever and highly original contribution. It moves beyond standard end-to-end benchmarks to offer a more insightful, fine-grained analysis of what LLMs actually learn in mathematics. The experimental design is robust, covering a wide range of relevant models and thoughtfully decomposing skills. The paper's most significant contribution comes from its interaction experiments, which reveal fascinating, non-obvious relationships between different mathematical abilities—for example, showing that training on algebra can boost geometry skills more effectively than training on geometry itself. These findings are not only interesting but also provide practical guidance for developing better training strategies.\n\n\n## Weaknesses: \n1）The paper's core idea is to \"decouple\" skills, but this is practically impossible to do perfectly. An algebra task might still rely on conceptual understanding, for example. The authors should acknowledge this limitation more directly and discuss how the inherent overlap between skills might affect their conclusions. It doesn't invalidate the work, but it's an important nuance to address.\n\n2）The claims about ability transfer (Tables 3, 4, 5) are based on a single training run. Given the stochastic nature of fine-tuning, these results could be due to chance. The conclusions would be far more convincing if the authors ran each experiment multiple times (e.g., with different seeds) and reported means and standard deviations. Without this, it's hard to be certain that the observed performance gains are statistically significant.\n\n3）The finding that models do better on hard topology problems than easy ones is intriguing but left underdeveloped. The authors suggest a data distribution mismatch, which is a reasonable hypothesis, but they don't provide any evidence to back it up. A few concrete examples of the problems in question or a quick analysis of relevant training data would make this point much stronger and less speculative.", "was_revised": true, "final_justification": "This is a technically solid and well-executed paper that offers meaningful insights with potential impact in the area of AI in education.", "rebuttal": "# Response to Reviewer Hb5o\nWe are grateful for your thoughtful comments and suggestions. We trust that our detailed response has helped to clarify your concerns.\n\n> Weakness 1: The paper's core idea is to \"decouple\" skills, but this is practically impossible to do perfectly. An algebra task might still rely on conceptual understanding, for example. The authors should acknowledge this limitation more directly and discuss how the inherent overlap between skills might affect their conclusions. It doesn't invalidate the work, but it's an important nuance to address.\n\nWe sincerely apologize for the confusion caused by the atomic ability classification schema presented in our paper. We would like to clarify that the field-level atomic abilities and the logical reasoning atomic abilities represent two different perspectives for categorizing mathematical atomic abilities. **Specifically, we propose four key field-level atomic abilities from a field-centric perspective, and three core logical reasoning abilities from a cognitive skill-centric perspective.** *Since these categorizations are based on different classification criteria, our primary concern is to ensure that each categorization is comprehensive and non-overlapping within its own dimension, rather than strictly non-overlapping across dimensions.* For example, it is natural that algebraic tasks may rely on foundational conceptual understanding. We promise to clarify this classification schema in our camera ready version.\n\nIn each dimension of our classification, we strive to cover all essential abilities required in mathematics while minimizing overlap between abilities within the same schema. For a more detailed explanation regarding the decoupling of abilities, please refer to our response to Question 1.\n\n\n> Weakness 2: The claims about ability transfer (Tables 3, 4, 5) are based on a single training run. Given the stochastic nature of fine-tuning, these results could be due to chance. The conclusions would be far more convincing if the authors ran each experiment multiple times (e.g., with different seeds) and reported means and standard deviations. Without this, it's hard to be certain that the observed performance gains are statistically significant.\n\nWe greatly appreciate your suggestion. Indeed, conducting more experiments with different random seeds is an effective way to validate our hypothesis. Due to time constraints during the rebuttal phase, we have conducted a supplementary set of experiments on a representative subset of tasks using five different random seeds, and we report the average performance and standard deviation. The results, shown in the table below, provide strong statistical evidence supporting our hypothesis, demonstrating that the observed effects are not due to random fluctuation from a single run.\n\n| Model               | Algebra (Low)   | Algebra (High)  | Analysis (Low)  | Analysis (High)| Geometry (Low)  | Geometry (High)| Topology (Low)  | Topology (High) |\n|---------------------|-----------------|-----------------|-----------------|----------------|-----------------|----------------|-----------------|-----------------|\n| Qwen-base           | 80.5            | 65.2            | 67.7            | 66.5           | 52.1            | 53.4           | 52.1            | 53.4            |\n| Qwen-train-Algebra  | 80.2 ± 0.4      | 69.7 ± 0.5      | 75.8 ± 0.6      | 71.5 ± 0.7     | 65.7 ± 0.3      | 57.5 ± 0.6     | 56.0 ± 0.5      | 62.3 ± 0.9      |\n| Qwen-train-Geometry | 79.6 ± 0.5      | 68.1 ± 0.4      | 69.8 ± 0.6      | 59.5 ± 0.8     | 57.3 ± 0.6      | 56.0 ± 0.5     | 52.8 ± 0.4      | 60.9 ± 0.3      |\n\n\n> Weakness 3: The finding that models do better on hard topology problems than easy ones is intriguing but left underdeveloped. The authors suggest a data distribution mismatch, which is a reasonable hypothesis, but they don't provide any evidence to back it up. A few concrete examples of the problems in question or a quick analysis of relevant training data would make this point much stronger and less speculative.\n\nThank you for the very helpful feedback. To provide a more direct comparison as you suggested, we can examine two problems that are of the same type—both require calculating a specific numerical value.\n\n**Example 1**: An \"Easy\" L1 Problem (Incorrect Result).  \n**Question**:Find the coefficient of (x^{18}) in the expansion of ((1+x^5+x^7)^{20}).  \n**Correct Answer**: 0. \n**Model's Correct Answer**: 369922   \n**Analysis of Model's Reasoning**: The correct solution relies on the algebraic insight that the equation (5j + 7k = 18) has no non-negative integer solutions for j and k. Because of this, it's impossible to form an (x^{18}) term, so its coefficient must be 0. However, the model's output suggests a different approach was taken, one that seems to show a misunderstanding of the multinomial theorem. It appears the model used a flawed computational heuristic instead of identifying the key algebraic constraint.\n\n**Example 2**: A \"Hard\" L2 Problem (Correct Result). \n**Question**: Eleanor has 100 marbles, each of which is black or gold. The ratio of the number of black marbles to the number of gold marbles is $1: 4$. How many gold marbles should she add to change this ratio to $1: 6$ ?\n**Correct Answer**: 40 \n**Model's Correct Answer**: 40 \n**Analysis of Model's Reasoning**: The model performed well on this problem, perhaps because it can be successfully deconstructed into a sequence of more straightforward algebraic steps. The model correctly:\n\n1. Set up and solved a system of equations for the initial state.\n2. Formulated a new equation for the final state.\n3. Solved for the final answer accurately.\n4. This process shows a strength in applying structured, step-by-step logic.\n\nA Possible Explanation for the Performance Gap\n1. These examples might offer a more detailed explanation for the \"data distribution mismatch\" hypothesis. It seems the issue may not be \"difficulty\" itself, but rather the underlying problem-solving paradigm.\n2. It appears that many L1 problems tend to require non-obvious computational insights. It functions like a computational puzzle, where success depends on spotting a specific \"trick\" or constraint. The model's failure on such problems could indicate that its training was less focused on this puzzle-like style.\n3. In contrast, many L2 problems seem to be solvable with structured, formal reasoning. It rewards a methodical approach: translating words into a formal system (equations) and then executing a standard procedure. The model's success here and on other L2 problems involving formal proofs or definitions could suggest its training was more aligned with this style.\n4. To conclude, the performance gap might be better understood if we think about the problem sets in a different way. Instead of \"easy vs. hard,\" it could be a distinction between \"computational puzzles\" and \"structured reasoning problems.\" \n\n\n> Question 1: The \"decoupling\" of skills is central to your work, but it's hard to do perfectly. How did you control for skill overlap when creating your datasets? For example, how did you isolate \"forward reasoning\" from \"conceptual understanding\"?\n\nThe issue you raised is indeed very important. As we mentioned in the paper when analyzing the nature of atomic thinking, **solving a mathematical problem often requires the coordination of multiple atomic abilities. The correctness of an answer typically reflects the interplay of various abilities.** This is especially true for more fundamental abilities, such as conceptual understanding, which are likely to manifest across a wide range of problems.\n\nTherefore, **when constructing our dataset, the goal was not to ensure that each mathematical problem reflects only a single atomic ability, but rather to highlight a primary ability.** In other words, if a model lacks a specific ability, it will struggle to solve problems designed to emphasize that ability. For instance, in tasks requiring forward reasoning, conceptual understanding certainly helps (which aligns with our findings on interaction effects), but if the model lacks forward reasoning ability, it will still fail to solve these complex multi-step reasoning problems effectively.\n\n\n> Question 2: Your interaction experiments (Table 3, etc.) are based on single training runs. Can you run these experiments multiple times with different seeds and report the mean and standard deviation? Without this, it's difficult to know if the results are just due to chance. This is my main concern.\n\nThank you again for your insightful suggestion. We have included the new experimental results as part of our response to Weakness 2. Due to the time constraints, we only selected a few representative cases for this supplementary experiment, and we commit to including more comprehensive results in the revised version of the paper.\n\n\n> Question 3: The result where models are better at hard topology problems than easy ones is interesting. Can you provide some examples of these problems? Or, even better, show some evidence from training data that might explain this strange result? Right now, it feels too speculative.\n\nThank you for your valuable advice! We hope the examples provided in the response to weakness 3 above help to address your concerns. If it would be beneficial, we promise to summarize this analysis in a chart or table for inclusion in the revised paper."}, {"review_id": "zjAxbUZZdE", "reviewer": "Reviewer_ezVn", "summary": "This paper introduces \"atomic thinking,\" a new paradigm for analyzing the mathematical reasoning abilities of Large Language Models (LLMs). The authors argue that current models, often trained on long and complex problems, may be memorizing reasoning chains rather than truly understanding fundamental concepts. To address this, they propose decoupling mathematical ability into two dimensions: field-specific capabilities (Algebra, Geometry, Analysis, Topology) and logical capabilities (conceptual understanding, forward reasoning with formal language, and backward reasoning with counterexamples. The study finds that models generally excel in algebra and analysis but struggle with geometry and topology. Furthermore, it highlights that strong conceptual understanding significantly boosts other reasoning abilities, suggesting a shift in training strategy from \"question drilling\" towards a more foundational, concept-based approach.", "questions": "1. The paper defines \"field capabilities\" like Algebra and Geometry as atomic units.  However, these are themselves complex domains. Could you elaborate on the process for selecting this level of granularity? Were finer-grained decompositions (e.g., breaking algebra into \"equation solving\" and \"proof by induction\") considered, and what were the primary trade-offs that led to the current four-field structure?\n\n2. The interaction experiments (Tables 3-5) offer great insights but are based on fine-tuning a single model, Qwen2.5-Math-7B.  How confident are you that these interaction effects (e.g., the foundational role of algebra and conceptual understanding) are fundamental principles of mathematical reasoning in LLMs, rather than artifacts of the specific model's architecture or pre-training? Would you expect similar results if you fine-tuned a model from a different family, such as Llama?", "limitations": "In Section 6, \"Conclusion, limitation, and future directions,\" the authors explicitly state that they \"have not explored more advanced strategies to stimulate a specific atomic capability, such as curriculum learning or reinforcement learning.\"  This is a fair and important acknowledgment.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- Clarity: The paper is well-written and organized. The core idea of \"atomic thinking\" is introduced and motivated clearly, and Figure 1 provides an excellent visual summary of the paper's entire scope and key findings.  The research questions are explicitly stated in the introduction and systematically answered in the analysis sections, making the paper easy to follow and understand.\n\n- Significance & Originality: In my opinion, the paper's primary strength lies in its novel and insightful framing of LLM mathematical reasoning. As we see models achieve near-superhuman scores on some benchmarks, it's becoming crucial to ask how they are solving these problems. This paper provides a structured methodology for doing just that. Moving the focus from \"can it solve the problem?\" to \"what fundamental skills does it possess?\" is a significant step forward.\n\n- Quality: The empirical evaluation is comprehensive and of high quality.  The authors benchmark a diverse and modern set of LLMs, including powerful proprietary models like OpenAI o1, Deepseek-R1, and Gemini 2.5-pro, which lends credibility and relevance to their findings. The methodology for creating the datasets is systematic, and the analysis of the results is thorough, supported by clear tables and illustrative case studies. \n\n## Weaknesses\n\n- While the proposed decoupling is a major strength, the choice of what constitutes an \"atom\" is debatable. Fields like \"Algebra\" or \"Geometry\" are vast and could be considered composites of many smaller, more fundamental skills (e.g., symbolic manipulation, spatial transformation, etc.). The authors briefly justify their choice to avoid over-complicating the interaction analysis, but the framework's claims to \"atomicity\" could be softened.", "was_revised": true, "final_justification": "While I understand that there are currently no math-instruct models in the LLaMA series, I believe there are several available math-instruct models based on LLaMA. I would appreciate seeing more experiments included in the camera-ready version.\n\nOverall, this is still an impressive contribution, and I will maintain my initial score.", "rebuttal": "# Response to Reviewer ezVn\nWe sincerely appreciate your insightful feedback and constructive suggestions. We hope our clarifications have resolved the issues you raised.\n\n> While the proposed decoupling is a major strength, the choice of what constitutes an \"atom\" is debatable. Fields like \"Algebra\" or \"Geometry\" are vast and could be considered composites of many smaller, more fundamental skills (e.g., symbolic manipulation, spatial transformation, etc.). The authors briefly justify their choice to avoid over-complicating the interaction analysis, but the framework's claims to \"atomicity\" could be softened.\n\nWe sincerely apologize for the confusion caused by our categorization of atomic abilities. In fact, the current division into four major field-level atomic abilities—algebra, geometry, analysis, and topology—was made for the following reasons.\n\nFirst, **we consulted with domain experts in mathematics (individuals holding at least a Ph.D degree in mathematics) and determined that adopting the four universally recognized fields within the mathematical community** would provide a complete and non-overlapping classification framework. Second, while each of these fields indeed contains numerous subfields—we initially considered over ten field categories—such fine-grained divisions introduced several challenges. Classifying data into highly specific subfields made data collection significantly more difficult, as data availability for certain fields was extremely limited, resulting in imbalances across different categories and insufficient samples for effective training and evaluation.\n\nMoreover, many of these fine-grained subfields show minimal or no interaction with one another. For instance, equation solving in algebra and spatial relations in geometry may involve completely disjoint abilities. As a result, any attempt to analyze interactions across such isolated subfields would demand exponentially more experimental resources, while the insights gained may be marginal, leading to wasted effort.\n\n> The paper defines \"field capabilities\" like Algebra and Geometry as atomic units. However, these are themselves complex fields. Could you elaborate on the process for selecting this level of granularity? Were finer-grained decompositions (e.g., breaking algebra into \"equation solving\" and \"proof by induction\") considered, and what were the primary trade-offs that led to the current four-field structure?\n\nWe apologize again for the confusion. As explained in our response to Weakness 1, the field-level classification we adopted is based on expert consultation and conforms to widely accepted categorizations in mathematics. In the data collection process, we re-mapped the original subfield labels to our four major fields. This was accomplished either through keyword matching or with the assistance of LLM-as-judge-based classification. As also discussed in our earlier response, this coarser field-level categorization allows us to explore the interaction effects between atomic abilities more efficiently, while minimizing training and computational overhead.\n\n> The interaction experiments (Tables 3-5) offer great insights but are based on fine-tuning a single model, Qwen2.5-Math-7B. How confident are you that these interaction effects (e.g., the foundational role of algebra and conceptual understanding) are fundamental principles of mathematical reasoning in LLMs, rather than artifacts of the specific model's architecture or pre-training? Would you expect similar results if you fine-tuned a model from a different family, such as Llama?\n\nWe deeply appreciate your valuable suggestion. The Qwen series includes a variety of open-source models across different parameter scales, offering both base and instruct versions, which makes it particularly suitable for research purposes. Among them, Qwen2.5-math-instruct demonstrated excellent performance in our atomic ability disentanglement experiments and thus became a focal model for our further investigation.\n\n**In response to your suggestion, we have now included InternLM2-math-plus-7B as an additional baseline for training and evaluation, aiming to further strengthen our claims.** The reason why we don't choose llama as the training backbone is due to the absence of the math-instruct model in llama series. Given the limited time during the rebuttal period, we selected only a few key training and evaluation scenarios. The results, shown in the table below, are consistent with our observations on Qwen2.5-math-instruct and provide a strong complement to our previous findings.\n\n| Model                     | Algebra (Low)       | Algebra (High)      | Analysis (Low)      | Analysis (High)    | Geometry (Low)      | Geometry (High)    | Topology (Low)      | Topology (High)     |\n|---------------------------|---------------------|---------------------|---------------------|--------------------|---------------------|--------------------|---------------------|---------------------|\n| InternLM2-math-plus-7b    | 49.2                | 35.9                | 33.0                | 31.4               | 41.9                | 41.5               | 27.2                | 37.0                |\n| InternLM2-train-Algebra   | 48.1 (-1.1)         | 37.8 (+1.9)         | 35.8 (+2.8)         | 33.6 (+2.2)        | 43.1 (+1.2)         | 42.1 (+0.6)        | 27.8 (+0.6)         | 37.9 (+0.9)         |\n| InternLM2-train-Geometry  | 48.6 (-0.6)         | 36.3 (+0.4)         | 32.2 (-0.8)         | 30.5 (-0.9)        | 44.3 (+2.4)         | 44.0 (+2.5)        | 28.7 (+1.5)         | 38.8 (+1.8)         |\n\nLimitations:\n\n> In Section 6, \"Conclusion, limitation, and future directions,\" the authors explicitly state that they \"have not explored more advanced strategies to stimulate a specific atomic capability, such as curriculum learning or reinforcement learning.\" This is a fair and important acknowledgment.\n\nIn this work, we focus primarily on the atomic ability thinking paradigm, a novel perspective on mathematical reasoning, and present a series of exploratory experiments. Building on our current findings, we hope to pursue more precise training strategies for activating atomic abilities in future work. For example, after discovering interactions among field-level atomic abilities, we could apply a curriculum learning approach, gradually introducing different field abilities layer by layer—starting with foundational support fields and progressing to more advanced ones—to achieve better generalization across fields.\n\nAdditionally, we plan to explore reinforcement learning-based methods that reward the model for correctly understanding and applying mathematical concepts. This could improve the model’s reasoning patterns, especially in forward and backward reasoning scenarios."}, {"review_id": "3DOfzW65iP", "reviewer": "Reviewer_6cMW", "summary": "The paper presents an approach that focuses on fundamental atomic capability units to enhance the mathematical reasoning in LLMs.  The authors study field specific abilities vis-a-vis four mathematical fields and logical abilities at different  reasoning levels. The authors also develop training and evaluation datasets for each of the atomic mathematical units. The paper presents an extensive comparative analysis of different types math reasoning LLMs and their own developed supervised models taking various factors into consideration.", "questions": "1. The authors should mention the criteria of selection of the baseline LLMs both the open-source and commercial ones.\n2. The details of the data construction should be detailed with examples in the appendix.\n3. Human evaluation should be conducted for this task.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper proposes the paradigm of atomic thinking which is intertwined with the cognitive abilities of humans while solving mathematical problems. Although current LLMs owe their mathematical reasoning capabilities to large scale of training data and implicit step-by-step supervision, this paper  breaks the reasoning task at two levels of field atomic units and logical reasoning based on three core reasoning capabilities. The coverage of experiments is vast and the paper analyzes the performance of the models in terms of fields and level of difficulty. The paper also discusses the transfer capabilities of fields and provides empirical evidence for it. Training and evaluation datasets for each of the atomic capability units have been developed which is a significant contribution to the field.\n\nThe authors do not mention the criteria of selection of the baseline LLMs. The construction of the dataset is not fully explained which makes things unclear. It is not clear what the authors mean by Level 1 and Level 2 problems. Although the paper relies on automatic evaluation metrics, a thorough human evaluation is required as a math problem can be solved in multiple ways using different kinds of mathematical axioms and concepts. It should be mentioned how the difficulty level is determined for a problem.", "was_revised": true, "final_justification": "The authors have satisfactorily answered my queries regarding the choice of baseline LLMs, and problem types based on difficulty levels.", "rebuttal": "# Response to Reviewer 6cMW\nThank you very much for your feedback and valuable suggestions. We hope our response adequately addresses your concerns.\n\n## Response to weaknesses\n\n> The authors do not mention the criteria of selection of the baseline LLMs. \n\nAs noted in Page 5, Lines 180–182, we have explained the rationale behind our model selection. Specifically, we chose proprietary models with superior training and architectural designs for mathematical tasks, and evaluated them alongside state-of-the-art models currently available. We also took into account models of varying parameter scales, considering both those widely adopted in industry and academia, to enable a more comprehensive assessment of the atomic abilities of current mathematical models.\n\n> The construction of the dataset is not fully explained which makes things unclear.\n\nWe apologize for the confusion. During the data construction phase, we first gathered a large volume of mathematical test data to support robust evaluation. For field-specific data, we primarily adopted a classification-based annotation strategy. We re-mapped the original subfield labels from the collected datasets to our four-field schema (algebra, geometry, analysis, topology). For data without clear subfield annotations or with ambiguous labels, we designed a keyword-based matching strategy, further supplemented by LLM-as-judge to classify the data into the appropriate field.\n\nIn contrast, the construction process for logical reasoning ability data followed a different approach. For conceptual understanding, we created cloze-style questions by masking keywords in natural language descriptions of mathematical definitions, theorems, and formulas. For forward reasoning, we focused on multi-step reasoning tasks involving formalized mathematical language. We collected mathematical problems originally expressed or annotated in formal language and used formal language transformation tools to convert natural language inputs into formal representations for reasoning. For backward reasoning, we collected a set of mathematical propositions and judgments that are particularly suitable for reasoning via counterexamples or proof by contradiction, and curated them with corresponding reasoning steps and justifications.\n\n> It is not clear what the authors mean by Level 1 and Level 2 problems. \n\nWe apologize for any confusion caused by our task difficulty labeling. The Level 1 and Level 2 distinctions correspond to low-difficulty and high-difficulty tasks within each field-level atomic ability, respectively. \n\n> Although the paper relies on automatic evaluation metrics, a thorough human evaluation is required as a math problem can be solved in multiple ways using different kinds of mathematical axioms and concepts. \n\nThank you for raising this point. \n\nWe did conduct a human evaluation on a sample of the data during the initial phase of our experiments in order to verify the accuracy of our automatic evaluation metrics. To do this, we selected two models, Gemini and Qwen, and sampled 20 problems from each dataset for manual review. Furthermore, since the majority of the problems are calculation-based, the final answers have a relatively fixed format. This, combined with our constraint that the answer must be in the `\\\\boxed{}` format, greatly simplified the validation process and made it easier to check for accuracy. \n\nThe table below shows the number of cases (out of 20) where the automatic evaluation aligned with the human evaluation for each model and dataset:\n\n| Model | Algebra-l1 | Algebra-l2 | Geometry-l1 | Geometry-l2 | Analysis-l1 | Analysis-l2 | Topology-l1 | Topology-l2 | Attr. | Def. | Forward Rea.  |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| Qwen2.5-Math-Instruct-7B | 20/20 | 20/20 | 20/20 | 20/20 | 20/20 | 20/20 | 19/20 | 19/20 | 18/20 | 20/20 | 20/20 | \n| Gemini2.5-pro | 20/20 | 20/20 | 20/20 | 20/20 | 19/20 | 20/20 | 20/20 | 20/20 | 19/20 | 20/20 | 20/20 |\n\n> It should be mentioned how the difficulty level is determined for a problem.\n\nThe difficulty assignment was based primarily on the source of the data (e.g., middle school vs. university-level problems) and any original difficulty tags associated with the dataset. For example, a problem originating from elementary school math would naturally be categorized as low-difficulty. In many cases, our collected datasets included original difficulty labels, which we also used as a reference to guide our difficulty classification.\n\n## Response to Questions\n\n> The authors should mention the criteria of selection of the baseline LLMs both the open-source and commercial ones.\n\nThank you for your thoughtful suggestions. We have further elaborated on our baseline selection criteria in the response to Weakness 1, and we will provide additional clarification in the revised version of the paper to help readers better understand our choices.\n\n> The details of the data construction should be detailed with examples in the appendix.\n\nWe also appreciate your feedback on the data construction process. As discussed in the response to Weakness 2, we have now included additional details regarding our data annotation pipeline. We provide the data and code in our supplementary materials for review. Below, we present several example data entries (in simplified form) for illustration. We plan to include more comprehensive data samples in the Appendix of the revised version. Furthermore, we commit to open-sourcing our dataset to facilitate better transparency and encourage broader participation and research based on our work.\n\n## Example 1\n**Atom**: Field Atom Ability\n**Task**: Algebra (Level1)\n**Question**: \"What is the greatest common divisor of all of the members of the set containing all numbers that are the product of four consecutive positive integers?\",\n**Solution**: \"These numbers are all in the form $n(n+1)(n+2)(n+3)\\\\pmod 4$, there will be one number of each residue, so one of the numbers will be divisible by 2 and another divisible by 4, leaving the product divisible by 8. Similarly, one of the numbers will be 0\\\\mod 3, so the product will be divisible by 3. The GCD must therefore be divisible by 3\\\\cdot8=24. It also must be less than or equal to the smallest number in the set, 1\\\\cdot2\\\\cdot3\\\\cdot4=24, so it must be \\\\boxed{24} exactly.\",\n**Final_answer**: \"24\"\n\n## Example 2\n**Atom**: Logic Atom Ability\n**Task**: Conceptual Understanding\n**Question**: \"Let f: S \\\\to T$ be a mapping. Let f^{-1} \\\\subseteq T \\\\times S be the inverse of f: :f^{-1} := \\\\set {\\\\tuple {t, s}: \\\\map f s = t} Let f^{-1} itself be a mapping :\\\\forall y \\\\in T: \\\\tuple {y, x_1} \\\\in f^{-1} \\\\land \\\\tuple {y, x_2} \\\\in f^{-1} \\\\implies x_1 = x_2 and :\\\\forall y \\\\in T: \\\\exists x \\\\in S: \\\\tuple {y, x} \\\\in f Then f^{-1} is called the '''   '''.\"\n**Solution**: \"Definition:Inverse Mapping\"\n**Final_answer**: \"inverse mapping of $f$\"\n\n> Human evaluation should be conducted for this task.\n\nThank you for your suggestion! We have provided the supplementary experimental results above. We promise to add this discussion in our revised manuscripts."}, {"review_id": "O7RHfgwnvS", "reviewer": "Reviewer_uVY9", "summary": "This study creates a benchmark that investigates LLMs' atomic-level math reasoning performance by dividing the capability with fields (algebra, geometry, analysis, and topology) and logical reasoning abilities (Conceptual understanding, Forward reasoning with formal language, Backward reasoning with counterexamples). The evaluation with a wide range of math LLMs reveals several insights such as strength and weakness of each LLM, knowledge transfer characteristics between fields and reasoning types.", "questions": "See \"Strengths And Weaknesses\".", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "It is a novel approach to categorize the math capability into sub-types, create the benchmark, and evaluate each LLMs performance.\nIn addition, the evaluation results show us several insights which will provide important clues for improving the performance of mathematical LLMs in the future.\n\nHowever, there are some weaknesses and points to be improved.\n- While the findings are interesting, they are not surprising. For example, we often find knowledge transfers between close fields (For example, https://neurips.cc/virtual/2024/poster/93733). In addition, LLMs are not good at Geometry because it cannot treat vision.\n- At training for atomic capability interaction, only one model (Qwen2.5-Math-Instruct-7B) is used for the experiment. No less than two models are needed to insist the universality of the findings.\n- VLMs are not included within the scope of the analysis. In particular, geometry needs vision information to completely grasp the problems.\n- The Supplementary zip file could not be unzipped due to some errors.", "was_revised": true, "final_justification": "Although there are some concerns (e.g., VLM is not a target of analysis for geometry, major open/close LLMs are not used for atomic capability interaction experiments), I think that the core motivation and the current results are sufficient for borderline acceptance.", "rebuttal": "# Response to Reviewer uVY9\nWe greatly value your feedback and the important points you raised. We hope our explanations have sufficiently clarified the concerns you mentioned.\n> While the findings are interesting, they are not surprising. For example, we often find knowledge transfers between close fields. In addition, LLMs are not good at Geometry because it cannot treat vision.\n\nThank you very much for your valuable suggestion. We carefully reviewed the paper you referenced and compared it with our own work. While both studies explore the knowledge transfer capabilities of LLMs across different fields, it is important to highlight that our research is not solely focused on transfer learning.\n\nThe core motivation and contribution of our work lies in proposing a new paradigm of atomic thinking for mathematical models, and in investigating the disentanglement of distinct atomic abilities as well as their interaction effects. Our findings go beyond field similarity-based transfer; we also observed phenomena such as complementary knowledge transfer between disparate fields, catastrophic forgetting of higher-level field knowledge, and the fundamental role of conceptual understanding as a foundational ability that supports reasoning in all other fields. These insights offer valuable inspiration for future training and inference strategies in mathematical reasoning models. Regarding knowledge in the geometry field, we plan to further explore how richer multimodal information (e.g., visual or spatial inputs) may enhance model performance in mathematical reasoning.\n\n> At training for atomic capability interaction, only one model (Qwen2.5-Math-Instruct-7B) is used for the experiment. No less than two models are needed to insist the universality of the findings.\n\nWe deeply appreciate your valuable suggestion. The Qwen series includes a variety of open-source models across different parameter scales, offering both base and instruct versions, which makes it particularly suitable for research purposes. Among them, Qwen2.5-math-instruct demonstrated excellent performance in our atomic ability disentanglement experiments and thus became a focal model for our further investigation.\n\nDue to time and resource constraints, we regret that we were unable to extend this discovery to a broader set of models. In response to your suggestion, we have now included InternLM2-math-plus-7B as an additional baseline for training and evaluation, aiming to further strengthen our claims. Given the limited time during the rebuttal period, we selected only a few key training and evaluation scenarios. The results, shown in the table below, are consistent with our observations on Qwen2.5-math-instruct and provide a strong complement to our previous findings.\n\n| Model                     | Algebra (Low)       | Algebra (High)      | Analysis (Low)      | Analysis (High)    | Geometry (Low)      | Geometry (High)    | Topology (Low)      | Topology (High)     |\n|---------------------------|---------------------|---------------------|---------------------|--------------------|---------------------|--------------------|---------------------|---------------------|\n| InternLM2-math-plus-7b    | 49.2                | 35.9                | 33.0                | 31.4               | 41.9                | 41.5               | 27.2                | 37.0                |\n| InternLM2-train-Algebra   | 48.1 (-1.1)         | 37.8 (+1.9)         | 35.8 (+2.8)         | 33.6 (+2.2)        | 43.1 (+1.2)         | 42.1 (+0.6)        | 27.8 (+0.6)         | 37.9 (+0.9)         |\n| InternLM2-train-Geometry  | 48.6 (-0.6)         | 36.3 (+0.4)         | 32.2 (-0.8)         | 30.5 (-0.9)        | 44.3 (+2.4)         | 44.0 (+2.5)        | 28.7 (+1.5)         | 38.8 (+1.8)         |\n\n> VLMs are not included within the scope of the analysis. In particular, geometry needs vision information to completely grasp the problems.\n\nWe sincerely appreciate your suggestion. In this work, our primary focus was to propose a novel paradigm centered on mathematical atomic abilities, conduct systematic evaluations of atomic ability disentanglement, and analyze interaction effects among abilities across a range of mathematical models. We believe these insights offer meaningful guidance for future model development and training strategies. As for visual understanding, we agree that it can be regarded as a new atomic ability relevant to mathematical problem-solving. We are incorporating this perspective into our future work, with the goal of further enriching and expanding the taxonomy of atomic abilities. We warmly welcome more researchers to engage with and contribute to this ongoing exploration, and together advance the development of mathematical reasoning in large language models.\n\n\n> The Supplementary zip file could not be unzipped due to some errors.\n\nWe are sorry to hear about the issue you encountered. We have repeatedly tested the supplementary file and confirmed that it can be downloaded and opened successfully. We suspect this may be related to an OpenReview system issue. Nonetheless, we promise that we will open-source relevant data and code, and actively encourage broader participation in this line of research."}]

Author rebuttal:

Meta-review: This paper introduces a novel paradigm termed “Atomic Thinking” to evaluate and understand the mathematical reasoning abilities of Large Language Models (LLMs) at a more granular level. Experiments on both closed- and open-source LLMs reveal insightful findings, such as the fine-grained weaknesses of each model under the “Atomic Thinking” paradigm. Overall, the paper is professionally presented.

For the revision, the authors should incorporate clarifications on some key terms, address concerns regarding decoupling control, provide more data construction examples, include human evaluation, and conduct a deeper investigation into the findings—for instance, the better performance on hard topology problems.
Decision: Accept (poster)