Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages
Authors: Zui Chen
Tianqiao Liu
Tongqing
Mi Tian
Weiqi Luo
Zitao Liu
Venue: ICLR 2025 Poster (2025)
Keywords: LLM continue pretrain;math problem solving;data synthesis
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Mathematical reasoning remains a challenging area for large language models (LLMs), prompting the development of math-specific LLMs such as LLEMMA, DeepSeekMath, and Qwen2-Math, among others. These models typically follow a two-stage training paradigm: pre-training with math-related corpora and post-training with problem datasets for supervised fine-tuning (SFT). Despite these efforts, the improvements in mathematical reasoning achieved through continued pre-training (CPT) are often less significant compared to those obtained via SFT. This study addresses this discrepancy by exploring alternative strategies during the pre-training phase, focusing on the use of problem-solving data over general mathematical corpora.
We investigate three primary research questions: (1) Can problem-solving data enhance the model's mathematical reasoning capabilities more effectively than general mathematical corpora during CPT? (2) Are synthetic data from the same source equally effective, and which synthesis methods are most efficient? (3) How do the capabilities developed from the same problem-solving data differ between the CPT and SFT stages, and what factors contribute to these differences?
Our findings indicate that problem-solving data significantly enhances the model's mathematical capabilities compared to general mathematical corpora. We also identify effective data synthesis methods, demonstrating that the tutorship amplification synthesis method achieves the best performance. Furthermore, while SFT facilitates instruction-following abilities, it underperforms compared to CPT with the same data, which can be partially attributed to its poor learning capacity for more challenging problem-solving data. These insights provide valuable guidance for optimizing the mathematical reasoning capabilities of LLMs, culminating in our development of a powerful mathematical base model called MathGPT-8B.

Published as a conference paper at ICLR 2025
ADVANCING MATHEMATICAL REASONING IN LAN- GUAGE MODELS: THE IMPACT OF PROBLEM-SOLVING DATA, DATA SYNTHESIS METHODS, AND TRAINING STAGES
Zui ; Liu"; Mi Tian', Qing Tong', Weiqi Luo', Zitao Liu't Jinan University "AL Education Group "ShanghaīTech University [chenzui3, liutianqia sittianiaal,, tianmi, tongging)8tal.com [liuzitao, 1 jnu. edu. en
ABSTRACT
Mathematical reasoning remains a challenging area for large language models (LLMs), prompting the development of math-specific LLMs such as LLEMMA, DeepSeekMath, and Qwen2-Math, among others. These models typically follow a two-stage training paradigm: pre-training with math-related corpora and post- training with problem datasets for supervised fine-tuning (SFT). Despite these efforts, the improvements in mathematical reasoning achieved through continued pre-training (CPT) are often less significant compared to those obtained via SFT. This study addresses this discrepancy by exploring alternative strategies during the pre-training phase, focusing on the use of problem-solving data over general mathematical corpora. We investigate three primary research questions: (1) Can cablem-solving data enhance the model's mathematical reasoning capabilities more effectively than general mathematical corpora during CPT? (2) Are syn- thetic data from the same source equally effective, and which synthesis methods are most efficient? (3) How do the capabilities developed from the same problem- solving data differ between the CPT and SFT stages, and what factors contribute to these differences? Our findings indicate that problem-solving data significantly enhances the model's mathematical capabilities compared to general mathemati- cal corpora. We also identify effective data synthesis methods, demonstrating that the tutorship amplification synthesis method achieves the best performance. Fur- thermore, while SFT facilitates instruction-followe compared to CPT with the same data, which can be partially attributed to its poor learning capacity for more challenging problem-solving data. These insights pro- vide valuable guidance for optimizing the mathematical reasoning capabilities of LLMs, culminating our development of powerful mathematical base model called MathGPT-8B"
abilities, it underperforms
INTRODUCTION
To address the challenge of insufficient mathematical reasoning capabilities in large language mod- els (LLMs), various math-specific LLMs are developed. These include models that enhance per- formance from the pre-training stage, such as LLEMMA (Azerbayev et al., 2023). MeepSeekMath (Shao et al. 2024). InternLM-Man (Ying al. 2024), and Qwen2-Math (Yang et al. 2024a), as well as models that improve through post-training, such as MetaMath (Yu ct al. 2023). WizardMath [Luo et al. 2023), and KwaiYYiMath (Fu al., (023). These models generally follow common training paradigm. During the pre-training stage, math-related corpora are filtered from extensive internet data augment the model's mathematical knowledge. During the post-training stage, they
'These authors contributed equally. Work done during Zui Chen's intersship at Guangdong Institute of Smart Education. Jinan University, Guangzhou, China.
The corresponding author.
Model is available at/ttps: /nnctts://huggiiiii ng co/ai 4ed Wed/MathhPP1eee 88

---

Published as a conference paper at ICLR 2025
typically utilize problem datasets and their augmented versions, such as Program-of Thought (PoT) (Chen et al. 2022), evol-Instruct (Xu et al. 2023). and Tool-Integrated Reasoning (TIR) (Gou et al, 2023 Yin et al.] 2024). , to construct supervised datasets for Supervised Fine-Tuning Tuning (SFT). This
enables the models to follow instructions and produce outputs in the desired format. Recently, there is growing foeus on constructing preference datasets for the solution process to perform Step-DPO (Lai ct al. 2024) or online-RLHF (Dong et al. (204)). These approaches aim to obtain more accurate reasoning pathways, thereby significantly enhancing the mathematical reasoning capabilities of the models.
Due to the intrinsic distinction between mathematical knowledge and general world knowledge, different strategies are required for their effective acquisition and application. The primary challenge in acquiring world knowledge lies in memorizing and understanding vast amounts of information, necessitating large corpora during the pre-training phase to enhance knowledge reserves (Roberts) jtt al. 2020, Petroni et al., 2019] | et al., 2024). In contrast, mathematical knowledge involves relatively limited set of elements, concepts, axioms, and theorems that need to be memorized and understood. The real challenge often lies not recalling the relevant knowledge but in using this knowledge for reasoning planning (Hao et al. 2023).
From previous studies, it might seem that the continued pre-training (CPT) stage contributes less to mathematical reasoning abilities. However, recent studies, such Physies of LLM (Allen-Zhu & Li 2023) and MiniCPM (Hu et al.] (024), highlight the importance of teaching models how utilize memorized knowledge during the pre-training stage. These findings question the effec- tiveness of the prevalent paradigm for enhancing mathematical reasoning abilities, which primarily focuses on memorizing more mathematical knowledge during the pre-training phase and developing reasoning abilities in the post-training phase. Therefore, we propose that alternative strategies utiliz- ing mathematical problems and their reasoning steps-reeerred to as solving aata-during the pre-training phase, to teach the model how to apply its memorized knowledge rather than sim- ply increasing the volume of relevant data, could potentially lead to significant improvements in mathematical reasoning capabilities. With these considerations, we aim to explore the following fundamental research questions (RQs):
During the CPT stage, providing problem-solving data more effectively enhance the model's s mathematical reasoning capabilities compared to using general mathematical corpora?
RQ1:
RQ2: If problem-solving data can enhance mathematical reasoning capabilities, are synthetic data from the same source equally effective, and what synthesis methods are most efficient?
RQ3: How do the capabilities developed from the same problem-solving data differ between the CPT and SFT stages, and what factors contribute to these differences?
We address these three research questions separately. In Section 3 we explore RQI by comparing the impact of using problem-solye data and examining various math data mixture ratios, which leads to Result 1. In Section 4 investigate RQ2 by delving into four data synthesis techniques: response diversification, query expansion, retrospective enhancement, and tutorship amplification, resulting in Result 2. In Section [.11 we address RQ3 by first identifying. from holistic perspec- tive, the differences learning mathematical capabilities between the CPT and SFT stages using problem-solving data. Subsequently, in Section 5.2 and Section 5.3 we further analyze RQ3 by dividing the problem-solving data into subsets based on data distribution and difficulty level to in- vestigate the sources of these differences, ultimately leading to Results 3-5.
Result 1: Providing math problem-sole solving data significantly enhances the model's mathematical capabilities compared to general mathematical corpora and higher proportion of problem-sole data is more effective.
Response diversification, query expansion, and tutorship amplification were effective. Among these, tutorship amplification methods emerged as distinctly superior, leveraging teacher model to identify and correct errors based on the student model's responses, aiming to equip the model with self-correction capabilities.
Result 2:
Result 3: Overall, while SFT can facilitate some learning of mathematical capabilities, has a clear disadvantage compared to CPT.
2

---

Published as a conference paper at ICLR 2025
Result 4: From the perspective of data distribution, both SFT and CPT primarily develop capabilities aligned with their data distributions. However, SFT's in-domain (IND) learning ability is weaker than that of CPT. Regarding out-of-domain (OOD) capability learning, the conclusions less clear, with only the observation that SFT is more susceptible to disturbances from data distribution compared to CPT.
Result 5: From the perspective difficulty level, providing more challenging problem-solving data enables more effective learning, with this advantage being particularly evident in CPT compared to SFT. This may be the primary source of the learning capability differences between CPT and SFT. Therefore, we recommend preparing more challenging problem-solving data for the CPT phase.
After addressing our three RQs, we identify the optimal strategy combination and apply it to the Llama3-8B model (Dubey et al. 2024), resulting in the highly efficient MathGPT-SB. MathGPT-SB surpasses various math-specific models including
Math--aaeMat--Bae---- B (Shao et al. 2024) and and the recently released @wen2..5-Math-77. (Yang et al. 20244). We introduce only 100B mathematical tokens, equivalent to 1/10 of Qwen2.5-Math-75 and perform CPT based on a weaker base model. This validates that our proposed method is more efficient approach for enhancing mathemati- cal capabilities compared to existing paradigms. Additionally, MathGPT-8B retains strong general knowledge capabilities, post-training is conducted, we are releasing the base version of MathGPT-8BB allowing the research community to perform further post-training enhance its capabilities.
Qwen2-Math-77 (Yang et al.) all.| 2024a), and exhibits capabilities comparable to Owen2-Math-77
confirmed by MMLU (Hendrycks et al. 2020) benchmarks. Since no
2 EXPERIMENTAL PREPARATION
In this section, we provide a comprehensive overview of experimental prepartiions, including data, baseline models, and metrics.
Training Data. The training data is categorized into three groups: (1) General corpus, which includes scientific texts from the ArXiv subset of RedPajama (Weber et al. 2023), code datasets from AlgebraicStack (Azerbayev et al., 2023) and StarCoder (Li et al. (023). and natural language datasets from the C4 and Wikipedia subsets of RedPajama (Weber et al. 2023), to prevent catas- trophic forgetting and maintain robustness. (2) Mathematical corpus, which utilizes corpus on mathematical content like
(Paster et al. 2023) to improve mathematical proficiency. (3) Problem-solving data, which includes NuminaMath (Li et al. 2024), Lila (Mishra et al., 2022), and proprictary data, with 14 million pieces used for synthetic data augmentation. Our experi- ments employ 48.3B tokens from the general corpus, 13.7B from the mathematical corpus, 7.2B from problem-solving data, and 30.54B from synthetic data. Detailed descriptions are provided in Appendix A.
Base Model. We select Llama2 (Touvron al. 2023) as our base model ensure robustness in our findings, as it predates the release of Open WebMath (Paster et al.) 2.23). By choosing a model that existed before the introduction of recent mathematical corpora, we effectively mitigate the risk of contamination from these newer datasets. More details are provided in Appendix A.2
Evaluation Set. To minimize contamination of the data set and expand the capacity assessment, we expand our evaluation set to include GAOKAO and ZHONGKAO, along with GSM8K (Cobbe et al., 202T) and MATH (Hendrycks et al., 2021). GAOKAO and ZHONGKAO datasets, developed after the release of Llama2, enable the measurement of a wider range of abilities. Detailed descriptions of the data sets are provided in Appendix A.33
Deduplication and Decontamination. We use the MinHash deduplication (Lee et work to enhance training data quality by removing documents with significant duplicate content. This process includes setting specific byte thresholds for deduplication and decontamination, effec- tively climinating contaminated documents, particularly from OpenWebMath (Paster et al. 2023).
2022 frame-
More details provided in Appendix A.4
Evaluation Metrics. Our evaluation follows a three-stage process: model inference using zero- shot and fhw-shot prompts, answer comparison to handle irregular outputs, and statistical scoring to determine accuracy. In the statistical scoring stage, we select the higher accuracy between the zero-shot and few-shot approaches for each dataset to ensure the reliability and robustness of the
3

---

Published as a conference paper at ICLR 2025
results, given that models perform better in zero-shot settings while others prefer few-shot settings. We report the arithmetic mean accuracy scores across datasets. Detailed methodologies discussed in Appendix A.5
IMPROVING REASONING ABILITY IN CPT WITH PROBLEM-SOLVING DATA
We believe that, compared to simply remembering and understanding more mathematical knowledge from vast corpora, the focus of mathematical knowledge acquisition during the pre-training phase primarily lies in learning to apply this knowledge for reasoning or planning. The intuitive approach is to provide corresponding data for problem-solving. Therefore, in this section, validate RQI, speciffccll the effectiveness of providing problem-solving data during the CPT phase. This serves not only validation of main argument but also the foundation for subsequent research questions. We then continue to explore the impact of the proportion of groblem-solving data to determine an appropriate data ratio and verify the efficiency of providing groblem-solving data.
first aim to
Experiments. We design four experimental groups, including one base group and three test groups. Our goal is to demonstrate the effectiveness of providing problem-solving data by comparing the base group with the test groups, while exploring suitable data mixing ratios through comparisons among the three test groups. Specifically, the total amount of math data used in the base group and test groups remains the same, with the base group utilizing the math corpus as its math data. In contrast, the test groups employ mix of the math corpus and problem-solving data their math data, with the mixing ratios varied among the three test groups. The specific data details are as follows: where the data mixture ratio indicates the mixing proportion of general data to math data, and the math data mixture ratio reflects the blending proportion of the math corpus to problem- solving data. More experimental design discussions can be found in the Appendix
Basel: Using 48.3B general corpus and 14.7B math corpus, mixed in 4:6 ratio.
Test1: Using 48.3B general corpus, 7.5B math corpus, and 7.2B problem-solving data, with data
mixture ratio 4:6, math data mixture ratio 5:5.
Test2: Same as Testl, but using a math data mixture ratio of 3:7.
Test3: Same Testl, but using math data mixture ratio of 7:3.
Training Details. We utilize Llama2 Touvroo et al. 2023 as the base model and perform CPT for 25,000 steps, with a global batch size of 1024 and context length of 4096 tokens. The learning rate warmed up le-4 and then decays to le-5 using a cosine schedule (Loshchilov & Hutter (022)). The training data is split into 95% for training and 5% for validation. After completing the 25,000 steps, select the checkpoint with the lowest validation loss for evaluation the result.
Average Accuracy Comparison
40
Basel
Testl
35
Tast3
30
20
15
10
10
15
20
25
Step/K)
Figure 1: The average accuracy of the four groups varies with the number of steps.
Results. As shown in Figure I the blue line, representing the reference group following the current training paradigm, indicates that CPT using the math corpus effectively improves problem-solving accuracy. However, compared to the other three curves, even though Base utilizes the same number of tokens, the trend and extent of improvement in mathematical capabilities significantly lower
4

---

Published as a conference paper at ICLR 2025
than those of the three test groups. For the three test groups, the green line in Figure I shows that as the number of steps increases, its average accuracy consistently surpasses the other two. Notably, we do not introduce new tokens but simply alter the math data mixture ratio. In Appendix D we include an additional test group, Test4, which uses only problem-solving data. Although it does not utilize any math corpus tokens, it achieves performance comparable or even higher than that of the green line. Additionally, we report the accuracy of four evaluation sets for further comparison. Thus, we achieve Result 1: Providing math problem-solving data significantly enhances the model's mathematical capabilities compared to general mathematical corpora, and higher proportion of problem-solving data is more effective.
4 EXPLORATION OF EFFICIENT DATA SYNTHESIS METHODS
In the preceding sections, Results highlights the effectiveness of problem-solving data. However, the limited availability of such data compared to internet data underscores the need for efficient data synthesis methods. Additionally, it not yet fully researched whether further synthesis from the same soolvm-golviggggg data during the pre-training stage can enhance model performance. To address these issues and RQ2, we explore four data synthesis methods: response diversification, query expansion, retrospective enhancement, and tutorship amplification. Our aim is to validate the effectiveness of synthesized data and identify the most efficient synthesis method. Below, we briefly introduce the data synthesis methods used in our study.
Response Diversification aims to enhance model capabilities by generating diverse reasoning paths through methods like rejection sampling. Since it does not alter the answers, response diversification does not require additional labeling, making it easy to implement. The effectiveness of response data synthesis is established through various implementations (Yuan et al., 2023] Yu et al. 2023 Chen jet al., 2024). Instead of using a sampling-this seaiingtthen-eeee deduplllupiin approach, we require the model to follow two steps to improve the efficiency of response diversification: (1) Generate
distinct solutions based on the question and the original answer; (2) Select the solution with the correct final answer serve one diversified training sample.
Query Expansion aims to enhance model capabilities by expanding the question set. However, generating high-quality questions directly is challenging. Existing methods (e.g.. Yu al.) 2023 and Mitra et al.) 2024) leverage the concept of reshaping, which involves generating new questions based on existing questions and answers through rephrasing, reversing statements, and other tech- niques. The synthesis of new questions focuses on ensuring: (1) the accuracy of the newly generated questions, and (2) the accuracy of their corresponding answers. We integrate existing methods and emphasize these key points by requiring the LLM to perform augmentation in four steps based on the input question and solution: (I) transform the question into a statement, (2) generate new ques- tions based on the statement, (3) provide answers for the new questions, and (4) evaluate the answers and explain the reasoning. Our approach improves quality through three main aspects: first, we pro- vide the original questions and answers; second, steps 1 and 2 ensure that the generated questions are valid and solvable; and third, steps 3 and 4 involve self-evaluation to assess the quality of the answers to the new questions.
Retrospective Enhancement Ye et al. (004) posits that teaching the model to directly correct mis- takes is beneficial. They employ low-resource construction method that involves directly inserting subsequent steps into preceding ones, allowing models to retry upon regret. A special [back] token used for identification, which is why we refer
retrospective enhancement. This method is validated on GSM8K using small parameter model with minimal pre-training. Our scenario differs in two key ways: (1) we utilize more diverse question set, with some questions significantly differ- ent from the simpler forms in GSM8K; (2) we perform CPT on mainstream model that possesses certain level of mathematical capability. We aim to validate the effectiveness of this straightforward method.
Tutorship Amplification is inspired by the real-life practice of teachers guiding students to rectify mistakes. As evidenced by ApenAl (2024), models can be trained to spot errors. This agrees with Ye et al. (2024). who suggest that while models can detect errors, they lack opportunities for correction. Tutorship amplification simulates realistic error correction process. In this process, "strong" model, acting as a teacher, aids a "weak" model, representing a student. After the student model generates answer to a problem, the teacher model performs the following actions: it checks
5

---

Published as a conference paper at ICLR 2025
whether the student's answer is correct. If the answer is correct, it responds affirmatively. Otherwise, it points out the erroneous steps and continues solving from that point. We aim for this process to achieve three objectives: first, to construct realistic enrors that are likely to occur; second, to enable self-evaluation and error identification; third, to facilitate timely correction of identified mistakes. We believe these three elements will aid the model in learning self-correction and enhancing its reasoning accuracy.
Synthetic Data. seed set is created by filtering subsets from the original problem-solving data, based on the completeness of data and the number of reasoning steps involved. Following this, four data synthesis methods are applied to the seed set. Details regarding the quantity of the resulting synthetic data and associated token counts are provided in Table
Experiment. We utilize a control group, Base2, which comprises 48.3B general corpus tokens, 14.7B math corpus tokens, and 7.2B problem-solving data. In addition to the data used in Base2, we introduce extra tokens generated from the four data synthesis methods establish four experimental groups. These models are continuous pre-trained from the raw Llama2 base model. Each data combination trained for at most 25, 5,0000 steps, and the checkpoint at which the validation set loss converged is selected. The final accuracy is then evaluated based on this chosen checkpoint. Other training parameters are consistent with those in Section 3
Model
Num
Tokens
GSM8K
MATH
GAOKAO
ZHONGKAO
Average
Base2
47.84
20.12
22.98
67.05
39.50
Res-Div
14,018,544
6.82B
52.99
23.22
23.83
65.15
41,30
Query-Exp
24,459.192
4.78B
51.25
23.08
27.23
69.13
42.67
Retro-Enh
14,707,792
5.04B
45.11
21.72
22.98
66.67
39.12
Tutor-Amp
11,942,328
13.90B
64.44
35.88
32.77
69.32
50.60
Table I: Performance comparison of four experimental groups using different synthetic data meth-
ods and one control group across four evaluation sets. "Num" denotes the count of problem-solving questions and corresponding solutions used, while "Tokens" indicates the total number of tokens. The model abbreviations represent: Res-Div (Response Diversification) Query-Exp (Query Expan- sion), Retro-Enh (Retrospective Enhancement), and Tutor-Amp (Tutorship Amplification).
Results. The experimental results for the four combinations of synthetic data are presented in Table [ From this, we derive Result 2: Response Diversification, Query Expansion and Tutorship Amplification emerge as effective data synthesis techniques, with Tutorship Amplification reg- istering particularly pronounced effects. Conversely, Retrospective Enhancement appears to ex- ert minimal influence. We postulate that this could be attributed to the fact that the erroneous data constructed is not grounded in actual sampling, resulting in lower likelihood of occurrence and thereby inhibiting the model's capacity for eтror detection and rectification learning. We also no- tice that query expansion and response diversification yield limited enhancements. We propose one hypothesis for this observation: during data generation, the model's self-evaluation might have failed to identify its errors, thereby constraining the quality of the synthesized data. As for the effectiveness of Tutorship Amplification, our hypotheses are twofold: first, the model acquires a reasoning framework for self-checking, error detection, and correction through the tutorship amplifi- cation data; second, the tutorship amplification data facilitates the learning of knowledge application to correctly resolve problems via error correction.
ABILITIES ACQUISITION COMPARISON OF CPT AND SET STAGES
In the previous two sections, we have demonstrated that providing problem-solving data during the CPT phase efficiently teaches the model to apply mathematical knowledge and enhances its reasoning ability. However, how does this differ from developing mathematical reasoning skills during the SFT phase? In this section, we first verify that the change in the training stage indeed raises the upper limits of the model's capability, not merely due the data. Then, investigate the sources of differences in mathematical learning between the CPT and SET phases from two perspectives: data distributions and difficulty levels.

---

Published as a conference paper at ICLR 2025
5.1 COMPARISON ABILITIES ACQUISITION
In this section, we explore how the stage at which problem-solving data is used (CPT vs. SFT) significantly affects the model's ultimate capabilities. We have a total of problem-solving data, which can be allocated at either the CPT or SFT stage. Additionally, we sample 0.072B problem- solving data for 1%-SFT endow the model with cntrructioo-oolowinng ability. We propose the following experimental settings to compare the acquisition of learning capabilities between the CPT and SFT stages:
Basel: CPT with 48.3B general corpus and 14.7B math corpus.
Base2: CPT with 48.3B general corpus, 7.5B math corpus, and 7.2B problem-solee solving data.
Basel-SFT: SFT with 7.2B problem-solving data based on Basel.
Basel-1 SFT: SFT with 0.072B problem-sol data based on Basel.
Base2-1%SFT: SFT with 0.072B problem-solving data based on Base2.
important to note that we perform SFT on both Basel and Base2 using 1% of the problem- solving data. This setup allows to isolate the impact of instruction-followii capability improve- ments and thereby assess the true enhancement mathematical reasoning ability brought about by introducing problem-solving data the CPT stage.
Experiment Details. During the SFT stage, we set batch size of 256 and a learning rate that decayed from le-5 le-6 following cosine schedule. We train for epochs. ensuring that the training loss converged. After convergence, we select the optimal result from 10 checkpoints for reporting, which typically occurred around the checkpoints at 2 epochs. More experimental design discussions can be found in the Appendix
Base2-1%SFT. 13.544
Instruction Following
Base2, 39.5%
CPT 4g. 19 81%
Basel-SFT, 35.62%
SFT 11. 89%
CPT 4,, 20. 39%
Base1-1%SFT, 23.73%
Instruction Following
Basel, 19.11%
Figure 2: Comparison of the acquisition of learning capabilities between the CPT and SFT stages
Results. The evaluation results across the four datasets can be found in Appendix E Their average accuracy is illustrated in Figure 2 First, we observe the red and blue shaded areas, where a small amount of SFT data brought similar improvements results, this improvement stems from significant reduction in the model's previously inconsistent and repetitive outputs. We believe this result of the supervised approach SFT, leading to leading to an interesting conclusion: A small amount of SFT data is sufficient to enhance the model's ability follow instructions.
both Basel and Base2. From the evaluation
Next, we compare the results after removing the influence of instruction following capabilities. At this point, the differences, denoted as SFT A and CPT Ag. can viewed the improvements in mathematical reasoning ability obtain during the SFT and CPT phases, respectively. Given that both use the same data, but the capability gain in SFT is only about 60% that achieves during CPT. Additionally, comparing B-ST and Base2, despite using the same data, 1-sc also gains the ability to follow instructions, yet its performance is still inferior to Base2. Thus we conclude Result 3: Overall, while SFT can facilitate some learning of mathematical capabilities, it has clear disadvantage compared to CPT.
To better understand SFT's impact learning capabilities, add three additional experimental groups, where we perform SFT with 10%, 20%, and 50% splits of the problem-solving data. We compare these with Basel, 1% SFT, and 100% SFT analyze the effect of SFT data volume

---

Published as a conference paper at ICLR 2025
reasoning improvement. The results shown in Figure We observe a significant increase in average accuracy at the 1% SFT markgroup, followed by a logarithmic- data volume and accuracy improvement. This further validates that small amount of SFT data enhances the model's ss ability to follow instructions. Moreover, increasing the SFT data may continue to logarithmically improve the model's 's reasoning ability.
relationship between
5.2 IMPACT OF DIFFERENT DATA DISTRIBUTIONS
In the previous section, we observe that the reasoning capability learned during the SFT phase is significantly weaker compared CPT. In this section, we aim to explore the source of this differ- ence. Our intuition that data distributions might have different impacts on capability learning each stage, with CPT possibly contributing enhanced out-of-distribution (OOD) performance. However, our findings contradict this hypothesis. Both CPT and SFT primarily develop capabilities aligned with the data distributions they are trained on.
Experiment. We design our experiments by segmenting the training data based on evaluation sets. Specifically, we select one evaluation set to represent in-distribution (IND) capabilities, with the remaining sets are considered out-of- outt----distiittiinn (OOD). Correspondingly, we retain only the por- tions of the training data aligned with IND capabilities. However, it is important to note two key challenges: first, during the decontamination process, we already exclude any data that overlapped with the evaluation sets; second, the scope of mathematical abilities inherently includes overlap and coverage across different areas. Due to these factors, it is challenging to perfectly match training data to specific capabilities. Therefore, we utilize knowledge point labels from the original problem- solving data segment out 0.83B middle school data, corresponding to ZHONGKAO capabilities, and 0.89B high school data, corresponding to GAOKAO as its IND capabilities. The OOD capabilities represented by the remaining evaluation sets that do not align with these IND capabilities. More experimental design discussions can be found in the Appendix C The specific experimental design is follows:
its IND
Base As described in Section 3 CPT with 48.3B general corpus and 14.7B math corpus. Middle-school-SFT: SFT with 0.83B middle school data on Basel.
Middle-school-CPP CPT with Basel data and middle school data.
High-school-SFT:: SFT with 0.89B high school data on Basel High-school-CPT: CPT with Basel data and high school data.
Model
GSM8K
MATH
GAOKAO
ZHONGKAO
Average
Basel
28.20
9.48
8.09
30.68
19.11
Middle-school-SFT
22.67 (5553)
16.36 (+6.88)
10.21 (+2.12)
52.28 (+21.60)
25.38 (+6.27)
Middle-school-CPP
29.42 (+1.22)
15.04 (+5.56)
8.09 (0.00)
54.71 (+24.03)
26.81 (+7.70)
High-school-STT
19.11 (-9.99)
13.48 (+4.00)
16.60 (+8.51)
36.78 (46.10)
21.49 (+2.38)
High-school-CPT
23.96 (-4.24)
13.82 (+4.34)
22.98 (+14.89)
34.19 (43.51)
23.74 (+4.63)
Table 2: Learning capabilities analysis across various data distributions.
Results. As shown in Table 2 for the IND capabilities represented by bolded evaluation results, learning during the CPT stage consistently leads greater improvements compared to learning dur- ing the SFT stage. This effect is especially evident in the learning of more challenging high school- level knowledge. In addition, for OOD capabilities, learning during the SFT stage experiences significantly more disruption. This is particularly noticeable for GSM8K (see the data distribution and capability dimension chart in Appendix B). which has the largest distributional difference. After SFT, the model's performance on OOD tasks suffers more compared to CPT. Thus, we achieve Re- sult 4: Both SET and CPT primarily develop capabilities aligned with their data distributions. However, SFT's IND learning ability is weaker than that of CPT. Regarding OOD capability learning, the conclusions are less clear, with only the observation that SFT is more susceptible to disturbances from data distribution compared CPT.
8

---

Published as a conference paper at ICLR 2025
5.3 IMPACT OF DIFFERENT DIFFICULTY LEVELS
In the previous section, although we clarify that both CPT and SFT involve in-domain capability learning, remains unclear what cause SFT's learning performance to be weaker than CPT's. How- ever, conclusions in Result 4 are more evident in the high school training data compared to middle school, prompting us to explore the difference in learning capabilities between CPT and SFT with varying difficulty levels problem-solving data.
Experiment. We select a 5B subset of our problem-solving data and categorize based on the num- ber of solution reasoning steps: data requiring 1-3 steps is classified easy, 4-7 steps medium, and 8 or more steps as hard. The distribution of samples account for 36.0%. 38.4%, and 25.6% of the total data, respectively, while token counts make up 23.0%, 36.0%, and 41.0%, respectively. Given the unavoidable inaccuracies in this method of categorization, we focus solely on easy data and hard data for the CPT and SFT comparison experiments. More experimental design discussions can be found in the Appendix C The experimental groups are designed as follows:
Basel: As described in Section 3
CPT with 48.3B general corpus and 14.7B math corpus. Easy-SFT: SFT using the easy data subset on top of Base
Easy-CPT: CPT incorporating both the Basel data and the easy data subset.
Hard-SFT: SFT using the hard data subset on top of Basel.
Hard-CPT: CPT incorporating both the Basel data and the hard data subset.
Model
GSM8K
MATH
GAOKAO
ZHONNGKA
Average
Easy
Medium
Hard
Basel
28.20
9.48
8.09
30.68
19.11
14.86
6.69
4.85
Easy-SFT
31.31
14.46
14.04
48,30
27.03
22.52 (17.6)
10.68 (14.00
6.94 (12.09)
Easy-CPT
37.98
15.70
17.02
52.46
30.79
27.61 (11275)
13.33 (16.4)
6,27 (1142)
Hard-SIT
31.39
17.40
15.32
54.55
29.66
24.37 (1951)
11.93 (15.24)
6.84 (+1.99)
Hard-CPT
45.79
23.96
26.38
69.89
41.51
35.78 (42009)
20.17 (41344)
9.32
Table 3:
Performance comparison of CPT and SFT models on different difficulty levels. The table shows the average and specific performance on easy, medium, and hard data subsets.
Results. The results in the left half of Table 3 which is divided by vertical lines, show that CPT models consistently outperform SFT models, with some relative improvements specifically indi- cated. Notably, Hard-CPT exhibits greater relative enhancements compared to Easy-CPT, and these improvements are not limited to just the hard domain accuracy but are observed across all datasets. Moreover, regardless of whether it is SFT or CPT, training on hard data consistently yields better results compared to training on easy data. This suggests Result S: Providing more challenging problem-solving data enables more effective learning, and this advantage is particularly ev- ident in CPT compared to SFT. This may be the primary source of the learning capability differences between CPT and SFT. Therefore, given limited computational resources, we rec- ommend preparing more challenging problem-solving data for the CPT phase.
The results in right half of Table 3 indicate that both SFT and CPT models achieve their highest improvements Easy problems, with reduced gains problem difficulty increases. For example, Easy-SFT and Easy-CPT show signiticant improvements of +7.66 and +12.75 on Easy problems, but only +2.09 and +1.42 on hard problems, respectively. Similarly, Hard-SFT and Hard-CPT exhibit their largest gains on easy problems (+9.51 and +20.92) compared to hard problems (+1.99 and +4.47). These patterns suggest that Regardless of the training data's 's difficulty, both SFT and CPT primarily focus on learning to solve simpler, fewer-step problems.
TRAINING STRONG MATH-SPECIFIC MODEL
To further validate the effectiveness of our empirical results, we aim to train a strong math-specific model based on the Llama3-8B (Dubey et al. 2024), named MathGPT-SB. We follow the conclu- sions from the three RQs outlined earlier: (1) We maintain a 3:7 ratio of mathematical corpus to problem-solving data; (2) We synthesized data from Query Expansion, Response Diversifica- tion, and Tutorship Amplification, with a focus on expanding data using the most efficient Tutorship Amplification method; (3) We filter and expand the raw data by focusing on problems with more than
9

---

Published as a conference paper at ICLR 2025
five reasoning steps, using these seed data generate additional synthesized data. In addition, we incorporate newly released mathematical corpora (Han et al., 2024) into the training. Ultimately, use 39.6B general corpus tokens, 46.7B mathematical corpus tokens, and 51.1B problem-solving data and synthesized data tokens to train MathGPT-SB for 25,000 steps, with a global batch size of 1024 and context length of 8192 tokens. The learning rate is warmed up 1e-4 and then decayed to le-5 using a cosine schedule.
Results. As presented in Table 4 compared the base model, significantly enhance the founda- tional capabilities of Llama3-8B. even surpassing larger models such as Llama3. 1-70B and Qwen2- 72B, which have over 70 billion parameters. Additionally, we evaluate our model using the Gao al. (2224) on the MMLU (Hendrycks et al. (2200) benchmarks, achieving a score of 0.6222 compared to Llama3-8B's 0.621 demonstrating that maintained its general knowledge capabilities.
Compared to math-specific base models, MathGPT-8B outperforms Metp DeepSeekMath-Base--78 (Shao al. 2024) and Qwen2-Math-7BB (Yang al., 2024a), and exhibits capabilities comparable to Owen2-Math-745 and the recently released Qwen2.5-Math-71 (Yang al., (024b). Compared to Owen2.5-Math-77B, MathGPT-SB trained on only 140 billion tokens (100 billion of which are math-related), while Qwen2.5-Math-77 utilizes I trillion tokens, as reported. Additionally, MathGGT-8B starts from a weaker base model. These findings validate our proposed method as efficient approach enhancing mathematical capabilities compared to existing paradigms. Fur- ther discussions on related work can be found in Appendix F Since we do not perform a complete post-training process,
releasing the base version of model. This allows the research community to conduct further post-training to enhance its capabilities as needed.
Model
GSM8K MATH
GAOKAO
ZHONGKAO
Average
General Models
Llama3-8B
58.38
17.04
13.62
42.61
32.91
Llama3-70B
82.34
38.42
28.09
64.02
53.21
Llama3. 1-8B
56.79
19.70
11.49
44.70
33.17
Llama3. 1-70B
81.73
39.66
31.06
64.77
54.31
Qwen2-7B
80.44
47.82
27.23
70.45
56.49
Qwen2-72B
56.88
45.11
73.67
65.56
Qwen2.5-7B
84.61
53.22
45.53
80.30
65.92
Qwen2.5-72B
90.60
59.38
56.60
82.95
72.38
Math-specific Models
LLEMMA-7B
41.47
18.94
14.89
45.08
30.10
DeepSeekMath-Base-77
65.73
33.40
23.83
62.69
46.41
Owen2-Math--7
80.67
53.02
42.13
77.08
63.22
Qwen2-Math-27
88.63
61.88
51.91
81.25
70.92
Qwen2.5-Math
85.44
59.10
53.19
78.79
69.13
Awen2.5-Math-725
88.70
67.10
53.62
81.63
72.76
(MathGPT-8B)
81.20
60.38
60.43
80.49
70.62
Table Model performance metrics (General and math-specific models)
CONCLUSION
In this study, we investigate the enhancement of mathematical reasoning capabilities in LLMS through alternative pre-training strategies. Our findings lead to the development of MathGPT-8B, competitive model that outperforms most 7B models and exhibits capabilities comparable to much larger models despite being trained fewer tokens. Future work should expand in two key areas. First, we need to refine the data synthesis methods. Although we have demonstrated the effective- ness of synthetic data, current approaches relatively naive. Second, should explore the role and impact of alignment processes during post-training Investigating these aspects will help further improve the mathematical reasoning capabilities of the model.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by National Key R&D Program of China, under Grant No. 2022YFC3303600; in part by NFSC under Grant No. 62477025: in part by Key Laboratory of Smart Education of Guangdong Higher Education Institutes, Jinan University (2022LSYS003) and part by Beijing Municipal Science and Technology Project under Grant No. 2241100001324011
REFERENCES
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. In Forty-first International Conference on Machine Learning. 2023.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. In The Twelfth International Conference on Learning Representations, 2023.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompt- ing: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2022.
Zui Chen, Yezeng Chen, Jiaqi Han, Zhijie Huang, Ji Qi, and Yi Zhou. An empirical study of data ability boundary in lims' math reasoning. arXiv preprint arXiv:2403.00799 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10, /4168 2021.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong. and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint axXiv:2405.0088 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint 24Xi: 1783, 2024.
Jiayi Fu, Lei Lin, Xiaoyang Gao, Pengli Liu, Zhengzong Chen, Zhirui Yang, Shengnan Zhang, Xue Zheng, Yan Li, Yuliang Liu, et al. Kwaiyimath: Technical report. arXiv preprint arXiv:2310.07488 2023.
Lco Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 2024.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. Tora: tool-integrated reasoning agent for mathematical problem solving, In The Twelfth International Conference Learning Representations, 2023.
Xiaotian Han, Yiren Jian, Xuefeng Hu, Haogeng Liu, Yiqi Wang, Qihang Fan, Yuang Ai, Huaibo Huang, Ran He, Zhenheng Yang, al. Infimm-wed webmath-40b. Advancing multimodal pre- training for enhanced mathematical reasoning. In The 4th Workshop on Mathematical Reasoning and Al NeurIPS 24, 2024.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong. Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 8154-8173, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. The Eighth International Conference Learning Representations, 2020.
11

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.00 06395, 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 01222199-22211 2022.
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step- wise preference optimization for long-chain reasoning of Ilms. arXiv preprint (rXi::2406.18629 2024.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pp- 2022.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024.
Raymond Li, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, LI Jia, Jenny Chim, Qian Liu, et al. Starcoder: may the source be with you! Transactions Machine Learning Research, 2023.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In Interna- tional Conference on Learning Representations, 2022.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qing- wei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09555 2023.
Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal. Peter Clark, et al. Lila: A unified benchmark for mathematical reasoning. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp. 5807-5832, 2022.
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math. arXiv preprint arXiv:2402.14830, 2024.
OpenAl. Finding GPT-4's mistakes with GPT-4, 2024. URL httpp::/ com /ndm//
Finding- 4p -ptting--pt-ss
mistakes wwwthwtthe -pt-pppt----
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. In The Twelfth International Conference on Learn- ing Representations, 2023.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp. 2463-2473, 2019.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge you pack into the parameters of a language model? Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pp. 5418-5426 2020.
12

---

Published as a conference paper at ICLR 2025
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint a1Xhv:2402.03300, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Maurice Weber, Daniel Y Fu, Quentin Gregory Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao. Virginia Adams, et al. Redpajama: an open dataset for training large language models. In The Thirty-eight Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 2023.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2024a.
An Yang. Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jian- hong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409. 2024b.
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of language models: Part 2.2, how learn from mistakes on grade-school math problems. arXiv preprint arXiv:2408. 16293, 2024.
Shuo Yin. Weihao You, Zhilong Ji, Guoqiang Zhong, and Jinfeng Bai. Mumath-code: Combin- ing tool-use large language models with multi-perspectivy data augmentation for mathematical reasoning. arXiv preprint arXiv:2405.0755/ 2024.
Huaiyuan Ying. Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, al. Internlm-math: Open math large language models toward verifiable reasoning, arXiv preprint arXiv:2402.063333 2024.
Longhui Yu, Weisen Jiang, Han Shi. YU Jincheng, Zhengying Liu, Yu Zhang, James Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2023.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. Scal- ing relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.011 0125, 2023.
APPENDIX
A DETAILED EXPERIMENT PREPARATION
A. TRAINING DATA DETAILS
The training data utilize in our study is categorized into three distinct groups: (1) General cor- pus, encompassing scientific texts from the ArXiv subset of RedPajama (Weber et 2023), code datasets from AlgebraicStack (Azerbayev et al., 2023) and StarCoder (Li et al. 2023). along with natural language datasets from the C4 and Wikipedia subsets of RedPajama (Weber et al. 2023). The
inclusion of general data helps prevent the model from experiencing catastrophic forgetting, where it might lose previously acquired knowledge during specialized training. Moreover, maintaining a broad base of general knowledge ensures the stability and robustness of the model, enabling it to retain well-rounded understanding and perform effectively across various tasks. (2) Mathemat- ical corpus is designed enhance the model's proficiency in mathematies, primarily comprising
13

---

Published as a conference paper at ICLR 2025
general mathematical content extracted from sources like CommonCrawl web pages. The main objective is to imbue the pre- pre-trained model with foundational mathematical knowledge, including terminology, theorems, proofs, etc. To achieve this, we utilize OpenWebMatte (Paster et al. 2023), a resource shown to effectively improve mathematical capabilities, as demonstrated in (Azerbayey et al. 2023). (3) Problem-solving data, which believe can more efficiently enhance the model's reasoning abilities. We collect 25 million pieces of uroblem-solving
data, including those from open-source resources such as NuminaMath (Li al. 2024) and Lila (Mishra et al. 2022), well as proprietary data. Among them, 14 million pieces are used as seed data for augmentation to create our synthetic data. Overall, using the Llama2 (Touvron al. 2023) conduct experiments on RQs, we employ 48.3B tokens from the general corpus, 13.7B from the mathematical corpus, 7.2B from problem-solving data and 30.54B from synthetic data.
A.2 BASE MODEL SELECTION
The selection of the base model is pivotal in shaping our conclusions, as it directly influences the reliability and applicability of our findings. To ensure that exploration of research questions yields practically valuable insights, we have chosen to base our study on mainstream models. Con- sidering that OpenWebMate may have been widely incorporated into recent LLMs, introducing this mathematical corpus might not produce the desired effect. Therefore, we select Llama2 (Touvron al. 2023), which is released prior to Open WebMath (Paster et al. 2023). as our base model. This decision aims to enhance the robustness of our conclusions.
A.3 EVALUATION DATASETS
Considering both the risk of dataset contamination and the scope of capabilities, we expand the evaluation set to include GAOKAO and ZHONGKAO, in addition to GSM8K (Cobbe et al.) (021) and MATH (Hendrycks al. 2021I. The GAOKAO dataset comprises both GAOKAO-2023 and GAOKAO-2024, derived from the most recent Chinese National College Entrance Examinations.
We convert the problem format into math word problems, translate the questions, and retain 235 items after review. Similarly, the ZHONGKAO dataset is sourced from the 2023 Chinese High- School Entrance Examination and includes 658 translated math word problems. Both GAOKAO and ZHONGKAO datasets are created after the release of Llama2 (Touvron et al., 2023), which strengthens our conclusion. These additional datasets provide coverage of different dimensions of ability compared to GSM8K and MATH. From the perspectives of general knowledge, math knowl- edge, and reasoning steps. GAOKAO is similar to MATH but demands more general knowledge, while ZHONGKAO is akin GSM8K but may require more mathematical knowledge and fewer reasoning steps. Detailed ability dimensions can be found in Appendix B We believe this expanded evaluation will lead to more comprehensive assessment and serve valuable reference for subsequent improvement.
A.4 DEDUPLICATION AND DECONTAMINATION
We use the MinHash deduplication Lee et al. (2022) framework remove entire documents con- taining duplicate text that exceeds a certain threshold from the training data. Specifically, we set threshold of 2048 bytes for deduplication to improve the quality of the training data. Additionally, we set a threshold of 100 bytes to remove any data from the training set that contains more than 100 bytes of overlapping text with subsets of the train and test sets in the evaluation data. We believe this can account for some contamination caused by simple paraphrasing. (Notably, in the case of Open- WebMath (Paster et al.) (2023), we remove 2594 contaminated documents, which have significant impact on the conclusions during our initial experiments.)
A.5 EVALUATION METRICS
The evaluation process comprises three stages: model inference, answer comparison, and statistical scoring. During model inference, we utilize both zero-shot and few-shot prompt templates for each dataset. For the zero-shot approach, employ simple Chain-of-Thouge (CoT) prompt (Kojimal et al. 2022). In the few-shot approach, we use 8-shot and 4-shot settings for the GSM8K and MATH datasets, respectively, and apply the same few-shot settings from GSM8K and MATH to
14

---

Published as a conference paper at ICLR 2025
the ZHONGKAO and GAOKAO datasets. For answer comparison, model to address issues related to the irregular output of the base models, such as inconsistent stopping criteria and extracting answers from CoT prompts. In the statistical scoring stage, select the higher accuracy between the zero-shot and few-shot approaches for each dataset to ensure the reliability and robustness of the results, given that some models perform better in zero-shot settings while others prefer few-shot settings. Finally, we report the arithmetic mean of the accuracy scores across the four datasets the average accuracy.
answer comparison
B ABILITY DIMENSIONS OF THE FOUR EVALUATION SETS
GSM8K, MATH, ZHONGKAO, and GAOKAO, four evaluation sets, are introduced to enrich the dimensions of the evaluation, as shown in Table 3 with example problems.
To preliminarily understand the differences in capabilities across various dimensions of the evalua- tion process, we attempt to define three capability dimensions: general knowledge, math knowledge, and reasoning steps. As Table [ illustrates, each capability dimension is divided into three levels, with requirements progressively increasing from Level to Level 3. General knowledge describes the demands for understanding common sense, such as the fact that a day consists of 24 hours; math knowledge refers to the complexity of mathematical knowledge, including arithmetic, elementary, and advanced mathematics; reasoning steps describe the depth of reasoning. Figure 3 displays the performance of the four evaluation sets across different dimensions. Overall, GAOKAO and MATH represent similar capability dimensions, but GAOKAO might require some general knowledge for certain problems. ZHONGKAO and GSM8K both demand higher level of general knowledge, but differ in their requirements for math knowledge and reasoning steps.
Furthermore, shown in Figure
analyze the data distribution of problems in the datasets to clarify the data distribution of different evaluation sets and the impact of different data distributions on
(OOD) capabílities as discussed in Section 5.2 Specifically, we sample up to 1,000 problems from the evaluation sets and used t-SNE for dimensionality reduction, with the visualization shown in 4(a) and the cosine similarity situation in 4()..
evident that MATH, ZHONGKAO, AO, and GAOKAO have certain correlations, whereas GSM8K exhibits the largest distri-
butional difference. This may also explain why different evaluation sets perform differently in terms of OOD capabilities, discussed in Table 2 and related conclusions.
Dataset
Problem
GSM8K
Josh decides to try flipping house. He buys house for $80,000 and then puts in $50,000 in repairs. This increased the value of the house by 150%. How much profit did he make?
MATH
How many vertical asymptotes does the graph of y x242 have?
ZHONGKAO
What is the opposite number of 4?
GAOKAO
Given the sets M {a| æ + 03. N [x|o 1 0}. what is M NON== =?
Table 5: Example problems from four evaluation sets
C DISCUSSION ON EXPERIMENTAL SETTINGS
Our experimental design generally adheres to the principles of comparative experiments, forming control groups to test hypotheses by introducing variations. Below, we elaborate on the considera- tions behind the design of each experimental group.
In Section 3 to mitigate the influence of the total amount of math data used on the experimental conclusions, as described in the main text, we control the total amount of math data used in both the base group and the test groups to be the same. Specifically, the base group uses the entire 14.7B math
http:s//wwaggggiiieee//////////r -aamaea-Caaaaa-CCaar-eeee- Compare Answer
15

---

Published as a conference paper at ICLR 2025
Competency Dimension
Level
Definition
Involves minimal General Knowledge
General Knowledge
2
Less than 50% of the problems require General Knowledge
More than 50% of the problems require General Knowledge
Basic arithmetic operations
Math Knowledge
Requirements for the Chinese High School Entrance
Examination
3 Requirements for the Chinese National College Entrance
Examinations
Within 1-3 steps
Reasoning Steps
Within 3-5 steps
More than 5 steps
Table 6: Definitions of Competencies Across Different Levels
GSMBK
Math Knowladge
ZHONGKAO
GADKAO
Level
Level
General Kndwledge
Reasohing
Figure Ability dimensions of four evaluation sets
corpus as the math data, while the groups use 7.2B problem-solving data and split the remaining 7.5B math corpus to ensure that the total math token was also 14.7B. Subsequently, in three test groups, we adjust the math data mixture ratio to further verify the effectiveness of problem-sol data and determine the optimal mixture ratio for subsequent experiments. Specifically, based on the token ratio 7.5::7.2B, we start with math data mixture ratio of 5:5. Then, we adjust the ratio to 3:7, where the smaller 7.5B math corpus could be fully utilized within 10K steps and used more than twice within 25K steps. We believe this ensures the full utilization of data. Finally, we complement the experiment with reverse ratio of 7:3.
In Section to delineate the impact of different synthetic data, we introduce a control group, Base2, which used the entire problem-solvine data top of Basel. The experimental group further incorporate synthetic data into this setup. We aim to verify that the synthetic data contributed new value rather than merely substituting the original data.
16

---

Published as a conference paper at ICLR 2025
Similartty
CORIDK MATH
DAOCAG ZHONDCAD
DSPER
00
-04
(a)
(b)
Figure 4: (a) Data distribution of problems of the four evaluation sets. (b) Dataset similarity based on data distribution calculation.
In Section 5.1 we compare how the stage at which problem-solving data is introduced (CPT vs. SFT) significantly affects the model's ultimate capabilities. First, follow the setups of Basel and Base2 and conduct SFT using the same data on Basel to create comparative experiment Base 1- SFT. We hypothesize that Basel-SFT would benefit from enhanced instruction-followinng ability, which Base2 might lack. To validate this, we partition 1% of the data, assuming it has limited impact on reasoning ability contributes to instruction-followinn ability. Subsequently, we apply this 1% data for SFT both Basel and Base2 groups. By comparing Basel-1 Basel-I%SFF with Basel- SFT, we evaluate the reasoning ability gained from SFT, and by comparing Basel-Ie 11ST with SaST,, assess the reasoning ability gained from CPT.
we primarily focus on differences capabilities across the same evaluation dataset representing the data distribution various training stages. To this end, Basel is reused to define the improvement in the experimental groups' abilities and then we introduce two experimental groups, Middle-school-SFT and Middle-school-CCPT, which use Middle-school data subset from the train- ing set for SFT and CPT, respectively, forming a comparison to evaluate the IND learning differences between SFT and CPT on specific evaluation dataset. Additionally, the differences in OOD learn- ing on other evaluation datasets are also analyzed. Subsequently, we replace the Middle-school sub- with a High-school subset in the training set, implementing two additional experimental groups and repeating the same experiments to strengthen the robustness of the conclusions.
In Section 5.2
In Section 5.3 the experimental design is similar that in Section 5.2 with key difference: we focus on variations in learning ability of training data with different difficulty levels at different training stages. Thus, besides comparing SFT and CPT using easy and hard subsets of the training data separately, we also contrast the performance of different training subsets within the same train- ing stage. Comparisons based on evaluation dataset distributions are used only as supplementary analysis.
D DETAILED RESULTS OF PROBLEM-SOLVING DATA EFFECTIVENESS EXPERIMENT
We hypothesize that simply performing CPT with the problem-solving data could be sufficient, and add an experimental group, Test4, which uses only problem-solving data as the math data, specifically:
Test4: Using 48.3B general corpus and 7.2B problem- data, with data mixture ratio 4:6.
The results shown in the Figure 5 We find that even with smaller amount of math data, the purple line corresponding to Test4 and the green line corresponding to Test2, which represent the experimental groups using the smallest math data mixture, demonstrate consistent even superior
17

---

Published as a conference paper at ICLR 2025
Average Accuracy Comparison
40
Teat1
35
Testd
25
20
15
10
15
20
Step1k)
Figure
The average accuracy of the five groups varies with the number steps.
Model
GSM8K
MATH
GAOKAO
ZHONGKAGAverage
Llama2-7b
14.40
5.10
4.26
16.48
10.06
Basel
28.20
9.48
8.09
30.68
19.11
Testl
44.88
19.72
20.00
66.29
37.72
Test2
48.29
20.78
23.40
67.05
39.88
Test3
42.15
19.48
22.55
63.26
36.86
Test4
40.11
25.12
29.79
68.18
40.80
Table 7: Accuracy of the all experimental groups across the four evaluation set.
performance. This strongly complements Result 1, highlighting the effectiveness of data, which can even surpass the impact of adding large number of new tokens.
The detailed results of Basel, Testl, Test2, Test3 and Test4 are in Table 7 Notably, when the math corpus is not used, the specific metrics of Test4 across the four evaluation datasets no longer align with those of Testl-3. This shift data distribution undermines the improvement in GSM8K performance while enhancing the improvements in MATH and GAOKAO capabilities.
E DETAILED RESULTS OF COMPARISON OF ABILITIES ACQUISITION
The evaluation results across the four datasets are in Table g And the relationship between average accuracy and SFT data quantity is in Figure 6
F RELATED WORK
We discuss the related work on math CPT. LLEMMA (Azerbayev et al., 2023) initially focused on continuing pre-training to enhance mathematical reasoning capabilities, collecting open-source data including from OpenWebMate (Paster et al. 2023) and providing the Proof-Pile-2 dataset. They made preliminary attempts at continuous pre-training the mathematies domain and shared their experiences. DeepSeekMath (Shao et al. 2024) advanced the effects of mathematical con- tinuing pre-training by improving data quality, primarily training a fastText model to recall more OpenWebMathe WebMath-like mathematical web pages and iterating this process, which also provided reliable experience for research beyond mathematical reasoning. InternLM-Math (Ying et al.) 2024) uti- lized open-source datasets and internal datasets and trained a scoring model to identify high-quality datasets. Qwen2-Math (Yang et al., 2024a) and the more recent Qwen2.5-MM (Yang et al. (024b) have begun to focus using synthetic data, effectively achieving significant improvements.
18

---

Published as a conference paper at ICLR 2025
Model
GSM8K
MATH
GAOKAO
ZHONGKAOAverage
Basel
28.20
9.48
8.09
30.68
19.11
31.08
12.10
12.34
39.39
23.73
Basel-10%%S
32.37
13.74
11.49
42.42
25.01
Basel -20%STT
34.65
16.26
13.62
46.40
27.73
Base 150%SFT
36.92
19.34
14.04
57.20
31.88
Basel-SFT
42.84
21.88
18.30
59.47
35.62
Base2
47.84
20.12
22.98
67.05
39.50
Base2-1 1FFT
51,40
27.10
25.96
69.70
43.54
Table 8: Model performance metrics with SFT
Average Accuracy vs SFT Data Size
35.0
32.5
30.0
Accuracy (%)
25.0
22.5
20.0
0%
20%
40%
60%
80%
100%
SFT Data Size
Figure 6: The relationship between average accuracy and SFT data quantity.
19

---

Published as a conference paper at ICLR 2025
G
SYNTHETIC DATA PROMPT
Response Diversification
You math teacher, please complete the following task using English.
Your task is: for the math problem below, in addition to the given solution, provide two more different solutions.
If you can provide two different solutions, start with < response >cccept< /response >, then present the additional solutions beginning with 'Solution2:' and 'Soolutio3:"" respec- tively.
If you believe you cannot offer two different solutions the solutions provided might be inaccurate, start with <, response >refuse<, /response >
Please ensure the solutions correct and distinct. If you doubt the correctness of your solutions, or it seems the problem does not allow for multiple solutions, directly indicate refusal by starting with response >refuse< /response and then explain the reason.
Query Expansion
Your goal is to create different math word questions and their solutions from given question and its solution. You should follow these steps:
1. Convert the question into statement and fill in < statement >> FILL IN HERE statement >.
2. Create
question based the statement and fill in question > FILL IN HERE < /question >.
Provide solution the new question and fill in solution FILL IN HERE
< /souution >.
Check the solution and report < check Accept< /check or check Refuse /check >. And then fill the reason in < reason > FILL IN HERE /reason >
Repeat the process for total of 2 questions and solutions.
Tutorship Amplification
As mathematics teacher, please check the solution to the following math problem. the solution is correct, please only reply < check >correct< /check >.
If the solution is incorrect, please first respond with < check >wrong<< /check >, then identify the erroneous steps, correct the errors, and continue to provide the correct solution.
Please note that your response must include check >correct<" /check check >wrong< /check > at the beginning of the response.
20


Reviews:
[{"review_id": "qoOfiOC7xo", "reviewer": "Reviewer_fBsg", "summary": "This paper investigates the impact of incorporating problem-solving data, various data synthesis techniques, and different training stages on enhancing mathematical reasoning capabilities in large language models (LLMs). The authors examine whether problem-solving data improves continued pre-training (CPT) effectiveness over traditional mathematical corpora and explore the optimal mixture ratios of such data. The study also assesses four data synthesis methods—response diversification, query expansion, retrospective enhancement, and tutorship amplification—highlighting the latter as particularly effective. Additionally, the authors compare the mathematical skills acquired during CPT versus supervised fine-tuning (SFT), finding that CPT leads to stronger mathematical reasoning capabilities, especially on complex, multi-step problems. These insights are applied to create JiuZhang-8B, a model that achieves competitive performance on mathematical reasoning benchmarks against other math-specific models.", "questions": "See weakness for questions\n\n1. The authors claims of collecting 25 million problem solving data. Assuming authors collected approximately 930,000 pieces from NuminaMath [1] and 140,000 from Lila [2], Does it mean that the proprietary problem solving data alone contains ~24 million pieces? or are the pieces calculated differently from the dataset websites?\n\n2. A brief summary on how Figure 3 (Ability dimensions of four evals) is computed would be helpful in understanding Section 5.2.\n\n[1] https://huggingface.co/collections/AI-MO/numinamath-6697df380293bcfdbc1d978c\n\n[2] https://lila.apps.allenai.org", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper provides an in-depth analysis of the effects of problem-solving data on mathematical reasoning, particularly through comparisons of CPT and SFT, with valuable insights into the role of SFT’s instruction-following capabilities.\n\n2. The authors explore and  compare different data synthesis methods, which is useful. The insights shared on the performance difference for the four different methods would be helpful for the LLM Reasoning community.\n\n3. The comparison between models trained on SFT and CPT at varying difficulty levels adds further depth to understanding model capabilities.\n\n4. Most of the sections are clearly written and well-structured, with detailed methodology on dataset curation and training, making it accessible and informative for the research community.\n\n5. The final results of JiuZhang-8’s performance compared to SOTA LLMs is impressive, validating the insights and techniques shared earlier. The release of the model’s base version would greatly support community development.", "weaknesses": "1. The section on exploring the impact of different data distributions (5.2) is confusing and needs more clarification:\n    1. While the authors note the inherent overlap in mathematical knowledge and the challenge in aligning data distribution with specific capabilities, the extent of overlap between knowledge points remains unclear. A dataset similarity analysis could be helpful to address questions raised below.\n    2. The capability analysis in Figure 3 suggests that GAOKAO’s knowledge capabilities  subsumes MATH, while ZHONGKAO encompasses questions requiring higher general knowledge capabilities than MATH. This should mean that a model trained to perform better on GAOKAO should perform even better in MATH than a model trained to perform better in ZHONGKAO. However this is not the case, The authors designate middle school data as IND to ZHONGKAO and high school data as IND to GAOKAO, yet models trained on middle school data (both SFT and CPT) outperform high school-trained models on the MATH evaluation set (see Table 2). This raises questions about: (a) how the authors define OOD capabilities—whether based on the absence of similar problems in training or lack of shared underlying concepts; and (b) the effectiveness of the knowledge point labels used for segmenting middle and high school data.\n   3. Figure 3 indicates that ZHONGKAO includes more advanced (level 3) general knowledge questions than GAOKAO. Since both NuminaMath and Lila contains math problem-solving data, assuming the proprietary dataset also contains mostly Math problem solving data,  a more detailed analysis of why high school math-trained models perform significantly worse on the ZHONGKAO evaluation would be beneficial. \n   4. More granular difficulty-level analysis of the middle school and high school datasets, as applied in Section 5.3, would enhance clarity.\n\n\n2. Result 8 appears somewhat trivial and lacks novelty, given that prior research has shown LLMs—and even simpler neural networks—tend to learn simpler representations first.\n\n3. Results 4 and 6 appear redundant. Result 4 (line 377) highlights that SFT is less effective than CPT in learning mathematical skills, while Result 6 (lines 427-428) conveys a similar conclusion, stating that SFT's in-domain learning ability is weaker than CPT’s.\n\n4. It seems to be that the majority of the problem solving data seems to be proprietary, thus making it hard to reproduce.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1YOYE889us", "reviewer": "Reviewer_DwwE", "summary": "This paper attempts to advance the mathematical reasoning capabilities of LLMs through  problem-solving data and exploring different data synthesis methods during pre-training and post-training stages. The authors propose findings on the efficacy of using problem-solving data and propose a new JiuZhang-8B model that outperforms baseline models.", "questions": "I feel that a lot of my questions are already contained in the weaknesses above. Here are some more:\n- How do you ensure representative and high-quality data?\n- Can you provide more intuition or theoretical support for these empirical results?\n- Beyond releasing JiuZhang-8B, what are the practical implications of this research?\n- The paper makes significant claims about the advantages of CPT over SFT in developing reasoning capabilities. How do the authors justify these findings theoretically? Could these results be influenced by the specific model architecture or training configuration used?\n- Is there a detailed error analysis provided for cases where the model underperforms, especially on more complex mathematical problems?", "limitations": "", "rating": 1, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "Here are some strengths:\n- I appreciate the clear introduction and quickly getting to the point on issues with the current two-stage paradigm.\n- The authors include information for reproducibility, such as hyperparameters and data mixture ratios.\n- The authors develop and release a new model, JiuZhang-8B, which could add value to the community.", "weaknesses": "This paper needs some work. Here is a list of areas for improvement:\n- I recommend briefly describing in the abstract and introduction what you mean by problem-solving data.\n- The paper outlines three research questions but fails to present a clear rationale or context for each. Why are these research questions important to answer?\n- I find the bolding to be a bit hard to read in the introduction. I recommend laying out your contributions in a bulleted list to avoid having to bold “Result X”.\n- I recommend providing a brief description of the techniques mentioned in Result 3 or instead summarizing what types of techniques are the most effective rather than specifying specific names.\n- In the abstract, you say “improvements in mathematical reasoning achieved through continued pre-training (CPT) are often less significant compared to those obtained via SFT.” If this is true, then why do you say in result 4 that “While SFT can facilitate some learning of mathematical capabilities, it has a clear disadvantage compared to CPT”?\n- I think listing Results 1 to 8 is ineffective as it is easy for the reader to get lost and hard to understand the overall bigger picture. I recommend consolidation the results by topic to ensure clarity.\n- I recommend explaining why you chose the data mixture ratios that you did and moving the less important training details like batch size to the appendix.\n- Adding figures throughout would greatly enhance the argument. For example, add a main pipeline figure of the overarching framework you are implementing, and potentially new figures for sections 3, 4, and 5.\n- The methodology appears insufficiently rigorous, particularly in the way data distribution and difficulty levels are analyzed.\n- The experiments lack comprehensive justification, and the comparisons between different training setups (CPT vs. SFT) come across as arbitrary. What are the hypotheses leading to these decisions?\n- Throughout the paper, there are bold claims about the effectiveness of problem-solving data and specific synthesis methods like Tutorship Amplification. I recommend relying less on superficial observations and instead adding more deep analysis of why certain methods are effective.\n- I recommend adding some theoretical backing or intuition as to why these methods work.\n- The paper repeatedly discusses the distinction between CPT and SFT stages, yet the insights presented do not meaningfully advance the reader's understanding. I recommend reducing redundancy and focusing on meaningful results.\n- I am concerned about data quality. It seems there is heavy reliance on synthetic data and vague descriptions of the datasets. What is the data cleansing process?\n- The real-world applicability or significance of JiuZhang-8B is not discussed.\n\n\nMinor:\n- “We employed the MinHash deduplicationLee et al. (2022) framework to enhance training data quality by removing documents with significant duplicate content.” should have a space between “deduplication” and “Lee”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0BKScpM2gr", "reviewer": "Reviewer_awJ3", "summary": "The paper aims to understand how different aspects of training affect the ability of LLMs to do mathematical reasoning. This includes the tension between continuous pretraining and SFT, the use of problem solving data directly during CPT, the use of synthetic data generation and so forth. The authors provide a variety of fairly standard methods to test this, including some creative uses of synthetic data generation ( Query Expansion, Response Diversification, and Tutorship Amplification). They use LLAMA2 with the standard math evals, supplemented by a few that were introduced after the checkpoint in LLAMA2.  They apply their optimal methodologies to LLAMA3 to produce a new model JiuZhang-8B. JiuZhang-8B which has good performance especially relative to its size (Table 4)", "questions": "What is the major advance of this paper?\nWhat surprised you?\nWhat did you learn that might generalize out of this narrow problem domain?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors apply standard methods though in different variants than other work in this field. Their methods for synthetic data augmentation echo things I've seen before but not in this specific context.  They show some gains relative to larger models which is good.", "weaknesses": "This is solid work but does not feel like a major advance -- it is a set of good work with attention to evals and training procedures, but does not make any new conceptual advances.  IF ICLR were filled with papers like this one, the meeting wouldn't be intellectually exciting. This is very much \"in the weeds\" in engineering mixtures of LLM research", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pey8UVk21V", "reviewer": "Reviewer_hewE", "summary": "This work investigates strategies to improve mathematical reasoning in large language models (LLMs). It explores the effects of:\n\nProblem-solving Data: Focuses on using math-specific problem-solving data during the pre-training phase, rather than general mathematical text, to enhance reasoning skills.\n\nData Synthesis Methods: Evaluates four synthesis methods—response diversification, query expansion, retrospective enhancement, and tutorship amplification—for creating synthetic problem-solving data to overcome data scarcity.\n\nTraining Stages: Compares the impact of applying problem-solving data during the continued pre-training (CPT) phase versus the supervised fine-tuning (SFT) stage.\n\nResults suggest that problem-solving data enhances math reasoning more effectively than general data. The paper introduces JiuZhang-8B, a math-specific LLM trained with insights from these strategies, which reportedly achieves comparable or superior performance to existing math-focused models despite fewer training tokens.\n\nThere are several interesting takeaways. For example, even if one provides many hard problems in the training data, expect improvements primarily in the easy/medium regime. Overall, this work is a valuable contribution to the community.", "questions": "Can the proposed synthetic data methods, especially \"tutorship amplification\", be effectively scaled or adapted to non-math domains?\n\nHow well would the identified CPT benefits translate to domains requiring other forms of reasoning (e.g., logical or scientific)?\n\nGiven the focus on hard data and high-volume problem-solving datasets, are there any recommendations for making this approach feasible for smaller labs with limited resources?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This work tackles several research questions and addresses them clearly and effectively. \n\nSpecifically:\n\nThe study covers multiple aspects of training and data synthesis for LLMs in math reasoning, providing valuable insights into data handling (generation and sharding) strategies.\n\nThe analysis identifies tutorship amplification as particularly effective, offering practical value for generating synthetic data that improves reasoning abilities.\n\nJiuZhang-8B demonstrates that the proposed methods lead to significant performance improvements, providing a practical model output as a proof of concept.\n\nThis work contributes to understanding how data complexity influences learning outcomes by segmenting data into difficulty levels and testing different mixtures.", "weaknesses": "While this paper summarizes several empirical results that are useful to the community, it makes limited efforts to advance understanding of the mechanics behind these observations. Future work in this direction would be of great value.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L2nMMxZa3H", "reviewer": "Reviewer_3zrA", "summary": "This paper aims to improve mathematical reasoning capabilities in large language models, investigating whether problem-solving data can outperform general mathematical corpora in improving these capabilities during pre-training. The study explores three research questions: (1) effectiveness of problem-solving data, (2) data synthesis methods, and (3) comparison between pre-training (CPT) and supervised fine-tuning (SFT) stages. Results indicate that problem-solving data significantly improves reasoning performance, with the \"tutorship amplification\" data synthesis method proving particularly effective. The paper also introduces JiuZhang-8B, a model based on these findings which performs comparatively with other math LLMs.", "questions": "* Is it really clear why SFT underperforms continual pretraining based on the experiments you performed?\n* Does JiuZhang-8B learned math skills transfer to interdisciplinary tasks combining math with other fields?\n* Could tutorship amplification be effective in other problem-solving domains?\n* Is there any error analysis in JiuZhang-8B's problem-solving tasks?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper nicely states the research questions. Also arranging results along the corresponding research question smoothly helps with reading the paper.\n* The level of details in the dataset, data mixture, and experiments is very good.\n* The introduced model JiuZhang-8B achieves competitive performance with fewer tokens, highlighting efficient training and well curated training tokens.\n* The results are clear and I like the attempts to explain why behind the results, particularly the advantage of CPT over SFT for reasoning tasks.", "weaknesses": "* The paper shows promising results for mathematical reasoning but could benefit from testing on datasets that combine math with other fields, like physics. This would help assess if the model’s reasoning improvements generalize to interdisciplinary tasks requiring both math and domain knowledge, strengthening the case for broader applicability of the techniques.\n* More analysis and clarity on why SFT underperforms CPT would strengthen the paper. Is the difference due to data volume limitations, catastrophic forgetting, or other factors?\n* It would also be good to show some samples of the data, especially the generated or synthetic data. Also more details around data generation would be good like prompts or models used to generate data and if a human was used in the process for verification for example.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Je28YU2Jci", "reviewer": "Reviewer_v6B9", "summary": "This paper aims to study the mathematical abilities of LLMs especially the effects of the type of training data (problem solving data vs general math data), using synthetic data in conjunction with real data, the effect of SFT vs CPT on the mathematical abilities of an LLM. The paper proposes training insights for optimizing the mathematical abilities of LLMs. The paper also proposes many synthetic data creation techniques like query expansion, retrospective enhancements, tutorship amplification for creating mathematical data. Finally they train a strong math specific model based on the findings of the previous questions.", "questions": "- For response diversification, it could happen that the model could generate incorrect logic leading to the correct answer, do you have simple checks in place to ensure that this does not happen?\n- For results in table 2, how will the results change if you apply inference time techniques like Best-of-N sampling or using a reward models during inference? It could be that the conclusion changes significantly. One recent work (https://arxiv.org/pdf/2410.02725) showed that simply restarting the inference can change the accuracy on math tasks significantly.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written, and the overall direction of the paper is extremely important in the current times.\n- Usually the data mixture ratio is one of the key secret sauces to training strong models, and the experiments in the initial section of the paper are important.\n- The datasets used in the experiments are publicly available, and the experiments described in section 2 are extensive and cover the important cases.\n- Synthetic data generation techniques can actually improve the robustness of the LLMs with respect to modifications (reframing of questions, or symbolic modifications) which could break the mathematical abilities of LLMs.", "weaknesses": "- For figure 1, have you tried ablating the percentage of the math mixture to extremes like 2:8, 1:9 or even complete problem solving data. For the tasks considered in the paper, it could be the case that simply doing CPT with the problem solving data could be sufficient.\n- Overall I could not find much weakness in the paper, the experiments seem easy to understand and sound.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SkaZmhluwD", "reviewer": "Reviewer_XMnz", "summary": "The paper titled **\"Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages\"** explores the limitations of current pre-training strategies for mathematical reasoning in large language models (LLMs). It addresses three primary research questions related to improving the model’s mathematical reasoning capabilities: (1) Whether problem-solving data during the pre-training phase is more effective than general mathematical corpora, (2) The efficacy of synthetic data, and (3) How these strategies differ between continued pre-training (CPT) and supervised fine-tuning (SFT). The authors present a variety of data synthesis methods and analyze their impact on model performance. The results suggest that problem-solving data significantly enhances performance, particularly when coupled with effective data synthesis techniques like tutorship amplification, leading to the development of a competitive model, JiuZhang-8B.", "questions": "- Could the authors provide more specific examples of problem-solving data used during pre-training and demonstrate how it differs from general mathematical corpora?\n- Could the authors provide more insights into the limitations of the retrospective enhancement technique? What additional methods or adjustments could make this approach more effective?\n-  Have the authors considered applying the problem-solving data approach to other reasoning-heavy domains, such as physics or logic-based tasks? How would the methods transfer to these fields?\n- How do the proposed data synthesis methods, such as tutorship amplification, perform when applied to larger LLMs like 72B? Is the impact equally significant?\n- Given the success of the tutorship amplification method, do the authors plan to further refine this approach? Could it be applied in real-time tutoring systems or adaptive learning models for broader applications?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces new insights into the importance of problem-solving data and explores under-researched data synthesis methods like tutorship amplification. It delves into an interesting research question regarding the effect of data mix on performance. Additionally, the study examines the differing impacts of continual pretraining and supervised fine-tuning on final mathematical performance.\n\n  These findings are likely to be of  importance for further research in the fields of pretraining and supervised fine-tuning. The paper's exploration of these topics contributes valuable knowledge to the ongoing development of machine learning models and their application in problem-solving tasks.\n\n- The experiments are thorough, with detailed comparisons between different data types, synthesis methods, and training stages.  Although the technical depth is substantial, the paper is mostly well-written, with a clear exposition of the research questions and a logical progression through the results.\n\n- This work could have a  impact on the development of LLMs for complex reasoning tasks, especially in mathematics. The introduction of JiuZhang-8B as a model trained on fewer tokens is interesting.", "weaknesses": "- **Limited Practical Examples**: The paper could be enhanced by including more detailed, practical examples of how problem-solving data improves model performance. Concrete examples from the dataset would clarify the real-world impact of the proposed techniques.\n\n- **Underexplored Larger Model Performance**:  The paper's experimental focus on mid-sized models like JiuZhang-8B, while informative, leaves a significant gap in understanding how the proposed methods perform and scale with larger language models. This limitation potentially understates the full impact and scalability of the presented approach.\n\n  Expanding the analysis to include larger models in the 13B, 32B, or even 70B parameter range would provide crucial insights into the scalability and effectiveness of the proposed techniques. Such an expansion could reveal whether the benefits observed in mid-sized models persist, amplify, or perhaps even diminish when applied to more powerful language models. \n\n  Moreover, the inclusion of larger models in the study might necessitate an increase in the volume of training data. This presents an opportunity to further explore the interplay between data quality and quantity in the context of more capacious models. It could potentially reveal new insights into the optimal balance between these factors for different model sizes, providing valuable guidance for future research。\n\n  \n\n- **Generalization to Other Domains**: The focus is heavily on mathematical reasoning, but there is little discussion on how the proposed techniques might generalize to other domains that also require reasoning.\n\n- **Data Synthesis Techniques**: Although the tutorship amplification method was shown to be most effective, the explanations for why certain methods (e.g., retrospective enhancement) performed poorly could be expanded. Further exploration of why specific synthesis methods yielded limited results would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)