Skywork-Math: Data Scaling Laws for Mathematical Reasoning in LLMs — The Story Goes On

Authors: liang  zeng
skywork  ai
liangjun  zhong
Venue: NeurIPS_cc_2024_Workshop_MATH-AI
Published: 2024-09-19 16:21:29

Abstract: In this paper, we investigate the underlying factors that potentially enhance the mathematical reasoning capabilities of large language models (LLMs). We argue that the data scaling law for math reasoning capabilities in modern LLMs is far from being saturated, highlighting how the model's quality improves with increases in data quantity. To support this claim, we introduce the Skywork-Math model series, supervised fine-tuned (SFT) on common 7B LLMs using our proposed 2.5M-instance Skywork-MathQA dataset. Skywork-Math 7B has achieved impressive accuracies of 51.2\% on the competition-level MATH benchmark and 83.9\% on the GSM8K benchmark using only SFT data, outperforming an early version of GPT-4 on MATH. The superior performance of Skywork-Math models contributes to our novel two-stage data synthesis and model SFT pipelines, which include three different augmentation methods and a diverse seed problem set, ensuring both the quantity and quality of Skywork-MathQA dataset across varying difficulty levels. Most importantly, we provide several practical takeaways to enhance math reasoning abilities in LLMs for both research and industry applications.

Main paper: 
 introduction: 
 More is different.--Philip W. Anderson, 1972 Reasoning ability is a hallmark of human intelligence [14, 11, 27] . Although Large Language Models (LLMs) have recently demonstrated significant capabilities in various tasks such as conversation [1, 3, 18] and summarization [28, 30, 21, 2] , they often struggle with complex reasoning tasks [11, 17, 29] . One particularly challenging area is mathematical reasoning [13, 9, 32, 4, 12] , which requires the ability to solve mathematical problems and derive logical conclusions in a step by step manner [27, 20, 23, 31, 24] .Two prevailing beliefs guide researchers and practitioners in enhancing mathematical reasoning abilities of LLMs. The first belief posits that complex reasoning abilities, especially mathematical reasoning, are emergent abilities that exist in large language models but not in small models [27, 26] . Typically, models with more than 30 billion parameters exhibit the strong mathematical reasoning ability [7] . The second belief is the seminal "superficial alignment" hypothesis [33] , which asserts that "A model's knowledge and capabilities are learnt almost entirely during pre-training, while alignment teaches it which sub-distribution of formats should be used when interacting with users.". According to this hypothesis, the alignment process, primarily through supervised fine-tuning (SFT), does not inject new knowledge or improve inherent abilities but rather adjusts the output response format. This implies that the strong mathematical reasoning ability may not be significantly improved by a large amount of synthetic SFT data. GPT-4-0314MetaMath-7B MetaMath-13B WizardMath-70B InternLM2-Math-7B LEMA-LLaMA2-7B LEMA-LLaMA2-13B MetaMath-Mistral-7B MAmmoTH-7B MAmmoTH-13B MAmmoTH-70B WizardMath-7B WizardMath-13B MetaMath-70B LLaMA3-8B GPT-3.5-Turbo LEMA-LLaMA2-70B DeepSeekMath-Instruct-7B InternLM2-Math-20B Xwin-Math-13B Xwin-Math-Mistral-7B Skywork-Math-LLaMA2-7B Skywork-Math-DeepSeekMath-7B Skywork-Math-Mistral-7B ChatGLM3-Math-SFT-32B/Xwin-Math-7BTop@1 Accuracy (%)Figure 1 : Top1 accuracy on GSM8K [9] and MATH [13] using only SFT techniques, without using external toolkits and voting techniques. Following MetaMath [31] , we employ a zero-shot chainof-thought evaluation framework. Skywork-Math models achieve state-of-the-art accuracy among models smaller than 10B parameters using only synthetic SFT data and surpass an early version of GPT-4 on MATH.In this paper, we re-examine these two common beliefs mentioned above regarding mathematical reasoning abilities of LLMs. For the first belief, we introduce the Skywork-Math model series, which are supervised fine-tuned (SFT) on common 7B pre-trained LLM models without employing other complex alignment techniques such as RLHF [6, 8] and DPO [19] . Skywork-Math 7B models have achieved impressive accuracies of 51.2% on the competition-level MATH [13] benchmark and 83.9% on the GSM8K [9] benchmark, notably outperforming an early version of GPT-4 on MATH.Our empirical findings, consistent with the conclusions in [16] , suggest that strong mathematical reasoning ability can indeed exist in common 7B language models. Moreover, scaling up synthetic SFT data can further enhance the mathematical reasoning ability of Skywork-Math 7B models.For the second belief, we propose Skywork-MathQA high-quality SFT dataset containing 2.5 million instances, which is much larger than open-sourced dataset of its kind to date, such as Meta-MathQA [31] containing 395K samples. We empirically observe that the scaling law curve on the SFT alignment for mathematical reasoning in modern LLMs is far from being saturated (ref.Figure 3 ). We have carefully scaled the Skywork-MathQA SFT dataset with diverse and high-quality samples specifically within the mathematical domain to enhance the model's capability in understanding and solving mathematical problems.Due to the scarcity of high-quality and challenging mathematical data, various pipelines and prompts have been employed to generate synthetic mathematical data [31, 23, 16, 24, 27, 25] . To address this deficiency, we employ GPT-4 to generate a substantial amount of synthetic data through a novel two-stage data synthesis pipeline, in conjunction with the corresponding model SFT process. In stage 1, our objective is to obtain normal synthetic problems to enhance the models' general comprehension of mathematical problems. To maintain the diversity in data selection process, we utilize the core-set approach [22] on enlarged seed problems. However, as the data volume increases, we empirically Highlighted Takeaways• The potential for math reasoning capabilities in modern LLMs is far from exhausted. The quality of LLMs can significantly improve with increases in data quantity (ref.Figure 3 ). Skywork-Math 7B models already demonstrate strong mathematical reasoning abilities by SFTing on common 7B pre-trained LLM models. • The learning process for accessing the math reasoning ability involves multiple stages.Training LLM models in a meaningful order, from the easy problems to the hard ones, can provide performance improvements. • When scaling the synthetic SFT dataset, increasing the diversity of seed problems and augmentation methods can improve the math reasoning performance of LLMs. • Selecting influential data with high-quality from a large dataset is non-trivial [10] . Our empirical results indicate that some straightforward methods to select the so-called "highquality" data may not increase (and can even hurt) LLMs' performance compared to randomly selecting data. The selection process involves multiple constraints, and the "highquality" data could significantly decrease the difficulty level of problems, thus negatively impacting the performance of LLMs. • The LLM models have strong knowledge transfer capabilities for mathematical reasoning across bilingual benchmarks (i.e., English and Chinese). We hypothesize that this can be attributed to the inherent nature of symbols and numbers in math problems, which retain their intrinsic meaning regardless of the language used. • Although Skywork-Math 7B models have achieved considerable improvement in robustness tests compared to other open-source LLM models, they remain sensitive to the distractors in math word problems compared with proprietary GPT-4 models. • Sparse MOE models cannot clearly exceed the performance upper bound of their dense counterparts through SFT alignment in the context of math reasoning. • Two subtle but crucial practical implementation techniques-preventing data leakage and considering the influence of model maximum length-significantly impact the final performance of LLM models.observe that the relationship between performance and data quantity begins to plateau. Accordingly, in stage 2, we diversify the dataset further by introducing a proportion of augmented hard problems, thereby exposing the model to more challenging mathematical questions. Without continual pretraining on a large-scale math corpus [23, 5] , Skywork-Math models achieve impressive performance with just supervised fine-tuning on common pre-trained LLMs containing only 7B parameters.Most importantly, we provide valuable insights and practical takeaways to enhance the mathematical reasoning ability in LLMs, benefiting both research and industry communities 2. 
method: 
 In this section, we present the methodology of Skywork-Math 7B models, as illustrated in Figure 2 . Skywork-Math models aim to enhance math reasoning abilities during the model alignment process, particularly in the SFT stage, using common and publicly available 7B pre-trained models. We employ a two-stage SFT approach, in conjunction with two data synthesis pipelines to produce high-quality data. In stage 1, we feed base pre-trained models with our generated normal synthetic problems (2.1M instances) to produce an intermediate model. In stage 2, to mitigate the diminishing returns in LLMs' performance as the quantity of data increases, we generate hard synthetic problems (0.4M instances) and develop our Skywork-Math models. To ensure the quality of data, we primarily utilize GPT-4-1106-preview [1] to generate 2.5M-instance synthetic Skywork-MathQA dataset. Due to space constraints, detailed methods and experimental results can be found in the appendix.  
data scaling laws in sft on mathematical reasoning: 
 In Figure 3 , we illustrate the relationship between synthetic SFT dataset size and model performance on GSM8K and MATH. The curve clearly exhibits a scaling law relationship between the size of SFT data and model's performance. Here are some in-depth observations:Quantity Breeds Quality. To enhance the mathematical reasoning abilities in LLMs, increasing the quantity of synthetic data can significantly improve the quality of model performance. This scaling trend implies that, while SFT with a small amount of data could achieve decent results [33] , utilizing a larger scale of synthetic SFT data can further improve math reasoning performance.Diminishing Returns from Continual Pre-Training. The DeepSeekMath-Base [23] 7B model, which has been continually pre-trained with 120B math-related tokens sourced from the web, initially demonstrates superior performance. However, as we increase the synthetic dataset size in the Skywork-MathQA dataset, this advantage diminishes and is eventually surpassed by the Mistral [15] 7B base model. As the amount of SFT data increases, Skywork-Math-Mistral-7B and Skywork-Math-LLaMA2-7B catch up in performance to the Skywork-Math-DeepSeekMath-7B. This suggests that while specialized pre-training provides a strong initial boost, its benefits are not consistently scalable and can be matched by increasing the quantity of synthetic SFT data.Effect of Problem Difficulty. The accuracy performance for Skywork-Math 7B model series significantly increases as the synthetic data size expands from 2.1M to 2.5M, corresponding to the stage 2 in our data synthesis pipeline. This performance improvement in the final stage of data scaling indicates that incorporating more complex problems-ranging from Level 3 to Level 5 in the MATH dataset-has a substantial positive impact on model performance. This finding underscores the importance of not only generating a large quantity of data but also including more challenging problems to push the limits of math reasoning abilities of LLM models.  
conclusion: 
 We study how to empower mathematical reasoning abilities for common 7B pre-trained LLM models. We propose the Skywork-MathQA dataset, consisting of 2.5 million diverse and high-quality SFT instances, implemented through our novel two-stage data synthesis pipeline. We introduce Skywork-Math model series, demonstrating that common small-scale 7B language models can stimulate strong mathematical reasoning ability using only synthetic SFT data. Skywork-Math models achieve state-of-the-art accuracy among models smaller than 10B parameters using only synthetic SFT data, surpassing 70B LLM models and an early version of GPT-4 on MATH. These results suggest that the data scaling law for mathematical reasoning in LLM models remains significant and promising. Notably, this research provides several valuable insights and practical takeaways to advance our understanding of the capabilities and limitations of LLMs in mathematical reasoning. 
 
 Appendix: 38th Conference on Neural Information Processing Systems (NeurIPS 2024). 



Reviews: review: title: title: Skywork-Math Review
main_review: main_review: Summary
* This work introduces a 2.5M instance Skywork-MathQA dataset, which is generated by a novel data pipeline for generating quality instances for SFT
* They show that models fine tuned using the synthetic data can achieve SOTA accuracy even on models smaller than 10B parameters and that SFT with high quality data helps models build reasoning capabilities

Strengths
* Skywork-Math models are able to achieve state-of-the-art accuracy among models smaller than 10B parameters on the GSM8K and MATH datasets
* Scaling law findings are valuable, especially that there are diminishing returns to pre-training and that varying problem difficulty is important

Weaknesses
* Since the description of methods is so brief, the methods and the main contributions in the methodology are not very clear from the main body of the paper. It would help to move some of the Appendix B content into the main body.
* Missing analysis of the limitations/tradeoffs of the proposed two-stage method. Is there a ceiling where increasing the quantity of synthetic SFT data stops helping?
title: title: Outstanding in all aspects; therefore, accepted.
main_review: main_review: The paper explores the potential of LLMs to solve mathematical problems from a data-centric perspective, finding that scaling laws remain applicable.

Strengths:
1. The article is well-structured, well-written, and easy to understand.
2. The volume of experiments is substantial, with thorough experimental analysis, exceeding typical workshop standards.

Weaknesses: Some conclusions, such as the discussion on data quality and the impact of data scale on LLMs solving mathematical problems, lack novelty.
title: title: The paper seems good, but the first 4 pages don't have enough information
main_review: main_review: The paper has a total of 35(!) pages with the first 4 having very little information except high level conclusions. I didn't read the full 35 pages, so my review is based mostly on the first 4 plus a couple of figures and tables I selectively chose from the appendix.

The only figure related to the presented highlights in the first 4 pages is Figure 3. I think it's an interesting and practically important observation that continued training does not necessarily have much impact after SFT. I wonder if the hyperparameters for those separate models have been tuned separately or if the hyperparameters (most importantly LR) for Mistral/Llama have been used for DeepSeek directly. This might invalidate conclusions as models after continued training might need very different HPs for best performance. 

The only other ablation mentioned in the first 4 pages is on the problem difficulty level, which is an interesting result. Although the numbers presented in Table 2 (in the appendix) are a bit inconclusive if we look at all 3 models. E.g. Mistral-7b seems to get an impressive 4 points boost in level 5 problems, while llama and deepseek slightly decrease. Also to make a fair comparison that indeed hard problems are the key, it's important to compare not with the baseline of 2.1M samples, but with 2.5M where we add random or easy problems (since maybe it's just the extra 0.4M of data that lead to this accuracy increase). Maybe this comparison is presented somewhere in the appendix, but I didn't read all 35 pages.

Overall, I think it's a good paper, but the first 4 pages have so little information that it's hard to judge that for sure.
title: title: Official Review
main_review: main_review: ## Paper Summary 
The paper investigates how mathematical reasoning in LLMs can be improved by scaling the quantity of training data. It introduces the Skywork-Math model series, fine-tuned on a synthetic dataset, which demonstrates superior performance on mathematical benchmarks and surpasses early versions of GPT-4.
## Pros
* **Novel Data Scaling Approach:** Overall, the paper has a high novelty in its proposed method. It introduces the Skywork-Math model series, which adopts a two-stage SFT approach with data synthesis pipelines to effectively demonstrate how scaling SFT data can enhance mathematical reasoning in LLMs. This focus on data scaling laws offers valuable insights for future research and industry applications.

* **Scalable and Efficient Use of Small Models:** A highlight in the paper is that the Skywork-Math models, with only 7B parameters, achieve results on par with larger models (up to 70B parameters) by leveraging fine-tuning and data scaling.  The results demonstrate that the method can maintain high accuracy and is resource-efficient.

* **Impressive Results on Benchmarks:** The Skywork-Math models outperform several state-of-the-art models, including an early version of GPT-4, on the MATH benchmark. Achieving accuracies of 51.2% on the MATH benchmark and 83.9% on GSM8K using only SFT data is a strong testament to the effectiveness of the method.

## Cons 
* **Additional Benchmarks for broaden applicability:** This paper can be further improved by testing the method on additional math benchmarks to evaluate the proposed method's reasoning ability such as logical reasoning.