Common 7B Language Models Already Possess Strong Math Capabilities

## Abstract

Abstract Mathematical capabilities were previously believed to emerge in common language models only at a very large scale or require extensive math-related pre-training. This paper shows that the LLaMA-2 7B model with common pre-training already exhibits strong mathematical abilities, as evidenced by its impressive accuracy of 97.7% and 72.0% on the GSM8K and MATH benchmarks, respectively, when selecting the best response from 256 random generations. The primary issue with the current base model is the difficulty in consistently eliciting its inherent mathematical capabilities. Notably, the accuracy for the first answer drops to 49.5% and 7.9% on the GSM8K and MATH benchmarks, respectively. We find that simply scaling up the SFT data can significantly enhance the reliability of generating correct answers. However, the potential for extensive scaling is constrained by the scarcity of publicly available math questions. To overcome this limitation, we employ synthetic data, which proves to be nearly as effective as real data and shows no clear saturation when scaled up to approximately one million samples. This straightforward approach achieves an accuracy of 82.6% on GSM8K and 40.6% on MATH using LLaMA-2 7B models, surpassing previous models by 14.2% and 20.8%, respectively. We also provide insights into scaling behaviors across different reasoning complexities and error types.

## 1 Introduction

Mathematical capabilities have long been considered so challenging that they are thought to emerge in common language models only at a very large scale. For instance, studies by Wei et al. (, ) suggest that only models with size exceeding 50 billion parameters can attain meaningful accuracy or benefit from chain-of-thought processing on math problems. A strategy to equip smaller language models with mathematical abilities involves creating math-specific base models trained on hundreds of billions of math-related pre-training data Lewkowycz et al. (); Azerbayev et al. (). However, the accuracy of such models remains modest; for example, Llemma-7B Azerbayev et al. () only achieves 36.4% on the GSM8K dataset Cobbe et al. () and 18.0% on the MATH dataset Hendrycks et al. ().

Figure: Figure 1: The orange star markers represent the accuracy achieved by selecting the best response from 256 random generations of the LLaMA-2 7B model. The high accuracy on the MATH (left) and GSM8K (right) benchmarks (72.0% and 97.7%, respectively) suggest that the LLaMA-2 7B already possesses strong mathematical capabilities, although the stability in generating correct answers could be enhanced. This paper demonstrates that by scaling synthetic SFT data, the stability can be significantly improved as evidenced by the curves. Through this straightforward scaling of SFT data, the top-performing model has exceeded an early GPT-4 model by 10.3% on the MATH benchmark.
Refer to caption: extracted/5422178/figures/teaser.png

**Table 1: Comparison of SFT data scaling with real versus synthetic math questions. It reveals that synthetic math questions are nearly as effective as real ones.**
| Data size | GSM8K-real | GSM8K-syn | MATH-real | MATH-syn |
| --- | --- | --- | --- | --- |
| 0.94K | 26.7 | 25.9 | 4.2 | 3.9 |
| 1.88K | 32.8 | 31.9 | 5.6 | 4.9 |
| 3.75K | 43.3 | 42.2 | 6.6 | 6.0 |
| 7.50K | 50.2 | 49.5 | 8.4 | 7.9 |

In this paper, we demonstrate that common language models of small size, such as the LLaMA-2 7B model Touvron et al. (), already possess strong mathematical capabilities without specific pre-training on math-related data. Surprisingly, we find that with supervised fine-tuning on just thousands of math questions (noting that the SFT stage does not enhance capabilities as stated in Bai et al. (); Ouyang et al. ()), the model can correctly solve 97.7% of GSM8K questions and 72.0% of MATH questions, when selecting the best answer from 256 random generations, as indicated by the orange star marks in Figure 1. It is noteworthy that the accuracy has even outperformed those reported for the GPT-4 model, which achieved 92.0% on GSM8K and 42.5% on MATH (^1^11The accuracy numbers are reported in the GPT-4 technical report OpenAI (). GPT-4 models are continuously being improved. The latest GPT-4 Turbo (1106) API has increased accuracy to 94.8% on GSM8K and 64.5% on MATH. However, the LLaMA-2 7B model using the best of 256 generations still outperforms the latest GPT-4 models.). Therefore, we conclude that the LLaMA-2 7B model has indeed developed strong mathematical capabilities. The primary issue is the lack of guarantee that the correct answer will be digged out, as most generations are incorrect. In fact, the accuracy drops to 49.5% on GSM8K and 7.9% on MATH if we consider only one random generation per question. We refer to this as the *instability issue*.

To address the instability issue, we first observe that the accuracy improves almost in linear or even super-linear with exponentially increased supervised fine-tuning (SFT) data. Moreover, we note that the accuracy is far from reaching a plateau when utilizing all available GSM8K and MATH training data (as shown in Table  1). This observation encourages us to further scale up the SFT data. However, we face a challenge as there is a lack of publicly accessible real data to support this continuous scaling.

To overcome this limitation, we turn to synthetic data, employing a prestigious language model, namely GPT-4 Turbo, to produce synthetic math questions. We find that a straightforward “brand-new” generation strategy, which prompts the GPT-4 Turbo to create a completely new question based on preference ones and then applies a simple verifier (also GPT-4 Turbo based), has been highly effective. Specifically, as indicated in Table  1, the use of synthetically generated math questions can achieve accuracy nearly on par with that of real questions, highlighting the potential of synthetic SFT math questions for the scaling purpose.

Leveraging synthetic data has allowed us to scale our SFT data significantly, for instance, from 7.5K to 960K on GSM8K and from 7.5K to 480K on MATH. This data scaling shows nearly perfect scaling behavior, as drawn in Figure 1. Specifically, by simply scaling the SFT data, our model has become the first to exceed 80% and 40% accuracy on GSM8K and MATH, respectively, using a standard LLaMA-2 7B base model (achieving 82.6% and 40.6% respectively)(^2^22Concurrently, DeepSeek-MATH-7B Shao et al. () also surpasses 80% accuracy. However, their approach relies on a much stronger base model extensively pre-trained on math-related corpora and a sophisticated RL algorithm. Our results are complementary to theirs.).

The straightforward synthetic SFT data proves effective from stronger base models as well, such as LLaMA-2 70B, which achieves 90.6% on GSM8K and 52.8% on MATH. To the best of our knowledge, this is the first open-source model to exceed 90% accuracy on GSM8K. It is also the first open-source model to outperform GPT-4 (i.e., GPT-4-0314) on the MATH benchmark, demonstrating the efficacy of our simple synthetic scaling method.

In addition to the strong results, we have also gleaned insights into the effectiveness of our approach: 1) As the scale of SFT data increases, the model’s accuracy tends to plateau when utilizing 256 attempts; however, there is a marked increase using 1 response. This indicates that while the model’s upper capability limit remains fairly constant, the performance gains are primarily due to enhanced stability in generating correct answers. 2) The accuracy of solving math problems follows a power law with respect to the number of chain-of-thought (CoT) steps with different SFT data quantities. An expanded SFT dataset improves the reliability of each reasoning step. Further increasing the proportion of training samples with longer CoT steps through resampling can significantly improve the accuracy of the model for difficult questions. 3) An analysis of error types during the scaling process reveals that calculation errors are more readily mitigated compared to reasoning errors.

## 2 Examine Math Capability of Language Models

#### Metrics

We employ two metrics to examine the math capabilities of language models.

The first is a Pass@N metric

$$ $\text{Pass@N}=\mathop{\mathbb{E}}_{\text{Problems}}\left[\min(c,1)\right],$ (1) $$

where $c$ represents the number of correct answers out of $N$ responses. This metric considers a question to be solved if at least one correct answer is produced from $N$ random generations. We employ this metric to reflect the potential or capability of a model in solving a math question. To enhance the diversity of the $N$ generations, we set the temperature of the generation process to 0.7(^3^33It is worth noting that most math models utilize a greedy generation strategy with the temperature set to 0. However, the impact of this difference is minimal.).

The second is a PassRatio@N metric

$$ $\text{PassRatio@N}=\mathop{\mathbb{E}}_{\text{Problems}}\left[\frac{c}{N} \right],$ (2) $$

which measures the percentage of correct answers within the $N$ generated answers. This metric is somewhat equivalent to Pass@1, but with reduced variance.

#### Observations

Based on these two metrics, we examine the performance of the LLaMA-2 models on the GSM8K and the MATH benchmarks(^4^44Following Lightman et al. (), we utilize a subset of 500 test samples from the MATH benchmark for experimental efficiency.) as shown in Figure 1. To adapt models for these two benchmarks in instruction-following settings, we use their SFT versions, which are trained with a limited amount of SFT data (i.e., 7.5K). As demonstrated in Bai et al. (); Ouyang et al. (), the SFT stage does not enhance capabilities (and may even lead to a reduction, as mentioned in the context of “alignment taxes”). Therefore, employing the SFT version provides a fair assessment of the models’ mathematical capabilities.

We first observe that the Pass@256 metrics for the LLaMA-2 7B model on both benchmarks are remarkably high: 97.7% on GSM8K and 72.0% on MATH. This suggests that the LLaMA-2 7B model possesses a strong capability for solving mathematical problems.

We then notice that the PassRatio@256 is significantly lower than that of Pass@256, being 48.2% on GSM8K and 7.9% on MATH. This suggests that while the correct answers to most math questions are present within 256 random generations, there is no assurance that the correct answers will consistently be extracted, a phenomenon we refer to as an "instability issue".

In the following, we will present a simple approach to significantly reduce the *instability issue*.

## 3 Scaling SFT Data using Synthetic Math Questions

In this section, we first demonstrate that scaling up the limited real SFT data can significantly alleviate the instability issue. We also observe that the accuracy has not yet plateaued when using the full available GSM8K and MATH training data. We consider further scaling up SFT data using synthetic math questions. To this aim, we introduce a straight-forward method for synthetic data generation utilizing the GPT-4 Turbo API. The synthetic data proves to be as effective as real math questions. Consequently, we boldly scale the synthetic SFT data to 960K on GSM8K and 480K on MATH, respectively, resulting in nearly perfect scaling behavior, and reach state-of-the-art accuracy.

#### Scaling using Real Math Questions

We begin by examining the scaling behavior of real math questions across the entire GSM8K and MATH training sets.
As indicated in Table  1, we observe a consistent accuracy improvement, increasing from 26.7% to 50.2% on GSM8K, and from 4.2% to 8.4% on MATH, with no signs of saturation.

#### Synthetic SFT Data Generation

Since the real data has been exhausted, we contemplate further scaling up SFT data using synthetically generated math questions.

We introduce a straightforward three-step approach with the assistance of the GPT-4 Turbo API:

- •
*Step 1. Generate a new math question.*
We request the GPT-4 Turbo API to generate a brand-new question using a reference math question as a starting point. To improve the validity of the new questions, we incorporate three rules into the prompt: Firstly, the new question must obey common knowledge; secondly, it should be solvable independently of the original question; and thirdly, it must not include any answer responses. Besides, we have set specific formatting requirements for questions and answers tailored to various target datasets.
- •
*Step 2. Verify the question.* We further enhance the quality of the generated questions by validating and refining them through attempted solutions. By integrating solving and verification steps into a single prompt, we have found that this approach consistently elevates the validity of questions across different benchmarks.
- •
*Step 3. Generate chain-of-thought (CoT) answers.* We request GPT-4 Turbo to produce a chain-of-thought (CoT) answer response for each newly generated question.

The detailed prompt designs are shown in Appendix A.

#### Comparison of Synthetic SFT Data versus Real Data

To assess the quality of the synthetically generated math questions, we evaluate their effectiveness against real questions from the GSM8K and MATH training sets, utilizing a LLaMA-2 7B model, as detailed in Table  1. The results indicate that the synthetic math questions are nearly as effective as the real ones.

We also explored various other synthetic methods as proposed in previous works Xu et al. (); Yu et al. (); An et al. (). These methods also prove to be effective, though marginally less so than the our approach, as illustrated in Figure 6.

#### Scaling to about a Million SFT Math Data

Considering the effectiveness of the synthetic approach, we substantially increase the scale of the SFT data for both GSM8K and MATH problems, to 960K and 480K, respectively. Figure 1 presents the main reasults utilizing various sizes of the LLaMA-2 series. The straightforward scaling strategy yields state-of-the-art accuracy.

It is also worth noting that the accuracy has not yet reached its peak. Exploring the effects of additional scaling will be left as our future research.

## 4 Experiments

### 4.1 Datasets and Evaluations

We conduct experiments on 5 benchmarks to evaluate the efficacy of the proposed method.

GSM8K Cobbe et al. (). This is a high-quality, linguistically diverse math dataset, whose math knowledge mainly covers grade school level. It includes 7,473 training examples and 1,319 test cases. In this work, we use its training set as the given questions to generate new synthetic data.

MATH Hendrycks et al. ().
This dataset focuses on competitive-level math problems that requires high levels of reasoning ability and mathematical knowledge. It consists of 7,500 training examples and 5,000 test cases. We use the training examples to generate synthetic data.

SVAMP Patel et al. ().
This dataset comprises elementary-level math problems. We utilize all 1,000 of its test cases to assess the cross-dataset performance of our models.

ASDiv Miao et al. (). This dataset contains a set of math problems with diverse language patterns and types of questions. We adopt the test set of 2,305 problems as evaluation benchmark.

Hungarian National High School Exam This evaluation benchmark is first introduced by Grok-1 xAI (), which is designed for evaluating the out-of-domain capability of math models. It consists of 33 challenging problems.

It is worth noting that the final answers of Hungarian National High School Exam dataset is annotated by human, while other benchmarks are labelled using automatic scripts, similar to previous works
Luo et al. (); Gou et al. ().

### 4.2 Implementation Details

In data synthesis, we utilize the GPT-4 Turbo API, setting the temperature to 1.0 for both question and answer generation.

For supervised fine-tuning, we employ the Adam optimizer with a cosine learning rate schedule spanning a total of 3 epochs of training. The maximum learning rate is set 2e-5 (except that 2e-6 for the Mistral-7b model) and there is a 4% linear warm-up. The maximum token length is set 2048, and the Vicuna-v1.1 Zheng et al. () system prompt is used. All experiments are conducted on 8$\times$Nvidia H100 GPUs. Our most resource-intensive experiment, involving a 70B model and 960K data points, takes 1900 H100 GPU hours.

For evaluation, we use the same prompt as used in SFT and set the maximum sequence length to 2048. The vLLM Kwon et al. () is used in answer generation.

**Table 2: Math reasoning performances of various LLMs.**
| Model | GSM8K | MATH |
| --- | --- | --- |
| Closed-source models |  |  |
| GPT-4 Turbo (1106) | 94.8 | 64.5 |
| GPT-4-0314 | 94.7 | 52.6 |
| GPT-4 Achiam et al. () | 92.0 | 42.5 |
| Claude-2 Anthropic () | 88.0 | - |
| GPT-3.5-Turbo OpenAI () | 80.8 | 34.1 |
| Open-source models LLaMA-2-7B |  |  |
| WizardMath-7B Luo et al. () | 54.9 | 10.7 |
| MuggleMath-7B Li et al. () | 68.4 | - |
| MetaMath-7B Yu et al. () | 66.5 | 19.8 |
| LEMA-LLaMA-2-7B An et al. () | 54.1 | 9.4 |
| Xwin-Math-7B (ours) | 82.6 | 40.6 |
| Open-source models Mistral-7B |  |  |
| WizardMath-7B-v1.1 Luo et al. () | 83.2 | 33.0 |
| MetaMath-Mistral-7B Yu et al. () | 77.4 | 28.2 |
| Xwin-Math-Mistral-7B (ours) | 89.2 | 43.7 |
| Open-source models Llemma-7B |  |  |
| MetaMath-Llemma-7B Yu et al. () | 69.2 | 30.0 |
| Xwin-Math-Llemma-7B (ours) | 84.2 | 47.2 |
| Open-source models LLaMA-2-13B |  |  |
| WizardMath-13B Luo et al. () | 63.9 | 14.0 |
| MuggleMath-13B Li et al. () | 74.0 | - |
| MetaMath-13B Yu et al. () | 72.3 | 22.4 |
| LEMA-LLaMA-2-13B An et al. () | 65.7 | 12.6 |
| Xwin-Math-13B (ours) | 88.1 | 44.9 |
| Open-source models LLaMA-2-70B |  |  |
| WizardMath-70B Luo et al. () | 81.6 | 22.7 |
| MuggleMath-70B Li et al. () | 82.3 | - |
| MetaMath-70B Yu et al. () | 82.3 | 26.6 |
| LEMA-LLaMA-2-70B An et al. () | 83.5 | 25.0 |
| Xwin-Math-70B (ours) | 90.6 | 52.8 |

### 4.3 Main Results and Comparison with State-of-the-art Models

In this comparison, we examine both in-domain benchmarks, GSM8K/MATH, and out-of-domain benchmarks, such as the Hungarian National High School Exam. For in-domain evaluation of each benchmark, we utilize data synthesized from its respective training samples. For GSM8K, 960K synthetic data is employed, while for MATH, 480K synthetic data is used. For out-domain evaluation, we test models trained using GSM8K, MATH, or a mixed of two synthetic sets.

For base models, we consider both common language models, i.e., LLaMA-2 7B/13B/70B/Mistral-7B, and math-specific models, such as Llemma-7B, to assess the generality of the proposed approach.

#### In-Domain Results

Table  2 presents a comparison of the proposed approach with the state-of-the-art open and closed-source models. Across all base models, our method significantly outperforms the previous best approaches that use the same pre-trained base model.

On LLaMA-2-7B, our approach exceeds the prior best by absolutely +14.2 on GSM8K (compared to MuggleMath-7B Li et al. ()), and by +20.8 on MATH (compared to MetaMath-7B Yu et al. ()), respectively. It even surpasses several latest 70B models dedicated for math capabilities, such as WizardMath-70B (Luo et al., ) (82.6 versus 81.6 on GSM8K). On LLaMA-2-13B, the improvements are +14.1 on GSM8K (compared to MuggleMath-13B Li et al. ()) and +22.5 on MATH (compared to MetaMath-13B Yu et al. ()), respectively. On LLaMA-2-70B, the gains are +7.1 on GSM8K (compared to LEMA-LLaMA-2-70B An et al. ()) and +26.2 on MATH (compared to MetaMath-70B Yu et al. ()), respectively.

On a stronger common language model, i.e., Mistral-7B, the improvements are +6.0 on GSM8K and +10.7 on MATH (compared to WizardMath-7B-v1.1 Luo et al. ()), respectively.

On a math-specific base model, such as Llemma-7B, the gains are +15.0 on GSM8K and +17.2 on MATH (compared to MetaMath-Llemma-7B Luo et al. ()), respectively.

It is also noteworthy that our LLaMA-2-70B model achieves competitive accuracy with early versions of GPT-4 on GSM8K and MATH. To our knowledge, this is the first LLaMA-based model to outperform GPT-4-0314 on MATH.

These results demonstrate the significant effectiveness and broad applicability of scaling synthetic math SFT data.

#### Out-of-Domain Results

We test the models trained using GSM8K, MATH, or a mixed of two synthetic sets on an out-of-domain benchmark, Hungarian National High-School Exam Test, following the practice in xAI ().

Table  3 shows the results. Our model trained on the mixing data (240K MATH synthetic data + 240K GSM8K synthetic data) ranked as the second, just behind the GPT-4 and much better than other models. Additionally, we plot the correlation between GSM8K and Hungarian national high-school exam scores in Appendix B. The results show that there is no significant benchmark overfitting in our model.

**Table 3: Hungarian national high school exam test result of various LLMs.**
| Model | Test Score (%) |
| --- | --- |
| GPT-4 Achiam et al. () | 68 |
| Grok-1 xAI () | 59 |
| Claude-2 Anthropic () | 55 |
| GPT-3.5 Turbo OpenAI () | 41 |
| DeepSeek-LLM-67B-Chat Bi et al. () | 58 |
| Xwin-Math-70B (480K GSM8K) | 22 |
| Xwin-Math-70B (120K MATH) | 51 |
| Xwin-Math-70B (480K MATH) | 59 |
| Xwin-Math-70B (480K Mix) | 65 |

Figure. 2 (Left) presents the results of the model trained on GSM8K synthetic data, while Figure. 2 (Middle) presents the results of the model trained on MATH. We find that the accuracy of other benchmarks also improves as the amount of data increases for models trained with either GSM8K or MATH synthetic data. We also note that the generalization behaviors differ for GSM8K and MATH models: 1) SVAMP and ASDiv benefit more from GSM8K models than from MATH models. 2) While MATH models perform relatively well on the GSM8K benchmark, GSM8K models perform considerably worse on MATH benchmarks.

Figure. 2 (Right) shows the results of models using a mixture of GSM8K and MATH in a 1:1 ratio. These models exhibit balanced scaling behaviors in both in-domain and out-of-domain benchmarks.

Figure: Figure 2: Comparing the increase in SFT data scale using either a single dataset or mixed datasets.
Refer to caption: extracted/5422178/figures/mix.png

Figure: Figure 3: The Pass@256 and PassRatio@256 curve with increasing data size on GSM8K and MATH benchmark.
Refer to caption: extracted/5422178/figures/ratio.png

### 4.4 What Happens behind Performance Improvements?

Pass@256 v.s. PassRatio@256
To deepen the understanding behind the performance improvements, we tracked Pass@N metric and PassRatio@N metric under different data size. The results are shown in Figure 3. With very limited synthetic data (e.g. 7.5K samples), the Xwin-Math-70B model already has very high Pass@256, indicating the strong ability to generate correct answers through multiple attempts. Meanwhile, the Pass@256 metric only changed slightly with increasing the amount of used data. In contrast, PassRatio@256, which reflects the stability to generate correct answer, increases significantly with the amount of synthetic data, and its growth trend is similar to that of Pass@1. This result confirms our hypothesize that the performance improvements is mainly caused by the better stability in answer generation rather than stronger ability to answer the question.

Estimated Single-step Reasoning Accuracy
Because of the Chain-of-Thought (CoT) are adopted in inference, the process of answer mathematical problems is completed by a multi-step reasoning process. Therefore, we hypothesize that the increase in final answer accuracy can be interpreted by the improvement in single-step reasoning accuracy. Based on this assumption, if one question can be theoretically answered by $s$ reasoning steps in CoT, the final answer accuracy can be approximate by the power function of the single-step reasoning accuracy:

$$ $\text{Acc}_{\text{final}}=\text{Acc}_{\text{step}}^{s}$ (3) $$

With this equation, step accuracy can be estimated from the final answer accuracy. We experimented on GSM8K. For each question in the test set, we generated 256 responses and used the number of steps in the GSM8k test set’s CoT annotations as the theoretical CoT steps. We draw the curve to show the relationship between the number of CoT reasoning steps and mean final answer accuracy and show the fitted curve based on Equation. 3. We test Xwin-Math-7B models with different synthetic data, and the results are shown in Figure 4. The solid line is fitted using all seven points and Table  4 shows the estimated single-step accuracy when using different amounts of data using all data points, and it can be seen that the single-step accuracy improve significantly with more data.

Figure: Figure 4: Left: The relationship between the mean accuracy on the GSM8K and the number of annotated CoT steps with data increasing. The solid line is fitted using all seven points, while the dashed line is fitted using the first four points. Right: Changes in mean accuracy when resampling is used to increase the CoT lengeh of training data.
Refer to caption: extracted/5422178/figures/resample.png

However, when we fit based on Equation. 3 to the first four points, as shown in dashed lines, we found that the latter three points were significantly below the curve. We believe this phenomenon may be related to the smaller proportion of more complex problems in the training data. Therefore, we resampled the 960K synthetic data according to the number of sentences in CoT solution. As can be seen from Figure 4 (right), when the proportion of complex problems is increased, the accuracy for simpler problems remains virtually unchanged, but the accuracy for more complex problems can be significantly improved. Moreover, the utilization of data resampling can increase the model’s PassRatio@256 from 71.1 to 72.8. This experimental result provides new insights into data selection for mathematical reasoning tasks.

In addition, we further used the GPT-4 Turbo to find the position where the first step in our answer was wrong and normalized that position by the total number of steps in each answer. As the estimated single-step accuracy gets higher, the first error position of the normalization is postponed.

**Table 4: The estimated single-step reasoning accuracy and the average normalized first error position by GPT-4 Turbo in Xwin-Math-7B on GSM8K benchmark.**
| Data size | Estimated $\text{Acc}_{\text{step}}$ | Normalized first error position |
| --- | --- | --- |
| 7.5K | 78.9 | 67.1 |
| 120K | 89.7 | 83.9 |
| 960K | 94.2 | 90.9 |

The Improvement in the Accuracy of Numerical Calculations is More Significant than Logical Reasoning
The performance of the model gradually improves as the synthetic data increases. For a deeper understanding, we analyze the error proportion for different types of errors on GSM8K.
We categorized errors into two types: reasoning errors and calculation errors. Reasoning errors primarily encompass issues such as loss of conditions and concept confusion, while calculation errors include incorrect analysis of quantitative relationships and numerical computation mistakes. Based on the experimental results illustrated in Figure 5, we observe a gradual decrease in the percentage of calculation errors, suggesting that GSM8K is correcting calculation errors at a faster rate than reasoning errors.

Figure: Figure 5: Changes in the proportion of calculation and reasoning mistake during data increased.
Refer to caption: extracted/5422178/figures/mistake_type.png

### 4.5 Ablations on the Data Synthetic Schema

Comparison with Other Data Synthetic Methods
We compared our approach with following common used data synthetic methods:

Add Constraint. Adding one more constrain to the original question while keeping others unchanged, which is used in WizardMath and MuggleMath.

Change Numbers. Changing the numbers that appear in the problem while keeping the context intact. which is used in MuggleMath.

Change Background. Changing the background in the question while keeping others the same.

The Combination of Changing Numbers and Background. A hybrid approach that combines changing both numbers and background.

MetaMath Approach. The synthetic methods proposed in MetaMath, including answer augmentation, rephrasing question, self-verification question and FOBAR question. In experiments, we follow the implementation of MetaMath but use GPT-4 Turbo instead of GPT-3.5 Turbo to generate response data using their released questions.

The experimental results in the Figure 6 show that when the data size is relatively small, e.g., 7.5k and 30k samples, the performance gap between the different methods is negligible. However, as the data size increases, our method and the method with added constraints show stronger performance. This suggests that the choice of data synthetic strategy becomes more critical as the data size increases, and that some methods can scale the data more efficiently, thus improving the performance.

Figure: Figure 6: GSM8K and MATH performance of different synthetic methods.
Refer to caption: extracted/5422178/figures/prompt.png

**Table 5: Ablation of question verification on MATH.**
| Model | Pass@1 (%) |
| --- | --- |
| Xwin-Math-70B (7.5K data) | 28.9 |
| Xwin-Math-70B (7.5K data) w/o verification | 28.1 (-0.8) |
| Xwin-Math-70B (30K data) | 37.6 |
| Xwin-Math-70B (30K data) w/o verification | 36.6 (-1.0) |

Effects of Question Verification.
The question verification is used to further improve the generation quality. In our experiments, we found it can improve the performance on MATH benchmark, the results are shown in Table  5, while we do not see significantly impact on GSM8K dataset.

## 5 Related Works

#### Large Language Models

Large language models Brown et al. (); Achiam et al. (); Touvron et al. (, ) have made significant achievements, with impressive performance on a wide range of tasks. Currently, closed-source large language models, represented by GPT Brown et al. (); Achiam et al. (), Gemini Team et al. (), Grok xAI (), and Claude-2 Anthropic (), are the most advanced models in terms of performance. However, open-source models, represented by LLaMA Touvron et al. (), LLaMA-2 Touvron et al. () and Mixtral Jiang et al. (), have also progressed rapidly, and have even shown competitive performance with the closed-source models on some tasks. Our work, which aims to improve the performance of open-source LLMs on mathematical tasks by fine-tuning them on synthetic data.

#### Reasoning Framework for Improving Mathematical Capability

Chain-of-thoughts Wei et al. () encourages the LLMs perform multi-step reasoning by specific designed prompts and can improve reasoning performance. Based on this work, many subsequent works suggesting further improvements Fu et al. (); Zhang et al. (); Kojima et al. (). The above works focus primarily on how to improve performance through better prompt design or inference strategies without fine-tuning the model, whereas our work focuses on how to improve the model itself, and thus these approaches are complementary to ours.

#### Fine-tuned LLM for Math

Another sort of works Lightman et al. (); Luo et al. (); Azerbayev et al. (); Yue et al. (); Yu et al. (); An et al. (); Li et al. (); Gou et al. () try to improve performance directly by training the model on mathematical data.
A direct way is to use fine-tuning to improve models. One widely used method is to use synthetic data, which is very close to our approach: MetaMath Yu et al. () presents to bootstrap questions to augment data. LeMA An et al. () collects mistake-correction data pairs by using GPT-4 as a corrector. And MuggleMath Li et al. () augments the GSM8K dataset by incorporating GPT-4 with a series of pre-defined operations. Compared to these synthetic data based efforts, our data synthetic method is much simpler and more scalable due to introduce less prior and constraint.

#### SFT Data Scaling

Recently, some research efforts have focused on the data scale for supervised fine-tuning. For instance, LIMA Zhou et al. () mentions that fine-tuning with 1,000 high-quality instructions can yield impressive results in various general tasks. Other studies have indicated that performance scales with data size in mathematical and coding tasks Dong et al. (). Recent work Bi et al. () even uses 1.5 million data for instruct fine-tuning to obtain top performance. However, the intrinsic reasons behind this scaling effect have not been thoroughly investigated.

## 6 Conclusion

This study reveals that common 7B language models, such as LLaMA-2 7B, already exhibit strong mathematical capabilities, challenging the previous belief that advanced mathematical reasoning is exclusive to larger, more extensively pre-trained models. By significantly scaling up SFT data, we have markedly improved the stability of the model’s mathematical problem-solving skills. Our methodology has enabled the Xwin-Math models to reach performance levels comparable to, and in some instances surpassing, those of their larger counterparts. Our analysis also indicates that the enhancements are primarily attributable to heightened accuracy in single-step reasoning and a extra resampling of training data can improve the accuracy of harder questions. Additionally, we see more substantial reduction of calculation errors as opposed to logical reasoning errors. Our research contributes valuable insights into the mathematical capabilities of large language models.

## Acknowledgments

Chen Li and Nanning Zheng were supported in part by NSFC under grant No. 62088102. Thank Shengnan An at IAIR, Xi’an Jiaotong University for his valuable advice on this work.

## Appendix A Synthetic Prompt on GSM8K

[frametitle=Prompt 1: Question Generation]
Please act as a professional math teacher.

Your goal is to create high quality math word problems to help students learn math.

You will be given a math question. Please create a new question based on the Given Question and following instructions.

To achieve the goal, you have three jobs.

# Please generate a similar but new question according to the Given Question.

# Check the question by solving it step-by-step to find out if it adheres to all principles.

# Modify the created question according to your checking comment to ensure it is of high quality.

You have five principles to do this.

# Ensure the new question only asks for one thing, be reasonable, be based on the Given Question, and can be answered with only a number (float or integer). For example, DO NOT ask, ‘what is the amount of A, B and C?’.

# Ensure the new question is in line with common sense of life. For example, the amount someone has or pays must be a positive number, and the number of people must be an integer.

# Ensure your student can answer the new question without the given question. If you want to use some numbers, conditions or background in the given question, please restate them to ensure no information is omitted in your new question.

# Please DO NOT include solution in your question.

# If the created question already follows these principles upon your verification. Just keep it without any modification.

Given Question: given question

Your output should be in the following format:

CREATED QUESTION: <your created question>

VERIFICATION AND MODIFICATION: <solve the question step-by-step and modify it to follow all principles>

FINAL CREATED QUESTION: <your final created question>

[frametitle=Prompt 2: Answer Generation]
Please act as a professional math teacher.

Your goal is to accurately solve a math word problem.

To achieve the goal, you have two jobs.

# Write detailed solution to a Given Question.

# Write the final answer to this question.

You have two principles to do this.

# Ensure the solution is step-by-step.

# Ensure the final answer is just a number (float or integer).

Given Question: given question

Your output should be in the following format:

SOLUTION: <your detailed solution to the given question>

FINAL ANSWER: <your final answer to the question with only an integer or float number>

[frametitle=Prompt 3: Question Generation w/o verification]
Please act as a professional math teacher.

Your goal is to create high quality math word problems to help students learn math.

You will be given a math question. Please create a new question based on the Given Question and following instructions.

To achieve the goal, you have one job.

# Please generate a similar but new question according to the Given Question.

You have four principles to do this.

# Ensure the new question only asks for one thing, be reasonable, be based on the Given Question, and can be answered with only a number(float or integer). For example, DO NOT ask, ‘what is the amount of A, B and C?’.

# Ensure the new question is in line with common sense of life. For example, the amount someone has or pays must be a positive number, and the number of people must be an integer.

# Ensure your student can answer the new question without the given question. If you want to use some numbers, conditions or background in the given question, please restate them to ensure no information is omitted in your new question.

# You only need to create the new question. Please DO NOT solve it.

Given Question: given question

Your output should be in the following format:

CREATED QUESTION: <your created question>

## Appendix B Additional Results

Figure: Figure 7: Xwin-Math’s aggregate performance on these two benchmarks is second only to GPT-4, demonstrating our model’s robust generalization capabilities.
Refer to caption: x1.png

**Table 6: To validate benchmark data leakage during data generation, we compare LM loss on: 1) a training subset, which is a small subset with 256 samples from all synthetic data; 2) a regenerated training subset, where we maintain the original questions from the training subset and use GPT-4 Turbo to rewrite answers; 3) a regenerated test set, where we keep the questions from the evaluation metrics unchanged and use GPT-4 Turbo to rewrite answers; 4) a reference test set, where we use the test set as seed to generate new questions and answers via GPT-4 Turbo. Referring to Skywork, we also report two key metrics: $\Delta_{1}=L_{\text{test-regen}}-L_{\text{test-ref}},\Delta_{2}=L_{\text{test- regen}}-L_{\text{train-regen}}$, As $\Delta_{1}$ is close to 0 and $\Delta_{2}$ is significantly greater than 0 in two benchmarks, we believe that there is no leakage during the process of data synthesis.**
| Dataset | $L_{\text{test-regen}}$ | $L_{\text{test-ref}}$ | $L_{\text{train}}$ | $L_{\text{train-regen}}$ | $\Delta_{1}$ | $\Delta_{2}$ |
| --- | --- | --- | --- | --- | --- | --- |
| GSM8K | 0.52 | 0.50 | 0.11 | 0.33 | 0.02 | 0.19 |
| MATH | 0.59 | 0.58 | 0.23 | 0.39 | 0.01 | 0.20 |

## Appendix C Case Study of Synthetic Data

### C.1 GSM8K

[frametitle=Original Question]
Randy has 60 mango trees on his farm. He also has 5 less than half as many coconut trees as mango trees. How many trees does Randy have in all on his farm?

[frametitle=Synthetic Question w/o Verification]
Maria has 80 apple trees in her orchard. She also has 4 less than a third as many cherry trees as apple trees. How many trees does Maria have in all in her orchard?

[frametitle=Verification Comment]
Let’s check the created question:

- Maria has 80 apple trees.

- She has 4 less than a third as many cherry trees as apple trees. A third of the apple trees is 80/3, which is approximately 26.67.

- Four less than that would be around 22.67, but since we cannot have a fraction of a tree, we round it to the nearest whole number, which is 27.

- The modified question should ask for the number of cherry trees without the decimal. So we adjust the question to have a whole number as a result.

- Since 80 is not divisible by 3, to get a whole number of cherry trees, we can change the number so that it is divisible by 3. Let’s use 81 apple trees instead of 80.

[frametitle=Synthetic Question with Verification]
Maria has 81 apple trees in her orchard. She also has 4 less than a third as many cherry trees as apple trees. How many trees does Maria have in all in her orchard?

[frametitle=Synthetic Solution and Final Answer]
First, let’s determine the number of cherry trees Maria has. We are told she has 4 less than a third as many cherry trees as apple trees.

Since Maria has 81 apple trees, let’s find one third of that number:

1/3 of 81 apple trees = 81 apple trees / 3 = 27 apple trees.

Now, since she has 4 less than this amount in cherry trees, we subtract 4 from 27 to find the number of cherry trees:

27 cherry trees - 4 = 23 cherry trees.

To find the total number of trees Maria has in her orchard, we add the number of apple trees to the number of cherry trees:

81 apple trees + 23 cherry trees = 104 trees.

The answer is 104.

### C.2 MATH

[frametitle=Original Question]
The value of $\log_{10}{28471}$ is between the consecutive integers $a$ and $b$. Find $a+b$.

[frametitle=Synthetic Question w/o Verification]
The value of $\log_{2}{16384}$ is between the consecutive integers $x$ and $y$. Find $x+y$.

[frametitle=Verification Comment]
To ensure the question can be solved, we need to check if the logarithm given can be evaluated and falls between two consecutive integers.

1. Identify the value of $2^{n}$ that equals $16384$.

2. If $16384$ is a power of $2$, then $\log_{2}{16384}$ will be an integer.

3. Check to ensure the result is an integer and find the consecutive integers around it.

Let’s perform the check:

$2^{0}=1$, $2^{1}=2$, $2^{2}=4$, $2^{3}=8$, $2^{4}=16$

…

$2^{14}=16384$

So, $\log_{2}{16384}=14$.

The consecutive integers around 14 are (13 and 14) or (14 and 15).

Since the logarithm gives an exact integer, to make the question valid, we will modify it to use a number that is not an exact power of 2.

MODIFIED CREATED QUESTION:

The value of $\log_{2}{10000}$ is between the consecutive integers $m$ and $n$. Find $m+n$.

Now, let’s check the modified question:

1. Since 10000 is not a power of 2, $\log_{2}{10000}$ will not be an integer.

2. The goal is to find two consecutive integers between which this logarithm value lies.

3. Knowing that $2^{13}=8192$ and $2^{14}=16384$, $\log_{2}{10000}$ is between 13 and 14.

The consecutive integers around $\log_{2}{10000}$ are 13 and 14.

[frametitle=Synthetic Question with Verification]
The value of $\log_{2}{10000}$ is between the consecutive integers $m$ and $n$. Find $m+n$.

[frametitle=Synthetic Solution and Final Answer]
To find the consecutive integers between which $\log_{2}{10000}$ lies, we need to find the powers of $2$ that are immediately less than and greater than $10000$.

First, let’s find the largest power of $2$ that is less than $10000$. Starting with $2^{1},2^{2},\ldots$ and so on, we find:

$2^{10}=1024$, $2^{11}=2048$, $2^{12}=4096$, $2^{13}=8192$, $2^{14}=16384$

Here, we see that $2^{13}<10000<2^{14}$, which means that $\log_{2}{10000}$ is greater than $13$ but less than $14$.

Therefore, the consecutive integers $m$ and $n$ between which $\log_{2}{10000}$ lies are $m=13$ and $n=14$.

Now we will add these two integers to find the required sum:

$m+n=13+14=27$.

The answer is 27.

## Appendix D Case Study of Model Response

### D.1 GSM8K

[frametitle=Question and Ground Truth Solution]
Question: A treasure hunter found a buried treasure chest filled with gems. There were 175 diamonds, 35 fewer rubies than diamonds, and twice the number of emeralds than the rubies. How many of the gems were there in the chest?

Solution:
There were 175 - 35 = 140 rubies in the treasure chest. There were 140 x 2 = 280 emeralds in the treasure chest. So, the total number of gems in the treasure chest is 175 + 140 + 280 = 595.

[frametitle=Response – Xwin-Math-7B (7.5k data) Greedy Decode]
First, let’s find out the number of rubies. We are told there are 35 fewer rubies than diamonds. Since there are 175 diamonds, we can calculate the number of rubies as follows:

Number of rubies = Number of diamonds - 35 = 175 - 35 = 140

Next, we need to find out the number of emeralds. We are told there are twice the number of emeralds than the rubies. So we can calculate the number of emeralds as follows:

Number of emeralds = 2 × Number of rubies = 2 × 140 = 280

Now, we can find the total number of gems by adding the number of diamonds, rubies, and emeralds together:

Total number of gems = Number of diamonds + Number of rubies + Number of emeralds = \dbox175 + 140 + 280 = 605

The answer is: 605. Incorrect
Wrong Calculation

[frametitle=Response – Xwin-Math-7B (7.5k data) One Sample from Temperature=0.7]
Let’s find the number of each type of gem in the chest step by step.

First, we are given the number of diamonds:

Number of diamonds = 175

Next, we are told there are 35 fewer rubies than diamonds. To find the number of rubies, we subtract 35 from the number of diamonds:

Number of rubies = Number of diamonds - 35 = 175 - 35 = 140

Then, we are told there are twice the number of emeralds than the rubies. To find the number of emeralds, we multiply the number of rubies by 2:

Number of emeralds = 2 * Number of rubies = 2 * 140 = 280

Now, we add the number of diamonds, rubies, and emeralds together to find the total number of gems in the chest:

Total number of gems = Number of diamonds + Number of rubies + Number of emeralds = 175 + 140 + 280 = 595

The answer is: 595. Correct

[frametitle=Response – Xwin-Math-7B (960k data) Greedy Decode]
Let’s break down the information given in the problem step by step.

1. We know there are 175 diamonds.

2. There are 35 fewer rubies than diamonds. Since there are 175 diamonds, we can calculate the number of rubies by subtracting 35 from 175.

Number of rubies = Number of diamonds - 35 = 175 - 35 = 140

3. There are twice the number of emeralds than the rubies. We can find the number of emeralds by multiplying the number of rubies by 2.

Number of emeralds = 2 × Number of rubies = 2 × 140 = 280

Now, to find the total number of gems in the chest, we add the number of diamonds, rubies, and emeralds together.

Total number of gems = Number of diamonds + Number of rubies + Number of emeralds = 175 + 140 + 280 = 595

The answer is: 595. Correct

### D.2 MATH

[frametitle=Question and Ground Truth Solution]
Question: If $f(x)=\frac{3x-2}{x-2}$, what is the value of $f(-2)+f(-1)+f(0)$? Express your answer as a common fraction.

Solution:
$f(-2)+f(-1)+f(0)=\frac{3(-2)-2}{-2-2}+\frac{3(-1)-2}{-1-2}+\frac{3(0)-2}{0-2}=
\frac{-8}{-4}+\frac{-5}{-3}+\frac{-2}{-2}=2+\frac{5}{3}+1=$$\frac{14}{3}$

[frametitle=Response – Xwin-Math-7B (7.5k data) Greedy Decode]
To find the value of $f(-2)+f(-1)+f(0)$, we need to evaluate the function $f(x)$ at these three values of $x$.

First, let’s find $f(-2)$:
$f(x)=\frac{3x-2}{x-2}$
Plug in $x=-2$:
$f(-2)=\frac{3(-2)-2}{-2-2}=\frac{-6-2}{-4}=\frac{-8}{-4}$

Now, let’s find $f(-1)$:
Plug in $x=-1$:
$f(-1)=\frac{3(-1)-2}{-1-2}=\frac{-3-2}{-3}=\frac{-5}{-3}$

Finally, let’s find $f(0)$:
Plug in $x=0$:
$f(0)=\frac{3(0)-2}{0-2}=\frac{-2}{-2}=1$

Now, we can find the sum:
$f(-2)+f(-1)+f(0)=\frac{-8}{-4}+\frac{-5}{-3}+1$

To add these fractions, we need to find a common denominator, which is the least common multiple (LCM) of the \dboxthree denominators (-4, -2, and -3) . The LCM of these denominators is -12.

Now, we can rewrite each fraction with the common denominator:

$\frac{-8}{-4}=\frac{4}{3}$ , $\frac{-5}{-3}=\frac{5}{3}$, $1=\frac{12}{12}$

Now, we can add the fractions:
\dbox$\frac{4}{3}+\frac{5}{3}+\frac{12}{12}=\frac{4+5+12}{3}=\frac{19}{3}$

The answer is: $\frac{19}{3}$. Incorrect

[frametitle=Response – Xwin-Math-7B (7.5k data) One Sample from Temperature=0.7]
To find the value of $f(-2)+f(-1)+f(0)$, we first need to evaluate each function at these specific values of x.

For $f(-2)$:
$f(x)=\frac{3x-2}{x-2}$
Substitute $x=-2$ into the equation:
$f(-2)=\frac{3(-2)-2}{-2-2}=\frac{-6-2}{-4}=\frac{-8}{-4}=2$

For $f(-1)$:
$f(x)=\frac{3x-2}{x-2}$
Substitute $x=-1$ into the equation:
$f(-1)=\frac{3(-1)-2}{-1-2}=\frac{-3-2}{-3}=\frac{-5}{-3}=\frac{5}{3}$

For $f(0)$:
$f(x)=\frac{3x-2}{x-2}$
Substitute $x=0$ into the equation:
$f(0)=\frac{3(0)-2}{0-2}=\frac{-2}{-2}=1$

Now we can add the values of $f(-2)$, $f(-1)$, and $f(0)$:
$f(-2)+f(-1)+f(0)=2+\frac{5}{3}+1=\frac{14}{3}$

The answer is: $\frac{14}{3}$. Correct

[frametitle=Response – Xwin-Math-7B (480k data) Greedy Decode]
To find the value of $f(-2)+f(-1)+f(0)$, we need to evaluate the function $f(x)$ at each of the given points and then sum the results.

First, let’s find $f(-2)$:
$f(-2)=\frac{3(-2)-2}{(-2)-2}=\frac{-6-2}{-2-2}=\frac{-8}{-4}=2$

Next, let’s find $f(-1)$:
$f(-1)=\frac{3(-1)-2}{(-1)-2}=\frac{-3-2}{-1-2}=\frac{-5}{-3}=\frac{5}{3}$

Finally, let’s find $f(0)$:
$f(0)=\frac{3(0)-2}{0-2}=\frac{0-2}{-2}=\frac{-2}{-2}=1$

Now, we can sum these values:
$f(-2)+f(-1)+f(0)=2+\frac{5}{3}+1$
To add these, we need a common denominator, which is 3:
$2+\frac{5}{3}+1=\frac{2\cdot 3}{3}+\frac{5}{3}+\frac{1\cdot 3}{3}=\frac{6}{3}+
\frac{5}{3}+\frac{3}{3}$

Now, we add the numerators:
$\frac{6}{3}+\frac{5}{3}+\frac{3}{3}=\frac{6+5+3}{3}=\frac{14}{3}$

The answer is: $\frac{14}{3}$. Correct

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Potential Overfitting to Evaluation Benchmarks: Given the synthetic supervised fine-tuning (SFT) data generation process and its similarity to GSM8K and MATH benchmark questions, there is a concern that the model may be overfitting to these evaluation datasets. Could the authors elaborate on any measures taken to prevent overfitting, and whether they tested on additional, unrelated benchmarks to ensure that the model’s mathematical abilities generalize beyond GSM8K and MATH?
soundness: 2
strengths: The authors uncover an interesting and less-explored phenomenon: by generating a large number of responses (up to 1024) per question, even small-scale language models like LLaMA-2 7B can achieve remarkably high accuracy on mathematical benchmarks.
Comprehensive Experimental Evaluation: The authors conduct extensive experiments across multiple benchmarks, including GSM8K, MATH, etc.
Well-Structured Presentation: The paper is organized logically, guiding the reader through the motivation, methodology—including the novel use of multiple response generations—results, and analysis in a coherent manner.
summary: The paper asserted that  mathematical abilities of small-scale language models, LLaMA-2 7B, inherently possess strong mathematical capabilities. They prompt the pre-trained LLaMA-2 7B model to generate multiple answers (up to 1024) for each question in math benchmarks like GSM8K and MATH. By selecting the correct answer from these numerous attempts (an "oracle" approach), they find that the model can achieve high accuracy (97.6% on GSM8K and 70% on MATH). To improve consistency, they hypothesize that increasing the amount of supervised fine-tuning data can help the model learn to produce correct answers more reliably. They use a stronger language model, GPT-4 Turbo, to generate a large number of synthetic math problems and their solutions. However, the reference questions are from  GSM8K and  MATH. In other words, the new questions are similar to the questions in GSM8K and MATH. Which ends up with overfitting towards to solving GSM8K and MATH.
weaknesses: Lack of Novelty: The approach of enhancing language models using synthetic data and multiple answer generation has been explored in previous studies. And the way they generate the SFT data may cause overfitting towards the evaluation..
Practical Limitations: Generating 1024 responses per question to select the correct answer is impractical for real-world applications due to computational inefficiency.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: weaknesses
soundness: 1
strengths: Strengths:
Scaling Approach: The paper demonstrates that scaling synthetic data can effectively improve performance for smaller models, which challenges the assumption that only large-scale models excel in complex reasoning tasks.
High Benchmark Performance: The LLaMA-2 7B model's performance on GSM8K and MATH benchmarks is commendable and suggests that synthetic data can be a viable alternative to real data in specific contexts.
Experiment Setup: The paper includes various data scaling scenarios and compares performance against state-of-the-art models, adding insight into the practical applications of synthetic data for improving mathematical capabilities in language models.
summary: This paper explores the mathematical capabilities of the LLaMA-2 7B model, suggesting that small language models can achieve high performance in mathematical reasoning when fine-tuned with synthetic data generated by GPT-4 Turbo. The LLaMA-2 7B model achieves 82.4% accuracy on the GSM8K benchmark and 40.1% on the MATH benchmark, outperforming some larger models, including earlier versions of GPT-4. By scaling up the supervised fine-tuning (SFT) data with GPT-4-generated questions, the authors claim that the synthetic data performs comparably to real data. The approach purportedly avoids performance saturation and shows competitive results across benchmarks.
weaknesses: 1 Training Data Overlap Concerns: There’s a significant possibility that LLaMA-2's pretraining data includes or paraphrases the benchmarks used in evaluation, especially given the unknowns around its training corpus. This overlap could mean that the results simply "unlock" capabilities that were already in the model due to exposure during pretraining. Testing on a model where pretraining data is known and does not include the benchmarks would provide a more credible assessment.
2 Limited Benchmark Selection: The focus on GSM8K and MATH restricts evaluation to high school and grade school mathematics. More challenging, diverse benchmarks like MMLU (especially college-level sections) and OlympiadBench could provide a stronger test of the model's robustness. Success on OlympiadBench (IMO) [3], for example, would be a stronger indication of broader mathematical reasoning abilities.
3 Overstated Claims of Generality: The paper frequently implies that its findings apply to "common 7B LMs," yet the experiments focus solely on LLaMA-2 7B. Generalizing these results to other 7B models without further testing is premature.
4 No Discussion on Synthetic Data Risks: The paper lacks a discussion on "model collapse" from synthetic data, an issue documented in the literature [2]. While using GPT-4 may mitigate some risks, the absence of model degradation across such extensive data scaling is unusual. This aspect needs further exploration to explain the improvement consistency.
5 Large Data Volume Requirements: The use of 1024 generations per problem (yielding up to 76M examples) raises questions about the efficiency and feasibility of this approach. Such large data requirements, combined with the potential overlap between synthetic and benchmark data, make the setup less compelling.
6 Reliance on GPT-4 as Judge: The use of GPT-4 to evaluate generated responses without validating its correlation with human evaluations weakens the reliability of the reported improvements. A smaller-scale human evaluation or citation demonstrating alignment between GPT-4 judgments and human judgments would improve credibility.
7 Power Law Claim without Evidence: The claim of "power law scaling" is unsupported by a functional form or detailed analysis. Without a clearly defined functional relationship, the observed improvements should be framed as scaling trends, not power laws.
Reservations - Writing Concerns:
8 Misleading Introduction on Emergence: The introduction on emergence theory is confusing (It’s known emergence is caused by metrics now and in addition this is not how we grade people) and does not add value to the main narrative. It distracts from the paper's core contributions and should be revised or strongly suggested its removed.
9 Lack of Quantitative Detail in the Abstract: Statements like "significant improvement" are vague without absolute or relative performance changes. Quantitative results and implications should be included in the abstract to clearly communicate the impact of the findings.
10 Clarity and Readability: The paper would benefit from a thorough review by a native English speaker. There are multiple areas where phrasing is unclear or lacks precision, detracting from the scientific rigor of the presentation.
11 Captions and Figures: The captions do not provide sufficient context or summaries of the figures' main takeaways. Including summary values (like percentages) in the captions or directly in the figures would make it easier for readers to interpret the results.
12 “Table 2 shows the results.” This type of sentence raises major concerns about the writing of the whole paper.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: Below I've started w/ a few major questions and then follow w/ a number of small suggestions:
1. Will xwin and the associated dataset be open sourced? That would substantially increase the significance of this work and may influence my overall rating.
2. Where did the xwin name come from? what does it mean ?
3. lines 15-16: "achieved by selecting the oracle response from 1024 generations". You mean pass@k?Please try to use standard terminology or reword this part.
4. lines 16-17: "Equipped GPT-4 Turbo as an additional verification"  This has a grammatical error and doesn't make sense nor stand on it's own. Please correct the grammar on these lines, describe in a straightforward way, and improve the clarity of the methods section on this topic in the main body of the paper.
5. lines 48-51: you’re comparing pass@1024 to pass@1. Please clarify for readers.
6. line 100: “based on preference ones” doesn’t make sense
7. lines 106-107: "This data scaling shows excellent scaling behavior". The wording “excellent scaling behavior” is subjective and not defined. Please make clear what you mean here or move it to the discussion or remove it.
8. Eq 3: This equation is a nice observation and could be  connected more deeply via experiment or theory  to explain how training on synthetic drives the model to select the correct generation.
9. Section 4.3 and Figure 5: While an easy analysis, i could equally conclude that this is more about the model getting better at long problems than improving reasoning chains. Actual analysis of step wise accuracy via an LLM grader or humans or an existing dataset (like PRM800k if possible) would be  more conclusive
soundness: 3
strengths: The paper has interesting empirical findings that may be of interest to researchers & developers on the math performance of small LLMs on large amounts of synthetic distillation data. The paper is generally well written and easy to follow, for the most part. The methods are detailed especially nicely to ensure ease of understanding and reproducibility.
summary: The authors present interesting findings that llama2-7b models can be fine-tuned to the level of SOTA closed source models on the MATH and GSM8K benchmarks when trained on large amounts of synthetic data from these SOTA models in a straight forward distillation process.
weaknesses: The main weaknesses are:
1. Lack of significant novelty or surprising results: Prior work has shown that extensive math-related data pre-training improves the math capabilities of 7B scale models. These results, illustrated even w/in the first sentence of the abstract, reduces the novelty of this work. It follows that if 7B scale models have stronger math performance after pretraining on more math related data, that we can similarly distill strong model math perf into smaller models in post-training. 
2. Lack of depth of analysis: you make an important observation that "The primary issue is the lack of guarantee that the correct answer will be dug out, as most generations are incorrect.", but this point is not deeply studied in the rest of the paper. The initial motivating observation that 7B models high have pass@1024 is under explored in empirical experiments, theoretical analysis, and final conclusion.
3. Severe lack of breadth:
 a. the lack of evaluation on llama3 is a glaring miss that the authors likely foresaw would arise during reviews. Given that the empirically strong performance of distilled llama2 is one of the strongest conclusions of this paper, running the analysis on llama3 is left as an obviously open question
b. The analysis conducted only on math and gsm8k. It's not clear how limited the author's claims are to these benchmarks. Further, section 4.3 would be much stronger using LLM or human stepwise grading or if an open source process-supervision dataset such as PRM800k could be leveraged.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: 1- For generating new questions, the authors use an LLM via prompts in the appendix, but it's unclear where the seed/original questions come from. For instance, for GSM8K, do the original questions come from the train set or test set?
soundness: 2
strengths: 1- Understanding reasoning capabilities of models, including mathematical capabilities, is an important research direction.   
2- The paper is well-written and the arguments are presented well.
summary: The paper investigates the mathematical capabilities of LLMs with a focus on GSM8K and MATH datasets. First, the authors show that with a large number of response generations (e.g., generating 1024 responses), the accuracy, measured by the probability of at least one generated response being correct (e.g., pass@1024), improves significantly. The paper claims that the presence of correct answers within these 1024 generated responses suggests that the model already possesses strong math abilities, and the observed performance issues are due to instability in extracting responses, which the paper refers to as a "ranking issue". To alleviate this, the authors propose using an external LLM (e.g., GPT-4) to generate synthetic data for finetuning, which can increase accuracy on various math benchmarks.
weaknesses: 1- The observation that increasing the number of generations improves the probability of obtaining a correct answer is not new to this work. For example, figure 4 from [1] reports the same observation on GSM8K.

2- Most importantly, the use of pass@1024 to claim a model's "capability" is methodologically questionable. Consider this extreme example for clarity: Imagine the target task is finding a path from start to end in a maze. If I write a program that randomly tests paths in the maze, with enough generations (depending on the maze size and difficulty), one of the outputs will be correct. Does that mean my program is capable of solving mazes? While LLMs don't output random tokens, and this is an extreme example for demonstration purposes, the overall situation is similar.

3- The paper assumes that higher accuracy on GSM8K indicates better 'mathematical reasoning' capabilities. Recent works such as [2] suggest that models do not understand mathematical concepts and rely on pattern matching and memorization, which could explain the reported improved results from using synthetic data for finetuning in this work. Note that the experiments in this paper does not provide any evidence for reasoning improvement, including the OOD experiments. For instance, the Hungarian National High School Math Exam is very similar to the MATH benchmark, and we observe from figure 2 that training on 120K synthetic examples from MATH leads to ~30% more accuracy than training on 480K synthetic examples from GSM8K. This raises questions about whether this benchmark is truly OOD for MATH, and whether the improvements reflect enhanced "mathematical reasoning" or simply exposure to similar questions in the synthetic data, especially given that we cannot rule out GPT-4's prior exposure to these questions.

4- Regarding the generation/use of synthetic data, there are no particularly new insights. There is no formal mechanism to guarantee the correctness of problems and solutions. In fact, Figure 8 of the paper (in Appendix) shows that the reasoning mistakes of the model increase as more synthetic data is used. This relates to my previous comments that there is no evidence (neither in the literature nor this work) that these LLMs possess "mathematical understanding/reasoning capabilities". However, there are several pieces of evidence, including this work, showing that the models rely on seeing similar questions and memorizing overall patterns to solve similar questions.

Overall, I find many of the paper's assumptions and claims to be incorrect, and the experiments do not adequately address these issues.

----
References:  
[1] Shi, Chufan, et al. "A thorough examination of decoding methods in the era of llms." arXiv preprint arXiv:2402.06925 (2024).
[2] Mirzadeh, Iman, et al. "Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models." arXiv preprint arXiv:2410.05229 (2024).  
[3] Zhang, Hugh, et al. "A careful examination of large language model performance on grade school arithmetic." arXiv preprint arXiv:2405.00332 (2024).