MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoning (2024)

Abstract: In math reasoning with large language models (LLMs), fine-tuning data augmentation by query evolution and diverse reasoning paths is empirically verified effective, profoundly narrowing the gap between open-sourced LLMs and cutting-edge proprietary LLMs. In this paper, we conduct an investigation for such data augmentation in math reasoning and are intended to answer: (1) What strategies of data augmentation are more effective; (2) What is the scaling relationship between the amount of augmented data and model performance; and (3) Can data augmentation incentivize generalization to out-of-domain mathematical reasoning tasks? To this end, we create two new dataset AugGSM8K and AugMATH, by complicating and diversifying the queries and sampling multiple reasoning paths from GSM8K and MATH. We obtained a series of LLMs called MuggleMath by fine-tuning LLaMA models on AugGSM8K and AugMATH. MuggleMath substantially achieves new state-ofthe-art on GSM8K and MATH. A log-linear relationship and a segmented log-linear are presented between MuggleMath’s performance and the amount of augmented data on GSM8K and MATH, respectively. We also find that it is weak in out-of-domain math reasoning generalization from AugGSM8K to MATH and from AugMATH to GSM8K, which suggests that augmenting queries that cover a broader range of subjects is more beneficial for generalization.

heading: 1 Introduction
text: The emergence of large language models (LLMs) (Ouyang et al., 2022; Anil et al., 2023; OpenAI, 2023) has profoundly revolutionized the field of natural language processing, exhibiting
*Work done during internships at Alibaba Group. †Corresponding author ‡Xiang Wang is also affiliated with Institute of Dataspace,
Hefei Comprehensive National Science Center.
versatile performance in various tasks like code generation (Chen et al., 2021; Luo et al., 2023b; Wang et al., 2024b; Wei et al., 2023), instruction following (Longpre et al., 2023; Zhou et al., 2023a; Lei et al., 2024), long context question answering (Tworkowski et al., 2023; Luo et al., 2024), and math reasoning (Wei et al., 2022; Taylor et al., 2022; Lewkowycz et al., 2022a). Math reasoning as a representative reasoning task is widely studied to access the reasoning abilities in LLMs (Cobbe et al., 2021; Hendrycks et al., 2021a). Proprietary LLMs, such as GPT-3.5, and GPT4 (OpenAI, 2023) have shown exceptional mathematical reasoning abilities, while there remains a substantial gap between open-source LLMs, such as GPT-J (Wang and Komatsuzaki, 2021) and LLaMA (Touvron et al., 2023a,b)) and the cutting-edge proprietary models.
To enable better mathematical reasoning abilities in open-sourced LLMs, they generally undergo a fine-tuning stage on supervised reasoning datasets. A series of efforts are committed to enhancing the mathematical reasoning capabilities of open-source LLMs, where a mainstream approach involves first augmenting new mathematical problems and answers, followed by supervised fine-tuning on the augmented dataset (Yuan et al., 2023a; Luo et al., 2023a; Yu et al., 2023). This type of approach has achieved good results, and in this paper, we would like to explore what are the key factors affecting the effectiveness of data augmentation for mathematical reasoning tasks and the scaling relationship between the amount of data augmentation and model performance. Specifically, with the help of proprietary models (GPT-3.5 and GPT-4), we applied five types of mathematical problem augmentation methods based on human experience in creating variations of mathematical problems similar to Luo et al. (2023b,a). We further generated multiple reasoning paths for each augmented problem since distinct reasoning paths can also enhance chain-of-
10230
thought reasoning (Huang et al., 2022; Zhu et al., 2023a; Yuan et al., 2023a). We obtained two new datasets called AugGSM8K and AugMATH after data augmentation on two widely used mathematical reasoning datasets GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021b). By supervised fine-tuning on the open-source LLaMA (Touvron et al., 2023a) and LLaMA-2 (Touvron et al., 2023b) LLMs on AugGSM8K and AugMATH, we obtained a series of models dubbed MuggleMath. We find that with sufficient amounts of data, MuggleMath achieves a new state-of-the-art on GSM8K and MATH. In addition to this, we find a loglinear relationship between the performance of MuggleMath and the amount of data augmentation over a range of data volumes on GSM8K and a segmented log-linear relationship on MATH.
Although MuggleMath achieves strong performance on the GSM8K and MATH test set, the rationales for performance improvement by data augmentation remain unclear. We are therefore interested in the specific reason behind the performance improvement and whether it brings enhancement in LLMs’ mathematical reasoning capabilities generally.
To validate the generalization of MuggleMath, we conduct multi-task learning and analyze the transferability with AugGSM8K and AugMATH. We found that LLMs trained with supervised learning after data augmentation on GSM8K only bring marginal improvements to performance on MATH and the similar conclusion is fit for AugMATH and GSM8K. By visualizing the data distribution in the embedding space of LLaMA-2-7B, we observe that the embedding distribution of problems in AugGSM8K is very close to that of GSM8K, but significantly different from the problem distribution in the MATH dataset. The reason behind can be attributed to the fact that GSM8K and MATH have different reasoning difficulty, response style and require different mathematical knowledge.
The main contributions of our work can be summarized as follows:
• By augmenting GSM8K and MATH with various queries and multiple reasoning paths, we curate GSM8K and MATH to two new datasets named AugGSM8K and AugMATH.
• We utilize AugGSM8K and AugMATH for fine-tuning the LLaMA and LLaMA-2 models to obtain MuggleMath, which greatly improves the in-domain performance of the
open-sourced LLMs on GSM8K and MATH, achieving new state-of-the-art performances.
• We find a log-linear relationship between the accuracy of the model on the test set and the amount of data augmentation within a certain range while the coefficient is similar to augmenting new human-written samples on GSM8K. When it comes to MATH, a a segmented log-linear relationship is found.
• We demonstrate that the performance gains from data augmentation on GSM8K and MATH are difficult to generalize to each other, which indicates a need of diverse original queries in augmenting math data.

heading: 2 Related Works
text: Mathematical Reasoning for Large Language Models Mathematical reasoning is a crucial ability to examine large language models (Cobbe et al., 2021; Hendrycks et al., 2021a; Wei et al., 2022; Yuan et al., 2023b). The mathematical reasoning ability of LLMs can be enhanced by math-related pre-training (Hendrycks et al., 2021a; Lewkowycz et al., 2022a; Taylor et al., 2022; Lightman et al., 2023a) and math-related supervised finetuning (Yuan et al., 2023a; Luo et al., 2023a; Yue et al., 2023b; Yu et al., 2023). Query augmentation (Luo et al., 2023a; Yu et al., 2023) and response augmentation (Huang et al., 2022; Zelikman et al., 2022; Ni et al., 2023; Zhu et al., 2023b; Yuan et al., 2023a) are useful techniques to improve math in-domain performances during SFT. Query augmentation methods usually generate rephrased, easier, or harder problems and use proprietary LLMs to generate answers. Response augmentation methods generate new reasoning paths for problems in the training set. They could rely on answers in the training set to filter the generated reasoning paths. Yuan et al. invests the scaling relationship on supervised LLMs math performance with pre-train loss, supervised data amount, and augmented reasoning path amount. Our work is further investigate on scaling relationships with query augmentation and out-of-domain generalization. MetaMath(Yu et al., 2023) is a contemporary work that is similar to us in the augmentation method. The distinction lies in MetaMath’s focus on rewriting original questions to create new ones using the questions’ mathematical relationships. In contrast, our efforts are centered on generating new problems with equal or
greater difficulty levels. Moreover, our work investigates the quantitative relationship between query and response augment amounts and in-domain and out-of-domain performances.
Data Augmentation for LLM Data augmentation is a common technique to improve downstream task performance in NLP (Feng et al., 2021). In the era of large language models, data augmentation is usually used for generating instruction following SFT datasets (Wang et al., 2023b; Taori et al., 2023; Xue et al., 2023). Queries (Ding et al., 2023; Xu et al., 2023) and responses (Mukherjee et al., 2023) of SFT datasets can both be augmented by prompting state-of-the-art proprietary LLMs. Compared with their work, we are concentrated on augmenting math SFT dataset and we are more interested in scaling relationships on in-domain and out-ofdomain generalizations.
Out-of-Distribution Generalization The challenge of out-of-distribution (OOD) generalization has garnered widespread attention across various domains (Karras et al., 2018; Wang et al., 2021; Song et al., 2023; Zhou et al., 2023b; Peng et al., 2024) in machine learning. This issue arises when the distribution of data encountered by a model during testing diverges from that of the training phase, leading to a decline in model performance. The OOD problem is multifaceted (Lipton et al., 2018; Schölkopf et al., 2012; Tran et al., 2022; Cai et al., 2023), with subcategories such as covariate shifts and concept shifts, among others. To mitigate the effects of OOD scenarios, a diverse array of strategies has been developed, including unsupervised domain generalization (Wang et al., 2021; Zhou et al., 2023b), stable learning (Shen et al., 2020; Kuang et al., 2020), invariant representation learning (Creager et al., 2021), causal learning (Peters et al., 2015), and invariant risk minimization (Mao et al., 2023), multi-task Learning (Dong et al., 2024; Wang et al., 2024a) and contrastive learning (Peng et al., 2023). Recent trends in the community have shown a growing preference for performance enhancement through data augmentation during the Self-supervised Finetuning (SFT) stage in large-scale models. However, the extent of OOD issues associated with this method and their severity remain underexplored. This study aims to fill this gap by conducting empirical experiments and providing a visual analysis to assess the impact of data augmentation on OOD generalization in the context of large models.

heading: 3 Experiments
text: We first introduce our experimental setup in Section 3.1 and dataset augmentation method in Section 3.2. Then we conduct analyses spanning several aspects of data augmentation to answer the three research questions in abstract. For space saving, we mainly analyze the augmentation on GSM8K and detailed discussion of augmentation on MATH are list in Appendix A.

heading: 3.1 Experimental Setup
text: Problem Definition We define the math reasoning SFT dataset as D = {qi, ai}i, where qi is a question and ai is a reasoning path with an answer. We augment the SFT dataset to a new dataset D′ = {q′i, a′i}i. We apply SFT on the pre-trained language models and measure the augmented SFT dataset based on the accuracy of the in-domain test set Din and out-of-domain test set Dout. We calculate the accuracy based on greedy decoding.
Datasets GSM8K (Cobbe et al., 2021) is a dataset with elementary school math word problems with 7,473 training problems and 1,319 testing problems. The test set of GSM8K is viewed as in-domain test datset for AugGSM8K and out-of-domain test dataset for AugMATH. MATH (Hendrycks et al., 2021a) is a dataset with challenging high-school math problems. Problems are classified into the following topics: Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus. Problems in MATH are harder and more diverse than in GSM8K. We use test set of MATH as our in-domain test dataset for AugMATH and out-of-domain test dataset for AugGSM8K.
Training We employ state-of-the-art opensource LLMs for fine-tuning, including LLaMA-1 7B (Touvron et al., 2023a), LLaMA-2 7B, LLaMA2 13B, and LLaMA-2 70B (Touvron et al., 2023b), all of which undergo full fine-tuning. We adopt system prompt from (Taori et al., 2023) for finetuning and listed in Appendix B. We use AdamW for optimization. The training proceeds for three epochs with a learning rate of 2e-5, a warmup ratio of 0.03, and a cosine learning rate scheduler. We do not apply early stops to choose checkpoints. The hardware setup involves 32 NVIDIA A100 GPUs.

heading: 3.2 Dataset Augmentation
text: Query Augmentation To generate new queries for GSM8K, we use gpt-3.5-turbo-0613 and gpt-4-0613 as the generator. Inspired by EvolInstruct (Luo et al., 2023b,a), we find that the diversity and complexity of queries in augmented datasets play a vital role in improving math reasoning benchmark performance. We employ human knowledge from authors in modifying mathematical problems for query augmentation. Below are five query augmentation methods used in our experiments: Change specific numbers; Introduce fractions or percentages; Combine multiple concepts; Include a conditional statement; Increase the complexity of the problem. The examples and detailed prompts we used for query augmentation are listed in Appendix C. The examples of augmented queries are shown in Table 10.
Response Augmentation Instead of using the trained SFT model proposed by Yuan et al., we use gpt-3.5-turbo-0613 and gpt-4-0613 to augment more reasoning paths . The main reason is that we can not filter out wrong reasoning paths without final answers. Thus we need to use a model that is as accurate as possible which is the state-ofthe-art LLMs ChatGPT. We use a 1-shot prompt to ensure augmented response formats. The response prompt we used for query augmentation is listed in Appendix D. Augmented responses can result in some unconventional answers, such as excessively long reasoning paths and reasoning paths that do not contain an answer at their end. We devise manual rules to filter out these corresponding query-response pairs and manual rules are detailed in Appendix F. The examples of augmented responses are shown in Table 11.
Augmented Dataset The original GSM8K training set has 7,473 samples. We augment 5 more queries for each query in the training set and yield 7, 473× 5 = 37, 365 augmented queries. We run this query augmentation three times with D1,D3 by GPT-3.5 and D2 by GPT-4, and ∥Di∥ = 37, 365. Then we generate one response for each augmented query for Di and apply response filtering. We consider the query-response pairs after filtering as Dji . We obtain approximately 30,000 query-response pairs for each Dji . To explore the performance differences of different augmented settings, we generate five responses on the augmented queries D1 with GPT-4’s temperature set to 1.0 (D11 ∼ D51),
one response with GPT-4’s temperature set to 0.0 (D61), and one response with GPT-3.5’s temperature set to 1.0 (D71). We also try a zero-shot response generation named D81. We use GPT-4 to augment responses as D12,D13. Since D12 is significantly larger than other subsets, we downsample it to D̂12. We refer to the union of all augmented data and the original GSM8K training set as AugGSM8K, upon which we conduct experiments using various subsets. Detailed augmented dataset notations are listed in Table 1.

heading: 3.3 RQ1. What Strategies of Data Augmentation are More Effective
text: Query Augmentation Types We want to examine whether query augmentation works for math reasoning SFT since (Luo et al., 2023a) applies PPO which cannot make an apple-to-apple comparison. Each query in the original training dataset is augmented with 5 different types. We cluster these queries based on the query types. We apply SFT on the original training set (D), each query type augmentation, and a combination of them (D+D11). Results are shown in Figure 1 and Table 20, where mixed-augmentation represents 1/5 of the aug-
mented data of each strategy being randomly sampled and then combined. Compared with no augment, each query augment method can improve the in-domain performance. Increase complexity augmentation method improves most among all of them. This suggests that enhancing the complexity of queries is one of the key factors influencing the sample efficiency of data augmentation. We randomly sampled one-fifth of the combined dataset and named it Mixed-augmentation. We can observe that the Mixed-augmentation method exhibits performance marginally lower than that of the best Increase Complexity approach. However, it can be seen from Table 16 that with larger data size, mixed-augmentation can reach the best performance. Therefore, it is necessary to mix different types of queries to improve the diversity of augmented questions. There are more discussions in Appendix I about improving existing data augmentation methods.
Query and Response Sources Here we examine how the query and response quality influence the augmented model performance. We list results in Table 2, and draw the following conclusions: (a) Comparing D̂12 and D61, we find that the queries generated by GPT-4 and GPT-3.5 have no significant impact on SFT performance. (b) Comparing D11 and D61, we can conclude that when using GPT-4 to generate responses, the temperature has no significant impact on SFT performance. (c) Comparing D11 and D81, we can conclude that, compared to the zero-shots generation method, the response augmentation prompt we propose plays a substantial role in enhancing the quality of the generated data (+3.6 for LLaMA-7B, +3.7 for LLaMA-2-7B, +3.3 for LLaMA-2-13B). The main reason we consider this is our 1-shot setting stabilizes the response format. (d) Comparing D71 (25K) and D11 × 0.8 (24K), we can conclude that, compared to GPT-3.5, the response augmented using GPT-4 yields significantly better results for SFT.

heading: 3.4 RQ2. What is the Scaling Relationship between the Amount of Augmented Data and Model Performance
text: Query Augmentation Amount We examine how query augmentation amount affects the indomain performance. We examine seven data volume configurations including partitioning D11 into proportions of 0, 0.2, 0.4, 0.6, 0.8, 1.0, as well as D11 + D̂12 and D11 + D̂12 + D13 as the augmented
datasets. Each augmented query only has one augmented response. They are mixed with GSM8K D to apply SFT. From Table 2 and Table 21, we can find that within the data volume range of 13-97K, the in-domain performance exhibits a log-linear relationship with the query amount. We employ linear regression to approximate this relationship. As shown in Figure 3, pre-training models with better initial math reasoning capabilities exhibit a smaller slope which is consistent with (Yuan et al., 2023a). This suggests it is harder to improve reasoning ability for a better pre-trained model. We also conduct validations on our fitted scaling law with an interpolate point at a query amount of 17K (D+D11 × 0.3) and an extrapolate point at a query amount of 104K (D+D11 +D12 +D13), discovering that the regression offers accurate predictions of model performances. We should notice this scaling law cannot be correct within all dataset size ranges since the test set accuracy is bounded.
Besides, the fitted regression shows when query augmentation amount doubles, LLaMA-7B models will improve 10.7×log(2) = 7.4, LLaMA2-7B will improve 9.8 × log(2) = 6.8, and LLaMA213B will improve 7.6 × log(2) = 5.3. As shown in Yuan et al. (2023a), it is estimated that when human-written sample amount doubles, LLaMA7B models will improve 6.5 score, LLaMA2-7B will improve 6.6 score, and LLaMA2-13B models will improve 5.5 score. Query augmentation is similarly effective to human-written samples in term of in-domain performance. This demonstrates that query augmentation benefits from the performing proprietary LLMs on GSM8K, thus the sample quality generated by query augmentation is as high as those of human-written samples. When it comes to the relationship of model performance and augmented data size, a segmented log-linear relationship is presented. The detailed discussion
is in Appendix A.
Response Augmentation Amount We further investigate under the data augmentation setting, if we keep the number of queries constant and increase the number of responses, how the in-domain performance changes. We use D1 as augmented queries and vary the response amount from 1 to 5 per augmented query. We also try majority voting (Wang et al., 2023a; Huang et al., 2022) to filter the augmented response since we cannot know the correct answer. In Figure 3 and Table 22, we find for LLaMA-7B and LLaMA-2-7B models, once the response data volume reaches 97K (3 responses per query), further increase the number of responses do not yield performance improvement. Before this point, model performance improves as the response amount increases. Thus, query augmentation with accurate responses seems more effective than only response augmentation with the augmented data size scales up. As for the LLaMA-2-13B model, within the data volume range of 37K to 157K, model performance consistently rises in a roughly linear fashion with increasing data volume, with a slower rate than the 7B model within the ascending interval. We then investigate the performance impact brought by majority voting filtering. If all responses have different answers, we discard the corresponding query-response. Surprisingly, we find that after applying majority voting, the model performance at the same data scale is generally lower than not applying filtering. A possible explanation is that even wrong responses generated by GPT-4 are useful for worse models (LLaMA) to improve their abilities. Another explanation is the reduction in the number of queries, as we discard
the corresponding query when all response answers are different. To study the relationship between response quality and model performance, we discuss more in Appendix J.
Combination We investigate how the combination of query augmentation and response augmentation in AugGSM8K will affect the model’s performance. Results are listed in Table 4. It demonstrates that query augmentation and response augmentation can complement each other to a certain extent to improve in-domain performance. We conduct SFT on AugGSM8K and AugMATH to obtain MuggleMath which effectively improves the in-domain accuracy on GSM8K and MATH test set in Table 4 and Table 5. These models outperform previous state-of-the-art open-sourced models with a very large margin for 7B and 13B models. Model comparisons of MuggleMath and a broader range of state-of-the-art approaches are in Table 26. Case studies of MuggleMath are listed in Table 12. Moreover, our work suggests that high-quality math training data can be synthesized through powerful large language models by relying solely on simple prompt, without the need for complex designs.

heading: 3.5 RQ3. Can Data Augmentation Incentivize Generalization to Out-of-domain Mathematical Reasoning Tasks?
text: We have found that query and response augmentation significantly improves in-domain math reasoning performance. But we really interested in whether we can improve performances on out-ofdomain distribution. We employ multi-task learning and transfer learning to see how models fine-
.
tuned on AugGSM8K perform on the MATH test set, where we sample 500 questions as test test like (Lightman et al., 2023b). We list results in Table 6. We find that (1) Multi-task learning and transfer learning outperform single-task supervised fine-tuning on LLaMA2-7/13B and do not improve on LLaMA-7B. (2) Although augmenting more query and response can improve GSM8K significantly, it has little to no help in improving MATH performance which indicates that in-domain augmentation data on the GSM8K dataset yields only marginal benefits for the MATH dataset in this setting. Case studies of models performed on MATH are listed in Table 13.
To further investigate why AugGSM8K helps little on the MATH dataset, we use t-SNE in Figure 4 to visualize the hidden representation of problems encoded by LLaMA2-7B, that is the 15-th layer of last token representation of the problem . We find GSM8K and MATH are separated in hidden space and only some of the problems in MATH are laid in the span of GSM8K. The augmented GSM8K problems are laid in the same span of GSM8K which makes sense why it improves little for MATH.
To investiate if the proposed augmentation method improve performance on this subset of MATH, we find that while there is an overlap in the embedding space distribution between MATH and GSM8K, it is relatively small compared with that between GSM8K and AugGSM8K. In the transfer learning setting, training first on GSM8K and then on MATH with LLaMA-13B-2 does provide some benefits for certain subsets, such as Prealgebra, Algebra, and Geometry. However, if we train first on AugGSM8K and then on MATH, this benefit is not only marginal but may even lead to a decrease in performance on other subsets, like Geometry and Prealgebra, which could be related to the data proportions. Overall, the performance improvement on the MATH dataset from augmentation on GSM8K is minimal, even on subsets like Prealgebra, where there is some overlap. For 7B size and multi-task learning setting, we can draw the similar conclusion. Detailed resulsts are listed in Table 28 to Table 33. This suggests if we want to improve math reasoning benchmark performances for LLMs, we can choose to apply augmentation on diverse math subjects. For the generalization from AugMATH to GSM8K, the detailed experiments
and explanation are in Appendix A. Moreover, we conduct experiments on more out-of-distribution mathematical dataset in Appendix A.3.

heading: 4 Discussion

heading: 4.1 Training set vs. Test set accuracy
text: Query and response augmentation generate similar problems for the training set which leads to better training set accuracy. We have shown augmentations improve the accuracy of the in-domain test set. We want to investigate the relationship between the accuracy of the training set and the test set to find if the accuracy of the training set can be a performance indicator. We sample 500 samples from the original training set to calculate the accuracy. From Figure 8, the training and test accuracy generally exhibit a positive correlation across different augmented data which shows the training accuracy could be an indicator of in-domain performance unless deliberately overfitting.

heading: 4.2 Make more augmentation on harder problems
text: During the query augmentation process, it is crucial to understand which kind of queries should be augmented. Augmenting too many easy problems may not be effective since the model may have mastered this level of problems. Here we examine if the model improves more when we augment more on harder or wrong problems. We define hard problems based on the number of equations, specifically, problems with fewer than three reasoning steps as easy, those with exactly three steps as medium, and those with more than three steps as hard(see more details in Appendix H). We define wrong problems
if the SFT model solves them incorrectly. We apply SFT on subsets of AugGSM8K with augmented queries on easy, medium, hard, wrong, and random problems. From Figure 5 and Figure 24, it is evident that for LLaMA-2-7B and LLaMA-2-13B, the performance gain from augmenting hard problems is significantly higher than that from augmenting other types of problems. From Figure 6 and Figure 25, we find that augmenting incorrect problems on three models consistently improves more than random query augmentation. In addition, we have conducted an analysis of our model’s performance on test set problems of varying difficulty. Our analysis of the 7B-2 model’s performance on the test set, shows accuracy rates of 0.55, 0.42, and 0.21 for easy, medium, and hard problems, respectively, while MuggleMath achieves higher accuracy rates of 0.73, 0.70, and 0.64 for the same problem categories. This significant performance boost on difficult questions can be attributed to the fact that the augmented problems we generated are generally more complex than the original problems.

heading: 4.3 Result on the Perturbed Test Set
text: We have perturbed two new test sets based on the original GSM8K test set, Change-Test and AugTest in in Table 27.
Upon evaluating our model on these two perturbed test sets, we found that the performance of MuggleMath consistently and significantly exceeds that of the model fine-tuned on GSM8K alone. This observation suggests that our data augmentation techniques not only enhance the model’s ability to solve the original problems but also contribute to its improved performance on varied and perturbed inputs, thereby indicating a robust generalization capability in in-domain dataset.

heading: 5 Conclusion
text: In this paper, we investigate the scaling property of query and response augmentation with respect to math reasoning in-domain and out-of-domain performance. We find that query and response augmentation can improve in-domain performance very effectively which has a similar improvement to human-written query-response pairs augmentation. Although we can obtain state-of-the-art indomain performance by such augmentation, We find that the math reasoning ability improved by the in-domain augmented data is hard to generalized to out-of-domain datasets. Therefore, when using synthetic data, increasing the diversity of the queries to improve the overall mathematical reasoning ability of the LLMs is crucial. In the future, how to use synthetic data to enhance the overall mathematical reasoning ability of the model is a very important research topic.
Limitations
In this study, we focused on the domain of mathematical reasoning—a key area of interest for the large language model (LLM) research community. We investigated the effectiveness of data augmentation techniques on both in-domain and out-ofdomain performance. While our work provides insights into the performance scalability and generalizability of Chain-of-Thought (COT) enhanced models in mathematical reasoning, it’s important to acknowledge certain limitations: (1)Depth of Generalizability Research: Although our study is among the initial efforts to evaluate the generalizability of COT-augmented models in math reasoning, the study did not extensively explore solutions to enhance out-of-domain performance across a broader spectrum of mathematical reasoning. (2)Unique Data Augmentation Process: We utilized generative models, specifically gpt-3.5turbo-0613 and gpt-4-0613, for data augmentation, which means that others using the same prompts may not be able to replicate our exact dataset. However, by varying the generation models’ temperature setting, we have verified the robustness of our methodology. This suggests that even if the data augmentation process were to be repeated, similar performance enhancements could be achieved, underscoring the reproducibility of the results despite the unique nature of the data generation.

heading: 6 Acknowledgement
text: We thank all anonymous reviewers for their helpful comments and suggestions. This research is supported by the National Science and Technology Major Project (2023ZD0121102) and National Natural Science Foundation of China (92270114).

heading: A Augmentation on MATH: AugMATH
text: A.1 Query and response augmentation on MATH
We conducted simple query and response enhancements for MATH whose training set has 7,500 problems. Each query in the MATH dataset was rewritten in five different ways, and we used GPT-4 to sample each variation eight times with a temperature setting of 0.7. Ultimately, we generated 7, 500 × 5 × 8 = 300, 000 synthetic MATH data points, Considering that MATH encompasses a broader range of topics—such as elementary calculus and geometry—rather than mainly algebraic problems like GSM8K, we used our GSM8K query augmentation method as a guide and allowed GPT4 to reference our approach during the query augmentation process without being strictly confined to those five methods. Details on the augmentation prompts can be found in the Appendix E. Since the MATH dataset comprises 7,500 questions that include seven subjects—Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus—we can decompose MATH into seven subsets each covering different mathematical subjects. Therefore, our augmented dataset, AugMATH, has already undergone augmentation across various mathematical topics. The results on LLaMA-2-7B are listed in Table 7.
A.2 Relationship of performance and augmented data size
We conduct SFT of LLaMA-2-7B on subsets of AugMATH, the results are evaluated on MATH test set with 5,000 problems.
We observe from Table 8 a segmented log-linear relationship in the data size’s impact on the model’s performance. This relationship is evident when the data volume is either less than 37.5k or greater than 82.5k. We conduct log-linear fittings for these two distinct segments separately, denoting x as the data size in thousands and y as the accuracy on the MATH test set. In the range of 7.5k to 37.5k, the log-linear relationship can be fitted as y = 1.47+2.45 log(x), with coefficient of determination 0.992. In the range of 82.5k to 307.5k, it can be fitted as y = −13.56 + 6.33 log(x), with coefficient of determination 0.985. We can see that once the data volume reaches a certain threshold, the performance gains from doubling the data become much more substantial.
A.3 Generalizability of AugMATH to GSM8K
From Table 8, we observe that as the volume of AugMATH data increases, the performance on GSM8K does not continuously rise but rather exhibits significant fluctuations. However, on the whole, when the full AugMATH dataset (300K) is utilized, the fine-tuned performance on GSM8K improves from 40.3 to 45.4. This increase is relatively modest, especially considering that just 30K of AugGSM8K can elevate the model’s performance on GSM8K to a score of 58.
In summary, we can see that our proposed simple data augmentation method is equally effective on the MATH dataset, but exhibits a piecewise logarithmic-linear property; although it appears that augmentation on MATH provides certain gains for GSM8K, these gains are very slight relative to the amount of data augmented.
A.4 Generalizability on other OOD datasets
To further illustrate generalizability, we conducted tests on five additional datasets: GSM-Hard (Gao et al.), SVAMP (Patel et al., 2021), TabMWP (Lu et al., 2023), ASDiv (Miao et al., 2020), and MAWPS (Koncel-Kedziorski et al., 2016).
It can be seen from Table 9 that MuggleMATH not only outperforms the base model and SFT model on other datasets, but also exhibits a significant performance advantage compared to the state-of-the-art augmentation method, WizardMath. Overall, by performing augmentation on both GSM8K and MATH, we can obtain robust capabilities across a wide range of datasets. The reason behind may be that these out-of-ditribution datasets are actually similar to the distribution of GSM8K and are not so out-of-ditribution.
B Instruction prompt for training and inference
Here is the instruction prompt used for the training and inference stage.
Fine-tuning system prompt
Below is an instruction that describes a task. Write a response that appropriately completes the request.### Instruction: **Query.** ### Response:

heading: C Query augmentation prompt for GSM8K
text: Here is the query augmentation prompt we use for GSM8K. We require the models to generate five different augmented problems with our provided example. We use gpt-3.5-turbo-0613 and gpt-4-0613 APIs with a temperature of 1.0 to obtain augmented problems.
Query augmentation prompt
I want you to act as a math teacher. I will provide a grade school math question and you will help to to create more challenging math questions by given ways. Given the question: “James writes a 3-page letter to 2 different friends twice a week. How many pages does he write a year?”, you will modify it by following ideas: 1. Change specific numbers: James writes
a 2-page letter to 2 different friends 3 times a week. How many pages does he write in 4 years? 2. Introduce fractions or percentages: James writes a 3-page letter to 2 different friends twice a week. Each week, he adds 50% more pages to each letter. How many pages does he write in a month? 3. Combine multiple concepts: James writes a 3-page letter to 2 different friends twice a week. He uses both sides of the paper and each side can hold 250 words. If James writes 100 words per minute, how long does it take for him to write all the letters in a week? 4. Include a conditional statement: James writes a 3-page letter to 2 different friends twice a week. If it’s a holiday, he writes an additional 5-page letter to each friend. Considering there are 10 holidays in a year,
how many pages does he write in a year? 5. Increase the complexity of the problem: James writes a 3-page letter to 2 different friends twice a week. In addition, he writes a 5-page letter to 3 other friends once a week. How many pages does he write in a month, assuming there are 4 weeks in a month? Now you are given the question: **A new math problem here.**

heading: D Response augmentation prompt for GSM8K
text: We use this prompt to generate responses to ensure the response format which can be viewed as 1- shot setting. We use gpt-3.5-turbo-0613 and gpt-4-0613 with temperature 0.0 or 1.0.
Response augmentation prompt
I want you to act as an excellent math solver. You will solve the given math question step by step. You need to reply with a python dictionary in the same format as the given examples. Retain decimals to three decimal places. The formulas in the process need to use the format: 48/2 = «48/2=24»24 clips. The end of response needs to be: #### {answer}. Examples: {“query”: “Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?”, “response”: “Natalia sold 48/2 = «48/2=24»24 clips in May.Natalia sold 48+24 = «48+24=72»72 clips altogether in April and May.#### 72”}. The given question: **A new math problem here.**

heading: E Query and response augmentation prompt for MATH
text: We use this prompt to augment MATH dataset and use gpt-4-0613 with temperature 0.0.
Response augmentation prompt
I want you to act as a math teacher. You should think of some ways to help students do variation training for challenging com-
petition mathematics problems. For example, for a question-solution pair, Question0: James writes a 3-page letter to 2 different friends twice a week. How many pages does he write in a year? Solution0: He writes each friend 3× 2 = 6 pages a week. So he writes 6 × 2 = 12 pages every week That means he writes 12 × 52 = 624 pages a year ##end0” we can propose 5 types of variation exercises, and response with: 1. Change specific numbers: Question1: James writes a 5-page letter to 3 different friends 4 times a week. How many pages does he write in 3 years? Solution1: To calculate the total number of pages James writes in 3 years, let’s first figure out how many pages he writes each week and then multiply that by the number of weeks in 3 years. He writes each friend a 5-page letter, so for 3 friends, that’s 5 × 3 = 15 pages per writing session. He writes 4 times a week, so the weekly total is 15 × 4 = 60 pages. There are 52 weeks in a year, so in one year, he writes 60× 52 = 3120 pages. Finally, over the course of 3 years, he writes 3120× 3 = 9360 pages. ##end1 2. Introduce fractions or percentages: Question2: James writes a 3-page letter to 2 different friends twice a week. Each week, he adds 100% more pages to each letter. How many pages does he write in a month? Solution2: Let’s take this step by step: In the first week, James writes a 3-page letter to 2 friends twice a week, which is 3 × 2 × 2 = 12 pages in total for the first week. n the second week, he writes 100% more pages, thus doubling the number of pages in each letter. So he writes 6 × 2 × 2 = 24 pages in total for the second week. In the third week, he again writes double the previous week’s pages, so 12 × 2 × 2 = 48 pages in total for the third week. In the fourth week, the number of pages doubles again, which results in 24×2×2 = 96 pages in total for the fourth week. Now, we’ll add up the pages from all four weeks to find out how many pages he writes in a month: 12+24+48+96 = 180 pages. Therefore, in a month (assuming a 4-week month), James writes 180 pages.
##end2 3. Combine multiple concepts: Question3: James writes a 3-page letter to 2 different friends twice a week. He uses both sides of the paper, and each side can hold 250 words. If James writes at a speed of 100 words per minute, how long does it take him to write all the letters in a week? Solution3: To find out how long it takes James to write all the letters in a week, we first calculate how many words he writes in total. Each letter is 3 pages long, and he writes to 2 friends, which is 3× 2 = 6 pages per writing session. Since he writes twice a week, the total number of pages per week is 6 × 2 = 12 pages. Considering each page has two sides and each side holds 250 words, the number of words on one page is 250× 2 = 500 words. Therefore, the total number of words James writes in a week is 500 × 12 = 6000 words. Given James writes at a speed of 100 words per minute, the time it takes him to write all the letters in a week is calculated by dividing the total number of words by his writing speed: 6000 words÷100 words/minute = 60 minutes. So, James takes 60 minutes to write all the letters in a week. ##end3 4. Include a conditional statement: Question4: XX Solution4: XX ##end4 5. Increase the complexity of the problem: Question5: XX Solution5: XX ##end5 Now, find five suitable variation training methods for the new problem. Be careful not to let existing methods limit your thinking. Instead, propose variation training methods that are specifically tailored to the given problem: Question0: **A new math problem here.** Solution0: **corresponding solution here here.** Please response with the given example format(including Questions and solutions)

heading: F Response filter
text: We filter out generated responses by following rules.
• Delete the responses without an answer.
• Delete the responses that are excessively lengthy(> 1500).
• Remove superfluous characters beyond the reasoning path and the answer.

heading: G Case Study of GSM8K
text: There are examples of different methods for generating new queries in Table 10 and different reasoning paths for the same query in Table 11. Some examples of MuggleMath-13B answering questions from the GSM8K test set are in Table 12 and wrong reasoning processes are labeled in red.

heading: H Difficulty level definition on GSM8K
text: We conducted a statistical analysis of the reasoning paths required for 7,473 questions in the GSM8K training set, categorizing them as hard, medium, and easy. Specifically, we defined questions with more than three formulas, exactly three formulas, and less than three formulas as hard, medium, and easy, respectively. This categorization yielded a balanced distribution with 2,357 easy, 2,360 medium, and 2,756 hard problems. This approach ensures a relatively equal number of problems in each category.

heading: I Discussion of augmentation methods for GSM8K
text: I.1 How to improve existing data augmentation methods?
If one wants to further improve using these ideas, there are two relatively direct approaches to enhance our current methodology.
• The first approach involves thinking of additional methods beyond the initial five. Although it is impossible to exhaust all the methods of rewriting queries, it is relatively easy to propose some similar rewriting approaches and examples, akin to the five schemes we have suggested. For example, "Introduce unknown quantities", "Introduce a range", "Finding averages" may be reasonable. Limited by resources, we only employ "Introduce unknown quantities" to expand the dataset.
• The second approach is to augment the queries we have expanded again. We conducted further enhancement on "mixed-augmentation" while maintaining the same data volume. We called it "Mixed-augmentation-second".
Taking the prompt in the question as an instance, the original problem is: "James writes a 3-page letter to 2 different friends twice a week. How many pages does he write a year?" Our five methods are as follows:
• Change specific numbers: James writes a 2- page letter to 2 different friends 3 times a week. How many pages does he write in 4 years?
• Introduce fractions or percentages: James writes a 3-page letter to 2 different friends twice a week. Each week, he adds 50% more pages to each letter. How many pages does he write in a month?
• Combine multiple concepts: James writes a 3-page letter to 2 different friends twice a week. He uses both sides of the paper, and each side can hold 250 words. If James writes 100 words per minute, how long does it take for him to write all the letters in a week?
• Include a conditional statement: James writes a 3-page letter to 2 different friends twice a week. If it’s a holiday, he writes an additional
5-page letter to each friend. Considering there are 10 holidays in a year, how many pages does he write in a year?
• Increase the complexity of the problem: James writes a 3-page letter to 2 different friends twice a week. In addition, he writes a 5-page letter to 3 other friends once a week. How many pages does he write in a month, assuming there are 4 weeks in a month?
In addition to these, we can conceive of other methods:
• Introduce unknown quantities: James writes some pages to his friends every week. After 4 years, he has written 1,440 pages. How many pages does James write each week?
• Introduce a range: James writes about 150- 200 words per page. If he writes a 4-page letter, what is the maximum and minimum number of total words he writes?
• Finding Averages: James wrote 5 letters last week. The number of pages were 2, 3, 4, 3, and 2, respectively. What was the average number of pages per letter?
We can observe that the "Introduce Unknown Quantities" strategy is essentially as effective as the other five methods from Table 14. Continuing to augment the expanded queries, however, yields slightly less effective results compared to the original five methods. A possible reason for this could be that the problems created through secondary augmentation are too complex, resulting in a lower accuracy rate in the responses provided by GPT-4, thereby impacting the efficacy of the augmentation.
I.2 Will mixed-augmentation has advantages over increase the complexity if we enlarge the size of data?
Increasing the dataset size to observe whether the mixed-augmentation approach can yield better results due to its diverse enhancements is indeed necessary. We categorized the data of different query augmentations in D11 , D12 ,D13 and performed random sampling from these mixed datasets to obtain a comparable quantity of data. We conduct a SFT on LLaMA-2-7B, the results are list in Table 15.

heading: J The effect of augmentation response
text: quality of GSM8K
J.1 Under what circumstances can wrong answers have a positive effect?
To investigate the effect of augmentation on response quality, we aim to collect datasets with responses that are uniformly incorrect, uniformly correct, and partially correct. We amalgamate D11 , D21 ,D31,D41 , D51 ,D61,D71, and categorize the combined dataset into three groups:
• The first category consists of cases where all seven responses are the same.
• The second category includes instances where each of the seven responses is distinct.
• The third category encompasses queries with 2 to 6 varied answers.
To mitigate the influence of the query, we eliminate the first and second categories and focus on the third category, where we vote on the answers. The answer with the majority of votes is deemed the correct response, while the others are classified as incorrect. We randomly select one correct and one incorrect response for each query from the third category to form the sets Dcorrct and Dwrong , ensuring both sets contain identical queries. Subsequently, we randomly sample half of the queries from Dcorrct and pair the remaining queries from Dwrong to create Dhalf , which shares the same queries as Dcorrct and Dwrong in Table 16. Although the estimated accuracy may not be precise, it serves to indicate the trend.
We conduct Supervised Fine-Tuning (SFT) on these three datasets on two models with varying accuracy: LLaMA-2-7B-SFT (41.6 on GSM8K test set), which is fine-tuned from the GSM8K training set, and MuggleMATH-7B (68.4 on GSM8K test set), which is fine-tuned from five subsets of AugGSM8K. The results are as follows:
From Table 16, we can draw some conclusions: (1)For LLaMA-2-7B-SFT, the higher the accuracy of the fine-tuned dataset, the more substantial the performance gain. There is an improvement of 4.5 percentage points even when fine-tuned on a dataset consisting solely of incorrect responses. This may be attributable to the presence of correct reasoning steps within the responses with wrong answers, as mentioned in (Lightman et al., 2023b). (2)For MuggleMATH-7B, the lower the accuracy
of the dataset, the more significant the performance degradation.
In summary, erroneous data still contributes to performance improvement for models with poorer performance, but it has a detrimental effect on models with superior performance.
J.2 The relationship between response quality and data volume.
To delve deeper into the relationship between the quality and quantity of augmented data, we partition Dcorrct ,Dwrong and Dhalf into different fractions and conduct Supervised Fine-Tuning (SFT) on LLaMA-2-7B-SFT (41.6 on GSM8K test set).
From these results in table 17, we can draw some insightful conclusions: (1)As the volume of data is increased, even datasets of poor quality can still yield performance gains for LLaMA-2-7BSFT. (2)The higher the quality of the responses, the more substantial the performance improvements achieved with each doubling of the data volume.
J.3 The Accuracy and difficulty of different query augmentation categories
In an effort to discern the performance disparities among various query augmentation techniques, we evaluate the accuracy of responses across different categories. By employing majority voting to determine the reference answer from D11, D21, D31, D41, D51, D61 and D71 , we consider queries that yield seven distinct answers as incorrect. Additionally, we compute the average number of reasoning steps for each augmentation method—a formula such as "30+90 = «30+90=120»120" is counted as a single step.
From the Table 18, we observe a general tradeoff between the number of reasoning steps and accuracy. As corroborated in section 4.2, the augmentation yields greater benefits for more challenging problems. Notably, the "Increase Complexity" method, while exhibiting the lowest dataset accuracy, involves the highest number of reasoning steps. This suggests that augmenting with more complex problems can lead to more substantial benefits for the model.

heading: K A more detailed comparison of GSM8K, MATH, AugGSM8K, and AugMATH
text: In GSM8K and AugGSM8K, a formula such as "30+90 = «30+90=120»120" is counted as a single step.
Finding a reasonable way to calculate reasoning steps is challenging in the MATH and AugMATH datasets, as the answers in the MATH dataset are in LaTeX format and vary greatly. We consider one sentence in the response (ending with a period or semicolon) as one step of reasoning. An interesting finding is that the number of reasoning steps in AugMATH is significantly lower than in MATH. One possible explanation could be an issue of linguistic style; GPT-4 and humans might use a different number of sentences for the same number of reasoning steps. Therefore, the reasoning steps in MATH are difficult to compare. The second reason may be that GPT-4 has only a 42% accuracy rate on MATH problems, and on the more challenging MuggleMATH, it provides incorrect responses for most questions. These guys incorrect responses may have fewer reasoning steps than are actually required for the problems.
The reason why a model fails to generalize from one dataset to another after data augmentation might be influenced by many factors. From the perspective of human learning, no matter how much one studies elementary school mathematics, it would be quite difficult to solve high school math problems. Therefore, we believe that simply performing data augmentation during the SFT (Supervised Fine-Tuning) phase to increase accuracy on a dataset, while it may yield significant performance gains, might not lead to as substantial improvements for large models across a wide variety of reasoning tasks as it may appear.

heading: L Detailed Experimental results
text: We list the detailed experimental results of different settings here.

heading: Future Work
text: In the future, how to use synthetic data to enhance the overall mathematical reasoning ability of the model is a very important research topic. Limitations
In this study, we focused on the domain of mathematical reasoning—a key area of interest for the large language model (LLM) research community. We investigated the effectiveness of data augmentation techniques on both in-domain and out-ofdomain performance. While our work provides insights into the performance scalability and generalizability of Chain-of-Thought (COT) enhanced models in mathematical reasoning, it’s important to acknowledge certain limitations: (1)Depth of Generalizability Research: Although our study is among the initial efforts to evaluate the generalizability of COT-augmented models in math reasoning, the study did not extensively explore solutions to enhance out-of-domain performance across a broader spectrum of mathematical reasoning. (2)Unique Data Augmentation Process: We utilized generative models, specifically gpt-3. 5turbo-0613 and gpt-4-0613, for data augmentation, which means that others using the same prompts may not be able to replicate our exact dataset. However, by varying the generation models’ temperature setting, we have verified the robustness of our methodology. This suggests that even if the data augmentation process were to be repeated, similar performance enhancements could be achieved, underscoring the reproducibility of the results despite the unique nature of the data generation