JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models

Authors: kun  zhou
beichen  zhang
jiapeng  wang
zhipeng  chen
wayne xin zhao
jing  sha
zhichao  sheng
shijin  wang
ji-rong  wen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:31:01

Abstract: Mathematical reasoning is an important capability of large language models~(LLMs) for real-world applications.
To enhance this capability, existing work either collects large-scale math-related texts for pre-training, or relies on stronger LLMs (\eg GPT-4) to synthesize massive math problems. Both types of work generally lead to large costs in training or synthesis.
To reduce the cost, based on open-source available texts, we propose an efficient way that trains a small LLM for math problem synthesis, to efficiently generate sufficient high-quality pre-training data.
To achieve it, we create a dataset using GPT-4 to distill its data synthesis capability into the small LLM.
Concretely, we craft a set of prompts based on human education stages to guide GPT-4, to synthesize problems covering diverse math knowledge and difficulty levels.
Besides, we adopt the gradient-based influence estimation method to select the most valuable math-related texts.
The both are fed into GPT-4 for creating the knowledge distillation dataset to train the small LLM.
We leverage it to synthesize 6 million math problems for pre-training our JiuZhang3.0 model. The whole process only needs to invoke GPT-4 API 9.3k times and use 4.6B data for training.
Experimental results have shown that JiuZhang3.0 achieves state-of-the-art performance on several mathematical reasoning datasets, under both natural language reasoning and tool manipulation settings.
Our code and data will be publicly released in \url{https://github.com/RUCAIBox/JiuZhang3.0}.

Main paper: 
 introduction: 
 Large language models (LLMs) have shown remarkable capabilities on a variety of tasks [1] [2] [3] . However, they still struggle in solving complex mathematical problems [4] . Recent work has shown that it is an effective approach to training LLMs on math-related data for improving the mathematical reasoning ability [5, 6] . Typically, they either collect the math-related data from the available corpora (e.g., webpages and books) for pre-training [7] [8] [9] , or rely on stronger LLMs to synthesize high-quality math problems for fine-tuning [10] [11] [12] . Despite the success, existing approaches would generally cause large training or inference costs. Due to the complexity and diversity of mathematical problems, the former type of work mostly needs to collect a large-scale corpus (e.g., 120B data for Deepseek-Math) for training, which greatly increases the training cost [5, 7, 8] . Similarly, to guarantee the knowledge coverage and effectiveness of the synthetic problems, the latter type of work relies on stronger LLMs with larger scales (e.g., GPT-4) to create massive math problems, leading to larger inference cost [6, 10, 11] . In Figure 1 , we show our estimated total costs of re-implementing two math-related LLMs (>$40000), details are in Appendix A Expense ($) In this work, we aim to develop a relatively lowcost data synthesis approach for improving the mathematical reasoning abilities of LLMs. Our key idea is that the data synthesis capability can be well learned by small LLMs. Here, small is a relative wording, which is in contrast with the extremely large or costly data synthesis models used in prior studies [6, 13] , such as GPT-4 or Qwen-72B. Actually, existing work [10, 14, 15] has extensive evidence of strong learning and adaptation abilities of small LLMs for new tasks and domains with suitable strategies (e.g., training with high-quality supervised data), including math, science, and complex multimodal tasks. However, this exploration has been neglected in prior efforts on data synthesis. This attempt can be essentially generalized to a broader problem: whether a small (or weak) model can produce high-quality data that is useful for training a large (or strong) model? Inspired by this motivation, we seek to train a relatively small yet powerful LLM for synthesizing high-quality math-related data.However, due to the diverse and complex nature of math problems, it is challenging to train wellperforming small LLMs for synthesizing high-quality ones. Although we can leverage GPT-4, it is not efficient to use it for synthesizing a large-scale knowledge distillation (KD) dataset. Specifically, we aim to build the KD dataset through a low-cost strategy, but it can sufficiently capture diverse and useful knowledge about math problem synthesis. Thus, we should guarantee the knowledge coverage and usefulness of the instances within the KD dataset. To achieve it, we first craft a set of prompts, and each prompt corresponds to an education stage of humans, e.g., middle school and college. Using the above prompts, the instances within the dataset can well cover broad mathematical knowledge and different difficulty levels. Besides, for usefulness, we estimate the influence of the available math-related texts and downstream tasks, by computing the gradient similarity between their corresponding synthetic data and the task instances. Then, we select the top-ranking math-related texts into the KD dataset, which are high-value ones with more positive influence on downstream tasks. By feeding the texts with prompts into GPT-4, we collect the outputs to build the KD dataset.Based on the KD dataset, we train DeepSeekMath-7B as our data synthesis model, which is much smaller than other commonly-used LLMs in existing work [6, 12, 13] , e.g., GPT-4 and Qwen-72B. Owing to our data selection strategy, we only require GPT-4 to generate 9,335 instances based on the selected most valuable texts for training it. Then, we utilize the crafted prompts to guide it for synthesizing high-quality problems. Benefiting from the strong data synthesis capability of the small model, we only need to synthesize 5,984,584 high-quality math problems (4.6B tokens) for pre-training our JiuZhang3.0. Thus, the total inference and training cost is much less than existing work, as shown in Figure 1 . After pre-training, we also collect open-source math instructions to fine-tune JiuZhang3.0. The experimental results have shown that JiuZhang3.0 can mostly outperform state-of-the-art methods across 18 evaluation datasets, in both the natural language reasoning and tool manipulation settings. Our contributions are summarized as follows:(1) our research provides compelling evidence that it is feasible to efficiently train a small LLM (7B) for synthesizing training data to improve the mathematical reasoning of LLMs. As results shown in Section 4.3, its synthetic data is more useful than larger LLMs in improving the performance.(2) we propose an efficient solution for training LLMs to improve mathematical reasoning, which only needs to invoke GPT-4 API 9.3k times and pre-train on 4.6B high-quality synthetic data, with nearly 20% total cost of existing state-of-the-art methods.(3) JiuZhang3.0 achieves state-of-the-art performance among open-source LLMs on several tasks and settings, e.g., 52.8 (JiuZhang3.0-7B) vs. 50.2 (DeepSeekMath-7B-RL) on MATH, 89.8 (JiuZhang3.0-8×7B) vs. 86.4 (MAmmoTH2-8×7B-Plus) on GSM8k in the natural language reasoning setting. 
related work: 
 Large Language Models. LLMs have demonstrated remarkable capabilities in a variety of NLP tasks, and commercial LLMs like ChatGPT, Claude, and Gemini [2, 16, 17] , represent cutting-edge capabilities. Meanwhile, the performance of open-source models (e.g., LLaMA-3, Mixtral) has also developed rapidly [18, 19] . To further improve the capability of LLMs on special tasks or domains, existing work mainly focuses on optimizing the following aspects: (1) prompt engineerings such as chain-of-thought and tree-of-thought [20, 21] ;(2) continual pre-training on a domain-specific or task-specific corpus, improving the model to deal with downstream tasks [5, 7, 8, 22, 23] ;(3) supervised fine-tuning, which involves fine-tuning the model on related instruction datasets, enhancing LLMs to follow special task instructions [24, 25] ; (4) other strategies including RLHF [26] , tool augmentation [27] , decoding optimization [28] and et al. We aim to efficiently improve the capability of LLMs for mathematical reasoning by pre-training on synthetic data.Mathematical Reasoning. Despite the impressive progress, mathematical reasoning remains a weak aspect of LLMs. To enhance LLMs' ability in mathematical reasoning, researchers have proposed a surge of methods from the aspects of prompting, pre-training and fine-tuning. For prompting, the chain-of-thought (CoT) prompts have been widely used to guide LLMs for performing multi-step reasoning on complex math problems [20] . Based on it, following work utilizes tools [27, [29] [30] [31] [32] [33] and verifiers [9, [34] [35] [36] , to further improve the accuracy of the mathematical reasoning process. For pre-training, existing work [5, 7, 9, 37, 38] collects a large-scale math-related corpus and continually pre-training open-source LLMs on it. Supervised fine-tuning methods focus on using relatively less high-quality data for training the LLM, which are typically math-related instructions [10, 12, 39] . Recent studies show that the complexity of mathematical reasoning demands high-quality instruction pairs, leading to reliance on advanced LLMs like GPT-4 for data synthesis [6, 40, 41] .The pre-training and fine-tuning methods generally lead to large training and data annotation costs, respectively. Our work aims to train a small LLM specially for math problem synthesis, which can efficiently produce sufficient data for training.Data Synthesis. For complex tasks and scenarios (e.g., mathematical reasoning), it is necessary to collect a substantial amount of data for training the LLM to enhance it. However, the available data may not be sufficient, hence researchers have explored using automatically synthetic data with consistent distribution to real data, to enrich the training corpus [42] [43] [44] [45] [46] [47] [48] . For data synthesis on mathematical reasoning tasks, existing work can be roughly categorized into the following two types, according to their based guided information. The first type of work starts with existing problems or math-related texts to synthesize similar problems or solutions [10, 40, 41, 49, 50] . The other type of work relies on available knowledge points, and devises special prompts to guide LLMs for synthesizing related problems with the solutions [6, 51] . As correctness is important, the two types of work generally design rules to check and remove wrong ones. In this work, based on the data synthesis model, we also construct the multi-source math corpora, and craft several prompts to guide it in producing diverse and useful math problems. 
approach: 
 In this section, we present our approach that aims to train a small LLM for synthesizing math problems. First, we initialize the data synthesis model by training it on the KD dataset, composed of crafted prompts, randomly sampled math-related texts, and the corresponding synthetic problems and solutions from GPT-4. Then, we improve its data synthesis capability by retraining it on the updated knowledge distillation dataset, where we add the high-value math-related texts selected by gradient-based influence estimation strategy. Finally, we utilize the model to synthesize massive high-quality math problems for training JiuZhang3.0, based on the multi-source math-related corpus.We focus on training a small data synthesis LLM, for synthesizing high-quality math problemsolution pairs to pre-train LLMs and improve its mathematical reasoning capability. To guarantee the quality of the synthetic data, we utilize GPT-4 to create the knowledge distillation (KD) datasetD KD = {p i , t i , qi , ŝi } N i=1for training the small LLM, where the math-related text t i and the prompt The pipeline of our approach. We first initialize the data synthesis LLM by distilling the knowledge from GPT-4 on randomly sampled data, then boost it using the high-value data selected by gradient-based value estimation strategy, finally utilize it for synthesizing data to train JiuZhang3.0.p i are the input of GPT-4, qi and ŝi are its synthetic math problem and solution respectively. Then, we train the small LLM on D KD to imitate the data synthesis ability of GPT-4. Finally, we leverage it to synthesize the pre-training dataset D = {q i , s i } M i=1 based on all the collected math-related texts with randomly selected prompts {p i , t i } M i=1 , which are used for training our JiuZhang3.0.In this work, we consider the natural language reasoning and tool manipulation settings, where LLMs require solving the problem by generating a natural language solution [20] and an executable program with external interpreters [52] , respectively. Thus, we train the data synthesis LLMs on the initial KD dataset, containing special prompts, math-related texts, and GPT-4 outputs for the two settings.We aim to craft a prompt set that can well cover the knowledge points and difficulty levels in human math education. Thus, for the natural language reasoning and tool manipulation settings, we manually craft prompt templates respectively, and each corresponds to a certain education stage. Based on the prompts, we build the KD dataset to train our data synthesis LLM. We randomly sample 5,336 math-related texts, and concatenate each one with a randomly selected prompt, to compose the input i.e., [p i ; w i ]. Then, we feed it into GPT-4, and extract the synthetic math problem qi and solution ŝi from its output using regular expressions, to compose the KD dataset D KD = {p i , w i , qi , ŝi } Nini i=1 . Next, we utilize it to train the synthesis model, and the learning objective is:EQUATIONwhere θ syn denotes the parameters of the data synthesis LLM. In this way, we can teach it to generate new math problem-solution pairs based on prompts and math-related texts by imitating GPT-4.After initialization, we further improve the data synthesis LLM using high-value KD data. However, it would be costly if we first utilize GPT-4 to generate candidates and then select high-value ones. Therefore, we propose an efficient way that leverages the data synthesis LLM for generating candidates, and then selects valuable ones to feed into GPT-4. Specifically, we incorporate the gradient-based method [53] to estimate the influence of each synthetic instance for downstream math-related tasks, and select the top-ranking ones to update the KD dataset for retraining the data synthesis LLM.According to the influence formulation [54] , at a certain training step of a model parameterized by θ, the influence of a training instance z on another instance z ′ can be estimated by computing the similarity between their produced gradients, denoted as:EQUATIONBy using it, we can measure the value of each synthetic data by computing its gradient similarity with downstream math-related task data. Concretely, we first train a reference model using LoRA, then compute its gradients on LoRA parameters as the features, to help estimate the data value.Training Reference Model using LoRA. Inspired by existing work [53] , we train a LLM for mathematical problem solving using LoRA [55] as the reference model. As LoRA only requires to optimize the low-rank adapters in the LLM, we can efficiently train the reference model on limited computation resources, and reduce the number of trainable parameters for efficient computation of gradient similarity. Besides, to further reduce the training cost, we randomly select a subset of synthetic math problems generated by the data synthesis model, denoted as D lora = {q i , s i } M l i=1 . Then, we train the reference model to predict the solution based on the given problem, denoted as:EQUATIONwhere θ lora denotes the parameters of LoRA, and M l is the number of training data.After training the reference model, we compute the gradients of LoRA parameters as the feature of each synthetic instance. As their dimension is large, we follow existing work [56] that performs random projection to obtain the low-dimensional features as:EQUATIONwhere z = ⟨p i , w i , q i , s i ⟩ denotes a synthetic instance, Π ∈ R d ′ ×d is a projection matrix initialized by the Rademacher distribution, its entries are -1 or 1, d ′ and d are the dimensions before and after projection, respectively. According to the Johnson-Lindenstrauss Lemmas [57] , this operation can nearly preserve the gradient distances, ensuring the usefulness of the low-dimensional features.Estimating Data Value. By using Eq. 4, we can compute the gradient features for synthetic instances. Then, we randomly sample M D instances from the training sets of downstream mathrelated datasets {z ′ i } M D i=1 , where z ′ i = ⟨q i , si ⟩, and also compute their gradient features. Next, we estimate the value of each synthetic instance by computing the similarity between its gradient feature and the average feature of all the sampled downstream instances as:EQUATIONwhere Cosine(x, y) computes the cosine similarity between the two vectors. In this way, the instance with higher data value would lead to a more positive influence on the downstream math-related tasks.Based on the estimated values, we can rank all the synthetic instances, and the top-ranking N add ones can be regarded as the most valuable data {⟨p i , w i , q i , s i ⟩} N add i=1 for improving downstream math-related tasks. Thus, we utilize GPT-4 to regenerate the synthetic math problems based on their prompts and original math-related texts, to acquire corresponding more high-quality math problems and solutions. Then, we add the new GPT-4 synthetic data {p i , w i , qi , ŝi } N add i=1 into the KD dataset, and the new data is capable of guiding the small LLM to generate more useful math problems for downstream tasks. Next, we retrain the data synthesis LLM with the updated KD dataset using Eq. 1.After training the data synthesis LLM, we construct the multi-source corpus containing rich mathrelated texts to cover more knowledge and scenarios. Then, we synthesize massive math problems based on it, which are used for pre-training JiuZhang3.0.Constructing Multi-source Corpus. We consider the following data types and select the corresponding open-source datasets to compose the math-related multi-source corpus.• Webpages: we use the OpenWebText corpus [58] , which consists of 6.3M math-related web documents extracted from Common Crawl.• Books: we use the Mathpile-textbook dataset [59] , including 4K educational textbooks, lecture notes and synthetic books.• Papers: we use the Mathpile-Arxiv dataset [59] , and select the high-quality ones according to the estimated scores (0.6-0.9), which are released by AutoMathText [60] .• QA Data: we select the StackExchange subset of the MMIQC dataset [41] , which contains 1.2M processed real-world math question-answering pairs.• Wikipedia: we use the Mathpile-Wikipedia dataset [59] , consisting of 106K documents from math-related entries in Wikipedia.Data Synthesis for Training JiuZhang3.0. For each instance within the multi-source corpus, we randomly select a prompt from the prompt set and embed the text into the prompt to compose the input. Then, we feed inputs into the data synthesis model, to generate the math problems and solutions for composing the synthesis dataset D = {q i , s i } M i=1 . Here, we follow existing work [5, 13] to filter out the instances with 10-grams overlap to both inputs and outputs from test sets of downstream evaluation tasks. We synthesize about 6M math problems (4.6B tokens) in total, which are used for pre-training JiuZhang3.0 to predict the solution based on the given problems. 4 Experiments 
experimental settings: 
 For our JiuZhang3.0, we follow existing work [13] that train the 7B, 8B and 8×7B versions based on Mistral-7B [61] , LLaMA-3-8B [18] , and Mixtral-8×7B [19] . During training, we first pre-train it on our synthetic 4.6B math problem-solution pairs and then fine-tune it on the collected multiple open-source instruction datasets. We evaluate JiuZhang3.0 in two settings, i.e., natural language reasoning and tool manipulation. More details about the fine-tuning data, evaluation datasets, baseline methods, and implementation details are in Appendix B, C, D and E, respectively.Natural Language Reasoning. The results of this setting are shown in Table 1 . First, the baseline methods trained on math-related data perform better than others. Among them, DeepSeekMath-7B is the best-performed base LLM, and DeepSeekMath-7B-RL also performs better than other baselines, since they have been pre-trained on 120B corpus containing rich math-related data. Besides, KPMath-DSMath-7B and MAmmoTH2 also perform well. Concretely, KPMath-DSMath-7B is trained on nearly 1M synthetic math problems produced by GPT-4, and MAmmoTH2 also utilizes the GPT-4, Mixtral-8×7B, and Qwen-72B to extract and refine the problems existing in the webpages. The acquired problems can greatly improve their performance in math problem solving. In our approach, we also utilize synthetic math problems to train our JiuZhang3.0-7B and 8B models. Differently, our used synthesis model is a much smaller 7B LLM, which has been trained by distilling the data synthesis capability from GPT-4. Thus, it can guarantee the quality of the synthetic data, and helps JiuZhang3.0 models perform the best across most of the dataset. The higher quality also reduces the data amount requirement for pre-training. Owing to our designed high-value data selection strategy, we can also reduce the times of invoking GPT-4 API for knowledge distillation. As noted in Figure 1 , the total cost of our approach is nearly only 20% of the compared baselines, indicating its efficiency.The results of other datasets with different data formats or related to other fields are shown in Table 2 . As the listed datasets focus on evaluating the different aspects, the performance of LLMs also differ a lot. For TabMWP, AQuA, and OCW-Math, our JiuZhang3.0-8B and JiuZhang3.0-8×7B achieve the best performance. The three datasets require the understanding of table data, algebra, and undergraduate-level science knowledge respectively, which may have been covered in our synthetic math problems guided by the multi-source corpus. However, our JiuZhang3.0 models perform not well on MMLU-STEM. It indicates the shortcoming of our approach that our prompts and math-related texts might not well cover the knowledge from other subjects. Tool Manipulation. The results are shown in Table 3 . We can see that our JiuZhang3.0-7B and 8B models outperform all the baseline methods by a large margin, indicating the effectiveness of our approach in this setting. The reason is that we synthesize massive math problems in this format, which can teach JiuZhang3.0 models to accurately utilize tools by generating programs. Besides, the mixed synthetic math problem from the natural language reasoning setting can also benefit the required capabilities for this setting. Different from the natural language reasoning setting, JiuZhang3.0-8B (based on LLaMA-3-8B) performs better than the 7B version (based on Mistral-7B). The reason may be that LLaMA-3-8B owns stronger code synthesis and tool manipulation capability than Mistral-7B.Performance w.r.t. Pre-training Data Amount. In this part, we study how the scaling of synthetic data amount affects the model performance. We train Mistral-7B and LLaMA-3-8B using varying ratios of our synthetic entire dataset, i.e., 20%, 40%, 60%, 80%, 100%, and report the performance on GSM8k, MATH, and ASDiv under the natural language reasoning setting. For comparison, we also show the results of the best-performed base LLM, i.e., DeepSeekMath-Base-7B.As shown in Figure 3 , with the increasing of the training data ratio, the performance of our model improves consistently. Based on Mistral-7B, it can outperform the best-performed baseline using only 80% or 60% of the pre-training data, indicating the high quality of our synthetic pre-training data.Based on LLaMA-3-8B, it can perform better than the baseline using 40% or even 20% data, and the performance is consistently better than using Mistral-7B. It demonstrates that LLaMA-3-8B can better adapt into our synthetic data. Besides, the performance of our model can surpass the baseline more on MATH, which is a very complex dataset consisting of competitive problems, exhibiting the superiority of our method for improving the advanced mathematical reasoning capability.Ablation Study. We conduct the ablation study to verify the effectiveness of key components in our proposed method. We test the following variations based on our approach, i.e., (1) (5) w/o using GPT-4 for Boosting: directly uses the high-value instance for boosting data synthesis model instead of using GPT-4. Limited by the computing resource, we conduct the ablation study under the natural language reasoning setting, and use 100k synthetic instances and randomly select 50k instructions from the instruction set. We report the results on GSM8K, MATH, ASDiv and CARP-en.As shown in Table 4 , all the variations mostly underperform the original model, indicating the effectiveness of all the components. Besides, the variation w/o using GPT-4 for Boosting performs slightly better in GSM8k and CARP, but degrades a lot in MATH (32.8-→27.8). A possible reason is that it can benefit from the selected high-value data. But without the help of GPT-4, it can not synthesize helpful complex math problems for the competitive problems within MATH dataset.Variation Study for Data Synthesis LLMs. To verify the effectiveness of our trained data synthesis LLM, we conduct the variation study using other existing LLMs for synthesizing the pre-training data. We select the following four LLMs, i.e., ChatGPT, Mixtral-8×7B, DeepSeekMath-RL-7B, and LLaMA-3-8B-Instruct to replace our data synthesis LLM. We follow the efficient test setting in the ablation study, and report the results on GSM8K, MATH, ASDiv and CARP-en.As shown in Table 4 , all the variations mostly perform worse than the original model. It demonstrates that existing LLMs without adapted training might not be suitable to directly synthesize the data for pre-training. Besides, the performance of all the variation degrades a lot in MATH, which consists of complex competitive problems. It indicates that these existing LLMs are hard to synthesize the data that is useful for improving the performance in solving complex math problems.Variation Study for Data Selection Strategies. To study the effectiveness of our gradient-based data selection strategy, we implement the following variations that replace it by other methods, i.e., (1) Random Sampling: randomly samples the same number of instances; (2) Perplexity: selects the instances with lowest perplexity evaluated by Mistral-7B; (3) Reward Model: uses a well-trained reward model [36] for scoring; (4) One-shot ICL: concatenates the synthetic math problem and solution with the downstream task data to construct the one-shot in-context learning (ICL) example, and computes the decrease of loss as the estimated value [62] . We follow the efficient test setting.As shown in Table 4 , our original model mostly performs the best among all the variations, indicating the superiority of our gradient-based strategy. Whereas, the variation using one-hot ICL performs relatively better than others, and achieves the best performance on GSM8k. As the problems in GSM8k typically require more natural language reasoning steps, the ICL loss can well detect the instances with helpful context for solving these problems. However, it performs not well on MATH, where the math problems are complex and require using more math symbols and formulas.Solving for k:k = 4 2 = 2Thus, the equation P (P (x)) = (P (x)) k holds for k = 2. Now let's verify this: For k = 2:P (P (x)) = x 4 (P (x)) k = (x 2 ) 2 = x 4Since both sides of the equation are equal:x 4 = x 4This equation holds true for all real numbers x. Therefore, the only value of k that makes P (P (x)) = (P (x)) k true for all real numbers x is k = 2. 
conclusion: 
 In this paper, we proposed an efficient way to improve the mathematical reasoning of LLMs, where we trained a small LLM to synthesize sufficient high-quality math problems for pre-training. Concretely, we crafted a set of prompts that cover the knowledge and difficulty levels of human education stages, and selected the high-value math-related texts for downstream math-related tasks via the gradientbased strategy. Then, we fed them into GPT-4 to create the knowledge distillation dataset, which can better teach the data synthesis model to generate diverse and useful math problems. We utilized the synthetic data to pre-train JiuZhang3.0, and the whole process only required to invoke GPT-4 API 9.3k times and pre-train on 4.6B data. JiuZhang3.0 achieved state-of-the-art performance on several datasets under the natural language reasoning and tool manipulation settings, surpassing competitive LLMs that requires much larger cost on data synthesis or pre-training. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We promise the code and the training data will be publicly available. Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Please refer to Appendix B, Appendix C, Appendix D, and Appendix E.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Our evaluation experiment is conducted under zero temperature for generation to avoid randomness. We report 18 different settings of evaluation datasets to demonstrate the performance.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Please refer to Appendix A.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA]Justification: Our research is focused on LLMs for math reasoning, which will not cause societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our research is focused on designing LLMs for math reasoning which will not have a high risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Please refer to Appendix D, Appendix C, Appendix E.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: The released code, data, and models are well documented including details about training, license, limitations.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: To estimate the expenses of previous work and our proposed JiuZhang3.0, we survey the price of potential service during the entire procedure, including calling the OpenAI API for GPT-4 and renting the AWS GPU server for LLMs training. The details of the price are presented in Table 5 . For a fair comparison, we assume that GPT-4 is utilized to synthesize training data, and 8 nodes of 8 × A100 GPU servers (64 GPUs in total) are leveraged for LLMs training.For our data synthesis process, the average length of the prompting (including selected math-related texts) is about 300 tokens, and the average length of problems and solutions is 877 tokens. Under the setting of 64 GPUs, we spend 4 hours selecting valuable data for natural language reasoning and tool manipulation in total, and 10 hours synthesizing the 4.6B pre-training corpus. We train the JiuZhang3.0-7B model for 10 hours, including both pre-training and fine-tuning.As the dataset and the construction process of KPMath are not publicly available, we adopt the average length of synthesis prompts, problems, and solutions in MetaMathQA for estimating the input and the output tokens, respectively. We estimate the training time on 120B tokens for a 7B model as 160 hours. Since the training details in the RLHF stage of DeepSeekMath-7B-RL are not publicly available, we do not count the fine-tuning cost for the KPMath model and the DeepSeekMath model. In this case, we estimate the expenses of the LLMs training process as follows,EQUATIONServer Expenses = Num Nodes × Price per Node × Training Time,Total Expenses = API Expenses + Server Expenses.The details and estimation of the expenses for different LLMs are present in Table 6 .After pre-training, we collect a set of open-source math-related instructions to fine-tune JiuZhang3.0. For natural language reasoning, we collect the training sets of MATH [63] , GSM8k [34] , CARP [33] , and open-source synthetic datasets based on them, i.e., MetaMATH [10] , MMIQC [41] , Math-Shepherd (without problems from original MATH test sets) [64] , Orca-MATH [11] . Besides, we also collect the positive examples from the PRM800k dataset (without problems from original MATH test sets) [35] , and the TAL-SCQ5K dataset consisting of multi-choice questions 3 . Whereas, we observe that the varying data styles in the above datasets might cause the LLM outputs to be irregular. Thus, we utilize a unified prompt DeepSeekMath-7B-RL, to synthesize 700k solutions for the problems from the above datasets.For tool manipulation, we use the synthetic datasets, i.e., OpenMathInstruct-1 [65] and MMOS [66] , consisting of a mixture of text reasoning and code blocks executed by a Python interpreter.We test our JiuZhang3.0 and baseline methods in the following two settings for evaluating the mathematical reasoning capability.• Natural Language Reasoning: we prompt LLMs to perform multi-step reasoning via natural language, and select the following publicly avaibable datasets: GSM8k [34] contains grade school math problems to test the basic arithmetic and reasoning ability. MATH [63] and CARP-en [33] consist of complex competition-level problems, and CARP-en is the English version of the original CARP dataset. ASDiv [67] , MAWPS [68] and SVAMP [69] are grade-school math word problem (MWP) datasets, and SVAMP focuses on the robust reasoning ability. Besides, we also consider the following datasets with different data formats or related to other interdisciplinary field, i.e., TabMWP, AQuA, SAT-Math, MathQA, MMLU-STEM. AQuA [70] , SAT-Math [71] , MMLU-STEM [72] are composed of multiple-choice questions for human exams across math and other STEM disciplines. OCW-Math [73] is a challenging dataset containing undergraduate-level math and science problems.• Tool Manipulation: we prompt LLMs to manipulate external tools via Python to solve the problems, and select GSM8k, MATH, GSM-Hard, SVAMP, TabMWP, ASDiv, and MAWPS for testing. GSM-Hard [27] replaces the numbers in the questions of GSM8K with larger numbers to increase the difficulty of calculation. TabMWP [74] is an open-domain MWP dataset containing tabular data.We consider diverse types of baseline methods for comparison.• Closed-source LLMs: ChatGPT and GPT-4 [2] ;• Larger LLMs (>20B): Qwen-1.5-110B [75] , Qwen-1.5-72B [75] , Deepseek-LM-67B [76] , Mixtral-8×7B [19] , Llemma-34B [7] , Intern-Math-20B [9] , MAmmoTH2-8×7B-Plus [13] , ChatGLM-Math-32B [77] ;• Smaller LLMs (<10B): DeepSeek-7B [76] , Qwen-1.5-7B [75] , Mistral-7B [61] , LLaMA-3-8B [18] , Gemma-7B [78] , and CodeLLama [79] ;• LLMs pre-trained on Math Corpus (<10B): Llemma-7B [7] , InternLM-Math-7B [9] , Rho-1-Math-7B [80] , DeepSeekMath-7B [5] ;• LLMs fine-tuned on Math Instructions (<10B): MetaMath-Mistral-7B [10] , WizardMath-7B-1.1 [49] , Abel-7B-002 [81] , Mistral-7B-MMIQC [41] , Math-Shepherd-Mistral-7B-RL [64] , DeepSeekMath-7B-Instruct [5] , DeepSeekMath-7B-RL [5] , Llama-3-8B-Instruct [18] , MAm-moTH2 [13] , KPMath-DSMath-7B [6] • LLMs fine-tuned on tool-augmented math instructions (<10B): MAmmoTH-7B-Mistral [39] , MathCoder-7B-CL [82] , ToRA-7B-Code [12] , MARIO-OVM-7B [83] , MMOS-CODE-7B [66] , OpenMath-Mistral-7B [65] , Rho-1-Math-7B-Code [80] .Our evaluation framework and in-context examples follow the existing work [6, 12, 84] . For general and math domain base models, we adopt the few-shot prompting method. For fine-tuned models, we adopt the zero-shot prompting method for open-ended natural language reasoning and tool manipulation tasks, and the few-shot prompting method for multiple choice problems. We cite the performance results reported in existing work [6, 12, 39, 65, 66, 77, 80, 82, 83] .Data Synthesis Models. We train two data synthesis models for the natural language reasoning and tool manipulation settings, respectively. We first initialize them by training DeepSeekMath-7B-RL with 4k and 1.3k KD datasets, respectively. Then, we utilize the synthesis models to generate 100k problem-solution pairs for each one. During value estimation, we adopt the training set of GSM8k and MATH for natural language reasoning, and the 5k subset from a mixture of OpenMathInstruct and MMOS for tool manipulation, as the instances from downstream math-related tasks. We select 2k the most valuable math texts in each setting for GPT-4 to boost the quality, and then add them into the KD dataset. During training, following existing work [14] , we adopt a cosine learning rate schedule with a 0% warm-up ratio and select a learning rate of 1e-5 for 5 epochs and 10 epochs for natural language reasoning and tool manipulation, respectively. JiuZhang3.0. Before training, we first filter out the instance from the synthetic dataset that have 10grams overlap to the test set data, and also deduplicate data and the synthetic tool manipulation data containing unexecutable code. Then, we follow existing work that trains 7B, 8B, and 8×7B versions based on Mistral-7B, LLaMA-3-8B, and Mixtral-8×7B [13] . During training, we first pre-train it on our synthetic 4.6B math problem-solution pairs and then fine-tune it on the multi-source instruction set. We reuse the optimizer to initialize the fine-tuning stage and adopt the Warmup-Stable-Decay learning rate scheduler [85] with 3% warm-up ratio and 85% stable training ratio for 1 epoch in the whole training process. We set the maximum learning rate to 1e-5 and the minimum learning rate to 1e-6 with a total batch size of 512. To boost the training efficiency, we pack multiple instances in the same context window of the model and modify the attention to avoid mutual interference among difference instances. The maximum length of model is set to 2048. We train all models with BFloat16 numerical format, Flash Attention 2.0, DeepSpeed Stage 2 for 7B and 8B models, and Stage 3 for 8×7B models.In this part, we conduct the experiments about tuning two important hyper-parameters, i.e., pretraining data proportion, and the number of high-value data.Pre-train Data Proportion. In the synthetic pre-training data proportion, we should determine the ratio between the data from the natural language reasoning and tool manipulation settings (NLC : TM). We set it to 1:1, 2:1, 4:1, and 8:1, and control the total data amount unchanged (200k) for comparison. We follow the efficient test setting in Section 4.3, and report the MATH and GSM8k results under the tool manipulation setting, as there is only slight performance changes on natural language reasoning setting. As shown in Figure 4, 2: 1 is more suited and leads to better performance than other proportions. The reason may be that smaller or larger proportion for tool manipulation data would cause underfitting or overfitting on the tool manipulation data, affecting the corresponding capability.Number of Selected High-Value Data. We also study the effect of changing the number of selected high-value data according to the ranking, as more data requires larger cost for invoking the GPT-4 API. We set it to 2k, 4k, 6k, and 8k, and do not change other settings for fair comparison. We follow the efficient test setting in Section 4.3. As shown in Figure 4 , using the top 2k high-value data achieves the best performance, and more data even causes the performance degradation. It indicates that the data quality is more important than data amount for boosting the data synthesis capability, as using more data would inevitably involve low-quality data.First, although we train a strong yet small LLM that can synthesize high-quality math problems for training, its capabilities on synthesizing the data for other domains or tasks might be relatively weaker. The reason is that we only use the math-related data to train it. In future work, we will try to train a general-purpose model, to enable the data synthesis for other requirements. Second, in this work, we only focus on mathematical reasoning capability, and our trained JiuZhang3.0 is also mainly for solving math problems. Limited by the computation resource, we do not test its performance on other complex reasoning tasks, e.g., planning, commonsense reasoning. We will also conduct the corresponding experiments in the future. Third, the 4.6B pre-training data is still a large scale for training, and we do not perform data filter to control its quality. Future work should focus on reducing its scale by proposing better data filter strategy. Forth, we only utilize GPT-4 for knowledge distillation, but do not use other well-performed LLMs, e.g., Claude 3, GLM-4, and the latest GPT-4o. More experiments should be conducted on these LLMs to study the effect of the teacher LLMs. Fifth, for cost estimation, our reported results are all estimated according to our experience, without the re-implementation of these methods. We also appeal to report the true cost of training LLMs in existing work.4. **Vertex of the Parabola:** The vertex of a parabola y = ax 2 + bx + c is given by:EQUATIONHere, a = -1 5 , b = 6, and c = 0:EQUATIONSo, the maximum height occurs at t = 15 seconds. However, this t value exceeds the range of our data points. In practical interpolation methods, this would indicate a correction point is needed beyond the given data. But for demonstration, let's assume t = 15 is within bounds (extrapolation for students): Plug t = 15 back into the original equation with adjusted coefficients:y = - 1 5 (15) 2 + 6(15) = - 1 5 (225) + 90 = -45 + 90 = 45Thus, the maximum height is **45 meters**Type: Natural Language Reasoning -High Math-Related Texts:Let k be a positive integer. Find all polynomials with real coefficients which satisfy the equation P (P (x)) = (P (x)) k . Let k be a positive integer. Find all polynomials with real coefficients which satisfy the equationP (P (x)) = (P (x)) k .I simply don't even know how to think about this problem. I've tried simple stuff just to get my head on the problem. For example for P (x) = x n I have P (P (x)) = (P (x)) n , and I think that any polynomial P (x) = x n + x n-1 + • • • + c can't be a solution as I would have P (x) = P (x)q1(x) + R. After that I simply stare at the problem. Can you guys give some help ? Note: I would like to understand how to tackle these kind of problems, so I would be really grateful if you would explain the thinking process behind the solution. (This is optional, so feel free to give an answer as you prefer.) Thanks in advance.First we note that only the constant polynomials P ≡ 0 and P ≡ 1, as well als P ≡ -1 for odd k, satisfy the equation. (This holds true for k > 1; for k = 1 any constant polynomial will do.) Now if P is not constant, then the range Y = {P (x) : x ∈ R} of P is an infinite set. For every y ∈ Y we have P (y) = y k , which means that the polynomial Q(x) = P (x) -x k is zero on Y . Since Y is infinite, this implies that Q is the zero polynomial. In conclusion, P (x) = x k . Synthesis QA: Question: Consider the polynomial P (x) = x 2 . We want to find out how P behaves when we apply it twice, P (P (x)), and compare it with P (x) raised to some power, (P (x)) k . Let's explore the following scenario: Given P (x) = x 2 , find all values of k such that the equation P (P (x)) = (P (x)) k holds for all real numbers x. Answer: First, we evaluate P (P (x)):P (P (x)) = P (x 2 ) = (x 2 ) 2 = x 4Next, we need to express P (x) raised to some power k:(P (x)) k = (x 2 ) k = x 2kWe want P (P (x)) = (P (x)) k :x 4 = x 2kThis equation holds for all real numbers x if and only if the exponents are equal. Therefore, we set up the following equation:Type: Tool Manipulation -Grade School Math-Related Texts: Refer to a unit in the Units package by indexing the name or symbol with the context, for example, bit[information] or bps[information]; or, if the context is indicated as the default, by using only the unit name or symbol, for example, bit or bps. The units of information are defined as follows. A crumb is defined as 2 bits. A nibble is defined as 4 bits.Synthesis QA: Question: In a digital communication system, information is transmitted in units called bits. A crumb is defined as 2 bits, and a nibble is defined as 4 bits. If a user sends a sequence of bits, and the sequence can be broken down into whole crumbs and nibbles, how many different ways can the user send exactly 18 bits? Answer:def count_ways_to_send_bits(total_bits): ways = 0 for crumb in range(total_bits // 2 + 1): for nibble in range(total_bits // 4 + 1): if crumb * 2 + nibble * 4 == total_bits: ways += 1 return ways total_bits = 18 ways_to_send = count_ways_to_send_bits(total_bits) print(ways_to_send) "'output 5 "' There are 5 different ways the user can send exactly 18 bits.Type: Natural Language Reasoning -AMC12 Math-Related Texts: Julia Robinson Mathematics Festival The Julia Robinson Mathematics Festival (JRMF) is an educational organization that sponsors locally organized mathematics festivals and online webinars targeting K-12 students. The events are designed to introduce students to mathematics in a collaborative and non-competitive forum. [ '25. 98"W Key people Nancy Blachman, Founder Daniel Kline, Executive Director Websitejrmf.org History In the 1970s, Saint Mary's College of California produced a mathematics contest that was popular with secondary schools throughout the San Francisco Bay Area. [4] In 2005, Nancy Blachman attended an education forum sponsored by the Mathematical Sciences Research Institute (MSRI) and remembered how the Saint Mary's contest had inspired her as a student. [5] Unfortunately, the contest no longer existed. Seeking to possibl Synthesis QA: Question: At the Julia Robinson Mathematics Festival, students are challenged with a geometry puzzle. The problem is as follows: A square ABCD with side length 10 is inscribed in a circle. Points P and Q lie on the circle such that P is on the same side of line AD as C, and Q is on the same side of line AB as C. If AP = 16 and AQ = 12, find the length of P Q. Answer: To solve this problem, we'll start by finding the radius of the circle, using the fact that the square is inscribed in the circle. Then, we'll use properties of circles and the given lengths to determine the length of P Q. 1. **Finding the radius of the circle:** Since the square ABCD is inscribed in the circle, the diameter of the circle is equal to the diagonal of the square. The diagonal of a square with side length s can be calculated using the Pythagorean theorem:Diagonal = s √ 2For the given square with side length 10, the diagonal is:EQUATIONThus, the radius of the circle, which is half the diagonal, is: EQUATIONEQUATIONThus, the length of P Q is 16 √ 3 units. This problem challenges students to use geometric principles (properties of circles and squares) and algebraic reasoning to solve for lengths in a geometric configuration, enhancing analytic and problem-solving skills suitable for the AMC 12 level.Type: Tool Manipulation -Secondary School Competition Math-Related Texts: 0Nick D. Jan 5 '17 at 14:48 • I guess it comes from a wifi code.Are you a chinese student? -0Renascence5. Jan 6 '17 at 8:50 HINT: Observe that the function is odd and the interval [-5, 5] is symmetric about 0. See more in this question: Definite integral of an odd function is 0 (symmetric interval)x 4 + 2x 2 + 1 = x 2 + 12 so we have an integral of the formx 3 (x 2 + 1) 2 sin 2 xdxwe can clearly see thatf (x) = x 3 (x 2 + 1) 2 sin 2 x =⇒ f (-x) = -f (x)i.e. "' Guidelines for each section:Synthesis QA:Question: Consider the function f (x) =x 3 (x 2 +1) 2 sin 2 x over the interval [-5, 5] . We are interested in evaluating the definite integral of this function with respect to x. Given the properties of the function f (x), which is odd, and the symmetric interval about the origin, determine the value of the integral. [Problem]: Craft a concise math word problem suitable for grade school, focusing on basic arithmetic operations (addition, subtraction, multiplication, division), number sense, simple shapes, or introductory measurements. Use relatable, real-world scenarios appropriate for the age group. Ensure the problem is purely text-based and solvable without images. [Solution]: Provide a clear, step-by-step solution to the problem using simple language that a grade school student could understand. Explain the reasoning behind each step.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Justification: Our research is focused on LLMs for math reasoning, conforming with the NeurIPS Code of Ethics in every respect.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).https://github.com/math-eval/TAL-SCQ5K 


Decision: This paper proposes a method of enhancing mathematical reasoning in LLMs through cost-efficient data synthesis. It also provides extensive experiments and comprehensive ablation studies to validate the effectiveness of the proposed pipeline, demonstrating superior performance on various mathematical reasoning datasets. It reveals that data synthesis abilities of large models can be effectively distilled into smaller models, enhancing efficiency. Additionally, the paper is pioneering in scaling instruction data over 5M pairs, validating the feasibility and effectiveness of this approach for reasoning tasks.  Reviewers like the proposed cost-efficient method. 
At the same time, please follow the promise of releasing the code and data to the research community.

Reviews: review: paper_summary: paper_summary: The paper addresses the high cost of training large language models (LLMs) for mathematical reasoning by proposing a more efficient method for generating high-quality pre-training data using a smaller LLM. Instead of relying on extensive pre-training data or using powerful models like GPT-4 for large-scale synthesis, the authors train a small LLM to synthesize math problems. They create a dataset by distilling GPT-4's synthesis capability into the small LLM using carefully crafted prompts covering various educational stages. Additionally, they employ a gradient-based influence estimation method to select the most valuable math-related texts for creating a knowledge distillation dataset. This approach significantly reduces the need for GPT-4 API calls and large-scale pre-training data. The resulting model, JiuZhang3.0, achieves state-of-the-art performance on several mathematical reasoning benchmarks.
strength_weakness: strength_weakness: Some details of the paper are not clear, please refer to the following questions.
questions: questions: 1. Table 1 shows that as the model size increases, the performance does not improve significantly, and some metrics even decline.
2. The accuracy of the pre-trained synthetic data raises concerns about the noise introduced by numerous erroneous samples.
3. Performance comparison of pre-training using original data versus synthetic data.
4. Differences between using gradient descent and semantic similarity methods to filter high-quality data.
5. Both Table 1 and Table 3 indicate that reasoning with CoT and Tool assistance yields similar performance.
6. The loss calculation method for the pre-trained model and its few-shot performance on some benchmarks, like GSM8k and MATH.
7. Figure 3 illustrates that as the data volume increases, the performance of two base models initially rises and then falls, rather than continuously improving.
8. Is fine-tuning Llama3 more suitable for your proposed synthetic method compared to Mistral? Which is more appropriate during the SFT phase: Llama3 or Mistral?
9. Why was 10k selected for training the synthetic model? Could more data be used?
10. Comparing the synthetic data capabilities of different LLMs in Table 4 seems unfair, as the study used distilled GPT-4 for synthetic data generation.
11. The performance did not significantly improve when comparing different data selection strategies in Table 4.
12. The effect of using SFT data on the Mistral-7B base model.
13. There is an error in citation [50].
limitations: limitations: Yes
paper_summary: paper_summary: This paper proposes a framework for training large-scale mathematical reasoning models using synthetic data, introducing JiuZhang3.0 based on this framework. Compared to manually collected or LLM-generated training data, this framework significantly reduces resource costs. The authors trained three different scales of JiuZhang3.0 models based on Mistral-7B, LLaMa-3-8B, and Mixtral-8*7B, demonstrating the effectiveness of this framework through extensive experiments.
strength_weakness: strength_weakness: 1. Novelty: Overall, I believe the innovation of this work is limited. The core of this work is a framework for generating high-quality training data. However, using GPT-4 to generate data to fine-tune smaller models is not innovative, as shown in [1]. Additionally, in the Boost Synthesis LLM stage, the work essentially just uses the methods from Xia [2] et al., lacking its own innovation. This framework seems more like a fusion of multiple works.
2. The authors conducted many ablation experiments, but there is a lack of in-depth analysis. For example, why is directly using GPT-4 to generate questions and answers not as effective as the original method? In the choice of Synthesis LLM, why is there such a large discrepancy? I find the authors' explanations vague.
3. There are some minor writing flaws worth noting, such as the uniformity of m_l and the capitalization in equation 3 on line 163.
questions: questions: 1. Regarding the title. I think the contribution of this paper is a framework that can generate high-quality training data, not the model itself. The title JiuZhang3.0 may mislead readers into thinking the core contribution is the model. I don't understand why the authors chose this title.
2. It seems that Initialize Synthesis LLM and Boost Synthesis LLM can be iterative. Can multiple iterations train a stronger Data Synthesis Model? Would this help improve the performance of JiuZhang3.0?
3. Regarding lines 135-136, the math-related texts used by the authors. Could the authors provide a more detailed explanation of how these texts were obtained? The ablation experiment results indicate it has a significant impact on the overall framework, so I think it is necessary to provide a detailed explanation.
limitations: limitations: Yes. The authors adequately addressed the limitations.
paper_summary: paper_summary: This work proposes a low-cost yet efficient method to synthesize QA pairs using a small LLM (e.g., DeepSeek-Math-7B model) instead of GPT-4, significantly reducing cost and improving performance in mathematical reasoning.

Concretely, it begins by initializing the data synthesis LLM through distillation from GPT-4 using randomly sampled data. It then enhances the model using high-value data selected via a gradient-based value estimation strategy and regenerates QA pairs from GPT-4, ultimately synthesizing around 6M QA pairs on diverse math pretraining corpus.

The synthetic QA pairs are applied to pretrain open-source LLMs, which are then fine-tuned on collected SFT data, resulting in improved performances across benchmarks such as GSM8K and MATH.
strength_weakness: strength_weakness: - The core idea involves distilling GPT-4’s synthesis capability into a smaller yet robust 7B LLM (e.g., Deepseek-Math-7B), which remains within the scope of distilling GPT-4 to open LLMs. The pipeline is somewhat compositional and complex, especially concerning the gradient-based value estimation.
- The main experimental result (Table 1) only displays the final performance, omitting a comparison of pretraining performance. This omission may lead to confusion about whether improvements result from the pretraining or SFT data.
- The work focuses on synthesizing QA pairs. The validity of the questions and the correctness of the solutions are crucial aspects of this line of work. This work does not address these aspects but only provides several examples in Appendix I.
questions: questions: - Regarding Weakness 2: Is there a fair comparison with Deepseek-Math-7B-Base in terms of pretraining performance in Table 1?
- Regarding Weakness 3: How do you consider the validity of the questions and the correctness of the solutions in your pipeline?
- Fine-tuning Data: In Appendix D, you mention using a unified prompt for DeepSeekMath-7B-RL to synthesize 700k solutions for problems from your instruction tuning datasets. Does this mean that no original solutions are used in your fine-tuning data, potentially losing useful data points? For example, MetaMath introduces augmented answers for GSM8K and MATH and shows benefits.
- Data Synthesis Models: In Appendix G, you mention initializing the data synthesis models with DeepSeekMath-7B-RL. Why choose DeepSeekMath-7B-RL rather than DeepSeekMath-7B-Base? Is there any experimental comparison?
limitations: limitations: See Weaknesses
paper_summary: paper_summary: The paper proposes distilling the data synthesis ability from the strong (and large) model to the small model and using the small model to synthesize (continual) pre-training data, focusing on mathematical problem-solving with natural language reasoning and tool-integrated reasoning.

Specifically, the paper 1) elaborately designs a prompt set covering different human education stages/difficulty levels, prompts GPT-4 with the designed prompts and diverse math-related texts to synthsize mathematical QA pairs, 2) uses the data synthesis samples (math-related text + designed prompt + QA pair) from 1) to train the small data synthesis model from DeepSeekMath-7B, leverages it to generate large-scale candidates, select the most valuable samples using LESS and utilizes GPT-4 to re-generate the valuable QA pairs, 3) re-trains the data synthesis model with the data merged from 1) and 2) and uses it to synthesize large-scale QA pairs, 4) continually pre-trains various base models with the data from 3), 5) finally, fine-tunes the continually-pre-trained model with mixed open-source instruction datasets.

The paper

- proposes an effective and efficient pipeline to synthesize large-scale high-quality instruction data;
- empirically validates the effectiveness of scaling up instruction data for reasoning tasks;
- conducts extensive ablation studies, providing many insights like:
  - diversity of prompts for data synthesis, reference text, and strong/specialized data synthesis models are important;
  - re-training with self-generated data synthesis samples is actually harmful.
strength_weakness: strength_weakness: - The paper achieves its best performances with a 2-stage instruction tuning on pre-trained models. Still, it doesn’t involve continual pre-training, which should be rather important for models’ reasoning abilities as proved by works like DeepSeekMath. It might need further consideration about **whether continual pre-training on large-scale synthetic instruction data before fine-tuning is compatible with or necessary to be added to existing training pipelines to achieve the best final performance (for reasoning tasks)**. Pre-training and RL should be out of this work’s scope. However, it would be better to further clarify the impacts of 1) continual pre-training on domain-specific corpora, 2) continual pre-training on large-scale instruction data, 3) final fine-tuning on additional instruction datasets, and their combinations.
  - According to the baseline results from DART-Math, directly fine-tuning on DeepSeekMath-7B-RL-synthesized data only can produce competitive performance (on GSM8K and MATH under the natural language reasoning setting). **Ablation studies of performance by 2) and 3)** only should be done to better validate the compatibility/necessity of the pipeline.
  - **To consider 1) continual pre-training on domain-specific corpora**, it is unrealistic to conduct by yourselves, but a possible workaround could be to make full use of the resources provided by DeepSeekMath: DeepSeekMath-7B is continually pre-trained from DeepSeek-Coder-Base-v1.5. By comparing performances on reasoning benchmarks of applying 2/3/2+3 on DeepSeek-Coder-Base-v1.5/DeepSeekMath-7B and the two models themselves, a more comprehensive study on the impacts of these training stages can be done.
- The abilities of the data synthesis model should be an important factor and the size of the knowledge distillation (KD) dataset could have a significant impact on the data synthesis abilities, but no related ablation studies have been done. It would be better to investigate **the scaling behavior of the KD dataset size**.
- Continual pre-training is usually done with LM loss without masking the prompt tokens (e.g. in DeepSeekMath). It is not mentioned **which type of loss is used for (continual) pre-training** in the paper and worth corresponding ablation studies.
- The upper limit of synthetic data is important. The student model could be weaker than the teacher model for data synthesis, so the results in the paper might not be the upper limit. It is so expensive and unrealistic for the authors to scale synthetic data to the same scale using GPT-4, but **more comparisons with dataset size controlled with previous works training on large-scale GPT-4-synthesized data (e.g. KPMath)** should be also helpful.

---

# After rebuttal and discussion

The authors resolve almost all the concerns.

I would improve my rating to 8 because JiuZhang3.0 has been validated to replace the continual pre-training stage in the standard SotA pipeline and shows great potential to combine further with continual pre-training like DeepSeekMath to effectively push forward the upper limit.

The main limitation is that the pipeline still relies on strong DeepSeekMath models for data synthesis, not fully eliminating the necessity of continual pre-training on large-scale corpora, which might be a bottleneck for improving the best models like GPT-4. So I could not give a rating of 9.
questions: questions: Suggestions:

- The current setting is more like distillation from DeepSeekMath (we can substitute GPT-4 with human experts for the small KD dataset). The method could be more promising if it could help self-improvement/weak-to-strong generalization. I highly recommend adding experiments on training DeepSeekMath or stronger models in future versions.

Confusions:

- How is the reward model (Eurus-RM should be for reward to correctness) specifically used for data selection?
- How is the fine-tuning dataset synthesized with DeepSeekMath-7B-RL?
- How is data deduplication done for the (continual) pre-training data?

Typo:

- Line 734: remove deduplicate → deduplicate?
limitations: limitations: The limitations of this work are acceptable and the authors point out potential directions to address the limitations for future works.

Comments: title: New SOTA Results on 23 Datasets by Scaling our Synthetic Data into 10B
Sincerely Thank for AC
Sincerely Thank for Reviewer trb5
thanks
Sincerely Thank for Reviewer rcgh
Sincerely Thank for Reviewer FBVa
Urgent Reminder of Rebuttal
Reviewer Response
Urgent Reminder of Rebuttal
Urgent Reminder of Rebuttal
New Experiments on 2 Settings and 17 Datasets for Testing Generalization Capability
Kind Reminder of Discussion
New SOTA Results by Scaling our Synthetic Data into 10B
Reply to Official Comment by Reviewer rcgh (part-2)
Reply to Official Comment by Reviewer rcgh
Rebuttal by Authors (part-3)
Rebuttal by Authors (part-2)
Rebuttal by Authors (part-2)
Rebuttal by Authors (part-2)
Rebuttal by Authors (part-3)
Rebuttal by Authors (part-2)
comment: Dear Reviewer 1oiT,

We sincerely appreciate your thoughtful feedback and your time to review our work. Here, we would like to address your concerns by clarifying the innovation of our approach or reporting new experimental results:

1. Cost-Effectiveness: Our method achieves significant cost reduction compared to baseline works. We use only 4.6B synthesized pre-training tokens, whereas DeepSeekMath uses 120B math corpus, and KP-Math uses 865k training samples generated from GPT-4. As a result, our method achieves better performance on evaluated datasets with nearly 20% of the total cost of the existing state-of-the-art methods. Besides, our method achieves significantly better performance compared to pre-training on the original pre-training corpus for synthesizing data.

2. Scalability: Our method is not only efficient but also highly scalable. When we scaled our training corpus to 10B tokens, we observed further improvements in performance, outperforming baselines in a large margin. We report the results on the following 23 datasets, where the first two table is for natural language reasoning and the last is for tool manipulation settings, respectively. With 10B PT data, our method can lead to great performance gain on Mistral-7B, achieving new SOTA on all the datasets. It demonstrates the potential of our approach to yield even better results with increased data amount.

|                                | GSM8k | MATH | SVAMP | ASDiv | MAWPS | CARP |
| ------------------------------ | ----- | ---- | ----- | ----- | ----- | ---- |
| DeepSeekMath-RL                | 88.2  | 50.2 | 87.3  | 91.8  | 95.5  | **51.6** |
| Mistral-7B + 4.6B-PT + SFT     | 88.6  | 52.8 | 90.4  | 92.6  | **97.3**  | 51   |
| Mistral-7B + 10B-PT + SFT      | **90.4**  | **56.2** | **91.1**  | **93.3**  | 97.1  | 51.1 |


|           | MWPBench/Collge Math                  | MWPBench/GaokaoBench | MWPBench/Fresh Gaokao 2023 | Gaokao2023 Math En | OlympiadBench | DeepMind Mathematics | GSM Plus | Math Odyssey | TheoremQA | Minerva Math |
|---------------------------------------|----------------------|----------------------------|--------------------|---------------|----------------------|----------|--------------|-----------|--------------|-------|
| DeepSeekMath-7B-RL                   | 37.5                 | 42.7                       | 36.7               | 43.6          | 19.3                 | 57.3     | 66.7         | **36.2**      | **37.9**         | 20.6  |
| Mistral 7B + 4.6B PT + SFT                        | 38.2                 | 38.8                       | 43.3               | 48.8          | 22.2                 | 57.2     | 68.4         | 33.9      | 29.3         | 20.2  |
| Mistral 7B + 10B PT + SFT             | **41.7**                 | **46.1**                       | **56.7**               | **51.2**          | **25.6**                 | **67.0**     | **71.2**         | 34.4      | 32.5        | **25.4**  |

|   | GSM8K                                 | MATH  | GSM Hard | SVAMP   | TabMWP | ASDiv  | MAWPS  |
|---------------------------------------|-------|----------|---------|--------|--------|--------|-------|
| DeepSeekMath-7b-RL           | 78.5  | 52.8     | 61.3    | 85.9   | **80.3**   | 87.0   | 95.8  |
| Mistral 7B + 4.6B PT + SFT                        | 82.4  | 53.0     | **64.9**    | 89.2   | 75.6   | 88.3   | 96.6  |
| Mistral 7B + 10B PT + SFT             | **91.1**  | **54.8**     | 61.2    | **91.3**   | 79.6   | **93.4**   | **97.4**  |

As we will publicly release all the synthetic data, our method would be a promising solution to reduce the cost of achieving the SOTA performance for LLM. Besides, we believe that we have not even reached the effectiveness ceiling of our method. With more computing resource, it is also promising to further scale up the synthetic dataset or fine-tune stronger LLMs as the data synthetic model.
Thanks for your detailed response. It addressed some of my concerns. However, Questions 12 does not show the significant improvement of GSM8k, SVAMP, MATH, CARP and other benchmarks after using a large amount of pre-training corpus. The results are not very impressive, so I will doubt the core innovation and effectiveness of this article. I will maintain my current score. And further discuss with other reviewers.
Dear AC oBmP,

Thank you so much for your attention! After discussion with the three reviewers, we have obtained massive insightful and constructive suggestions. We will revise our paper according to the valuable suggestions!

For reviewer 1oiT, we would like to know whether our responses have addressed your concerns. We are very glad to discuss with the reviewer for clarifying the concerns and improving our work.

Thank you once again for taking the time to our paper.

Best regards,

The Authors
Dear Reviewer trb5,

We sincerely thank you for your response and valuable feedback that helped enhance our paper significantly. If you have any additional queries or concerns about preventing you from raising your score, please don't hesitate to inform us, and we'll do our best to respond promptly.

Thank you once again for taking the time to review our paper and read our comments. 

Best,

Authors
Thank you for your patience!
I will notify reviewer 1oiT.
I would like to keep the score as positive.
Dear Reviewer rcgh,

We sincerely thank you for the updated score.

We appreciate your valuable feedback, which helped enhance our paper significantly. We will revise our paper according to your valuable suggestions! For your further comments, they are also constructive suggestions and we will study them in the future work. 

Thank you once again for taking the time to review our paper and discussing with us!

Best,

Authors
Thanks for your reply! It is also happy for me to see the impressive results. I will update my official review.
Dear Reviewer FBVa,

We sincerely thank you for the updated score, and appreciate your valuable feedback. We will revise our paper according to your valuable suggestions! 

Thank you for taking the time to review our paper and read our comments. If you have any additional queries or concerns, please don't hesitate to inform us, and we'll do our best to respond promptly. 

Best,

Authors
Dear Reviewer trb5,

Sorry to disturb you for the last time, but only one day is left until the end of the reviewer-author discussion stage. We still do not know if you have received our response. To address your concerns, we wrote all the responses in details and added new experiments to support it, including:
- (1) clarify our contributions in Reply to W1;
- (2) train Mistral-7B only using the pre-trained or SFT data in Reply to W2 & Q1;
- (3) empirically validate that LLMs are robust to correctness noise in Reply to W3 & Q2;
- (4) clarify the used prompt for the SFT data in Reply to Q3;
- (5) train our DeepSeekMath-7B-base as data synthesis model in Reply to Q4;

You know, conducting the above mentioned experiments within the limited rebuttal period was quite challenging. We would like to know whether our responses have addressed your concerns. If you still have other concerns, please give us an opportunity to clarify them.

Sincerely hope that you can take a moment to reply to us, as it is very important for researchers and their efforts on this work.

Best regards, 

The Authors
I have read the response. Thanks for your clarification.
Dear Reviewer FBVa,

Sorry to disturb you for the last time, but only one day is left until the end of the reviewer-author discussion stage. We still do not know if you have received our response. To address your concerns, we wrote all the responses in details and added new experiments to support it, including:
- (1) clarify our novelty and contributions in Reply to W1;
- (2) provide in-depth analysis about the ablation study experiments in Reply to W2;
- (3) conduct experiments about 1-round Iteration on our method in Reply to Q2;
- (4) other detailed replies and modification for other suggestions;

We would like to know whether our responses have addressed your concerns. If you still have other concerns, please give us an opportunity to clarify them.

Sincerely hope that you can take a moment to reply to us, as it is very important for researchers and their efforts on this work.

Best regards, 

The Authors
Dear Reviewer 1oiT,

Sorry to disturb you for the last time, but only one day is left until the end of the reviewer-author discussion stage. We still do not know if you have received our response. To address your concerns, we added a number of new experiments to clarify our novelty and contributions, including 
- (1) data noise effect analysis; 
- (2) pre-training using original data; 
- (3) semantic similarity methods to filter data;
- (4) few-shot performance;
- (5) different data selection strategies with more data;
- (6) using SFT data on the Mistral-7B;

You know, conducting the above mentioned experiments within the limited rebuttal period was quite challenging. We would like to know whether our responses have addressed your concerns. If you still have other concerns, please give us an opportunity to clarify them.

Sincerely hope that you can take a moment to reply to us, as it is very important for researchers and their efforts on this work.

Best regards, 

The Authors
We sincerely thank the reviewer for the positive feedback and the patient about our discussion. Following the suggestion, we test the generalization capability of our newly trained LLMs in two settings, i.e., natural language reasoning and tool manipulation, totally  17 datasets. We report the results of our methods and baselines in the following two tables, where the first table is for natural language reasoning and the second is for tool manipulation settings, respectively.

First, with 10B PT data, DeepSeekCoder can perform better than DeepSeekMath-7B-RL on most evaluation datasets from the two settings. It indicates that the scaling of pre-training data is an effective approach to improving the general capability of LLMs.

Besides, 10B PT data also leads to great performance gain on Mistral-7B, achieving new SOTA on all the datasets. It further demonstrates that our method is a promising solution to simplify the SOTA pipeline.


|           | MWPBench/Collge Math                  | MWPBench/GaokaoBench | MWPBench/Fresh Gaokao 2023 | Gaokao2023 Math En | OlympiadBench | DeepMind Mathematics | GSM Plus | Math Odyssey | TheoremQA | Minerva Math |
|---------------------------------------|----------------------|----------------------------|--------------------|---------------|----------------------|----------|--------------|-----------|--------------|-------|
| DeepSeekMath-7B-RL                   | 37.5                 | 42.7                       | 36.7               | 43.6          | 19.3                 | 57.3     | 66.7         | **36.2**      | **37.9**         | 20.6  |
| DeepSeekCoder-7B-base + 10B PT + SFT | 37.8                 | 40.7                       | 40.0               | 48.1          | 20.0                 | 60.0     | 67.1         | 30.5      | 26.9         | 21.0  |
| Mistral 7B + 4.6B PT + SFT                        | 38.2                 | 38.8                       | 43.3               | 48.8          | 22.2                 | 57.2     | 68.4         | 33.9      | 29.3         | 20.2  |
| Mistral 7B + 10B PT + SFT             | **41.7**                 | **46.1**                       | **56.7**               | **51.2**          | **25.6**                 | **67.0**     | **71.2**         | 34.4      | 32.5        | **25.4**  |

|   | GSM8K                                 | MATH  | GSM Hard | SVAMP   | TabMWP | ASDiv  | MAWPS  |
|---------------------------------------|-------|----------|---------|--------|--------|--------|-------|
| DeepSeekMath-7b-RL           | 78.5  | 52.8     | 61.3    | 85.9   | **80.3**   | 87.0   | 95.8  |
| DeepSeekCoder-7b + 10B PT + SFT | 81.7  | 52.0     | 64.7    | 87.9   | 78.5   | 89.8   | 96.9  |
| Mistral 7B + 4.6B PT + SFT                        | 82.4  | 53.0     | **64.9**    | 89.2   | 75.6   | 88.3   | 96.6  |
| Mistral 7B + 10B PT + SFT             | **91.1**  | **54.8**     | 61.2    | **91.3**   | 79.6   | **93.4**   | **97.4**  |

After we publicly release our data, everyone can utilize it to pre-train the LLM, which greatly reduces the data amount requirement and achieve exciting performance. As the rebuttal DDL is approaching, we have to leave more interesting investigation about testing the ceiling of our approach into future work, including further scaling up the synthetic dataset or fine-tuning stronger LLMs for data synthesis. 

Thus, we sincerely appreciate it if you would raise the score, as it is indeed very important for our work.
Thanks for your reply! Your newest results are really impressive!

My last concern is **the potential generalization issue introduced by LESS with in-domain samples of the test tasks**. Could you test the SotA models and their most comparable baselines on some out-of-domain tasks, e.g. TheoremQA, MWPBench/CollegeMath/Test, MWPBench/GaokaoBench, Odyssey-Math?

I would consider improving my rating to 8/9 based on the generalization results.
Dear Reviewers,

We sincerely thank all the reviewers for their positive feedback and constructive suggestions. Now, we want to send you a friendly reminder that the discussion stage will be complete soon. We have carefully replied to all the reviewers' concerns and supported our claim with added new experiments and clarification on the rebuttal column. We really want to know whether our responses address your concerns. If there are still any unsolved concerns, please feel free to let us know, and we would be glad to discuss with you.

Thanks,

Authors
Thank you for your positive feedback and acknowledgment of our effort put into addressing your concerns. Actually, we will appreciate it if you would consider raising the score after seeing our new SOTA experimental results. 

Inspired by your concern about if our method can simplify or enhance the SOTA pipeline, we carefully rethink all the settings in our method. Surprisely, we find that due to our limitation on the pre-training data amount (4.6B), we have not even reached the effectiveness ceiling of our method. Concretely, we scale up the amount of the synthetic pre-training data from 4.6B to 10B, and utilize it to train DeepSeekCoder-v1.5 and Mistral-7B then fine-tuning on the original SFT data. We report the results in the following table, where a great performance improvement can be observed in the two LLMs. 

Firstly, more pre-training data helps DeepSeekCoder-Base-v1.5 outperform DeepSeekMath-RL on all the evaluation datasets. It indicates that we achieve the goal of using rather less data (10B) to reach the performance of SOTA large-scale pre-trained LLMs (DeepSeekMath using 120B data).

Second, more pre-training data improves the performance of Mistral-7B a lot. We  achieve new SOTA among open-source LLMs, and greatly narrow the performance gap between open-source LLMs and closed-source GPT-4. 

As we will publicly release all the synthetic data, we believe our method would be a promising solution to simplify the SOTA pipeline. Everyone can utilize our data to efficiently pre-train the LLM, which greatly reduces the data amount requirement and can achieve better performance. With more computing resource, it is also promising to further scale up the synthetic dataset or fine-tune stronger LLMs as the data synthetic model.

|         | GSM8k | MATH |	SVAMP	| ASDiv | MAWPS | CARP |
|------------|------------|------|---------|------------|------------|------|
| DeepSeekMath-RL | 88.2 | 50.2 | 87.3 | 91.8 | 95.5 | 51.6 |
| DeepSeekCoder-Base-v1.5 + 4.6B-PT + SFT|86.2|	47.4|87.2|90.9|	96.8|	49.6| 
| DeepSeekCoder-Base-v1.5 + 10B-PT + SFT|88.4 | 53.0|88.6 |92.6|97.6|51.7 | 
| Mistral-7B + 4.6B-PT + SFT |	88.6	|52.8|	90.4	|92.6	|97.3|	51|
| Mistral-7B + 10B-PT + SFT |	90.4	|56.2|	91.1	|93.3	|97.1|	51.1|
| GPT-4 | 92.2 | 65.4 | 92.9 | 94.3 | 96.6 | 53.6 |
Thanks for your high-quality reply! You have almost resolved all my concerns!

But sorry that I **would not like to raise my score >= 8 if a method could not be effectively justified to be able to simplify / improve the SotA pipeline**. Though results based on Mistral-7B&Llama-3-8B are impressive, it is highly possible that combining DeepSeekMath and JiuZhang3.0 could achieve better results but not clear if JiuZhang3.0 is truly necessary. The only way that I come up with for now to validate this is to conduct ablation studies on DeepSeekCoder-base-v1.5, but if you would not like to explore this, it would be fine.

Considering the results so far, I would keep my score as 7. However, your dedicated discussion indeed makes the submission of a higher quality. Again, thanks for your efforts in the discussion!
We sincerely thank the reviewer for the positive feedback and new insightful comments. We list our new responses for W3, W4, Q3, Minor Question and Typo below:

**W3**: Because we are focusing on the effect on masking in continual pre-training, I want to check whether masking is applied to SFT consistently for the setting “w/o Mask”. If so, the results should resolve my doubts.

- **Reply to W3**: Yes, masking is applied in the SFT process, following the classic settings in existing work[1,2]. In the above experiments, we only use or remove the Mask in the PT process.

[1] Long Long Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zheng Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. ArXiv, abs/2309.12284, 2023

[2] Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, Yudong Wang, Zijian Wu, Shuaibin Li, Fengzhe Zhou, Hongwei Liu, Songyang Zhang, Wenwei Zhang, Hang Yan, Xipeng Qiu, Jiayu Wang, Kai Chen, and Dahua Lin. Internlm-math: Open math large language models toward verifiable reasoning. ArXiv, abs/2402.06332, 2024

**W4**: The comparison might be slightly unfair because KPMath only conducts one-stage SFT on KPMath-Plus, while you keep the number of the “pre-training” samples the same as KPMath-Plus and conduct two-stage training, with the total number of your training samples more than KPMath-Plus. If possible, I would recommend showing the scaling behavior of the final SFT model performance against your training data amount for more comprehensive analysis.

- **Reply to W4**: Actually, we guarantee the total number of training samples for our method (including PT+SFT data) and KPMath-Plus to be the same. Thus, according to the results, we can compare the effectiveness of our approach and KPMath-Plus, with the same training cost.
Following the suggestion of the reviewer, we also scale the amount of pre-training data from 500k, 1M, 2M and 6M, and control the SFT data amount unchanged. We report the final model performance in the following table, where we can see that the performance increases w.r.t. the increasing of pre-training data amount. 

|         | GSM8k | MATH |	SVAMP	| ASDiv |
|------------|------------|------|---------|------------|
| Mistral + 500k CoT PT + All SFT |	88.2|	49|	87.9|	90.9|
| Mistral + 1M CoT PT + All SFT |	87.9	|49|	89.6|	91.3|
| Mistral + 2M CoT PT + All SFT | 88.4	 | 50.8	|89.9|	91.4|
| Mistral + 6M CoT PT + All SFT |	88.6	|52.8|	90.4|	92.6|

**Q3**: Your data selection method using the reward model seems to ignore the prompt information, which might be unreasonable, but still shows competitive performance on MATH and ASDiv. It might be meaningful to try to control the prompt distribution when adopting the “response reward model”.

- **Reply to Q3**: In fact, the Reward Model does not ignore prompts, as it measures whether the response has well solved the question in the prompt. It is a rather useful aspect to estimate the quality of the whole sample, and enables to identify more helpful samples for training. 
Following the suggestion of the reviewer, we also conduct the experiment to compare the selected samples with or without controling the prompt distribution when adopting the reward model. Concretely, we control that the rates of selected samples for different prompts are the same, i.e., 5%. We find that there is a large overlap between the selected samples from the two variations, nearly 95%. Also, the selected samples using different prompts by our original method naturally own similar proportion as the original dataset. Both indicate that it is unnecessary to control the prompt distribution during data selection.

**Minor Question**: Is the “Pre-training Data Proportion” calculated with token or sample?

- **Reply to Minor Question**: It refers to the proportion of samples, as it is hard to accurately control the token number for matching the selected data proportion.

**Typo**: Line 213: “4.6B math problem-solution pairs” → “… math problem-solution pairs of 4.6B tokens in total”

- **Reply to Typo**: Thank you for your carefully reading. We will refine it in the revised version of our paper.
We sincerely thank the reviewer for the positive feedback and new insightful comments. We list our new responses for W1, P.S. and W2 below:

**W1**: Your extensive experiments resolve my doubts in W1. However, it’s a pity to see that your “pre-training” data can not fully substitute (DeepSeekCoder-Base-v1.5 + PT SFT << DeepSeekMath-Base + SFT) nor bring significant improvement over SFT (DeepSeekMath-Base + PT SFT improves marginally over DeepSeekMath-Base + SFT) based on continual pre-training like DeepSeekMath. So I would still take this as a limitation for failing to effectively simplify / push forward the upper limit of the standard pipeline.

- **Reply to W1**: In this work, we do not focus on replacing the large-scale pre-training stage by our method, but seeking new low-cost approaches for learning mathematical reasoning capability. Our experiments have verified that based on strong LLMs (e.g., Mistral-7B and LLaMA-3), our method can achieve new SOTA performance, enabling them to surpass DeepSeekMath-RL. For DeepSeekCoder-base-v1.5, as it is a relatively weaker code-based LLM, it can not fully elicit the potential of our method. 
Actually, as our approach utilizes DeepSeekMath for synthesizing the pre-training corpus, our synthetic data might be full of its well-learned knowledge. Thus, it might not provide enough new knowledge for greatly improving itself. In contrast, for other models like Mistral-7B and LLaMA-3-8B, our pre-training data can boost the performance a lot. We show the performance in the following table.

|         | GSM8k | MATH-OAI* |	SVAMP	| ASDiv | MAWPS | CARP |
|------------|------------|------|---------|------------|------------|------|
| Mistral-7B + SFT	|87.5|	45.8|	87.9|	89.7	|97|	44.4|
| Mistral-7B + PT + SFT |	88.6	|52.8|	90.4	|92.6	|97.3|	51|
| LLaMA-3-8B + SFT|	87.3	|45.8	|86.9	|89.1	|96.8	|44|
| LLaMA-3-8B + PT + SFT|	88.6	|51	|89.4	|92.6	|97.1	|50.9|

**P.S**. I do not see any response to Q1, which is annotated as replied to. Do I have a misunderstanding?

- **Reply to P.S**: We have merged the response to Q1 into Reply to W1-b & Q1, and we also add new experimental results here to better discuss the self-improvement/weak-to-strong generalization.
For self-improvement, we conduct the experiments on DeepSeekMath, as we use it for data synthesis. As the results shown in the following table, our approach can further boost DeepSeekMath, surpassing DeepSeekMath-RL. It indicates the effectiveness of our approach for self-improvement. 
For weak-to-strong generalization, we utilize our data to pre-train and fine-tune Mixtral-8X7B, the SOTA LLM with Mixture-of-Expert Architecture. As the shown results, our approach can also greatly boost its performance, indicating its effectiveness for weak-to-strong generalization.

|         | GSM8k | MATH |	SVAMP	| ASDiv | MAWPS | CARP |
|------------|------------|------|---------|------------|------------|------|
| DeepSeekMath-Base	|64.1|	34.2|	73.7	|82.7	|92.7|	44.4|
| DeepSeekMath-RL | 88.2 | 50.2 | 87.3 | 91.8 | 95.5 | 51.6 |
| DeepSeekMath-Base + Ours	|87.6|	52.6	|88.6|	93.1|	96.7|	52|
| Mixtral-8X7B | 74.4 | 29.0 | 76.5 | 78.5 | 93.9 | 38.8 |
| Mixtral-8X7B + Ours | 89.8 | 53.8 | 90.2 | 93.1 | 96.7 | 52.3|

**W2**: I take experiments in Figure 4 more of showing the importance of selecting high-quality data, instead of “the scaling behavior of the KD dataset size” (of similar quality). It would be helpful to modify the size of the whole raw KD dataset, select the high-quality data subset in a similar way and compare the results.

- **Reply to W2**: Following the suggestion of the reviewer, we modify the size of the raw dataset to be 100k, 200k, 500k, 1M for scaling the KD dataset amount into 5k, 10k, 25k, and 50k. In this way, we keep the sampling rate of high-quality data to be 5%, to control the similar data quality. We utilize the new KD datasets to fine-tune the data synthesis models. Next, we utilize these models to synthesize top-ranked 100k samples for pre-training Mistral-7B, and then fine-tune on 50k instructions. We report the results on the following table, where all the models achieve similar performance and more data leads to marginal improvement in part of evaluation datasets. It indicates that 5k high-quality KD data is sufficient for training the data synthesis model, and scaling KD data can not greatly improve the performance.

|         | GSM8k | MATH |	ASDiv | CARP |
|------------|-----------|--------|------------|------------|
| Ours with 5k KD data | 78.6 | 32.8 | 84.5 | 36.2 |
| Ours with 10k KD data | 78.8 | 32.7 | 84.1 | 36.3 |
| Ours with 25k KD data | 78.9 | 33.0 | 84.6 | 35.9 |
| Ours with 50k KD data | 78.8 | 32.8 | 84.5 | 36.5 |
Thanks to the authors for your high-quality clarification! Below are my follow-up comments:

**W1**: Your extensive experiments resolve my doubts in W1. However, it’s a pity to see that your “pre-training” data can not fully substitute (DeepSeekCoder-Base-v1.5 + PT SFT << DeepSeekMath-Base + SFT) nor bring significant improvement over SFT (DeepSeekMath-Base + PT SFT improves marginally over DeepSeekMath-Base + SFT) based on continual pre-training like DeepSeekMath. So I would still take this as a limitation for **failing to effectively simplify / push forward the upper limit of the standard pipeline**.

P.S. I do not see any response to Q1, which is annotated as replied to. Do I have a misunderstanding?

**W2**: I take **experiments in Figure 4 more of showing the importance of selecting high-quality data, instead of “the scaling behavior of the KD dataset size” (of similar quality)**. It would be helpful to **modify the size of the whole raw KD dataset**, select the high-quality data subset in a similar way and compare the results.

**W3**: Because we are focusing on the effect on masking in continual pre-training, I want to check whether masking is applied to SFT consistently for the setting “w/o Mask”. If so, the results should resolve my doubts.

**W4**: The comparison might be slightly unfair because **KPMath only conducts one-stage SFT on KPMath-Plus**, while you keep the number of the “pre-training” samples the same as KPMath-Plus and conduct **two-stage training, with the total number of your training samples more than KPMath-Plus**. If possible, I would recommend showing the scaling behavior of the final SFT model performance against your training data amount for more comprehensive analysis.

**Q3**: Your data selection method using the reward model seems to ignore the prompt information, which might be unreasonable, but still shows competitive performance on MATH and ASDiv. It might be meaningful to try to control the prompt distribution when adopting the “response reward model”.

**Minor question**:

- Is the “Pre-training Data Proportion” calculated with token or sample?

**Typo**:

- Line 213: “4.6B math problem-solution pairs” → “… math problem-solution pairs of 4.6B tokens in total”
We sincerely thank the reviewer for the insightful suggestion and positive feedback, and list our responses for the questions below:

**Q2**. How is the reward model (Eurus-RM should be for reward to correctness) specifically used for data selection?
- **Reply to Q2**. The used reward model Eurus-RM can produce the reward score that reflects the human preference. For each question and its solution, we regard the question as the user input instruction, and utilize Eurus-RM to estimate whether the solution is helpful for it. In ablation study, we use the above way to replace our gradient-based value estimation strategy, to score all the synthetic candidates from the initialized synthesis model. Then, the top-ranked 2k samples are used to boost the model, and finally we use the boosted model to synthesize large-scale data for pre-training.

**Q3**. How is the fine-tuning dataset synthesized with DeepSeekMath-7B-RL?
- **Reply to Q3**. We directly feed the questions from all the collected instruction datasets into DeepSeekMath-7B-RL, and use a unified prompt to guide it. Fo each question, we first synthesize eight solutions. Then, if the dataset has provided the answer, we use rules to filter out the solutions with wrong answers. If not, we regard the mostly common answer from all the solutions as the correct one, and filter others. Finally, we keep at most 3 solutions per question, total 700k samples. The synthetic 700k samples help regulate the output format of LLMs, to avoid misleading by diverse formats from collected instruction datasets. 

**Q4**. How is data deduplication done for the (continual) pre-training data?
- **Reply to Q4**. We use the Minhash algorithm on the question part for all the synthetic samples, and set the threshold into 0.7 for filtering the duplicate questions. Concretely, we use the open-source tool https://github.com/ChenghaoMou/text-dedup for implementing it.

**Q5**. Line 734: remove deduplicate → deduplicate?
- **Reply to Q5**. Thank you for your carefully reading. We will refine it in the revised version of our paper. We will also carefully read and check the possible typos in our paper.
We sincerely thank the reviewer for the insightful suggestion and positive feedback, and list our responses for the weaknesses 2-4 below:

**W2**. The abilities of the data synthesis model should be an important factor and the size of the knowledge distillation (KD) dataset could have a significant impact on the data synthesis abilities, but no related ablation studies have been done. It would be better to investigate the scaling behavior of the KD dataset size.
- **Reply to W2**. We report how the scaling the KD dataset size affects the model performance in Figure 4 from Appendix. Concretely, we set the number of selected high-value data into 2k, 4k, 6k, and 8k, and follow the test setting in ablation study using 100k and 50k samples for pre-training and SFT, respectively. As shown in Figure 4, using the top 2k high-value data achieves the best performance, and more data even causes the performance degradation. It indicates that the data quality is more important than data amount for boosting the data synthesis capability, as using more data would inevitably involve low-quality data. Besides, it also indicates that data synthesis is not a hard-to-learn capability for small LLMs. Therefore, we only use 10k data for training, which also reduces the cost of using GPT-4 APIs. 

**W3**. Continual pre-training is usually done with LM loss without masking the prompt tokens (e.g. in DeepSeekMath). It is not mentioned which type of loss is used for (continual) pre-training in the paper and worth corresponding ablation studies.

- **Reply to W3**. We follow existing work[1,2] that adds a mask on the question tokens and only computes loss on the solution tokens during continual pre-training. Following the suggestion of the reviewer, we also test whether removing the mask would lead to better performance. Concretely, we follow the setting in the ablation study that conduct the experiments on Mistral-7B using 100k pre-training and 50k SFT samples. As shown in the following table, we can see that without the mask the performance degrades. The reason may be that the question tokens only depict the given condition of the math problems, which is not useful for improving the problem solving capability of LLMs.

[1] Yue, Xiang, et al. "Mammoth2: Scaling instructions from the web." arXiv preprint arXiv:2405.03548 (2024).

[2] Cheng, Daixuan, et al. "Instruction Pre-Training: Language Models are Supervised Multitask Learners." arXiv preprint arXiv:2406.14491 (2024).

|         | GSM8k | MATH |	ASDiv | CARP |
|------------|-----------|--------|------------|------------|
| **Ours** |78.6|	32.8|	84.5|	36.2|
| **Ours w/o Mask** | 78.5	|31.2|	79.7	|35.5|

**W4**. The upper limit of synthetic data is important. The student model could be weaker than the teacher model for data synthesis, so the results in the paper might not be the upper limit. It is so expensive and unrealistic for the authors to scale synthetic data to the same scale using GPT-4, but more comparisons with dataset size controlled with previous works training on large-scale GPT-4-synthesized data (e.g. KPMath) should be also helpful.
- **Reply to W4**. Following the suggestion of the reviewer, we control the number of training sample in KPMath and our approach being same, and compare the performance of using our approach and KPMath. We keep the scale of SFT data unchanged, but reduce the number of pre-training samples, until the same as KPMath. We report the results in the follow Table, where we can see that our approach can even perform better than KPMath. It indicates the quality of our synthetic data. The reason may be that our devised prompts and collected math-related text data help better guide the data synthesis process, enables it to outperform KPMath using GPT-4-synthesized data.

|         | GSM8k | MATH |	SVAMP	| ASDiv | MAWPS |
|------------|------------|------|---------|------------|------------|
|**Ours - Same Data Size** |87.9	|49|	89.6	|91.3	|97.2|
|**KPMath**|	83.9|	48.8|	81.5	|88.9|	94.8|
We sincerely thank the reviewer for the insightful suggestion and positive feedback, and list our responses for the questions below:

**Q3**. Fine-tuning Data: In Appendix D, you mention using a unified prompt for DeepSeekMath-7B-RL to synthesize 700k solutions for problems from your instruction tuning datasets. Does this mean that no original solutions are used in your fine-tuning data, potentially losing useful data points? For example, MetaMath introduces augmented answers for GSM8K and MATH and shows benefits.
- **Reply to Q3**. We use the original solutions of GSM8k and MATH in our fine-tuning data, as they are human-annotated high-quality data. In addition to it, we also use a unified prompt for DeepSeekMath-7B-RL to synthesize 700k solutions, which help regulate the output format of LLMs, to avoid misleading by diverse formats from collected instruction datasets. 

**Q4**. Data Synthesis Models: In Appendix G, you mention initializing the data synthesis models with DeepSeekMath-7B-RL. Why choose DeepSeekMath-7B-RL rather than DeepSeekMath-7B-Base? Is there any experimental comparison?
- **Reply to Q4**. According to the original paper, DeepSeekMath-7B-RL is trained with more instructions and math questions, it also performs much better than DeepSeekMath-7B-base. Thus, it would own better math reasoning capability than the base model. Besides, we have also conducted experiments to study which one is a better data synthesis model. Concretely, we replace DeepSeekMath-7B-RL by DeepSeekMath-7B-base in our pipeline, and utilize them to synthesize the same number of samples for pre-training. Then, we pre-train Mistral-7B on the synthetic samples, and then fine-tune on our collected SFT data. As shown in the following Table, using DeepSeekMath-7B-RL can achieve slightly better performance than using DeepSeekMath-7B-base. Thus, we choose DeepSeekMath-7B-RL in our work.

|         | GSM8k | MATH |	SVAMP	| ASDiv | MAWPS | CARP |
|------------|------------|------|---------|------------|------------|------|
| **Ours + DeepSeekMath-7B-base** 	|89 |	52.6	|89.9|	91.4|	97.3|	50.4|
| **Ours + DeepSeekMath-7B-RL** |	88.6	|52.8|	90.4	|92.6	|97.3|	51|
We sincerely thank the reviewer for the insightful suggestion, and list our responses for the questions below:

**Q1**. Regarding the title. I think the contribution of this paper is a framework that can generate high-quality training data, not the model itself. The title JiuZhang3.0 may mislead readers into thinking the core contribution is the model. I don't understand why the authors chose this title.
- **Reply to Q1**. Thank you for your valuable suggestion. This title is named due to our consideration of our JiuZhang series language models for math problem solving. We will change it to avoid possible misunderstanding about this work. 

**Q2**. It seems that Initialize Synthesis LLM and Boost Synthesis LLM can be iterative. Can multiple iterations train a stronger Data Synthesis Model? Would this help improve the performance of JiuZhang3.0?
- **Reply to Q2**. Following the suggestion of the reviewer, we also have conducted the experiments that iterate the process for further boosting the data synthesis model. Concretely, we repeat the 2nd stage that first selects 4k high-value data and then uses it for distilling knowledge from GPT-4 into data synthesis model. We repeat the above process once again, and then synthesizes 100k samples for pre-training. Following the setting in our ablation study, we then fine-tune the LLM on 50k samples. As the results shown in the following table, iteration can not further boost the performance, even causing performance degradation. The reason may be that data synthesis is not a hard-to-learn capability for small LLMs. The original 10k data is sufficient for well training the data synthesis model. 

|         | GSM8k | MATH |	ASDiv | CARP |
|------------|-----------|--------|------------|------------|
| **Ours** |78.6|	32.8|	84.5|	36.2|
| **Ours + 1 round Iteration**| 75.4	|31.8|	83.0|	35.9|

**Q3**. Regarding lines 135-136, the math-related texts used by the authors. Could the authors provide a more detailed explanation of how these texts were obtained? The ablation experiment results indicate it has a significant impact on the overall framework, so I think it is necessary to provide a detailed explanation.
- **Reply to Q3**. Thank you for your constructive suggestion. As written in Section 2.4, we select the commonly-used math-related data from multiple resource, consisting of OpenWebText, Mathpile, StackExchange. OpenWebText is widely-used for pre-training math-related LLMs, which contains the math-related webpages from Common Crawl. Mathpile is a mixture of multi-source datasets, including math-related textbooks, Arxiv papers, and wikipedia, which contain rich math knowledge with diverse styles. StackExchange contains massive real-world math-related questions, which is also very useful for learning to master mathematical knowledge. We will open source all the above data and code, to help researchers reimplement our work.
We sincerely thank the reviewer for the insightful suggestion and positive feedback, and list our responses for the weaknesses 12-13 below:

**W12**. The effect of using SFT data on the Mistral-7B base model.
- **Reply to W12**. Following the suggestion of the reviewer, we directly fine-tune Mistral-7B and LLaMA3-8B on the SFT data (without the synthetic pre-training data), and report the results on the following table. It is obvious that only using the SFT data performs not better than our approach, especially on the more complex MATH and CARP datasets. It indicates the effectiveness of our synthetic pre-training data.

|         | GSM8k | MATH |	SVAMP	| ASDiv | MAWPS | CARP |
|------------|------------|------|---------|------------|------------|------|
| **Mistral-7B + SFT only**|87.5|	45.8|	87.9|	89.7	|97|	44.4|
| **JiuZhang3.0-7B** |	88.6	|52.8|	90.4	|92.6	|97.3|	51|
|**Llama3-8B + SFT only**|	87.3	|45.8	|86.9	|89.1	|96.8	|44|
|**JiuZhang3.0-8B**|	88.6	|51	|89.4	|92.6	|97.1	|50.9|

**W13**. There is an error in citation [50].
- **Reply to W13**. Thank you for your carefully reading. We follow the suggestion in the official website of Cosmopedia https://huggingface.co/datasets/HuggingFaceTB/cosmopedia, and use the given citation format. We will check and refine it in the revised version of our paper.
We sincerely thank the reviewer for the insightful suggestion and positive feedback, and list our responses for the weaknesses 6-11 below:

**W6**. The loss calculation method for the pre-trained model and its few-shot performance on some benchmarks, like GSM8k and MATH.
- **Reply to W6**. We follow existing work[3,4] that only computes loss on the tokens in the solution. Following the suggestion of the reviewer, we also test the few-shot performance of our base model (without fine-tuning), and report the results of best-performed baseline DeepSeekMath-base for comparison. As shown in the following table, the few-shot performance of our JiuZhang3.0 also performs better, indicating the effectiveness of our approach.

[3] Yue, Xiang, et al. "Mammoth2: Scaling instructions from the web." arXiv preprint arXiv:2405.03548 (2024).

[4] Cheng, Daixuan, et al. "Instruction Pre-Training: Language Models are Supervised Multitask Learners." arXiv preprint arXiv:2406.14491 (2024).

|         | GSM8k | MATH |	SVAMP	| ASDiv | MAWPS | CARP |
|------------|------------|------|---------|------------|------------|------|
| DeepSeekMath-Base	|64.1|	34.2|	73.7|	82.7|	92.7|	44.4 |
| JiuZhang3.0-7B-Base | 68.7 |40.2	|80.4	|86.5	|93.5	|40.2|
| JiuZhang3.0-8B-Base | 73.4	|41.2	|83.3	|88.8	|93.4	|38.3|


**W7**. Figure 3 illustrates that as the data volume increases, the performance of two base models initially rises and then falls, rather than continuously improving.
- **Reply to W7**. As shown in Figure 3, we can see that the performance mostly converges well when using 100% data for Mistral-7B, and only falls a little for LLaMA-8B. The reason may be that LLaMA-8B is pre-trained on rather large-scale data (15TB tokens), reducing its data requirement in continual pre-training.

**W8**. Is fine-tuning Llama3 more suitable for your proposed synthetic method compared to Mistral? Which is more appropriate during the SFT phase: Llama3 or Mistral?
- **Reply to W8**. According to the results in our paper, Mistral-7B with our method can perform better on math problem solving tasks (in Table1), while LLaMA-3 is better on math-related interdisciplinary tasks (in Table2) and tool manipulation tasks (in Table3). The reason may be that Mistral-7B has focused on the math problem solving capability during training, and LLaMA-3 has been pre-trained on more data about interdisciplinary knowledge and code data.

**W9**. Why was 10k selected for training the synthetic model? Could more data be used?
- **Reply to W9**. As shown in Figure 3 right, the increasing of the training data does not always lead to performance improvement. According to the results, 10k data is sufficient for training the synthetic model, which indicates that data synthesis is not a hard-to-learn capability for small LLMs. Therefore, we only use 10k data for training, which also reduces the cost of using GPT-4 APIs. 

**W10**. Comparing the synthetic data capabilities of different LLMs in Table 4 seems unfair, as the study used distilled GPT-4 for synthetic data generation.
- **Reply to W10**. In Table 4, we aim to study whether small LLMs could be better data synthesis models than other commonly-used larger ones (e.g., ChatGPT, Mixtral-8X7B), for efficiently generating pre-training data. Thus, we focus more on the performance rather than the technical implementation, and maybe the other baselines also have used more high-quality data provided by human or GPT-4. The experimental results have shown that the synthetic data from our model can improve the performance more on the evaluation benchmarks, indicating its effectiveness and verifying our hypothesis.

**W11**. The performance did not significantly improve when comparing different data selection strategies in Table 4.
- **Reply to W11**. In Table 4, we conduct the experiments on 100k pre-training and 50k fine-tuning samples, due to the limitation on GPU resource. The limited training data causes the performance improvement to be not large, but most of the improvements are significant (with p<0.05) according to our t-test. Besides, with the increasing of the training data scale, the performance gap will become larger. As shown in the following table, when we use 500k pre-training samples and 50k SFT samples, the performance gap between our approach and the variation using random sampling is obvious. 

|         | GSM8k | MATH |	ASDiv | CARP |
|------------|-----------|--------|------------|------------|
| **Random Sampling PT 100k + SFT 50k**	|78.8 |31.0| 83.4| 34.3|
| **Ours PT 100k + SFT 50k**	|78.6|	32.8|	84.5|	36.2|
| **Random Sampling PT 500k + SFT 50k**	|83.2	|38.4|	88.3|	43.6|
| **Ours PT 500k + SFT 50k**	|85.9	|43.8|	90.0|	45.9|