JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models
Authors: Kun Zhou
Beichen Zhang
jiapeng wang
Zhipeng Chen
Xin Zhao
Jing Sha
Zhichao Sheng
Shijin Wang
Ji-Rong Wen
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Mathematical Reasoning
Data Synthesis
Primary area: natural_language_processing

Abstract: Mathematical reasoning is an important capability of large language models~(LLMs) for real-world applications.
To enhance this capability, existing work either collects large-scale math-related texts for pre-training, or relies on stronger LLMs (\eg GPT-4) to synthesize massive math problems. Both types of work generally lead to large costs in training or synthesis.
To reduce the cost, based on open-source available texts, we propose an efficient way that trains a small LLM for math problem synthesis, to efficiently generate sufficient high-quality pre-training data.
To achieve it, we create a dataset using GPT-4 to distill its data synthesis capability into the small LLM.
Concretely, we craft a set of prompts based on human education stages to guide GPT-4, to synthesize problems covering diverse math knowledge and difficulty levels.
Besides, we adopt the gradient-based influence estimation method to select the most valuable math-related texts.
The both are fed into GPT-4 for creating the knowledge distillation dataset to train the small LLM.
We leverage it to synthesize 6 million math problems for pre-training our JiuZhang3.0 model. The whole process only needs to invoke GPT-4 API 9.3k times and use 4.6B data for training.
Experimental results have shown that JiuZhang3.0 achieves state-of-the-art performance on several mathematical reasoning datasets, under both natural language reasoning and tool manipulation settings.
Our code and data will be publicly released in \url{https://github.com/RUCAIBox/JiuZhang3.0}.

JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Models
Kun Zhou 19 Beichen Zhang?*, Jiapeng Wang", Zhipeng Chen', Wayne Xin Zhao?t,
Jing Sha', Zhichao Sheng", Shijin Wang*.41 Ji-Rong Wen
School of Information, Renmin University of China
Gaoling School of Artificial Intelligence, Renmin University of China
FFLLTEK Research "iFLYTEK AI Research (Central China) latnncis_kuuzzzhuueee com, {zhangbei cweggbecchen2224 (aweeen....aaaaallleeeeee.. batmanfly))@gaiill com, (wangjp1010, wha.ipp...0,.rippg....i chen, , jrwen edu. cn, siigsshassyaang338fflyteee
Abstract
Mathematical reasoning is an important capability large language models (LL.Ms) for real-world applications. To enhance this capability, existing work either collects large-scale math-related texts for pre- pre-training, or relies on stronger LLMs (e.g., GPT-4) to synthesize massive math problems. Both types of work generally lead to large costs training or synthesis. To reduce the cost, based on open-source available texts, we propose an efficient way that trains small LLM for math problem synthesis, to efficiently generate sufficient high-quality pre-training data. To achieve into the small LLM. Concretely, we craft a set of prompts based on human education stages to guide GPT-4, synthesize problems covering diverse math knowledge and difficulty levels. Besides, we adopt the gradient-based influence estimation method select the most valuable math-related texts. The both are fed into GPT-4 for creating the knowledge distillation dataset to train the small LLM. We leverage it to synthesize million math problems for pre-training our JiuZhang3.0 model. The whole process only needs to invoke GPT-4 API 9.3k times and use 4.6B data for training. Experimental results have shown that JiuZhang3.0 achieves state-of- the-art performance on several mathematical reasoning datasets, under both natural language reasoning and tool manipulation settings. Our code and data will be publicly released in httth::///ttiiiii http://ggiihuuu..m//..caaa. Tacoox/JJuZhaan JiuZhang3.0
create dataset using GPT-4 to distill its data synthesis capability
Introduction
Large language models (LLMs) have shown remarkable capabilities variety of tasks [13]. However, they still struggle in solving complex mathematical problems [4. Recent work has shown that it is an effective approach training L.MSs math-related data for improving the mathematical reasoning ability [5] (.. Typically, they either collect the math-related data from the available corpora (e.g., webpages and books) for pre-training 77.), rely stronger LLMSs synthesize high-quality math problems for fine-tuning (10H22). Despite the success, existing approaches would generally cause large training or inference costs. Due to the complexity and diversity of mathematical problems, the former type of work mostly needs to collect large-scale corpus (e.g., 120B data for Deepseek- Math) for training, which greatly increases the training cost 5 7 g). Similarly, guarantee the knowledge coverage and effectiveness of the synthetic problems, the latter type of work relies on
"Equal contributions.
t Cooresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

stronger LLMs with larger scales (e.g, GPT-4) to create massive math problems, leading to larger
inference cost 66] TO, M.. In Figure J we show our estimated total costs of re-implementing two math-related LLMs (>$40000), details are in Appendix A
In this work, we aim develop a relatively low- cost data synthesis approach for improving the mathematical reasoning abilities of LLMs. Our key idea is that the data synthesis capability can be well learned by small LLMs. Here,
Benchmark Results
Estimated Cost
58.0 92.6
00206 1oo0 moc
Accuracy (%)
80-
(S) peucdde: F
small is a relative wording, which is in con-
70
trast with the extremely large or costly data
synthesis models used in prior studies (6 [3],
60-
such as GPT-4 or Qwen-72B. Actually, exist- ing work [10 14 15 has extensive evidence of strong learning and adaptation abilities of small LLMs for new tasks and domains with suit- able strategies (e.g,, training with high-quality supervised data), including math, science, and complex multimodal tasks. However, this explo- ration has been neglected in prior efforts on data synthesis. This attempt can be essentially generalized to a broader problem: whether a small (or weak) model can produce high-quality data that is useful for training large (or strong) model? Inspired by this motivation, we seek to train a relatively small yet powerful LLM for synthesizing high-quality math-related data.
15.8 10.2
$2.4
40
GSM8k
MATH
ASDiv DespocakMMath-Be TB-R-
Model
KPMait-DSMull-76
The comparison of existing work and our method in task performance and the total cost.
Figure 1:
However, due to the diverse and complex nature of math problems, it is challenging to train well- performing small LLMs for synthesizing high-quality ones. Although we can leverage GPT-4, it is not efficient to use it for synthesizing a large-scale knowledge distillation (KD) dataset. Specifically, we aim to build the KD dataset through a low-cost strategy. but it can sufficiently capture diverse and useful knowledge about math problem synthesis. Thus, we should guarantee the knowledge coverage and usefulness of the instances within the KD dataset. To achieve it, we first craft set of prompts, and each prompt corresponds to an education stage of humans, e.g., middle school and college. Using the above prompts, the instances within the dataset can well cover broad mathematical knowledge and different difficulty levels. Besides, for usefulness, we estimate the influence of the available math-related texts and downstream tasks, by computing the gradient similarity between their corresponding synthetic data and the task instances. Then, we select the top-ranking math-related texts into the KD dataset, which are high-value ones with more positive influence on downstream tasks. By feeding the texts with prompts into GPT-4, we collect the outputs to build the KD dataset.
Based on the KD dataset, train DeepSeekMath-7B as data synthesis model, which is much smaller than other commonly-used LLMs in existing work (6 [3) e.g., GPT-4 and Qwen-72B. Owing our data selection strategy, only require GPT-4 to generate 9,335 instances based on the selected most valuable texts for training it. Then, we utilize the crafted prompts to guide it for synthesizing high-quality problems. Benefiting from the strong data synthesis capability of the small model, we only need to synthesize 5,984,584 high-quality math problems (4.6B tokens) for pre-training our JiuZhang3.0. 3.0. Thus, the total inference and training cost is much less than existing work, as shown in Figure [ After pre-training, we also collect open-source math instructions to fine-tune JiuZhang3.0. The experimental results have shown that JiuZhang3.0 5. can mostly outperform state-of-the-ar methods across 18 evaluation datasets, in both the natural language reasoning and tool manipulation settings. Our contributions are summarized as follows:
(I) our research provides compelling evidence that it is feasible to efficiently train a small LLM (7B) for synthesizing training data to improve the mathematical reasoning of LLMs. As results shown in Section 4.3 its synthetic data is more useful than larger LLMs in improving the performance.
(2) we propose an efficient solution for training LLMs to improve mathematical reasoning, which only needs to invoke GPT-4 API 9.3k times and pre-train on 4.6B high-quality synthetic data, with nearly 20% total cost of existing tate-off-he-ar methods.
(3) JiuZhang3.0 3. achieves of-te--o-the--rr performance among soun-s LLMs on several tasks and settings, e.g., 52.8 (JiuZhang3.,, 33 vs. 50.2 (DeepScekMath-7B..e on MATH, 89.8 (JiuZhang3.0 - 8x vs. 86.4 (MAmmoTH2- xB-2-8xxB-PPP) 7B-Plss) on GSM8k in the natural language reasoning setting.
2

---

2 Related Work
Large Language Models. LLMs have demonstrated remarkable capabilities variety of NLP tasks, and commercial LLMs like ChatGPT, Claude, and Gemini [6 17], represent cutting-edge capabilities. Meanwhile, the performance of open-source models (e.g. LLaMA-3, Mixtral) has also developed rapidly [18] 19]. To further improve the capability of LLMs on special tasks or domains, existing work mainly focuses on optimizing the following aspects: (1) prompt engineerings such as chain-of-tie -ftin-of-thought and tree-of-thougee [20] 21: (2) continual pre-training
Coomain-speciffi
or task-specific corpus, improving the model to deal with downstream tasks [5] 7
g 22 23]; ; (3)
supervised fine-tuning, which involves tee-tuning the model related instruction datasets, enhancing
LLMs to follow special task instructions [24
(4) other strategies including RLHF [26]. tool
augmentation (27]. decoding optimization (8 and et al. We aim to efficiently improve the capability
of LLMs for mathematical reasoning by pre-training on synthetic data.
Mathematical Reasoning. Despite the impressive progress, mathematical reasoning remains a weak aspect of LLMs. To enhance LLMs' ability in mathematical reasoning. researchers have proposed surge of methods from the aspects of prompting, pre-training and fine-tuning. For prompting, the chain-of-thoughh (CoT) prompts have been widely used to guide LLMs for performing multi-step reasoning on complex math problems [20). Based on it, following work utilizes tools (27 33 and verifiers [9] 36), to further improve the accuracy of the mathematical reasoning collects a large-scale math-related corpus LLMs on it. Supervised fine-tuning methods focus using relatively less high-quality data for training the LLM, which are typically math-related instructions (I0]
process. For pre-training, existing work [5] 7
9 37 3S
and continually pre-traae troinirinng
12 39]. Recent studies show that the complexity of mathematical reasoning demands high-qualits instruction pairs, leading to reliance on advanced I.LMs like GPT-4 for data synthesis [6] 40) 41).
The pre-training and fine-tuning methods generally lead to large training and data annotation costs, respectively. Our work aims to train a small LLM specially for math problem synthesis, which can efficiently produce sufficient data for training.
Data Synthesis. For complex tasks and scenarios (e.g., mathematical reasoning), it is necessary to collect a substantial amount of data for training the I.LM to enhance it. However, the available data may not be sufficient, hence researchers have explored using automatically synthetic data with consistent distribution to real data, to enrich the training corpus 42]. For data synthesis on mathematical reasoning tasks, existing work can be roughly categorized into the following two types, according to their based guided information. The first type of work starts with existing problems
or math-relates texts to synthesize similar problems or solutions [TO 40 41 49) 550. The other
type of work relies on available knowledge points, and devises special prompts to guide LLMs for synthesizing related problems with the solutions [6 51]. As correctness is important, the two types of work generally design rules to check and remove wrong ones. In this work, based on the data synthesis model, also construct the multi-source math corpora, and craft several prompts guide it in producing diverse and useful math problems.
3 Approach
In this section, we present our approach that aims to train a small LLM for synthesizing math problems. First, initialize the data synthesis model by training it on the KD dataset, composed of crafted prompts, randomly sampled math-related texts, and the corresponding synthetic problems and solutions from GPT-4. Then, we improve its data synthesis capability by retraining it on the updated knowledge distillation dataset, where we add the high-value math-related texts selected by gradient-based influence estimation strategy. Finally, utilize the model to synthesize massive high-quality math problems for training JiuZhang3.0, O, based on the multi-soure math-related corpus.
3.1 Preliminary
We focus on training small data synthesis LLM, for synthesizing high-quality math problem- solution pairs to pre-train LLMs and improve its mathematical reasoning capability. To guarantee the quality of the synthetic data, we utilize GPT-4 to create the knowledge distillation (KD) dataset DKD {p., ti, qi, W for training the small LLM, where the math-related text t; and the prompt
3

---

Multi-source Corpus
[oo
Khen Academy Blog
hitlpsubllog thanscademmy.or
Sampled 3.3k Terts
m
Inttry Whenewe comn
Training Reference Model
ar iiv
MATH
WIKIPEDIA Behre
OpenAI ChalGPT44
Value Estimation using Gradients
Symbheti 4.68 Data
Stack Exchange
Test Sets Filter
Natural Language
Tool Manipulation
Initial KD Dataset
Reasoning Prompts
Prompts
8X7
4k High Value Data
GridlMMdddleHigg
Grade School
School
AIME
College
Secondary School
Juzhhnn 3.0- 78/88/8x78
8/10112
1.Initialize Synthesis LLM
2.Boost Synthesis LLM
3.Pre-train RuZZhang3.0
Figure 2: The pipeline of approach. We first initialize the data synthesis LLM by distilling the knowledge from GPT-4 on randomly sampled data, then boost using the high-value data selected by gradient-based value estimation strategy, finally utilize for synthesizing data to train JiuZhang3.00
P; are the input of GPT-4, q and & are its synthetic math problem and solution respectively, Then, train the small LLM on D&D imitate the data synthesis ability of GPT-4. Finally, we leverage it to synthesize the pre-training dataset D = {9: 8; fimmm based on all the collected math-related texts with randomly selected prompts {pr, 44 1 which are used for training our JiuZhang3.0.
3.2 Initializing Data Synthesis Model
In this work, we consider the natural language reasoning and tool manipulation settings, where LLMs require solving the problem by generating natural language solution (00 and an executable program with external interpreters [52]. respectively. Thus, we train the data synthesis LLMs the initial KD dataset, containing special prompts, math-related texts, and GPT-4 outputs for the two settings.
3.2.1 Prompts for Math Problem Synthesis
We aim craft prompt set that well cover the knowledge points and difficulty levels in human math education. Thus, for the natural language reasoning and tool manipulation settings, we manually craft prompt templates respectively, and each corresponds to certain education stage.
Prompts for Natural Language Reasoning. We consider the following human education stages and 4 worldwide competitions, i.e., Grade School, Middle School, High School, and College; AMC (American Mathematics Competition) 8, AMC 10, AMC 12, and AIME (American Invitational Mathematics Examination). Based on these, we design 8 prompts with corresponding instructions and guidelines. We show an example of grade school math problem synthesis:
## Instruction: Create age-appropriate math word problem for grade school students based the provided math content.
## Guidelines:
Craft concise math word problem suitable for grade school, focusing busic arithmetic operations, number sense, simple shapes, [Solution]: Provide a clear, step-by-step solution to the problem using simple language that grade schwol sludent could understand,
[Problem]:
Prompts for Tool Manipulation. We consider 2 types of math problems.i.e., Grade School and Secondary School Competitions, as the competition math problems may need tools for advanced math operations (e.g., integral computation), while grade school math problems are much easier and be solved by basic operations (e.g., +-*/. As tool manipulation solves the problem via executable programs, we use more words in prompts to emphasize the data format and show an example as follows:
## Instruction: Please gain inspiration from the following random math content create a high- quality.. ... Present your output in two distinct sections; (Problem Description] and [Solution].

---

## Guidelines: [Problem]: This should be completely self-contained, providing all the contextual information one needs to understand. ... [Solution]: Offer comprehensive, correct solution that accurately addresses the [Problem] you provided using Python code,
## Example:| Problem Description] Janet buys 3 pounds of broccoli, ... [/Problem Description] [Solution] def spending(): cost = 4 ..../Soolution)
3.2.2 Knowledge Distillation from GPT-4
Based on the prompts, we build the KD dataset train our data synthesis LLM. We randomly sample 5,336 math-related texts, and concatenate each one with a randomly selected prompt, to compose the input [p:: w:]. Then, we feed into GPT-4, and extract the synthetic math problem qi and solution & from its output using regular expressions, to compose the KD dataset DKD = {pr, wa, qi 9, ) Next, we utilize it to train the synthesis model, and the learning objective is:
Nins
L(Osyn)
Σ
log & wx]).
where agn denotes the parameters the data synthesis LLM. In this way, we can teach it to generate new math problem-solution pairs based on prompts and math-related texts by imitating GPT-4.
3.3 Boosting Synthesis Model using High-Value Value Data
After initialization, we further improve the data synthesis LLM using high-value KD data. However, would be costly if we first utilize GPT-4 to generate candidates and then select high-value ones. There- fore, we propose an efficient way that leverages the data synthesis LLM for generating candidates, and then selects valuable ones to feed into GPT-4. Specifically, we incorporate the gradient- based method [53] to estimate the influence of each synthetic instance for downstream math-related tasks, and select the top-ranking ones to update the KD dataset for retraining the data synthesis LLM.
3.3.1 Gradient-base baadd Data Value Estimation
According to the influence formulation [54], at a certain training step of model parameterized by 8, the influence of training instance 2 another instance 2' can be estimated by computing the similarity between their produced gradients, denoted as:
Inf (z, x (V((()(,,,) v((='0)).
(2)
By using it, we can measure the value of each synthetic data by computing its gradient similarity with downstream math-related task data. Concretely, we first train reference model using LoRA, then compute its gradients on LoRA parameters as the features, to help estimate the data value.
we train LLM for mathematical problem solving using LoRA (55) as the reference model. As LoRA only requires to optimize the low-rank adapters in the LLM, we can efficiently train the reference model on limited computation resources, and reduce the number of trainable parameters for efficient computation of gradient similarity. Besides, to further reduce the training cost, randomly select a subset of synthetic math problems generated by the data synthesis model, denoted Diora {g:+ 8; i=1- Then, we train the reference model to predict the solution based on the given problem, denoted as:
Training Reference Model using LoRA. Inspired by existing work 153
Mi
Mc
Tree (Otora)
log P(silg))
1=1
where Dtora denotes the parameters of LoRA, and Mj is the number of training data.
Computing Gradient Features. After training the reference model, we compute the gradients of LoRA parameters as the feature of each synthetic instance. As their dimension is large, we follow existing work [56] that performs random projection to obtain the low-dimensional features as:
Vlres Olora) II Vlref Otora):
(4)
5

---

where 2 = (ps, Wi, 4s. 8t) denotes synthetic instance, II E IR is projection matrix initialized by the Rademacher distribution, its entries are 1 or d' and d are the dimensions before and after projection, respectively. According to the nearly preserve the gradient distances, ensuring the usefulness of the low-dimensional features.
Lemmas [57], this operation can
Estimating Data Value. By using Eq. we can compute the gradient features for synthetic instances. Then, we randomly sample MD instances from the training sets of downstream math- related datasets estimate the value of each synthetic instance by computing the similarity between its gradient feature and the average feature of all the sampled downstream instances as:
Mp {>/]<=,, where 2/ = (Ø: 8,), and also compute their gradient features. Next, we
Mr
(()) Cosine (Vrrene (2, 0lora) MD
1
Σ
Vrres (2/z Otora))
where Cosine( y) computes the cosine similarity between the two vectors. In this way, the instance with higher data value would lead more positive influence on the downstream math-relate related tasks.
3.3.2 Retraining Data Synthesis Model
Based on the estimated values, we can rank all the synthetic instances, and the top-ranking Nadd ones can be regarded as the most valuable data {pss Wy, q. { ji for improving downstream math-related tasks. Thus, we utilize GPT-4 regenerate the synthetic math problems based on their prompts and original math-related texts, to acquire corresponding more high high-quality math problems and solutions. Then, we add the new GPT-4 synthetic data {pi, W4, qi & ) Nads into the KD dataset, and the new data capable of guiding the small L.M to generate more useful math problems for downstream tasks. Next, we retrain the data synthesis LLM with the updated KD dataset using Eq.
3.4 Pre-training JiuZhang3.0 using Synthetic Data
After training the data synthesis LLM, construct the multi-source corpus containing rich math- related texts to cover more knowledge and scenarios. Then, we synthesize massive math problems based on it, which are used for pre-training JiuZhang3.0.
Constructing Multi-source Corpus. We consider the following data types and select the corre- sponding open-source datasets to compose the math-related multi- corpus.
Webpages: the OpenWebText corpus [58 which consists of 6.3M math-related web documents extracted from Common Crawl.
Books: we use the Mathpile-textbook dataset 59 including 4K educational textbooks, lecture notes and synthetic books.
Papers: we use the Mathpile-Arxiv dataset [59 and select the high-quality ones according to the estimated scores (0.6-0.9), which are released by AutoMathText (60].
QA Data: we select the StackExchange subset of the MMIQC dataset which contains 1.2M processed real-world math question-answering pairs.
Wikipedia: we use the -iikippeia dataset
59
consisting of 106K documents from
math-related entries in Wikipedia.
Data Synthesis for Training JiuZhang3.0. For each instance within the multi-source corpus, we randomly select a prompt from the prompt set and embed the text into the prompt to compose the input. Then, we feed inputs into the data synthesis model, to generate the math problems and solutions for composing the synthesis dataset D = (yi 8; LM Here, we follow existing work (5) [3] to filter out the instances with 10-grams overlap to both inputs and outputs from test sets of downstream evaluation tasks. We synthesize about 6M math problems (4.6B tokens) in total, which are used for pre-training JiuZhang3.0 0 to predict the solution based on the given problems.

---

Table 1: Results on datasets in the natural language reasoning setting. The best and second-best ones among LLMs with similar scales are marked in bold and underlined respectively.
Models
GSM8K
MATH
SVAMP
ASDiv
MAWPS
CARP
Avg.
ChatGPT
76.6
38.2
83.7
87.7
96.9
41.3
70.7
GPT-4
92.2
65.4
92.9
94.3
96.6
53.6
82.5
Owen-L.5-1101 1.0
85.4
49.4
86.2
85.1
94.3
53.6
75.7
Qwen-1.5 1.5-72B
77.6
39.4
83.1
85.1
95.8
72.3
Mixtral-8x7B
74.4
29.0
76.5
78.5
93.9
38.8
65.2
Llemma-34p
60.2
24.6
68.0
75.6
89.8
36.5
59.1
Intern-Math-200
64.9
27.4
74.9
79.6
94.4
42.3
63.9
ChatGLM-MMthh332
82.6
40.6
MAmmoTT12-8x7B-PMl
86.4
47.0
90.0
92.2
97.0
76.4
J (Ours)
89.8
53.8
90.2
93.1
96.7
52.3
79.3
DeepSeck-7B
13.6
4.8
40.8
52.1
65.4
10.3
31.2
Mistral-7B
41.2
13.6
64.7
68.5
87.5
14.9
48.4
LLLMA-3-8B
54.5
19.6
68.5
72.8
90.5
29.2
55.9
Gemma- 7B
54.1
19.6
69.7
74.2
89.0
30.5
56.2
Qwen-1.5-7B
60.5
28.2
64.9
74.9
90.1
38.6
59.5
Llemma-7B
39.2
18.4
56.9
69.0
82.7
31.8
49.7
MtternLM-Math-77
45.9
15.8
67.3
71.2
88.3
28.0
52.8
Rho-1-Math-
66.3
31.0
78.5
79.2
94.0
36.7
64.3
DeepSeekMath-7B
64.1
34.2
73.7
82.7
92.7
44.4
65.3
Mistral-7B-MMIQC
75.0
34.2
73.5
82.1
90.1
65.2
NetaMath-MMsstaal-7
77.8
29.6
79.6
81.2
93.7
30.5
65.4
Abel-7B-00)
80.4
29.6
78.8
82.7
93.5
33.2
66.4
WizardMath-7BBL1
82.2
32.8
80.7
84.2
93.8
31.9
67.6
Math-Shepherr-MMistraa-PP
84.3
34.4
82.9
82.8
92.5
32.9
68.3
JPMath-DSMath-
83.9
48.8
81.5
88.9
94.8
MAmmo MAmmoTH2-7B-- TH2-7B-Plus
84.2
46.2
90.3
90.3
95.8
44.3
75.2
TH2-SB-
84.4
41.2
89.9
89.9
97.1
44.8
74.6
DeepSeckMatth-B-Iimmg Inntsurut
82.3
45.8
83.7
90.1
95.7
45.8
73.9
DrepSeekMatt-77B..
88.2
50.2
87.3
91.8
95.5
77.4
JiuZhang (Ours)
88.6
52.8
90.4
92.6
97.3
$1.0
78.8
JiuZhang3.0-80 (Ours)
88.6
51.0
89.4
92.6
97.1
50.9
78.3
4
Experiments
4.1
Experimental Settings
For our JiuZhang3. O, we follow existing work |K] that train the 7B, 8B and 8x versions based on Mistral-7B [6]]. LLaMA-3-8B [18), and Mixtral-8x75 [19]. During training, we first pre-train it synthetic 4.6B math cooblem-solution pairs and then fine-tune on the collected multiple open-source instruction datasets. We evaluate JiuZhang3.0 in two settings, i.e., natural language reasoning and tool manipulation. More details about the fine-tuning data, evaluation datasets, bascline methods, and implementation details are in Appendix B and respectively.
4.2 Results and Analysis
Natural Language Reasoning, The results of this setting are shown in Table First, the bascline methods trained on math-related data perform better than others. Among them, DeepSeekMath- the best-performed base LLM, and DeepSeekMath- 70--BB-RRI RL also performs better than other baselines, since they have been pre-trained on 120B corpus containing rich math-related data. Besides, KPMath- DSMath-7B and MAmmoTHE also perform well. Concretely, JPMath-DSMath-775 is trained on nearly 1M synthetic math problems produced by GPT-4, and MAmmoTH2 also utilizes the GPT-4, Mixtral-8 7BB, and Qwen-72B extract and refine the problems existing in the webpages. The acquired problems can greatly improve their performance in math problem solving. In our approach, also utilize synthetic math problems to train JiuZhang3.0- 3.0-BB and 8B models. Differently.

---

Table 2: Results on 5 other datasets with different data formats or related to interdisciplinary fields, and we abbreviate MMLU-STEM into M-STEM. The best and second-best methods among LLMs with similar scales are marked in bold and underlined respectively.
Models
TabMWP
AQuA
SAT-Math
M-STEM
OCW-Math
Avg.
CharGPT GPT-4
82.0
53.9
78.1
63.5
11.0
57.7
90.8
76.9
96.9
77.1
26.5
73.6
Qwen- .,11 10B
80.5
64.6
87.5
71.5
14.0
63.6
Owen-1.5-72B
56.1
55.1
87.5
68.8
7.7
55,0
Mixtral-8xP0 8x7B
67.3
48.0
65.6
62.3
8.8
50.4
Llemma-34B
57.1
71.9
54.3
48.2
Intem--aath200B
63.4
44.1
65.6
62.3
48.5
HAmmmoTH2-8x77-P2
62.7
55.9
81.2
71.8
18.8
58.
JiuZhang3..m 0-8xxB (Ours)
84.7
65.4
81.2
66.9
23.5
64.3
Mistral-7B
37.3
34.3
56.2
49.5
36.1
LLaMA-3-8B
67.5
46.5
56.2
54.4
7.7
46.5
Gemma-7B
60.9
42.9
71.9
57.7
4.8
47.6
Llemma-7B
49.2
37.8
62.5
45.8
7.7
40.6
tho-1-Math--B
55.5
49.2
75.0
54.9
6.2
48.2
DeepSeekMath-7B
69.8
51.6
84.4
56.1
17.6
55.9
DeepSeekMatth-7-Innntee
70.5
60.6
84.4
57.9
58.6
MAmmoTH2-7BBPPHn
54.7
62.2
84.4
64.0
15.1
56.1
MAmmoTT2-8B-PUP
75.1
57.5
87.5
65.7
14.7
60.1
MuZZhang3.0-77 007B (Ours)
74.8
59.4
81.2
53.6
20.2
57.8
JiuZhang3.0-88 3.0-BB (Ours)
79.2
62.2
84.4
60.4
21.3
61.5
Table Results 6 mathematical reasoning datasets under the tool manipulation setting. The best and second-bes methods are marked in bold and underlined respectively.
Models
GSM8k
MATH
G-Hard
SVAMP
TabMWP
ASDiv
MAWPS
Avg.
ChatGPT (PAL)
78.6
38.7
67.6
77.8
79.9
81.0
89.4
73.3
GPT-4 (PAL)
97.0
69.7
77.6
94.8
95.9
92.6
97.7
89.3
CodeLLama
34.0
59.0
61.4
79.6
MAmmoTH-77--Missee
75.0
40.0
MathCoder-77Bcoo
67.8
30.2
70.7
ToRA-7B-Code
72.6
44.6
56.0
70.4
51.6
78.7
91.3
66.5
MARIO-OVM-F7
74.5
47.7
IMOS-CODE-NT
73.9
44.3
76.4
78.6
OpenMath- Mistral-7B
80.2
44.5
63.7
82.4
70.0
82.7
95.4
74.1
Rho- Mrt-MMtth-B-Coo
81.3
51.8
63.1
80.8
70.1
85.5
94.5
75.3
JiuZhang3.0 3.B-BBBBBBBBB... (Ours)
82.4
$3.0
64.9
89.2
75.6
88.3
96.6
78.6
JiuZhang3.0.mm (Ours)
82.9
53.4
64.4
89.2
79.9
87.5
97.3
79.2
our used synthesis model is a much smaller 7B LLM, which has been trained by distilling the data synthesis capability from GPT-4. Thus, it can guarantee the quality of the synthetic data, and helps JiuZhang3.0 models perform the across most of the dataset. The higher quality also reduces the data amount requirement for pre-training. Owing to our designed value data selection strategy, we can also reduce the times of invoking GPT-4 API for knowledge distillation. As noted in Figure [ the total cost of our approach is nearly only 20% of the compared basclines, indicating its efficiency.
The results of other datasets with difffrrent data formats related other fields are shown in Table As the listed datasets focus on evaluating the different aspects, the performance of LLMs also differ a lot. For TabMWP, AQuA, and OCW-Math, our JiuZhang3 00-B and JiuZhang3.0-8x 7B achieve the best performance. The three datasets require the understanding of table data, algebra. and undergradnate-Ievel science knowledge respectively, which may have been covered our synthetic math problems guided by the multi-source corpus. However, our JiuZhang3.0 models perform not well on MML.U-STMM. It indicates the shortcoming of our approach that our prompts and math-related texts might not well cover the knowledge from other subjects.

---

Mistral-7B
L-MM--3-8
Mistral-7B B
1LaMA-3-SB
Mistral
LLaMA-3-50
75
45
90
GSMSk
70
MATH
40-
ASDiv
85
DeepSekkMaahi 76
35
DoopSSook Mati-7
80
65
75
30
70
20% 40%
80% 100%
20% 40% 80% 100%
20% 40%
80% 100%
Pre-training Data Proportion
Pre-training Data Proportion
Pre-training Data Proportion
Figure Performance changes with the increasing of the pre-traiin pre-training data proportion for our approach. We also show the best-performed base LLM DeepSeckMath-7B using dashed line.
Table 4: Ablation and variation studies in the natural language reasoning setting. We randomly sample 100k synthetic data and 50k instruction data for efficient test.
Variation
Models
GSM8k
MATH
ASDiv
CARP
Ours
78.6
32.8
84.5
36,2
w/o Prompt Set
76.9
27.8
81.4
34.5
w/o Math-related Texts
76.4
28.6
83.8
31.9
Ablation
w/o Boosting Retraining
77.9
31.6
83.8
34,7
w/o Value Estimation
78.8
31.0
83.4
34.3
w/o using GPT-4 for Boosting
79.0
28.4
82.9
36.3
ChatGPT
77.0
26.6
83.0
34.3
Mixtral-8xX7
77.6
26.8
82.9
33.1
Synthesis LLM
DeepSeckMath-RI-11
77.1
27.2
32.9
IJaMA-3-8B-Instmmcc
75.7
26.2
81.5
31.4
Random Sampling
78.8
31.0
83.4
34.3
Perplexity
77.5
30.8
83.1
36.3
Data Selection
Reward Model
78.0
31.6
84.3
34.8
One-shot ICL
79.2
30.2
83.3
36.0
Tool Manipulation. The results are shown in Table g We can see that our JiuZhang3 00 and models outperform all the baseline methods by large margin, indicating the effectiveness of our approach in this setting. The reason is that we synthesize massive math problems this format, which can teach JiuZhang3.0 models to accurately utilize tools by generating programs. Besides, the mixed synthetic math problem from the natural language reasoning setting can also benefit the required capabilities for this setting. Different from the natural language reasoning setting, JiuZhang3 3.0BBB (based on LLaMA-3-8B) performs better than the 7B version (based on Mistral-7B), The reason may be that LLaMA-3-88 M-M---BBB 8B owns stronger code synthesis and tool manipulation capability than Mistral-7B.
4.3 Further Analysis
Performance w.r.t. Pre-training Data Amount. In this part, we study how the scaling of synthetic data amount affects the model performance. We train Mistral-7B and LLaMA-3-SB using varying ratios of our synthetic entire dataset, i.e., 20%, 40%, 60%, 80%, 100%, and report the performance on GSM8k, MATH, and ASDiv under the natural language reasoning setting. For comparison, we also show the results of the best-performed base LLM, i.e., DeepSeckMath-Baseeee
As shown in Figure with the increasing of the training data ratio, the performance of our model improves consistently. Based on Mistral-7B, it can outperform the best-performed baseline using only 80% or 60% of the pre-traiming data, indicating the high quality of our synthetic pre- pre-training data. Based on LLaMA-3-8BP it can perform better than the baseline using 40% or even 20% data, and performance consistently better than using Mistral-7B. It demonstrates that LLaMA-3--8 can better adapt into our synthetic data. Besides, the performance of our model can surpass the baseline more on MATH, which is very complex dataset consisting competitive problems, exhibiting the superiority of our method for improving the advanced mathematical reasoning capability.

---

Ablation Study. We conduct the ablation study to verify the effectiveness of key components in our proposed method. We test the following variations based on our approach, i.e. (1) w/o Prompt Set: uses simple prompt for guiding data synthesis instead of our crafted prompt set; (2) w/o Math- related Texts: directly synthesizes the math problems without math related texts; (3) w/o Boosting Retraining: uses the data synthesis model without retraining; (4) w/o Value Estimation: ignores the estimated value but randomly samples the instances for boosting training; (5) w/o using GPT-4 for Boosting: directly uses the high-value instance for boosting data synthesis model instead of using GPT-4 Limited by the computing resource, we conduct the ablation study under the natural language reasoning setting, and use 100k synthetic instances and randomly select 50k instructions from the instruction set. We report the results on GSM8K, MATH, ASDiv and CARP-en.
As shown in Table all the variations mostly underperform the original model, indicating the effectiveness all the components. Besides, the variation w/o using GPT-4 for Boosting performs slightly better in GSM8K and CARP, but degrades a lot in MATH (32.8 -27.8). A possible reason that can benefit from the selected high-value data. But without the help of GPT-4, it can not synthesize helpful complex math problems for the competitive problems within MATH dataset.
Variation Study for Data Synthesis LLMs. To verify the effectiveness of our trained data synthesis LLM, we conduct the variation study using other existing LLMs for synthesizing the pre-training data. We select the following four LLMs, i.e,, ChatGPT, fixtral-8x7B, LLaMA-3-8B-Instree to replace our data synthesis LLM. We follow the efficient test setting in the ablation study, and report the results on GSM8K, MATH, ASDiv and CARP-en.
MatpSeeeMMthRRL--- and
As shown in Table 4 all the variations mostly perform worse than the original model. It demonstrates that existing LLMs without adapted training might not be suitable to directly synthesize the data for pre-training. Besides, the performance of all the variation degrades a lot in MATH, which consists of complex competitive problems. It indicates that these existing LLMs are hard to synthesize the data that is useful for improving the performance in solving complex math problems.
Variation Study for Data Selection Strategies. To study the effectiveness of our gradient-based data selection strategy, we implement the following variations that replace it by other methods, i.e. (I) Random Sampling: randomly samples the same number of instances; (2) Perplexity: selects the instances with lowest perplexity evaluated by Mistral-7Bs (3) Reward Model: uses a well-trained reward model [36 for scoring: (4) One-shot ICL: concatenates the synthetic math problem and solution with the downstream task data to construct the one- one-shot in-context learning (ICL) example, and computes the decrease of loss as the estimated value [62]. We follow the efficient test setting.
As shown in Table our original model mostly performs the best among all the variations, indicating the superiority of our gradient-based strategy. Whereas, the variation using one-hot ICL performs relatively better than others, and achieves the best performance on GSMSk. As the problems in GSM8k typically require more natural language reasoning steps, the ICL loss can well detect the instances with helpful context for solving these problems. However, it performs not well on MATH, where the math problems are complex and require using more math symbols and formulas.
5 Conclusion
In this paper, we proposed an efficient way improve the mathematical reasoning of LLMs, where we trained a small LLM to synthesize sufficient high-quality math problems for pre-training. Concretely, we crafted a of prompts that cover the knowledge and difficulty levels of human education stages, and selected the high-value math-related texts for downstream math-related tasks via the gradient- based strategy. Then, we fed them into GPT-4 to create the knowledge distillation dataset, which can better teach the data synthesis model to generate diverse and useful math problems. We utilized the synthetic data to pre-train JiuZhang3.0, and the whole process only required to invoke GPT-4 API 9.3k times and pre-train on 4.6B data. JiuZhang3.0 achieved state-of-the-a performance on several datasets under the natural language reasoning and tool manipulation settings, surpassing competitive LLMs that requires much larger cost on data synthesis pre-training.
10

---

Acknowledgments and Disclosure of Funding
This work was partially supported by Beijing Natural Science Foundation under Grant No. L233008 and 4222027, National Natural Science Foundation of China under Grant No. 62222215.
References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 2020.
[2] OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
[3] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong. Yifan Du, Chen Yang. Yushuo Chen, Z. Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang. Zikang Liu, Peiyu Liu, Jianyun Nie, and Ji rong Wen. A survey of large language models. ArXiv, Abs/2303.18222, 2023.
[4] Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai- Wei Chang. survey of deep learning for mathematical reasoning. ArXiv, abs/2212. 10535, 2022.
[5] Zhihong Shao, Peiyi Wang, Qihao Zhu, R. Xu, Jun-Mei Song, Mingchuan Zhang, Y. K. Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv, abs/2402.03300, 2024.
[6] Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-point-driven data synthesis with its enhancement on mathematical reasoning. ArXiv, abs/2403.02333, 2024.
[7] Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics. ArXiv, 0bs/22110110333 2023.
[8] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. ArXiv, abs/2206. 14858, 2022.
[9] Huaiyuan Ying. Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, Yudong Wang. Zijian Wu, Shuaibin Li, Fengzhe Zhou, Hongwei Liu, Songyang Zhang, Wenwei Zhang, Hang Yan, Xipeng Qiu, Jiayu Wang, Kai Chen, and Dahua Lin. Internim-math: Open math large language models toward verifiable reasoning, ArXiv, abs/2402.06332, 2024.
[10] Long Long Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zheng Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. ArXiv, 0as/2309.128444 2023.
[11] Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of sims in grade school math. ArXiv, abs/2402. 14830, 2024.
[12] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang. Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrater reasoning agent for mathematical problem solving. ArXiv, abs/2309. 17452, 2023.
[13] Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. 2024.
[14] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Tyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, I.. Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment. ArXiv, abs/2305.11106 2023.
11

---

[15] Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. ArXiv, abs/2304.08485, 2023.
[16] Anthropic. The claude 3 model family: Opus, sonnet, haiku, Technical report, Anthropic, March 2024.
[17] Google Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ArXiv, abs/2403.05530, 2024.
[18] Meta. Introducing meta llama 3: The most capable openly available Ilm to date, April 2024.
https://ai. meta. www. wwtp///lo//meeaaaeeeeeeee 3/
[19]
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, l'elio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William EI Sayed. Mixtral of experts. ArXiv, abs/2401.04055 2024.
[20] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi. Quoc Lc, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201. 1903, 2022.
[21] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. ArXiv, abs/2305.10601, 2023.
[22] K. Singhal, Shekoofeh Azizi, Tao Tu, Said Mahdavi. Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Kumar Tanwani. Heather J. Cole-Lewis, Stephen J. Pfohl, P A Payne. Martin G. Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Scharli, Aakanksha Chowdhery, P. A. Mans- field, Blaise Aguera Arcas, Dale R. Webster, Greg S. Corrado, Yossi Matias, Katherine Hui-Ling Chou, Juraj Gottweis, Nenad Tomavsev, Yun Liu, Alvin Rajkomar, Joelle Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan. Large language models encode clinical knowledge. Nature, 620:172 180, 2022.
[23] Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony S. Hartshorn Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. ArXiv, abs/2211 1.09885, 2022.
[24] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners. ArXiv, abs/2109.01652, 2021.
[25] Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery. Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction- centuuttoonffnetunedd language models. ArXiv, 00s/2210.114111 2022.
[26]
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke F. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.0215.. 2022.
[27] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Graham Neubig. Pal: Program-aide language models. arXiv preprint arXiv:2211.00433 2022.
[28] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Annual Meeting of the Association for Computational Linguistics, 2022.
12

---

[29] Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. arXiv preprint arXiv:2205.1223 2022.
[30] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. ArXiv, abs/2211.12585, 2022.
[31] Timo Schick, Jane Dwivedi-Yuu Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.0476/1 2023.
[32] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. ArXiv, abs/2210.03629, 2022.
[33] Beichen Zhang, Kun Zhou, Xilin Wei, Wayne Xin Zhao, Jing Sha, Shijin Wang, and Ji rong Wen. Evaluating and improving tool-augmented computation-intent -intennivee math reasoning. ArXiv, abs/2306.024088, 2023.
[34] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 144466,, 2021.
[35] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Nya Sutskever, and Karl Cobbe. Let's verify step by step. 2023.
[36] Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng. Zhiyuan Liu, and Maosong Sun. Advancing Ilm reasoning generalists with preference trees. ArXiv, abs/2404.02078,, 2024.
[37] Wayne Xin Zhao, Kun Zhou, Zheng Gong, Beichen Zhang, Yuanhang Zhou, Jing Sha, Zhigang Chen, Shijin Wang, Cong Liu, and Ji rong Wen. Jiuzhang: A chinese pre-trained language model for mathematical problem understanding. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022.
[38] Wayne Xin Zhao, Kun Zhou, Beichen Zhang, Zheng Gong, Zhipeng Chen, Yuanhang Zhou, Ji rong Wen, Jing Sha, Shijin Wang, Cong Liu, and Guoping Hu. Jiuzhang 2.0: A unified chinese pre-trained language model for multi-task mathematical problem solving. Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023.
[39] Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. ArXiv, abs/2309.05653 2023.
[40] Chen Li, Weiqi Wang, Jingeheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7b language models already possess strong math capabilities. ArXiv, abs/2403.04706, 2024.
[41] Haoxiong Liu, Yifan Zhang, Yifan Luo, and Andrew Chi-Chih Yao. Augmenting math word problems via iterative question composing. ArXiv, 200.22401..0003. 2024.
[42] Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. Cosmopedia, 2024.
[43] Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code is all you need. ArXiv, abs/2312.021200 2023.
[44] Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, and Andrew M. Dai. Best practices and lessons learned on synthetic data for language models. ArXiv, abs/2404.07503, 2024.
[45] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. ArXiv, abs/2304. 12244, 2023.
13

---

[46] Run Ze Fan, Xuefeng Li, Haoyang Zou, Junlong Li, Shwai He. Ethan Chern, Jiewen Hu, and Pengfei Liu. Reformatted alignment. ArXiv, abs/2402.122199 2024.
[47] Saumya Gandhi, Ritu Gala, Vijay Viswanathan, Tongshuang Wu, and Graham Neubig. Better synthetic data by retrieving and transforming existing datasets. 2024.
[48] Ruida Wang, Wangchunshu Zhou, and Mrinmaya Sachan. Let's synthesize step by step: Iterative dataset synthesis with large language models by extrapolating errors from small models. ArXiv, 1is/2310.11661, 2023.
[49] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-Guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. ArXiv, abs/2308.09583, 2023.
[50] Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. Mathgenie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of Ilms. ArXiv, abs/2402.16352. 2024.
[51] Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang. Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang. Si- Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, and Furu Wei. Synthetic data (almost) from scratch: Generalized instruction tuning for language models. ArXiv, abs/2402. 13064, 2024.
[52] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. ArXiv, abs/2305. 1138, 2023.
[53] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. ArXiv, abs/2402.04333, 2024.
[54] Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 33:19920-19930 2020.
[55] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 096885,,06,,668, 2021.
[56) Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Mądry. Trak: attributing model behavior at scale. In Proceedings of the 40th International Conference on Machine Learning, pages 27034-27113, 2023.
[57] William B. Johnson and Joram Lindenstrauss. Extensions of lipschitz mappings into hilbert space. Contemporary mathematics, 26:1 18:18662 1984.
[58] Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. ArXiv, abs//310.067886 2023.
[59] Zengzhi Wang, Rui Xia, and Pengfei Liu. Generative ai for math: Part i - mathpile: A billion-token-scall pretraining corpus for math. ArXiv, abs/2312.17120, 2023.
[60] Yifan Zhang, Yifan Luo, Yang Yuan, and Andrew Chi-Chih Yao. Autonomous data selection with language models for mathematical texts. 2024.
[61] Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lu- cile Saulnier, L'elio Renard Lavaud, Marie- Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. ArXiv, abs/2310.00s 066225. 2023.
[62] Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxi Yang, Min Yang. Lei Zhang, Shuzheng Si, Junhao Liu, Tongliang Liu, Fei Huang, and Yongbin Li. One shot learning as instruction data prospector for large language models. ArXiv, abs/2312.1030) 2023.
14

---

[63] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Xi- aodong Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. ArXiv, abs// 103.03874 2021.
[64] Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y.W,, and Zhifang Sui. Math-shepherd: Verity and reinforce Ilms seep-by-step without human annotations. ArXiv, 12s//332..0895, 2023.
[65] Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. Openmathinstructtll: A 1.8 million math instruction tuning dataset. ArXiv, abs/2402.10176. 2024.
[66] Zui Chen, Yezeng Chen, Jiaqi Han, Zhijie Huang, Qi, and Yi Zhou. An empirical study of data ability boundary in lims' math reasoning. ArXiv, abs/2403.00799, 2024.
[67] Shen- Yun Miao, Chao-Chun Liang, and Keh-Yih Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. ArXiv, abs/2106.15772 2020.
[68] Rik Koncel-Kedziorsee Subhro Roy, Aida Amini, Nate Kushman, and Hannanch Hajishirzi. Mawps: A math word problem repository. In North American Chapter of the Association for Computational Linguistics, 2016.
[69] Arkil Patel, S. Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? In North American Chapter of the Association for Computational Linguistics, 2021.
[70] Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 158-167, 2017.
[71] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied Sanosi Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. ArXiv, abs/2304.06364, 2023.
[72] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ArXiv, abs/2009.03304 2020.
[73] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Sole Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. ArXiv, abs/2206. 14858, 2022.
[74] Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, fei Chang, Ying Nian Wu, Song-Chui Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. ArXiv, abs/2304.09842, 2023.
[75] Jinze Bai. Shuai Bai, Yunfci Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenhang Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, K. Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang, Jian Yang. Shusheng Yang. Yang Yao, Bowen Yu, Yu Bowen, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xing Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. ArXiv, abs/2309. 1tts//309..6009, 2023.
[76] DeepSeek-AL Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng. Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wen-Hui Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, A. X. Liu, Bo Liu, Wen Liu, Xiaodong I Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nie, Tian Pei, Yishi Piao, Junjic Qiu, Hui Qu, Tongzheng Ren,
15

---

Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Jun-Mei Song, Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Min Tang, Bing-Li Wang, Peiyi Wang, Shiyu Wang. Yaohui Wang. Yongji Wang. Tong Wu, Yu Wu, Xin Xie, Zhenda Xie, Ziwei Xie, Yi Xiong, Hanwei Xu, Ronald X Xu, Yanhong Xu, Dejian Yang, Yu mei You, Shuiping Yu, Xin yuan Yu, Bo Zhang, Haowei Zhang. Lecong Zhang. Liyue Zhang, Mingchuan Zhang. Minghu Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, and Yuheng Zou. Deepseek Ilm: Scaling open-source language models with longtermism. ArXiv, abs/2401.0253 2024.
[77] Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan Zeng, Zhengxiao Du, Wenyi Zhao, Jie Tang, and Yuxiao Dong. Chatglm-math: Improving math problem-solving in large language models with a self-critique pipeline. ArXiv, 0bs/2404.02895. 2024.
[78] Gemma Team Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, L. Sifre, Morgane Riviere, Mihir Kale, J Christopher Love, Pouya Dehghnn Tafti, L'eonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros Ros. Ambrose Slone, Am' clie H' eliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo Cl'ement Crepy, Daniel Cer, Daphne Ippolito. David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskly,, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptise Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Pier Giuseppe Sessa, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L Smith. Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De. Ted Klimenko. Tom Hennigan, Vladimir Feinberg, Wojeiech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Brian Warkentin, Ludovic Peran, Minh Giang, Cl'ement Farabet, Oriol Vinyals, Jeffrey Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology. ArXiv, 08829, 2024.
[79] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
[80] Zheng-Wen Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, and Weizhu Chen. Rho-1: Not all tokens are what you need. ArXiv, abs/2404.07965, 2024.
[81] Ethan Chern, Haoyang Zou, Xuefeng Li, Jiewen Hu, Kehua Feng, Junlong Li, and Pengfei Liu. Generative ai for math: Abel. htttps://itthi: com/ NLP/RR- 2023.
[82] Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in 1lms for enhanced mathematical reasoning. ArXiv, abs/2310.0373.1 2023.
[83] Minpeng Liao, Wei Luo, Chengxi Li, Jing Wu, and Kai Fan. Mario: Math reasoning with code interpreter output - a reproducible pipeline. ArXiv, abs/2401.08 2024.
[84] Xinzhe Ni, Yeyun Gong, Zhibin Gou, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Exploring the mystery of influential data for mathematical reasoning. ArXiv, abs/2404.01067, 2024.
[85] Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zhen Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chaochao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm: Unveiling the potential of small language models with scalable training strategies. ArXiv, abs/2404.06395, 2024.
16

---

Table 5: The price of the potential service during the training procedure.
Items
Details
OpenAT API
30.00 USD per 1M tokens for input, and 60.00 USD per 1M tokens for output.
AWS GPU Server
40.96 USD per x A100 per hour.
Table 6: The estimated cost of different LLMs using the official GPT-4 API and 8 nodes of Ax00
GPU servers for training.
#API Calling (Token)
#Server Time (Hour)
Models
Total Expenses
Input
Output
Data Num.
Synthesizing
Training
KPMath-DSMath-7B
1090
218
865K
39,599 USD
DeepSeckMath-7B-RI
160
52,428 USD
JiuZhang3.0-7B
300
877
10K
14
10
8,480 USD
A
Cost Estimation
To estimate the expenses of previous work and our proposed JiuZhang3.0, we survey the price of potential service during the entire procedure, including calling the OpenAl API for GPT-4 and renting the AWS GPU server for LLMs training. The details of the price are presented in Table For fair comparison, we assume that GPT-4 is utilized to synthesize training data, and 8 nodes of x A100 GPU servers (64 GPUs in total) are leveraged for LLMs training.
For our data synthesis process, the average length of the prompting (including selected math-related texts) is about 300 tokens, and the average length of problems and solutions is 877 tokens. Under the setting of 64 GPUs, we spend 4 hours selecting valuable data for natural language reasoning and tool manipulation in total, and 10 hours synthesizing the 4.6B pre-training corpus. We train the JiuZhang3.0-11 0-BB model for 10 hours, including both pre-training and fine-tuning.
As the dataset and the construction process of KPMath are not publicly available, we adopt the average length of synthesis prompts, problems, and solutions MetaMathQA for estimating the input and the output tokens, respectively. We estimate the training time on 120B tokens for a 7B model 160 hours. Since the training details in the RLHF stage DeepSeekMath- ---R are not publicly available, we do not count the fine- tne-tuning cost for the KPMath model and the DeepSeekMath model. In this case, we estimate the expenses of the LLMs training process as follows,
API Expenses = (Avg. Input Length x API Input Price
(6)
+ Avg. Output Length API Output Price) x Num Data, Server Expenses = Num Nodes x Price per Node x Training Time,
(7)
Total Expenses = API Expenses + Server Expenses.
(8)
The details and estimation of the expenses for different LLMs are present in Table
B Fine-tuning Data
After pre-training, we collect a set of open-source math-related instructions to fine-tune JiuZhang3.0 O.. For natural language reasoning, we collect the training sets of MATH (63), GSM8k (34). CARP [33. and open-source synthetic datasets based on them, i.e., MetaMATH 10., MMIQC 4T.. Math- Shepherd (without problems from original MATH test sets) (64] MATH 11 Besides, we also collect the positive examples from the PRM800k dataset (without problems from original MATH test sets) [35] and the TAL-SCQSK dataset consisting of multi-choice questions Whereas, we observe that the varying data styles in the above datasets might cause the LLM outputs to be irregular. Thus, we utilize a unified prompt DeepSeekMatth7BBBBB to synthesize 700k solutions for the problems from the above datasets.
17

---

For tool manipulation, we use the synthetic datasets, f.e., OpenMathhnssrrutt- 665 and MMOS [66].
consisting of mixture of text reasoning and code blocks executed by Python interpreter.
C Evaluation Datasets
We test our JiuZhang3.0 O and baseline methods in the following two settings for evaluating the mathematical reasoning capability.
Natural Language Reasoning: we prompt LLMs to perform multi-step reasoning via natural lan- guage, and select the following publicly avaibable datasets: GSM8K 34 contains grade school math problems to test the basic arithmetic and reasoning ability. MATH 6S]
and CARP-en 333 consist of complex competition-level problems, and CARP-en is the English version of the original CARP dataset. ASDiv 167), MAWPS [68] and SVAMP [69 are grade-school math word problem (MWP) datasets, and SVAMP focuses on the robust reasoning ability. Besides, we also consider the fol- lowing datasets with different data formats or related to other interdisciplinary field, i.e. TabMWP, AQuA, SAT-Math, MathQA, MMLU-STEM. AQuA (70], SAT-Math [7]), MMLU-STEM [72 composed of multiple-choice questions for human exams across math and other STEM disciplines. OCW-Math [73] a challenging dataset containing undergraduate-
math and science problems.
Tool Manipulation: we prompt LLMS to manipulate external tools via Python to solve the problems, and select GSM8K, MATH, GSM-Hard, SVAMP, TabMWP, ASDiv, and MAWPS for testing. GSM- Hard (77 replaces the numbers in the questions of GSM8K with larger numbers to increase the difficulty of calculation. TabMWP 74 is an open-domain MWP dataset containing tabular data.
D Baseline Methods
We consider diverse types of baseline methods for comparison.
Closed-source LLMs: ChatGPT and GPT-4 (2]:
Larger LLMs (>20B): Qwen-1.5-110B 75 Qwen-1.5-72B [75], Deepseek-LM-677 (76), Mixtral-
8x7B 19]. Llemma-34B Intem-Math-20BB [9, MAmmoTH22-8x7772 -owoooTHHHxxxx-PPPPP0 13 ChatGLM-Math-
32B 777:
Smaller LLMs (<10B): DeepSeck-7B (6). Qwen-1.5-7B [75), Mistral-7B [1], LLaMA-3-8B [8]
Gemma-7B 7B (78). and CodeLLama [79];
LLMs pre-trained on Math Corpus (<10B): Llemma-7B IntehL.M-MMat-7B Rho-I-Matte
7B 80 DeepSeekMath-78 [5]:
LLMs fine-tuned Math Instructions (<108): MetaMatt-MMistral-7 10
WizardMath-
7B-1.1 49), Abel-7B-002 81I. Mistral-7B-MMIQC [41], Math--hhepherdd-Mista---BBB [64).
DeepSeekMath-7B-nnstee (5). DeepSeekMath-7B-RL [5] Llama-3--B--Insrruet
18] MAm-
moTH2 13), KPMath-DSMath-7B (6]
LLMs fine-tuned tool-augmented math instructions (<10B): WAmmoTH--B-Mistee
39
MathCoder-7B-CI 82), ToRA-7B-Code 12), MARIO-OVM-7B [83] MMOS-CODE-7. [66]
OpenMath-Mistral-77 165), Rho- Boao---Math-BBB-C BoCd [80].
Our evaluation framework and in-context examples follow the existing work 6 For general and math domain base models, we adopt the few-shot prompting method. For fine-tuned models, we adopt the zero-shot prompting method for open-ended natural language reasoning and tool manipulation tasks, and the few-shot prompting method for multiple choice problems. We cite the performance results reported existing work 6
E
Implementation Details
Data Synthesis Models. We train two data synthesis models for the natural language reasoning and tool manipulation settings, respectively. We first initialize them by training DeepSeekMath- 7B-RL with 4k and 1.3k KD datasets, respectively. Then, we utilize the synthesis models to generate 100k problem-solution pairs for each one. During value estimation, we adopt the training set of GSM8k and MATH for natural language reasoning, and the 5k subset from mixture of OpenMathInstruct
18

---

Perforncee MATI/GSSM //SSMSS
74
Pe-farmace on MATH/ /CSMSK
79-
72-
MATH GSM8K
73
MATH GSMSK
44
32-
12
30+
40
38
1:1
4:1 NLC:TM Data Patio
2:1
8:1
0k
2k
4k
6k
8k
High-sslue Data Number
Figure 4: Hyper-parameter tuning results of the pre-training data proportion and high-value data amount, under the tool manipulation and natural language reasoning settings, respectively.
and MMOS for tool manipulation, as the instances from downstream math-related tasks. We select 2k the most valuable math texts in each setting for GPT-4 to boost the quality, and then add them into the KD dataset. During training, following existing work [14], we adopt cosine learning rate schedule with 0% warm-up ratio and select a learning rate of le-5 for 5 epochs and 10 epochs for natural language reasoning and tool manipulation, respectively.
JiuZhang3.0. Before training, we first filter out the instance from the synthetic dataset that have 10- grams overlap to the test set data, and also deduplicate data and the synthetic tool manipulation data containing unexecutable code. Then, we follow existing work that trains 7B, 8B, and 8x versions based on Mistral-7B, LLaMA-3-8B, and Mixtral-8- x 7B [13. During training,
first pre-train it on our synthetic 4.6B math problem- soluuioon pairs and then fine-tune it on the multi-source instruction set. We reuse the optimizer to initialize the fine-tuning stage and adopt the Warmup-Stain Stapp--talll--aa- Decay learning rate scheduler [85] with 3% warm-up ratio and 85% stable training ratio for 1 epoch in the whole training process. We set the maximum learning rate to le-5 and the minimum learning rate le-6 with a total batch size of 512. To boost the training efficiency, we pack multiple instances in the context window of the model and modify the attention to avoid mutual interference among difference instances. The maximum length of model is set to 2048. We train all models with BFloat 16 numerical format, Flash Attention 2.0, DeepSpeed Stage 2 for 7B and 8B models, and Stage 3 for 8B models.
F Hyper-parameter Tuning
In this part, we conduct the experiments about tuning important hyper-parameters, i.e., pre- training data proportion, and the number of high-value data.
Pre-train Data Proportion. In the synthetic pre-training data proportion, we should determine the ratio between the data from the natural language reasoning and tool manipulation settings (NLC TM). We set it to 1:1, 2:1, 4:1, and 8:1, and control the total data amount unchanged (200k) for comparison. We follow the efficient test setting Section 4.3 and report the MATH and GSM8K results under the tool manipulation setting, as there is only slight performance changes on natural language reasoning setting. As shown Figure 4 2:1 is more suited and leads to better performance than other proportions. The reason may be that smaller or larger proportion for tool manipulation data would cause underfitting or overfitting on the tool manipulation data, affecting the corresponding capability.
Number of Selected High-Vade Value Data. We also study the effect of changing the number of selected high-value data according to the ranking, as more data requires larger cost for invoking the GPT-4 API. We set it to 2k, 4k, 6k, and 8k, and do not change other settings for fair comparison. We follow the efficient test setting in Section 4.3 As shown Figure 4 using the top 2k high-value data achieves the best performance, and more data even causes the performance degradation. It indicates
19

---

that the data quality is more important than data amount for boosting the data synthesis capability, as using more data would inevitably involve low-quality data.
G
Limitation
First, although train strong yet small LLM that synthesize high-quality math problems for training, its capabilities on synthesizing the data for other domains or tasks might be relatively weaker. The reason is that only the math-related data to train it. In future work, we will try to train a general-purpose model, to enable the data synthesis for other requirements. Second, this work, only focus mathematical reasoning capability, and trained JiuZhang3.0 is also mainly for solving math problems. Limited by the computation resource, we do not test its performance other complex reasoning tasks, e.g., planning, commonsense reasoning. We will also conduct the corresponding experiments in the future. Third, the 4.6B pre-training data is still a large scale for training, and do not perform data filter to control its quality. Future work should focus on reducing its scale by proposing better data filter strategy. Forth, we only utilize GPT-4 for knowledge distillation, but do not use other well-performed LLMss, e.g., Claude 3, GLM-4, and the latest GPT-4o. More experiments should be conducted on these LLMs to study the effect of the teacher LLMs. Fifth, for cost estimation, our reported results are all estimated according our experience, without the c--implementation of these methods. We also appeal to report the true cost of training LLMs in existing work.
20

---

H
Case Study
Good Case
Type: Natural Language Reasoning Grade School Math-Related Texts:
Question
The Huka Falls on the Waikato River is one of New Zealand's 's most visited natural tourist attractions (see Figure 12.30). On avorage about 300,000
averages deep. What average speed of the river the gorge? (b) What is the average speed the water the river downstream of the falls when it widens to m and dopth increases an average of m²? Question Image
gorge,
narrows
1. 0.75 ms
2. 0.13
Solution Vidcoft OpeaStax College Physics Solution, Chapter 12, Problem (Problems & Exercisos) (1:14)
sample solutionat Calculator Screenshots
Video Transcript
This College Physics Answers with Shaun Dychko, the gorge where the Walkato River of New Zealand narrows width of and height 20 meters, we2019tc fre going figure out what its speed And sa the volume flow equal to the sectional area of the river multiply hy
Synthesis
Question: The Waikato River flows through gorge where it narrows down width of 20 meters and depth of 20 meters. The average flow rate the river 300,000 liters per second.
s* =*Question ** What is the average speed of the river in this gorge?
Ansser:
To find the average
the river in the gorge, need the formula for the volume flow
Volume Flow Rate Cross-Soctional Speed
First, lei201 calculate the sectional of the river:
Cross-Sectional Area Width x Depth
Given:
Width 20 meters
Depth meters
So.
Cross Sectional Area 20 meters
meters 400 square
Now, we know the volume flow rate and the cross sectional area. The volume flow rate (Q) 300,000 liters per second, which we need into cubic because square
cubic decimeter
So,
Q 300, 000 liters second 300, 000 0.001 cubic meters per second 300 cubic meters second
now
flow speed (1:):
Pross-Scctional Area x
Rearnnging
Cross- Cosss-Cctional Area
Substituting
300 cubie meters per second
meters per
400 square meters
So, the average speed of the river in the gorge **0.75 meters per *
21

---

Good Case 2
Type: Natural Language Reasoning Middle
Math-Related Texts:
Successive parabolic interpolation
Successive parabolic interpolation lechniqque linding extremum (mininmmm muximumi continuous uninnodal function by successively fiting parabolas (polynomials of degree twa) to function of one variahle at three unique points in general, a each iteration replacing the oldestpoint with extremum the fitted parabola.
function variables at I+n(n-332 points, Advantages
Only lunction values are used, and when this method converges to an extrenuum it does so with an order of convergence approximately 1.325. The superlinear rale convergence superior to that other methods with only linear convergence (such as line search). Morcover, not requiring the computation 4pproximation of function derivatives makes successive parabolic method).
interpolation popular allernative other Disadvantages
that do require them (such as gradient descent
the other hand, convergence (oven
Synthesis QA:
Question: Alex and Jamie are working on science project and need to find the maximunn height of rocket launched upward. They record the height of the moket three different times
f O secends, the height 0 meters. seconds, height
meters.
At 10 seconds, the height is 40 meters.
To fied the maximuum height, Alex and Jamic decide to use successive parabolic interpolation. They start by fitting parabola these three points and then linding the maximum point of this purabola.
1. Write the general form of a qundratic (paraholic) equation: y au? + har + 2. Use the three points create a system equations find the coefficients h and 3. Find the vertex the parahola determine the maximum height.
Anszer:
1. **eneral Forn of Quadratic Equation:
The general form of a quadratic equation
= +
2. **Creating System of Equations:**
We have three points:
(0,0), (5,25), (10,40)
Plugging these points into equation For (0, 0):
az bu +
0= a(u)" + b(0) +
For (5, 25):
25 a(5)² + b(5) = 25 25a + 5b (Equation 1)
For (10;
40 a(10)² b(10)
40 100a + 106
(Equation 2)
3. *"Solving the System of Equations:** Use Equation
25 = 25a + 56 (Equation 1)
Simplify:
5a
(Equation 3)
Use Equation 2:
10 = 100a + 105 (Equation 2)
Simplify:
4 10a (Equaation
Subtract Equation from Equation 3:
(5a 6) (10a +
-5
Plug back Equatiin
5 = 5
-5)
+
5 -1+ b
22

---

Good Case 2
4. *Veertex of the *
The veriex parabola axr bu + is given by:
b
2a
Here, = -g. G, and O:
6
15
(--)
So, maximum height occurs
seconds. However, value exceeds range data points. In practical
interpolation methods, this would indicate corrcction point needed beyond the given data. But for demonstration, let's assume 15 within bounds (extrapolation for students):
Plug back into original equation adjusted coefficients:
(15)² 6(15) (225)
y
5
Thux, the maximum height +h45 m
Good Case
Type: Natural I Langunge Reasoning High
Math-Related Texts:
Let he positive integer. Find all polynomial with real coefficients which satisfy the equation P(P(x)) (P(r))) hea positive integer. Find all polynomials with real coefficients which satisfy the equation
P(P(x)) (P(x))"
simply don't even know how think about this problem. I've tried simple stuff just to my head on the problem. For example for P(ar)
P(P(a)) (P(x))"
think any polynomial P(ar) R. After that I simply stare at the problem. Can guys give some help ? Note: I would
2t + ...
solution
as I would have P(x) = (()))1(2) like to understand how to lackle these kind problems, so would be really grateful if you would explain the thinking process behind solution. (This optional, feel free give
you prefer.) Thanks advance.
First we note that only the constant polynomials P = and P 1. as well als P = for odd k. satisfy the equation. (This any constant polynomial do.) every y E
rango (P(x) R} P(y) y which means the polynomial Q(2) P(x) x Since Y infinite, this implics that the polynomial. In conclusion, P(x) a
constant,
inlinite
Synthesss QA:
Question: Consider the polynomial P(x) x²
how beluves we apply twice, ((P(a))) and
compare P(ar)
some power, (P())))*
Let's explore the following scenario:
Given P(r) 22, find all values of k such that the equation P(P(x)) = (P(x))** holds for all real numbers x. Ansver:
P(P(x)):
P(P(x)) = P(x?) )
Next, we need express P(x) raised to some power k:
(P(x))) = (2²2²² 20
We want P(P(z)) (P(a))*:
2k
This equation holds for all real numbers if and only if the exponents are equal. Therefore, we set up the following equation:
21
Solving for &::
k=
Thus, the squation P(P(x)) (P(x))* holds & 2. Now let's verify this:
P(P(x)) = (P(#))* = (13/3=
Since both sides of the equation are equal:
This equation holds true for all real numhers Therefore, the only value of that makes P(P(z)) = (P(x))* true for all real numbers 2.
23

---

Good Case 4
Type: Tool Manipulation Grade School
Math-Related Texts:
Refer to unit in the Units package by indexing the name symbol with the context. for example, bitlinformation or bpslinformation]
the by using only units of information are defined as follows. crumh is defined as hits. A nibble defined as 4 bits.
symbul, example, bps.
Synthesis QA:
Question: digital communication system, information tramsmitted in units called bits. A crumb defined as bits, and a bits, and the sequence broken down into whole crumbs and nibbles, bow many different ways can the user send exactly 18 bits?
nibble is defined 4 bits. If user sends
Answer:
hount_ways__a tettttto__stsaaaaeeeeeeee bittttitllttttttaa
ways 0
for crumb in range(total_ (totallbiita // 2 1):
for nibble in range(total_bigg // 4 1): if crumb 2 nibble ways
cotal_bita
return ways
total_bits 18
ways__to_sent count_ count_ways_to_eeee htt.://aww._..eee..icttotoaaaaaa bits bitts) Frinttwwas_to_Bat
"oontput
There are 5 different ways the user can send exactly 18 hits.
24

---

Good Case 5
Type: Tool Manipulation Secondary School Competitonn Math-Related Texts:
The generalization of this 0: I = [l)) Mcan and variance of continuous random variables
the of " density probubility function
point. J= #p(x)
The variance the sum of distance to the mean times the density probability function at that point. d² -- (a (4) p()r()))))))))
Esporance of a probability distribution is the value we could expect to have if we repeat some experiment lot of times. It's the sum of the value the events ponderated by probability
Synthesis QA:
Question: Consider continuous random variable X with probubility density function (pdf) given p(x) fee a
1. Calculate the mean (μ4) of the distribution.
[brif < 0
2. Calculate the variance (o²) of the distribation.
Anever:
import sympy as sp
# Deféne the variable and the påf ap.Symbol ('x', real-True, positive=True) pdf (1/2) sp. ex.ex(2xx/2)
Calculate the mean mean sp. ixtegrateeex pdf, (x, O, sp))
Calculate the second moment about the origin
second moment = ap. integrate (I**2 pdf, (x, 0, sp.oo))
Calculate the variance variance second_mone moment mean**2
mean value mean. evalf() variance, value = variance. evalf
mean mean_valuee value, variance_ value
(.00.4.)
mean (j4) distribation
variance (a distribution
25

---

Bad Case 1
Type: Natural Language Reasoning AMC12
Math-RRelted Texts:
Julia Robinsen Mathematics Festival
organization sponsoes locally organized mathemalics festivals and online wehinars targeting K- K--12 students. 'The events are designed to introduce students to mathematics in a collaborative and non-competitvve forum. [[][2][3]
Robinsen Mathematics Festival (JRMF) an
Abbreviatoo/RME FounderNancy Blachman Location c/o SEE. 23564 Calabasas Road, Suite 201. Calabasas. CA 91302 Coordinates37" 48.57"N 121 54'25.98" Key people Nancy Blachman, Founder Daniel Kline, Executive Director Website- jrmf.og
History
Mary's College Califoniia produced mathematics contest was popular secondary schools throughout the San Francisco Bay Aren. [4] In 2005, Nancy Blachman attended an education forum sponsored by the Mathematical Sciences Research Institute (MSR) and remembered how Saint Mary's contest had inspired her as student, [5] Unfortunately, contest longer existed. Seeking possibl
the
Synthesis QA:
Question: Al the Julia Robinson Mathematics Festival, students are challenged with geametry puzzle. The problem is as follows: length inseribed circle. AD as C. and Q on the same side of line A B as C. If AP 16 and AQ 12, find the length of PQ. start by finding radius properties of circles and the given lengths determine the length of PQ.
square ABCD
Q
that the same line
Anewer: solve this problem.
circle, using the fact that square inseribed cirele.
Then,
1. *FFinding the radius
circle:se
circle, the diameter of the circle is equal the diagonal of the square. The diagonal of square with side length can be calculated using the Pythagorean thearem:
Since the square BCD is inseribed
Diagonal 2v2
For the given square with side length 10, the diagonal is:
Diagonal 10v2
Thus, the radius of the circle, which is half the diagonal, is:
10v2
Radius
5v2
2. **Using the Power
Theorem:**
The Power of'a Point Theerem states that for point P and a point Q in circle with center O and radius r. P and Q are on the passing through
product
Q CD equal to the square
distance from O C.D. In our case, CDD the diameter the circle, and O is the center circle. Point is 16 units from A on side AD, and Q is the circle and the diameter passes through the center
units from A side AB. AP 16 and 12. Since P and O, the Power of a Point Theorem gives:
AP AQ Op
Op
OP = /192 = 8v3
3. *"Determining the length of PQ:** Finally, the radius the circle and the calculated power of point distance to find (2P determine PQ. Since O the midpoint of PQ and OP OQ = 8v3:
PQ 2 OP 2 8v3 16v3
Thus, the length of PQ is 16 16 units. This problemi challenges
geometric principles (properties circles squares) algebraic reaseming solve lengths in a geometric configuration, enhancing analytic and problem-solving skills suitable for the AMC 12 level.
26

---

Bad Case 2
Type: Tool Manipulation Secondary School Competitonn
Math-Related Texts:
ONick D. Jan 5 '17 at 14:48
guess comes
you
ORenascenceS. '17
HINT:
Observe that function
interval [-6, 6j symmerric more in this question: Definite intogral of an odd function (symmetric interval)
+
2n
1= x +1)
have integral of form
x²
(x² sin zdx
clearly that
a²
f(x)
siu
f(-x) = ((())
1)2
L.c.
Guidelines for each section:
Syuthesis QA:
Question: Consider the function ((x (=#41)2 sin? x over interval [-5, 5]. We are interested in evaluating the definite
integral this function with respect Given the propenies of the function f(rr). which add, and symmetric interval about the origin, determing the value of the integral.
Anszer:
import sympу sp
Define the variable ap. symbols('x')
Define the function
f (x**3) (x**2 + 1)++2 * sx.)in(xx**2
# Since the function is odd over symmetric interval, the integral is zero integral_ ap. ip.integrateff, (x, -5, 5))
print (integrall
"'oulpu
The answer
I Prompts
L1 Prompts for Natural Language Reasoning Data Synthesis
Prompts for Grade School-Level Problem
Instruction
Create an age ape-apppoeeeite math word problem for grude school students hused on the provided math content.
Math Content
[Math Text Placebolder]
Guidelines
[Problem]: Crafl concise math word problem suitable for grade school, focusing basic arithmetic operations (additionn, subtraction, multiplication, division), number sense, simple shapes, or introductory measurements. Use relatable, real-woeld scenarios appropriate the group. Ensure problem purely text-based and solvable without images. [Solution]: Provide clear, step-by-step solution to the problem using simple languge that a grade school student could understand. Explain the reasoning behind each step.
27

---

Prompts for Middle School-Levv Problem
Instruction
Create an middle school level math problem and solution based on the provided math content excepp
Math Content
[Math Text Placeholder]
Guidelines
[Problem]: Create self-contained problem for middle school student that directly incorporates concept provided math content. Target difficulty level uppeopriate for grades 6-8 lages 11-14), asuming knowledge of arithmetie,, pre pre-algebra. busic images. concepts typically
probubility/siaaiiiiss and geometry. Ensure problem fully based covered by the end of 8th grade.
[Solution]: Provide detailed, stop-by-stop solution demonstrares mathematical reasoning from problem statement conclusion, around 250-350 words long. Utiizee LaTeX formatting for mathematical expressions. Explain each step to reinforce underlying math principles being applied.
Prompts for High-Level Problem
Instruction
Inspired by the provided math content extract, create high school level math problem that combines concepts from least math subjects.
Math Content
(Math Text Placeholder]
Guidelines
[Prohlem]: Draft self contained math problem for high school students based the math content. The problem should require knowledge from one of these subjects: Algebra and 11. Pre-Calculus Calculus, Geometry. Trigonometry, Statistics and Probability. problem fully text based and salvable without images. concepts typically covered by [Solution]: Provide detailed, step-hy-step solution that demonstrares the mathematical reasoning from problem statement to conclusion, around 250-350 words long. Utilize LaTeX formatting mathematical expressions. Explain cach stop to reinforee undertying math principles being applied.
grade.
Prompts for High-Leve Problem
Instruction
Inspired by the math content, create college-level math problem.
Math Content
[Math Text Placeholder]
Guidelines
[Problem]: Draft self-contained, college-level math problem inspired by the math content. should be intellectually stimulating and designed for un audience lamiliar with advanced mathemsticcs. such as Calculus, Linear Algebra, Abstract Algebra, etc. Ensure the problem includes all necessary information sulving Aim for problem statement around 100 150 words, [Solation]: Provide step-by-step solution to your problem, around 250-350 words long. The solution should clearly explain the reasoning, mathematical principles, and stops used. Call any theorems properties being applicd each step.
Prompts for AMC 8-Level Problem
Instruction
Inspired by the provided math content, craft math problem suitable for the AMC competition, engaging top grade students with challenge arithmetic, algebra, counting, geometry, logic, number theory, or probability. Ensure the problem stimulates problem-solving techniques and heuristics within text-based and solvable format, avoiding advanced subjects like calculus or physics.
Math Content
[Math Text Placeholder]
Guidelines
[Problem]: Design compelling, self-contained problem
contestants inspired by content, incorporating
elements of basic math disciplines. The problem should be approachable through logical reasoning and fundamental problem-solvinn strategies. Ensure the problem fully lext-based, solvable without the images, and has difliculty level appropriate for the AMC competition.
[Solution]: Provide detailed, hy-pehy-step solution that educational and tailored to the AMC 8 audience, around 250. 350 words Explain the logic reasoning behind each step thoroughly, using and ago-appropriat language and tenninology facilitate understaning. Highligt the use of cobblem-solving techniques and bouristics employed in the solution.
28

---

Prompts for AMC 12-1.vvll Problem
Instruction
Inspired by the provided math content, develop math problem suitable for the AMC 12 competition, targeting high-school students with cupabdities advanced areus algehru, geomeiry. trigonomeeryy. counting. probubility. and theory. problem should challenge students to utilize sophistcateed prohlem-solvin skills and mathematical reasoning. Focus concepes from the given content, but adapt them into an original problem. Avoid delving into higher-level college mathematics topics beyond AMC 12 scope.
Math Content
[Math Text Placebolder]
Guidelines
[Problem]: Formulate compelling and complex math problem for AMC participants inspired by the provided math content. The problem should encourage students employ advanced logical reasoning and problem- stralegies related given concepts. Cruft it to solvable in the AMC 12 formal and difficully level.
comprchensie, step-by-step solution that words kong. Clearly articulate the reasoning and methods used al each step. providing insight into the problem solving process. Use format any equations properly
[Solution]:
problem and
student. around 250-350
language challenges yet instructs high using LaTeX appropriate notation.
looking improve their skills,
Prompts for AIME-Level Problem
Instruction
Inspired by the math content, math problem appropriate for advanced high school mathematics competitions like the AIME. The problem should challenge students in core areas lested on AIME such algebra, geumelry, comhinatorics, number theory and prohability. Encourage creative problem-solving and mathematical thinking using pre-caleulus level techniques. Avoid delving into college-level topics like abstract/lincar algebra, topology, multivariable calculus, or advanced physics. Any physics concepts shoold be kept
mechanics
Math Content
(Math Text Placeholder]
Guidelines
[Problem]: Design an insightful and challenging problem suitable for AIME participants inspired by the math content. Focus on care AIMB lupies like algebra, geometry, combinatorics, number theory and prubability. The problem should solvable using clever applications of high school math and mathematical reasoning, without requiring knowledge of advanced college level mathematics or physics. Aim for a difficulty level on par with actual AIME questions. [Solution]: Provide detailed. Hep-by-stey solution that would be enlightening applied each step. Use language that is mathematically sophisticated yet understandahle to well-prepared AIME participant. Justify any key insights creative leaps in the solution process.
advanced high school student, around 250-350 words Explain the reasoning
and techniques
1.2 Prompts for Tool Manipulation Data Synthesis
Prompts for Tool Manipulation where Grade School or Secondary School Competition Level is Determined by Example
Instruction
Please gain inspiration from the following random math content to create higb-quality math problem and solve it with Python code. your output
sections: [Problem Description] (Solution].
Math Content
[Math Text Placeholder]
Guldelines
1. [Problem Description]: This should be **rompletely self-contained providing all the contextual information one needs to
understand solve the problem.
2. [Solution]: Offer a comprehensive, **correc*** solution that accurately addresses the [Prohlem Description] you provided using
Python code. Summarize in natural langunge in the end and pul the answer in /boxed. Example
[Problem Description]
[Example Question Placeholder]
I/Problem Description)
[Solution]
(Example Python Solution Placcholder]
[/Solution]
29

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: According to the contributions proposed in Section [] for the the first contribu- tion, please refer to the Section 3 and Section 4.3 for the second contribution, please refer to the Section and Appendix A for the third contribution, please refer to Section 4.2
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
30

---

Justification:
Please refer to Appendix G
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sellispecifiitionn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We illustrate the necessary details of implementation, including details of prompting, training, and evaluation. Please refer to Appendix B Appendix C Appendix [
Appendix B Appendix[]
Guidelines:
31

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We promise the code and the training data will be publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
wtt.ssumiiiiiioossosoooioioieiie for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https cttips.cc/public//uue
CodeeSumiiii wwabbbmmsssiinni ssisnooiccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
32

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Please refer Appendix B Appendix Appendix
D and Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our evaluation experiment is conducted under zero temperature for generation to avoid randomness, We report 18 different settings of evaluation datasets to demonstrate the performance.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
33

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: htttps://meet httpp//pcc///tthccsuuueeeeeeeee
Answer: [Yes]
Justification: Our research is focused on LLMs for math reasoning, conforming with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our research is focused on LLMs for math reasoning, which will not cause societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
34

---

Answer: [NA]
Justification: Our research is focused on designing LLMs for math reasoning which will not have high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes)
Justification: Please refer to Appendix D Appendix Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pwpprswithhaeecoccm..pppp.... thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pace re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The released code, data, and models are well documented including details about training, license, limitations.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
35

---

14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "AzYIRcP6SF", "reviewer": "Reviewer_1oiT", "summary": "The paper addresses the high cost of training large language models (LLMs) for mathematical reasoning by proposing a more efficient method for generating high-quality pre-training data using a smaller LLM. Instead of relying on extensive pre-training data or using powerful models like GPT-4 for large-scale synthesis, the authors train a small LLM to synthesize math problems. They create a dataset by distilling GPT-4's synthesis capability into the small LLM using carefully crafted prompts covering various educational stages. Additionally, they employ a gradient-based influence estimation method to select the most valuable math-related texts for creating a knowledge distillation dataset. This approach significantly reduces the need for GPT-4 API calls and large-scale pre-training data. The resulting model, JiuZhang3.0, achieves state-of-the-art performance on several mathematical reasoning benchmarks.", "questions": "1. Table 1 shows that as the model size increases, the performance does not improve significantly, and some metrics even decline.\n2. The accuracy of the pre-trained synthetic data raises concerns about the noise introduced by numerous erroneous samples.\n3. Performance comparison of pre-training using original data versus synthetic data.\n4. Differences between using gradient descent and semantic similarity methods to filter high-quality data.\n5. Both Table 1 and Table 3 indicate that reasoning with CoT and Tool assistance yields similar performance.\n6. The loss calculation method for the pre-trained model and its few-shot performance on some benchmarks, like GSM8k and MATH.\n7. Figure 3 illustrates that as the data volume increases, the performance of two base models initially rises and then falls, rather than continuously improving.\n8. Is fine-tuning Llama3 more suitable for your proposed synthetic method compared to Mistral? Which is more appropriate during the SFT phase: Llama3 or Mistral?\n9. Why was 10k selected for training the synthetic model? Could more data be used?\n10. Comparing the synthetic data capabilities of different LLMs in Table 4 seems unfair, as the study used distilled GPT-4 for synthetic data generation.\n11. The performance did not significantly improve when comparing different data selection strategies in Table 4.\n12. The effect of using SFT data on the Mistral-7B base model.\n13. There is an error in citation [50].", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed methodology for enhancing mathematical reasoning in LLMs through cost-efficient data synthesis has several advantages: 1. Cost Efficiency: Utilizing a smaller LLM for data synthesis drastically reduces training and inference costs, requiring only 9.3k GPT-4 API calls; 2. High-Quality Data: The approach ensures high-quality synthesized data by using carefully crafted prompts and selecting the most valuable math-related texts through gradient-based influence estimation; 3. Scalability: A small LLM can efficiently generate a large dataset (4.6B tokens) for pre-training, making the method scalable; 4. Performance: JiuZhang3.0 achieves state-of-the-art results on various mathematical reasoning datasets, outperforming larger models in natural language reasoning and tool manipulation settings; 5. Generalizability: Demonstrates that smaller models can effectively learn and adapt to complex tasks with proper training strategies, applicable to other domains beyond mathematics. The paper is well-written.", "weaknesses": "Some details of the paper are not clear, please refer to the following questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the insightful suggestion and positive feedback, and list our responses for the weaknesses 1-5 below:\n\n**W1**. Table 1 shows that as the model size increases, the performance does not improve significantly, and some metrics even decline.\n- **Reply to W1**. As shown in existing work (e.g., LLaMA-3-8B and Mistral-7B), small LLMs can also perform pretty well by training on larger high-quality corpus, even outperforming larger LLMs. Therefore, the performance does not always consistently increase w.r.t. the scaling of model size. This phenomenon can also be seen on other baselines in Table 1 (e.g., Qwen-1.5-110B and DeepSeekMath-7B-RL). Moreover, as the used backbones are not trained with the same data, their capacity and compatibility for our approach would also be different. But according to the average performance on Table 1,2,3, they roughly follow the tendency that JiuZhang3.0-8x7B > JiuZhang3.0-8B > JiuZhang3.0-7B.\n\n**W2**. The accuracy of the pre-trained synthetic data raises concerns about the noise introduced by numerous erroneous samples.\n- **Reply to W2**. Actually, it is inevitable that noisy samples are involved into the pre-training data of LLMs, e.g., erroneous and toxic ones[1,2]. Thus, the following SFT process is widely-used to reduce their influence and align LLM with humans. Therefore, after pre-training, we also collect a set of math-related instructions with reliable accuracy for fine-tuning LLMs. \nIn our empirical experiments, we have found that such a way can achieve similar performance as filtering the erroneous samples in pre-training data. Concretely, we utilize a well-trained LLM to annotate the accuracy of all the pre-training data, then only select the top-ranked 500k samples for pre-training Mistral-7B. Besides, we also randomly sample 500k samples for pre-training for comparison. After pre-training, we fine-tune the two methods on 50k instructions. As shown in the following table, the two variations achieve comparable performance, indicating that the noise is not an important concern.\n\n[1] Ouyang, Long, et al. \"Training language models to follow instructions with human feedback.\" Advances in neural information processing systems 35 (2022): 27730-27744.\n\n[2] Albalak, Alon, et al. \"A survey on data selection for language models.\" arXiv preprint arXiv:2402.16827 (2024).\n\n|         | GSM8k | MATH |\tSVAMP\t| ASDiv | MAWPS | CARP | Avg. |\n|------------|------------|------|---------|------------|------------|------|---------|\n| **PT Random 500k + SFT 50k** | 83.2 | 38.4 | 84.9 | 88.3 | 94.8 | 43.6 | 72.2 |\n| **PT Top-Acc. 500k + SFT 50k** | 82.7 | 40.8 | 84.1 | 87.9 | 94.2 | 42.2 | 72.0 |\n\n**W3**. Performance comparison of pre-training using original data versus synthetic data.\n- **Reply to W3**. Following the suggestion of the reviewer, we conduct the experiments that using the original math-related data as the pre-training corpus, and also reuse our SFT data. We train Mistral-7B on the above data, and report the results in the following Table. We can see that the models using our synthetic data perform consistently better than using the original data. It indicates that our synthetic data owns higher quality and is more useful for improving the mathematical reasoning capability. \n\n|         | GSM8k | MATH | SVAMP | ASDiv | MAWPS | CARP |\n|------------|------------|------|---------|------------|------------|------|\n| **Original PT text** | 36.9 | 15.4 | 60.6 | 66.5 | 82.6 | 26.2 |\n| **Original PT text + SFT** | 86.1 | 44.4 | 87.8 | 90.2 | 97.2 | 48.1 |\n| **Our PT text** | 68.7 | 40.2 | 80.4 | 86.5 | 93.5 | 40.2|\n| **Our PT text + SFT** | 88.6 | 52.8 | 90.4 | 92.6 | 97.3 | 51.0|\n\n**W4**. Differences between using gradient descent and semantic similarity methods to filter high-quality data.\n- **Reply to W4**. Following the suggestion of the reviewer, we implement a semantic similarity based methods for data selection. Concretely, we utilize GTE, https://huggingface.co/Alibaba-NLP/gte-large-en-v1.5, a well-performed sentence embedding model  to encode 1M candidates into embeddings, and then compute their semantic similarity with the downstream task data via cosine similarity. Finally, we select top-ranked 100k samples for pre-training, and then fine-tune on 50k instructions. As shown in the following table, our gradient-based method can achieve better performance, indicating its effectiveness for selecting high-quality data.\n\n|         | GSM8k | MATH |\tASDiv | CARP |\n|------------|-----------|--------|------------|------------|\n| **Ours + Embedding**\t|78.2|\t29.2|\t84.8|\t31.9|\n| **Ours**\t|78.6|\t32.8|\t84.5|\t36.2|\n\n**W5**. Both Table 1 and Table 3 indicate that reasoning with CoT and Tool assistance yields similar performance.\n- **Reply to W5**. Chain-of-thought (CoT) and tool manipulation are two different capabilities for solving reasoning tasks. Existing work mostly focuses on eliciting one of them for 7B LLMs, as it is hard to well master both. As shown in Figure 4 left, the improvement of one of the capabilities, might lead to the degradation of the other one. Therefore, in our work, we aim to ***balance the two capabilities***, and enable our LLM to ***master both well***. To this end, we carefully adjust the proportion of the two kinds of data (see Figure 4 left). Consequently, our LLM can perform better than baselines in the two settings with a balanced performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n0789gQh0v", "reviewer": "Reviewer_FBVa", "summary": "This paper proposes a framework for training large-scale mathematical reasoning models using synthetic data, introducing JiuZhang3.0 based on this framework. Compared to manually collected or LLM-generated training data, this framework significantly reduces resource costs. The authors trained three different scales of JiuZhang3.0 models based on Mistral-7B, LLaMa-3-8B, and Mixtral-8*7B, demonstrating the effectiveness of this framework through extensive experiments.", "questions": "1. Regarding the title. I think the contribution of this paper is a framework that can generate high-quality training data, not the model itself. The title JiuZhang3.0 may mislead readers into thinking the core contribution is the model. I don't understand why the authors chose this title.\n2. It seems that Initialize Synthesis LLM and Boost Synthesis LLM can be iterative. Can multiple iterations train a stronger Data Synthesis Model? Would this help improve the performance of JiuZhang3.0?\n3. Regarding lines 135-136, the math-related texts used by the authors. Could the authors provide a more detailed explanation of how these texts were obtained? The ablation experiment results indicate it has a significant impact on the overall framework, so I think it is necessary to provide a detailed explanation.", "limitations": "Yes. The authors adequately addressed the limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Proposed an effective training framework that can generate a large amount of high-quality training data with only a few GPT-4 calls.\n2. This work compares various open-source (Qwen series, Mixtral-8*7B) and closed-source models (ChatGPT, GPT-4), and conducts experiments on 18 datasets including GSM8k and MATH, with very rich experimental results.", "weaknesses": "1. Novelty: Overall, I believe the innovation of this work is limited. The core of this work is a framework for generating high-quality training data. However, using GPT-4 to generate data to fine-tune smaller models is not innovative, as shown in [1]. Additionally, in the Boost Synthesis LLM stage, the work essentially just uses the methods from Xia [2] et al., lacking its own innovation. This framework seems more like a fusion of multiple works.\n2. The authors conducted many ablation experiments, but there is a lack of in-depth analysis. For example, why is directly using GPT-4 to generate questions and answers not as effective as the original method? In the choice of Synthesis LLM, why is there such a large discrepancy? I find the authors' explanations vague.\n3. There are some minor writing flaws worth noting, such as the uniformity of m_l and the capitalization in equation 3 on line 163.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the insightful suggestion, and list our responses for the weaknesses below:\n\n**W1**. Novelty: Overall, I believe the innovation of this work is limited. The core of this work is a framework for generating high-quality training data. However, using GPT-4 to generate data to fine-tune smaller models is not innovative, as shown in [1]. Additionally, in the Boost Synthesis LLM stage, the work essentially just uses the methods from Xia [2] et al., lacking its own innovation. This framework seems more like a fusion of multiple works.\n- **Reply to W1 - Part1**. In this work, we aim to verify whether we can efficiently improve mathematical reasoning capability of LLMs by training small data synthesis models. As the reviewer mentioned, existing work has used GPT-4 to generate data for fine-tuning LLMs to improve their mathematical reasoning capability. However, it needs GPT-4 to generate a number of samples for achieving SOTA performance (e.g., KPMath-DSMath-7B[1] using GPT-4 865k times), which is not an efficient way and approximately spends 40k USD. In this work, we train a small LLM specially for data synthesis, and only need GPT-4 to synthesize few samples (i.e., 10k) for distilling the data synthesis capability into it. Our work demonstrates that such a way is applicable and can achieve new SOTA performance with only 20% cost (i.e., 8.5k USD), shown in Figure 1. Thus, our work is promising to inspire other researchers for exploring how to further reduce the cost for LLM to learn new capabilities, advancing the development of low-carbon LLMs and green AI.\n- **Reply to W1 - Part2**. Besides, the major technical contribution of this work is that we propose a low-cost solution for distilling the data synthesis capability from GPT-4 into small LLMs. To this end, we craft a set of prompts corresponding to human education stages for ensuring the knowledge coverage, and adopt an influence estimation strategy to select high-value math-related texts for feeding into GPT-4. The influence estimation strategy is indeed borrowed from existing work, but we find that the original method can not be directly applied into our approach in an efficient way. Therefore, we devise the following optimizations: (1) applying the selection method on math-related texts instead of GPT-4 synthetic data to reduce the cost; (2) adding the first stage for initializing synthesis LLM to guarantee the quality of candidates. In this way, we can ensure both the effectiveness and low cost of the influence estimation strategy in our work.\n\n**W2**. The authors conducted many ablation experiments, but there is a lack of in-depth analysis. For example, why is directly using GPT-4 to generate questions and answers not as effective as the original method? In the choice of Synthesis LLM, why is there such a large discrepancy? I find the authors' explanations vague.\n- **Reply to W2 - Part1**. Thank you for the constructive comment. Due to the page limit, we do not deeply analyze the experimental results. Here, we will do it and add the analysis in the revised version. \nFor ablation study, the performance of the variations that remove prompt set or math-related texts degrades a lot. It is because the prompts can control the synthetic samples corresponding to human educational stages, and the math-related texts contain rich math domain knowledge. Both are important to guarantee the knowledge coverage and quality of the synthetic data. Besides, without using GPT-4 for boosting, the variation degrades a lot in MATH. The reason is that it is not easy for the small LLM itself to synthesize complex math problems that are helpful for the MATH dataset, without the help of GPT-4.\n- **Reply to W2 - Part2**. For synthesis LLM, we select the four models owing to their better evaluation performance in Table 1,2,3, which implies that they own better math reasoning capability and may perform well on math problem synthesis. Among them, the variation direcly using ChatGPT performs not well. A possible reason is that ChatGPT is not specially trained for mathematical reasoning, its capability of math data synthesis is relatively weak. Furthermore, DeepSeekMath-RL-7B also performs not better than our approach. It indicates that distilling the data synthesis capability from GPT-4 is necessary for adapting math-related LLMs into proper data synthesis model.\n- **Reply to W2 - Part3**. For data selection, the variation using perplexity does not perform well. According to our case study, we find that the samples with lower perplexity are typically easier ones, which are not very helpful to learn to solve complex math problems. Additionally, the variation using one-hot ICL performs relatively better. We find the ICL loss are prone to assign high values for the solutions with more natural language tokens, which is more useful for GSM8k but not good for MATH with more math symbols and formulas. \n\n**W3**. There are some minor writing flaws worth noting, such as the uniformity of m_l and the capitalization in equation 3 on line 163.\n- **Reply to W3**. Thank you for your carefully reading. We will fix your mentioned writing flaws, and carefully check and refine the writing of our paper in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VRXhsyo2ZP", "reviewer": "Reviewer_trb5", "summary": "This work proposes a low-cost yet efficient method to synthesize QA pairs using a small LLM (e.g., DeepSeek-Math-7B model) instead of GPT-4, significantly reducing cost and improving performance in mathematical reasoning.\n\nConcretely, it begins by initializing the data synthesis LLM through distillation from GPT-4 using randomly sampled data. It then enhances the model using high-value data selected via a gradient-based value estimation strategy and regenerates QA pairs from GPT-4, ultimately synthesizing around 6M QA pairs on diverse math pretraining corpus.\n\nThe synthetic QA pairs are applied to pretrain open-source LLMs, which are then fine-tuned on collected SFT data, resulting in improved performances across benchmarks such as GSM8K and MATH.", "questions": "- Regarding Weakness 2: Is there a fair comparison with Deepseek-Math-7B-Base in terms of pretraining performance in Table 1?\n- Regarding Weakness 3: How do you consider the validity of the questions and the correctness of the solutions in your pipeline?\n- Fine-tuning Data: In Appendix D, you mention using a unified prompt for DeepSeekMath-7B-RL to synthesize 700k solutions for problems from your instruction tuning datasets. Does this mean that no original solutions are used in your fine-tuning data, potentially losing useful data points? For example, MetaMath introduces augmented answers for GSM8K and MATH and shows benefits.\n- Data Synthesis Models: In Appendix G, you mention initializing the data synthesis models with DeepSeekMath-7B-RL. Why choose DeepSeekMath-7B-RL rather than DeepSeekMath-7B-Base? Is there any experimental comparison?", "limitations": "See Weaknesses", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The feasibility of training a 7B LLM to replace GPT-4 for synthesizing QA pairs for mathematical pretraining is demonstrated.\n- The cost is significantly lower than previous state-of-the-art methods (e.g., Deepseek-Math).\n- The final performance, after pretraining and SFT stages, surpasses state-of-the-art methods in both natural language reasoning and tool manipulation settings.\n- Extensive ablation studies confirm the effectiveness of their pipeline.", "weaknesses": "- The core idea involves distilling GPT-4’s synthesis capability into a smaller yet robust 7B LLM (e.g., Deepseek-Math-7B), which remains within the scope of distilling GPT-4 to open LLMs. The pipeline is somewhat compositional and complex, especially concerning the gradient-based value estimation.\n- The main experimental result (Table 1) only displays the final performance, omitting a comparison of pretraining performance. This omission may lead to confusion about whether improvements result from the pretraining or SFT data.\n- The work focuses on synthesizing QA pairs. The validity of the questions and the correctness of the solutions are crucial aspects of this line of work. This work does not address these aspects but only provides several examples in Appendix I.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the insightful suggestion and positive feedback, and list our responses for the weaknesses below:\n\n**W1**. The core idea involves distilling GPT-4’s synthesis capability into a smaller yet robust 7B LLM (e.g., Deepseek-Math-7B), which remains within the scope of distilling GPT-4 to open LLMs. The pipeline is somewhat compositional and complex, especially concerning the gradient-based value estimation.\n- **Reply to W1**. In this work, we aim to verify whether we can improve mathematical reasoning capability of LLMs via a low-cost way. Thus, we need to distill GPT-4's synthesis capability into a small LLM for low-cost massive math problems synthesis. However, it is not easy to achieve this goal, as math problems are typically complex and diverse. Besides, we also should control the cost of invoking GPT-4 APIs to be acceptable. To this end, we expect to first select less high-value data, and then only use GPT-4 to create knowledge distillation dataset based on it. Therefore, we adopt the gradient-based value estimation method for measuring and selecting the high-value data. In our empirical experiments and the ablation study in Table 4, we can see that all the components are useful. Without the value estimation method, the LLM suffers performance degradation. Note that as we carefully design the pipeline to train the data synthesis model, it can be easily used for data synthesis, with no need for retraining the model. Also, our synthetic dataset can also be directly used to pre-train LLMs for improving their mathematical reasoning capability.\n\n**W2 & Q1**. The main experimental result (Table 1) only displays the final performance, omitting a comparison of pretraining performance. This omission may lead to confusion about whether improvements result from the pretraining or SFT data.\n- **Reply to W2 & Q1**. Following the suggestion of the reviewer, we train Mistral-7B only using the pre-trained or SFT data, respectively, and also report the performance of DeepSeekMath-Base and JiuZhang3.0-7B for comparison. As shown in the following Table, after using the pre-training data, our base model can outperform DeepSeekMath-Base, SOTA open-source backbone for mathematical reasoning, indicating the effectiveness of our pre-training data. Besides, by only using the SFT data, Mistral-7B+SFT underperforms JiuZhang3.0-7B in a large margin. It also demonstrates that our pre-training data can improve the potential of LLMs for learning the SFT data. \n\n|         | GSM8k | MATH |\tSVAMP\t| ASDiv | MAWPS | CARP |\n|------------|------------|------|---------|------------|------------|------|\n| DeepSeekMath-Base-7B\t|64.1|\t34.2|\t73.7\t|82.7\t|92.7|\t44.4|\n| JiuZhang3.0-7B-Base|\t68.7\t|40.2\t|80.4\t|86.5\t|93.5\t|40.2|\n| Mistral-7B + SFT\t|87.5|\t45.8|\t87.9|\t89.7\t|97|\t44.4|\n| JiuZhang3.0-7B |\t88.6\t|52.8|\t90.4\t|92.6\t|97.3|\t51|\n\n\n**W3 & Q2**. The work focuses on synthesizing QA pairs. The validity of the questions and the correctness of the solutions are crucial aspects of this line of work. This work does not address these aspects but only provides several examples in Appendix I.\n- **Reply to W3 & Q2**. In existing work[1,2], it is inevitable that erroneous and noisy samples are included into the pre-training data of LLMs. Whereas, its negative influence can be greatly reduced by the following supervised-finetuning process, using correct and human-aligned data[1,3]. For mathematical reasoning, our empirical experiments also found that LLMs are robust to the noise during pre-training. Concretely, we utilize a well-trained LLM to annotate the accuracy of all the pre-training data, then only select the top-ranked 500k samples for pre-training. Besides, we also randomly sample 500k samples for pre-training, to compare LLMs pre-training on the original or correct data. After pre-training, we fine-tune the two models on 50k instructions. As shown in the following table, the two variations achieve comparable performance, indicating that the noise is not an important concern.\n\n[1] Ouyang, Long, et al. \"Training language models to follow instructions with human feedback.\" Advances in neural information processing systems 35 (2022): 27730-27744.\n\n[2] Albalak, Alon, et al. \"A survey on data selection for language models.\" arXiv preprint arXiv:2402.16827 (2024).\n\n[3] Zhou, Chunting, et al. \"Lima: Less is more for alignment.\" Advances in Neural Information Processing Systems 36 (2024).\n\n|         | GSM8k | MATH |\tSVAMP\t| ASDiv | MAWPS | CARP | Avg. |\n|------------|------------|------|---------|------------|------------|------|---------|\n| **PT Random 500k + SFT 50k** | 83.2 | 38.4 | 84.9 | 88.3 | 94.8 | 43.6 | 72.2 |\n| **PT Top-Acc. 500k + SFT 50k** | 82.7 | 40.8 | 84.1 | 87.9 | 94.2 | 42.2 | 72.0 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rtHlUECZzs", "reviewer": "Reviewer_rcgh", "summary": "The paper proposes distilling the data synthesis ability from the strong (and large) model to the small model and using the small model to synthesize (continual) pre-training data, focusing on mathematical problem-solving with natural language reasoning and tool-integrated reasoning.\n\nSpecifically, the paper 1) elaborately designs a prompt set covering different human education stages/difficulty levels, prompts GPT-4 with the designed prompts and diverse math-related texts to synthsize mathematical QA pairs, 2) uses the data synthesis samples (math-related text + designed prompt + QA pair) from 1) to train the small data synthesis model from DeepSeekMath-7B, leverages it to generate large-scale candidates, select the most valuable samples using LESS and utilizes GPT-4 to re-generate the valuable QA pairs, 3) re-trains the data synthesis model with the data merged from 1) and 2) and uses it to synthesize large-scale QA pairs, 4) continually pre-trains various base models with the data from 3), 5) finally, fine-tunes the continually-pre-trained model with mixed open-source instruction datasets.\n\nThe paper\n\n- proposes an effective and efficient pipeline to synthesize large-scale high-quality instruction data;\n- empirically validates the effectiveness of scaling up instruction data for reasoning tasks;\n- conducts extensive ablation studies, providing many insights like:\n  - diversity of prompts for data synthesis, reference text, and strong/specialized data synthesis models are important;\n  - re-training with self-generated data synthesis samples is actually harmful.", "questions": "Suggestions:\n\n- The current setting is more like distillation from DeepSeekMath (we can substitute GPT-4 with human experts for the small KD dataset). The method could be more promising if it could help self-improvement/weak-to-strong generalization. I highly recommend adding experiments on training DeepSeekMath or stronger models in future versions.\n\nConfusions:\n\n- How is the reward model (Eurus-RM should be for reward to correctness) specifically used for data selection?\n- How is the fine-tuning dataset synthesized with DeepSeekMath-7B-RL?\n- How is data deduplication done for the (continual) pre-training data?\n\nTypo:\n\n- Line 734: remove deduplicate → deduplicate?", "limitations": "The limitations of this work are acceptable and the authors point out potential directions to address the limitations for future works.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The scaling effect of instruction data is an important empirical question. The paper is one of the first works to scale instruction data over 5M pairs, showing the feasibility and effectiveness of scaling up instruction data (for reasoning tasks).\n- Instruction data synthesis usually depends on ground truth answer supervision signals, which is not always accessible. The paper validates the effectiveness of scaling synthetic instruction data without ground truth answers to a large scale.\n- Direct distillation / data synthesis from strong but large models is effective but expensive. The paper reveals that the data synthesis abilities of strong but large models like GPT-4 can be effectively distilled into small models like DeepSeekMath-7B, making the data synthesis more efficient.\n- JiuZhang3.0 achieves performance superior to or comparable with previous SotA on various mathematical reasoning datasets, under both natural language reasoning and tool manipulation settings. Especially,\n  - JiuZhang3.0-7/8B are respectively based on Mistral-7B and LLaMA-3-8B, which are conventionally thought weaker than DeepSeekMath-7B, but achieve impressing performance slightly better than DeepSeekMath-7B-RL. (JiuZhang3.0 based on DeepSeekMath-7B should be expected to achieve even better performance.)\n  - Scaling curves of performance linear to the training dataset size until a rather large scale (\\~5M) are shown in Figure 3, which should be better than usual log-linear ones.\n- Extensive experiments on various base models and benchmarks and comprehensive ablation studies are done to validate the effectiveness of the pipeline.\n- The paper is generally well written to be clear and detailed.", "weaknesses": "- The paper achieves its best performances with a 2-stage instruction tuning on pre-trained models. Still, it doesn’t involve continual pre-training, which should be rather important for models’ reasoning abilities as proved by works like DeepSeekMath. It might need further consideration about **whether continual pre-training on large-scale synthetic instruction data before fine-tuning is compatible with or necessary to be added to existing training pipelines to achieve the best final performance (for reasoning tasks)**. Pre-training and RL should be out of this work’s scope. However, it would be better to further clarify the impacts of 1) continual pre-training on domain-specific corpora, 2) continual pre-training on large-scale instruction data, 3) final fine-tuning on additional instruction datasets, and their combinations.\n  - According to the baseline results from DART-Math, directly fine-tuning on DeepSeekMath-7B-RL-synthesized data only can produce competitive performance (on GSM8K and MATH under the natural language reasoning setting). **Ablation studies of performance by 2) and 3)** only should be done to better validate the compatibility/necessity of the pipeline.\n  - **To consider 1) continual pre-training on domain-specific corpora**, it is unrealistic to conduct by yourselves, but a possible workaround could be to make full use of the resources provided by DeepSeekMath: DeepSeekMath-7B is continually pre-trained from DeepSeek-Coder-Base-v1.5. By comparing performances on reasoning benchmarks of applying 2/3/2+3 on DeepSeek-Coder-Base-v1.5/DeepSeekMath-7B and the two models themselves, a more comprehensive study on the impacts of these training stages can be done.\n- The abilities of the data synthesis model should be an important factor and the size of the knowledge distillation (KD) dataset could have a significant impact on the data synthesis abilities, but no related ablation studies have been done. It would be better to investigate **the scaling behavior of the KD dataset size**.\n- Continual pre-training is usually done with LM loss without masking the prompt tokens (e.g. in DeepSeekMath). It is not mentioned **which type of loss is used for (continual) pre-training** in the paper and worth corresponding ablation studies.\n- The upper limit of synthetic data is important. The student model could be weaker than the teacher model for data synthesis, so the results in the paper might not be the upper limit. It is so expensive and unrealistic for the authors to scale synthetic data to the same scale using GPT-4, but **more comparisons with dataset size controlled with previous works training on large-scale GPT-4-synthesized data (e.g. KPMath)** should be also helpful.\n\n---\n\n# After rebuttal and discussion\n\nThe authors resolve almost all the concerns.\n\nI would improve my rating to 8 because JiuZhang3.0 has been validated to replace the continual pre-training stage in the standard SotA pipeline and shows great potential to combine further with continual pre-training like DeepSeekMath to effectively push forward the upper limit.\n\nThe main limitation is that the pipeline still relies on strong DeepSeekMath models for data synthesis, not fully eliminating the necessity of continual pre-training on large-scale corpora, which might be a bottleneck for improving the best models like GPT-4. So I could not give a rating of 9.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the insightful suggestion and positive feedback, and list our responses for the weaknesses 1 below:\n\n**W1**. It might need further consideration about whether continual pre-training on large-scale synthetic instruction data before fine-tuning is compatible with or necessary to be added to existing training pipelines to achieve the best final performance (for reasoning tasks). Pre-training and RL should be out of this work’s scope. However, it would be better to further clarify the impacts of 1) continual pre-training on domain-specific corpora, 2) continual pre-training on large-scale instruction data, 3) final fine-tuning on additional instruction datasets, and their combinations.\n- **Reply to W1**. Recently, continual pre-training on large-scale instructions has been widely used, due to its effectiveness in improving domain/task-specific capability[1,2]. After it, by further fine-tuning on less high-quality instruction, LLMs would be able to understand and follow more complex domain/task-specific instructions. In this work, we name our synthetic data as math-specific pre-training corpus, as its scale (4.6B tokens) is much larger than commonly-used instruction datasets, and we also conduct SFT after training on it. In fact, we think this corpus should not be directly used for instruction-tuning, as we can not guarantee the accuracy of its contained data. Thus, it is necessary to add the fine-tuning process with reliable and accurate data after training on it. In future work, we will explore how to guarantee the accuracy of the data and compress our dataset into a relatively less instruction dataset specially for fine-tuning.\n\n[1] Yue, Xiang, et al. \"Mammoth2: Scaling instructions from the web.\" arXiv preprint arXiv:2405.03548 (2024).\n\n[2] Cheng, Daixuan, et al. \"Instruction Pre-Training: Language Models are Supervised Multitask Learners.\" arXiv preprint arXiv:2406.14491 (2024).\n\n\n**W1-a**. According to the baseline results from DART-Math, directly fine-tuning on DeepSeekMath-7B-RL-synthesized data only can produce competitive performance (on GSM8K and MATH under the natural language reasoning setting). Ablation studies of performance by 2) and 3) only should be done to better validate the compatibility/necessity of the pipeline.\n- **Reply to W1-a**. Following the suggestion of the reviewer, we add the ablation studies that train Mistral-7B only using the pre-trained or SFT data, respectively, and also report the performance of DeepSeekMath-Base and JiuZhang3.0-7B for comparison. As shown in the following Table, after using the pre-training data, our base model can outperform DeepSeekMath-Base, SOTA open-source backbone for mathematical reasoning, indicating the effectiveness of our pre-training data. Besides, by only using the SFT data, Mistral-7B+SFT underperforms JiuZhang3.0-7B in a large margin. It also demonstrates that our pre-training data can improve the potential of LLMs for learning the SFT data. \n\n|         | GSM8k | MATH |\tSVAMP\t| ASDiv | MAWPS | CARP |\n|------------|------------|------|---------|------------|------------|------|\n| DeepSeekMath-Base-7B\t|64.1|\t34.2|\t73.7\t|82.7\t|92.7|\t44.4|\n| JiuZhang3.0-7B-Base|\t68.7\t|40.2\t|80.4\t|86.5\t|93.5\t|40.2|\n| Mistral-7B + SFT\t|87.5|\t45.8|\t87.9|\t89.7\t|97|\t44.4|\n| JiuZhang3.0-7B |\t88.6\t|52.8|\t90.4\t|92.6\t|97.3|\t51|\n\n**W1-b & Q1**. To consider 1) continual pre-training on domain-specific corpora, it is unrealistic to conduct by yourselves, but a possible workaround could be to make full use of the resources provided by DeepSeekMath: DeepSeekMath-7B is continually pre-trained from DeepSeek-Coder-Base-v1.5. By comparing performances on reasoning benchmarks of applying 2/3/2+3 on DeepSeek-Coder-Base-v1.5/DeepSeekMath-7B and the two models themselves, a more comprehensive study on the impacts of these training stages can be done.\n- **Reply to W1-b & Q1**. Following the suggestion of the reviewer, we apply our synthetic pre-training data and collected SFT data, to train DeepSeek-Coder-Base-v1.5 and DeepSeekMath-7B-base. We report the results in the following Table, where we can see a consistent trend that Backbone+PT+SFT > Backbone+SFT > Backbone+PT > Backbone. It indicates that our overall framework works well on different backbones. The reason why Backbone+SFT > Backbone+PT is that the SFT data is typically more like or even derives from the downstream evaluation datasets. With the help of our synthetic pre-training data, the math reasoning capability can be enhanced more after SFT. Besides, we can see that our approach leads to more improvement on DeepSeekMath-7B-base than DeepSeek-Coder-Base-v1.5. It demonstrates that continually pre-training on more domain-specific data can boost the model capability more. \n\n|         | GSM8k | MATH |\tSVAMP\t| ASDiv | MAWPS | CARP |\n|------------|------------|------|---------|------------|------------|------|\n| DeepSeekMath-Base\t|64.1|\t34.2|\t73.7\t|82.7\t|92.7|\t44.4|\n| DeepSeekMath-Base + SFT|\t87.1\t|51.2|\t86.5|\t91.2|\t96.6|\t52.2|\n| DeepSeekMath-Base + PT\t|77.6\t|46.6\t|87.4\t|90\t|94.8\t|50.4|\n| DeepSeekMath-Base + PT SFT|\t87.6|\t52.6\t|88.6|\t93.1|\t96.7|\t52|\n|DeepSeekCoder-Base-v1.5|\t41.3\t|17.4\t|68.5\t|71\t|85.2\t|32.6|\n|DeepSeekCoder-Base-v1.5 + PT\t|67.6|\t43.2|\t80.2|\t86.5\t|91.9|\t42.3|\n|DeepSeekCoder-Base-v1.5 + SFT\t|81.9\t|40.8\t|83.3\t|88.3\t|96.3|\t43.9|\n|DeepSeekCoder-Base-v1.5 + PT SFT|\t86.2|\t47.4|\t87.2|\t90.9|\t96.8|\t49.6| \n| JiuZhang3.0-7B |\t88.6\t|52.8|\t90.4\t|92.6\t|97.3|\t51|\n|JiuZhang3.0-8B\t| 88.6\t|51\t|89.4\t|92.6|\t97.1\t|50.9|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a method of enhancing mathematical reasoning in LLMs through cost-efficient data synthesis. It also provides extensive experiments and comprehensive ablation studies to validate the effectiveness of the proposed pipeline, demonstrating superior performance on various mathematical reasoning datasets. It reveals that data synthesis abilities of large models can be effectively distilled into smaller models, enhancing efficiency. Additionally, the paper is pioneering in scaling instruction data over 5M pairs, validating the feasibility and effectiveness of this approach for reasoning tasks.  Reviewers like the proposed cost-efficient method. 
At the same time, please follow the promise of releasing the code and data to the research community.
Decision: Accept (poster)