GRIP: A Graph-Based Reasoning Instruction Producer
Authors: Jiankang Wang
Jianjun Xu
Xiaorui Wang
Yuxin Wang
Mengting Xing
Shancheng Fang
Hongtao Xie
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Reasoning
Data Synthesis
Primary area: applications

Abstract: Large-scale, high-quality data is essential for advancing the reasoning capabilities of large language models (LLMs). As publicly available Internet data becomes increasingly scarce, synthetic data has emerged as a crucial research direction. However, existing data synthesis methods often suffer from limited scalability, insufficient sample diversity, and a tendency to overfit to seed data, which constrains their practical utility. In this paper, we present \textit{\textbf{GRIP}}, a \textbf{G}raph-based \textbf{R}easoning \textbf{I}nstruction \textbf{P}roducer that efficiently synthesizes high-quality and diverse reasoning instructions. \textit{GRIP} constructs a knowledge graph by extracting high-level concepts from seed data, and uniquely leverages both explicit and implicit relationships within the graph to drive large-scale and diverse instruction data synthesis, while employing open-source multi-model supervision to ensure data quality. We apply \textit{GRIP} to the critical and challenging domain of mathematical reasoning. Starting from a seed set of 7.5K math reasoning samples, we construct \textbf{GRIP-MATH}, a dataset containing 2.1 million synthesized question-answer pairs. Compared to similar synthetic data methods, \textit{GRIP} achieves greater scalability and diversity while also significantly reducing costs. On mathematical reasoning benchmarks, models trained with GRIP-MATH demonstrate substantial improvements over their base models and significantly outperform previous data synthesis methods.

GRIP: A Graph-Based Reasoning Instruction Producer
Jiankang Wang 1 Jianjun Xulr, Xiaorui Wang" Yuxin Wang',
Mengting Xing".
Shancheng Fang² Hongtao Xie University of Science and Technology of China
MetaStone Technology. Beijing, China
Langjiankang@mali ustc.
www.ll..cttuuu.........om edu. cn
Abstract
Large-scale, high-quality data is essential for advancing the reasoning capabilities of large language models (LLMs).. As publicly available Internet data becomes increasingly scarce, synthetic data has emerged However, existing data synthesis methods often suffer from limited scalability, insufficient sample diversity, and tendency to overfit to seed data, which constrains their practical utility. In this paper, we present GRIP, Graph-base braph-based Reasoning Instruction Producer that efficiently synthesizes figh-quality and diverse reasoning instructions, GRIP constructs a knowledge graph by extracting high-level concepts from seed data, and uniquely leverages both explicit and implicit relationships within the graph to drive large-scale and diverse instruction data synthesis, while employing open-source multi-model supervision to ensure data quality. We apply GRIP to the critical and challenging domain of mathematical reasoning. Starting from a seed set of 7.5K math reasoning samples, we construct GRIP-MATH, a dataset containing 2.1 million synthesized question-answer pairs. Compared similar synthetic data methods, GRIP achieves greater scalability and diversity while also significantly reducing costs. On mathematical reasoning benchmarks, models trained with GRIP-MATH demonstrate substantial improvements over their base models and significantly outperform previous data synthesis methods.
crucial research direction.
Sagerviees
Sentbee: Eraitode Palce
faurite Extra Exformation
Spethetic
Soed bend
Reweloge
(E.50)
The two methods on the reconstruct data based on either the seed itself or extra information. making them similar to the seed and difficult to scale. Our method first extracts key concepts to construct a concept graph, then synthesizes novel questions by leveraging multiple relationships within the graph.
Figure 1:
Introduction
In recent years, large language models (LLMs) have achieved remarkable performance across wide range of linguistic tasks 1
30 largely driven by access to large-scale, high-quality training data.
However, recent studies 23 34 have shown that the pool of high-quality publicly available data the Internet continues to diminish, the further advancement of LLMs may be limited by data scarcity.
Work done during the internship at MetaStone Technology.
Eqqual contribution. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

As a result, the exploration of synthetic data has attracted increasing attention. Several leading commercial models [27 38 39 have incorporated large amounts of synthetic data as indispensable components of their training pipelines. Nevertheless, the details of these synthesis processes are often undisclosed, or their resource requirements are prohibitively high. Therefore, developing a practical and highly scalable method for synthesizing high-quality data has become a critical challenge for the continued evolution of large language models.
One approach for building high-quality reasoning datasets is data filtering 22 involves extracting data from pre-training corpora such as Common Crawl and rewriting using advanced commercial models or human annotation. However, due to these corpora's 's vast scale and inherent noise, data filtering results in high protessnn costs and inconsistent data quality and
This method
distribution. A more efficient approach is data symthesis [14] [0 [33] 41 42]. Such approaches
often leverage prompts to -ouure-source models [
30 rephrasing seed data or generating analogous
questions to augment data, as shown in Figure 1 More advanced variants incorporate the knowledge
points or topics of the seed data into the prompts, guiding the model to generate new questions centered around this extra information. However, these methods have several inherent limitations. First, since they only rephrase or slightly modify seed questions, the expansion in data volume is inherently limited, and does not achieve orders-of-magnitue on seed data means that the generated questions are often highly similar to the original seeds, restricting the diversity of the synthesized data. Third, the total amount of data that can be synthesized is further restricted by the high cost and limited accessibility of closed-source models, making large-scale generation impractical.
growth. Second, the strong dependence
Inspired by human learning patterns, we move beyond directly manipulating seed data and instead focus on the underlying high-level concepts (such as topics and knowledge points) embedded within each example. Typically, a seed example consists of three or four key concepts. Prior methods, which rephrase questions or alter scenarios, in fact only rearrange the same set of core concepts, thus providing limited new information for the model to learn. To address this limitation, we first systematically extract all high-level concepts from the seed data and construct co-occurrence graph, where two concepts are connected if they ever appear together in the same example. By designing new examples that combine concepts which have never co-occurred in the original data, we able to synthesize truly novel data that lics outside the original seed distribution. We observe that the number of valid non-co-occurring concept pairs far surpasses co-occurring ones. Even modest increase in seed data leads to a dramatic growth of novel concept combinations, highlighting the strong scalability enabled by this idea. Building on this idea, we propose GRIP: graph- based framework to efficiently synthesize large-scale, high-quality and diverse reasoning data.
As presented in Figure I right, GRIP encompasses four steps: (1) Knowledge base construction: We begin by extracting key concepts (KCs) from seed data using a specialized model, such as mathematical theorems (e.g., the Pythagorean theorem). formulas (e.g., the quadratic formula), and key properties (c.g., the distributive law). This is followed by a filtering step to remove duplicates and low-quality key concepts. (2) Building the graph: The Key Concepts Relationship Graph (KCRG)
is designed to capture the interconnections among concepts. In the graph, we define two types of relationships to form the KCRG: explicir and implicis. Explicit relationships are represented by solid lines, connecting concepts that originate from the same example. Implicit relationships, shown as dashed lines, connect concepts that are not directly related in the seed but exist within certain distance of each other. These implicit relationships are further classified into two-hop and three-hop connections based on their distance. (3) Synthesis: The specialized model generates new samples by feeding its designed prompts and explicit and implicit concept combinations from the KCRG. (4) Data Evaluation: Multiple advanced open-source models are utilized to filter the synthesized data by jointly scoring the new samples.
To demonstrate the effectiveness of GRIP, we apply it to one of the most challenging reasoning domains: mathematics. We use 7.5K question-solution pairs from the MATH training set as seed data and synthesize new dataset, GRIP-MATH, which contains over 2.1 million math question-solution pairs. We use GRIP-MATH to train large language models (LLMs) with diverse architectures and parameter sizes, including Qwen weeel155.. 7. 33 Mistral-7B [17. LLaMA3-SB 22). LLaMA3.1-8P (10), Qwen2-1.5p 15 (37, and Qwen2-7B [37]. On mathematical reasoning benchmarks, models trained with GRIP-MATH demonstrate substantial improvements over their base models and significantly outperform previous data synthesis methods. Furthermore, GRIP-MATH also enhances scientific reasoning performance, highlighting the strong generalization ability of GRIP.

---

Knowledge Base Construction
Seed Data
Key Concepts
Key Concepts Filter Models
Quenten of we eagrasa
Model
supensdce ans no veites forry by asiltg end Réfreltiy app speeptitt
Selatlew the Wee
L.Quddettic Expeadiona Underatorda prructure stendard forms
that fom
as San (x--) -5
of
Extratt
Vertee Farm of Quadeetic Bowntine qudkettc fuccias the ferm ocx Ailet and sdentifyeg the
Construct
Key Concepts Relatonship Graph
Explore
Explict Implicit
Key Goncepts
(KCRG)
Mathematical Data Synthesis
Key Concepts Combinations of aode the to oppia engle crratent
High-quality Solutions
Saletie "o find the valus of snple and the other two onglka in "mangl 180 degress and the Loe of Sinca
2. Atgle Sun and difference Iderthlle The die of twa Auu axssn..
Lingg
Model
Vote
Canceges
Superstsion
Step Fiul engle we fird cagle c. 60-455cc1
fimm
step Fiet see is we are the Lew siees find side was ceer wiles clepitiie lilee ix of the sinea find where
Synthesttee
desieg pory wn oa 4ridero Wiich ampiific to su
High-quality Questions
MATH Model
Quasties Let's card der trangle wnd with coles a and and ms) Fedd The vake onglr and the onker hwe sites
MultiMMadel
Vate
Infermmcn
Pinal Answen Angle C-77, sick
Superstsion
Figure 2: The overview of the Graph- based Reasoning Instruction Producer (GRIP). GRIP begins with seed data and follows four-step process: (1) knowledge base construction, (2) key concepts relationship graph construction, (3) graph-based synthesis, and (4) evaluation by multiple models voting. After these steps, obtain the GRIP-MATH dataset.
2
Related work
2.1 LLMs and Mathematical Reasoning
Recent years have seen remarkable advances in large language models (LLMs) for mathematical reasoning. Various strategies have been proposed to further improve LLMs' math abilities, including chain-of-thought prompting [36], program synthesis (27 (5). and fine-tuning on high-quality math corpora 21]. More recently, proprietary models trained extensive curated datasets and large-scale synthetic data-such as DeepSeekMath (27] and Qwen2.5-Math (39 have achieved state-of-the results. However, the lack of transparency their data synthesis methods limit reproducibility. In response, research has increasingly focused on synthetic data generation (29 43) as a scalable and reproducible approach b oosting LLM performance on mathematical reasoning tasks.
2.2
Data Synthesis
With the imminent exhaustion of Internet data, data synthesis has drawn increasing research attention. In mathematical reasoning, data synthesis is primarily used for instruction fine-tuning, where each sample consists of a question text and its corresponding answer. Our method can synthesize large- scale, high-quality mathematical inference data from limited seed data, making it suitable for continued pre-training tasks. Research efforts primarily concentrate on two pivotal aspects: improving data quality and generating novel questions. Regarding the generation of novel questions, one approach [19 29 33 41 42) entails rewriting or generating similar questions based on seed data for data augmentation. Another approach 14 [5| 20 involves generating new questions using knowledge points, either by generating new knowledge points via GPT-4 or extracting them from existing knowledge point databases. However, these approaches often suffer from limited scalability, high cost, and high similarity to seed data, due to their reliance on explicit relationships and closed- source models. Our method addresses these limitations by exploring both explicit and implicit relationships between key concepts using the graph and leveraging open-source models for cost- effective data synthesis.
Proposed Method
As presented in Figure this section introduces a Graph-based Reasoning Instruction Producer (GRIP), unified synthetic data framework with four steps: Knowledge Base Construction, Key Concepts Relationship Graph Construction, Synthesis Based on Diverse Key Concept Combinations,
3

---

Multi-Mode Evaluation, and Dataset Statistics. Detailed descriptions and implementation steps for each component are provided in the subsequent sections.
3.1 Knowledge Base Construction
To enable large language models (LLMs) to more effectively process complex information such as mathematical problems, we first decompose each seed instance into its essential conceptual components, constructing knowledge base that provides a structured representation of the overall dataset. In principle, problem may be characterized by multiple hierarchical attributes, such as "Suujjeet" (e.g., "Mathematics"), "Topic" (e.g.. "Algebra"), and "Key Concept" (c.g., "Permutations and combinations" or "Difference of cosines formula"), For simplicity and to facilitate both concept extraction and automated modeling, focus exclusively extracting "Key Concept" as the key conceptual features for each question. Empirically, we find that this abstraction is sufficient to capture the salient mathematical characteristics of most problems.
GRIP uses the MATH training set as the seed data which consists of 7.5k math problems. We first extract more than 5 relevant key concepts (KCs) from each seed problem with prompt engineering of Qwen2.5-32B
(refer to Prompt A. 1 Appendix A.II. After extracting the KCs, we employ an embedding model 4 and Qwen2.5-7B (8) for dual filtering of the KCs. Initially, the LLM filters out KCs with vagueness, math errors, or excessive details. Subsequently, the embedding model clusters KCs with similar meanings, followed by second validation using the LLM. Finally, the most appropriate and accurate KC from each cluster is chosen to represent the cluster, resulting in 10K qualified key concepts. More details about dual filtering and KC examples can be found in Appendix(] D
3.2 KCRG Construction
To organize the disordered KCs in the knowledge base and explore their specific interconnections, designed a Key Concepts Relationship Graph (KCRG) to capture the associations between KC pairs.
In the KCRG, each node is represented as KC, and each solid edge represents that the connected KCs have co-occurred in the same problem. Specifically, the KCRG C can be represented as C (K, E). The nodes K, which refer to key concepts, are denoted as K (ki, k Kj J. The edges E are denoted as E [eex, Eim}. and there are two types: (1) Explicit (Be)): Key concepts that appear together in the same seed problem are connected by a solid edge. The explicit edges Eys can be denoted as Ee, = {(k, ( D(k, kj) 1), where the edge distance D(k,, k,) represents the number of solid edges in the shortest path between k; and k,. Additionally, the edge weight W W(k, k,) is recorded to denote the co-occurrence frequency between k; and ky. (2) Implicit (Lim): Key concepts with more than one solid edge between them are connected by dashed edge, as visualized in Figure The implicit edges Em can be denoted as E. {(k,, ((k,,,, k,) > 1}.
3.3 Synthesis Based on Diverse Key Concept Combinations
Different from previous methods that primarily relied on seed data and explicit relationships between KCs, fully integrate both explicit and implicit relationships generate more diverse synthetic data. We observed that incorporating overly distant implicit relationships tends to increase the proportion of low-quality data. This is because such KCs exhibit weaker correlations, making it difficult to synthesize meaningful questions. Therefore, we proposed four types of key concept relationships: one-hop, two-hop, three-hop. and community. Exploring implicit relationships has mined more key concept combinations, which is a key driver of GRIP's high scalability. Figure 3 illustrates an example of KCRG construction.
One-hop relationships represent explicit links between pairs of key concepts (KCs) that are directly connected by single edge in the graph. The edge weight reflects the frequency of their co-occurrence in the seed data. Since these combinations are already present in the seed set, they exhibit high semantic relevance.
Two-hop relationships capture implicit connections, involving pairs of KCs that are connected indirectly through an intermediate node-i.e., two KCs that share common neighbor. This indirect association often reflects shared characteristies or themes, thus preserving a moderate level of semantic relevance.

---

Knowledge Base
F
Key Concept Combinations
Quastion I00e
(3) "Pigerries
fieeriess of Ratational Syeecetry"
Hub Concept:
(A) Aegle of Raretiee"
One-hop: A-B, A-C, B-C,
(8) ane cocle and proppccctia
A-D,
Quastion
Two-hop: A-E, C-D, D-F
Regertic Autarional Syenetty"
(8) Tn a cessian gegonettry ond rnganometry
Three-hop: A-F
Question 4 KGs
(3) or courtatt poonunyy and trignonetrr"
Community:
Figure 3: An example of constructing the Key Concepts Relationship Graph (KCRG) from an existing knowledge base and identifying the four key concept combinations we proposed.
Three-hop further explores implicit relationships by considering pairs of key concepts (KCs) that are three edges apart in the graph. In this setting, we focus on hub concepts those nodes in the knowledge concept relationship graph (KCRG) that have the largest number of connections (i.e., high degree). Such hub concepts serve as central points in the knowledge network, indicating their broad relevance and significance across various topics. A three-hop combination defined as a pair consisting of a hub concept and another KC that is three edges away from it. As the graph grows larger, multiple hub concepts may exist. Since the semantic relevance between KCs tends to decrease as the path length increases, we restrict three-hop relationships to only those involving hub concepts to help maintain meaningfulness. Additionally, we filter out low-weight three-hop combinations to further ensure that the selected three-hop pairs are relevant and informative.
For example. assume that a hub concept is "calculus". In the seed data, problems only associate "calculus" with fundamental concepts such as "limits" and "derivatives". However, in the KCRG, "calculus" is likely not adjacent but close to KCs such as "Fourier transforms" and "complex functions", By integrating these three-hop KCs (the same applies to two-hop) to construct novel problems, we increase the diversity of the problem set.
Community represents explicit relationships involving three or four key concepts (KCs), where every pair within the group is mutually connected by edges, forming fully connected subgraph. Such communities indicate a strong correlation among the KCs and typically represent cohesive knowledge areas.
Accordingly, one-hop combinations are used to synthesize high-qualit variant problems directly related to the seed data. Implicit relationship combinations are used to synthesize new distribution data, increasing the diversity of the dataset. Community-basse combinations are used to synthesize integrative problems that require simultaneously applying multiple closely related key concepts.
As shown in Figure 3 after extracting the KCs from the seed, we construct the KCRG based on their co-occurrence. All key concept combinations in the graph that meet these four relationship types are extracted, and low-weight implicit relations are filtered out. We input the prompt and key concept combinations into the @wwn2.5-32BB (38) to synthesize new problems. Different from other methods, we do not include seed data in the prompt, as this would cause the model to generate problems too similar to them. Before solution generation, rating model assigns difficulty level to each problem. For medium and low-difficulty issues, Qwen2.5-Math-77 (39 generates the solutions, while Siven.55MMah--5 handles high-difficulty problems. The complete prompt template is provided in Appendix A.2 Prompt A.2.
3.4 Multi-Model Evaluation
To match the evaluation effectiveness of closed-source models, we employ a multi-model supervision framework using three state-of-the-art open-source mathematical LLMs: DeepSeek-RI -iitt------iii---iiei-e-
7B [1]. Qwen2.5-Man M-Mth-IInttrucee used to score and filter the synthesized data, ensuring high quality of both problems and solutions.
[39], and DeepSeek-Math
[27]. These models are jointly
For problem evaluation, we use a weighted scoring filtering strategy. Problems are evaluated on two criteria: logical completeness (absence of mathematical errors and accurate relation to provided key concepts) and presentational completeness (clarity, thoroughness, and absence of prompts or answers). Each model assigns score between 0 and for every problem. A weighted average
5

---

GRIESAATTI toun)
MuthCotetaainut
MataSSalbUA
Seale
Stadlarity SCREe
Studlarky Score
Stuilasity
Stadiality
Figure 4: Histogram of the similarity scores between synthesized data and the seed data, including a comparison of GRIP-MATH with three open-source datasets terms of seed similarity. The bars represent the frequency of the similarity scores, while the red line represents the cumulative distribution function (CDF) of the scores, can be observed that the similarity scores of GRIP-MATH are concentrated between 0.55 and 0.65, whereas those of the other three methods are more concentrated around 0.85 or even 1. This indicates that MATT exhibits lower seed similarity.
Table 1: Comparison of various methods in expansion ratio (x) and synthesis cost (10-² cents). The expansion ratio represents the proportion of synthesized data to seed data. Synthesis Cost indicates the expenses associated with closed-source models GPU usage for synthesizing single data sample. Details of cost calculation are provided Appendix
Method
Sonree
Synthesis Model
Total Data
Total Synthesized Data
Expansion Ratio
Cost
Novelty Rate
Memaath
5
GSMRK-MATH
GPT:3,5
15K
395K
26
23
17.9
MalSaalle
MWPBENCH
GPT:3.5
2M
WizardMtch
21
GSMMK
GPT4
I5K
96K
6.4
220
XwinMath
GSMBK+MATH
GPT-4
15k
1.4M
93
220
MAmmaTH
MAmmoTH dutasets
GPT4
270K
2622K
MadhCoder
35
CSMRK-MATHE
GPT-4
I5K
80K
5.3
220
9.1
GRIP
MATH
Open Model
2.1M
score is then computed, where weights are proportional each model's demonstrated mathematical ability. Problems with scores below 0.85 are discarded. For solution evaluation. we implement strict single-vote veto mechanism. A solution must be mathematically correct and fully address all aspects of the problem. The solution is scored as O or by each model, and only those unanimously approved by all models (perfect score) are retained. Any solution receiving a negative vote from any model is filtered out to maintain overall dataset quality. After evaluation of problems and solutions, we constructed a dataset named GRIP-MATH comprising 2. million high-quality mathematical questions at low cost.
3.5 Dataset Statistics
To demonstrate the superiority of our methodology, we compare various data synthesis methods from multiple dimensions. (1) For scalability. we compare the seed and synthesized data volumes among different methods. As illustrated in TT MathScale (29] is the largest dataset, utilizing GPT-3.5 to expand 20K seed data into 2M samples, reaching 100-fold expansion ratio. Methods employing GPT-4 for data synthesis demonstrate relatively low expansion ratios due their high operational costs. In contrast, our approach expands 7.5K seed data to 2. samples, reaching the highest expansion ratio of 280-fold. (2) For synthesis cost, we calculate the per-sample synthesis cost for each method. To simplify the comparison, we consider API ssagercosts for methods relying on commercial models and GPU computational cost:] for our approach. As shown in Table I our method's 's per-sample synthesis cost is merely 2% of GPT-3.5- methods and less than I 1% of GPT-4-based methods. (3) For data quality, we compare the similarity between open-source datasets and their corresponding seed data using embedding model. Specifically, we calculate the similarity between the embedded synthesized data and seed data to obtain the similarity score distribution for each dataset, visualized in Figure 4 The results show that the rewritten methods of MathCoder (35) and MetaMath [41 show extremely high similarity to the seed data. Although MathScale is not directly based on rewriting, it still exhibits relatively high similarity due to its reliance on explicit relationships in the seed data. In comparison, GRIP-MATH has over 50% of its data similarity below 0.65, with the majority under 0.75 and none exceeding 0.9. (4) For diversity, we further analyzed the diversity of generated datasets by calculating the proportion of questions whose key concept combinations were not present in the seed data. This metric reflects how many genuinely novel questions each method can generate. We found that for other methods, this proportion was consistently below 40% indicating that they generated relatively few truly novel questions. In contrast, thanks to GRIP's use of implicit relationships, we generated 1.5 million novel questions,
http:://oopeai..oo///a//pott bbtps://powwr.neemmndd
htwp./reeeeemiiiiiaaaaeantttooo ai/rentlntro

---

Table The performance of models on mathematical reasoning tasks. The results are sourced from the evaluation scripts of MAmmoTH2 and OpenCompass. GK denotes the 2010-2022 Math II MCQs from GAOKAO- AO-Eval, and GK represents the 2010-2022 Math MCQs. * denotes our reproduced results based on the officially released codes.
Model
GSMBK
GKI
SVAMP
AVG
Specific Models
Qwen2-Math
Qwen2
Qwen2-Math
Qwen2
50.4
78.9
62.5
88.1
72.2
Deepseek Math Instruct
DeepseeMMath
46.8
82.9
58.3
46.7
84.0
63.7
DespseskMath-RI.
DeepseekMath
7B
51.7
88.2
61.5
58.9
86.4
69.3
Base Models
Mistral-7B
7B
11.2
36.2
13.8
12.2
66.9
28.0
Qwen2
1.5B
21.7
58.5
29.8
28.5
67.4
41.2
QwenZ
45.2
80.3
66.5
52.8
87.5
66.5
Qwen1.5
56.4
53.7
73.4
LLaMA3
31.6
I.aMA3.1
23.1
54.9
10.6
10.8
70.1
33.9
Data symthesis method
MetaMath
Mistral
28.2
9.2
9.4
77.2
40.3
WizordMath
Mistral
7B
31.0
78.0
17.0
15.4
48.5
38.0
MathCuder-C CL
Mistral
7B
30.2
67.8
9.6
15.9
70.7
38.8
MathScale
Mistral
34.5
74.0
36.7
31.3
79.6
51.2
MathScale*
Qwen1.5
32.2
55.2
52.4
75.1
56,9
MAmmolHH
MAmmoTI12
Mistral
36.7
68.4
44.9
29.4
81.8
52.2
MAmmoTH2
LLaMA3
8B
35.8
70.4
33.5
24.3
78.6
48.5
only GRIP-MATH
GRIP
Mistral
7B
41.6
83.5
42.9
33.1
85.4
57.3
GRIP
Qwen1.5
7B
37.9
77.1
57.4
56.3
80.8
61.9
GRIP
LLaMA3
37.2
76.5
38.5
31.8
82.2
53.2
GRIP
LLaMA3.1
44.5
Qwen2
GRIP
Qwen2
53.4
86.0
68.4
54.2
88.8
70.2
with 71.8% of key concept combinations not found in the seed set. This demonstrates a substantial improvement diversity achieved by our approach.
4 Experiments
4.1 Training Setup
Qwen2- 1.5B (37]. and Qwen2-7B 337 as baseline models, and trained all of them exclusively on the GRIP-MATH dataset. The fine-tuning performed using the L.LMAFactoryy 44 framework over 2 epochs, with a learning rate of 5e-6, global batch size of 1 128, and maximum sequence length of 4096. A cosine schedule with a 3% warm-up ratio adopted to regulate the learning rate. For expedited and efficient training, we leveraged DeepSpeed (25) ZeRO Stage 3 and FlashAttention 2 [9] The synthesis with GRIP was completed 36 hours using NVIDIA 110 GPUs and vLLM [8]
We selected Qwen1.5-7B [3]. Mistral-7B [17], LLaMA3-8B 22). LLaMA3.1-SB [0..
4.2 Evaluation Datasets
To rigorously assess the enhancement in mathematical reasoning capabilities of models trained with GRIP-MATH, we employed a suite of mathematical evaluation datasets, including GSM8K [ MATH [3, GAOKAO-Eval [1 and SVAMP (24). In addition, to assess the impact of GRIP-MATEE
on the model's reasoning capabilities across other domains (e.g., physics, chemistry, coding, and
logic), we evaluate the model using range of scientific reasoning datasets, including ARC-C [6]
MMLU-STEM [33, GGOA-Diaaonn (26), BBH 28), TheoremQA (5), and MBPP 2 The results
sourced from the evaluation scripts of MAmmoTH2 43 and OpenCompass 8

---

Table 3: Results on scientific reasoning tasks.
Model
Base
Size
ARC-C
MMLU-STEM
GPQA-Diamond
BBH
TheoremQA
MBPP
AVG
Mistral
45,2
Qwen1.5
75.6
45.5
26.7
45.2
14.2
52.1
43.2
LLaMA3
8B
78.6
35.6
27.2
61.1
20.1
54.9
49.6
LLaMA3.1
8B
79.5
54.7
24.2
62.8
20,9
57.2
49.9
Qwen2
1.58
60.5
42,9
23.7
36.8
15.1
36.9
36.0
Qwen2
83.6
64.3
61.7
33.5
60.7
56.0
GRIP
Mistral
7B
78.6
58.6
31.7
61.1
26.5
54,9
51.9
GRIP
Qwenl.5
7B
77.2
56.9
30.0
51.2
22.4
53.7
48.6
GRIP
LLaMA3
8B
80.5
60.8
30.8
63.7
24.2
58.4
53.1
GRIP
LLaMA3.1
82.7
61.8
32.8
63.2
25.9
59.2
54.3
GRIP
Qwen2
84.3
65.9
33.4
62.5
34.8
65.9
57.7
4.3
Main Results
Table Summarizess the performance of various models on a suite of mathematical reasoning bench- marks. Our experimental findings highlight three main observations:
Substantial improvements over base models. All Graieera models significantly outperform their respective base models across all benchmarks. For example, Mistral-7B, when trained only on GRIP-MATH, achieves an average score 57.3, compared 28.0 for the vanilla Mistral- 7B-a nearly 30-point improvement. Similar trends are observed for Qwen2 and LLaMA3 families, indicating the large and robust gains brought by high-quality GRIP-MATH data regardless base model architecture parameter size.
Outperforms previous data synthesis methods by a large margin. Compared to other open- source data synthesis methods such as MetaMath [41). WizardMath [21] MathScale [29], and MHmmoTH2 [ our GRIP- trained models consistently obtain much higher scores. For instance, Mistral-7B with GRIP-MATH achieves 57.3 average, dramatically surpassing the best prior method (MathScale, 51.2). On the MATH benchmark, GRIP-7B reaches 41.6, 6, far beyond the scores of WizardMath or MetaMath (31.0 and 28.2, respectively), confirming the effectiveness of our approach in generating high-quality, diverse mathematical reasoning data.
Competitive with proprietary math specialist models. Remarkably, GRIP-trained models close much of the gap between open-source foundation models and proprictary specialist models that lever- age substantially more diverse and larger-scale training data (including web data, textbook corpora, exam problems, and Chinese data). key benchmarks such as MATH, GSM8K, and especially on challenging out-of-domain datasets (GK II, SVAMP), GRIP models achieve performance on par with, or even surpass, proprietary models like MeepSeekMath- [27] and Owwe2-MMah-78 relying solely GRIP-MATH synthetic data.
39 despite
Overall, our findings show that GRIP-MATH brings substantial gains base models, consistently outperforms prior data synthesis methods, and enables competitive or even superior performance to commercial specialist models on major benchmarks, all without using proprietary resources.
4.4 Results Scientific Reasoning Benchmark
GRIP-MATH constructed solely from the training set of MATH and does incorporate any data from other datasets. Nevertheless, experiments show that GRIP-MATH not only enhances the model's mathematical reasoning abilities, but also brings significant improvements on out-of-domain scientific reasoning tasks. As shown in Table we evaluate our models on several widely used datasets covering physics, biology, chemistry, and computer science. Across all benchmarks, GriP-trained models consistently outperform their respective base models, demonstrating strong cross-dae coos--omainn generalization. For example, GRIP-trained Qwen2-7B achieves an average score of 57.7, compared to 56.0 for the base model, with notable gains on MMLU-STEM, GPQA-Diamonte and MBPP. These results highlight the strong generalization ability of GRIP-MATH across scientific domains, even without explicit exposure to additional out-of-domain training data.

---

4.5 Ablation Studies about GRIP
Comparison between Multi-Model and GPT-4.1. To investigate the difference between multi-model quality evaluation and using GPT-4.1 alone, we conducted manual annotation of 500 synthesized math problems. Each sample was independently labeled as "qualified" or "unqualified" by human annotators. We then used various combinations of open-source models (DeepSeek-FF 1 7B [1], Sween.5--Mth-Instuuu same data: a problem was considered qualified if its question score was above 0.85 and its solution score was 1. Finally, we compared the model evaluation results with the manual labels, as shown in T T shows that three open-source models outperform G in accuracy, demonstrating that multi-model supervision an effective and economical substitute for closed-source evaluation.
[39] and DeepSeek-Mann Maaeek--Mtt-RRae RI. (77) and GPT-4.1 to score the
Table 4: Ablation on Hop Distance
Table 5: Ablation on Model Combinations
Method
AVGG Score
Model Combination
ACC
One-hop
0.94
GPT-4.1
94.3
Two-hop
0.72
wwenn.5-Math-Isstrrctta
81.4
Three-hop
0.62
Beeen..--Math--nstrruee DeepseekR1 1-7BB
90.5
Three-hop w/o hub KCs
0,39
Qwen2.5- Swen2.5-Math--ii Math-Insstruct7BB, DeepscekRI
Four-hop w/o hub KCs
0.21
DeepseckMath-RR
95.7
Key Concept Filtering. The quality of key concepts is cru- cial for subsequent data synthesis. We synthesize questions (excluding solutions) using both unfiltered and filtered key con- cepts, and evaluate them using the multi-model scoring system to calculate the average score for each setting. As shown in Table 6 applying key concept filtering increases the average question score by 0.2 points. This improvement is due to the removal of meaningless or incorrect key concepts, which would otherwise mislead the model and lead lower-quality questions.
Table 6: Ablation on Filtering
Method
AVG Score
Concept Filtering
0.83
w/o Concept Filtering
0.61
The quality of data synthesized from different hop distances. We compared the average scores of data synthesized from one-hop, two-hop, three-hop, and more distant. The results in Table 4 show that as the distance between key concepts increases, the average score of the synthesized data decreases. This is because meaningful combinations rarer between more distant concepts, leading to a drop in data quality. Nevertheless, it is still possible to synthesize high-quality data from these distant combinations, although this typically comes at higher synthesis cost. Moreover, utilizing hub concepts to facilitate high-hop data synthesis is an effective strategy for maintaining quality.
Impact of Implicit and Explicit Data on Model Per- formance. To investigate the effects of implicit and explicit data on model performance, comprehensive ablation study. We first randomly sam- pled 0.2 million examples from the one-hop (explicit) dataset and, based on the same key concept combina- tions, synthesized another 0.2 million one-hop samples. Additionally, we randomly selected total of 0.2 million samples from the two-hop and three-hop (implicit) datasets. We then trained Mistral-7B separately with the explicit data group and the implicit data group. As shown in Table 7 the results suggest that introducing novel types of questions is more beneficial for model training than continuing expose the model repeated samples of the same type.
Table 7: Ablation on Datasets
Datasets
Math GSMSK
conducted
hop. one-hop(duplication)
29.3
67.6
one-hop, two-hop, three-hop
32.6
71.2
5 Conclusion and Future Work
In this paper, we introduce GRIP, an efficient paradigm for synthesizing high-quality data. Utilizing this method, we construct the GRIP-MATH dataset, comprising 2.1 million high-quality question- solution pairs. By leveraging this dataset, GRIP models have demonstrated outstanding performance across mathematical and scientific reasoning benchmarks. Our research indicates that thoroughly exploring implicit knowledge relationships enables larger-scale and more diverse data synthesis; additionally, multi-model evaluation can approach closed-source performance while maintaining cost-effectiveeness. Intuitively, GRIP should be applicable to various domains where data can be decomposed into key concepts; however, our current experiments are limited to the mathematics domain, and its effectiveness in other areas is yet to be verified, discuss in detail in Appendix F

---

Acknowledgment
This work is supported by the National Nature Science Foundation of China (62425114, 62121002. U23B2028, 62232006). We thank MetaStone Technology for providing closed-source model APIs support and GPU computing resources. We acknowledge the support of the GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC, and USTC super-computing center for providing computational resources for this project.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108. 07732, 2021.
[3] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309, 166099 2023.
[4] Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3- embedding: Multi-lingual, multi-functionaal self-knowledge distillation, 2023.
multi gulti-granularity text embeddings through
[5] Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. Theoremqa: theorem-driven question answering dataset. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7889-7901, 2023.
[6] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.054. 2018.
[7] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 2021.
[8] OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. httpp:://gichub..com/pppppee ptpppencccompass//ppcccpppppap 2023.
[9] Tri Dao. Flashattention-22: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023.
[10] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788
2024.
[11] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
[12] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint trXiv:2009.03300. 2020.
[13] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
[14] Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-point-driven data synthesis with its enhancement on mathematical reasoning. arXiv preprint arXiv:2403.022. 02333, 2024.
10

---

[15] Yinya Huang, Xiaohan Lin, Zhengying Liu, Qingxing Cao, Huajian Xin, Haiming Wang, Zhenguo Li, Linqi Song, and Xiaodan Liang. Mustard: Mastering uniform synthesis of theorem and proof data. arXiv preprint PrXiv:2402.08990 2024.
[16] Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. C-eval: A multi-level multi musti-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36, 2024.
[17] Albert Q Jiang. Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2330. 10.0025. 2023.
[18] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 121166662220 2023.
[19] Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Peng. Common 7b language models already possess strong math capabilities. arXiv preprint 2403::2000 2024.
[20] Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, et al. Synthetic data (almost) from scratch: Generalized instruction tuning for language models. arXiv preprint arXiv:2402. 1306,, 2024.
[21] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical rea- soning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308. 09583, 2023.
[22] Al Meta. Introducing meta llaaa 3: The most capable openly available 11m to date, 2024. URL htlps://ai. meta. com/blog//eeta-lamm--3 Accessed on April, 26, 2024.
[23] Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358-500 2023.
[24] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07199) 2021.
[25] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & 2020.
Data Mining, pages
[26] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-leve google- proof q&a benchmark. arXiv preprint arXiv:2311.1202. 2023.
[27] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:24402.33: 033000000000..00..000 2024.
[28] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, et al. Challenging big- bench tasks and whether soain-of-thought can solve them. arXiv preprint 2rXiv:2210.09261. 2022.
[29] Zhengyang Tang, Xingxing Zhang, Benyou Wan, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning, arXiv preprint arXiv:2403.02888 2024.
11

---

[30] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean--appieeeeee--ee Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican. al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
[31] InternLM Team. Internim: A multilingual language model with progressively enhanced capabilities, 2023.
[32]
InternLM Team. Gaokao-eval: A comprehensive gaokao evaluation. htttps://githb. com/ open- Seen-coopass/AAADA (TAOKAO-Eval 2024. Accessed: 2024-06-05.
[33] Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Git- man. Openmathinstruct-II: A 1.8 million math instruction tuning dataset. arXiv preprint arXiv:2402.10176, 2024.
[34] Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Position: Will we run out of data? limits of Ilm scaling based on human-generated data. In Forty-first International Conference on Machine Learning. 2024.
[35] Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in Ilms for enhanced mathematical reasoning. arXiv preprint arXiv:2310.03731, 2023.
[36] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
[37] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024.
[38] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024.
[39] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via seff-improvement, arXiv preprint arXiv:2409, 2XX.:2449.11222.. 12122, 2024.
[40] Huaiyuan Ying. Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma. Jiawei Hong, Kuikun Liu, Ziyi Wang, et al. Intemlm-math: Open math large language models toward verifiable reasoning. arXiv preprint arXiv:2402.06332, 2024.
[41] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023.
[42] Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653, 2023.
[43] Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. arXiv preprint 2rX 2024.
[44] Yaowei Zheng. Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403. 13372, 2024.
12

---

A Prompts
Prompt A.I Prompt for Key Concepts Extraction
You will he given mathematics problem and its detailed solution. Please extract to 5 key concepts according to the strict requirements below:
1. Use precise mathematical terminology. Each key concept must be named using accurate and professional mathematical terms; avoid colloquial, vague, or general language.
2. Direct and exclusive relevance. Only include key concepts that are directl applied or explicitly referenced in both the problem and its solution. Exclude unrelated, unused, or general concepts, even if they appear incidentally.
3. Be as specific as possible. Do not use broad descriptions like "basic algebra" or "clementary mathematics." Specify particular theorems, formulas, properties, or standard techniques (c.g., "Pythagorcan theorem," "Closure property of multiples under addition").
4. No repetition or composite items. Each key concept must correspond to a single, unique, and atomic mathematical concept. Do not combine multiple theorems, laws, or properties into one item; do not list overlapping or paraphrased concepts.
5. Do not include procedural descriptions or general skills. Only extract conerete math- ematical facts, such as theorems, definitions, formulas, or standard properties. Do not include step-by-step methods or broad problem-solving strategies (e.g.. "expand the equation." "calculate the sum").
Prompt A.2 Prompt for New Problem Generation
Please design a brand-new mathematics problem that integrates both "[Key Concept A]" and "[Key Concept BJ", according the following requirements:
1. The problem must organically combine the two key concepts within a single mathematical scenario real-life context. Do not simply split them into two independent sub-questions
The content the problem should be free logical and mathematical errors or ambiguities; the conditions must be sufficient and clearly stated.
3. The problem should have certain level of difficulty, providing challenge and thoroughly testing students' ability to apply knowledge comprehensively.
4. The context can be realistic or innovative, but the question must remain coherent and natural. You may introduce additional key concepts needed create well-formed and mcaningful problem.
5. Expression should be concise and elearly structured, without unnecessary or irrelevant information.
6. The problem should have a unique and definite answer or solution path.
7. Choose appropriate question type (such free response, proof, sein-thh-blande multiple choice) according to the chosen key concepts.
B Calculation of Synthesis Cost
For synthesis cost, we posit that methods using closed-soures source models incur cost solely from the closed-source model C whereas for our method, we only need account for GPU usage cos Based on information from the web pages, the input cost of GPT-4 $ per 1 M tokens, and the output cost is $30 per 1M tokens. For GPT-3.5, the input cost is $1.5 per 1M tokens and the output cost $2 per 1M tokens. The cost of using one NVIDIA RTX A100 (80G) is $0.42 per hour.
According to our experiments, synthesizing and scoring problems and solutions requires at least 1000 input tokens and 400 output tokens (with slight differences between various methods). For data synthesis using GPT-4, the cost of synthesizing one data point is calculated as:
10 0.001 + 30 x 0.0004 0.022S
hhtps://opemaai.cm//ppppppr 'ntps://power.neemind ai/rentinno
13

---

In terms of 0.01 cents, the synthesis cost is 220.
For data synthesis using GPT-3.5, the cost of synthesizing one data point is calculated
1.5 x 0.001 + 2 x 0.0004 = 0.0023
In terms of 0.01 cents, the synthesis cost is 23.
For GRIP, we leveraged the vLL.M [1] and used 8 NVIDIA A100 GPUs for 36 hours to construct 2.1 million data points. The cost synthesizing one data point calculated as:
0.42 x 36
0.0000575
2123345
In terms of 0.01 cents, the synthesis cost is 0.57.
we were synthesize million math problems and solutions, it would cost $44000 using GPT-4, $4600 using GPT-3.5, but only $1 14 using GRIP. This gap becomes even more pronounced as the data volume increases.
C Benchmarks Overview
This section briefly introduces the datasets used in this paper, including the mathematical reasoning dataset, the scientific reasoning dataset, and the general ability dataset.
MATH is a new dataset of 12,500 challenging competition mathematics problems. Each problem MATH has full step-by-step solution which can be used teach models to generate answer derivations and explanations.
MATH 12::
GSM8K 77: This test dataset contains 1.32K diverse grade school math problems, intended to test basic arithmetic and reasoning ability in an educational context.
GAOKAO-Eval 32 GAOKAO-Eval is benchmark from China's Gaokao exam, covering various subjects and question types. Questions include multiple-choice, problem-solving reading comprehen- sion, and essay writing, with subjective answers scored by high school teachers. This paper evaluates only the mathematics test.
SVAMP [24: SVAMP is challenge set for evaluating models on elementary-level Math Word Problems (MWP). The dataset contains a total of 1, 1,000 problems. Each MWP consists of short natural language narrative that describes a state of the world and poses a question about some unknown quantities.
ARC includes questions derived from various grade-level science exams, testing models' ability to handle both straightforward and complex scientific queries. We use the challenge subset, which contains 1,172 test questions.
ARC-C [
Spanning 57 subjects across multiple disciplines, MMLU evaluates the breadth and depth of a model's 's knowledge in manner akin to academic and professional testing environments. We select the STEM subset of MMLU with 3.13K problems.
MMLU-STEM [3::
GPOA-Diamond 26
This dataset provides "Google-prooff questions in biology, physics, and chemistry, designed to test deep domain expertise and reasoning under challenging conditions. We use the diamond subset containing 198 hard problems.
Consisting of 23 tasks previously found challenging for language models from BIG-Bench (Srivastava al., 2023), BBH contains a total of 6511 challenging problems examining the capability of LLMs to solve them.
BIG-Bench Hard (BBH) 28
TheoremQA [5: Focused applying mathematical theorems to solve advanced problems in fields such as mathematics, physics, and engineering, TheoremQA includes 800 questions that test the theoretical reasoning capabilities.
MBPP 2 MBPP consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entry-level programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of task description, code solution, and automated test cases.
14

---

C-Eval is a comprehensive Chinese evaluation suite designed to assess the advanced knowledge and reasoning abilities of large language models. It includes multiple-choice questions across four difficulty levels (middle school, high school, college, and professional) and spans 52 diverse disciplines.
C-Eval [6]:
MMLU [3]: MMLU (Massive Multitask Language Understanding) is a benchmark that measures text models' multitask accuracy across 57 tasks, including elementary mathematics, US history, computer science, and law. It requires extensive world knowledge and problem-sol abilities, but even the best models still need significant improvements to reach expert-level accuracy.
D
Dual Filtering and KC Examples
D.1 Dual Filtering
Ensuring the quality of KCs is crucial, as using meaningless KCs can result low-quality synthesized problems while using overly similar KCs can lead to duplicated problems. These issues increase the computational and time costs for both problem synthesis and problem quality validation. We employ dual filtering strategy using both embedding models and LLMs to remove low-quality and duplicated KCs. The three main steps are as follows:
Eliminating Low- -uaaliyl KCs: LLMs are used to filter out KCs that are vague, contain mathematical emror,, overly detailed. This is because vague KCs can be broad in meaning, failing to standardize the model's output effectively. Erroneous KCs may lead the model to synthesize incorrect questions, while overly detailed KCs can overly constrain the model's output.
Categorization: We first use an embedding model to calculate pairwise similarity scores between KCs. KCs with similarity scores between 0.90 and 1.0 are deemed to have the same meaning, while those with scores between 0.70 and 0.90 undergo an additional check by the LLM to confirm they truly synonymous. KCs with scores below 0.70 treated as distinct. Based on this process, KCs are grouped into classes with similar KCs placed in the same class. These thresholds were determined through an analysis of the KC set.
Summarization: For each KC class, the LLM identifies the most representative KC to act as the class representative. If no existing KC in the class is suitable, the LLM synthesizes new KC to represent the class. Finally, we obtained 10K qualified key concepts.
When only the embedding model was used for de-duplication, the quality check revealed that only 26% of the synthesized problems met the quality standard. After introducing dual filtering with the LLM, this proportion increased to 45%. This demonstrates that the dual filtering process significantly improves dataset quality while reducing problem synthesis costs.
D.2 Examples of Bad Key Concepts
The LLM helps the embedding model classify KCs that appear similar but actually have different meanings. For example:
"Geometric sequence" vs. "Arithmetic sequence" (similarity score: 0.805)
"inn function in trigonometry" vs. "Cosine function in trigonometry" (similarity score: 0.865)
The LLM effectively removes vague, mathematically incorrect, or overly detailed KCs. For example:
Vague KCs:
"Problem-soovving strategies"
"Mathematical techniques"
Mathematically Incorrect KCs:
"The sum of the outer angles of a polygon depends on the number of sides"
"The matrix result of multiplying matrix by its inverse is the matrix itself" -"A series converges if its terms approach zero."
Some incorrect or incomplete KCs
15

---

Overly Detailed KCs:
"oovinn the quadratic equation x² + 5z + 6 = 0 by factoring...
Some specific problems
D.3 Examples of Key Concepts
To demonstrate the diversity and comprehensiveness our knowledge base, we randomly sampled 20 KCs:
"Angle of Rotation", "The unit circle and properties". "Solving Equations with Multiple Variables", "Right triangles in sphere", "Inversions in permutations", "Pi (x) as constant in geometry and trigonometry". "Perfect Cubes", "Area of Triangles and Squares", "Diophantine Approximation "Perimeter of triangle". "Abundant Number", "Graphing hyperbola". "Determining the base and height of Parallelogram", "Difference of cosines formula". "Quartic Polynomial", "Polynomial Inequalities", "Congruence of Integers". "Solving equations involving digits", "Sign Analysis", "Calculation of expected value for a fair eight-sided die".
E
Additional Experiments and Analyses
E.1
Performance on Additional Challenging Benchmarks
To further reasonably demonstrate the performance gains of GRIP on more difficult benchmarks, we have already added some relatively challenging test benchmarks (e.g.. GPQA-Diamond, TheoremQA) to Table X in the paper, where GRIP shows significant performance improvements compared to the base models. Furthermore, to further evaluate the models' ability to solve particularly complex mathematical problems, we have also introduced the AIME 2024 dataset and tested their pass @ 64 performance. These results, particularly on the notoriously difficult AIME benchmark, show a promis- ing signal that GRIP can enhance complex reasoning capabilities, even if the absolute performance remains a frontier challenge. This improvement from zero demonstrates GRIP's potential to unlock new abilities in base models.
Table 8: Performance additional challenging benchmarks. Results for GPQA-Diamond and TAeoremQA are also presented in the main text, but are included here for comprehensive comparison with AIME 2024.
Model
Size
GPQAADiamoud (Ace)
TheoremQA (Acc)
(pass(644)
Mistral
7B
24.7
19.2
0/30
MLMMA3
8B
27.2
20.1
0/30
Qwen2
7B
33.5
GRIP GRIP
Mistral
7B
31.7
26.5
4/30
LLaMA3
8B
30.8
24.2
3/30
GRIP
Qwen2
7B
33.4
34.8
6/30
E.2
Decontamination Analysis
To ensure the integrity of our results and address potential data contamination from benchmark test sets, we conducted a thorough decontamination analysis. Our synthesis method, GRIP, is fundamentally designed to generate novel problems by modeling key concepts and performing multi- hop combinations, rather than rephrasing or imitating existing examples. This design theoretically minimizes the risk of direct duplication. As demonstrated in our main analysis Table I our synthetic data exhibits low similarity to its seed data and achieves a high Novelty Rate of 71.8%.
To quantitatively verify the novelty of our dataset against standard benchmarks, performed formal n- n-gram overlap analysis between our GRIP-MATH training set and the official MATH test set. After normalizing the text of both datasets by lowercasing and removing punctuation, calculated the percentage of overlapping n- n-grams for various lengths of n.
The results, presented in Table show that the n-gram overlap rate is extremely low, particularly for longer sequences, indicating negligible level verbatim contamination. For instance, the 10-gram overlap is only 0.63%, and drops less than 0.01% for 15-grams.
16

---

Table 9: N-gram overlap analysis between the GRIP-MATH training set and the MATH test set.
Dataset
N=8
N=10
N=13
N=15
MATH
1.94%
0.63%
0.06%
<0.01% %
Furthermore, qualitative analysis of the most frequent overlapping n-grams reveals that they consist of common mathematical phrases, definitions, or generic question stems, rather than specific problem content that would suggest data leakage. The top five most frequent overlapping sequences are:
"What is the smallest possible value of the"
"How many zeros are at the end of
"digit is the same the units digit"
"digit of the sum of the squares of
"is the sum of the lengths of these"
In conclusion, this two-part analysis, combining our method's theoretical design with a direct empirical decontamination check, confirms that our synthesis process effectively avoids significant contamination from the benchmark test sets, thereby ensuring the validity of our evaluation results.
E.3 Validation of Knowledge Concept Adherence
A crucial aspect of our data synthesis pipeline is its ability to generate problems that faithfully adhere to the guiding knowledge concepts (KCs). quantitatively evaluate this, we conducted validatalii experiment. Specifically, employed the same concept extraction model used in our initial pipeline (Qwen2.5-32B- Innttuutt to perform "reverse" concept extraction on a sample of the synthesized problems. Subsequently, calculated the match ratio between the newly extracted concepts and the original concepts that were used to generate these problems.
We define two levels of adherence to measure the fidelity of the synthesis process. Full Match occurs if all of the original input concepts were successfully re-extracted from generated problem. A Partial Match registered if at least one of the original input concepts was found.
Table 10: Adherence ratio of synthe- sized problems to the guiding knowl- edge concepts.
Adherence Level
Adherence Ratio
Full Match Partial Match
88.65%
The results. shown in Table 10 demonstrate very high degree of fidelity. The Full Match ratio of 88 38.65% confirms that our method is highly effective at integrating all speci- fied concepts into a coherent problem. Furthermore, the 98.49% Partial Match ratio indicates that even cases where perfect combination is not achieved, the generated problems remain strongly relevant to the intended knowledge domains. This validation confirms that our data synthesis process not only scalable but also precise following the conceptual guidance provided by the knowledge graph.
98.49%
E.4 Performance Analysis Based on Dataset Scale
To provide more granular analysis of our method's performance, we present supplementary comparison that evaluates various data synthesis methods on a single model architecture (Mistral-7B), with datasets grouped by their approximate scale. The effectiveness of data synthesis method is typically evaluated on two key dimensions: the quality of the generated data and the scalability of the synthesis process. This analysis is designed offer further insight into GRIP-MATH'S 's performance along both of these dimensions. The results are presented in Table 11 which is divided into two categories: datasets with approximately 100K samples and those with over million samples.
Performance at a Smaller Scale (-100k). To facilitate a direct comparison of data quality against datasets of similar size, we randomly sampled 80K question-ansee answer pairs from our full dataset to create
As shown in Table [I] this allows for controlled evaluation where data quantity is normalized. The results demonstrate that the superior quality of data generated by our synthesis method leads to better model performance. Notably, a model trained on (43.0%) significantly surpasses one trained on MathCoder (38.8%) (38.8% which has the same number of samples.
17

---

Table 11: Performance comparison of various data synthesis methods on the Mistral-7B model, grouped by dataset scale. The 'AVG' column refers to the average score on the mathematical reasoning benchmarks detailed in the main paper.
Scale
Datasel
Total Seed Data
Total Synthesized Data
AVG
-100K
MetaMath
WizardMath
15K
96K
38.0
MAmmoTII
220K
262K
39.0
MathCoder
15K
80K
38.8
CRIP-MATH-mini
7.5K
80K
43.0
>/M
MathScale
20K
51.2
MAmmoTT12
>10
10M
52.2
GRIP-MATH
7.5K
2.1M
57.3
Performance at Larger Scale (>1M). The second part of the analysis highlights the key metric of scalability. When comparing our full 2.1 M--mmle dataset against other large-scale methods, GRIP-MATH demonstrates superior efficiency and quality. For example, compared to MAmmoTTH, which filters 10M samples from the massive Common Crawl corpus, our method achieves 5.1 1pontt higher score (57.3% vs 52.2%) using only 7.5K seed examples. Similarly, our method outperforms MathScale (which uses GPT-3.5) by a margin of 6.1 points. This analysis proves that our synthesis method not only scales data volume effectively but also ensures superior data quality scale.
F Limitations and Future Work
The primary limitation of this work is that the empirical validation of the GRIP framework is confined to the domain of mathematical reasoning. While our results demonstrate significant success within this area, the effectiveness of GRIP in other domains has not yet been verified.
Our decision initially focus on mathematics was deliberate. Mathematical reasoning represents a significant challenge for Large Language Models (LLMs), and success in this domain provides strong evidence of data synthesis method's '' efficacy. Furthermore, the well-defined, hicrarchical knowledge structure of mathematics offered an ideal environment to validate our core hypothesis: that novel and complex problems can be synthesized by combining foundational knowledge concepts.
Despite this specific focus, our work provides initial evidence of GRIP's broader applicability. As shown in Table models trained exclusively GRIP-MATH demonstrate improved performance on scientific reasoning benchmarks (e.g., BBH, GPQA-Diamond, MMLU-STEM), indicating a degree of cross-domain generalization. We posit that the core GRIP pipeline- "Concept Extraction - Knowledge Graph Construction - Concept Combination -> Data Synthesis - Filtering" fundamentally agnosiie. The key to extending GRIP to new domains lies in adapting the definition of "concept" to the target domain. We envision two primary directions for this extension:
For structured domains like STEM and programming, the mapping straightforward, as "concepts" can be directly defined as clear rules, principles, or library functions.
For less-structured domains like commonsense reasoning, we hypothesize this paradigm remains effective. Here, "concepts" can be defined as higher-level scenarios or activities. For instance, GRIP could combine two distinct but related concepts, such as "planning an international trip" " and "dealing with a lost passport," " create novel, complex reasoning problem that leverages their implicit relationships.
Therefore, a top priority for our future research to rigorously extend and validate the GRIP framework in these diverse domains, including commonsense and procedural reasoning. Verifying its effectiveness in these areas will crucial for establishing GRIP as a truly general-purpos data synthesis methodology.
18

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The detailed procedure of GRIP is presented in Section 3 and the data analysis of CRIP-MATH is discussed in Section 3.5 The experimental results in Section 4.3 further demonstrate the effectiveness of our approach.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
19

---

Intuitively, GRIP should be applicable to various domains where data can be decomposed into key concepts: however, our current experiments are limited to the mathematics domain, and its effectiveness in other areas is yet to be verified. (Appendix
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
demonstrate the high scalability, low cost, high quality and diversity of the synthetic data. In the experiments of Section 4.33 and Section 4.4] we show the effectiveness and strong generalization ability of GRTH,MATH, and the ablation studies in Section 4.5 further illustrate the importance of each component of GRIP.
Justification: In Section 3.5
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
20

---

Answer: [Yes]
Justification: In S we provide a detailed introduction to all the algorithmic proce- dures of GRIP well as the designed prompts. We will also open-source the code and dataset. Due to company intellectual property limits, we can only provide part of the dataset during the review process, but all code and datasets will be open-sourced immediately after the review period ends.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
We will open-source the code and dataset. Due to company intellectual property limits, can only provide part of the dataset in the supplementary materials during the review process, but all code and datasets will be open-sourced immediately after the review period ends.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
21

---

reproducible, they The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httppss.ccppulllcggiideeeessseee
sesbbbbmmssiiooo www.ssssiiiiiiiiiii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have described the experimental settings used in Sections 4.1 Sections 4.2 as well as in the ablation studies Section 4.5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: The paper does not report error bars or other measures of statistical significance for the experimental results.
Guidelines: This paper includes holdout experiments to assess statistical significance and provide error bars for the reported results.
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
22

---

8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: The computational resources used for the experiments are provided in Section and the cost information is detailed in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
The research is conducted with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e-g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: GRIP provides a practical and efficient method for data synthesis, which is crucial for enhancing the capabilities of large language models as the availability internet data decreases in the future.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
23

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual- usal-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: This paper cite the original papers such as dataset.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
This paper will release code for GRIP.
Guidelines:
The answer NA means that the paper does not release new assets.

---

Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: The paper provides detailed description of the large language model used in our experiments.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
25

---

Please refer to our LLM policy http:s//heerriisccc///ofrrrrrrsr for what should or should be described.
26


Reviews:
[{"review_id": "KCZ8PHw6Jk", "reviewer": "Reviewer_Fk5a", "summary": "This paper presents GRIP, a graph-based framework that synthesizes high-quality mathematical reasoning data by extracting key concepts from seed problems and leveraging both explicit and implicit concept relationships. From 7.5K seed problems, GRIP generates 2.1M diverse question-answer pairs at 99% lower cost than GPT-4 methods while achieving 71.8% novelty rate versus under 40% for existing approaches. Models trained on GRIP-MATH significantly outperform base models and previous synthesis methods on mathematical reasoning benchmarks, demonstrating the effectiveness of concept-driven data generation.", "questions": "See Weaknesses.", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The motivation is interesting, and the paper is easy to understand.\n\n2. Compared to other data synthesis methods, GRIP used less money to synthesize more data and achieve better results.\n\n## Weaknesses\n\n1. How do you validate the accuracy of the synthesized 2.1M questions?\n\n2. The comparison in Table 2 seems unfair. To properly validate a `training set`’s merit, a reasonable, fair setup should randomly sample the \"same number\" of Q-A pairs from different corpus, then apply SFT on the same LLM backbone. Table 2's evaluation appears unfair - why do other data synthesis methods experiment on Mistral and Qwen1.5 with fewer training tokens than GRIP?\n\n3. How do you store the constructed KB? Can you provide a visualization interface for all the knowledge comcepts extracted by GRIP and their relationships, using online KG like neo4j?\n\n4. What advantages does using open-source models for data synthesis have over GPT-series. Can you provide performance evidence? I don't think cost is the only reason, since other data synthesis methods using current powerful open-source models might achieve better results than GRIP, so this needs proof.", "was_revised": false, "final_justification": "", "rebuttal": "## On validating data accuracy\n\nThank you for raising this critical question. Ensuring the accuracy of a synthetically generated dataset of such a large scale (2.1 million) is paramount, and we designed our methodology with this challenge at its core.. Given that manual verification of the entire dataset is infeasible, we designed and implemented a rigorous, multi-stage automated quality control process to maximize the quality of the synthetic data, as detailed in Section 3.4 of our paper. This process is built on three main layers of validation:\n\n- **Evaluation and Filtering of Questions:** We employed a multi-model supervisory framework composed of three advanced open-source mathematical models. Each model scored every generated question based on two dimensions: logical soundness and expressive completeness. Only questions surpassing a high weighted-average score of 0.85 are retained, ensuring the clarity and validity of the problems themselves.\n\n- **Evaluation and Filtering of Solutions:** For the evaluation of solutions, we adopted a more stringent \"unanimous consent\" mechanism. A solution was accepted **only if** it was judged to be completely correct by all three evaluation models simultaneously. This high-standard design aims to maximize the accuracy of the problem-solving process and the final answers in our dataset.\n\n- **Validation of the Automated Evaluation Framework's Efficacy:** To demonstrate the reliability of the aforementioned automated evaluation process, we manually annotated 500 synthetic samples in our ablation study (Section 4.5, Table 5). The results showed that the judgment accuracy of our three-model framework reached **95.7%This result not only confirms the reliability of our process but also shows it outperforms the powerful, closed-source GPT-4.1 (94.3%) on the same task. This comparative experiment provides strong evidence that our quality control process is both reliable and efficient, even surpassing the accuracy of a costly single closed-source model.\n\nIn summary, through the separate evaluation of questions and solutions, strict filtering criteria (weighted scoring and unanimous consent), and validation via manual sampling, we have constructed a closed-loop quality assurance system to ensure the accuracy of the dataset to the greatest extent possible. We will further emphasize the details of this process in the revised manuscript to address your concerns more clearly. \n\n## On the fairness of the comparison in Table 2\n\nThank you for your insightful comment regarding the fairness of the comparison in Table 2. We agree that a controlled comparison is vital for validating a training set's merit.\n\nThe effectiveness of a data synthesis method is evaluated on two key dimensions: the **quality** of the generated data and the **scalability** of the synthesis process. A primary research goal in this field is to **generate large-scale, high-quality data from limited seed data at a low cost.** Our initial experiments, following the previous works, were designed to demonstrate this scalability by using our full dataset across multiple base models. To provide a more direct and fair comparison, as you suggested, we have isolated the results for all methods on the same model architecture  (Mistral-7B) and present a more granular analysis below. We have broken it down into two parts:\n\n|        | Datasets             | Seed Dataset Count | Dataset Count | Avg      |\n| :----- | :----------------- | :----------------- | :------------ | :------- |\n| ～100K | MetaMath           | 15K                | 395K          | 40.3     |\n|        | WizardMath         | 15K                | 96K           | 38.0     |\n|        | MAmmoTH            | 220K               | 262K          | 39.0     |\n|        | MathCoder          | 15k                | 80K           | 38.8     |\n|        | **GRIP-MATH-mini** | **7.5k**           | **80k**       | **43.0** |\n| >1M    | MathScale          | 20K                | 2M            | 51.2     |\n|        | MAmmoTH2           | >10M               | 10M           | 52.2     |\n|        | **GRIP-MATH**      | **7.5k**           | **2.1M**      | **57.3** |\n\n- **Performance at a Smaller Scale (~100k)**\n\nTo directly address your point about comparing datasets of the \"same number,\" we randomly sampled 80k Q-A pairs from our synthesized data to create **GRIP-math-mini**. This allows for an apples-to-apples comparison with datasets of a similar size, like MathCoder (80k). The results show that even when controlling for data quantity, our synthesis method's superior quality leads to better performance, outperforming existing methods in this category. For instance, **GRIP-math-mini (43.0%)** significantly surpasses **MathCoder (38.8%)**.\n\n- **Performance at a Larger Scale (>1M)**\n\nThis comparison highlights the second key metric: scalability. When comparing our full 2.1M dataset against other large-scale methods, GRIP-math demonstrates superior efficiency and quality. For example, compared to MAmmoTH2, which filters 10M samples from the massive Common Crawl, our method achieves a 5.1% higher score (57.3% vs 52.2%) using only 7.5k seed examples. Similarly, our method outperforms MathScale (which uses GPT-3.5) by 6.1%. This proves that our synthesis method not only scales data volume effectively but also ensures superior data quality.\n\nWe will update the manuscript to include this detailed table and analysis. This will make the evaluation more transparent, comprehensive, and address your valid concerns about fairness. Thank you again for this valuable suggestion.\n\n## On Knowledge Base storage and visualization:\n\nThank you for your question. We store the constructed Knowledge Base in a structured and portable **JSON format**. A sample entry looks like this:\n\n```\n{\n  \"key_concept1\": [1, 2, 8, 15],\n  \"key_concept2\": [2, 8, 22, 31, 57]\n  ...\n}\n```\n\nThis indicates, for example, that key_concept1 appears in problems 1, 2, 8, and 15; key_concept2 appears in problems 2, 8, 22, 31 and 57. If two key concepts co-occur in any problem (e.g., both appear in problems 2 and 8), we add an edge between them. The edge weight is defined as the number of problems where both appear together (here, weight = 2). \n\nWe completely agree that visualization is crucial. To best enable this, we chose a foundational JSON format. Its simplicity and universality make the knowledge base maximally portable and accessible. More importantly, by using Python scripts based on the methodologies described in Sections 3.1-3.3 of our paper, this JSON data is programmatically processed into a **weighted undirected graph**. This graph structure explicitly defines the nodes (concepts), the relationships between them, and the weights of these relationships. Because these attributes are well-defined, the entire Knowledge Base can be easily imported into various graph analysis tools or databases, such as Neo4j, for further exploration.\n\nDue to rebuttal guidelines that prohibit image uploads, we are unable to provide visualizations or a web interface. However, **inspired by your valuable feedback**, we will incorporate high-resolution static visualizations of key subgraphs in the final version of our paper. These images will serve to intuitively illustrate the structure of our knowledge base, showcasing elements such as the neighborhoods of specific concepts, multi-hop reasoning paths, and concept clusters.\n\n## On the advantages of using open-source models:\n\nThanks for your question. To be clear, the open-source models we use for data synthesis and evaluation are weaker than the GPT series, which is the tool used by the competing methods listed in Table 1. While the significant cost savings are a major benefit (as noted in Table 1), the advantages of using a well-designed open-source framework extend far beyond cost. We have summarized the following key advantages:\n\n**Robust Quality**:  Even if the models used in our data synthesis process are weaker than the GPT series, a carefully designed multi-model open-source framework can match and even surpass the quality of a single, powerful closed-source model. We provide direct performance evidence for this in our ablation study (Section 4.5, Table 5). In a critical comparison, our collaborative evaluation framework, which utilizes three open-source models, achieved an accuracy of **95.7%**. This was notably higher than the **94.3%** accuracy achieved when using the powerful, closed-source GPT-4.1 model alone for the same task. \n\n**Stability, Control, and Scalability**: Open-source models provide the stability and control necessary for massive data synthesis projects. Proprietary models like ChatGPT and Gemini have significant usage restrictions, including non-permissive licenses, regional availability limitations, and strict API rate limits. These factors can hinder large-scale generation. Our multi-model collaborative method circumvents these issues, enabling the rapid, stable, and low-cost synthesis of large datasets while avoiding potential licensing and operational risks.\n\n**Flexibility**: The open-source ecosystem is evolving rapidly with the constant emergence of more advanced models (e.g., from DeepSeek, Qwen, Kimi, Llama). Our framework is inherently modular, which grants us the flexibility to easily swap or integrate different state-of-the-art open-source models based on specific task requirements or performance needs. \n\n**Furthermore, data diversity is what truly distinguishes GRIP from other synthesis pipelines.** While other methods might simply rephrase seed data, GRIP's graph-based approach explores **implicit relationships** between knowledge concepts to generate truly novel problems. This is proven by our **71.8% novelty rate (Table 1)** and the **lower seed data similarity visualized in Figure 4**, both confirming that GRIP-MATH is not just a rehash of the original data.\n\n## Paper Formatting Concerns:\nThanks! The 'App. ??' error points to Appendix D."}, {"review_id": "9lyv2QYeWI", "reviewer": "Reviewer_3rLC", "summary": "The paper introduces a method of synthetic data generation that leverages concepts graphs to produce novel and diverse data. GRIP builds a key concept graph from seed data and then synthesizes new problems by prompting an LLM with concept combinations selected from this graph.", "questions": "* The paper does not mention data decontamination, which is a standard procedure for synthetic data generation. Could the authors perform and report the results of a standard n-gram decontamination analysis between synthetic dataset and all benchmark test sets? If significant overlap is found, please include a discussion on how this might affect the paper's conclusions. \n\n* The paper would be strengthened by a more explicit justification for the choice of models and benchmarks for experiments. Given that benchmarks like GSM8K are largely considered solved by frontier models, could the authors please elaborate on why this setting was chosen?", "limitations": "No. The authors have not included a limitations as a separate section and briefly mention them in the section 5. A key limitation that should be discussed is the choice of non-reasoning models and simple benchmarks for evaluation. An honest discussion would acknowledge that while the generated data shows promise on these models, its utility for state-of-the-art long-reasoning models remains unverified.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n* The paper is generally well-structured, with a clearly stated motivation and reasonable experiments and ablation studies.\n* The described method is novel and makes a significant contribution to the field of synthetic data generation. In particular, its ability to generate diverse problems while avoiding the direct use of seed questions and the use of implicit concept connections makes the approach valuable.\n* The authors' commitment to release their code and the 2.1M problem dataset is another important contribution.\n\n## Weaknesses\n\n* **Focus on the non-reasoning models** \n\n    The evaluation is conducted on benchmarks (e.g., GSM8K, SVAMP) that are considered largely saturated and with models that are far behind the current frontier reasoning models, such as QwQ-32B and DeepSeek-R1. While the experimental validation of the method is thorough, its capabilities are not demonstrated on the more complex benchmarks (e.g. AIME24/25, HMMT24/25, GPQA Diamond) that are more conventional now. The lack of justification for this experimental design choice weakens the paper's overall claims.\n\n* **Lack of decontamination**\n\n    Even though the presented method does not directly rephrase seed problems, a standard concern for synthetic data generation is potential contamination with benchmark test sets. The paper does not provide any details regarding decontamination.\n\n* **Incomplete cost comparison**\n\n    The cost comparison in Table 1 does not include any open-source-based data generation methods. This leads to an incomplete comparison, as all listed baselines rely on proprietary models (e.g., GPT-3.5, GPT-4), making it difficult to fully assess GRIP's cost-effectiveness against non-proprietary alternatives.", "was_revised": true, "final_justification": "I am satisfied with the authors’ clarifications; they resolve most of my concerns. I will update my score.", "rebuttal": "## Justification of Benchmark and Model Choices in Evaluation\n\nThank you for your insight on our experimental design choices. We chose benchmarks like GSM8K and MATH, as well as foundational models, based on the following considerations.\n\n**Evaluation consisency**：As shown in Table 2, We initially focused on benchmarks such as GSM8K and MATH using widely adopted foundation models (e.g., Mistral, Qwen, Llama) to maintain consistency with prior data synthesis studies (e.g., MathScale, MetaMath). These benchmarks are still broadly accepted in evaluating mathematical reasoning and allow fair comparison across methods under equivalent training conditions. Importantly, our goal was to isolate and validate the effectiveness of the GRIP data synthesis strategy, independent of model capabilities.\n\n**Evaluation on Challenging Benchmarks**:  To further reasonably demonstrate the performance gains of GRIP on more difficult benchmarks, we have already added some relatively challenging test benchmarks, e.g. GPQA TheoremQA, to Table 2 in the paper, where GRIP shows significant performance improvements compared to the base models. Furthermore, to further evaluate the model's ability to solve particularly complex mathematical problems, we have also introduced the AIME 2024 dataset and tested their pass@64 performance. These results, particularly on the notoriously difficult AIME benchmark, show a promising signal that GRIP can enhance complex reasoning capabilities, even if the absolute performance remains a frontier challenge. This improvement from zero demonstrates GRIP's potential to unlock new abilities in base models.\n\n|              | GPQA-Diamond（Acc） | TheoremQA（Acc） | AIME2024（pass@64） |\n| ------------ | ------------------- | ---------------- | ------------------- |\n| Mistral   | 24.7                | 19.2             | 0/30                |\n| GRIP-Mistral | 31.7                | 26.5             | 4/30                |\n| LLaMA3       | 27.2                | 20.1             | 0/30                |\n| GRIP-LLaMA3   | 30.8                | 24.2             | 3/30                |\n| Qwen2        | 32.3                | 33.5             | 3/30                |\n| GRIP-Qwen2   | 33.4                | 34.8             | 6/30                |\n\n**Complementary to reasoning-based approaches**: GRIP is designed as a general, scalable data synthesis framework that is **model- and benchmark-agnostic**. Our core contribution lies in enabling large-scale generation of diverse and high-quality reasoning data by systematically modeling concept-level relationships, rather than relying on specific CoT traces or frontier model outputs. This approach **complements existing methods** that improve reasoning through CoT distillation or reinforcement learning. While our current experiments focus on commonly used seed data and foundation models to ensure evaluation consistency, GRIP is fully compatible with stronger seeds and models. As part of ongoing work, we plan to apply GRIP to more advanced datasets (e.g., DeepScaleR) and frontier models such as DeepSeek-R1 and Qwen3-235B to assess its effectiveness under high-capacity reasoning settings. Due to rebuttal time constraints, we could not complete such experiments, but we will clearly acknowledge this in the paper’s “**Limitations**” section and treat it as a key direction for future work.\n\n\n\n## Lack of decontamination\n\nThank you for your valuable comment regarding data decontamination. While our paper indirectly demonstrated the novelty of our GRIP dataset compared to the seed data, we acknowledge that a direct analysis of potential contamination from benchmark test sets is necessary. Our synthesis method, which is based on modeling **key concepts and performing multi-hop combinations,** is fundamentally designed to generate new problems rather than rephrasing or imitating existing ones. This theoretically minimizes the risk of direct duplication. For example, the \"two-hop\" and \"three-hop\" problem structures generated by our method are largely absent from the original seed dataset. Our analysis in **Section 3.5 (Figure 4, Table 1)** already supports this, showing that our synthetic data exhibits **low seed similarity** and a **high Novelty Rate** of 71.8%.\n\nTo directly address your concern about benchmark contamination, we conducted a formal n-gram overlap analysis between our GRIP-math training set and the MATH test set. After normalizing the text of both datasets, we calculated the overlap rate for various n-gram lengths. The results are as follows:\n\n| N    | 8     | 10    | 13    | 15     |\n| ---- | ----- | ----- | ----- | ------ |\n| Math | 1.94% | 0.63% | 0.06% | <0.01% |\n\nThe results clearly show that the n-gram overlap rate between the GRIP-math training set and test benchmark is extremely low, especially for longer sequences, indicating a negligible level of verbatim contamination. Furthermore, a qualitative analysis of the most frequent overlapping n-grams reveals that they consist of common mathematical phrases, definitions, or generic question stems, rather than specific problem content. Here are the top 5 overlaps:\n\n```\n\"What is the smallest possible value of the\"\n\"How many zeros are at the end of\"\n\"digit is the same as the units digit\"\n\"digit of the sum of the squares of\"\n\"is the sum of the lengths of these\"\n```\n\n\nIn conclusion, both our method's theoretical design and this direct decontamination analysis confirm that our synthesis process effectively avoids significant contamination from the benchmark test sets. We will add this detailed analysis to the revised manuscript to make this explicit.\n\n\n\n## Incomplete cost comparison\n\nThank you for your valuable suggestion regarding the cost comparison in Table 1.  A direct cost comparison with a few methods using open-source models is indeed challenging. Unlike proprietary models, where costs can be directly calculated from API pricing, the cost of running open-source models depends heavily on implementation details, hardware, and specific synthesis settings, making a direct, apples-to-apples comparison difficult.\n\nHowever, to address your concern, we incorporate a **rough cost comparison** in our analysis. We have selected the **MAmmoTH2** method as a representative open-source baseline, as it synthesizes data by filtering and rewriting pre-training corpora using an open-source model. To quantitatively compare the cost-effectiveness of the two methods, we establish the following assumptions for cost comparison: \n\n- **Base Cost Unit:** The cost of processing *n* tokens with a 7B model is defined as 1 unit.\n- **Task Length:** The average processing length for a standard question-answering task is *n*, while for a raw document processing task, it is *5n*.\n- **Model Cost Multipliers:** 7B=1.0x, Mixtral-8x22B=5.5x(for activated parameters), 32B=4.5x, 72B=10.0x.\n\n**GRIP Method Cost Analysis**:\n\n- Question Generation (32B model): 3M tasks × 4.5x = 13.5M units\n- Solution Generation (7B & 72B models): (2M × 1.0x) + (1M × 10.0x) = 12.0M units\n- Filtering (3x7B models, 2 stages): 6M tasks × 3 × 1.0x = 18.0M units\n- **Total**: ~43.5M units for 2.1M items -> **20.7 units/item**\n\n**MAmmoTH2 Method Cost Analysis**:\n\n- Document Processing (72B model): 18M tasks × 5x length × 10.0x = 900.0M units\n- Data Augmentation (Mixtral 8x22B & 72B): (5M × 5.5x) + (5M × 10.0x) = 77.5M units\n- **Total**: ~977.5M units for 10M items -> **97.75 units/item**\n\nTherefore,  the per-sample synthesis cost of MAmmoTH2 is **4.7 times** higher than that of GRIP. It is crucial to note that this comparison only covers the open-source computation. MAmmoTH2’s pipeline additionally incurs substantial costs from proprietary APIs (GPT-4 and GPT-3.5) for its initial filtering, a cost that GRIP entirely avoids. Therefore, the total cost-effectiveness gap is even larger than our conservative 4.7x estimate. We will add this detailed comparison to the revised manuscript to fully address your concern.\n\n## Paper Formatting Concerns:\nThank you for your careful review and for pointing out the formatting error on Line 130. This was indeed a LaTeX label rendering issue where \"Appendix ??\" was displayed incorrectly. We have now corrected this cross-reference, and it correctly points to Appendix D in the revised manuscript. We appreciate your valuable feedback."}, {"review_id": "rMaeR91yfK", "reviewer": "Reviewer_Bfeb", "summary": "This paper introduces **GRIP** (Graph-based Reasoning Instruction Producer), a novel framework for synthesizing large-scale, high-quality, and diverse mathematical reasoning data. In contrast to traditional data synthesis approaches that rephrase or augment seed examples, GRIP departs from direct manipulation of seed data. Instead, it constructs a knowledge graph from extracted key concepts and leverages both explicit and implicit relationships within the graph to synthesize new questions. A multi-model supervision mechanism is employed to filter the generated data, ensuring quality without relying on expensive closed-source models. Applied to the MATH dataset, GRIP synthesizes a 2.1M dataset (GRIP-MATH) from 7.5K seeds, achieving a 280× expansion while significantly reducing generation cost. Empirical evaluations show that models trained on GRIP-MATH outperform prior data synthesis baselines and even rival proprietary math-specialist models in both mathematical and scientific reasoning tasks.", "questions": "1. Figures 1 and 2 are difficult to read due to small font and dense layout. Could the authors improve the resolution, enlarge the key annotations, or separate parts of the diagrams for better interpretability?\n\n2. Lines 45–57 discuss the scalability benefits of using non-co-occurring concept pairs. Is there any empirical analysis or ablation study that directly supports this observation (e.g., concept pair count vs. effective data generation rate)?\n\n3. The current evaluation is limited to mathematical reasoning tasks. Could the authors comment on how GRIP might generalize to other domains (e.g., science, programming, commonsense QA), possibly by leveraging semantic graphs or domain-specific ontologies?\n\n4. The construction process of the KCRG appears heuristic. Is there a formally defined algorithm or pseudo-code for how the graph is constructed, especially for identifying and weighting implicit relationships?\n\n5. The evaluation primarily focuses on benchmark accuracy, but lacks qualitative insights.\n   Could the authors include a small-scale human evaluation or case study to highlight strengths and potential failure modes in the generated data?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The paper conducts comprehensive experiments across multiple benchmarks and model architectures, with detailed ablations that validate the effectiveness and scalability of the proposed method.\n\n2. GRIP introduces a novel graph-based synthesis strategy that systematically combines both explicit and implicit concept relationships, enabling diverse and large-scale generation beyond prior methods.\n\n## Weaknesses\n\n1. The method is only evaluated on mathematical data, and its applicability to domains lacking structured concept representations (e.g., commonsense reasoning) remains unverified.\n\n2. Data quality tends to degrade as the graph hop distance increases, which may reduce the reliability of more diverse combinations even with filtering.\n\n3. The evaluation focuses primarily on quantitative metrics, with limited qualitative analysis of synthesized examples, making it difficult to assess failure modes or interpretability aspects.", "was_revised": true, "final_justification": "After carefully considering the author’s rebuttal, the AC’s input, and the points raised in the other reviewers’ discussions, I find that my original assessment remains valid. While the rebuttal addressed certain concerns, it did not sufficiently resolve the key issues I raised. Therefore, I will keep my original score.", "rebuttal": "## Generalization to Other Domains\n\nThank you for this insightful comment. We agree that the generalization of GRIP is a crucial point. We would like to address this from three perspectives: \n\n**Rationale for Starting with Mathematics**: Following the existing works, our experiments with GRIP are focused on the domain of mathematical reasoning. We chose mathematics as the initial application area for two primary reasons. First, mathematical reasoning is one of the most challenging tasks for LLMs; success in this domain provides strong evidence of a data synthesis method's efficacy. Second, the well-defined, hierarchical knowledge structure of mathematics offered an ideal environment to validate our core hypothesis: generating novel and complex problems by combining foundational knowledge concepts.\n\n**Generalization Potential of the GRIP Framework:** While mathematics was our primary focus, our work already provides initial evidence of GRIP's broader applicability. As shown in **Table 3**, models trained on GRIP-MATH demonstrate improved performance on scientific reasoning benchmarks (e.g., **BBH, GPQA, MMLU-STEM**), indicating cross-domain generalization.\n\nWe posit that the core GRIP pipeline—**“Concept Extraction → Knowledge Graph Construction → Concept Combination → Data Synthesis → Filtering”**—is fundamentally domain-agnostic. The key is adapting the **definition of a \"concept\"** to the target domain.\n\n- **For structured domains** like STEM and programming:  the mapping is straightforward, as they rely on clear rules and principles.\n\n- **For less-structured domains** like commonsense reasoning, we hypothesize this paradigm remains effective. Here, \"concepts\" can be defined as higher-level scenarios or activities. For instance, GRIP could combine two distinct but related concepts, such as **\"planning an international trip\"** and **\"dealing with a lost passport.\"** Their combination creates a novel, complex reasoning problem by leveraging implicit relationships, which is GRIP's core strength.\n\n**Acknowledged Limitations and Future Work:** We are transparent about the scope of our current experiments. As stated in our conclusion (**Section 5**), *\"Intuitively, GRIP should be applicable to various domains... however, our current experiments are limited to the mathematics domain, and its effectiveness in other areas is yet to be verified.\"* Extending and rigorously validating the GRIP framework in diverse domains, including commonsense and procedural reasoning, is a top priority for our future research.\n\n## Data Quality and Hop Distance\n\nThank you for this insightful comment. We agree and have also observed that data quality tends to decrease with greater hop distances. This is precisely why we **integrated specific strategies** to mitigate this effect while still harnessing the **diversity from implicit relationships**. \n\n**Direct Observation**: Our ablation study in **Table 4** confirms this trend. The average quality score drops from **0.94 (one-hop)** to **0.72 (two-hop)** and **0.62 (three-hop)** as distance increases.\n\n**Mitigation Strategies**: To ensure the reliability of more distant combinations, we implemented two key strategies:\n\n- **Hub Concept Prioritization**: For three-hop relationships, we exclusively generate combinations involving a **hub concept**—a central, highly-connected node in our knowledge graph. This maintains semantic relevance even across longer distances.\n- **Weight-Based Filtering**: We apply stricter filtering to multi-hop combinations, removing low-weight (i.e., less frequent or relevant) pairs to ensure the selected concepts are information-rich.\n\n**Effectiveness of Strategies**: The effectiveness of the hub concept strategy is validated in **Table 4**, which shows that three-hop combinations generated *with* our hub concept strategy achieve a score of **0.62**, whereas those generated *without* it perform significantly worse at just **0.39**.\n\n**Performance evidence**: Most importantly, the final model performance demonstrates the value of this curated diversity. As shown in **Table 7**, training on a dataset that includes these carefully filtered two-hop and three-hop implicit combinations leads to a notable improvement on the GSM8K benchmark (score improves from 67.6 to **71.2**) compared to using explicit data alone.\n\n\n\n## Qualitative Analysis of Synthesized Data\n\nThank you for your valuable suggestion. In the revised manuscript, we will add a new section to the appendix. This section will showcase concrete question-and-answer samples generated from different types of concept combinations (one-hop, two-hop, and three-hop), along with a brief analysis of them.\n\n## Empirical Support for Scalability\n\nThank you for your advice. We conducted an empirical analysis to precisely measure the scalability benefits and data contribution of using non-co-occurring (multi-hop) concept pairs. The results directly support our observation.\n\nThe analysis confirms that these multi-hop, non-co-occurring pairs are the primary driver of both scalability and final data volume. The number of available concept pairs expands dramatically when moving from direct co-occurrence to implicit, multi-hop relationships. While direct 1-hop pairs are highly effective, their limited number means they contribute only **16.64%** of the final dataset. In contrast, the non-co-occurring **2-hop and 3-hop pairs collectively generate over 71%** (49.85% + 21.98%)of the total effective data. \n\n|            | Combination Count | Effective Data Generated | Effectiveness Rate(%) | % of Total Data |\n| ---------- | ----------------- | ------------------------ | --------------------- | --------------- |\n| one-hop   | 450663            | 353123                   | 78.4                 | 16.6           |\n| two-hop   | 1771511           | 1058133                  | 59.7                 | 49.9           |\n| three-hop | 315744            | 466719                   | 47.6                 | 22.0           |\n| Community  | 285744            | 244485                   | 77.4                 | 11.5           |\n\n## Formalizing the Graph Construction Algorithm\n\nThank you for this important question. We appreciate the opportunity to clarify that the KCRG construction is a **deterministic, algorithm-driven process**, not a heuristic one. The core logic is detailed in Section 3.2. We are happy to outline the precise algorithmic steps below and will add this clarification to the final version of the paper.\n\nThe construction process follows these formal steps:\n\n**Step 1: Node Set (K) Construction**\n\n1. After extracting Key Concepts (KCs) from seed data, we apply a rigorous dual-filtering process to create a unique set of high-quality nodes, K.\n\n**Step 2: Explicit Edge (E_ex) Construction & Weighting**\n\n1. **Data Structuring:** After extracting all KCs, we store their occurrences in a key-value format (e.g., JSONL). Each line maps a KC to a list of problem IDs where it appears.\n\n   ```\n   {\n     \"key_concept1\": [1, 2, 8, 15],\n     \"key_concept2\": [2, 8, 22, 31, 57]\n     ...\n   }\n   ```\n\n2. **Identification:** An explicit (one-hop) edge `(k_i, k_j)` is created between two KC nodes if their problem ID lists have a non-empty intersection.\n\n3. **Weighting:** The weight of an explicit edge `W(k_i, k_j)` is strictly defined as the **count of co-occurring problems** (the size of the intersection of their problem ID lists). For instance, in the example above, \"key_concept1\" and \"key_concept2\" co-occur in `problem_2,problem_8`. If this is their only co-occurrence, the weight of the edge between them is 2.\n\n**Step 3: Implicit Relationship Identification & Weighting** Our method explores implicit relationships by identifying multi-hop paths. Crucially, we use the weights of the constituent explicit edges to filter for high-quality connections.\n\n1. **2-Hop Path Identification:** A 2-hop path is identified between nodes `(k_i, k_m)` if their shortest path distance is exactly two, i.e., `D(k_i, k_m) = 2`. These paths represent concept pairs sharing a common neighbor.\n2. **3-Hop Path Identification (Hub-Constrained):** A 3-hop path is identified between `(k_i, k_m)` if `D(k_i, k_m) = 3`, under the strict condition that the path must involve at least one **Hub Concept** .\n   - A Hub Concept is formally defined as a node whose degree falls within the **top 5%** of the entire graph's degree distribution. \n3. **Weighting for Implicit Relationship **: The weight of an implicit path is the ***minimum* weight** of any explicit edge along it. We filter out low-weight implicit paths to ensure a baseline of semantic coherence.\n\nHere is the pseudo-code\n\n```\nAlgorithm 1: KCRG Construction\n---------------------------------------------------------------------\nInput: Seed dataset D, Hub concept degree threshold k\nOutput: Graph G = (K, E_ex, W_ex, E_im)\n\n// Step 1: Node Set Construction\nK_raw ← {}\nfor problem p in D do\n  K_p ← ExtractKeyConcepts(p)\n  K_raw ← K_raw ∪ K_p\nK ← DualFilter(K_raw)\n\n// Step 2: Explicit Edge Construction & Weighting\nG ← InitializeGraph(Nodes=K)\nCo-occurrence_Map ← BuildProblemIDMap(D, K)\nfor k_i, k_j in pairs(K) where i < j do\n  Intersection_IDs ← Co-occurrence_Map[k_i] ∩ Co-occurrence_Map[k_j]\n  if |Intersection_IDs| > 0 then\n    weight ← |Intersection_IDs|\n    G.AddEdge(k_i, k_j, weight=weight, type='explicit')\n\n// Step 3: Implicit Relationship Identification\nHub_Concepts ← FindNodesWithDegree(G) > k // Top-k percentile\nfor k_i in K do\n  // 2-Hop\n  Neighbors_2_hop ← FindPaths(G, start=k_i, depth=2)\n  G.AddImplicitRelations(k_i, Neighbors_2_hop, type='2-hop')\n  // 3-Hop (Hub-Constrained)\n  Neighbors_3_hop ← FindPaths(G, start=k_i, depth=3, through=Hub_Concepts)\n  G.AddImplicitRelations(k_i, Neighbors_3_hop, type='3-hop')\n\nreturn G\n```\nThis rule-based process is reproducible and provides a solid foundation for our data synthesis. We will also release the code.\n\n## Readability\nWe will make revisions in the final version."}, {"review_id": "3P5oVYQ6Ny", "reviewer": "Reviewer_Juc8", "summary": "GRIP successfully addresses the limitations of existing data synthesis methods in terms of scalability, diversity, and cost-effectiveness through its innovative graph-based approach. This method not only achieves significant results on mathematical reasoning tasks but also demonstrates strong cross-domain generalization capabilities, providing a new paradigm for data synthesis in large language models.", "questions": "1. Line 189 mentions using a rating model to obtain problem difficulty. What specific model is used and how is it implemented?    \n2. When generating data, multiple open-source models are used to jointly score problems and answers, directly providing absolute score values. The article does not explain the method in detail - is it done by directly prompting the open-source models to give scores? If scoring directly, do open-source models truly possess the capability similar to reward models to directly assign definitive scores to problems and answers?    \n3. The article validates diversity by comparing the similarity distribution between generated data and seed data, but does not provide specific calculation methods. For example, GRIP generates through key concepts without directly corresponding seed data, while some other methods have directly corresponding data. Is the similarity calculated for all data, or only for corresponding parts, or taking the maximum value? Supplementing calculation details would make the results more convincing.", "limitations": "The article compares multiple data synthesis methods, but different methods use different source data, with significant variations in the amount of generated data, all of which are lower than the proposed method. On one hand, the ability to synthesize such a large amount of high-quality data is indeed an advantage of the proposed method, which deserves recognition; on the other hand, if more fair comparison results could be provided under the premise of controlling for the same source data and generated data volume, it would undoubtedly further demonstrate the effectiveness of the proposed method.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strength**\n1. This is an interesting work with a very clever idea (I have had similar thoughts before, and I'm glad you implemented and validated its effectiveness).\n2. A relatively large-scale mathematical dataset has been constructed, which is very meaningful for model training and evaluation. I believe this method also has certain scalability.\n3. The overall writing of the paper is relatively clear, although I still have some questions after reading it.\n\n**Weakness**\n1. Do the synthesized problems truly adhere to these knowledge concepts? I think it's necessary to perform knowledge concept extraction operations again to check whether the generated problems follow the provided knowledge concepts, and what is the adherence ratio?\n2. Specific questions are detailed in the \"Questions\" section.\n\n**Typos**\n1. There is a reference error on line 130.", "was_revised": true, "final_justification": "I am satisfied with the author’s response and keep my score unchanged.", "rebuttal": "## Validation of Knowledge Concept Adherence\n\n\nThank you for your highly insightful suggestion to \"reverse-validate\" whether the generated problems adhere to the given knowledge concepts. \n\nWe have adopted your suggestion and conducted this experiment. Specifically, we used the knowledge concept extraction model mentioned in our paper (Qwen2.5 32B) to perform \"reverse\" concept extraction on our synthesized problems. Subsequently, we calculated the match rate (i.e., the \"adherence ratio\" you mentioned) between the newly extracted concepts and the original concepts that were used to generate these problems. We define two levels of adherence:\n\n- **Full Match:** All of the original input concepts were successfully extracted from the generated problem.\n- **Partial Match:** At least one of the original input concepts was successfully extracted from the generated problem.\n\n|               | Adherence Ratio |\n| ------------- | --------------- |\n| Full Match    | 88.65%          |\n| Partial Match | 98.49%          |\n\nThese results demonstrate that our data synthesis process faithfully adheres to the guidance of the given knowledge concepts, ensuring the relevance and accuracy of the generated content. The high \"Partial Match\" rate also indicates that the generated problems remain highly relevant to the intended knowledge domains even when a perfect combination isn't achieved. We will add this crucial validation experiment to the revised version of our paper. Thank you again for your valuable feedback.\n\n\n\n## Specification of the Problem Difficulty Rating Model\n\nThank you for the question. We will provide a more detailed description in the revised manuscript. The model mentioned is an **internal model** that we developed, based on the BERT architecture. It was fine-tuned on a proprietary internal dataset. On this dataset, problems were assigned difficulty labels (easy, medium, or hard) through a combination of our own **in-house human annotation** and generation assisted by ChatGPT. Fine-tuning on our unique internal data enables the model to assign difficulty labels to new problems.\n\n## Methodology and Reliability of Multi-Model Quality Scoring\n\nThank you for your question, as it touches upon a core mechanism of our methodology. We designed a **Structured Evaluative Prompt**. By clearly defining the dimensions for evaluation and providing a few human-annotated samples as few-shot examples, we guide the models to perform a multi-dimensional, quantifiable assessment, rather than simply having them assign an absolute score directly.\n\nAs described in Section 3.4 of the paper, for the evaluation of **problems**, we focus on two core criteria: **Logical Completeness** and **Presentational Completeness**. These are designed to assess whether a problem contains mathematical errors or logical contradictions, whether it is closely related to the given knowledge concepts, and whether its phrasing is complete and clear. We then guide the model to produce a final score by enforcing a structured output.\n\nFor the evaluation of **solutions**, we employ a similar but stricter strategy. We also guide the models to assess the solution's quality via prompts. However, for higher accuracy, we first have the model generate a detailed reasoning process and then provide a binary \"correct\" or \"incorrect\" judgment based on that reasoning for the purpose of **rejection sampling**. Furthermore, GRIP implements a **\"unanimous consent\" mechanism**, where a solution is only accepted if all evaluation models unanimously judge it as correct. This greatly enhances data accuracy.\n\nRegarding your second, and more central, concern—**\"Do open-source models truly possess this capability?\"**—we provide an affirmative answer through a key ablation study.\n\nIn the ablation study in Section 4.5, we conducted an experiment specifically to validate the reliability of our multi-model evaluation framework. We randomly sampled 500 synthetic math problems and had human experts manually label them for quality (\"qualified\" or \"unqualified\"). Using this human-labeled set as the ground truth, we tested the accuracy of our three-model evaluation framework (and GPT-4.1 for comparison). As shown in **Table 5**, the collaborative evaluation accuracy of our three open-source models reached **95.7%**, a performance that even surpassed the accuracy of using GPT-4.1 alone for evaluation **(94.3%)**.\n\nThis result strongly demonstrates that through our designed framework—which is multi-model, multi-dimensional, and uses structured outputs—the open-source models, when used in concert, can indeed make highly reliable judgments on the quality of problems and answers. Their effectiveness is comparable to, and can even exceed, that of top-tier closed-source models. Additionally, regarding your point on reward models, some studies have already shown that generative reward models [1,2] can also achieve strong performance.\n\n[1] Mahan D, Van Phung D, Rafailov R, et al. Generative reward models[J]. arXiv preprint arXiv:2410.12832, 2024.\n\n[2] Li H, Dong Q, Chen J, et al. Llms-as-judges: a comprehensive survey on llm-based evaluation methods[J]. arXiv preprint arXiv:2412.05579, 2024.\n\n## Clarification on the Diversity Metric Calculation Method\n\nThank you for your question regarding the calculation details for our diversity metrics. To ensure an absolutely fair comparison, we applied **an identical and uniform evaluation protocol** to all methods. For each method, we used its own respective seed data and synthesized data for the analysis.\n\n**Similarity Score Calculation (for Figure 4):**\n\nIn Table 1, we list the source and scale of the seed data for all data synthesis methods. To calculate the similarity, we used the **BGE m3-embedding model** for each respective seed dataset.\n\nFor each individual problem in a synthetic dataset, we calculate its embedding similarity against **all** problems in its corresponding seed dataset. We then take the **maximum value** from these comparisons as the final similarity score for that synthesized problem. The distributions of these maximum scores are shown in Figure 4.\n\nThe calculation for the \"Novelty Rate\" metric is defined as: \"the percentage of problems whose combination of knowledge points did not appear in the seed dataset.\"\n\nWe will update the next version of our paper to include these specific calculation details to make our results more convincing.\n\n## Fair comparison\n\nWe have reorganized the experiments in Table 2 to improve readability and fairness.\n|        | Datasets             | Seed Dataset Count | Dataset Count | Avg      |\n| :----- | :----------------- | :----------------- | :------------ | :------- |\n| ～100K | MetaMath           | 15K                | 395K          | 40.3     |\n|        | WizardMath         | 15K                | 96K           | 38.0     |\n|        | MAmmoTH            | 220K               | 262K          | 39.0     |\n|        | MathCoder          | 15k                | 80K           | 38.8     |\n|        | **GRIP-MATH-mini** | **7.5k**           | **80k**       | **43.0** |\n| >1M    | MathScale          | 20K                | 2M            | 51.2     |\n|        | MAmmoTH2           | >10M               | 10M           | 52.2     |\n|        | **GRIP-MATH**      | **7.5k**           | **2.1M**      | **57.3** |\n\n- **Performance at a Smaller Scale (~100k)**\n\nTo directly address your point about comparing datasets of the \"same number,\" we randomly sampled 80k Q-A pairs from our synthesized data to create **GRIP-math-mini**. This allows for an apples-to-apples comparison with datasets of a similar size, like MathCoder (80k). The results show that even when controlling for data quantity, our synthesis method's superior quality leads to better performance, outperforming existing methods in this category. For instance, **GRIP-math-mini (43.0%)** significantly surpasses **MathCoder (38.8%)**.\n\n- **Performance at a Larger Scale (>1M)**\n\nThis comparison highlights the second key metric: scalability. When comparing our full 2.1M dataset against other large-scale methods, GRIP-math demonstrates superior efficiency and quality. For example, compared to MAmmoTH2, which filters 10M samples from the massive Common Crawl, our method achieves a 5.1% higher score (57.3% vs 52.2%) using only 7.5k seed examples. Similarly, our method outperforms MathScale (which uses GPT-3.5) by 6.1%. This proves that our synthesis method not only scales data volume effectively but also ensures superior data quality.\n\nWe will update the manuscript to include this detailed table and analysis. This will make the evaluation more transparent, comprehensive, and directly address your valid concerns about fairness. Thank you again for this valuable suggestion.\n## Paper Formatting Concerns:\nThank you for your careful review and for pointing out the formatting error on Line 130. This was indeed a LaTeX label rendering issue where \"Appendix ??\" was displayed incorrectly. We have now corrected this cross-reference, and it correctly points to Appendix D in the revised manuscript. We appreciate your valuable feedback."}]

Author rebuttal:

Meta-review: Summary: 

This paper focuses on data synthesis methods to advance the reasoning capabilities of LLMs. It presents GRIP that synthesizes high-quality and diverse reasoning instructions. GRIP constructs a knowledge graph by extracting high-level concepts from seed data, and leverages both explicit and implicit relationships within the graph to drive large-scale and diverse instruction data synthesis, and employs open-source multi-model supervision to ensure data quality. Compared with existing synthetic data methods, GRIP is shown to have achieved greater scalability and diversity and also significantly reduces costs.  On a variety of mathematical reasoning benchmarks, models trained with the synthesized GRIP-MATH demonstrate substantial improvements over their base models and outperform previous data synthesis methods.

Strengths:

1. The paper is well-written with a clear structure and motivation, making it easy to understand. 

2. The proposed method is novel and generates diverse problems without the direct use of seed questions or with a less severe risk of data contamination. 

3. The paper conducts extensive experiments on math reasoning to show the effectiveness of the approach and the benefit of the curated dataset. 

Weaknesses:

Before the rebuttal, the reviewers raised many clarification questions, most of which have been addressed. I  put a lower weight on the review from Fk5a, due to its lower confidence. In addition, the authors conducted more experiments and analyses during the rebuttal to address concerns related to evaluation on more challenging benchmarks, decontamination, and cost analysis. All reviewers (except Fk5a) are satisfied with the author responses and lean towards accepting the paper (borderline accept - accept). 

Additional comments: 

1. The authors should make changes in the revised version to reflect the discussions in the rebuttal period. Please add the clarifications, discussions on limitations and future work, as well as additional results/analysis in the revision. 

2. Based on the new results on the more challenging math benchmarks (e.g., AIME, provided in the response to reviewer 3rLC), it seems that the method is getting less effective on those benchmarks when using stronger base models such as Qwen2. Could you please comment on that in the revised version?

3. Please kindly release the constructed datasets and open-source the data synthesis approach to benefit the community, if not yet. 

Overall, I think this paper makes a valuable contribution and would recommend an acceptance.
Decision: Accept (poster)