A Graph-Based Synthetic Data Pipeline for Scaling High-Quality Data

Authors: Jiankang Wang, Jianjun Xu, Xiaorui Wang, Yuxin Wang, Mengting Xing, Shancheng Fang, Zhineng Chen, Hongtao Xie
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, Mathematical Reasoning, Data Synthesis

Abstract:
Synthesizing high-quality data for continual training has been proven to be effective in enhancing the performance of Large Language Models (LLMs). However, previous synthetic approaches struggle to easily scale up data and incur high costs in the pursuit of high quality. In this paper, we propose the Graph-based Synthetic Data Pipeline (GSDP), an economical and scalable framework for high-quality reasoning data synthesis. Inspired by knowledge graphs, we extracted knowledge points from seed data and constructed a knowledge point relationships graph to explore their interconnections. By exploring the implicit relationships among knowledge, our method achieves $\times$255 data expansion. Furthermore, GSDP led by open-source models, achieves synthesis quality comparable to GPT-4-0613 while maintaining $\times$100 lower costs. To tackle the most challenging mathematical reasoning task, we present the GSDP-MATH dataset comprising over 1.91 million pairs of math problems and answers. After fine-tuning on GSDP-MATH, GSDP-7B based on Mistral-7B achieves 37.7\% accuracy on MATH and 78.4\% on GSM8K, demonstrating the effectiveness of our method. The dataset and models trained in this paper will be available.

Decision: reject
Reviewer scores: 5, 3, 6

Reviews and discussion:
<thread>
creation date:
1730694402731

thread_type:
official_review

<thread_object>
creator: reviewer_deux

<content>
summary:
This paper proposes the Graph-based Synthetic Data Pipeline (GSDP), a scalable and cost-effective framework for synthesizing high-quality data by leveraging knowledge graphs to expand data and reveal implicit relationships. GSDP, shown to generate synthesis quality comparable to GPT-4-0613 at a fraction of the cost, achieves strong performance in mathematical reasoning tasks, with the GSDP-7B model reaching 37.7% accuracy on MATH and 78.4% on GSM8K.

strengths:
1. strong experimental performance
2. interesting research problem

weaknesses:
1. The motivation is not clear enough, authors point out the limitations of existing method, including limited scalability, high cost, and similarity to seed data. However, the similarity to seed data remains questionable and lack a quantitative investigation. Moreover, why the graph-based synthetic method can solve these limitations is not very clear in the introduction section.
2. In the KP extraction process, did authors check the quality of the generated KPs and the impact of each filter and clustering operation to the quality of KPs.
3. Would be nice to conduct study to present the diversity of the generated dataset is better than other synthetic methods.
4. The baseline results based on Qwen and Llama-3 models are missing, would be nice to present these results.
5. Would be nice to conduct case study to show how the KP graph works and show the superiority compared to existing methods.

questions:
please see the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714827253

thread_type:
official_review

<thread_object>
creator: reviewer_867h

<content>
summary:
This paper presents a novel approach for generating synthetic data related to mathematics and mathematical reasoning using a graph-based synthetic data pipeline (GSDP). It automatically extracts knowledge points from seed data to create a knowledge point relationship graph. Utilizing the MATH training set of 7,500 problems and answers as seeds, the GSDP-MATH dataset expands to over 1.91 million pairs of math problems and answers. The authors report achieving accuracies of 37.7% on the MATH dataset and 78.4% on GSM8K when tested with several 7B LLM models. However, the dataset and models are not available for verification.

strengths:
The idea of extracting knowledge points from seed data, forming a graph to explore their relationships, and then generating training data from these compressed concepts is intriguing. It aligns well with the principles of autoencoders. I believe this is probably a good angle for presenting your approach.

weaknesses:
While the idea is plausible, it requires validation. Overall, I'm uncertain how this approach compares to the recent work on Critical Plan Step Learning, which appears to demonstrate greater generalization capabilities and performs better on cross-domain tasks. In principle, once you generate 1.91 million training samples, you may lose some generalization power. Additionally, the paper contains several typos.

questions:
In principle, does generating 1.91 million data samples result in a loss of generalization power?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730829663544

thread_type:
official_review

<thread_object>
creator: reviewer_covk

<content>
summary:
Summary:

Training data comes at a premium when training generative language models to solve math (word) problems.  Semi-synthetic instance augmentation is commonly used in such cases.  This submission proposes math problem instance augmentation using "knowledge points".  I wasn't familiar with this term, but, going by the examples given, a "knowledge point" (KP) looks like a glossary entry of a mathematical term or concept like "Pythagorus theorem" or "completing the square".  These KPs can be connected by edges, based on whether/how they are related.  The proposed method samples small compact subgraphs of this KP-graph, and submits these to an LLM again to generate math problems that requires familiarity and expertise over these KPs.

strengths:
Strengths:

* Identifies data scarcity problem in training LLMs to solve math (word) problems (but this is not unknown).
* Proposes selection of interconnected knowledge points as a way to synthesize high-quality, diverse math problems.
* Demonstrates that this form of data synthesis can be used to boost the performance of smaller language models.

weaknesses:
Weaknesses:

* As in an increasing number of LLM-based papers, the "innovation" consists of creating a workflow and taking each task to an LLM with a suitably designed prompt.  E.g., a prompt is proposed for knowledge point (KP) extraction from a (problem, solution) pair.  There is very lightweight processing (based on cooccurrence) for graph formation among the KPs and the selection of small compact subgraphs from the KP-graph.  Then there is a prompt that sends this KP subset into an LLM again, asking it to generate a math problem involving those KPs.  Since LLMs are now basically wish-fulfillment machines, such LLM-based papers are incomparable to earlier NLP papers published at such venues.
* Perhaps as a partial consequence, the style of writing is quite foreign to students of pre-LLM ML and AI communities.  "Chain of thought" is itself a good example of how not to describe a "data structure" to a computer scientist, "knowledge point" perpetuates that style.
* We are not necessarily playing a number game here, but [https://paperswithcode.com/sota/math-word-problem-solving-on-math] lists the best MATH performance as 88.1 using a 72B LLM and 87.92 using GPT4-turbo.  In comparison the submission claims GPT-4-0613 is at 42.5 and gets it to 37.7 using a 7B LLM.  [https://paperswithcode.com/sota/arithmetic-reasoning-on-gsm8k] lists the best GSM8K performance as 96.7 using a 72B LLM and 96.4 using a 7B LLM.  In comparison the submission shows GPT-4-0613 at 92 and gets it to 76.5 using a 8B model.  I have no experience in how to give credit for lower performance using smaller models.

questions:
Comments and suggestions:

The title can be greatly improved. "Data" is too generic. If you are focusing on math word problems, say so in  the title itself.

Fig 2 caption should be much longer and explain the point of the diagrams without depending on distant text.

"Knowledge point" is too non-standard and nebulous.  Please be specific and precise right the first time you use this term and define it in terms of bits and bytes.

Overall, this may be better suited to a NLP or AI conference than an ML conference.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732539720722

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear reviewers,

We have carefully addressed your suggestions and questions, updated the paper, and submitted the revised version. The main changes are highlighted in different colors for your convenience. We kindly request you to review the revision.

As the deadline is approaching, we hope you can let us know if there are any further issues so that we can address them promptly.

Thank you very much!
</content>
</thread_object>

</thread>
<thread>
creation date:
1733107417446

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Key Improvements in the Rebuttal

comment:
Based on the valuable suggestions and questions from the three reviewers (**we sincerely appreciate the support of coVk and dEUx for our work**), we have conducted additional experiments and included them in the revised Appendix and main text. Below is a summary of the key changes:

**(1) Out-of-Domain Reasoning Ability Test**

In **Section 3.6** and **Appendix C**, we tested the out-of-domain reasoning ability of the GSDP model. We selected multiple test sets from various fields (such as ARC-C, MMLU-STEM, GPQA, BBH, TheoremQA, and MBPP) to evaluate the model's reasoning capability in physics, chemistry, biology, and computer science (We call them scientific reasoning). Although the model was trained only on GSDP-MATH, it showed an average improvement of over **5%** (with a maximum improvement of **10.8%**) in scientific reasoning ability tests, indicating that GSDP-MATH not only significantly enhances the model's mathematical reasoning ability but also boosts its out-of-domain reasoning skills. **Table 5 in the revision** and **Table 1 below this comment** show the improvement of our model on scientific reasoning tasks and its comparison with other models (**including the CPL method requested by Reviewer 867H. Additionally, as the deadline is approaching, we hope Reviewer 867H can provide feedback so that we can respond as soon as possible**).

---
*Table 1: Main results on scientific reasoning tasks. The Δ rows highlight the improvements of the GSDP models over their corresponding baseline models.*

| Model                | ARC-C | MMLU-stem | GPQA | BBH  | TheoremQA  | MBPP         | AVG  |
|----------------------|-------|-----------|------|------|------------|--------------|------|
| Mistral-7B           | 74.2  | 50.1      | 24.7 | 55.7 | 19.2       | 47.5         | 45.2 |
| LLaMA3-8B            | 78.6  | 55.6      | 27.2 | _61.1_ | 20.1     | _54.9_       | 49.6 |
| Qwen1.5-7B           | 75.6  | 45.5      | 26.7 | 45.2 | 14.2       | 52.1         | 43.2 |
| WizardMath-7B-V1.1   | 78.3  | 54.4      | 30.8 | 57.5 | 21.1       | 56.4         | 49.8 |
| MAmmoTH-7B-Mistral   | 72.1  | 48.1      | 25.3 | 48.5 | **31.5**   | 46.7         | 45.4 |
| MetaMath-Mistral-7B  | 76.7  | 53.0      | 28.8 | 51.2 | 19.1       | 46.3         | 45.9 |
| MathScale-Mistral    | 77.3  | 54.9      | **35.4** | 56.8 | 20.8    | 54.0         | 49.9 |
| CPL-final   | 56.1  | 54.9      | 34.3 | 60.5 | -   | - | -|
| GSDP-Qwen-7B         | _79.2_ | 56.3   | 29.8 | 50.3   | 21.6      | 52.5         | 48.3 |
| `Δ Qwen1.5-7B`         | `+3.6`    | `+10.8`       | `+3.1`   | `+5.1`  | `+7.4`        | `+0.4`   | `+5.1`   |
| GSDP-7B              | 78.8  | _58.3_  | _32.3_   | 60.3   | _25.6_  | 54.8         | _51.7_       |
| `Δ Mistral-7B`         | `+4.6`    | `+8.2`        | `+7.6`   | `+4.6`  | `+6.4`        | `+7.3`   | `+6.5`   |
| GSDP-8B              | **80.5**  | **60.8**  | 30.8 | **63.7** | 24.2 | **58.4** | **53.1** |
| `Δ LLaMA3-8B`          | `+1.9`    | `+5.2`        | `+3.6`   | `+2.6`  | `+4.1`        | `+3.5`   | `+3.5`   |
---

**(2) Quantitative Analysis of the Dataset**

In **Appendix E**, we conducted a quantitative analysis of the dataset's "seed similarity" and "data diversity." The results in **Figure 5** and **Table 6** of the **revision** show that GSDP-MATH has lower seed similarity and higher data diversity.

**(3) Other Modifications**

In the revision, we added a case study to better illustrate the GSDP workflow and its advantages over other methods. Additionally, we made it clearer in the introduction the importance of KPRG and how the Graph-based method addresses the issues of limited scalability, high cost, and similarity to seed data in existing methods. (**Appendix D.2, D.3, Figure 3 in the revision**)
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a Graph-based Synthetic Data Pipeline (GSDP) for generating synthetic data to improve mathematical reasoning capabilities in language models. The framework extracts knowledge points (KPs) from seed mathematical problems and creates a knowledge graph representing relationships between these concepts. Using the MATH training set (7,500 problems) as seeds, GSDP generated a dataset of 1.91 million problem-answer pairs (GSDP-MATH). When tested with 7B parameter LLM models, the approach achieved 37.7% accuracy on the MATH dataset and 78.4% accuracy on GSM8K. The paper claims to offer a more scalable and cost-effective approach compared to existing methods, producing synthesis quality comparable to GPT-4-0613.

The paper addresses an important challenge in the field - the scarcity of training data for mathematical reasoning tasks. The proposed approach of using interconnected knowledge points for problem synthesis is novel and theoretically well-motivated, drawing parallels to autoencoder principles. The method demonstrates the ability to boost the performance of smaller language models, potentially making mathematical reasoning more accessible with less computational resources. The experimental results show promising performance improvements, particularly considering the model size constraints.

However, the approach heavily relies on LLM-based workflows with prompt engineering, rather than introducing fundamental algorithmic innovations. The quality and impact of the KP extraction process and filtering operations are not thoroughly validated, and there is a lack of quantitative investigation into dataset diversity and comparison with other synthetic methods. The dataset and models are not available for verification, missing baseline results for important models (Qwen and Llama-3), and performance still lags behind state-of-the-art results on benchmark datasets. Additionally, there is insufficient motivation for why a graph-based approach addresses the stated limitations, lack of detailed case studies demonstrating the KP graph's effectiveness, use of non-standard terminology ("knowledge point") without precise definition, and several typos noted in the manuscript.

Based on the reviews and analysis, rejection is recommended for this paper. While it presents an interesting approach, the heavy reliance on LLM-based workflows with minimal algorithmic innovation raises concerns about the fundamental contribution to the field. The lack of rigorous validation of key components (KP extraction, filtering, dataset diversity) and missing baseline comparisons makes it difficult to fully assess the method's effectiveness. Although the performance shows promise for smaller models, it does not advance the state-of-the-art, and the inability to verify results due to unavailable data and models is concerning. The presentation and motivation issues suggest the work would benefit from substantial revision and additional experimental validation before being ready for publication. While the paper shows promise and addresses an important problem, it would benefit from addressing these limitations in a revised version, including more thorough empirical validation and clearer positioning of the technical contribution beyond LLM-based workflows.