DataGen: Unified Synthetic Dataset Generation via Large Language Models

Authors: Yue Huang, Siyuan Wu, Chujie Gao, Dongping Chen, Qihui Zhang, Yao Wan, Tianyi Zhou, Chaowei Xiao, Jianfeng Gao, Lichao Sun, Xiangliang Zhang
Primary area: foundation or frontier models, including LLMs
Keywords: large language model, evaluation, synthetic data

Abstract:
Large Language Models (LLMs) such as GPT-4 and Llama3 have significantly impacted various fields by enabling high-quality synthetic data generation and reducing dependence on expensive human-generated datasets. 
Despite this, challenges remain in the areas of generalization, controllability, diversity, and truthfulness within the existing generative frameworks. To address these challenges, this paper presents DataGen, a comprehensive LLM-powered framework designed to produce diverse, accurate, and highly controllable datasets. DataGen is adaptable, supporting all types of text datasets and enhancing the generative process through innovative mechanisms. To augment data diversity, DataGen incorporates an attribute-guided generation module and a group checking feature. For accuracy, it employs a code-based mathematical assessment for label verification alongside a retrieval-augmented generation technique for factual validation. The framework also allows for user-specified constraints, enabling customization of the data generation process to suit particular requirements. Extensive experiments demonstrate the superior quality of data generated by DataGen, and each module within DataGen plays a critical role in this enhancement. Additionally, DataGen is applied in two practical scenarios: benchmarking LLMs and data augmentation. The results indicate that DataGen effectively supports dynamic and evolving benchmarking and that data augmentation improves LLM capabilities in various domains, including agent-oriented abilities and reasoning skills.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730301211384

thread_type:
official_review

<thread_object>
creator: reviewer_eq9j

<content>
summary:
This paper proposes a new framework to synthesize high-quality datasets across various types. To ensure dataset quality, the framework integrates an attribute-guided generation module and a group-checking feature to enhance diversity and controllability. It also includes a code-based mathematical assessment and a retrieval-augmented generation module to improve truthfulness and factuality. Experimental results demonstrate the superior quality of the generated datasets in terms of semantics, diversity, and length distribution. By applying the framework to two scenarios, benchmarking LLMs and data augmentation, it validates the effectiveness of this framework.

strengths:
- By integrating different modules, DataGen ensures the quality of generated datasets, considering the diversity, truthfulness, controllability, and so on.
- To assess the quality of generated datasets, the authors design curated experiments and evaluate key factors including length distribution, semantics, diversity, and knowledge richness.
- Experiments on DataGen's effectiveness for data augmentation demonstrate significant benefits across various tasks, particularly in instruction-following scenarios.

weaknesses:
- In Table 1, the authors list a series of current dataset generation frameworks, highlighting that DataGen considers a broader range of factors. However, for downstream applications, especially data augmentation section, none of these methods are compared, which limits the demonstration of their effectiveness.
- As shown in Figure 4(a), while the length distribution of the generated data tends toward a normal distribution, longer-length samples are missing for HellaSwag and MMLU. In Figure 5, although the generated examples align with the original datasets, it is evident that the generated dataset represents only a partial subset of the originals.
- Table 7 shows that, without difficulty enhancements, LLMs perform better on generated benchmarks compared to the originals, which reduces DataGen's effectiveness and practical value.
- The proposed framework is complex, and in the ablation study presented in Table 4, the analysis may be too simplified to fully validate the effectiveness of each module.

questions:
please refer to weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730568388420

thread_type:
official_review

<thread_object>
creator: reviewer_s9ep

<content>
summary:
The paper introduces DateGen that uses LLM to generate synthetic dataset. DateGen overcomes limitations in generalization, controllability, diversity, and truthfulness, DATAGEN supports a variety of dataset formats and includes mechanisms like attribute-guided generation and group-checking to enhance diversity. It also employs mathematical code-based assessment and Retrieval-Augmented Generation (RAG) for accuracy and truthfulness. Experimentation confirms superior data quality, with applications in benchmarking LLMs and data augmentation, leading to improved model performance in domains like reasoning and agent capabilities

strengths:
1.  DataGen introduces novel elements like attribute-guided generation and the RAG-based validation, which distinguish it from existing synthetic dataset generation frameworks.
2. The modular design allows for customization and adaptability across diverse datasets.
3. The experiments with improved reasoning and agent-oriented tasks performance shows potential in this data generation framework.

weaknesses:
1. RAG-based validation is very high in cost (raising cost from 0.038 to 0.19, almost 5x increase). However, it is unclear how it affects the final data generation quality (like the results in Table 7). In other words, it would be nicer to ablate the modules in terms of metrics in Table 7, instead of the current reports in Table 4.
2. I am not convinced that the performance decline on GSM8K in your experiments can be concluded to that many LLMs may be overstated and overfit on the GSM8K dataset, would you please elaborate more on this?
3. Have you tried any experiments with open-sourced LLMs such as LLaMA 405B being the generating LLM? Would it be as beneficial as the GPT-4 or Claude?

questions:
My questions are included in the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730598158120

thread_type:
official_review

<thread_object>
creator: reviewer_1kec

<content>
summary:
This paper introduces DataGen, a comprehensive framework for generating high-quality (diverse, accurate, and controllable) datasets using large language models (LLMs). DataGen accepts diverse dataset and constraints as input, a comprehensive set of generation hints to reduce computational cost, augment diversity with hyperparameter setting / attribute guided generation, and increase evaluation quality using various reasoning techniques (self-refine) / strong code-based verifier / RAG. Evaluation shows DataGen is able to generalize to diverse set of domains and tasks, models.

strengths:
**Novelty and Significance**. The paper presents a novel technique and artifact for the field of synthetic data generation. DataGen is generalizable to other domains and tasks, though with additional overhead. Compared to other related work, DataGen is able to cover a wide range of features in real settings. The artifact is available and runnable. 

**Writing**. The writing is clear and well-organized, with clear visual / tables to summarizes the comparison, methodology, evaluation, and ablation studies. The motivation of the paper is very clear, the problem is well-defined, key contributions are listed and aligned with the structure of the paper. The visual elements in the paper are very helpful to understand the paper.

**Methodology**. The proposed framework is well-motivated, and the framework design is simple and easily generalizable to different domains and setups.

**Evaluation**. The evaluation is very comprehensive. It covers a wide range of tasks, models (open and closed source), and domains.

weaknesses:
**Data formatting.** In section 3.5 (error analysis), the paper mention sometimes the dataset strggles to follow instruction / format the data correctly. Using constrained decoding and similar techniques, this is a very much solved problem, but produce result that the LLM itself may not follow (hence potentially dropping quality of response). I recommend checking out related works in this field (e.g. Guidance[1], AICI[2], LMQL[3], etc.) to improve the data formatting issue. Further more, LLM engiens such as vLLM[4], SGLang[5] and other proprietary engines (Anthropic, Gemini) have provide structured output generation to support constrained decoding at the time of data generation.  

[1] https://github.com/guidance-ai/guidance
[2] https://github.com/microsoft/aici
[3] https://lmql.ai/
[4] https://vllm.ai/
[5] https://github.com/sgl-project/sglang


**Section 3.3 Effectiveness of Modules in DataGen**. 
- Can you explain more on remote-clique score? Why is it a good metric, how exactly is it calculated on the generated dataset (using embeddings, or other representation of the dataset)?
- The delta of remote-clique score of HellaSwag is significantly higher than other datasets. Why is that?

questions:
**Section 3.8 Benchmarking LLM**. The paper mentioned "challenging nature of Claude-3 generated dataset". Do different LLM uses the same / different prompts? 

**Section 3.3 Effectiveness of Modules in DataGen**. 
- Can you explain more on remote-clique score? Why is it a good metric, how exactly is it calculated on the generated dataset (using embeddings, or other representation of the dataset)?
- The delta of remote-clique score of HellaSwag is significantly higher than other datasets. Why is that?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703942785

thread_type:
official_review

<thread_object>
creator: reviewer_wunv

<content>
summary:
The authors present a framework for generating synthetic datasets that focus on generalization, controllability, diversity, and truthfulness by guiding the generation with attributes, checking diversity within a clique, performing code-based verification for reasoning tasks, and performing RAG to verify facts. The authors also show what types of synthetic benchmarks LLMs excel at and fail at.

strengths:
- There's comprehensive work into each of the target attributes (generalization, controllability, diversity, and truthfulness)
- The methodology is highly detailed, including comprehensive ablations, evaluations, and cost details.
- The details about what synthetic generations other LLMs perform well and poorly on are helpful for further work into synthetic benchmarks.

weaknesses:
- There could be more side-by-sides of questions from the original dataset and each generated dataset.

questions:
- What is the performance of each module given different generator models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732137224653

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thanks for all ACs' and Reviewers' efforts

comment:
First, we would like to thank the Area Chair (AC) and all the reviewers for their time and thoughtful feedback. In response to the reviewers' concerns, we have taken the following measures, which are summarized below:

1. **More Comprehensive Ablation Studies (Reviewer WuNv and Reviewer eQ9J)**: We have added ablation results for Claude-3, Llama3-70B, and Llama3.1-405B on code-based mathematical validation, diversity measurement, the RAG module, and the number of iterations in self-reflection. We have also conducted thorough analyses of these results to demonstrate the generalizability of DataGen.

2. **Inclusion of Additional LLMs in Benchmarking (Reviewer S9ep)**: We have included benchmark results for Llama3.1-405B and provided additional results for Llama3-70B. These results highlight the strong performance of DataGen in dynamic benchmarks.

3. **Justification for Metric Selection (Reviewer 1kEc)**: We have provided a detailed explanation of the calculation process for the remote-clique score and clarified why this metric was chosen to measure diversity. Additional experiment results on alternative metrics (APS and INGF) have also been included in our response.

4. **Effectiveness of RAG (Reviewer S9ep)**: To address the effectiveness of the RAG module in our work, we have presented results for GPT-4, Claude-3, Llama3-70B, and Llama3-405B, along with detailed explanations. Additionally, we have proposed feasible and easily implementable alternatives within our toolkit.

5. **Analysis of GSM8K Benchmark Results (Reviewer S9ep)**: We have incorporated the latest references to support our analysis of GSM8K benchmark results.

6. **Generated Dataset Length Distribution (Reviewer eQ9J)**: We conducted additional experiments to demonstrate that the length of generated datasets can be controlled within DataGen.

7. **Clarifications and Explanations (Reviewers WuNv, 1kEc, and eQ9J)**: We have provided clarifications on data formats, the difficulty enhancements module, and the consistency of prompts used in our experiments.

We hope that our responses address the reviewers’ concerns and further strengthen the understanding of our work. Once again, we sincerely appreciate the reviewers' valuable feedback and efforts.
</content>
</thread_object>

</thread>


Metareview:
metareview:
[Summary]  
The paper introduces DATAGEN, a unified framework leveraging large language models (LLMs) to generate diverse, accurate, and controllable textual datasets. The framework incorporates innovative modules such as attribute-guided generation, code-based label verification, and retrieval-augmented generation to address challenges in generalization, diversity, truthfulness, and user-defined constraints.

[Strengths]
  - Compared to related work, DATAGEN demonstrates the ability to cover a wide range of features in real-world scenarios.
  - The framework is well-motivated, with a simple yet generalizable design that can be applied across various domains and setups.

[Weaknesses]
  - The ablation study and module analysis are somewhat simplified, limiting the validation of each module's effectiveness.
  - The paper contains some ambiguities, particularly in areas such as data formatting, diversity metrics, length distribution of generated data, and difficulty enhancements.

[Decision]  
This paper is well-motivated and presents a comprehensive framework that effectively addresses the challenges of dataset generation using LLMs. Based on the reviewers’ recommendations (6: WuNv, 6: 1kEc, 6: S9ep, 6: eQ9J), I recommend accepting this paper.