DiffLM: Controllable Synthetic Data Generation via Diffusion Language Models

Authors: Ying Zhou, Xinyao Wang, Yulei Niu, Yaojie Shen, Lexin Tang, Fan Chen, Ben He, Le Sun, Longyin Wen
Primary area: generative models
Keywords: synthetic data generation, diffusion models, language model

Abstract:
Recent advancements in large language models (LLMs) have significantly enhanced their knowledge and generative capabilities, leading to a surge of interest in leveraging LLMs for high-quality data synthesis. However, synthetic data generation via prompting LLMs remains challenging due to LLMs' limited understanding of target data distributions and the complexity of prompt engineering, especially for structured formatted data. To address these issues, we introduce DiffLM, a controllable data synthesis framework based on variational autoencoder (VAE), which further (1) leverages diffusion models to reserve more information of original distribution and format structure in the learned latent distribution and (2) decouples the learning of target distribution knowledge from the LLM's generative objectives via a plug-and-play latent feature injection module. As we observed significant discrepancies between the VAE's latent representations and the real data distribution, the latent diffusion module is introduced into our framework to learn a fully expressive latent distribution. Evaluations on seven real-world datasets with structured formatted data (i.e., Tabular, Code and Tool data) demonstrate that DiffLM generates high-quality data, with performance on downstream tasks surpassing that of real data by 2\%–7\% in certain cases. Data and code will be released upon acceptance.

Decision: reject
Reviewer scores: 5, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730719699009

thread_type:
official_review

<thread_object>
creator: reviewer_1ul1

<content>
summary:
This paper investigates methods for generating synthetic data samples from a pre-trained large language model (LLM) using diffusion-based methods to approximate the underlying data distribution. The authors explore the limitations of standard fine-tuning approaches in capturing true data distribution and propose an alternative involving a VAE (Variational Autoencoder) setup where a latent representation encodes the distribution and a conditioned LLM decoder generates samples.

The approach can be summarized as follows:

1. Fine-tuning an LLM to generate samples often results in samples diverging from the original data distribution (cited in lines 90-96).
2. To address this, the authors propose using a VAE with a learned latent representation to capture the data distribution, conditioning the LLM decoder on these representations through a soft prompt generated by an MLP.
3. Standard VAE setups can face issues with strong decoders, where the encoder vector may be ignored. To mitigate this, the authors introduce denoising diffusion steps in the training and employ a $\beta$-VAE for additional control over the KL-divergence term via the adjustable $\beta$ parameter.

Experiments span three task types across seven datasets.

strengths:
The paper is structured clearly, addressing an important problem in synthetic data generation.

Experimental results indicate performance improvements, and the visualizations are well-crafted, aiding comprehension of the methods and outcomes.

Ablation studies on injection methods provide useful insights

weaknesses:
The work lacks a comprehensive ablation and stronger baseline comparisons. Although the authors conduct ablations, they focus on minor details like the $\beta$ adjustments and design choice of latent feature injection, rather than the main components of the proposed pipeline. Key design choices lack justification and comparative analysis, for instance the choice of doing denoising diffusion as I describe later. Despite references to the limitations of standard fine-tuning with sampling, the paper does not provide a thorough comparison of these methods against the proposed framework, showing enough evidence on what precise problem of finetuning and sampling is resolved (please check the questions for more on this.)

There is also a lack of evidence for key claims. In Lines 54-55, the paper mentions that standard VAE representations tend to be ignored by the LLM, driving the need for the diffusion-based approach. However, there is insufficient evidence or citation support for this assertion. Similar issues have been addressed in prior works (e.g., [2, 4, 5]).

Similarly in lines 195-196, if **the core issue** is a discrepancy between the encoder’s posterior and prior, citations or empirical evidence are needed to support this claim. Additionally, simpler solutions, such as adopting a more expressive prior distribution (e.g., Mixture of Gaussians), could have been discussed as alternatives to diffusion. I think the choice of diffusion models is not justified in the current write up.

**Some Minor Recommendations and Editorial Notes**

- I recommend citing the original $\beta$-VAE work since you're using this technique (reference [3]).
- **Conditional Generation with LLMs and VAEs**: Since the paper explores variational autoencoding with LLMs, you could consider to reference prior work on conditional generation with hard and soft tokens in LLMs with auto-encoders. Related work such as [1, 2, 4, 5] would provide readers with a broader context and highlight parallels to existing methods.

questions:
1. Line 153 mentions that $D \cap D_{syn} = \emptyset$, yet it is unclear to me how Figure 5 demonstrates this condition is met for the proposed method but not for LLM fine-tuning.
2. For the fine-tuning baseline, the results may be highly dependent on training parameters (e.g., number of epochs, memorization, sampling strategy). What steps were taken to ensure a fair comparison?


**References**

1. Kaiser and Bengio. "Discrete Autoencoders for Sequence Models."
2. Amani et al. "Symbolic Autoencoding for Self-Supervised Sequence Learning."
3. Higgins et al. "$\beta$-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework."
4. Havrylov and Titov. "Preventing Posterior Collapse with Levenshtein Variational Autoencoder."
5. Bowman et al. "Generating sentences from a continuous space".
</content>
</thread_object>

</thread>
<thread>
creation date:
1730724368792

thread_type:
official_review

<thread_object>
creator: reviewer_rfz8

<content>
summary:
The paper introduces a novel framework, named DiffLM, for controllable synthetic data generation using diffusion models and LLMs. It addresses challenges in synthetic data generation by combining a variational autoencoder (VAE) with a latent diffusion module to enhance the understanding of target data distributions. The model was tested across different type of datasets, including tabular, code, and tool data. Results showed that DiffLM could surpass real data performance in some cases.

strengths:
1. DiffLM effectively captures real-world data distributions through a combination of VAE and diffusion models, which enables the generation of high-quality synthetic data.
2. By using soft prompt injection, DiffLM allows for easy control over generated data characteristics without retraining the language model, which makes it adaptable across diverse data types and tasks.
3. DiffLM can generate various structured formats data.

weaknesses:
1. Some very related works are missing. There are some works that also combine PLMs, VAE and diffusion. The authors should do a concrete survey.
2. The contribution is limited in my opinion. I think the main contribution: the authors align a encoder for LLM and do the experiments on the task, since doing generation in the latent space of VAE by diffusion model is quite common.

questions:
1. For different data, do we need to train different VAE encoder? If yes, can we still call it as plug-and-play module?
2. As we know, diffusion models are good at generating diverse samples. So how to guarantee the generated samples are consistent? ( for exmaples, sometimes there will be some mismatches in the generated samples)
3. Besides training a VAE based on LLMs, what do you think is the major contribution of this work?
4. Besides specific data format, is it possible to use this method to do natural text generation? Will the generated text make sence?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731145943904

thread_type:
official_review

<thread_object>
creator: reviewer_p77k

<content>
summary:
The paper presents DiffLM, a novel framework for synthetic data generation that combines LLMs with diffusion models. The primary goal is to address the challenges of generating high-quality synthetic data with a specific focus on structured formats such as tabular, code, and tool data. DiffLM introduces a plug-and-play latent feature injection module to decouple the learning of target distribution knowledge from the generative objectives of LLMs. The framework is evaluated on seven real-world datasets, demonstrating that DiffLM can outperform real data in certain downstream tasks.

strengths:
1. The paper addresses the significant and relevant application of generative models: generating high-quality synthetic textual and structural data, which is an important field to study.
2. DiffLM is the first framework to combine diffusion models with LLMs for the purpose of high-quality data synthesis.

weaknesses:
1. In Table 1, DiffLM does not consistently outperform TabSyn across various columns. While the authors compare state-of-the-art results based on language models, there are concerns:
- GReaT is based on GPT-2, which is a relatively weak baseline. A more robust baseline, such as GPT-4 or at least Mistral 7B, should be considered.
- Since TabSyn is also a diffusion-based table generation method, the integration of LLMs in DiffLM does not significantly enhance performance compared to TabSyn, suggesting that DiffLM may be a less effective solution for table generation.
2. The paper claims to focus on data generation with only a small set of real-world distribution data. However, for table and code data, the dataset size (~24k) is substantial. Consequently, specialized models like TabSyn have sufficient training data to perform well without LLM assistance, which diminishes the perceived advantage of DiffLM.
3. The study does not explore how the quantity of real & synthetic data affects downstream task performance respectively, which is crucial for understanding the practical impact of synthetic data.

In conclusion, certain details remain unclear in the paper (see questions), and some settings are not fully addressed.

questions:
1. Is the injector module tuned in Figure 1? Clarity on this would be beneficial.
2. In Table 1, should the column "Beijing MLE" with a down arrow actually be an up arrow?
3. Which specific LLM is utilized in DiffLM presented in Table 1?
4. How many data samples were synthesized for each of the tasks evaluated?
5. What is the data volume used specifically for tool-related tasks?
6. How does the diversity of synthetic data impact downstream results? Can you control the generation diveristy of synthetic data?  An exploration of this aspect would be informative.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731186040396

thread_type:
official_review

<thread_object>
creator: reviewer_homx

<content>
summary:
This paper introduces DiffLM, a novel framework for controllable synthetic data generation using large language models (LLMs). DiffLM addresses the challenges of LLM-based data synthesis, such as limited understanding of target data distributions and complex prompt engineering, particularly for structured data. The key innovation lies in decoupling the learning of data distributions from the LLM's training objectives. This is achieved by employing a variational autoencoder (VAE) to learn a latent representation of the real data, which is then used to guide the LLM's generation process. To enhance the quality of the learned latent space, a diffusion model is incorporated, mitigating the limitations of traditional VAEs in capturing complex distributions. Furthermore, a soft prompt injection module seamlessly integrates the learned latent information into the LLM decoding process without retraining, preserving the LLM's inherent knowledge and reasoning abilities. The authors evaluate DiffLM on seven real-world structured datasets spanning tabular, code, and tool data. Their experiments demonstrate that DiffLM generates high-quality synthetic data, achieving comparable or even superior performance to existing methods on downstream tasks, and even surpassing the performance of real data in certain cases. The proposed framework offers a flexible and robust approach for controllable data synthesis, paving the way for wider adoption of LLMs in various data generation scenarios.

strengths:
- DiffLM introduces a combination of VAEs, diffusion models, and LLMs for synthetic data generation. The incorporation of a latent diffusion model within the VAE framework, coupled with the soft-prompt injection mechanism, distinguishes DiffLM from prior work that primarily focuses on either fine-tuning LLMs or using simpler latent variable models for text generation.
- The paper provides a thorough evaluation across a diverse set of structured datasets and tasks (tabular, code, and tool generation), demonstrating the robustness and adaptability of the proposed framework.
- The observation that synthetic data generated by DiffLM can outperform real data on certain downstream tasks is  compelling and suggests a potential for knowledge enhancement through synthetic data.
- The proposed DiffLM framework addresses a significant challenge in leveraging LLMs for data synthesis, namely, controlling the generated data's structure and distribution. The ability to generate high-quality synthetic data for structured formats has broad implications for various applications, including data augmentation, privacy preservation, and software testing.

weaknesses:
- This work suggests that the proposed approach can achieve controllable synthetic data generation, but the model didnt use any disentangled latent variables learning or other grounding techniques to achieve such controllability, can the authors clarify what exactly the controllability refers to?
- In experiments section the works shows that the synthetic data can outperform real data for continued pre-training, which is intriguing and interesting, it would be more helpful to provide more analysis and empirical insights on why and how this is made possible.
- How does the interpolation of latent space look like? How and where did the diffusion model improve the issues of regular VAE model as described in the main text?
- While automated metrics like DCR and downstream task performance are useful, they do not fully capture the nuances of data quality, especially for complex structured data. Incorporating human evaluation, particularly for code and tool generation, would provide a more holistic assessment of the generated data's usability and realism.

questions:
Please see comments above
</content>
</thread_object>

</thread>
<thread>
creation date:
1733211539487

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear Program Chairs, Area Chairs, and Reviewers,

First, we sincerely thank you for your valuable time, constructive feedback, and insightful guidance. We are delighted that the reviewers recognize the novelty and effectiveness of our proposed methods, as well as our exploratory efforts in high-quality text data synthesis. At the same time, we acknowledge the concerns raised, primarily regarding the contributions and scope of our work, and we provide a unified explanation below.

Following prior work[1,2], we define the controllability of text data synthesis as the ability to generate text that satisfies specific requirements (e.g., structure, topics, domains). Existing methods for structured textual data synthesis often face challenges in achieving this controllability. On one hand, LLM prompt-based methods that rely on prompt engineering or few-shot inference cannot ensure the diversity and scalability of synthetic data, even with intricate human-crafted designs[3]. On the other hand, controlling a language model (LM) by fine-tuning it with supervised data (SFT, RLHF) is not only computationally expensive but may also degrade the LM's general capabilities[1,4]. Our proposed method addresses these challenges by sampling in the latent space while leveraging the LLM's instruction-following abilities to preserve data structure.

Regarding contributions, our work makes the following key advancements:

- Motivation and Challenges: We identify the principles and challenges for high-quality structured text data synthesis, emphasizing the need for scalable and diverse synthetic data that integrates relevant knowledge while satisfying specific data requirements (structure, topic, domain). Current prompt-based and fine-tuning methods struggle to meet these objectives cost-effectively.

- Framework: To address these challenges, we propose the DiffLM framework, which decouples the task of learning the synthesis requirements from language modeling. By modeling these requirements in the latent space and injecting them into an unaltered LLM, our framework generates realistic data that combines the LLM’s extensive knowledge with the specific patterns learned from training data. This leads to superior performance on downstream tasks.

- Techniques: We introduce a training procedure that keeps the LLM decoder fixed, training only the VAE encoder and projector. Furthermore, we validate various methods of latent knowledge injection to ensure effectiveness.

Additionally, in response to the reviewers' suggestions, we have: 1) Enhanced our explanations of baseline training parameters. 2) Clarified how the Latent Diffusion model contributes to DiffLM's improvements. 3) Addressed other potentially ambiguous descriptions.

More comprehensive analyses, including those on the quantity of synthetic data, synthetic data outperforming real data, and human evaluation, have been provided in the rebuttal responses and the revised paper.

We also plan to open-source all intermediate data and code following the review process. We sincerely look forward to further discussions and appreciate your thoughtful consideration of our work.

Best regards,

The author(s) of Paper 10707

Reference:

[1] Keskar et al. CTRL: A Conditional Transformer Language Model for Controllable Generation. arXiv 2019.

[2] Li et al. Diffusion-LM Improves Controllable Text Generation. NeurIPS 2022.

[3] Long et al. On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey. ACL 2024.

[4] Borisov et al. Language Models are Realistic Tabular Data Generators. ICLR 2023.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a framework that combines large language models with diffusion models for synthetic data generation, particularly for structured data like tables, code, and tools. The key innovation lies in using a VAE with latent diffusion to capture data distributions and inject them into LLMs via soft prompts, without model retraining. The strengths include competitive performance across diverse structured data types, with synthetic data sometimes outperforming real data on downstream tasks, and a novel approach to maintaining LLM capabilities while controlling generation. However, several critical weaknesses emerge: (1) insufficient justification for choosing diffusion over simpler alternatives like expressive priors, (2) limited ablation studies on key architectural choices, (3) inadequate comparison with strong baselines using modern LLMs, (4) unclear evidence supporting claims about VAE representation being ignored by LLMs, and (5) some key relevant reference regarding latent discrete diffusion is missing, including Latent Diffusion for Language Generation (Lovelace et al. 2022) and PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model (Zhang et al. 2023). The recommendation to reject stems primarily from these methodological gaps that raise questions about the true advantages of the proposed approach over existing methods.