DiffLM: Controllable Synthetic Data Generation via Diffusion Language Models

## Abstract

Abstract Recent advancements in large language models (LLMs) have significantly enhanced their knowledge and generative capabilities, leading to a surge of interest in leveraging LLMs for high-quality data synthesis.
However, synthetic data generation via prompting LLMs remains challenging due to LLMs’ limited understanding of target data distributions and the complexity of prompt engineering, especially for structured formatted data.
To address these issues, we introduce DiffLM, a controllable data synthesis framework based on variational autoencoder (VAE), which further (1) leverages diffusion models to reserve more information of original distribution and format structure in the learned latent distribution and (2) decouples the learning of target distribution knowledge from the LLM’s generative objectives via a plug-and-play latent feature injection module.
As we observed significant discrepancies between the VAE’s latent representations and the real data distribution, the latent diffusion module is introduced into our framework to learn a fully expressive latent distribution.
Evaluations on seven real-world datasets with structured formatted data (i.e., Tabular, Code and Tool data) demonstrate that DiffLM generates high-quality data, with performance on downstream tasks surpassing that of real data by 2%–7% in certain cases.
The data and code will be publicly available upon completion of internal review.

## 1 Introduction

Data Synthesis has become an indispensable technique in current machine learning research, enabling rapid generation and modification of datasets (Bauer et al., ), allowing researchers to experiment with various scenarios and model architectures without the extensive processes associated with real-world data collection.
Meanwhile, with the rapid advancements in large language models (LLMs), recent research in natural language processing (NLP) has increasingly focused on leveraging LLMs for synthetic data generation.
Early efforts attempted to fine-tune LLMs to align with real data distributions (Keskar et al., ; Anaby-Tavor et al., ; Borisov et al., ).
As the in-context learning capabilities of LLMs have improved, some studies have explored zero-shot or few-shot prompting of LLMs to generate synthetic data (Ye et al., ; Wei et al., ).

Despite the progress achieved, generating high-quality synthetic textual data using LLMs remains challenging, particularly for structured data (Josifoski et al., ; Li et al., ).
First, LLMs often lack a global understanding of the target data distribution when generating synthetic data. Even after fine-tuning, it is difficult to inject information about complex and varied distributions into current LLM architectures, often resulting in outputs with low diversity and instances of data copying (Wu et al., ; Yu et al., ).
Moreover, existing LLM-based synthetic data generation methods typically involve complex pipelines and post-processing mechanisms, such as prompt engineering, multi-agent frameworks, and iterative sampling (Dekoninck et al., ; Wu et al., ). These complexities hinder the rapid adaptation of LLMs to new tasks, limiting their utility in dynamic research and industrial scenario.
Concurrently, the remarkable performance of variational autoencoders (VAEs) and diffusion models in image synthesis tasks (Betker et al., ; Rombach et al., ) has spurred interest in adapting these techniques to other modalities (Borisov et al., ; Li et al., ; Gong et al., ).
Although some works have introduced latent spaces into language models for simple tasks like style transfer or topic generation (Yang & Klein, ; Li et al., ), our preliminary experiments indicate that directly applying the latent distributions learned by VAEs often results in outputs that are unrelated to the real data. This challenges the direct application of these methods in more complex scenarios for synthetic data generation.

To address these challenges, we propose DiffLM, a novel framework that leverages a plug-and-play latent space to provide data distribution information for LLMs during data generation.
First, to decouple the learning of real data distributions from the LLM’s training objectives, we develop a latent space using a VAE to capture external information, mapping samples from the real dataset to latent vectors.
However, we observed that simply sampling from a Gaussian distribution obtained from naive VAE that cannot generate realistic results. To overcome the poor quality of data generated by sampling from VAE, we employ a latent diffusion method that linearly adds noise to the latent space over time. A denoising network is then trained to learn these noises in the reverse process, reducing efficiency loss in data synthesis due to sampling failures.
Finally, we design a soft prompting method to inject latent features into the LLM decoding process, resulting in controllable, high-quality synthetic data.
We evaluate our method on seven real-world structured formatted datasets, ranging from simple table synthesis to more complex code and tool synthesis tasks. Experiments demonstrate that DiffLM can generate realistic results, and ablation studies confirm the effectiveness of each component in our proposed method.
The contributions of this paper are threefold:

- •
Decoupling Data Distribution Learning: We proposed a new VAE-based LLM framework for data systhesis, which decouples the learning of real data distribution information from the training objectives of the LLM by introducing the a small projection network.
- •
High-Quality Synthetic Data Generation: Based on our observations, the meticulously designed VAE and diffusion structures effectively model the distribution of real data, enabling the generation of high-quality synthetic data. In all tasks, the quality of the generated data is comparable to or even surpasses that of the real data.
- •
Comprehensive Evaluation: We validate the high quality of data generated by DiffLM across three distinct scenarios and seven datasets, underscoring its robustness and adaptability in advancing synthetic data generation for natural language processing.

## 2 Related Works

#### Large Language Models in Data Synthesis.

The recent advancement in the generative capabilities of LLMs has motivated numerous exploratory works aiming to leverage these models for data augmentation in areas such as text classification (Ye et al., ; Li et al., ), information extraction (Tang et al., ; Josifoski et al., ), and tabular data generation (Borisov et al., ; Xu et al., ).
A comprehensive survey conducted by Long et al. () proposes a prompt-based generic workflow for synthetic data generation, curation, and evaluation.
And multiple advanced works have attempted to fine-tune language models for data synthesis in recent years (Anaby-Tavor et al., ; Kumar et al., ; Dinh et al., ; Borisov et al., ; Xu et al., ). Specifically, these methods involve fine-tuning LLMs on a small amount of gold data for language modeling, followed by the use of various sampling methods to generate data.
However, a major challenge remains in ensuring that synthetic data accurately reflects real-world distributions. Veselovsky et al. () has shown that LLM-generated data can sometimes diverge from actual data distributions, leading to unfaithful representations that may hinder model training.
Some studies have explored data selection (Puri et al., ) or data augmentation (Ye et al., ) to address this distribution gap, but there remains significant room for improvement.

#### Latent Variable Models in Text Generation.

Latent variable models have made significant advances in computer vision in recent years (Yu et al., ; Gu et al., ; Luo et al., ; Gulrajani et al., ), achieving high-quality generation results, flexibility and effectiveness, as well as robustness to noise perturbations.
In particular, latent diffusion models, such as DALL-E (Betker et al., ) and Stable Diffusion (Rombach et al., ), operate their diffusion processes in a latent space rather than directly in data space, enabling a near-optimal balance between generation quality and computational efficiency.
In text generation, several works (Wiseman et al., ; Ding & Gimpel, ; Li et al., ; Gu et al., ; Borisov et al., ) have attempted to combine latent spaces with language models to accomplish tasks such as sentence representation, text style transfer, and dataset augmentation.
Additionally, some studies have explored the use of diffusion for plug-and-play controllable generation (Li et al., ; Gong et al., ), aiming to steer the outputs of pre-trained language model using auxiliary modules.
While these works share a similar perspective with ours, we tackle a more challenging scenario of structured data synthesis and thoroughly investigate multiple methods of latent knowledge injection.
To the best of our knowledge, our work is the first to combine VAEs and diffusion models with LLMs for high-quality data synthesis.

Figure: Figure 1: Overview of our DiffLM. The trainable lanaguage model (LM) works as VAE encoder while the fixed LLM decoder serves as VAE decoder. We further (1) introduced a Diffusion module to learn the latent space, and (2) employ a latent feature injector with soft prompting to align latent vector space with LLM decoder.
Refer to caption: x1.png

## 3 Methodology

Figure 1 illustrates the main pipeline of our proposed DiffLM.
First, we define an encoder to map discrete data into a continuous latent space (Section 3.2).
Second, although text features are extracted and compressed, vanilla latent embeddings in VAEs often lead to decoding failures due to underutilized or empty regions in the latent space. To address this issue, we adopt a latent diffusion model (Section 3.3).
Finally, to incorporate the prior knowledge into the decoding stage of LLMs, we propose a novel soft prompt injection method to steer the decoding process (Section 3.4).

### 3.1 Problem Formulation

We begin by defining $\mathcal{D}$ as a known small set of real-world distribution data, where each element $x$ represents a real sample.
We define $G$ as the synthetic data generator, which learns the distribution of $\mathcal{D}$ and generates a set of synthetic samples, $\mathcal{D}_{syn}$, ensuring that the model does not simply memorize and reproduce the same real samples, meaning $\mathcal{D}\cap\mathcal{D}_{syn}=\varnothing$.
It should be noted that we focus on the task of unconditional data synthesising using LLMs, where $G$ generates synthetic samples independently of any additional context, i.e., without using explicit prompt text.

### 3.2 VAE-based Representation Learning

Feature Encoding: In standard VAEs, an encoder is typically employed to map input data into a latent space.
Given structured text data $s_{i}$, we utilize a learnable Transformer-based pre-trained language model (Vaswani et al., ; Devlin et al., ; Raffel et al., ) to obtain the representation vector $x_{i}\in\mathbb{R}^{d\times 2}$, which can be split into the mean and variance.
Using the re-parameterization trick (Kingma & Welling, ), we then obtain the latent feature $z\in\mathbb{R}^{d}$:

$$ $z=\mu+\sigma\odot\epsilon,$ (1) $$

where $\mu$ and $\sigma$ are the mean and standard deviation output by the encoder, and $\epsilon$ is sampled from a standard normal distribution $\mathcal{N}(0,I)$.

LLM Decoding: After generating the latent feature $z$, we employ a frozen-parameter LLM to reconstruct the input text $s$ in a causal language modeling manner.
The rationale for freezing the LLM parameters is to avoid retraining and to preserve its general knowledge and capabilities.
Consequently, aligning the two different modalities, whereas the latent space and the LLM input space, presents a significant challenge.
To address this, we propose a novel latent feature injector using soft prompting and design a corresponding injector network; specific details are provided in Section 3.4.

VAE Training Objective: The VAE model is typically trained using the Evidence Lower Bound (ELBO) loss function.
Following previous work (Burgess et al., ), we adopt the $\beta$-VAE training strategy, which introduces a weighting parameter $\beta$ to control the contribution of the KL divergence loss in the total loss function.
Specifically, when $\beta=0$, the model reduces to a standard autoencoder. For $\beta>0$, the KL constraint encourages learning a smoother latent space:

$$ $\displaystyle\text{ELBO}_{\beta}$ $\displaystyle=L_{rec}-\beta L_{kl},$ (2) $\displaystyle L_{rec}$ $\displaystyle=\mathbb{E}_{q_{\phi}(z|x)}\big{(}\log p_{\theta}(x|z)\big{)},$ (3) $\displaystyle L_{kl}$ $\displaystyle=D_{\text{KL}}\big{(}q_{\phi}(z|x)~{}\|~{}p(z)\big{)},$ (4) $$

where $p_{\theta}(x|z)$ is the language modeling reconstruction likelihood, $q_{\phi}(z|x)$ is the approximate posterior, and $p(z)$ is the prior over the latent space, i.e., Gaussian distribution.
In our model design, considering the denoising network of latent diffusion, we adopt an decreasing $\beta$ adjustment strategy.
We initially set a larger $\beta$ weight to enforce a strong regularization on the latent space.
As the reconstruction loss convergence slows, we decrease the $\beta$ value to allow the model to focus more on reconstruction accuracy.
Additionally, we employ an early stopping mechanism to prevent overfitting.

### 3.3 Latent Space Denoising

Although VAE can learns latent space representations of data, directly sampling from the prior distribution $p(z)$ often exhibit low quality generated samples.
In our preliminary experiments, we observed that directly utilizing the latent features learned by the VAE frequently produces text that is unrelated to the target data distribution.
This issue arises due to the discrepancy between the encoder’s learned posterior distribution $q_{\phi}(z|x)$ and the prior $p(z)$.
To address this problem, we introduce a diffusion model in the latent space to more accurately model the true distribution of the latent features.
Inspired by Zhang et al. (), we extract the latent vectors $z\in\mathcal{Z}$ from the trained VAE for each data point $x\in\mathcal{D}_{\text{train}}$.
Starting from the initial latent vector $z_{0}$, we progressively add noise over time following a linear schedule to get $z_{t}$.
During the reverse diffusion process, we employ a standard continuous denoising network to recover $z_{0}$ (Song et al., ).
For the training objective, we optimize the diffusion model through denoising score matching (Karras et al., ):

$$ $\displaystyle z_{t}$ $\displaystyle=z_{0}+\sigma(t)\epsilon,\epsilon\in\mathcal{N}(0,I),$ (5) $\displaystyle dz_{t}$ $\displaystyle=-\dot{\sigma}(t)\sigma(t)\nabla_{z_{t}}\log{p(z_{t})}dt+\sqrt{2 \dot{\sigma}(t)\sigma(t)}d\omega_{t},$ (6) $\displaystyle\mathcal{L}_{\text{diff}}$ $\displaystyle=\mathbb{E}_{t\mathrel{\mathchoice{\vbox{\hbox{$\scriptstyle\sim$ }}}{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}{ \vbox{\hbox{$\scriptscriptstyle\sim$}}}}p(t),~{}z_{0}\mathrel{\mathchoice{ \vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{ \hbox{$\scriptscriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}}p(z_ {0}),~{}\epsilon\mathrel{\mathchoice{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{ \hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}{\vbox{ \hbox{$\scriptscriptstyle\sim$}}}}\mathcal{N}(0,I)}\left\|\epsilon_{\theta}(z_ {t},t)-\epsilon\right\|^{2},$ (7) $$

In forward process Eq.5, $z_{t}$ is the latent variable at time $t$,
and $\sigma(t)$ is a time-dependent noise scale function.
As for backward process Eq.6,
$\dot{\sigma}(t)$ stands for the time derivative of $\sigma(t)$,
and $\nabla_{z_{t}}\log{p(z_{t})}$ is the gradient of the log probability density with respect to $z_{t}$, also known as the score function, and $d\omega_{t}$ is an increment of the Wiener process (standard Brownian motion).
For diffusion model training loss Eq.7,
$\epsilon_{\theta}(z_{t},t)$ is the neural network that predicts the noise $\epsilon$ given $z_{t}$ and $t$.
The detailed description for diffusion model could be found in Appendix A.1.

### 3.4 Latent Feature Injection

After constructing a latent space that captures the true data distribution, two challenges remain:
1)
Aligning latent space with LLM’s input space.
How can the decoding LLM process the latent vector $z$ to steer a powerful language model for realistic data generation?
2) Seamless Integration with LLM Knowledge: How can we integrate external information without disrupting the LLM’s internal knowledge?
Motivated by adapter training methods in LLM fine-tuning (Lester et al., ; Li & Liang, ; Houlsby et al., ; Liu et al., ), we consider the soft prompt latent injection approach to incorporate $z$ into LLM decoding without training the model weights.
Specifically, after obtaining the latent representation $z$, we use an upper MLP to map it into $k$ soft prompt token embeddings, denoted as $\mathbf{H_{\text{latent}}}\in\mathbb{R}^{k\times d}$. These soft embeddings serve as a steering vector, which is concatenated before the $<$BOS$>$ token to assist the LLM in decoding. The detailed process is illustrated in Figure 4.
We also conduct ablation experiments in Section 5.1 with the other two injection methods proposed by Li et al. (), which validated that our methods obtain the best reconstruction loss and downstream task performance.

## 4 Experiments

In this section, we evaluate the generation quality of the DiffLM method on multiple public benchmarks across three tasks:
1) Tabular Data Generation: We compare DiffLM with SoTA tabular generation algorithms, demonstrating its strong capability in structured data generation.
2) Code Generation: DiffLM showcases the ability to integrate structured data priors with its internal knowledge. The results on synthetic data are even better than real ones.
3) Tool Generation: DiffLM can quickly adapt to complex function call scenarios, highlighting its flexibility and adaptability.

**Table 1: Performance of downstream tasks using generated tabular data. We evaluate the quality from: performance in machine learning efficiency (MLE) task, and column-wise distribution density estimation ($\bm{\rho}$) task. $\uparrow,\downarrow$ indicate that higher (or lower) metrics correspond to better performance. Boldface indicates DiffLM surpasses the SoTA model based on language models. Red Boldface denotes DiffLM exceeds the MLE performance achieved using real data.**
| Method | Adult | Default | Magic | Shoppers | Beijing |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MLE $\uparrow$ | $\rho$ $\downarrow$ | MLE $\uparrow$ | $\rho$ $\downarrow$ | MLE $\uparrow$ | $\rho$ $\downarrow$ | MLE $\uparrow$ | $\rho$ $\downarrow$ | MLE $\downarrow$ | $\rho$ $\downarrow$ |  |
| Real | 0.927 | - | 0.770 | - | 0.946 | - | 0.926 | - | 0.423 | - |
| SMOTE | 0.899 | 1.60 | 0.741 | 1.48 | 0.934 | 0.91 | 0.911 | 2.68 | 0.593 | 1.85 |
| CTGAN | 0.886 | 16.84 | 0.696 | 16.83 | 0.855 | 9.810 | 0.875 | 21.15 | 0.902 | 21.39 |
| TVAE | 0.878 | 14.22 | 0.724 | 10.17 | 0.887 | 8.250 | 0.871 | 24.51 | 0.770 | 19.16 |
| GOGGLE | 0.778 | 16.97 | 0.584 | 17.02 | 0.654 | 1.900 | 0.658 | 22.33 | 1.090 | 16.93 |
| CoDi | 0.871 | 21.38 | 0.525 | 15.77 | 0.932 | 11.56 | 0.865 | 31.84 | 0.818 | 16.94 |
| TabSyn | 0.915 | 0.58 | 0.764 | 0.85 | 0.938 | 0.88 | 0.920 | 1.43 | 0.582 | 1.12 |
| GReaT | 0.913 | 12.12 | 0.755 | 19.94 | 0.888 | 16.16 | 0.902 | 14.51 | 0.653 | 8.25 |
| DiffLM | 0.894 | 9.16 | 0.793 | 9.33 | 0.910 | 7.04 | 0.912 | 14.43 | 0.717 | 6.05 |

### 4.1 Tabular Data Generation

Benchmarking. We selected five publicly available datasets for evaluation, encompassing both classification and regression tasks: Adult, Beijing, Default, Magic, and Shoppers. The properties of
datasets are presented in Table 5.
To assess the quality of synthetic data, we employed two perspectives: 1) Low-order statistical metrics, where we quantified column-wise density estimation using the Kolmogorov-Smirnov Test for numerical columns and the Total Variation Distance for categorical columns;
2) Downstream task performance, where we measured the predictive accuracy on test data of classifiers or regressors trained on the generated data.

Baselines. We selected a comprehensive set of classic and SoTA tabular data generation models with diverse architectures for comparison.
First, we consider the performance on real data as the upper bound for evaluation.
Secondly, we included the classic method, synthetic minority over-sampling technique (SMOTE) (Chawla et al., ), which generates new synthetic data patterns by performing linear interpolation between minority class samples and their $k$ nearest neighbors.
Additionally, for neural network-based tabular generation algorithms, we considered six baseline models across different architectures:
1) GAN-based models: CTGAN (Xu et al., ); 2) VAE-based models: TVAE (Xu et al., ), GOGGLE (Liu et al., ); 3) Diffusion-based models: Codi (Lee et al., ), TabSyn (Zhang et al., ); 4) LLM-based: GReaT (Borisov et al., ), which attempts to fine-tune a GPT-2 (Radford et al., ) for table synthesis.
It is worth noting that we compare with the current strongest generative models not to merely outperform them in tabular generation but to demonstrate that our flexible DiffLM architecture can achieve comparable performance while offering additional advantages.

Evaluation. Table 1 presents the quality assessment results of the generated data.
For different tabular datasets, we train a XGBoost classifier or a regressor using the synthetic data to predict the label column values, using AUC and RMSE to evaluate the accuracy, respectively.
From the results, DiffLM outperforms the current language-model-based SoTA (GReaT model) on most datasets.
Notably, on the Default dataset, the prediction accuracy using DiffLM’s synthetic data surpasses that obtained by training on real data.
This suggests that DiffLM’s approach of integrating the real data distribution with its own learned knowledge can provide richer information for downstream tasks while preserving the original data structure.
In other words, the synthetic data generated by DiffLM contains additional knowledge compared to real data, which is challenging to achieve with previous methods.
Moreover, our generated results achieve performance comparable to prior methods in column-wise distribution density estimation.
Although the TabSyn method attains superior performance on several datasets, it should be noted that our approach focuses on general, pluggable generation control for large language model, rather than training data synthesis models from scratch for specific domains.
Despite this, in tabular data generation, our method’s performance is on par with these domain-specific methods.

**Table 2: pass@k scores on HumanEval and MBPP. We follow Chen et al. () for estimating pass@k, where $n>k$ solutions are generated per problem with p = 0.95 and a temperature of 0.2 to calculate the success rate with zero-shot learning. Boldface indicates that DiffLM surpasses the performance achieved with real data. Red Boldface indicates that DiffLM surpasses the base model’s performance.**
| Model | Size | HumanEval | MBPP |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| pass@1 | pass@10 | pass@100 | pass@1 | pass@10 | pass@100 |  |  |
| GPT-4 | - | 67.00 | - | - | - | - | - |
| CodeLLaMA | 7B | 33.50 | 59.60 | 85.90 | 41.40^∗ | 66.70^∗ | 82.50^∗ |
| 34B | 48.80 | 76.80 | 93.00 | 55.00^∗ | 76.20^∗ | 86.60^∗ |  |
| Mistral-Base | 7B | 27.79 | 41.22 | 56.37 | 37.31 | 52.02 | 59.65 |
| 12B | 10.12^† | 20.91^† | 28.93^† | 43.38 | 61.44 | 69.09 |  |
| Mistral-Instruct | 7B | 36.09 | 52.95 | 64.18 | 38.45 | 50.77 | 59.17 |
| 12B | 7.08^† | 12.43^† | 16.14^† | 52.20 | 63.61 | 69.02 |  |
| Mistral-Real-Code | 7B | 28.58 | 42.24 | 54.24 | 27.15 | 42.21 | 48.14 |
| 12B | 36.97 | 52.04 | 60.95 | 34.79 | 45.49 | 50.22 |  |
| Mistral-DiffLM-Code | 7B | 35.37 | 47.36 | 54.38 | 32.70 | 41.65 | 47.39 |
| 12B | 42.24 | 56.02 | 61.97 | 44.42 | 52.35 | 55.70 |  |

### 4.2 Code Generation

Benchmarking. In the code generation scenario, to simplify the problem, we focus on Python code and use the Flytech(^1^11[https://huggingface.co/datasets/flytech/python-codes-25k](https://huggingface.co/datasets/flytech/python-codes-25k)) dataset as real data, which contains 24,813 unique real user queries and the corresponding Python code fulfilling those requests. We discard the user queries and use only the code to train DiffLM.
After generating synthetic code data, we continue pre-training the Mistral 7B v0.3 base model (Jiang et al., ) using a smaller learning rate, i.e., 1e-5, in a causal language modeling objective.
We then benchmark the trained model on code generation tasks, selecting two mainstream benchmarks: HumanEval (Chen et al., ) and MBPP (Austin et al., ).
To better understand the performance changes of the base model, we also experiment with base models of different sizes, i.e., Mistral Nemo with 12B parameters.

Baselines. We include baselines from recent code models.
First, we consider the CodeLLaMA (Rozière et al., ) series, which use approximately 600B tokens to continue pre-training the LLaMA-2 (Touvron et al., ) base model, injecting strong code capabilities through multi-task learning.
Additionally, we compare with the Mistral base model (Jiang et al., ) and its instruction-tuned variants, the latter could representing the upper bound of code capabilities for this architecture.

Evaluation. We report the code generation capabilities in Table 2. Specifically, Mistral-Real-Code and Mistral-DiffLM-Code denote models that were further pre-trained on real data and synthetic data generated by DiffLM, respectively.
The 7B models are based on Mistral-0.3-Base, and the 12B models are based on Mistral-Nemo-Base. Both models were trained for 3 epochs on the same amount of data using identical hyperparameters, effectively serving as a controlled experiment where the data source is the only variable.
The results indicate that simply continuing to pre-train the Mistral model with a small amount of code data leads to inconsistent impacts on code generation capabilities.
Specifically, Mistral-Real-Code shows a slight improvement on HumanEval but a significant decline on MBPP.
However, using our synthetic data to continue pre-training the base model yields better results than using real data. For instance, Mistral-DiffLM-Code-7B, achieved a 7 percentage point improvement over the base model, even outperforming the Code Llama 7B model that was trained with more extensive data.
In summary, in the code generation scenario, we focus on the differing impacts of real data and synthetic data, further demonstrating that DiffLM can generate synthetic data that is even more effective than real data in enhancing downstream task performance.

Figure: Figure 2: GPT-4 evaluation scores for tools from the ToolBench dataset and tools generated by DiffLM. The evaluation prompt considers aspects such as clarity, specificity, completeness, consistency, and applicability.
Refer to caption: x2.png

### 4.3 Tool Generation

Evaluation.
To address more complex structured data generation scenarios, we further conduct a tool synthesis task.
Specifically, we select the ToolBench (Qin et al., ) dataset as a benchmark for comparison, which is constructed based on the RapidAPI(^2^22[https://rapidapi.com/hub](https://rapidapi.com/hub)) platform by crawling APIs created by real users and synthesizing related dialogue SFT data using GPT(^3^33[https://chat.openai.com](https://chat.openai.com)).
We use the its toolset to train DiffLM and then sample an equal number of tools for comparison.
We assess the usability of the generated tools from two perspectives:
1) Single-Tool Quality: We use GPT-4 as an annotator to score the real and synthetic data across multiple dimensions on a scale from 0 to 10, where the results are illustrated in Figure 2.
2) Category-Level Preference: We collect all tools within the same category and use GPT-4 to perform preference scoring between real tools and synthetic tools, as presented in Table 3.
The specific evaluation prompts are provided in the appendix B.2.
From the results, DiffLM’s synthetic data achieves higher scores in the single-tool scoring task, indicating that leveraging the internal knowledge and generative capabilities of LLMs allows us to create tool descriptions and input/output parameter definitions of higher textual quality.
Additionally, in the category-level preference evaluation, nearly $1/3$ of the tool types surpass or are on par with real data in terms of diversity and usability. Since DiffLM can sample and generate tools indefinitely to increase coverage, we believe there is room for further improvement in this metric.

## 5 Analysis

Figure: Figure 3: Model loss curves under different latent feature injection methods and different $\beta$ adjustment strategies. The left is the KL-divergence loss trends, and the right is the language modeling reconstruction loss on a logarithmic scale. In the cyclical $\beta$ strategy, $\beta$ increases linearly from 0 to 0.2. The other methods employ a decreasing $\beta$, starting from a maximum value of 0.1 and decreasing to a minimum of 0.001. Our proposed injection and $\beta$ adjustment strategy achieves the lowest reconstruction loss.
Refer to caption: x3.png

Figure: Figure 4: Final data synthesis process. The comparison of different latent feature injection methods is shown in grey dashed box. Memory Injection introduces the latent features as past key-value (KV) memories into each attention layer of the LLM. Embedding Injection directly adds the latent features to the token embeddings.
Refer to caption: x4.png

### 5.1 Ablation Study

The effect of adaptive $\beta$ adjustment.
As described in Section 3.2, we use a decreasing $\beta$ adjustment strategy to train the VAE latent space.
Here, we compare this with another method that uses a cyclical schedule to anneal $\beta$ (Fu et al., ), evaluating both the loss decline curves and downstream task performance to demonstrate the effectiveness of our decreasing strategy.
Firstly, as shown in Figure 3, the KL-divergence loss trends under decreasing $\beta$ exhibit a pattern where the loss first increases, then decreases, and then increases again.
This indicates that during the early stages of VAE training, DiffLM uses a larger $\beta$ to focus on the divergence between the embedding distribution and the standard Gaussian. This helps quickly learn a standard latent space to stabilize the training of the LLM module.
Subsequently, when the reconstruction loss reaches a bottleneck, it gradually reduces the weight of the KL-divergence loss. At this point, the training objective shifts towards obtaining a decoder with stronger generative capabilities. As a result, the KL loss gradually increases and eventually stabilizes at a fairly low value.
From the results, our decreasing $\beta$ method achieves the lowest reconstruction loss.
Additionally, by introducing the latent diffusion process, we address the issue of distribution discrepancy. Therefore, as shown in Table 4, compared to the cyclical method, the decreasing $\beta$ strategy used in this paper results in stronger generative ability.

**Table 4: The results of MLE and $\rho$ under different latent feature injections and $\beta$ adjustments on Adult dataset.**
| Models | MLE $\uparrow$ | $\rho~{}\downarrow$ |
| --- | --- | --- |
| DiffLM-Cycle $\beta$ | 0.872 | 16.79 |
| DiffLM-Embed | - | - |
| DiffLM-Memory | 0.875 | 17.05 |
| DiffLM-Prompt | 0.894 | 9.16 |

The effect of latent feature injection. We also compare our proposed soft prompt latent feature injection method with previously explored methods such as KV memory injection and input embedding injection (Li et al., ); implementation details are illustrated in Figure 4.
Specifically, the loss convergence on the validation dataset for different injection methods are shown in Figure 3. The input embedding method leads to suboptimal training results, where the reconstruction loss ceases to decrease after reaching around 3.6.
This indicates that such a simple injection method struggles to effectively convey complex real distribution information to the LLM decoder.
Meanwhile, the soft prompt method slightly outperforms KV memory in terms of reconstruction loss.
However, as shown in Table 4, on downstream task performance using the Adult dataset, our proposed soft prompt approach achieves higher (2%) classification accuracy and better column density.

Figure: Figure 5: DCR results of the real test data, Codi, TabSyn, GReaT, and DiffLM on the Beijing and Default datasets. DiffLM exhibits a DCR distribution similar to the current SoTA method, TabSyn.
Refer to caption: x5.png

### 5.2 Training Data Plagiarism

Data copying is a significant challenge for overfitted generative models in practical applications.
To verify that the data generated by DiffLM is not merely copied from the training data, we compute the Distance to Closest Record (DCR) metric. Specifically, for each row in the tabular data, we represent the categorical columns using one-hot vectors and perform min-max normalization on the numerical columns.
We then define DCR as the minimum L1-distance between a synthetic data point and each training sample point:

$$ $\text{DCR}(x_{\text{syn}})=\min_{x_{\text{real}}\in\mathcal{D}_{\text{train}}} L_{1}(x_{\text{syn}},x_{\text{real}}).$ (8) $$

The DCR distribution is shown in Figure 5.
We observe that the LLM-based GReaT generates results that differ significantly from the training data, indicating that vanilla fine-tuning struggles to adapt LLMs to real data distributions and generate high-quality results.
DiffLM demonstrates a DCR distribution similar to that of the SoTA method TabSyn on both datasets.
This further indicates that our proposed general-purpose data synthesis framework can achieve performance on par with domain-specific models on specific tasks.

Figure: Figure 6: The t-SNE visualization of the latent space obtained by encoding evaluation data. DiffLM implicitly learns clustering relationships among different types of data.
Refer to caption: x7.png

### 5.3 Visualization

Figure 6 presents 2D t-SNE visualizations of the latent space for multiple datasets, including four categorical tabular datasets, one numerical tabular dataset, and one tool dataset.
We use DiffLM trained on the corresponding datasets to encode their validation sets, obtaining latent features.
It can be observed that data of the same class encoded by DiffLM exhibit clustering characteristics in the latent space, as seen in the Adult and Magic.
Notably, in the numerical dataset Beijing, different target values display a clear transitional distribution: the upper part of the 2D space corresponds to data with larger target values, i.e., 157 to 858, while the lower part corresponds to data with smaller target values, i.e., 1 to 23.
These results demonstrate that DiffLM’s latent space learning strategy can effectively capture the real data distribution.

## 6 Conclusion

In this paper, we introduce DiffLM, a novel framework designed to enhance LLM’s understanding of real-world data distributions in synthetic data generation tasks.
DiffLM leverages a VAE to map real data into a latent space, which is then injected into the decoding process of LLM, enabling end-to-end training through causal language modeling objective.
Additionally, we incorporate a diffusion process to further refine the learning of the latent distribution, mitigating the sampling failures caused by latent space discrepancies.
To flexibly and non-intrusively control the structure and quality of the generated data, DiffLM integrates real data information with LLMs’ internal knowledge by freezing the LLM parameters and using the latent features as plug-in modules.
Experimental results demonstrate that DiffLM produces highly robust and consistent outputs.
In all datasets, the performance of downstream models trained on the generated data is comparable to or even surpasses that of models trained on real data.

## Appendix A Details on Model Design

### A.1 Diffusion Process

In this section, we will introduce the general process of latent diffusion models.
Latent Diffusion Models (LDMs) are a class of diffusion probabilistic models that operate in the latent space of an autoencoder rather than directly on the high-dimensional data space. By performing diffusion in a compressed latent representation, LDMs significantly reduce computational complexity while maintaining high fidelity in data generation.
An LDM consists of two primary components:

- 1.
Autoencoder: Encodes input data $\mathbf{x}_{0}$ into a latent representation $\mathbf{z}_{0}=E(\mathbf{x}_{0})$ and decodes latent variables back to data space $\hat{\mathbf{x}}=D(\mathbf{z})$.
- 2.
Diffusion Model: Defines a diffusion process on the latent variables $\{\mathbf{z}_{t}\}_{t=0}^{T}$.

It should be noted that the variable used here is independent with main text.

#### Forward Process (Diffusion).

The forward diffusion process in latent space progressively adds Gaussian noise to the latent representation over $T$ timesteps. Starting from the initial latent code $\mathbf{z}_{0}=E(\mathbf{x}_{0})$, obtained by encoding the data $\mathbf{x}_{0}$, the forward process is defined as:

$$ $q(\mathbf{z}_{t}\mid\mathbf{z}_{t-1})=\mathcal{N}(\mathbf{z}_{t};\sqrt{1-\beta _{t}}\,\mathbf{z}_{t-1},\beta_{t}\mathbf{I}),$ (9) $$

where $\beta_{t}\in(0,1)$ is a predefined variance schedule that controls the amount of noise added at each step $t$, and $\mathcal{N}$ denotes a Gaussian distribution. By recursively applying this process, we can express $\mathbf{z}_{t}$ directly in terms of $\mathbf{z}_{0}$:

$$ $q(\mathbf{z}_{t}\mid\mathbf{z}_{0})=\mathcal{N}(\mathbf{z}_{t};\sqrt{\bar{ \alpha}_{t}}\,\mathbf{z}_{0},(1-\bar{\alpha}_{t})\mathbf{I}),$ (10) $$

where $\alpha_{t}=1-\beta_{t}$ and $\bar{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{s}$. This formulation allows efficient sampling of $\mathbf{z}_{t}$ at any arbitrary timestep $t$ without iterating through all previous steps.
In this paper, we adopt the Variance Exploding defined perturbation kernels, whereas setting $s_{t}=\sqrt{1-\beta_{t}}$ and $\sigma_{t}=\sqrt{\frac{\beta_{t}}{1-\beta_{t}}}$. Also, we set $s_{t}=1$ to directly add noise to the data rather than weighted mixing, convert Eq.10 to:

$$ $q(\mathbf{z}_{t}\mid\mathbf{z}_{0})=\mathcal{N}(\mathbf{z}_{t};\mathbf{0}, \sigma_{t}^{2}\mathbf{I})$ (11) $$

#### Reverse Process (Denoising).

The reverse diffusion process aims to recover $\mathbf{z}_{0}$ from a noisy latent variable $\mathbf{z}_{t}\sim\mathcal{N}(0,\mathbf{I})$. It is parameterized by a neural network $\bm{\epsilon}_{\theta}$, which predicts the noise component at each timestep:

$$ $p_{\theta}(\mathbf{z}_{t-1}\mid\mathbf{z}_{t})=\mathcal{N}(\mathbf{z}_{t-1}; \mu_{\theta}(\mathbf{z}_{t},t),\Sigma_{\theta}(\mathbf{z}_{t},t)).$ (12) $$

Typically, the model predicts the mean $\mu_{\theta}$ while the covariance $\Sigma_{\theta}$ is fixed or simplified. By leveraging the properties of the forward process, the mean can be parameterized to predict the original noise $\bm{\epsilon}$ added during the forward diffusion:

$$ $\mu_{\theta}(\mathbf{z}_{t},t)=\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{z}_{t} -\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\bm{\epsilon}_{\theta}(\mathbf{z}_ {t},t)\right).$ (13) $$

This formulation enables the model to denoise $\mathbf{z}_{t}$ step by step, ultimately reconstructing $\mathbf{z}_{0}$.

#### Learning Objective.

The training objective for LDMs focuses on minimizing the difference between the true noise $\bm{\epsilon}$ added during the forward process and the noise predicted by the model $\bm{\epsilon}_{\theta}$. The simplified loss function is:

$$ $\mathcal{L}_{\text{latent}}=\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon},t}\left[ \left\|\bm{\epsilon}-\bm{\epsilon}_{\theta}(\mathbf{z}_{t},t)\right\|^{2} \right],$ (14) $$

where $\mathbf{z}_{t}$ is sampled as:

$$ $\mathbf{z}_{t}=\sqrt{\bar{\alpha}_{t}}\,\mathbf{z}_{0}+\sqrt{1-\bar{\alpha}_{t }}\,\bm{\epsilon},\quad\bm{\epsilon}\sim\mathcal{N}(0,\mathbf{I}).$ (15) $$

This objective encourages the model to learn the conditional distribution $p_{\theta}(\mathbf{z}_{t-1}\mid\mathbf{z}_{t})$ by accurately predicting the noise component at each timestep.

#### Noise Scheduling.

The noise schedule $\{\beta_{t}\}_{t=1}^{T}$ plays a critical role in the diffusion process. It dictates how quickly noise is added in the forward process and, consequently, affects the difficulty of the reverse denoising task. Common strategies for setting $\beta_{t}$ include linear, cosine, and quadratic schedules. We use use linear noise schedule, i.e., the perturbation kernel $\sigma(t)=t$. As it is an effective schedule, ensuring that the data is sufficiently diffused by timestep $t$, while still allowing the model to learn meaningful reverse transitions.

## Appendix B Details on Experimental Setup

### B.1 Tabular Data Generation

**Table 5: Details of tabular dataset. For each dataset, #Num stands for the number of numerical columns, and #Cat stands for the number of categorical columns.**
| Datasets | #Num | #Cat | #Train | #Validation | #Test | Downstream Task |
| --- | --- | --- | --- | --- | --- | --- |
| Adult^1 | 6 | 9 | 29,304 | 3,257 | 16,281 | Binary Classification |
| Beijing^2 | 7 | 5 | 35,059 | 4,382 | 4,383 | Binary Classification |
| Default^3 | 14 | 11 | 24,000 | 3,000 | 3,000 | Binary Classification |
| Magic^4 | 10 | 1 | 15,216 | 1,902 | 1,902 | Binary Classification |
| Shoppers^5 | 10 | 8 | 9,864 | 1,233 | 1,233 | Regression |

### B.2 Tool Judgement Prompts

We present the evaluation prompts used for assessing tool generation quality in Figure 7 and Figure 8.

Figure: Figure 7: Evaluation prompt for single-tool quality. Used by GPT-4 with temperature=1.0.

Figure: Figure 8: Evaluation prompt for category-level perference. Used by GPT-4 with temperature=1.0.

### B.3 Instructions for Reproduction

In this section, we present the experimental details of DiffLM, including data preprocessing, training hyperparameter settings, and data post-processing filtering methods.

#### Data Preprocessing.

Real-world NLP datasets often exhibit inherent structures, such as the context, question, and answer in machine reading comprehension tasks, or key-value pairs in tabular generation tasks. In DiffLM, we convert all structured data into JSON format.
For instance, tabular data in a CSV file is transformed into lines of JSON, and tools from ToolBench are abstracted into JSON structures comprising tool_name, tool_description, api_name, and api_description.
For code data, we use the raw code directly without any preprocessing as input for DiffLM training.

#### Hyperparameter Settings.

- •
VAE Encoder: bert-base-uncased
- •
VAE Decoder: mistralai/Mistral-7B-Instruct-v0.3
- •
Soft Prompt Tokens $k$: 64
- •
Soft Prompt Embedding Dimension $d$: 4096
- •
$\beta_{\text{max}}=0.1$
- •
$\beta_{\text{min}}=0.001$
- •
Diffusion Noise Dimension: 4096

#### Generation Filtering.

For inputs in JSON format, we employ column names to filter the generated outputs. A generated result is considered valid only if it contains all required columns. For code generation tasks involving plain text, we do not apply any filtering. We utilize the same filtering criteria across all baseline models.

## Appendix C Synthetic Data Generated by DiffLM

In Table 6, we compare the real test data of the Adult dataset with the generated outputs from GReaT and DiffLM.
As discussed in Section 5.1, DiffLM produces more diverse samples that more closely align with the real data distribution.
Specifically, for columns like workclass and native-country, the outputs generated by the GReaT model are relatively homogeneous.

**Table 6: Comparison of real samples and synthetic data.**
| Methods | age | workclass | education | occupation | race | sex | native-country | income |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Real | 40 | Private | Some-college | Machine-op-inspct | Asian-Pac-Islander | Female | Japan | $>50K$ |
| 38 | Private | HS-grad | Other-service | White | Female | Canada | $<=50K$ |  |
| 59 | Private | HS-grad | Craft-repair | White | Male | England | $>50K$ |  |
| 29 | Self-emp-not-inc | Assoc-voc | Adm-clerical | White | Male | United-States | $<=50K$ |  |
| 26 | Private | Assoc-acdm | Prof-specialty | White | Female | Canada | $<=50K$ |  |
| GReaT | 27 | Private | Bachelors | Prof-specialty | White | Male | United-States | $<=50K$ |
| 22 | Private | HS-grad | Craft-repair | Black | Male | United-States | $<=50K$ |  |
| 41 | Private | HS-grad | Sales | Black | Male | United-States | $<=50K$ |  |
| 35 | Private | HS-grad | Adm-clerical | White | Female | United-States | $<=50K$ |  |
| 54 | Private | Doctorate | Prof-specialty | Asian-Pac-Islander | Male | India | $>50K$ |  |
| DiffLM | 34 | Private | Some-college | Craft-repair | White | Male | Canada | $<=50K$ |
| 53 | Local-gov | Some-college | Other-service | White | Female | Canada | $<=50K$ |  |
| 23 | Private | Bachelors | Adm-clerical | White | Male | England | $<=50K$ |  |
| 24 | ? | Some-college | ? | Asian-Pacific-Islander | Male | Canada | $<=50K$ |  |
| 32 | Local-gov | Bachelors | Adm-clerical | Asian-Pac-Islander | Male | India | $>50K$ |  |

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see comments above
soundness: 3
strengths: - DiffLM introduces a combination of VAEs, diffusion models, and LLMs for synthetic data generation. The incorporation of a latent diffusion model within the VAE framework, coupled with the soft-prompt injection mechanism, distinguishes DiffLM from prior work that primarily focuses on either fine-tuning LLMs or using simpler latent variable models for text generation.
- The paper provides a thorough evaluation across a diverse set of structured datasets and tasks (tabular, code, and tool generation), demonstrating the robustness and adaptability of the proposed framework.
- The observation that synthetic data generated by DiffLM can outperform real data on certain downstream tasks is  compelling and suggests a potential for knowledge enhancement through synthetic data.
- The proposed DiffLM framework addresses a significant challenge in leveraging LLMs for data synthesis, namely, controlling the generated data's structure and distribution. The ability to generate high-quality synthetic data for structured formats has broad implications for various applications, including data augmentation, privacy preservation, and software testing.
summary: This paper introduces DiffLM, a novel framework for controllable synthetic data generation using large language models (LLMs). DiffLM addresses the challenges of LLM-based data synthesis, such as limited understanding of target data distributions and complex prompt engineering, particularly for structured data. The key innovation lies in decoupling the learning of data distributions from the LLM's training objectives. This is achieved by employing a variational autoencoder (VAE) to learn a latent representation of the real data, which is then used to guide the LLM's generation process. To enhance the quality of the learned latent space, a diffusion model is incorporated, mitigating the limitations of traditional VAEs in capturing complex distributions. Furthermore, a soft prompt injection module seamlessly integrates the learned latent information into the LLM decoding process without retraining, preserving the LLM's inherent knowledge and reasoning abilities. The authors evaluate DiffLM on seven real-world structured datasets spanning tabular, code, and tool data. Their experiments demonstrate that DiffLM generates high-quality synthetic data, achieving comparable or even superior performance to existing methods on downstream tasks, and even surpassing the performance of real data in certain cases. The proposed framework offers a flexible and robust approach for controllable data synthesis, paving the way for wider adoption of LLMs in various data generation scenarios.
weaknesses: - This work suggests that the proposed approach can achieve controllable synthetic data generation, but the model didnt use any disentangled latent variables learning or other grounding techniques to achieve such controllability, can the authors clarify what exactly the controllability refers to?
- In experiments section the works shows that the synthetic data can outperform real data for continued pre-training, which is intriguing and interesting, it would be more helpful to provide more analysis and empirical insights on why and how this is made possible.
- How does the interpolation of latent space look like? How and where did the diffusion model improve the issues of regular VAE model as described in the main text?
- While automated metrics like DCR and downstream task performance are useful, they do not fully capture the nuances of data quality, especially for complex structured data. Incorporating human evaluation, particularly for code and tool generation, would provide a more holistic assessment of the generated data's usability and realism.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Is the injector module tuned in Figure 1? Clarity on this would be beneficial.
2. In Table 1, should the column "Beijing MLE" with a down arrow actually be an up arrow?
3. Which specific LLM is utilized in DiffLM presented in Table 1?
4. How many data samples were synthesized for each of the tasks evaluated?
5. What is the data volume used specifically for tool-related tasks?
6. How does the diversity of synthetic data impact downstream results? Can you control the generation diveristy of synthetic data?  An exploration of this aspect would be informative.
soundness: 2
strengths: 1. The paper addresses the significant and relevant application of generative models: generating high-quality synthetic textual and structural data, which is an important field to study.
2. DiffLM is the first framework to combine diffusion models with LLMs for the purpose of high-quality data synthesis.
summary: The paper presents DiffLM, a novel framework for synthetic data generation that combines LLMs with diffusion models. The primary goal is to address the challenges of generating high-quality synthetic data with a specific focus on structured formats such as tabular, code, and tool data. DiffLM introduces a plug-and-play latent feature injection module to decouple the learning of target distribution knowledge from the generative objectives of LLMs. The framework is evaluated on seven real-world datasets, demonstrating that DiffLM can outperform real data in certain downstream tasks.
weaknesses: 1. In Table 1, DiffLM does not consistently outperform TabSyn across various columns. While the authors compare state-of-the-art results based on language models, there are concerns:
- GReaT is based on GPT-2, which is a relatively weak baseline. A more robust baseline, such as GPT-4 or at least Mistral 7B, should be considered.
- Since TabSyn is also a diffusion-based table generation method, the integration of LLMs in DiffLM does not significantly enhance performance compared to TabSyn, suggesting that DiffLM may be a less effective solution for table generation.
2. The paper claims to focus on data generation with only a small set of real-world distribution data. However, for table and code data, the dataset size (~24k) is substantial. Consequently, specialized models like TabSyn have sufficient training data to perform well without LLM assistance, which diminishes the perceived advantage of DiffLM.
3. The study does not explore how the quantity of real & synthetic data affects downstream task performance respectively, which is crucial for understanding the practical impact of synthetic data.

In conclusion, certain details remain unclear in the paper (see questions), and some settings are not fully addressed.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. For different data, do we need to train different VAE encoder? If yes, can we still call it as plug-and-play module?
2. As we know, diffusion models are good at generating diverse samples. So how to guarantee the generated samples are consistent? ( for exmaples, sometimes there will be some mismatches in the generated samples)
3. Besides training a VAE based on LLMs, what do you think is the major contribution of this work?
4. Besides specific data format, is it possible to use this method to do natural text generation? Will the generated text make sence?
soundness: 3
strengths: 1. DiffLM effectively captures real-world data distributions through a combination of VAE and diffusion models, which enables the generation of high-quality synthetic data.
2. By using soft prompt injection, DiffLM allows for easy control over generated data characteristics without retraining the language model, which makes it adaptable across diverse data types and tasks.
3. DiffLM can generate various structured formats data.
summary: The paper introduces a novel framework, named DiffLM, for controllable synthetic data generation using diffusion models and LLMs. It addresses challenges in synthetic data generation by combining a variational autoencoder (VAE) with a latent diffusion module to enhance the understanding of target data distributions. The model was tested across different type of datasets, including tabular, code, and tool data. Results showed that DiffLM could surpass real data performance in some cases.
weaknesses: 1. Some very related works are missing. There are some works that also combine PLMs, VAE and diffusion. The authors should do a concrete survey.
2. The contribution is limited in my opinion. I think the main contribution: the authors align a encoder for LLM and do the experiments on the task, since doing generation in the latent space of VAE by diffusion model is quite common.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Line 153 mentions that $D \cap D_{syn} = \emptyset$, yet it is unclear to me how Figure 5 demonstrates this condition is met for the proposed method but not for LLM fine-tuning.
2. For the fine-tuning baseline, the results may be highly dependent on training parameters (e.g., number of epochs, memorization, sampling strategy). What steps were taken to ensure a fair comparison?


**References**

1. Kaiser and Bengio. "Discrete Autoencoders for Sequence Models."
2. Amani et al. "Symbolic Autoencoding for Self-Supervised Sequence Learning."
3. Higgins et al. "$\beta$-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework."
4. Havrylov and Titov. "Preventing Posterior Collapse with Levenshtein Variational Autoencoder."
5. Bowman et al. "Generating sentences from a continuous space".
soundness: 2
strengths: The paper is structured clearly, addressing an important problem in synthetic data generation.

Experimental results indicate performance improvements, and the visualizations are well-crafted, aiding comprehension of the methods and outcomes.

Ablation studies on injection methods provide useful insights
summary: This paper investigates methods for generating synthetic data samples from a pre-trained large language model (LLM) using diffusion-based methods to approximate the underlying data distribution. The authors explore the limitations of standard fine-tuning approaches in capturing true data distribution and propose an alternative involving a VAE (Variational Autoencoder) setup where a latent representation encodes the distribution and a conditioned LLM decoder generates samples.

The approach can be summarized as follows:

1. Fine-tuning an LLM to generate samples often results in samples diverging from the original data distribution (cited in lines 90-96).
2. To address this, the authors propose using a VAE with a learned latent representation to capture the data distribution, conditioning the LLM decoder on these representations through a soft prompt generated by an MLP.
3. Standard VAE setups can face issues with strong decoders, where the encoder vector may be ignored. To mitigate this, the authors introduce denoising diffusion steps in the training and employ a $\beta$-VAE for additional control over the KL-divergence term via the adjustable $\beta$ parameter.

Experiments span three task types across seven datasets.
weaknesses: The work lacks a comprehensive ablation and stronger baseline comparisons. Although the authors conduct ablations, they focus on minor details like the $\beta$ adjustments and design choice of latent feature injection, rather than the main components of the proposed pipeline. Key design choices lack justification and comparative analysis, for instance the choice of doing denoising diffusion as I describe later. Despite references to the limitations of standard fine-tuning with sampling, the paper does not provide a thorough comparison of these methods against the proposed framework, showing enough evidence on what precise problem of finetuning and sampling is resolved (please check the questions for more on this.)

There is also a lack of evidence for key claims. In Lines 54-55, the paper mentions that standard VAE representations tend to be ignored by the LLM, driving the need for the diffusion-based approach. However, there is insufficient evidence or citation support for this assertion. Similar issues have been addressed in prior works (e.g., [2, 4, 5]).

Similarly in lines 195-196, if **the core issue** is a discrepancy between the encoder’s posterior and prior, citations or empirical evidence are needed to support this claim. Additionally, simpler solutions, such as adopting a more expressive prior distribution (e.g., Mixture of Gaussians), could have been discussed as alternatives to diffusion. I think the choice of diffusion models is not justified in the current write up.

**Some Minor Recommendations and Editorial Notes**

- I recommend citing the original $\beta$-VAE work since you're using this technique (reference [3]).
- **Conditional Generation with LLMs and VAEs**: Since the paper explores variational autoencoding with LLMs, you could consider to reference prior work on conditional generation with hard and soft tokens in LLMs with auto-encoders. Related work such as [1, 2, 4, 5] would provide readers with a broader context and highlight parallels to existing methods.

Meta-review: This paper presents a framework that combines large language models with diffusion models for synthetic data generation, particularly for structured data like tables, code, and tools. The key innovation lies in using a VAE with latent diffusion to capture data distributions and inject them into LLMs via soft prompts, without model retraining. The strengths include competitive performance across diverse structured data types, with synthetic data sometimes outperforming real data on downstream tasks, and a novel approach to maintaining LLM capabilities while controlling generation. However, several critical weaknesses emerge: (1) insufficient justification for choosing diffusion over simpler alternatives like expressive priors, (2) limited ablation studies on key architectural choices, (3) inadequate comparison with strong baselines using modern LLMs, (4) unclear evidence supporting claims about VAE representation being ignored by LLMs, and (5) some key relevant reference regarding latent discrete diffusion is missing, including Latent Diffusion for Language Generation (Lovelace et al. 2022) and PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model (Zhang et al. 2023). The recommendation to reject stems primarily from these methodological gaps that raise questions about the true advantages of the proposed approach over existing methods.

Decision: Reject