Synthetic Text Generation for Training Large Language Models via Gradient Matching
Authors: Dang Nguyen
Zeman Li
Mohammadhossein Bateni
Vahab Mirrokni
Meisam Razaviyayn
Baharan Mirzasoleiman
Venue: ICML 2025 poster (2025)
Keywords: Synthetic data
Large language models
Gradient matching
ADMM
Primary area: deep_learning->large_language_models
TL;DR: We propose a new synthetic data generation approach with a theoretical guarantee using gradient matching.

Abstract: Synthetic data has the potential to improve the performance, training efficiency, and privacy of real training examples. Nevertheless, existing approaches for synthetic text generation are mostly heuristics and cannot generate human-readable text without compromising the privacy of real data, or provide performance guarantees for training Large Language Models (LLMs). In this work, we propose the first theoretically rigorous approach for generating synthetic human-readable text that provides convergence, performance, and privacy guarantees for fine-tuning LLMs on a target task. To do so, we leverage Alternating Direction Method of Multipliers (ADMM) that iteratively optimizes the embeddings of synthetic examples to match the noisy gradient of the target training or validation data, and maps them to a sequence of text tokens with low perplexity. In doing so, the generated synthetic text guarantees convergence of the model to a close neighborhood of the solution obtained by fine-tuning on real data and preserves their privacy. Experiments on various classification tasks confirm the effectiveness of our proposed approach. Our code is available at [https://github.com/BigML-CS-UCLA/GRADMM](https://github.com/BigML-CS-UCLA/GRADMM).

Synthetic Text Generation for Training Large Language Models via Gradient Matching
Dang Nguyen 12
Zeman 2i
Mohammadhossein Bateni
Vahab Mirrokni? Meisam Razaviyayn 2.3
Baharan Mirzasoleiman 12
Abstract
training data (Hartmann et al., 2023), ensuring the privacy of training examples hinders training the model directly on the training data. Thus, generating small subsets of synthetic data that can train an LLM to superior performance on the target task becomes handy. To do so, synthetic text should be generated a way that ensures similar dynamics to that of training on the real data. However, text discrete nature and optimization in the discrete space is very challenging.
Synthetic data has the potential to improve the performance, training efficiency, and privacy of real training examples. Nevertheless, existing ap- proaches for synthetic text generation are mostly heuristics and cannot generate human-readable text without compromising the privacy of real data, or provide performance guarantees for train- ing Large Language Models (LLLSs). In this work, we propose the first theoretically rigorous approach for generating synthetic human-readable text that provides convergence, performance, and privacy guarantees for fine-tuning LLMs target task. To do so, we leverage Alternating Direction Method of Multipliers (ADMM) that iteratively optimizes the embeddings of synthetic examples to match the noisy gradient of the target training or validation data, and maps them to a sequence of text tokens with low perplexity. In doing so, the generated synthetic text guarantees convergence of the model to close neighborhood of the solution obtained by fine-tuning on real data and preserves their privacy. Experiments on vari- ous classification tasks confirm the effectiveness of our proposed approach. Our code is available at http://ghaaa..ooomigili...cc....
Existing approaches for synthetic text generation mostly rely on advanced LLMSs such GPT-4 to generate synthetic text for the target categories (Ye et al., 2022; Meng et al., 2022; Li et al., 2023b; Gupta et al., 2023; Tao et al., 2024; Wu et al., 2024; Dekoninck et al., 2024; Yu et al., 2024). L.LM-generatee text either suffers from lack of diversity and faithfulness to real data (Ye al., 2022: Meng et al., 2022; Li et al., 2023b), requires meticulous prompt engi- neering and highly complex pipelines, such multi-agent frameworks, iterative sampling, and processing mechanisms (Gupta et al., 2023; Dekoninck et al., 2024; Wu et al., 2024). The complexity of the pipelines, efforts for manual prompt engineering, and the cost of querying advanced models limits the applicability of such approaches. few recent studies explored the of VAEs and diffusion for con- trollable text generation (Li al., 2022; Gong al., 2022; Zhou et al., 2024). But, training diffusion models is compu- tationally heavy and difficult in practice. Importantly, none of the above approaches guarantees performance of LLMs trained on synthetic text or preserve privacy of real data.
1. Introduction
The above limitations raise a key question: Can we generate small subset of synthetic text that train LLM with similar dynamics to that of real duta? For vision models, Dataset Distillation (DD) addresses the above question by generating small number of synthetic images that mini- mize the training loss (Wang et al., 2018; Loo et al., 2022; Nguyen et al., 2020), match the training gradient (Zhao al., 2020; Zhao & Bilen, 2021) or model's weight trajectory dur- ing training (Cazenavette al., 2022; Wang al., 2022). For images, gradient-based methods the pixel-wise continuous space. However, for LLMs, the discrete nature of text and the very large number of LLM's parameters make DD much more challenging. The few ex-
High-quality data is crucial for training Large Language Models (LLLMs) to superior performance (Yang al., 2024; Li et al., 2023b). However, collecting and curating high- quality data is often very expensive and hard to obtain in many domains. In addition, as L.MMs can memorize their
'Equal contribution 'Department of Computer Science, Univer- sity of California, Los Angeles GGoole Research 'epartment of Industrial & Systems Engineering, University of Southern Califor- nia. Correspondence to: Dang Nguyen <dangnth @ es. cs.ucla.edu>>
easily operate in
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Synthetic Text Generation for Training Large Language Models via Gradient Matching
isting approaches generate synthetic embeddings that mini- mize the training loss (Sucholutsky & Schonlau, 2021; Li & Li, 2021; Sahni & Patel, 2023; Mackawa et al., 2023), or by training a generator model to match the gradient of an LLM trained on target data (Maekawa et al., 2024). However, the synthetic embeddings are not readable and cannot be trans- ferred to train other LLMs, and synthetic data generated by matching dynamics of a LLM trained on target data may include real training examples and is not privacy preserving.
DD for Images. DD is originally proposed for images. Wang et al. (2018) initially proposed a meta-learning approach which synthesizes data by iteratively training model to convergence on the synthetic examples, and optimizing the synthetic data such that the trained model generalizes well on the real training data. Subsequent stud- ies tried to make this process more efficient by using kernel methods to approximate training the model on synthetic data in a closed form (Loo et al., 2022; Nguyen et al., 2020). More recent works generate synthetic data by matching the gradient (Zhao et al., 2020; Zhao & Bilen, 2021; Kim et al., 2022) or wright trajectory (Cazenavette et al., 2022; Wang et al., 2022) of the model trained on real data, or by matching the data distribution (Zhao & Bilen, 2023).
In this work, we propose the first theoretically-rigonon method to generate readable synthetic text that guarantees similar dynamics to that of fine-tuning on real data. First, we formulate a discrete optimization problem to find text embeddings that have a similar gradient to that of real data, under the constraint that the optimized embeddings should corrssond to tokens in the vocabulary. Moreover, to ensure readability, we add another constraint that requires the sequence to have low perplexity. Then, we solve this discrete optimization problem using Alternating Direction Method of Multipliers (ADMM) that iteratively optimizes the embeddings of synthetic data to match the average gradient of the real data, and maps them to a sequence of text tokens with low perplexity. To guarantee Differential Privacy (DP), we clip real data gradients and add controlled noise to their average before matching it. We prove that the synthetic text generated by our method guarantees convergence to a close neighborhood of the solution obtained by fine-tuning the model on real data.
DD for Text. There have been recent efforts in applying DD to text. For text datasets, existing methods (Sucholutsky & Schonlau, 2021; Li & Li, 2021; Sahni & Patel, 2023) apply the original meta-learning based method of (Wang et al., 2018), or minimize the KL-divergence between the self-attention probabilities of the model and the distilled at- tention labels across all layers and heads, for the first token Maekawa et al. (2023). As generating text in the discrete space is difficult, the synthetic data is generated as continu- ous input word embeddings instead of discrete text. Such embeddings cannot be used for training other models that have different word embedding weights, and are unreadable to humans, making them difficult to interpret and analyze. Sucholutsky & Schonlau (2021); Sahni & Patel (2023) trans- formed their distilled synthetic samples to text by finding words with the nearest neighbor embeddings. However, this results in unrelated words that are not meaningful.
We conduct extensive experiments evaluate the effective- ness of our approach, namely GRADMM, for generating syn- thetic data using Phi model for multiple classification tasks. First, we consider the case where only a small number of validation examples are available and we apply GRADMM to generate a larger fine-tuning data. We show that with only 5 to 50 examples, GR GRADMM can successfully gener- ate 100 synthetic data that outperform training on the real examples by up 31.5%. Next, we apply GRADMM to generate a small synthetic data based on an existing larger fine-tuning data. We show that the synthetic data generated by GRADMM outperforms zero-shot and few-shot genera- tion by LLMS as well as real examples selected by coreset selection methods by up to 13.1%, while ensuring the pri- vacy of the training data. We also confirm the transfcrability of GRADMM's generated text via Phi for fine-tuning other LLMs, including -l2ma-3.2--1B and OPT-I.3B.
To generate readable text, Maekawa et al. (2024) first trains a proxy language model from scratch to generate synthetic training data for different classes. Then, it fine-tunes a generator model to generate synthetic data by minimizing the gradient matching loss between generated and training data. Training the proxy model is bottleneck in scaling the method. Besides, as the distilled synthetic data may include real samples from the original dataset, this method cannot ensure privacy.
Notably, none of the existing DD methods scale beyond BERT (Devlin, 2018) to L.MSs with billions of parameters. In this work, we propose the first DD method that can gen- erate privacy-rreservig human-readable text, by matching gradients of L.MSs with billions of parameters.
2. Related Work
2.2. Synthetic Text Generation using Generative Models
2.1. Dataset Distillation (DD)
LLMs. A large body of recent work used LLMs to generate synthetic text data in the zero-shot or few shot setting (Meng et al., 2022: Li et al., 2023b). In the zero-shot setting, the LLM is directly prompted to generate text for categories of interests. In the few-shot setting, a few
DD aims to generate a small synthetic subset of examples that can achieve a similar generalization performance to that of training on the full real dataset.
2

---

Synthetic Text Generation for Training Large Language Models via Gradient Matching
real-world data instances are provided as examples to guide the LLM in generating the synthetic data. In our work, we use the zero-shot and few-shot approaches as our baselines. LLM-generated text is often very repetitive and lacks diver- sity (Holtzman et al., 2019; Keskar et al., 2019). Besides, it does not capture the distribution of the target task and may contain incorrect or hallucinated examples (Ye et al., 2022; Meng et al., 2022; Gupta et al., 2023; Li et al.. 2023b; Wu et al., 2023). To address these issues, recent methods rely on extensive prompt engineering to inject semantic diversity for each target category (Gupta et al., 2023) and design highly complex pipelines, such as model arithmetic which composes and biases multiple LLMs (Dekoninck et al., 2024), multi-step meticulous prompt engineering to inject domain knowledge, iterative sampling, and self-correction to rectify inaccurately labeled instances (Gupta et al., 2023), and
over the whole dataset D as ((D,0) = - Σ i= ((s 0).
Problem formulation. Given a subsets of real examples from the fine-tuning data Dreal C Dr. our goal is to generate k 1== q' & DD Vi, containing synthetic examples that do not belong to Dr. such that fine-tuning the model on Dsy minimizes the loss on Dreat Formally,
synthetic data Dsyo {q¹)
arg min E(Dreat,0"), s.t. 0 E arg min (Dsyn: 0). Daya Pagul ISr
0
(1)
Readability constraint. Importantly, we want the synthetic data to be human-readable That is we want every synthetic example to be a sequence of words in the vocabulary. Be- sides, to ensure that the sequence is meaningful, we require that the synthetic data has low perplexity. Thus, we wish to solve the following constrained optimization problem:
generation techniques (Wu et al., 2023). Such pipelines require large number of queries to advanced LLMs such as GPT-4 (OpenAI, 2023) and Claude3-Opus (Anthropic, 2023). This incurs a large financial cost and makes such approaches difficult to apply in practice. While synthetic data generated by L.LMs are human-readable, LLMs may memorize and generate their training data (Hartmann et al., 2023). Hence, the synthetic data generated by LLMs do not preserve the privacy of their training data. Besides, it does not provide any theoretical guarantee for the performance of LLMs trained on them.
arg min E(Dreal, 0*), s.t. 0 E arg min (( syn 0),
arg
Dyn. P.yll .ET pe VsEDge
(2)
where r= (s= E V) is the set of all prompts and responses that consist of words in vocabulary V.
Use cases for synthetic data generations. The above for- mulation is applicable to two settings: (1) Data is scarce for the target task, and we want to generate a larger synthetic fine-tuning data based on a small number of examples from the target task. (2) A relatively large supervised fine-tuning data is available, and we wish generate a smaller syn- thetic data to replace the real data to preserve the privacy of training examples or to improve the training efficiency.
VAE and Diffusion. A few recent studies explored the use of VAEs and diffusion for controllable text generation (Li et al., 2022; Gong et al., 2022; Zhou et al., 2024). Such approaches train diffusion model from scratch and parame- terize structural and semantic controls by different classifiers and update the latent variables to satisfy the controls (Li & Li, 2021), or to add noise and denoise embeddings of real data (Zhao et al., 2020). This process is computationally very heavy and difficult in practice. Similar to LLMs, syn- thetic data generated by VAEs and diffusion models do not provide guarantee for the performance of the trained model.
4. Method
Next, we discuss our proposed method for generating read- able synthetic text for fine-tuning LLMs on a target task.
4.1. Text Generation via Gradient Matching
3. Problem Formulation
An effective way to solve Eq 2 is via gradient matching. Specifically, we generate a synthetic data Dsyn that has a similar gradient to that of the real dataset:
Here, we formalize the problem of generating small human- readable synthetic text that can fine-tune an L.M with sim- ilar dynamics to that of real data. We also discuss two common use cases where such synthetic data is useful.
arg min D(Vol(Deynn 0), Vol(Drral 0)).
(3)
Setting. Consider a pretrained L.M with parameters 0 and vocabulary V = for..... , }jvl} containing all the words it has been trained to recognize and use. Consider a super- vised fine-tuning dataset D7 = {s*), }. where each example s¹ (p', r') a pair of prompt p' and response r contain- ing words in the vocabulary. The negative log likelihood loss is defined as {(s' O)
Dans Sn, SET pp((s){ VeEDon
where D(., -) is distance between two gradients. Following (Deng et al., 2021; Geiping et al., 2020), we use 1 - [-coos(.) .) as our distance metric, where cos is the cosine similarity. If such a synthetic data can be generated, training on it with gradient methods directly minimizes the loss on real data.
Hog(r'lp' Ip). The fine-tuning objective is thus to minimize the negative log likelihood loss
3

---

Synthetic Text Generation for Training Large Language Models via Gradient Matching
Fine-tuning is often short and changes the model to smaller extent than pre-training. Fine-tuning for longer results in forgetting the pretrained information and harms the perfor- mance (Gekhman et al., 2024). Since fine-tuning loss is often smooth and has a bounded curvature, we solve the above problem by generating a synthetic data that matches the gradient of real data at the pretrained parameters. We prove that training on such a subset converges a close neighborhood of the solution found by training on real data.
(ADMM) (Glowinski & Marroco, 1975; Gabay & Mercier, 1976). By forming the augmented Lagrangian function, ADMM decomposes the original problem into subproblems that can be solved separately and iteratively. While ADMM was originally introduced for convex optimization under linear constraints, more recently it has been successfully applied to solving mixed integer non-linear programs (Leng et al., 2018; Lin et al., 2019), with convergence guarantees (Huang al., 2021).
Challenges of Readable Text Generation. Solving Prob- lem 3 is very challenging, as the set of feasible solutions is sparse, the space is discrete, and LLMs are non-linear and high-dimensional. Specifically, the constraint set is formed by the Cartesian product of many discrete sets, each restricting a word to belong to the vocabulary. Among sequences that satisfy this condition, only those that are readable-measured by a low perplexity value-are valid. Thus, solving Problem 3 is NP-hard as it requires going through all the possible sequences of words in the vocabu- lary and finding readable sequences that best match the real data gradient. The number of such sequences exponential in the size of the vocabulary. This makes it computationally infeasible to find the optimum solution. Finally, calculating the similarities in the gradient space of LLMs with billions of parameters is computationally very expensive.
Constrained Gradient Matching in the Embedding Space. To apply ADMM to our discretely constrained non-convex problem 4, we convert it to a non-convex optimization with convex linear constraints. To do so, we introduce an aux- iliary variable Z and rewrite our objective with an extra equality constraint so that the embeddings are constrained to be from the vocabulary, but not subject to that restriction:
minx x (Xx + IZ) 8.L. X Z.
(5)
The indicator function (Z) is defined as (Z) 0 if 2j E & Vj (i.e., if the embedding of each synthetic example can be mapped to sequence of words in the vocabulary), and Ie (Z) = +o0 otherwise. The augmented Lagrange of Eq. 5 for parameter > O, can be formulated as:
Lang (X 2,A = f(x) + Io + (A, X - Z)
P X Z 12
4.2. Alternating Between Text and Embedding Spaces
+
(6)
2
To solve the above discretely constrained non-convex op- timization problem, we first transfer it to the continuous embedding space, where one can optimize the embeddings of synthetic data to match the target gradient, under the con- straint that the optimized embeddings belong to the set of all token (words, subwords, or characters) embeddings in the vocabulary. If such embeddings can be found, they can be directly mapped to sequence of words in the vocabulary.
where E [R (Pp denotes the Lagrangian multipliers. With simple algebraic manipulations, Eq.6 can be written as:
- Lagg (X Z, - - ((x) 1 Z - A1
ADMM solves the above problem by minimizing primal variables X, Z and maximizing dual variable A at each iteration t, using the following update rules:
Formally, let x E Raxd be the embedding matrix of syn- thetic sample 8 with n tokens, where row xt E Rdd is the jth token embedding. By stacking the embedding matrices of all synthetic samples in Dayn: we obtain an embedding tensor X E IR Paull xxxd With an abuse of notation, we de- note f(zz 0) = E(s, 0) and 8(X, 0)
Primal update:
x = arg min Lang X,Z','') (8)
x
z argmin Caue (X++1 2, A') (9)
Papl Σ imi ((x", 0).
Parl
2
Dual update: A 1 A' + p(x
+1 -2 1
(10)
We rewrite Problem 3 as:
arg min (XX) s.t. D(V))
0)),
which are respectively the proximal step. projection step, and dual update. The proximal step optimizes the embeddings to match the target gradient, and the projection step maps the embeddings words in the vocabulary. Eq, 8 requires solving unconstrained optimization problem. When p is large, the function is strongly convex in X. In practice, stochastic gradient descent algorithms such as Adam (Kingma, 2014) can obtain an approximate solution, which is sufficient for the convergence of ADMM (Huang et al., 2021). Next, we discuss the projection step.
P PuulSr.
E.E2 pp V
(4)
where & = ((1,, e e denote the vocabulary embed- ding, i.e. the set of all token embeddings in the vocabulary V of model 0 where e; E d and is the embedding dimension.
To solve the above constrained optimization problem we apply the Alternating Direction Method of Multipliers

---

Synthetic Text Generation for Training Large Language Models via Gradient Matching
Algorithm 1 GRADient matching ADMM (GRADMM)
4.3. Dealing with High-dimennioo Gradients
1: Input: Constant p O, ADMM steps T, proj param k;
Calculating similarities in the very high-dimensional gra- dient space of LLMs with billions of parameters is compu- tationally very expensive. Besides, such gradients contain many small and noisy dimensions which makes calculat- ing gradient similarities inaccurate. An effective way to tackle this issue is to leverage lower-dimensions gradient estimates (Mirzasoleiman et al., 2020). Various weight initialization (Glorot & Bengio. 2010) and activation nor- malization methods (loffe, 2015) uniformize the activations across samples. Thus, the variation of the gradient norm is mostly captured by the gradient of the loss with respect to the model's 's last layer (Katharopoulos & Fleuret, 2018).
DP param e
2: Step 1: Initialization
3: Random sample X T
4: Initialize X = arg minx f(X, E,
5: Initialize 2 = X and A E I (Pal (Paa|xxxd
6: Step 2: ADMM
7: for 1. =1=00 0, 111...... ...,. T do
8: Update X: X
arg minx x C(X A',,,,,)
9: Update Z: Z -Pemp (x++1 + p- --)
10: Update A: A+t1= = =A' + +( 1
11: end for
12: S = Pe )X"
13: Step 3: Filtering
Based on the observation, we generate synthetic data by only matching the last-layer gradient of the model. Let 0r. denote the last layer of model 0 with L layers, the last-layer gradient distance between synthetic and real data in Eq 4 is:
14: Drop samples in S that do not belong to their category
15: Select rsamples S with lowest gradient matching loss
16: Drop examples with highest loss from categories that have higher average gradient matching loss
17: Output: Remaining synthetic texts in S.
arg min D(Vol(X,01), Vol(Dres, 0z))
(14)
Deai Sn. ES ppl(a)) va
Projecting the Embeddings into the Vocabulary Space. Equation (9) can be written as (Huang et al., 2021):
Matching the last layer gradient is much cheaper than the (11) full gradient and allows generating synthetic data with supe- rior performance, as we will confirm in our experiments.
z = arg min Cangl ( (Z'1, -.A')
2
arg min Le (Z) + +11 - -p --1A'
(12)
z
= PPe( =Pe(X'+p-1NN -
(13)
4.4. Filtering the Generated Examples
While the top-k projection enables generating human- readable text, it can negatively affect the performance due to the following reasons: (i) It may change the category of the synthetic example by including words that are most relevant to other categories. (ii) It may significantly increases the gradient matching loss of some synthetic examples. (iii) It may result in much higher gradient matching loss for some categories compared to the rest. To address the above issues, we filter the low-quality synthetic examples as follows. First, we drop examples that do not belong to the correct category by running a simple few-shot evaluation (Li et al., 2023b), as detailed in Appendix B. Next, for every category we se- lect r synthetic examples with the lowest gradient matching loss. Finally, we ensure similar gradient matching loss for all categories by dropping examples with highest loss in categories with a higher average loss compared to the rest. The above steps significantly boost the performance of the synthetic data, as we will confirm in our experiments.
For the vocabulary embeddings E, the projection (r.) of an embedding vector x; E Rd into the vocabulary space is the embedding vector 2; == arg min ||2; - ell corre- sponding to the token in the vocabulary that is closest to x; in Euclidean space. In practice, 2 can be found by looping over the vocabulary and finding the closest token. This oper- ation can be vectorized efficiently. For an embedding matrix E R consisting of embedding vectors, we project each embedding vector z; independently to get the matrix embedding (x) = [Pe(11) Pe (x2) ... P=(2)) Similarly, for the embedding tensor X, the projection operation can be vectorized efficiently to find Z.
Ensuring Readability of the Projected Text. Projecting embeddings to tokens in vocabulary independently does not yield meaningful text. To address this, we leverage the idea of top-k decoding to enforce the readability of generations (Fan et al.. 2018). Consider an embedding matrix x E ga consisting of n embedding vectors. For every embedding 24. we find the top k most probable tokens from the vocabulary condition on the previously projected tokens. Formally, we find the top i tokens that minimize EnEF (x/ and denote them by Expx. Then, we project D; into the space of the top-k selected tokens by solving 2; : Perpa (2:)- = arg min.eespp || - ell?.
Remark. Due to top-k decoding, the filtered synthetic examples do not match the target gradient very accurately. Thus, we generate each synthetic example to match the target gradient independently, not conditioned on each other. We will confirm in our experiments in Sec. 5.3 that the synthetic data generated independently by GRADMM matches the real data gradient closely during fine-tuning.
5

---

Synthetic Text Generation for Training Large Language Models via Gradient Matching
90
85
Accuracy 196
85
80
75
80
75
80
Accuracy
70
GRADMM
70
65
GRADMM 0.051
65
75
GRADMM
60
GRADMM
55
55
70
50
10
20
50
10
20
50
5
10
20
50
Number of validation samples
Number validation samples
Number validation samples
(a) SST-2
(b) Tweet emotions
(c) Rotten tomatoes
Figure/ 1. Data-scarce regime. Generating 100 synthetic samples with GRADMM, based 5, 10, 20, 50 examples from target task. Synthetic data generated based on only 5 real examples outperforms the real data by 15.7%. 29.7%, 76, and 31.5% on the three datasets.
4.5. Making GRADMM Differentially Private
of synthetic and real data during the fine-tuning.
Differential privacy (DP) (Dwork al., 2006) rigor- mathematical framework that ensures no single data point can be identified or inferred from the output of sta- tistical or machine learning model. To make GRADMM differentially private, we inject controlled noise a into the clipped gradient of the real data in Equation (4). Specifi- cally, GR ADM first computes per-sample gradients of the threshold of C. These
Lemma 4.1. Assume that the fine-tuning losses of the real L and synthetic data L are 3-smooth. The synthetic data generated by GRADMM that captures the grudient of real data by an error of (V((0) - (0)( S e at the pre- trained parameters 0, has a bounded gradient error at any point t during fine-tuning
[V (O))l S 2,36+ +
(15)
real data and clips their l2 norm then averaged, and Gaussian noise, drawn from ((0 o²), is added to this average. The added noise scale a is defined as follows:
clipped gradients
where 8 2 10 0.ll upper-bounds the norm of change to the parameteres during fine-tuning.
Next, we analyze the convergence of fine-tuning with gradi- ent descent on the synthetic subset generated by GRADMM. Theorem 4.2. For a p-PL" loss function C, under the as- sumptions of Lemma 4. I, gradient descent on the synthetic data converges with the same rate as that of real data. More- over, at every step t, the difference between the fine-tuning loss synthetic and real data is upper bounded by:
C 2lg 123
0 < e (Dwork et al., 2014)
Clotve+ts) /2elPral
i > 1 (Lowy & Razaviyayn, 2021)
where c
log v and the clipping threshold
C'= for all experiments. Based on the composition the- orem (Dwork et al., 2010), GRADMM achieves (e, 8)DP.
[C(0,) - L=) S <(2V - E)/2pp
(16)
We denote the new optimization problem and its augmented Lagrangian objective as 6)) and C(X, Z. &.A, .) re- spectively. As -> oc, the privacy constraint is relaxed and f(X,,, + f(X), yielding the original optimization problem. The new problem retains the same structure and can be solved using the ADMM procedure described before.
where { = 238 + and V is an upper bound on the gradient norm during fine-tuning.
The next corollary shows that fine-tuning on real data and synthetic data found by GRADMM yields similar models.
Pseudocode of our method, is illustrated in Alg
Corollary 4.3. Consider strongly convex loss (i.e. ||]] >
a > 0) with unique minimizer 0. and let L(0.)= =0. Then fine-tuning with any optimizer on real and synthetic data generated by GRADMM yield similar models:
4.6. Convergence Analysis
Next, we theoretically analyze the convergence of fine- tuning on the synthetic examples generated by GRADMM. As discussed in See. 3, fine-tuning short and changes the model a small extent compared pretraining. Effectively, the fine-tuning loss is relatively flat and can be modeled by 8-smooth (i.e., with bounded Hessian H) and p-PPL* (i.e., / (I) 2 ())( function.
00_ - -2 VE(2V -E)/au
(17)
Thus, the fine-tuned models will have similar performance.
5. Experiments
5.1. Experimental settings
For a synthetic subset generated by GRADMM via matching the gradient of real data at the pretrained model parameters, the following lemma bounds the error between the gradient
Datasets. We apply GRADMM to different text classifica- tion datasets including SST-2 movie reviews (Socher et al.,
6

---

Synthetic Text Generation for Training Large Language Models via Gradient Matching
Table 1. Fine-tuning Phi on synthetic examples generated by GRADMM, vs -.MM-generate zero-shot and few-shot synthetic data, vs real examples selected with herding. K K-center, and Random baselines. Synthetic data generated by GRADMM outperforms the baselines by up 10.4% and the only method that can preserve the privacy of the training data, GRADMM'S s synthetic data has similar log perppeexy (ppl) to that of real data, and higher ppl than LLM-generated synthetic data, confirming its more diverse nature.
Privacy V
LLM generated, no privacy X
Real data, privacy X
GRADMM
Zero-shot acc
Few-shot acc
Herding acc
K-center acc
Random
Rand 1K
Dataset
data
DC
0.05
ppl
ppl
ppl
ppl
ppl
acc
ppl
86.5 10.5
84.210.1
5.8
71.6144
2.5
71.8
3.0
61.21s.7
6.6
75.310.
5.5
52.912.4
7.7
SST-2
10
87.4 17.9
86.2118
5.1
79.116x
2.3
72.010.3
2.6
78.810.9
6.7
82.011.0
5.5
16.11115
7.3
20
87,9 27.9
87. 87.2+0.s
5.3
82.2+22
2.2
77.5.0.5
2.7
83.342..
6.6
86.220.
5.7
86.2
7.0
91.3
Base acc:
50
89.7
88.010.1
5.7
83.0g11
2.3
80.61a.9
2.6
86.020.4
6.6
88 88.1.10
5.5
87.8
87.
6.8
#0.4
69.6%
100
89.7
88.6
5.2
87.5
2.3
77.4+1.7
2.5
88.9
6.7
89.3
5.8
88 88.8
6.7
Tweet
5
83.8
80.4
3.5
70.7+10.1
2.7
52.5+2.
3.0
56.3
5.8
55.4
5.1
56.2=o.a
5.9
emotions
10
84.1
81.413.
2.8
70.3111.
2.3
47.510.3
3.2
58.410.4
5.6
61.0 61.028.5
5.6
62.1
5.8
20
85.21...
83.5114
3.4
81.711.9
2.2
68.213.1
3.3
65.819.4
5.8
73.5
5.5
70.6
5.8
96.1
Base acc:
50
85,8
83.2+2.0
4.3
82.7+1.9
2.2
79.0+2.7
3.1
76.7+2.9
5.6
83.9=1.2
5.1
77 77.4-3.1
5.6
#0.2
43.7%
100
86.5 10.1
83.9
3.8
84.2
2.3
83.511.
3.4
85.710.4
5.5
84.6
5.1
80.8
5.5
Rotten
82.2.0.3
12.6
4.5
72.6128
2.5
55.7128
3.8
69.81a.
4.9
60.0
6.4
70.414.2
5.4
tomatoes
10
82.9
81.820.9
5.5
75.3
2.3
63.7a29
3.0
71.512.
5.2
61.1
5.5
74.5
5.8
20
84,4
83.220.6
7.0
78.0
2.2
75.7224
3.1
79.
5.7
07.5.0.7
5.2
80.6g0.
5.7
88.1
Basc acc:
50
84.9
83.1
4.6
77.5
2.3
78.7
2.9
81.2
5.6
78.7=1.
5.1
81.121.8
5.6
#0.3
65.8%
100
85.0
83.2+0.7
4.5
81.3+1a
2.3
82.3+0.3
2.9
82.8
5.6
82.1
5,1
83.7=1.1
5.6
2013), Tweet emotions (Mohammad al., 2018), and Rot- ten tomatoes (Pang & Lee, 2005b).
5.2.1. GENERATING LARGER SYNTHETIC FINE-TUNING DATA IN DATA- SCARCE REGIME
Model. We use the Phi model (Li et al., 2023a) to generate synthetic data and for supervised fine-tuning,
First, consider the case where data is scarce for the tar- get task, and we wish to generate larger synthetic training data based on a small number of examples from the target task. Figure shows the result of applying GR GRADMM to generate 100 synthetic examples based on only 5, 10, 20, 50 examples randomly selected from the validation data of SST-2, Tweet emotions, and Rotten tomatoes. We see that GRADMM successfully generates high-quality supervised fine-tuning data that can train Phi to a superior performance over that of training on the available validation data. No- tably, GR ADMM generated synthetic data based on only 5 real examples outperform the real data by 15.7%, 29.7%, and 31.5% on the three datasets. This confirms the effec- tiveness of GRADMM in the data-scarce regime.
Fine-tuning settings. We fine-tune each model for 200 steps with Adam optimizer (Kingma, 2014) and batch size of 16. The learning rate follows a linear scheduler with the initial learning rate selected from {7c- 6, 1e - 5, 1.5e - 5).. We run an evaluation every 50 steps and report the best test classification accuracy among all the checkpoints.
Baselines. We compare our method with L.M-generated synthetic data with zero-shot and few-shot methods (Li et al., 2023b). We also compare to popular coreset selection meth- ods, namely Herding (Welling. 2009), K-center (Farahani & Hekmatfar, 2009), and Random.
Hyperparameters. The number of synthetic tokens is set to the average token length of all samples. For ADMM, the number of updates T is set to 30 and p is chosen from {0.001, 0.05, 0.01..... 103. To update X, we run 50 itera- tions of Adam with Ir 0.008. For the top-k projection, we use k= = 200. For DP, we use 8 = le-4 and 0.05.
5.2.2. GENERATING SMALL SYNTHETIC DATA BASED ON LARGER FINE-TUNING DATA
Next, we consider the case where a relatively large super- vised fine-tuning data is available, and we generate a smaller synthetic data to replace the real data to preserve the privacy of training examples or to improve the training efficiency.
5.2. Main results
GRADMM outperforms baselines and preserves privacy. Table 1 compares the performance of fine-tuning on syn- thetic data generated by GRADMM to that of zero-shot and few-shot techniques. It also shows the performance of fine- tuning on subsets of real data selected by herding, K-center, and Random baselines. We note that among all the methods, only the synthetic data generated by GRADMM can pre-
In our experiments, we consider the two scenarios discussed in Sec. 3. First, we apply GRADMM generate synthetic training data based target task. Then, we apply set of synthetic data by distilling an existing training data.
small number of examples from generate small
7

---

Synthetic Text Generation for Training Large Language Models via Gradient Matching
Table 2. Fine-tuning Llama-3.2-11 1B and OPT-1.3B on 20 synthetic samples generated by matching the gradient of a pretrained Phi.
Model
Dataset
Pretrained
GRADMM
Zero-shot
Few-shot
Herding
K-centers
Random real
SST-2
68.6
89.4
82.4
79.7
85.4
64.6
88.4
Llama-3.2-1B
Tweet emotions
43.7
85.8
83.4
74,4
76.1
88.5
83.9
Rotten tomatoes
67.5
87.8
80.5
78.5
73.6
87.8
84.3
SST-2
62.3
87.0
83.9
85.6
85.8
73.8
88.7
OPT-1.3B
Tweet emotions
43.7
78.5
77.8
76.9
75.3
74.7
77.7
Rotten tomatoes
63.1
87.9
74.9
80.6
80.8
84.8
85.5
Normalized grad error
0.50
urdd
Mear: 7.2
92
9.55
6:18
Mockan: 7.1
(39
8.9
Probability
0.09
2.000
2.05
86
0.04
833
GRADMM 20
0.02
0.35
50
0.2
0.4
0.0 0.5
0.2
10
80
Training Progress (0-1)
Distance to closest real data
Number of validation samples
(a) Last layer gradient difference
(b) L2 embedding distance
(c) Effect of
Figure Fine-tuning Phi on synthetic texts generated by GRADMM for SST-2. (a) Normalized last-layer gradient error during fine-tuning on synthetic data. Data generated by GRADMM yields significantly smaller gradient errors compared to the zero-shot baseline, indicating closer alignment with real data, (b) L2 embedding distance between GRADMM'S 's synthetic texts to their closest real training data, (c) Effect of e on the performance of fine-tuning Phi on GR GRADMM's synthetic data.
normalized last-layer gradient error, i.e. Vl Ve, C Ve, (B.)l))))// ((O))] at the pretrained parameters small, and this relation holds during fine-tuning. Notably, GRADMM generated data has much smaller gradient error than the zero-shot baseline during fine-tuning, corroborating its superior performance. Similar results for other datasets and full gradient error can be found in Appendix D.
Table SST-2, Divergence (FID) between the (i) training data distribution (Train), (ii) distribution of the few available real exam- ples (Val), (iii) distribution of the 100 GRADMM synthetic data, (iv) distribution of 100 zero-shot synthetic data (Zero).
# data (Train
Val) (Train || GRADMM) (Train || Zero)
71.8
44.2
10
59.8
43.3
56.0
51.6
39.8
GRADMM's synthetic data is close to real data. Table compares (for Figure la) the embedding divergence (in terms of FID) between the (i) training data distribution, (ii) the distribution of the few available real examples, (iii) the distribution of the 100 synthetic data generated by GRADMM and (iv) the distribution of 100 synthetic data generated using the zero-shot approach. Our synthetic data has a smaller FID, confirming that it has a more similar dis- tribution to that of real training data, compared to baselines. This corroborates the superior performance of GR GRADMMM While the effecctiveness of GRADMM depends on the di- versity of the available real examples, our empirical results show that a small number of randomly selected examples can be leveraged to effectively reduce the expected loss.
50
40.8
39.7
serve the privacy of training data. We see that GRADMM outperforms all the baselines across various datasets and data sizes, by up to 13.1% 1.. Notably, the synthetic data gen- erated by GRADMM has similar perplexity to that of real data, while having higher perplexity than LLM-generated synthetic data with zero-shot and few-shot methods. This confirms the more diverse nature of the synthetic data gen- crated by GRADMM, compared to LLM generated data.
GRADMM's synthetic data transfer to other LLMs. Ta- ble 2 shows the performance of fine-tuning Llama-3.2- 1B and OPT-L.3B on 20 synthetic examples generated with GRADMM by matching gradient of pretrained Phi model. We see that the data generated by GRADMM outperforms zero-shot and few-shot methods and the real data selected by herding and K-center baselines. This confirms the trans- ferability of the synthetic data generated by GRADMM.
GRADMM'S synthetic data is yet different from real data. Figure 2b shows the histogram of the distances of synthetic examples their closest real training data. None of the synthetic examples generated by GRADMM are very similar to the real training examples, confirming that our synthetic data is not identical to real examples.
5.3. Analysis
GRADMM preserves privacy. We apply bass-based MIA (Shokri et al., 2017) to the model fine-tuned on GRADMM'S synthetic data generated for SST-2 (Table
GRADMM yields similar gradient to real data during fine-tuning. For fine-tune Phi on GRADMM'S 's synthetic data generated from SST-2, Figure 2a illustrates that the
8

---

Synthetic Text Generation for Training Large Language Models via Gradient Matching
only important for interpretation and transferability of the re- sults, but it is crucial for obtaining satisfactory performance.
Table SST-2. Matching the gradient of last-layer yields higher performance with smaller number of synthetic data. Mapping the optimized embeddings to text via top-k projection (readable text)
Matching last-layer gradient boosts the performance, memory and speed of generation. Table 4 shows that matching the gradient of last-layer yields a higher perfor- mance with smaller number of synthetic data, At the same time, it reduces the generation memory by 2.6x (from 44.6G to 17.3G) and reduces the generation time by 2.3x (from 4.6 hours to 2 hours on one H100 GPU).
yields 9.2% higher accuracy than La projection (unrelated words).
Method
Acc
#data
ppl
GRADMM
90.0
68
5.2
GRADMM with full grad
89.6
89
5.5
GRADMM w/o top-k projection
80.8
57
13.3
Table 5. SST-2, Our filtering strategies effectively reduce the size of synthetic data from 200 to 68 and yield 1.9% higher accuracy.
Filtering improves the performance of synthetic data. Table shows the effect of the three filtering strategies dis- cussed in Sec. 4.4 to obtain a subset of at most r = 100 synthetic examples from the 200 synthetic data generated by ADMM. We observe that (i) removing examples that belong to the wrong category effectively improves the performance; (ii) selecting the top r = 100 examples with the lowest loss in every category effectively reduces the size of the syn- thetic data without harming its performance; (iii) dropping examples with highest loss in categories that have a larger average loss further reduces the size of the synthetic data while improving its performance. The filtering strategies reduce the size of the synthetic data from 200 to 68, while yielding 1.9% improvement in the fine-tuning performance.
Method
Acc
#data
ppl
ADMM
88.1
200
4.6
Removvng wrong labels
89.4
169
4.6
Selecting data with lowest loss
89.4
100
5.4
Balancing avg loss of categories
90.0
68
5.2
Great movic review is must experi- ence that will leave you in a state of all time high with the brilliant acting and the stunning production. Positive: The movic truly left me completely moved and in better place than when started it because of its well thought out and impactful way.
Positive:
Effect of e. Figure 2c compares the performance of syn- thetic texts generated with different values. As expected, increasing e, i.c., relaxing the privacy constraint, generally leads to better performance. Interestingly, when the number of validation samples is limited to just 5, the DP version with E = 4 outperforms the non-DP version (s = oo). We hypothesize that the injected noise may enhance generaliza- tion during training, consistent with observations from prior work (He et al., 2021).
Negative: The overall quality of action in this movie was not impressive enough to keep me away from the action center.
Negative: Terribly bad and boring to me as a person who values quality content and good storyline over mind.
Figure 3. Synthetic examples generated by GRADMM from SST-2.
1). We select N = 100 member samples from the training data and N non-member samples. Then, we find the optimal threshold that maximizes the advantages, defined as 2 x (ace- 50%), on these 2.N samples. Finally, we test loss-based MIA with optimal threshold on another 2N samples consisting of N members and N non-members. Averaged advantage scores over 10 runs (smaller absolute values indicate better privacy) are -.5% for s = 0.05 and 2.9% + 5.0 for 5 = x0. We see that even our non-DP version retains strong privacy, performing only slightly worse than the explicitly differentially private version.
Qualitative results. Fig. 3 shows examples of generated synthetic text by GRADMM from positive and negative classes of the SST-2, We see that the synthetic data is mean- ingful and semantically aligns with the target categories.
6. Conclusion
We proposed the first theoretically-rigoron method for gen- erating privacy-preserving synthetic readable text data by matching the gradient of real examples from a target task. We formulated this problem as a discretely constrained non- convex optimization in the embedding space and applied the Alternating Direction Method of Multipliers (ADMM) to iteratively optimizes the embeddings of synthetic exam- ples to match the noisy target gradient, and map them to a sequence of text tokens with low perplexity. We proved that the generated synthetic text can guarantee convergence of the model to close neighborhood of the solution obtained by fine-tuning on real data. Our experiments on various clas- sification tasks confirmed the effectiveness of GRA ADMM.
5.4. Ablation study
Generating readable text improves the performance. Ta- ble 4 shows that mapping the optimized embeddings to text via top-k projection yields readable synthetic text with low log-perplexity (ppl). In contrast, synthetic examples gener- ated via Le projection have considerably higher ppl, as they contain a set of unrelated words. Notably, the top-k pro- jection yields 9.2% better performance than L2 projection. This confirms that readability of the generated text is not

---

Synthetic Text Generation for Training Large Language Models via Gradient Matching
Impact Statement
Fan, A., Lewis, M., and Dauphin, Y. Hierarchical neural 2018.
story generation. arXiv preprint
This paper proposes theoretically-rigoouus method to gen- erate synthetic text data for fine-tuning LLMs. This can improve the privacy and training efficiency, and be applied to generate synthetic data in scenarios where real data is expensive or hard to obtain. There are many positive poten- tial societal consequences of our work, none which we feel must be specifically highlighted here.
Farahani, R. Z. and Hekmatfar, M. Facility location: con- cepts, models, algorithms and case studies. Springer Science & Business Media, 2009.
Gabay, and Mercier, B. dual algorithm for the solu- tion of nonlinear variational problems via finite element approximation. Computers & mathematics with applica- tions, 2(1):17-40, 1976.
Acknowledgements
Geiping. J., Bauermeister, H., Dröge, H., and Moeller, M. Inverting gradients-how easy is it to break privacy in federated learning? Advances in neural information processing systems, 33: 13:16937-16947, 2020.
This research was partially supported by the National Sci- ence Foundation CAREER Award 2146492, CAREER Award 2144985, the NSF-Simons AI Institute for Cosmic Origins, and an Okawa Research Award. We thank Tianjian Huang for their helpful discussion.
Gekhman, Z.,, Yona, G., Aharoni, R., Hyal, M., Feder, A., Reichart, R., and Herzig, J. Does fine-tuning Ilms on new knowledge encourage hallucinations? arXiv preprint arXiv:2405.00 05904, 2024.
References
Anthropic. Claude3. com/claudes 2023.
Glorot, X. and Bengio, Y. Understanding the difficulty training deep feedforward neural networks. In Pro- ceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Cazenavette, G., Wang. T., Torralba, A., Efros, A. A., and Zhu, J.-Y. Dataset distillation by matching training tra- jectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4750- 4759, 2022.
Glowinski, R. and Marroco, A. Sur I' l'approximation, par éléments finis d' d'ordre un, et la résolution, par iitalisstiionduallin d' une classe de problèmes de dirich- let non linéaires. Revue française d' d'automatique, infor- matique, recherche opérationnelle. Analyse numérique, (R2):41-76, 1975.
Dekoninck, J., Fischer, M., Beurer-Kellner, L., and Vechev, M. Controlled text generation via language model arith- metic. In The Twelfth International Conference on Learn ing Representations, 2024.
Deng, J., Wang, Y., Li, J., Shang, Liu, H., Rajasekaran, S., and Ding, C. Tag: Gradient attack transformer- based language models. arXiv preprint arXiv:2103.068/1 2021.
Gong, S., Li, M., Feng. J., Wu, Z., and Kong, L. Diffuseq: Sequence to sequence text generation with diffusion mod- els. arXiv preprint arXiv:2210.089333 2022.
Devlin, J. Bert: Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint 2018
Gupta, H., Scaria, K., Anantheswaran, U., Verma, S., Par- mar, M., Sawant, S. A., Baral, C., and Mishra. Tar- gen: Targeted data generation with large language models. arXiv preprint arXiv:2310.1788 2023.
Dwork, C., McSherry, F., Nissim, K., and Smith, A. Cal- ibrating noise to sensitivity in private data analysis. Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pp. 265-284. Springer, 2006.
Hartmann, V., Suri, A., Bindschaedler, V., Evans, D., Tople, S., and West, R. Sok: Memorization in general-purpose large language models. arXiv preprint arXiv:2310. 18362, 2023.
Dwork, C., Rothblum, G. N., and Vadhan, Boosting and differential privacy. In 2010 IEEE 5/ annual symposium on foundations of computer science, pp. 51-60. IEEE, 2010.
He, F., Wang, B., and Tao, D. Tighter generalization bounds for iterative differentially private learning algorithms. In Uncertainty in Artificial Intelligence, pp. 802-812. PMLR, 2021.
Dwork, C., Roth, et differential privacy. Foundations and Trends@ in Theo- retical Computer Science, 9(3-4):211 4077 2014.
et al. The algorithmic foundations of
Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. arXiv preprint 1904 2019.
10

---

Synthetic Text Generation for Training Large Language Models via Gradient Matching
Huang, T., Singhania, P., Sanjabi, M., Mitra, P., and Raza- viyayn, M. Alternating direction method of multipliers for quantization. In International Conference Artificial Intelligence and Statistics, pp. 208-216. PMLR, 2021.
progressive admm. arXiv preprint arXiv: 1905.0079. 2019.
Liu, C., Zhu, L., and Belkin, M. Toward a theory of op- timization for over wrer-parametrriee systems of non-linear equations: the lessons of deep learning. arXiv preprint arXiv:2003.0083 2020.
loffe. S. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. arXiv preprint 150.::300220000000 2015.
Loo, N., Hasani, R., Amini, A., and Rus, D. Efficient dataset distillation using random feature approximation. Advances in Neural Information Processing Systems, 35:
Katharopoulos, A. and Fleuret, F. Not all samples are cre- ated equal: Deep learning with importance sampling. In International conference on machine learning, pp. 2525- 2534. PMLR, 2018.
13877-13891, 2022.
Lowy, A. and Razaviyayn, M. Output perturbation for differentially private convex optimization: Faster and more general. arXiv preprint (tti::2222.04400 2021.
Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., and Socher, R. Ctrl: A conditional transformer lan- guage model for controllable generation. arXiv preprint arXiv: (eXiv:/909.055 05858, 2019.
Maas, A. L.. Daly, R. E., Pham, P. T.,, Huang, D., Ng, A. Y., and Potts, C. Learning word vectors for sen- timent analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguis- tics: Human Language Technologies, pp. 142-150, Port- land, Oregon, USA, June 2011. Association for Com- putational Linguistics. URL http::h/t/.. www.waclwee.... org/ pegganthology//// PP1-1015..
Kim, J.-H., Kim, J., Oh, S. J., Yun, S., Song, H., Jeong, J. Ha, J.-W., and Song, H. O. Dataset condensation via efficient synthetic-data parameterization. In Inter- national Conference on Machine Learning, pp. 11102- 11118. PMLR, 2022.
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv: Det:::442299990 2014.
Mackawa, A., Kobayashi, N., Funakoshi, K., and Okumura, M. Dataset distillation with attention labels for fine- tuning bert. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 119-127, 2023.
Langley, P. Crafting papers on machine learning. Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (CCML pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Mackawa, A., Kosugi, S.,, Funakoshi, K., and Okumura, M. Dilm: Distilling dataset into language model for text-level dataset distillation. arXiv preprint urXiv:2404.0000 002444, 2024.
Leng, C., Dou, Z., Li, H., Zhu, S., and Jin, R. Extremely low bit neural network: Squeeze the last bit out with admm. In Proceedings of the AAAI conference on artificial intel- ligence, volume 32, 2018.
Meng, Y., Huang, J., Zhang, Y., and Han, J. Generating training data with language models: Towards zero-shot language understanding. Advances in Neural Information Processing Systems, 35:462-477 202.
Li, X., Thickstun, J., Gulrajani, I., Liang, P. S., and Hashimoto, T. B. Diffusion-Im improves controllable text generation. Advances in Neural Information Process- ing Systems, 35:4328 030884333.222 2022.
Mirzasoleiman, B., Bilmes, J., and Leskovee. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6950 6900. PMLR, 2020.
Li, Y. and Li, W. Data distillation for text classification. arXiv preprint
2021.
Li, Y., Bubeck, S., Eldan, R., Del Giorno, A., Gunasekar, S., and Lee, Y. T. Textbooks are all you need ii: phi- 1.5 technical report. arXiv preprint arXiv:2309.05-00 2023a.
Mohammad, S., Bravo- Sravo-Marquee, F., Salameh, M., and Kir-
itchenko, S. Semeval-2018 task 1: Affect in tweets. In Proceedings of the 12th international workshop se- mantic evaluation, pp. 2018.
Li, 7., Zhu, H., Lu, 7.., and Yin, M. Synthetic data genera- tion with large language models for text classification: Po- tential and limitations. arXiv preprint 210.02231 2023b.
Nguyen, T., Chen, Z., and Lee, Dataset meta- learning from kernel ridge-regression. arXiv preprint arXi::2011.00050 2020.
Lin, S., Ma, X., Ye, S., Yuan, G., Ma, K., and Wang, Y. To- ward extremely low bit and lossless accuracy in dnns with
OpenAL. Gpt-4. https: / openai com/product// chatgpt, 2023.

---

Synthetic Text Generation for Training Large Language Models via Gradient Matching
Pang, B. and Lee, L. Seeing stars: Exploiting class relation- ships for sentiment categorization with respect rating scales. In Proceedings of the ACL, 2005a.
Yang. Y., Mishra, S., Chiang. J. N., and Mirzasoleiman, B. Smalltolarge (s21): Scalable data selection for fine-tuning large language models by summarizing training trajecto- ries of small models. arXiv preprint arXiv:2403.07.88 2024.
Pang, B. and Lee, L. Seeing stars: Exploiting class relation- ships for sentiment categorization with respect to rating scales. In Proceedings of the ACL, 2005b.
Ye, J., Gao, J., Li, Q., Xu, H., Feng, J., Wu, Z.,, Yu, T., and Kong, L. Zerogen: Efficient zero-shot learning via dataset generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11653-11669, 2022.
Sahni, S. and Patel, H. Exploring multilingual text data distillation. arXiv preprint WwXh::2308.04982 2023.
Shokri, R., Stronati, M., Song, C., and Shmatikov, V. Mem- bership inference attacks against machine learning mod- els. In 20/7 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Yu, Y., Zhuang, Y., Zhang. J., Meng, Y., Ratner, A. J., Kr- ishna, R., Shen, J., and Zhang, C. Large language model as attributed training data generator: A tale of diversity and bias. Advances in Neural Information Processing Systems, 36, 2024.
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631- 1642, Seattle, Washington, USA, October 2013. Asso- ciation for Computational Linguistics. URL https: //www. wwwwwb.wwwbb.... wtw.weeb.orgg....... http:///.r/aattoooog///a/------/
Zhao, B. and Bilen, H. Dataset condensation with differen- tiable siamese augmentation. In International Conference on Machine Learning, pp. 12674-12685. PMLR, 2021.
Zhao, B. and Bilen, H. Dataset condensation with distri- bution matching. In Proceedings of the IEEF/CVF Win- ter Conference on Applications of Computer Vision, pp. 6514 2514-,523, 2023.
Sucholutsky, I. and Schonlau, M. Soft-label dataset distil- lation and text dataset distillation. In 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2021.
Zhao, B., Mopuri, K. R., and Bilen, H. Dataset condensation with gradient matching, arXiv preprint arXiv:2006.05599, 2020.
Tao, Y., Kong, L., Kan, A., and Callot, L. Textual dataset distillation via language model embedding. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 12557-12569, 2024.
Zhou, Y., Wang. X., Niu, Y., Shen, Y., Tang, L., Chen, F., He, B., Sun, I.., and Wen, L. Difflm: Controllable synthetic data generation via diffusion language models. arXiv preprint arXiv:2411.033500 2024.
Wang. K., Zhao, B., Peng. X., Zhu, Z., Yang, S., Wang, S., Huang, G., Bilen, H., Wang, X., and You, Y. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12195-1220,, 2022.
Wang, T. Zhu, J.-Y,, Torralba, A., and Efros, A. A. Dataset distillation. arXiv preprint 181 10959, 2018.
Welling, M. Herding dynamical weights to learn In Pro- ceedings of the 26th annual international conference on machine learning, pp. 1121-1128, 2009.
Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D., and Mann, G. Bloomberggpt: A large language model for finance. arXiv preprint wxX.::3333.17666 2023.
Wu, S., Huang, Y., Gao, C., Chen, D.. Zhang, Q. Wan, Y., Zhou, T., Zhang, X., Gao, J., Xiao, C., et al. Unigen: A unified framework for textual dataset generation using large language models. CoRR, 2024.
12

---

Synthetic Text Generation for Training Large Language Models via Gradient Matching
A. Convergence
First we have the following lemma on the upper-bound the gradient difference the end of training:
A.1. Proof of Lemma 4.1
Proof. Let 0g be the parameter of the model after / steps of training. Let V2(00) - and define Rd b IR as
(0) = (d. V VL)
For a fixed e IRd the mean value theorem implies that h()) = h(0o) + (Vh(E), 0 - 0o) for some & Rd on the line segment connecting and Po. Using the definition of the function h(-). we obtain
(d. VL(0) vC> vC((0) (00))
v2 d Oo!
Setting 0 = O., we get
(d, VC())) - + -1 0, - -00
(d,
(18)
$ d [V - (C) +[V - ( ||d||2 100 00ll2,
(19)
where in the last line, we used the
Schwwrtz inequality, the inequality ||ABB|2 S ||A|22 and the fact that
= 02 0g- Plugging the value in the LHS of equation 18 and equation 19, we obtain
[[V - (.. = ( VC IVC v=*(())l:
+ <| - v (E)l2 IV
- v V (0.)ll))l 10, 0ool2
Dividing both sides by [[,)) - (0))l1) and using the fact that [|v* - ( S | +
[[) S 23, we get
[VV (.)(l < (V))0)) - ( + 23 - 2 - 236,
(20)
which completes the proof.
A.2. Proof of Theorem 4.2
Next, we prove the convergence of GD on the real data vs synthetic data generated by GRADMM.
Proof. For Lipschitz continuous g and p-PL condition, gradient descent on the real data yields
[(0++1) [(0.)
" g: 1 (0).(())
(21)
and,
((0.) < (1 -
(22)
which was shown in (Liu et al., 2020).
For the synthetic data we have
L (0.+1) (²(0,) |g::l
(23)
By substituting Eq. (20), and assuming & S ||g: have:
L (0 (0.))) <
17
(1lg: || E
(24)
2
77
(llg. 22 + 2 2g||g:
(25)
2
(llg: 1| + g² 2(V)
(26)
2
(2nL(0.) 5 -2V))
(27)
2
13

---

Synthetic Text Generation for Training Large Language Models via Gradient Matching
where V is upper bound on the norm of ge in Eq. (25), and Eq. (27) follows from the JrPL condition.
Hence,
[(0++)) S (1-1))))((0)) -22-222 -5" v
(28)
Since, E)=o(1 m/)5 $ mp for a constant learning rate 7) we get
C(0++1) S S (1-mmu)+ m1)(tt(((0)))) [()0) 1 -(5 25V) -VV)
(29)
2μ
A.3. Proof of Corollary 4.3
Proof. If H
2 a > and [(O.) = 0, we have that ||0 - -011 < 2 (()) From Theorem 4.2 we get:
2 (()) <
@ 0.11 < a
(2 - -² = = - E3/ap
(30)
2p
B. Prompts
B.1. Zero/Few-shot prompts
Figure summarizes the zero-shot prompts (Li et al., 2023b) to generate synthetic samples. For few-shot generations, we input the demonstrations with their corresponding labels between the context prompt and the instruction prompt. addition, we add a sentence "You should imitate the example have provided, but you cannot simply modify or rewrite the example I have given." to the instruction part prevent the LLMs from simply rewording the given examples. The few-shot prompts can be found in Figure 5.
SST2 and Rotten Tomatoes: You are now movie critic. You are provided with sentiment label. You need to write one unique sentence that reflects the given sentiment about movie. Your writing style should be consistent with typical movie reviews. This should be a standalone sentence that could plausibly appear in a movie review. Ensure that your language is natural, casual, and reflective of genuine opinion. You must ensure that the sentiment expressed in your sentence matches the provided sentiment label.
Remember keep your appropriate and not violate any laws social ethics. Please be creative and write only one sentence. The sentiment of the movie review is {label}. Answer: Tweet Emotions: You are now person using twitter. You are provided with emotion, and you need to write tweet expressing that emotion. Your writing style must be consistent with the tweets on twitter. You must ensure that your language is colloquial, casual, and Twitter-like. You are given a length requirement. You must ensure that the emotion conveyed your tweet matches the emotion provided and meets the length requirement. This an academic study and the content you generate will not be used for anything that violates the law or social ethics. Write tweet expressing the emotion and ensure the tweet is within the usual length. Remember make sure that your language is colloquial, casual, and Twitter-like. Please be creative and write only one unique tweet. The emotion of twitter is (label}. Answer:
Figure 4. Zero-shot prompts for different datascts.
B.2. Few-shots Evaluation prompts
Figure presents the evaluation prompts used filter out synthetic data with incorrect labels.
14

---

Synthetic Text Generation for Training Large Language Models via Gradient Matching
You are now a movie critic. You are provided with sentiment label. You need to write one unique sentence that reflects the given sentiment about movie. Your writing style should be consistent with typical movie reviews. This should be standalone sentence that could plausibly appear in a movie review. Ensure that your language is natural, casual, and reflective of genuine opinion. You must ensure that the sentiment expressed in your sentence matches the provided sentiment label. {Few-shot examples)
SST2 and Rotten Tomatoes:
Remember to keep your tone appropriate and not violate any laws or social ethics. Please be creative and write only one sentence. The sentiment of the movie review is [label]. You should imitate the example I have provided, but you cannot simply modify or rewrite the example have given. Answer.
Tweet Emotions: You tweet expressing that emotion. Your writing style must be consistent with the tweets on twitter. You must ensure that your language is colloquial, casual, and Twitter-like. You are given a length requirement, You must ensure that the emotion conveyed in your tweet matches the emotion provided and meets the length requirement. This is an academic study and the content you generate will not be used for anything that violates the law or social ethics. [Few-shot examples]
person using twitter. You provided with emotion, and you need write
Write tweet expressing the emotion and ensure the tweet within the usual length. Remember to make sure that your language colloquial, casual, and Twitter-like. Please be creative and write only one unique tweet. The emotion of twitter is {label). You should imitate the example have provided, but you cannot simply modify or rewrite the example have given. Answer:
Figure Few-shot prompts for different datasets.
C. Generation Samples
C.I. Synthetic SST2 Samples by GRADMM
C. 1.1.1. POSITIVE LABEL
Great movie review is must experience that will leave you in state of all time high with the brilliant acting and the stunning production
Great action and special effects combined with compelling emotional connection with the the show characters made it a one of the best ever watched
The movie truly left completely moved and in better place than when started it because of its well thought out and impactful way
The new action movie was absolutely thrilling and had me on the outside of my skin throughout the entire two acts of the first two and
The action movie kept me sitting Jane and was on the point of wanting to leave the entire time but the way the story was told
1.2. NEGATIVE LABEL
The action action not well executed and the plot not as it should in a science or
The movie was a not so great film that would not want to see second time because the the
The overall quality of action in this movic was not impressive enough to keep me away from the action center of
Terribly bad and boring to me as a person who values quality content and good storyline over mind
The new movie was a not so great and disappointing experience for me since did not keep up with the
C.2. Synthetic Rotten Tomatoes Samples by GRADMM
1.2.1. POSITIVE LABEL
The action-adventure movie was thrilling and had way keeping me right on the of the
15

---

Synthetic Text Generation for Training Large Language Models via Gradient Matching
The movie was fantastic and thrilling!
Label: positive
I hated the film; it was boring and slow. Label: negative
What a masterpiece, truly inspiring! Label: positive
The plot dull and characters uninspiring- Label: negative
[Evaluation sample}
Label: positive/negative
Figure Few-shot evaluation prompts. text indicates the labels predicted by the model.
The suspense in the final act left room for the most important and most thrilling of all parts of the movie
The new 'Innate Robots' is a must-see for anyone who loves the latest in the field technology
The critically acclaimed action movie "Fast and Far East City" is a work of the highest caliber ever to be made
The action-movie Ace Driver' a "wins' masterpiece that will leave you feeling 'th
C.2.2. NEGATIVE LABEL
The action level plot of the movie not up to mark. The use cases
engaging and the the use of the provided
The movie was terrible. You writing page story set in world where people only see the world
The new movie was absolutely not enjoyable. The over-dilting of the water made it real downer experience
The action-adventure filled movie was disappointment. The excessive of the 't' sound
he quality of the action in Bad Movie is not up to the standard set by the other 'g' and movie.
C.3. Synthetic Tweet Emotions Samples by GRADMM
C. POSITIVE LABEL
just got a new, high-end phone. It's got new
Joyful and sharing a good time with my friends. Life is so much better now
Joy is a feeling that makes you feel like you are on the up and up
just got a new job working at a local Use
thrilled to new member of the twitterhub.
C.3.2. NEGATIVE LABEL
am so sad today. Sad is the word that I would use to write this
just received some sad and life news. I can't 't believe it. am so am so over it. I can't 't even believe it's over. can't
just received news that my best-loved, and most-licked-at
just got back from a long day at work. can't help
16

---

Synthetic Text Generation for Training Large Language Models via Gradient Matching
92
Accuracy 1%6)
89
86
76
GRADMM
73
GRADMM
Val
80
10
20
50
5
10
20
50
Number vaiidation samples
Number validation samples
(a) IMDB
(b) Sentence polarity
Figure 7. Data-scarce regime. Generating 100 synthetic samples with GRADMM, based 10, 20, 50 examples from target task.
Synthetic data generated based on only 5 real examples outperforms the real data by 8.9% and 12.5% on the two datasets.
Hornalized grad enrer
GMD Zeroshat
Normalized grad erro
Reai-SMA3MM ReahiZershht grad der
0.50
3.30
0.45
0.4
0.35
0.40
0.3
GRADIIM grad grad Training Progress (0-1)
04
0.55
Training Progress
Training Progress
(a) SST-2
(b) Tweet emotions
(c) Rotten tomatoes
Figure 8. Last-layer gradient differences during fine-tuning synthetic real data. Data generated by GRADMM yields significantly smaller gradient errors compared to the zero-shot baseline, indicating closer alignment with real data.
Nomalized grad etror
Sas GRAVVN di Sea Zeronbot gndd dit
Normal ttred grad error
0.00
0.075
0.75
6-7
8..70
0.70
0.55
0.50
godd
0.35
Training Progress (0-1)
Training Progress (0-1)
Training Progress (0-1)
(a) SST-2
(b) Tweet emotions
(c) Rotten tomatoes
Figure 9. Full gradient differences during fine-tuning synthetic vs. real data. Data generated by GRADMM yields significantly smaller gradient errors compared to zero-shot baseline, indicating closer alignment with data.
Mean: 7.0
0.12
Mean: 7.2
0.08
Median: 7.0
Median: 7.1
0.10
Probability
Probability
0.08
0.06
0.04
0.04
0.00
11
0.00
12
Distance to closest real data
Distance to closest real data
(a) Tweet emotions
(b) Rotten tomatoes
Figure 10. L2 embedding distance between GRADMM's synthetic texts to their closest real training data in (left) Tweet emotions and (right) Rotten tomatoes.
17

---

Synthetic Text Generation for Training Large Language Models via Gradient Matching
D. Additional experiments
Additional datasets. We include two additional datasets namely IMDB (Maas et al., 2011) and Sentence polarity (Pang & Lee. 2005a) in the setting of Figure Section 5.2.1. Figure shows the result of applying GRADMM to generate 100 synthetic examples based on only 10, 20, 50 examples randomly selected from the validation data of IMDB and Sentence polarity. We that GRADMM successfully generates high-quality supervised fine-tuning data that train Phi to superior performance over that of training the available validation data. Notably, ARADMM generated synthetic data based on only 5 real examples outperform the real data by 8.9% and 12.5% on the two datasets. This confirms the effectiveness of GRADMM in the data-scarce regime.
Last-layer gradient error. Figure demonstrates the normalized last-layer gradient error with respect to real data, i.e. (i (().) Ve, (ee))ll)) [()))l. remains low at the pretrained parameters at the pretrained parameters and continues stay low throughout fine-tuning. Notably, the data generated by GRADMM yields significantly lower gradient error than the zero-shot baseline during fine-tuning, supporting its better performance.
Full gradient error. Figure 9 shows that the normalized full gradient error with respect to real data, i.e. Vll())
(O))))//IV While GRADMM only matches the last-layer gradients of real and synthetic samples, we observe the same trend as that last-layer gradient error in Figure This reinforces our last-layer argument in Section 4.3.
Embedding distance. Figure 10 presents histogram of the distances between synthetic examples and their nearest real training samples. The absence of synthetic examples that are extremely close real data indicates that GRADMM does simply replicate real training instances.
18


Reviews:
[{"review_id": "VmoF18X0se", "reviewer": "Reviewer_Xke2", "summary": "This paper discussed a method for generating synthetic data to train LLMs, and aims to create a synthetic dataset that can train a similar dynamics to the real data. Theories and experiments are provided to justify the effectiveness.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. However, we disagree with their evaluation of our work, as we discuss below. \n\n----\n\n**Data scarce regime**: Fig 1 provides strong evidence for the applicability of our method to the data scarce regimes. The 100 synthetic data generated by GRADMM based on as few as 5 to 20 *randomly selected real examples* already reach 98%, 89%, 96% the performance of training on the *full training data* (last col in Tab 1), and outperform training on the 5-20 real examples with a big margin. \n\nThe following [table](http://anonymous.4open.science/r/gradmm/fid_score.png) compares (for SST2, Fig 1a) the divergence (in terms of FID) between the (i) training data distribution, (ii) the distribution of the few available real examples, (iii) the distribution of the 100 synthetic data generated by GRADMM and (iv) the distribution of 100 synthetic data generated using the zero-shot approach.\n\nOur synthetic data has a smaller FID, confirming that it has a more similar distribution to that of real training data, compared to baselines. This corroborates the superior performance of GRADMM. While the effectiveness of GRADMM depends on the diversity of the available real examples, our empirical results show that a small number of randomly selected examples can be leveraged to effectively reduce the *expected loss*. We do not claim that GRADMM *perfectly minimizes* the expected loss with limited information (a few available real examples), but it is considerably more effective than other baselines in the data scarce regime.\n\n**Privacy concerns**: Using real examples as a guide to generate synthetic data **does not compromise the privacy of real training examples**. As discussed in Sec 2, GRADMM is the first *dataset distillation (DD)* method able to generate human readable text. DD methods provide differential privacy guarantees [1]. Specifically, for two datasets that differ in only 1 example, the parameters of the models trained on the distilled version of the two datasets are provably highly similar [1]. Intuitively, as dataset distillation methods generate data by techniques such as matching the mean of the data distribution or average gradient of a real training data, as long as synthetic data is not initialized by real training examples, there is no information leakage about individual training examples [1]. GRADMM does not initialize synthetic data with real training examples, and thus effectively preserves the privacy of the training data.\n\nTo confirm, we conducted the loss-based Membership Inference Attack (MIA) to the model trained on synthetic SST2 data generated by GRADMM. Specifically, we select N=100 member samples from the training subset and N non-member samples. Then, we find the optimal threshold that maximizes the advantages (2 x (acc - 50%)) on these 2N samples. Finally, we test the loss-based MIA with optimal threshold on another 2N samples consisting of N members and N non-members and report the advantage score. We repeated the whole process 10 times and the advantage score (%) is only 1.75 $\\pm$ 1.4, demonstrating the effectiveness of GRADMM against loss-based MIA.\n\nFinally, the following [figures](https://anonymous.4open.science/r/gradmm/embedding_distance/) show the histogram of the distances of synthetic examples to their closest real training example. Indeed, none of the GRADMM synthetic examples are very similar to the real training examples, further confirming that our synthetic data is not very similar to real examples.\n\n**Theoretical assumption**: Our gradient matching approach ensures a high similarity to the target gradient. The following [figures](https://anonymous.4open.science/r/gradmm/grad_diff/) confirm the validity of our theoretical assumption, by showing that the gradient error, i.e. $\\| \\nabla \\mathcal{L}(\\theta_t) - \\nabla \\mathcal{L}^s(\\theta_t) \\| \\leq \\| \\nabla \\mathcal{L}(\\theta_t) \\|$ at the pretrained parameters and this relation holds during fine-tuning. Crucially, GRADMM synthetic data has a much smaller gradient error compared to the zero-shot baseline during fine-tuning, which is the reason for its superior performance. \n\n**Empirical evidence**: The following [figures](https://anonymous.4open.science/r/gradmm/figure1) show std for our experimental results (based on three runs) and confirms the strong performance of GRADMM on the 3 dataset in our paper in addition to **two new datasets**, namely IMDB [Maas et al., 2011] and Sentence polarity [Pange et al., 2005].\n\nWe hope that our rebuttal could address the reviewers’ doubts. Our work is the first to show the applicability of dataset distillation to generate readable synthetic text via gradient matching with strong empirical performance, and provides a promising avenue for future research. Thus, we hope that the reviewer considers supporting our work to be accepted.\n\n[1] Privacy for free: How does dataset condensation help privacy? ICML’22.", "claims_and_evidence": "I think the evidence is not very convincing, especially regarding the theory. \n\nHere is the argument: the authors try to create synthetic data that can mimic the dynamics of read data **samples** regarding training dynamics. But whether this is really useful? I am highly skeptical. The purposes of generating synthetic data are mainly for the following two perspectives: 1. help the mitigate the issue of limited real data; 2. privacy concerns. But the proposed method can not solve either of them. In the case the real data samples are scarce, generating synthetic data cannot fix the gap of generalization (notice that the ultimate goal is to train to optimize the **expected loss**). In the case of privacy, the synthetic data directly use real data to help generation, so privacy is also violated. \n\nI really doubt the usefulness of the proposed method. The experimental results are also quite weak and not comprehensive. 5.2.1 only uses 3 datasets and the evolution metrics are not comprehensive. Effective sample size and ideally confidence intervals for multiple runs should be considered.", "methods_and_evaluation_criteria": "Not comprehensive enough. See Claims And Evidence.", "theoretical_claims": "Highly doubt the theoretical results. Besides the statements mentioned above in Claims And Evidence, lemma 4.1 will rely on pertaining errors on synthetic data and real-data trained model. I think this is not guaranteed to be small. I am not thinking the statement is making much sense given this presumption.", "experimental_designs_or_analyses": "The evaluation is not comprehensive, For instance, Figure 1 only includes single trials without confidence interval.", "relation_to_broader_scientific_literature": "Synthetic data is quite important to scientistic discovery, but this paper is not particularly related to that.", "essential_references_not_discussed": "NA.", "other_strengths_and_weaknesses": "As mentioned above, I really doubt the usefulness of the proposed method. The experimental results are also quite weak and not comprehensive. 5.2.1 only uses 3 datasets and the evolution metrics are not comprehensive. Effective sample size and ideally confidence intervals for multiple runs should be considered.", "other_comments_or_suggestions": "NA.", "questions_for_authors": "NA.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ne6smckBwH", "reviewer": "Reviewer_RncU", "summary": "This paper improves on the SOTA synthetic data for LLM method by imposing a readability constraint in (2). This makes it necessary to 4.2 Alternating Between Text and Embedding Spaces. The experiments are convincing.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for supporting our work and acknowledging our convincing experiments.\n\n----\n\n1. Can this scheme used to generate math, logic, and code?\n\nThe idea of our work (generating synthetic text via gradient matching using ADMM) can be applied to math, logic or code. However, this requires incorporating additional structure in the generated text. Controllable text generation which requires the text to follow a particular structure, syntactic or semantic properties has been a topic for several recent research, including [Li et al’22, Gong et al’22, Zhou et al’24 (references are in Line 133 from the paper)] to generate text in the form of tables, code, etc. Such techniques can be incorporated in our method to generate structured synthetic text. Considering that our method is the first to show the applicability of gradient matching to generate readable synthetic text, it lays the ground for many exciting future work, including controllable text generation via gradient matching.", "claims_and_evidence": "They are good", "methods_and_evaluation_criteria": "They are good", "theoretical_claims": "They are good", "experimental_designs_or_analyses": "They are good", "relation_to_broader_scientific_literature": "I didn't review this aspect carefully", "essential_references_not_discussed": "I didn't review this aspect carefully", "other_strengths_and_weaknesses": "No", "other_comments_or_suggestions": "No", "questions_for_authors": "Can this scheme used to generate math, logic, and code?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YWtMPs3Pwc", "reviewer": "Reviewer_FREm", "summary": "This paper presents a novel approach for generating synthetic human-readable text to train Large Language Models (LLMs) via gradient matching. The authors propose a method called GRADMM (GRADient matching with ADMM) that leverages the Alternating Direction Method of Multipliers (ADMM) to iteratively optimize synthetic text embeddings to match the gradient of real training data. The goal is to generate synthetic text that not only preserves the privacy of real data but also ensures similar training dynamics and performance when used to fine-tune LLMs.\nThe key contributions of this work include:\n1. A theoretically rigorous framework for generating synthetic text that guarantees convergence and performance comparable to fine-tuning on real data.\n2. The use of gradient matching in the embedding space to ensure that the synthetic text has similar training dynamics to real data.\n3. A method to project optimized embeddings into human-readable text while maintaining low perplexity.\n4. Experimental validation showing that GRADMM-generated synthetic text outperforms existing methods in terms of training efficiency and privacy preservation.\n\nThe authors demonstrate the effectiveness of GRADMM through extensive experiments on various text classification tasks, including SST-2, Tweet emotions, and Rotten tomatoes. The results indicate that GRADMM can generate high-quality synthetic data even with limited real examples, achieving significant performance improvements over baseline methods. Additionally, the generated synthetic text is shown to be transferable to other LLMs, further validating the method's practicality and versatility.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the novelty of our method, our theoretically-rigorous framework, our well-supported claims and extensive experiments.\n\n----\n\n**Experimental validation**:\n- Problem formulation: gradient matching in Eq 3 applies to both the data scarce regime (Fig 1) and Dataset Distillation (DD) (Table 1). In both, the synthetic data is generated by matching the average gradient of the few available examples (Fig 1) or a larger training data (Table 1). \n\n- Evaluation: while accuracy is an accurate metric to measure the performance in both cases, we include the following [table](http://anonymous.4open.science/r/gradmm/fid_score.png) for Fig 1a (SST2) which shows the divergence (FID) between the (i) training data distribution, (ii) distribution of the few available real examples, (iii) distribution of the 100 GRADMM synthetic data, (iv) distribution of 100 zero-shot synthetic data. Our synthetic data has a smaller FID, confirming its more similar distribution to that of real training data, compared to the baselines. This corroborates the superior performance of GRADMM.\n- FID is not generally used to evaluate DD (Table 1), which aims at generating a **small** subset of synthetic data with similar dynamics to a large training data. This is because: (i) FID requires both distributions to have a large sample size such that they resemble gaussian distributions, and otherwise is less accurate. (ii) distribution of the small generated data is not comparable to the large real training data (in particular, their variances are not comparable due to their very different sizes). Hence, FID is not commonly used for DD (c.f. the DD related work in Sec 2.1), but is more common when generating large data with diffusion models.\n\n**Theoretical bounds**: \n- The following [figures](https://anonymous.4open.science/r/gradmm/grad_diff/) confirm the validity of our theoretical assumption, by showing that the gradient error, i.e. $\\| \\nabla \\mathcal{L}(\\theta_t) - \\nabla \\mathcal{L}^s(\\theta_t) \\| \\leq \\| \\nabla \\mathcal{L}(\\theta_t) \\|$ at the pretrained parameters, and this relation holds during fine-tuning. Crucially, the GRADMM generated data has a much smaller gradient error compared to the zero-shot baseline during fine-tuning, corroborating its superior performance. \n- We generated 200 synthetic data in Figure 1a. The accuracy of the model fine-tuned on this larger subset is 90.1 $\\pm$ 0.1, which is higher than 89.8 $\\pm$ 0.4 of training on the original 100 synthetic data, and is closer to the last column in Table 1. \n- We compared the $L_2$ normed difference in model parameters when trained on real training data vs 100 synthetic data generated for SST2. The normed difference for GRADMM is 1.99, which is smaller than the 2.27 for zero-shot. This further confirms the validity of our theory.\n\n**Privacy**:\n- Thanks! We will add discussion of the importance of privacy, data leakage and benefits of synthetic data to our revised version.\n- GRADMM is the first *dataset distillation* method able to generate human readable text. DD methods provide differential privacy guarantees [1]. Specifically, for two datasets that differ in only 1 example, the parameters of the models trained on the distilled version of the two datasets are provably highly similar [1]. Intuitively, as dataset distillation methods generate data by techniques such as matching the mean of the data distribution or average gradient of a real training data, as long as synthetic data is not initialized by real training examples, there is no information leakage about individual training examples [1]. GRADMM does not initialize synthetic data with real training examples, and hence effectively preserves the privacy of the training data. \n- We conducted the loss-based MIA to the model trained on GRADMM synthetic data for SST2. Specifically, we select N=100 member samples from the training subset and N non-member samples. Then, we find the optimal threshold that maximizes the advantages (2 x (acc - 50%)) on these 2N samples. Finally, we test the loss-based MIA with optimal threshold on another 2N samples consisting of N members and N non-members and report the advantage score. We repeated the whole process 10 times and the advantage score (%) is only 1.75 $\\pm$ 1.4, demonstrating the effectiveness of GRADMM against loss-based MIA.\n- The following [figures](https://anonymous.4open.science/r/gradmm/embedding_distance/) shows the histogram of the distances of synthetic examples to their closest real training data. None of the synthetic examples generated by GRADMM are very similar to the real training examples, further confirming that our synthetic data is not identical to real examples.\n\nWe will add the above discussion and results for all the datasets to our revised version. We hope that our rebuttal has addressed the reviewer's concerns and they can consider further supporting our work to be accepted.", "claims_and_evidence": "The claims made in the submission are generally well-supported by clear and convincing evidence. The authors provide a comprehensive theoretical framework and extensive experimental validation to back up their assertions. Below is a detailed evaluation of the key claims and the evidence provided:\n\n1.\tGRADMM generates synthetic text that guarantees convergence and performance comparable to fine-tuning on real data.\nThe authors provide a rigorous theoretical analysis, including Lemma 4.1 and Theorem 4.2, which bound the gradient difference and prove the convergence of fine-tuning on synthetic data generated by GRADMM. This theoretical foundation supports the claim that GRADMM can produce synthetic text with similar training dynamics to real data.\nThe experiments on various text classification tasks (SST-2, Tweet emotions, and Rotten tomatoes) demonstrate that GRADMM-generated synthetic text consistently outperforms or matches the performance of fine-tuning on real data. \n\n2.\tGRADMM preserves the privacy of real training data.\nThe synthetic text generated by GRADMM is guaranteed to be different from real data, ensuring privacy. The authors emphasize that GRADMM does not directly use real data samples but rather matches the gradient of real data, which inherently preserves privacy. But I think the evidence is weak and should be more experiments to demonstrate it.\n\n3.\tGRADMM-generated synthetic text is human-readable and semantically meaningful.\n The authors use a top-k projection mGRADMM, which are shown to be meaningful and semantically consistent with the target labels. For instance, the synthetic movie reviews and tweets generated.\n\n4.\tGRADMM is computationally efficient.\nGradient Matching: The authors argue that matching the gradient of the last layer of the model significantly reduces the computational cost compared to matching the full gradient. This approach allows for faster and more memory-efficient generation of synthetic data.\nExperimental Validation: The paper reports that GRADMM reduces the generation time by 2.3x and memory usage by 2.6x compared to matching the full gradient, demonstrating its computational efficiency.", "methods_and_evaluation_criteria": "The GRADMM method contains three important parts:\n1. Alternating Direction Method of Multipliers (ADMM): The use of ADMM to iteratively optimize synthetic text embeddings to match the gradient of real data is a appropriate choice for this problem. ADMM is well-suited for solving constrained optimization problems and provides a theoretical characteristic for evaluating the model training on synthetic data.\n2. Gradient Matching in the Embedding Space: Matching the gradients of synthetic and real data in the embedding space is a clever way to ensure that the synthetic text captures the essential training dynamics of real data. \n3. Top-k Projection for Readability: The method of projecting optimized embeddings into human-readable text using top-k decoding ensures that the generated text is both meaningful and semantically aligned with the target categories. This technique balances the need for readability with the constraints of the vocabulary and perplexity.", "theoretical_claims": "Yes, I reviewed the proofs of Lemma 4.1 and Lemma 4.2. In Lemma 4.1, there is a small error in the transition from equation (19) to (20): specifically, \\(d = \\theta_t - \\theta_0\\).", "experimental_designs_or_analyses": "Yes. I checked the sec. 5.2 and 5.3.", "relation_to_broader_scientific_literature": "Yes.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths\n+ The paper introduces a novel approach to synthetic text generation using gradient matching, which is a creative and effective way to ensure that the synthetic data captures the essential training dynamics of real data. This method is particularly innovative in its application to text data, which is inherently discrete and challenging to optimize directly.\n+ The authors provide a strong theoretical foundation for their method, including convergence guarantees and bounds on the gradient difference. This theoretical analysis adds credibility to the proposed approach and differentiates it from heuristic methods that lack formal guarantees.\n+ The paper demonstrates the practical applicability of GRADMM by showing its effectiveness in generating synthetic text for fine-tuning LLMs. The results on various text classification tasks highlight the potential of this method for real-world applications, especially in scenarios where real data is scarce or privacy is a concern.\n\nWeaknesses\n1. Insufficient Experimental Validation:\n   - Inconsistent Problem Formulation: The experiments in Sections 5.2.1 and 5.2.2 address different aspects of the problem (data scarcity vs. data distillation) but are compared using the same metrics (accuracy and perplexity). This approach may not fully capture the nuances of each scenario. \n   - Lack of Specific Metrics: The paper could benefit from more specific metrics tailored to each experimental setting. For example, in the data distillation scenario, metrics such as the similarity between synthetic and real data distributions (e.g., using divergence measures) could better illustrate the effectiveness of GRADMM.\n\n2. Theoretical Bounds vs. Practical Insights:\n   - Lack of Empirical Validation for Theoretical Bounds: While the theoretical bounds provided in the paper are valuable, they could be complemented with empirical evidence to demonstrate their practical relevance. For example, visualizing the gradient difference over training iterations or comparing the theoretical bounds with actual performance metrics could provide more intuitive insights.\n3. Privacy Importance and Metrics:\n   - Importance of Privacy: The paper does not sufficiently emphasize the importance of preserving training data privacy. A detailed discussion on the potential risks of data leakage and the benefits of using synthetic data in this context would strengthen the paper's motivation.\n   - Lack of Privacy Metrics: The paper claims that GRADMM preserves privacy but does not provide specific metrics or experiments to validate this claim. Including metrics such as membership inference attack success rates or differential privacy guarantees could provide a more concrete assessment of the method's privacy-preserving capabilities.\n\n\nOverall, the paper presents a novel and theoretically sound approach to generating synthetic text for training LLMs. The strengths of the paper lie in its innovative use of gradient matching, rigorous theoretical analysis, and practical applicability. However, the weaknesses identified suggest areas for improvement, particularly in experimental validation, privacy assessment, and clarity of presentation. Addressing these issues could significantly enhance the paper's impact and applicability to real-world problems.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1.  How to evaluate that GRADMM preserves privacy?\n2.  Why did you generate only 100 synthetic data points based on 5, 10, 20, and 50 real data points? Why not generate a larger dataset comparable to the last column in Table 1?\n3.  In practice, are there experiments that can demonstrate whether the parameters of a model trained on synthetic data are indeed within a certain neighborhood of those trained on real data?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sUoHd1O3vi", "reviewer": "Reviewer_HXUe", "summary": "This paper introduces GRADMM (Gradient Matching with ADMM), a novel method for generating synthetic, human-readable text to train large language models (LLMs) efficiently while preserving privacy. The approach leverages gradient matching to ensure that synthetic text replicates the training dynamics of real data, using the ADMM to optimize embeddings in a continuous space and project them into discrete, readable text sequences with low perplexity via top-k decoding. A key conceptual contribution is the theoretical guarantee that fine-tuning an LLM on this synthetic text converges to a solution close to that of real data, overcoming limitations of prior methods like heuristic LLM-generated text or unreadable embeddings from dataset distillation.\n\nThe main findings show GRADMM’s effectiveness in two scenarios: generating substantial synthetic training data from a small set of real examples, and replacing larger real datasets with a compact synthetic set that maintains privacy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the novelty of our work and our thorough experiments.\n\n----\n\n1. Unverified assumption $\\xi \\leq |g_t|$ in theory.\n\nThe following [figures](https://anonymous.4open.science/r/gradmm/grad_diff/) confirm the validity of our theoretical assumption, by showing that the gradient error, i.e. $\\| \\nabla \\mathcal{L}(\\theta_t) - \\nabla \\mathcal{L}^s(\\theta_t) \\| \\leq \\| \\nabla \\mathcal{L}(\\theta_t) \\| = \\| g_t \\|$ at the pretrained parameters and this relation indeed holds during fine-tuning. Crucially, the data generated by GRADMM has a much smaller gradient error compared to the zero-shot baseline during fine-tuning, which is the reason for its superior performance.", "claims_and_evidence": "C1: GRADMM-generated synthetic text guarantees convergence to a close neighborhood of the real data fine-tuning solution\nTheoretical analysis is provided in Section 4.5 and Appendix A, including Lemma 4.1, Theorem 4.2, and Corollary 4.3\nExperimental results (Section 5.2, Table 1) show that on datasets like SST-2, fine-tuning with synthetic data achieves accuracy (e.g., 90.0%) close to or exceeding real data baselines (e.g., Random 1K at 91.2%).\n\n\nC2: GRADMM-generated synthetic text is human-readable\nThe method uses top-k projection (Section 4.2) to map embeddings to vocabulary token sequences, with perplexity (ppl) as a readability metric. Table 1 shows synthetic data ppl (e.g., 5.2-5.8) close to real data (6.6-7.7), compared to 13.3 without top-k (Table 3).\nQualitative results (Figure 2 and Appendix C) provide examples like “Great movie review is a must see experience...” (positive) and “Terribly bad and boring to me...” (negative).\n\nC3: GRADMM outperforms existing methods in performance\nI'm not satisfied about the selected baselines, but include the random gold data make it acceptable.", "methods_and_evaluation_criteria": "Its evaluation criteria—classification accuracy, perplexity, comparisons with baselines (zero-shot/few-shot LLM generation, coreset methods), and benchmark datasets (SST-2, Tweet Emotions, Rotten Tomatoes)—tested across Phi, Llama-3.2-1B, and OPT-1.3B, appropriately measure performance, readability, and competitiveness. These methods and criteria align well with the problem.", "theoretical_claims": "The proof for Theorem 4.2 is mostly correct mathematically but hinges on an unverified assumption $\\xi \\leq \\|\\mathbf{g}_t\\|$", "experimental_designs_or_analyses": "I think the experiment is very thorough.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This submission addresses how to generate readable synthetic text sequences that are either a distilled version of a much larger dataset that retains performance, or a larger version of a smaller dataset that helps improve performance. The key idea is to match the gradients of the loss on synthetic examples to that on the real text, and to make this tractable by working in the embedding spaces and enforcing the embeddings to be from the vocabulary. Additional tricks (top-k decoding, filtering and matching only the last-layer gradients) were used. Overall, the empirical evaluation is thorough and demonstrates well the utility of the proposed approach.

The reviewers (except one) did not engage in the rebuttal and post-rebuttal discussion, unfortunately. The AC noted that most questions/concerns have been addressed. The AC also read the paper and found the approach to be sound, experimentally validated and of interest to a large community (working on optimisation, coresets, synthetic data), thus recommended acceptance.
Decision: Accept (poster)