Synthesizing Privacy-Preserving Text Data via Finetuning *without* Finetuning Billion-Scale LLMs
Authors: Bowen Tan
Zheng Xu
Eric P. Xing
Zhiting Hu
Shanshan Wu
Venue: ICML 2025 poster (2025)
Keywords: synthetic data
language model
differential privacy
Primary area: applications->language_speech_and_dialog
TL;DR: We propose a novel framework for generating privacy-preserving synthetic data without extensive prompt engineering or billion-scale LLM finetuning.

Abstract: Synthetic data offers a promising path to train models while preserving data privacy. Differentially private (DP) finetuning of large language models (LLMs) as data generator is effective, but is impractical when computation resources are limited. Meanwhile, prompt-based methods such as private evolution depend heavily on the manual prompts, and ineffectively use private information in their iterative data selection process. To overcome these limitations, we propose CTCL (Data Synthesis with **C**on**T**rollability and **CL**ustering), a novel framework for generating privacy-preserving synthetic data without extensive prompt engineering or billion-scale LLM finetuning. CTCL pretrains a lightweight 140M conditional generator and a  clustering-based topic model on large-scale public data. To further adapt to the private domain, the generator is DP finetuned on private data for fine-grained textual information, while the topic model extracts a DP histogram representing distributional information. The DP generator then samples according to the DP histogram to synthesize a desired number of data examples. Evaluation across five diverse domains demonstrates the effectiveness of our framework, particularly in the strong privacy regime. Systematic ablation validates the design of each framework component and highlights the scalability of our approach.

Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMs
Bowen Tan Zheng Xu²
Eric Xing 13 Zhiting Hu
Shanshan Wu²
Abstract
ple, models are improved by adapting to the typing text in user's mobile virtual keyboard (Hard et al., 2018: 10 Xu et al., 2023), and aligning with user preference in a chatbot (Ope- nAI, 2024; Google, 2024; Llama Team, 2024). However, training models on user data raises privacy concerns, par- ticularly domains involving highly sensitive information, such healthcare records (Milmo & Stacey, 2025) and chat messages (Silva, 2025). Researchers have shown that the training data can be memorized and potentially extracted from models (Carlini et al., 2021; Nasr et al., 2023; Carlini et al., 2023). Synthesizing emerged as promising approach to mitigating these pri- vacy risks. A popular approach is to differentially-priven (DP) finetune generative language model (LM) on user data, followed by generating synthetic data using the fine- tuned model (Bommasani et al., 2019; Putta et al., 2022; Mattern et al., 2022a; Yue et al., 2023). Benefiting from the development of open-sourced billion-scale large language models (LLMs) such as Llama (Touvron et al., 2023), DP- finetuned generators have demonstrated effectiveness the downstream classification (Kurakin al., 2023) and instruc- tion tuming tasks (Yu al., 2024). However, DP finetuning is both computationally expensive and resource-intensive because it requires per-sample gradient operations in every training batch and large batch size to get good privacy-utility trade-off (Ponomareva et al., 2023). This results in higher memory usage and slower training speeds compared the non-DP finetuning. Moreover, when the user data are de- centralized across their own devices, and centralized data collection is allowed following the data minimization privacy principle (McMahan et al., 2017; Kairouz et al., 2021; Daly et al., 2024), the devices performing local com- putations typically lack the necessary resources to finetune billion-scale LLMs.
Synthetic data offers promising path to train models while preserving data privacy. Differen- tially private (DP) finetuning of large language models (LLMs) as data generator is effective. but is impractical when computation resources are limited. Meanwhile, prompt-base based methods such as private evolution (Xie et al., 2024; Hou et al., 2024) depend heavily on the manual prompts, and ineffectively use private information in their iter- ative data selection process. To overcome these limitations, we propose CTCL (Data Synthesis with ConTrollabilliy and CLustering), novel framework for generating privacy-preserving syn- thetic data without extensive prompt engineering or billion-scale LLM finetuning. CTCL pretrains a lightweight 140M conditional generator and a clustering-basee topic model on large-scale pub- lic data. To further adapt to the private domain, the generator is DP finetuned on private data for fine-grained textual information, while the topic model extracts DP histogram representing distri- butional information. The DP generator then sam- ples according to the DP histogram to synthesize a desired number of data examples. Evaluation across five diverse domains demonstrates the ef- fectiveness of our framework, particularly the strong privacy regime. Systematic ablation vali- dates the design of each framework component and highlights the scalability of our approach.
user data has
1. Introduction
Many artificial intelligence (AI) applications improves their model performances by leveraging user data. For exam-
To address the resource limitations, recent work has ex- plored generating synthetic data that only require LLM API access, exemplified by the Private Evolution (PE) frame- work (Lin et al., 2024; Xie et al., 2024; Hou et al., 2024). These methods use an iterative process where samples are drawn from the LLMS using human-crafted prompts, and then filtered based on their similarity to the private data. This line of work has several limitations. First, they require prompt writers to have deep domain knowledge of the pri-
Work done during an internship at Google.
CCamegie Mellon University *Google Research 'Mohamed bin Zayed University of Artificial Intelligence "UC San Diego Correspondence to: Bowen Tan, Shanshan Wu <btan2 andrew.cmu.cde shanshanw @gwoole..omm...........
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Synthesizing Privacy-Pree Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMs
A. Developing
and
once and for all with public corpora
Aspests
10 keywords for each topic
2B LLM
Pretrain
Embed
Clustering
Pretrain Docs
Extracted Aspects (Keywords, Doc type, ... for every its condition
CTCe-Generator (140M)
Wikipedia
CTCL-Topic
Learning the Private Domain with
C. Synthesizing Data with
1
High-feve! Topic-wise info
Private Data
Identify Topics
(Keywords, Document) Pairs
Topic Histogram
Generate
CTCL-Generator
Synthetic Data
Fine gra-igraine Textual Info
(Finetuned)
No billion-scale DP Training
(Keywords, Document) Pairs
Finetune
CTCL- TeC--Generttor
No domein-speciffc prompt engineering
(Finetuned)
No constraints synthetic data size
Figure 1. Overview of CTCL (Data Synthesis with Con Trollabbbyyy and CLustering) framework: (A) A universal topic model and lightweight 140M generator with strong controllability are developed once and for all on large-scale public corpora ($3.1 and 83.2); (B) To learn the private domain, we collect DP topic histogram, and DP finetune the generator on the private data (33.3); (C) Privacy-preserving synthetic data is generated based on the topic histogram and the finetuned generator ($3.4).
vate data, requirement that can be unrealistic across diverse scenarios. They also heavily rely on the LLM's creativity and extensive prompt engineering tailored to the specific LLMs. More critically, the PE framework only uses the pri- vate data in the embedding space for example selection and filtration, and fails fully leverage the fine-grained word- level information. This inherently limits the performance of the synthetic data in the downstream tasks, particularly the challenging generative tasks. As we will show in our experiments, unlike the standard classification tasks, these generative tasks are evaluated by next-word prediction ac- curacy, and hence, demand finer- aeeen-g approximation of the private data distributions.
privacy costs, because of the post- proseprocessing property of DP (Dwork et al., 2014).
We validate our framework across five diverse downstream domains, including the medical contexts and human-to- human conversations, covering both generative tasks evalu- ated by the next-word prediction accuracy, and the standard classification tasks. Our framework demonstrates significant ad advantages over previous approaches, particularly under strict privacy constraints. Through a comprehensive analy- sis, we highlight the importance of each component in our design, and demonstrate the scalability of CTCL. compared to prompting- based methods such PE (Xie al., 2024).
In this work, introduce CTCL (Data Synthesis with Con ConTrollability and CLustering), a novel framework for data without fine-
2. Related Work
Differential Privacy (DP) Our operations the private data adhere to the standard (e, 6)-DP guarantee (Dwork et al., 2006), ensuring that the inclusion or exclusion of single record has minimal impact on the algorithm's 's out- put. This constraint limits the model to learning generaliz- able patterns rather than memorizing individual data points. Specifically, we employ DP-Adam (Li et al., 2022; Yu et al., 2022) for DP finetuning, which clips per-sample gradients and injects Gaussian noise into each gradient update during training. We also add Gaussian noise every bin when collecting DP histogram. For more details about the DP mechanism and the DP paremeters used our experiments, see Appendix and E.
generating synthetic privacy-preservie tuning billion-scale LLMs or domain-specific prompt engi- necring. As illustrated in Figure I, CTCL comprises two key components: lightweight 140M parameter generator and universal topic model. Both components are pre-trained on the large-scale public corpora, SlimPajama (Soboleva et al., 2023) and Wikipedia (Foundation, 2023), respec- tively. When adapting to the private domains, the topic model produces a DP topic histogram to capture high-level distributional information, while the generator DP fine- tuned learn fine- fine-graing grained, textual information. During the data generation phase, the DP-fitntueed generator is pled proportionally for each topic according to the DP topic histogram. An arbitrary amount of synthetic data can be generated by our CTCL-generator without paying additional
*Code available
https://github.com
2

---

Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMs
Prompt for OpenReview on GPT-3.5 (Xie et al., 2024)
Prompt for GBoard Dialogues on PaI.M (Wu et al., 2024a)
Given the area and final decision research paper. you are required to provide a **ectailed and long** review consisting of the following content:
Imagine you are a female at age 23. You are using the Android Messages APP to message your family on your mobile phone on the afternoon of a vacation day. You want to chat about the following topie: I can't 't wait come home and tell you all about it. Generate the conversation between you and your message receiver.
1. briefly summarizing the paper in 3-5 sentences;
2. listing the strengths and weaknesses of the paper in details;
3. briefly summarizing the review in 3-5 sentences.
Prompt used in our pretraining data construction on Gemma-2-22 ($3.1)
Describe this document in multiple aspects. Make sure "Document Type" and "Keywords" are two of the aspects. (document)
Table The prompts used in existing synthetic data approaches versus our pretraining data construction. Prompts existing work usually requires in-depth domain knowledge and intensive prompt engineering specific to dataset and the LLM being prompted, while the one used our data construction is simple and generally applicable on whatever types of documents in pretraining corpus.
Synthetic Data via DP Finetuning of LMs This line of work DP finetunes an LM on the private data, and the fine- tuned LM is then used to generate synthetic data (Bom- masani et al., 2019; Putta et al., 2022; Mattern et al., 2022a; Yue et al., 2023; Kurakin et al., 2023; Yu et al., 2024; Wang et al., 2024; Ochs& & Habernal, 2024; Carranza et al., 2024). To preserve model capability under the DP training noise, these approaches often rely billion-scale models, particu- larly for generative tasks. For instance, Yu et al. (2024) fine- tune LLaMA-7B (Touvron et al., 2023) with DP to generate short (usually single-sentence) maachiee instruc- tions. In contrast, our framework incorporates a carefully designed learning process on the private data while using significantly smaller backbone LM with only 140M pa- rameters in DP finetuning. This substantially reduces the computational costs, making the approach more feasible for real-world consrrcecnnstriiee applications.
cantly greater scalability compared to the PE approach. We discuss more related work including private inference in Appendix B.
3. CTCL Framework
In this work, we propose CTCL (Data Synthesis with ConTrollability and CLustering), framework for generat- ing synthetic private data without requiring billion-scale DP finetuning or domain-specift prompt engineering. Figure I and Appendix H give an overview of CTCL..
Our framework consists of two key components: CTCL- Generator and CTCL-Topic. Both are developed only once using the large-scale public corpora. CTCI-Genneator is lightweight 140M-parameter conditional generator that supports free-form text input, allowing users to specify at- tributes such as keywords and document type ($3.1). The second component, CTCL--Topcc, is a topic model that cat- egorizes given document into a predefined topic, repre- sented by ten keywords ($3.2). To use these two compo- nents for learning specific private domain: the topic model constructs topic-wise histogram to capture high-level dis- tributional information, while the generator is DP finetuned on private training data to retain low-level textual details ($3.3). After that, we use the DP finetuned generator and the DP topic histogram to produce an arbitrary number of synthetic samples without additional privacy costs ($3.4).
Synthetic Data via LLM API Prompting This line of research explores data synthesis using only LLM inference APIs, typically leveraging prompt engineering with domain- specific knowledge, such as specifying document structures or assuming roles (Wu et al., 2024a; Zhang et al., 2025). The Private Evolution (PE) framework (Lin et al., 2024; Xie et al., 2024; Hou et al., 2024) integrates the private informa- tion into the synthetic data through an iterative sample se- lection process. Specifically, the APl-generated outputs are selected based their proximity private data measured by the differentially private nearest neighbors (DP-NN). In this setup, DP-NN serves as the sole mechanism for extract- ing information from the private data, limiting the extent to which its information is fully captured. Furthermore, the synthetic data size (which determines the number bins in the DP-NN histogram) is often constrained in order to better tolerate the DP noise (see discussion in Appendix B). For instance, the synthetic datasets in (Xie et al., 2024) contain typically 2,000 to 5,000 examples across the experiments. Unlike the grompting-based methods, our framework does not require prompt engineering and prior domain knowl- edge when applied to downstream data. Additionally, our synthetic dataset size is not constrained, offering signifi-
The design of our framework offers several advantages. First, compared with the existing billion-scale LLM DP finetuning, our backbone LM contains only 140M parame- ters, making DP finetuning practical for real-world resource- constrained applications. Second, unlike the prompting- based approaches that depend on hand-crafted domain- specific prompts that require in-depth expertise, our frame- work is applicable to any private domain regardless of prior domain knowledge. Third, PE- methods need to bal- ance between data quality and synthetic data size (see discus- sions in Appendix B), while our framework naturally allows for unlimited data samples using the DP finetuned generator,
3

---

Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMs
Document
example of the generated document description. Notably, our prompt encourages the inclusion of "Document Type" and "Keywords" as aspects in the prompting output. This is designed to match how we use topic keywords to obtain high-level topic distributions, when adapting to a specific private domain ($3.3 and $3.4). Additionally, the document type is encouraged because it the simplest high-level information to extract from the private data domain.
MORGANTOWN, Va. (November 11, 2015) - West Virginia University goll coach Sean Covich announced Wednesday that Ty Olinger (Blacksburg, Va.//North Cross IIS) and Etienne Papineau (St-Jean sur Richelieu, Quebee St-Lawrence) have committed to joining the Mountaineers starting in the fall of 2016. [...]
Extracted Aspects by Gemma-2-2B
Tone:
Informative, positive, celebratory, and official.
Style
Simple, straightforward, and direct.
Keywords West Virginia University, Golf, Recruiting, College
Pretraining Setup We perform continual pretraining on top of BART-base (Lewis, 2019), 140M-parameter requence-to-sequenee LM previously pretrained in an un- supervised manner. The model's encoder-decoder archi- tecture is well suitable for conditioning on inputs through the encoder while generating outputs via the decoder. Opti- mization was performed using the AdamW optimizer with batch size of 4096 and cosine learning rate schedule starting at 5 x 10-5. The implementation of the pretraining is based on RedCoast (Tan et al., 2024) using 6 mixed precision and the pretraining takes approximately 24 hours on 256 TPU-v4 cores (Jouppi et al., 2023).
Purpose
To announce a new recruiting class for WVU golf.
Structure Follows standard journalistic format
Document Type:
Article, Sports News [...]
Table 2. Example of the generated document description. It is used form the condit iin part in our pretraining (condit tinn document) ) data corpus. This task of extracting existing informa- tion from the document doesn't require strong creativity, and hence, can be done by a relatively small LLM. The aspects marked in blue are automatically generated instead of pre-defined.
without additional privacy costs during generation.
3.2. CTCL-TTpic
The remainder of this section provides a detailed explanation of CTCL, covering its components (53.1, $3.2), and the private learning and data synthesis processes ($3.3, $3.4).
Another key component of CTCL is high-quality and di- verse clustering schema: a universal topic model based on document embeddings. This model is designed to identify a topic index for a given document, along with 10 representa- tive keywords associated with the identified topic.
3.1. CTCL-Generator
In our framework, CTCL-Generator is a lightweight (140M- parameter) conditional LM designed for strong controllabil- ity. Specifically, it accepts one or more feature assignments as input, and generates documents that adhere to these spec- ifications. The assignments can include free-text inputs, such as "Document Type: daily dialogue." " To enable this functionality within small LM, we construct large-scale dataset and perform continual pretraining of an unsupervis- edly pretrained LM.
The topic model is used to capture high-level distributional information from the private training data. To ensure uni- versality, the model is designed to generalize well across a wide range of downstream documents, always identifying a relevant topic. To achieve this, we constructed the topic model using Wikipedia (Foundation, 2023), a large-scale, diverse, and commonly recognized high-quality corpus.
Topic Model Setup Specifically, we utilized the Novem- her 2023 version of Wikipedia, which contains over 6 mil- lion pages. A publicly available 20M-parameter document embedding model* was applied to the entire Wikipedia cor- pus. followed by HDBSCAN clustering (McInnes et al., 2017), resulting in the identification of 1 1,300 clusters, each treated as a distinct topic. To represent each topic, we employed the KeyBERT (Sharma & Li. 2019) to annotate 10 keywords for each cluster. The implementation of the pipeline above is based on BERTopic (Grootendorst, 2022).
Pretraining Data Curation We introduce a simple yet ef- fective approach for constructing a large-scale condition-to- document corpus. Our method builds on SlimPajama (Sobol- eva et al., 2023), a large unsupervised pretraining corpus, and leverages relatively small LLM, Gemma-2-2B (Team et al., 2024). Specifically, we employ domain-agnonn agnostic and L.MM-independent prompting strategy for each document in SlimPajama: "Describe the document in multiple aspects." " This document description task is straightforward and not requiring the LLM's creativity, making it well-suited for small LLM like Gemma-2-2B to efficiently handle the data construction process. As result, we generate large- scale pretraining dataset comprising 430M (condi i on, document) ) pairs, where the Gemma-2-2B generated doc- ument description is used as the condit t ion part.
3.3. Learning the Private Domain
When applying CTCL to the downstream private domains (shown by Part B in Figure 1), we use CTCL-Topic to
'tt wttp://tttttttteeeeeaaaaaaaaaaaa
Tables 1 and present the exact prompt we use and an
'https: 'https://maarteng:: http:///aaateggggggi..i//./////a io..ehop..////
4

---

Synthesizing Privacy-Preeerving Text Data via Finetuning without Finetuning Billion-Scale LLMs
PubMed (Medical Paper Abstract)
Setting
X
4
BERT
BERTs Small
BERT Mini
BERT Small
BERT Misi
BERT
BERT Mini
BERT Small
GPT2x.-1.3 1.5B (Upper Bound)
39.6
42.9
37.7
40.5
37.3
40.2
36,8
39.7
GPT2a-11.5B8 -L5B-LLAA (Upper Bound)
39.4
42.5
34.7
37.7
34.9
37.9
34.9
37.9
Downstream DPFT (No Synthetic Data)
44.3
46.0
30.7
34.1
28.9
32.5
26.7
30.4
Private Evolution (PE) (Lin al., 2024)
29.7
31.8
29.6
31.8
29.7
31.9
29.8
31.9
AUG-PE Mixtral-8x7B (Xie al., 2024)
24.9
27.6
24.5
27.1
AUG-PE GPT-3.5 (Xie et al., 2024)
30.4
32.7
30.3
32.5
30.2
32.5
30.1
32.4
GPT2smal (Yue ct al., 2023)
38.1
41.6
35.0
37.4
32.0
34.4
26.8
29.3
GPT2 + Resample (Yu al., 2024)
39.0
42.4
35.3
37.5
33.0
35.1
27.6
29.1
(Yue BART
2023)
40.9
43.9
30.5
32.4
28.9
30.8
26.7
28.5
BART Resample (Yu 2024)
44.2
30.7
32.5
29.0
30.7
26.5
28.0
Ours
41.5
44.6
35,9
38.1
35.4
37.6
34.5
36.7
Table Performance of PubMed evaluated by next-word prediction accuracy of downstream models MBRRTMai and BERT
smaller privacy budget (e) corresponds stricter privacy constraint. See $4.1.2 for details of different baselines.
Chatbot Arena (Human-to -o-Machin Instructions)
Multi-Session Chat (Long-Term Human-Human Dialogues)
Next-Word Prediction Accuracy
Next: Next-Word Word Prediction Accuracy
20
30
16
bb
12-
15
10
Downstream DPFT (No Syn. Data)
BARTBas
BARTass Resample CTCL (Ours)
Downstream DPFT GPT2smmh
GPT2sma!! Resample CTCL CTCL(Ours)
(No Syn. Data)
Figure 2. word prediction accuracy of the downstream model BERT Mini in the Chathot Arena Instruction and Multi-Sennt Session Chat domains. Comparing the the blue and yellow bars, our framework demonstrates greater improvements over the baselines under the stricter privacy constraint compared the setting of
capture the high-level distributional information across the entire private corpus (via DP topic histogram), and adapt CTCL-Generator (via DP finetuning) to learn fine-grained text information from the private data.
used to DP finetune the CTCL-Generator. Note that the condi tion part is slightly different between the con- structed pretraining data in $3.1 and private finetuning data here. The pretraining data has free-form text condition (ob- tained from Gemma-2-2B) while the finetuning data has 10 keywords the condition. That said, if available, addi- tional domain-specific knowledge, such document types, can be incorporated into the condition as well. These con- structed citittiion-dcmmentt pairs align with the pretraining condition-to--ocumeen task in $3.1. This alignment is key to benefit from our pretraining, which enables the model to effectively learn private information while being more robust to the noise in training compared vanilla DP fine- tuning (Yue et al., 2023; Kurakin al., 2023).
DP Topic Histogram Using the topic model built by CTCL-Topic, we first assign topic to each document in the private training data, by applying the same 20M document embedding modelt on every document and finding the clos- est topic embedding (among the 1,300 topic embeddings obtained from $3.2). A histogram representing the topic- wise distribution of the private corpus (i.e., the proportion of documents associated with each topic) is then constructed. Gaussian noises are added properly to every bins the histogram, and the result is a private topic histogram.
3.4. Synthetic Data Generation
DP Finetuning After the "DP Topic Histogram" process, each document in the private dataset has been assigned topic. Recall that in CTCL-TTopic, each topic is represented by 10 keywords. Now we transform the private dataset into the (condition, document) pairs, where the condit part consists keywords corresponding to the topic assigned the document. This dataset then
The DP finetuned CTCL- TTCL-Generatto is sampled to generate synthetic data based on the DP topic histogram (see Part C in Figure 1). Specifically, given the desired size of the synthetic dataset (say, N) and the topic proportions specified by the DP topic histogram (say, 2% for Topic 1, 7% for Topic 2, etc), know the number of target samples for
5

---

Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMS
each topic (i.e., NN for Topic 1, yN for Topic 2, etc). For each topic, we use the corresponding 10 keywords input to the DP finetuned CTCL-Gennrator to generate data.
Vanilla DP Finetuning We conduct standard DP fine- tuning (Yue al., 2023) on BART Base (Lewis, 2019) and GPT2small (Radford et al., 2019), both of which have com- parable O(100M) model sizes as that of the generator in our framework. Additionally, we include DP finetuning of GPT2xl-1.55 (Radford et al., 2019) as an upper bound. Given prior findings that LoRA finetuning can outperform full-model finetuning under DP constraints (Kurakin et al., 2023), also evaluate a LoRA DP-finetuned variant of GPT2x1-11.5 as another upper bound. Notably, while LoRA reduces trainable parameters, it does not significantly decrease resource demands since backpropagation is still required through the full backbone LLM.
4. Experiments
4.1. Experimental Setup
4. 4.1.1. DOWNSTREAM TASKS
Our experiments contain three generative tasks and two clas- sification tasks The downstream generative tasks are eval- uated by the next-word prediction accuracy, which needs the synthetic data to preserve fine-graing -ieeggaaddd textual information from the private data. In contrast, the downstream classifi- cation tasks usually rely on co-occurrence patterns between labels and words in the synthetic data. Therefore, generative tasks tend to be more challenging than classification tasks.
Post-Generation Resampling Yu et al. (2024) proposes to refine the synthesizd dataset by resampling technique, in order to better align with statistical properties derived from the private data.
Generative Tasks Three generative downstream tasks are chosen to cover a diverse set of the practical scenarios. Specifically, we include PubMed (Yu al., 2023) to repre- sent the academic medical domain, Chatbot Arena (Zheng et al., 2023) for numan-to-maching interactions, and Multi- Session Chat (Xu, 2021) for tomman-to-hume everyday dialogues. Following the evaluation setup in (Xie et al., 2024), we train 10M-level downstream causal LMs on the synthetic datasets, and use next-word prediction accuracy on the real test data as the primary quality metric.
Private Evolution (PE) We include results from the origi- nal PE (Lin et al., 2024) and its augmented variant, AUG-PE (Xie et al., 2024), as the examplar of L.LM prompting based data synthesis approach.
4. 1.3. HYPERPARA METERS
DP Finetuning and Sample Generation For all settings involving DP finetuning, we use DP-Adam for 2000 steps with a batch size of 4096, a gradient norm clip of 1 1.0, and a weight decay of 0.1. The learning rate follows a linear decay schedule with 100 warmup steps, and the peak learning rate is selected from the range [1, 4] x [10-3, 10-1, 10-5] based on validation performance. The privacy budget accounts for both DP model finetuning and the collection of DP topic histogram statistics. We apply a Gaussian noise multiplier of 10 to the DP topic histogram. The noise multipliers for DP finetuning vary across settings depending on the training data size and the presence of topie histogram 9. For the sample generation process,
Classification Tasks We conduct experiments on two classification tasks: Yelp (Yelp, Inc.) and OpenReview (Xie et al., 2024), both of which are 5-way classification, with Yelp focusing on business reviews and OpenReview on academic paper reviews. The performance is measured by the accuracy of a downstream classifier trained on the synthetic data.
To mitigate concerns regarding data contamination, we use a search engine (Liu et al., 2024) indexed on RedPajama (Computer. 2023) (a superset of our pretraining corpus) to identify potential overlaps between our downstream and pretraining data. Our analysis detects no overlap between our training data and the five downstream datasets. Addi- tionally, for the PubMed dataset, all included samples are dated within August 2023, ensuring they were published after the release of our pretraining corpus in June 2023.
generate 400K synthetic examples using nucleus sampling with top-p = 0.95 and a maximum sequence length of 512 tokens. For upper-bound experiments with GPT2x1 we reduce the batch size to 256 to mitigate computational costs. The implementation of DP finetuning is based on RedCoast (Tan et al., 2024) using full fp32 precision.
Downstream Model Training and Evaluation We fol- low the evaluation of (Xie et al., 2024) for both generative and classification tasks. For generative tasks, we train the causal versions of BERT and BERT Small using a linear learning rate schedule from 0.0003 to O, a batch size of 64, and a total of 6000 steps, with weight decay of 0.01. For classification tasks, we finetune a RoBERTa-bann der the same hyperparameter settings in generative tasks above, except for learning rate of 3 x 10-5.
4.1.2. BASELINES
Direct DP Finetuning Downstream Models A straight- forward approach to obtain a downstream model is to di- rectly perform DP finetuning of the downstream model on the private data, without using the synthetic data. For sim- plicity, we refer to this baseline as "Downstream DPFT" throughout this paper.
model un-
More details of noise multipliers and privacy budget alloca- tion can be found in Appendix E. and D.
Trrininn data sizes can he found Appendix C.
6

---

Synthesizing Privacy- Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMs
Yelp
OpenReview
Setting
6
00
6
6
0o
c
4
c
GPT2xt-LL35 (Upper Bound)
71.1
69.4
68.2
68.2
49.1
46.6
46.3
45,4
G.T2xx-L.5B-Lok (Upper Bound)
70.3
67.7
67.6
67.7
51.0
46.2
45.3
46.0
Downstream DPFT (No Synthetic Data)
76.0
67.5
67.2
66.8
50.8
32.0
32.0
32.0
Private Evolution (PE) (Lin et al., 2024)
67.9
67.1
67.2
67.6
42.4
43.5
43.7
42.9
AUG-PE (Xie et al., 2024)
68.4
68.1
67.8
67.9
43.5
44.6
44.5
43.1
GPT2smati (Yue et al., 2023)
71.0
68.2
67.9
67.9
52.1
41.1
38.5
35.1
BART Base (Yue et al., 2023)
70.7
66.3
66.9
66.9
52.6
44.7
42.2
25.7
Ours
70.5
68.1
68.0
67.7
53.9
46.5
47.1
46.2
Table 4, Accuracy of downstream models in the classification tasks. constraint. See $4.1.2 for details of different baselines.
smaller privacy budget (4) corresponds to a stricter privacy
PubMed
Next-Word Accuracy
PubMed (E = 1)
Yelp (==4)
Next- Next-Word -Wodd Accuracy
68
35
40
30
Accuracy
67
32
25-
*
X
66
28
-R- AUG-PE
20
AUG-PE
x
AUG-PE
Ours
Ours
Ours
15
65
8=1 2=2
E=
2K
10K20K
200K
5K
10K
100K
Privacy Budget (Increasing)
Synthetic Samples
Synthetic Samples
Figure 3. Scalability investigation results. The x-axes represent the increasing privacy budget or the number of synthetic examples, while
the y-axes indicate performance of downstream models trained on synthetic data.
4.2. Results
to effectively capture information the private data. More- over, a comparison different LMs within the AUG-PE framework reveals significant performance gap between GPT-3.5 API and the open-source Mistral-8x7B, despite the latter also being considered strong model. This suggests that the effectiveness of PE methods heavily relies on the exceptional capacity and creativity of the backbone LLM.
4.2.1. GENERATIVE TASKS
Table 3 and Figure 2 present the results of three generative tasks II, Our framework consistently outperforms baselines under different DP constraints and achieves performance close the upper bounds. Moreover, as shown in Figure 2, the performance gap between framework and the baselines widens under tighter privacy constraints (i.e., com- paring the patterns of blue and yellow bars), highlighting its robustness. This can be attributed to our framework's 's abil- ity to simultancously learn both high-level and fine-graine -re-grained information from private data.
4.2.2. CLASSIFICATION TASKS
As shown in Table 4, our model still achieves performance that is either superior to or on par with the best-performing methods. PE-based approaches demonstrate stronger results in classification tasks compared to their performance on generative tasks. This may be because that classification primarily relies on synthetic data capture associations between labels and specific words or phrases, which is an objective that PE methods can effectively achieve by prompt- ing LLMs properly. In contrast, generative tasks require a deeper resemble of finr-grained textual information from private data, which poses greater challenges for PE methods.
Our results also reflect the limitations of the baselines. Specifically, when there is DP constraint, direct down- stream finetuning on the real data (without synthetic data) achieves the best performance across all three tasks. How- ever, adding DP training noise leads substantial performance drop, indicating the vulnerability towards DP noise of small downstream models. Additionally, the performance of PE methods (Xie et al., 2024; Lin al., 2024) remains almost unchanged across different privacy constraints, which also indicates that these methods do not fully exploit the in- creased privacy budget. This limitation may stem from their constrained capacity (i.e., only via the nearest neighbors)
4.3. Analysis and Ablation Study
4.3.1. 1. SCALABILITY
The privacy budget and the size of the generated synthetic data are two key factors influencing the performance of data synthesis. In this study, we examine the effect of these
A complete result table is available in Appendix K.
7

---

Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMS
Setting
= 00
<=
BART-Base (Downstream Model Performance: 30.5%); We explored the relationship between molccular interaction, NCT-2 and NCT-3 (NCT-1). 1). NCT-4, NCT and NCT-1.5 (NCT), NCT-, NCT-6, and NCT[4). In recent clinical trial, we described an enzyme in NCT-10 10 that enabled novel processes to explore novel approaches for NCT- 2.3 to NCT-III. [...]
BART Base
63.1
572.9
BART Buse + Keywords
49.3
291.6
BART Base + Keywords+ + Pretraining (Ours)
48.4
125.6
Table Ablation study results evaluated by the downstream lan- guage model's perplexity (lower values indicate hetter perfor- mance). A privacy budget of = oo means no DP training noise during the finetuning of the data generator.
GPT2xt-11551 (Upper Bound, Downstream Model Perfor- mance: 37.7%) The ability of leptin to induce weight loss, to stimulate ectothermic thermogenesis, and to augment activity of the AMPK system and the AMPK -.PPK-dependent lipoprotein lipase activity, was examined. Circulating concentrations leptin were assessed the femoral adipose fat pad of the lean and ohese [...]
0.4
AUG-PE + GPT-3.5 (Downstream Model Performance: 30.3%) An increasing incidence of aneurysmal subarachnoid hem- orrhage (SAH) remains high, necessitating prompt intervention. The recognition and treatment options, including both surgical and endovascular approaches, have emerged as key components of tertiary management. [...]
MAUVE Score
0.3
0.2
Ours (Downstream Model Performance: 35.9%): To develop a therapeutic formula to reduce rates of morbidity that occur in people with a combination of cardiovascular problems. We used multi-state, multidisciplinary approach to the research of the clinical manifestations of cardiovascular problems with the intro- duction of a biocontrol. [...]
0.1
GPT2small -P52x-11.5B AUG-PE
Ours
Figure 4. MAUVE scores PubMed dataset under 4. Note that only the relative rankings instead of the absolute scores matters here, because the score scale can change a lot with slightly different evaluation configurations (e.g., use a different embedding model).
Table 6 Synthetic data samples PubMed under 4. Ran- domly Sampled. Obvious disfluent cuts are highlighted in red.
factors, focusing on a comparative analysis between our framework and AUG-PE, an exemplar prompting- trappinn--based ap- proach. To investigate the impact of synthetic data size, we follow the experimental setup of (Xie et al., 2024) and extend AUG-PE's sample sizes to 200K for the PubMed dataset and 100K for the Yelp dataset. The PubMed expan- sion is achieved by combining two runs of data synthesis using GPT-3.5 and -aama-3--b-Instruct while the Yelp ex- pansion uses Garge as reported in (Xie et al., 2024).
tuning. Specifically, starting from standard DP finetuning, we sequentially introduce these components and measure the downstream model's perplexity. The results, presented in Table 5, demonstrate the following: first, a comparison between "BART Rose and "BART Base + Keywords" reveals that incorporating keywords during finetuning significantly improves performance, regardless of the presence of DP training noise. Second, comparison between "BART Base + Keywords" and "BART Base + Keywords + Pretraining" in- dicates that pretraining offers limited benefits in noise-free settings but provides a clear advantage when the DP training noise added.
Regarding privacy budget scalability, as illustrated in the leftmost plot of Figure 3 and briefly discussed in $4.2.1, AUG-PE does not benefit from an increased privacy bud- get, whereas framework continues to improve under the same conditions. For synthetic data size, the second and third plots in Figure 3 show that when the number of synthetic examples remains in the thousand-level range, AUG-PE produces higher-quality datasets. However, its per- formance plateaus beyond 10K examples. In contrast, our framework exhibits continuous improvement as the dataset size increases. These findings align with our discussion in $2 and Appendix B on the size limitations of PE method.
We also compute the MAUVE score (Pillutla et al., 2021) to measure the distribution similarity between the gener- ated synthetic data and the PubMed test set. As shown in Figure 4, our method achieves the highest MAUVE score among the compared methods, showing the effectiveness of our topic-wise distribution aligmment during the generation process (53.1). Moreover, a comparison between GPT2x1.- 1.5B and GPT2small reveals that DP finetuning on larger model better captures high-level distributional patterns. Fur- thermore, we find that the high-level distribution similarity measured by MAUVE is not the sole determinant of syn- thetic data quality. For instance, while the synthetic data from GPT2xL- has a lower MAUVE score than that of our approach, the model trained on it achieves higher downstream performance (37.7% vs. 35.9%) in Table 3.
Overall, our approach demonstrates superior scalability in terms of both privacy budget and synthetic data size.
4.3.2. ABLATION STUDY
In this study, we validate the importance of two key com- ponents in our framework: 1) pretraining the generator and 2) incorporating keyword-based conditions during DP fine-
8

---

Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMs
4.3.3. SYNTHETIC SAMPLES
References
Table 6 presents synthetic samples generated by our frame- work and several baselines. Under the DP training noise, the BART Base model tends to produce repetitive content. In con- trast, our framework, built on the same lightweight model architecture, maintains the sentence fluency well. Interest- ingly, while the AUG-PE method generates fluent sentences using the powerful GPT-3.5, its downstream performance is only comparable to that of the DP-finetuned BART Base. This suggests that in the context of data synthesis, the quality of the surface form (e.g., fluency and coherence) may not be the most critical factor. Generating synthetic data that is useful for the downstream model development is more important than generating fluent data.
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communiccations Security, ACM, 2016. doi: 10. 00.114529976744.2978816 URL http:/ /xxp/xxxx////////e dox.doi..oo org/orgg///// 114...11791/9/999999 2978318.
Amin, K., Bie, A., Kong, W., Kurakin, A., Ponomareva, N., Syed, U., Terzis, A.. and Vassilvitskii, S. Pri- vate prediction for large-scale synthetic text genera- tion. In Al-Onaizan, Y., Bansal, M., and Chen, Y.- N. (eds.), Findings of the Association for Computa- tional Linguistics: EMNLP 2024, Miami, Florida, USA, November 2024. Association for Computational Linggsi- tics. URL https: /at///aaaanttaooa org.or////00 2024. f Eindings-emn1p 425/.
5. Conclusion
In this work, propose novel framework CTCL for syn- thesizing private domain data, which integrates a universal topic model with a lightweight 140M conditional language model. This framework captures both high-level, topic- specific information and fine-graine graineraidd context-sensitive de- tails of the private domain in modular and efficient man- ner. Through evaluations across five diverse downstream domains, we demonstrate that the synthesized data gener- ated by CTCL outperforms baseline methods, including vanilla differential privacy finetuning and prompting- based approaches such as private evolution.
Bassily, R., Smith, A., and Thakurta, A. Differentially private empirical risk minimization: Efficient algorithms and tight error bounds. 2014. URL https: / /arxiv. org/ abs / 140// 7085.
Bommasani, R., Wu, Z. S., and Schofield, A. Towards pri- vate synthetic text generation. In NeurlPS 2019 Machine Learning with Guarantees Workshop, 2019.
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert- Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. Extracting training data from large language models. In 30th USENIX Security Symposium 2021.
While the proposed CTCL framework outperforms existing methods in the downstream tasks, one limitation is that the synthetic data generated by our 140M- parameter still lack the fluency and coherence compared to those generated by prompting the latest billion-size LLMs. How to leverage LLMs for data synthesis in the resource- constrained setting (the setting which this paper focuses on), would be one important area for future exploration.
(USENIX Security 21), Pp.
Carlini, N., Hayes, J., Nasr, M., Jagielski, M., Sehwag, V., Tramer, F., Balle, B., Ippolito, D., and Wallace, E. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 5253-5270 2023.
Another interesting direction for future work is to general- ize our framework the multi-modal settings, e.g., data synthesis for the vision-language tasks. This requires appro- priate architectural adaptations, e.g., by replacing language models with diffusion- based models for image generation.
Carranza, A., Farahani, R., Ponomareva, N., Kurakin, A., Jagielski, M., and Nasr, M. Synthetic query generation for privacy-preserving deep retrieval systems using dif- ferentially private language models. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Con- ference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 3920-3930, Mexico City, Mexico, June 2024. Association for Compu- tational Linguistics. doi: 10.18653//1/22224.maas 217. URL maacl-1ong.22
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Carvalho, R. S., Vasiloudis, T., and Feyisetan, O. Tem: High utility metric differential privacy on text. In Pro- ceedings of the 2023 SIAM International Conference on Data Mining (SDM), pp. 883.890. SIAM, 2023.

---

Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMs
Computer, T. Redpajama: An open source recipe to reproduce Ilama training dataset, 2023. URL
Hard, A., Rao, K., Mathews, R., Ramaswamy, S., Beaufays,
F., Augenstein, S., Hichner, H., Kiddon, C., and Ramage,
https: http::///ithub..ooo httpahuccc/c/oogeeeerrmprpeeee RedPa jaaa- aaaaa.
D. Federated learning for mobile keyboard prediction.
arXiv preprint (ate::111100000 2018.
Daly, K., Eichner, H., Kairouz, P., McMahan, H. B., Ram- age, D., and Xu, Z. Federated learning in practice: re- flections and projections. In 2024 IEEE 6th International Conference on Trust, Privacy and Security in Intelligent Systems, and Applications (TPS-ISA), pp. 148-156. IEEE, 2024.
Hou, C., Shrivastava, A., Zhan, H., Conway, R., Le, T., Sagar, A., Fanti, G., and Lazar, D. Pre-text: Training language models on private federated data in the age of llms. ICML, 2024.
Hou, C., Wang, M.-Y., Zhu, Y., Lazar, D., and Fanti, G. Private federated learning using preference-optimizee synthetic data. arXiv preprint arXiv:2504 164338 2025.
DeSalvo, G., Kagy, J.-F., Karydas, L.,, Rostamizadeh, A., and Kumar, S. Softsrv: Learn to generate targeted syn- thetic data. In Will Synthetic Data Finally Solve the Data Access Problem? Workshop at ICLR 2025, 2025.
Jouppi, N., Kurian, G., Li, S., Ma, P., Nagarajan, R., Nai,
L., Patil, N., Subramanian, S., Swing, A., Towles, B., Young, C., Zhou, X., Zhou, Z., and Patterson, D. A. Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for em- beddings. In Proceedings of the 50th Annual Inter- national Symposium Computer Architecture, ISCA '23, New York. NY, USA, 2023, Association for Com- puting Machinery. ISBN 9798400700958.
DP Team. 2022.
Google's differential privacy libraries., /ttp::////ttubbbub wtw.hhubcooooococcoooomoooooo.oe google/ di differential-priieey
Duan, H., Dziedzic, A., Papernot, N., and Boenisch, F. Flocks of stochastic parrots: Differentially private prompt learning for large language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https::/ /ttp:////ppeeeeieeeeeee net forum? id=u6Xv3FuF8N.
doi: 10. I 1145/3579371 URL htttp:://oi.orrs 10. 10.1145/3579377 0tt.:/5577771.358833333 3589350.
Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14(1-2):1-210, 2021.
Dwork, C., Kenthapadi, K., McSherry, F., Mironov, I., and Naor, M. Our data, ourselves: Privacy via distributed noise generation. In Annual International Conference on the Theory and Applications of Cryptographic Tech- niques, 2006.
Kurakin, A., Ponomareva, N., Syed, U., MacDermed, L., and Terzis, A. Harnessing large-language mod- els to generate private synthetic text. arXiv preprint DXXiv:33060016: 2023.
Dwork, C., Roth, A., et al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 93:44):21-440 2014.
Lewis, M. Bart: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension. arXiv preprint arXiv: 13461, 2019.
Feyisetan, O., Balle, B., Drake, T., and Diethe, T. Privacy- and pellity-preserving textual analysis via calibrated mul- tivariate perturbations. In Proceedings of the 13th Inter- national Conference on Web Search and Data Mining, WSDM '20, pp. 178-186, 2020.
Li, X., Tramèr, F., Liang, P., and Hashimoto, T. Large lan- guage models can be strong differentially private learn- ers. In International Conference on Machine Learning (ICML), 2022.
Foundation, W. Wikimedia downloads, 2023, URL https: //dumps. wikiiediiii org.
Lin, Z., Gopi, S., Kulkarni, J., Nori, H., and Yekhanin, S. Differentially private synthetic data via foundation model apis Images. ICLR, 2024.
Gao, T. Wettig, A., He, L., Dong, Y., Malladi, S., and Chen, D. Metadata conditioning accelerates language model pre-training. arXiv preprint arXiv:2501.0199 2025.
Liu, J., Min, S., Zettlemoyer, L., Choi, Y., and Hajishirzi, H. Infini- gram: Scaling unbounded n-gram language models to trillion tokens. arXiv preprint arXiv:2401.17377 2024.
Google, Gemini: A family of highly capable mul- timodal models, 2024. URL
abs/2312 11805.
Llama Team, M. models, 2024. meta. com/ the-1lama- -f--mma-----dd-------ooo-
The Ilama 3 herd of
Grootendorst, M. Bertopic: Neural topic modeling with a based tf-idf procedure. arXiv preprint arXiv:2203.00 055944, 2022.
URL https:/ /ai.
10

---

Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMs
Mattern, J., Jin, Z., Weggenmann, B., Schoelkopf, B., and Sachan, M. Differentillly private language models for secure data sharing, In Conference Empirical Methods in Natural Language Processing (EMNLP),
machine learning with differential privacy. Journal of Artificial Intelligence Research, 77, 2023. ISSN 1076- 9757. doi: 10.1613/jair. 1.14649, URL http:/ //dx.
Noi.org/10.111 1613/ jair ...r.l.......................... 1...1.4....
Mattern, J., Weggenmann, B., and Kerschbaum, F. The limits of word level differential privacy. In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V. (eds.), Findings of the Association for Computational Lin- guistics: NAACL 2022, pp. 867-881, Seattle, United States, July 2022b. Association for Computational Linguistics. doi: 10. 10.18653/v1/2222.fmddig
Putta, P., Stecle, A., and Ferrara, J. W. Differentially private conditional text generation for synthetic data production. 2022. URL https: http:///ppppeeeeeeeee wet.tiiwwweeeeeeee........e..eee htt.:/eeww.eeeoonnt forum?
id=10g133000 3FOF
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
65. URL https: / //c//caaantt wwwllnnthlooyyo..oooooo org.:/y...rg2220 Sandings-naacl-.. 6.//
Sharma, and Li, Y. Self-supervised contextual keyword and keyphrase retrieval with self-labelling. 2019.
McInnes, L., Healy, J., Astels, S., al. hdbscan: Hierar- chical density based clustering. J. Open Source Softw., 2 (11):205, 2017.
Silva, d. J. Linkedin accused of using private messages to train ai. BBC, 2025. URL https: //t:://wwwww wwww .bc. com/ ntw./eww///ae/eeeee wtwws/rrticlen http////eeeexxxyxxeeoeoo Accessed: 2025-01-29.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. Communication--ffiien learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017.
Soboleva, D., Al-Khateeb, F., Myers, R., Steeves, J. R., Hestness, J., and Dey, N. SlimPajama: A 627B token cleaned and deduplicated version of RedPa- jama, 2023, URL https: https: https:://uggingfacee co/ datasets/ accasett//cerebrrr/SSiiiee
Milmo, D. and Stacey, K. What does ai plan mean for nhs patient data and is there cause for cern? The Guardian, 2025. URL https://www. theguardian pttpuarrdaanncomoooooooo. http:/ooooo...c/////////////////
667B.
www.-aaciioaaaaa---------------- Accessed: 2025-01-29.
labour-ai--e antt--aiiaciionn ion pwapppaaan-- nhs wattss-pptieetttttteeeee
Song. S., S. Ri Sarwate, A. D. Stochastic gradient descent with differentially private updates. In 2013 IEEE Global Conference Signal and Information Processing, pp. 245-248, 2013.
Nagesh, S., Chen, J. Y., Mishra, N., and Wagner, T. Private text generation by seeding large language model prompts. In GenAl for Health: Potential, Trust and Policy Com- pliance, 2024. URL https: / httpp/pwwneeeeeeee../ forum? dd=mm_200rwww2
Tan, B., Zhu, Y., Liu, L., Wang, H., Zhuang, Y., Chen, J., Xing, E., and Hu, Z. RedCoast: A lightweight tool to automate distributed training of LLMs on any GPU/TPUs. In Proceedings of the 2024 Conference of the North Amer- ican Chapter of the Association for Computational Lin- guistics: Human Language Technologies (Volume 3: Sys- tem Demonsirations), pp. 137-147, Mexico City. Mexico, June 2024. Association for Computational Linguistics. doi: 2...8663//1122222.maadddd 14. URL https:/ aclanthology oro.or//////// 2024. www.::............m............. -eeaee---demm.mmmm 14/.
Nasr, M., Carlini, N., Hayase, J., Jagielski, M., Cooper, A. F., Ippolito, D., Choquette- Choo, C. A., Wallace, E., Tramèr, F., and Lee, K. Scalable extraction of training data from (production) language models, arXiv preprint arXiv:2311. 2023.
Ochs, S. and Habernal, I. Private synthetic text generation with diffusion models. 2024. URL https: //axxiv.. org/abs hts//bbs/24000 2499722971.....
Tang, X., Shin, R., Inan, H. A., Manoel, A., Mireshghallah, F,, Lin, Z., Gopi, S., Kulkarni, J., and Sim, R. Privacy- preserving in-context learning with differentially private few-shot generation. In The Twelfth International Confer- ence Learning Representations, 2024. URL https:
OpenAI. Gpt-4 technical report, 2024. URL https:// arxiv. www.oogg///////eee 22t.b.///333..00 08774.
Pillutla, K., Swayamdipta, S., Zellers, R., Thickstun, J., Welleck, S., Choi, Y., and Harchaoui, Z. Mauve: Mea- suring the gap between neural text and human text using divergence frontiers. In NeurlPS, 2021.
i /poenreeveww.. net
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00 00/000//8/ 2024.
Ponomareva, N., Hazimeh, H., Kurakin, A., Xu, Z., Denison, C., McMahan, H. B., Vassilvitskii, S., Chien, S., and Thakurta, A. G. How to dp-fy ml: A practical guide to

---

Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMs
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.139777 2023.
Yelp, Inc. Yelp Open Dataset. https:
/ //business yelp. www.cooo/caa// resttrcesssrrce///// open dataset/.
Yu, D., Naik, S., Backurs, A., Gopi, S., Inan, H. A., Kamath, G., Kulkarni, J., Lee, Y. T.,, Manoel, A., Wutschitz, L., Yekhanin, S., and Zhang, H. Differentially private fine- tuning of language models. In International Conference on Learning Representations (ICLR), 2022.
Utpala, S., Hooker, S., and Chen, P.-Y. Locally differ- entially private document generation using zero shot prompting. In Bouamor, H., Pino, J., and Bali, K. (eds.). Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 8442-8457, Singa- pore, December 2023. Association for Computational Linguistics. doi:
Yu, D., Backurs, A., Gopi. S., Inan, H., Kulkarni, J., Lin, Z., Xie, C., Zhang, H., and Zhang, W. Training private and efficient language models with synthetic data from llms. In Socially Responsible Language Modelling Research, 2023.
566. URL htwpt:ttp/// peattts://aclaan /wwpaatooannoooo.o.oooo org/2023. Eindings-emnip..55 566/.
Wang, W., Liang, X., Ye, R., Chai, J., Chen, S., and Wang, Y. KnowledgeSG: Privacy-prestt prreeeriin synthetic text gen- eration with knowledge distillation from server. In Al- Onaizan, Y., Bansal, M., and Chen, Y.N.. (eds.), Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 7677-7695, Miami, Florida, USA, November 2024. Association for Compu- tational Linguistics. doi: 10.18663/ main. 438. URL https: : wwwlaaannalggooooooooooooooooloe emn p co.main.... 438/.
Yu, D., Kairouz, P., Oh, S., and Xu, Z. Privacy-preserving instructions for aligning large language models. ICML, 2024.
Yue, X., Inan, H. A., Li, X., Kumar, G., McAnallen, J., Shajari, H., Sun, H., Levitan, D., and Sim, R. Synthetic text generation with differential privacy: A simple and practical recipe. ACI., 2023.
Zhang, Y., Xu, Z., Wu, S., Zhang, Y., and Ramage, D. Synthesizing and adapting error correction data for mo- bile large language model applications. arXiv preprint arXiv: 18488, 2025.
Wu, S., Xu, Z., Zhang, Y., Zhang, Y., and Ramage, D. Prompt public large language models to synthesize data for private on-device applications. COLM, 2024a.
Zhao, W., Song. S., and Zhou, C. Generate synthetic text approximating the private distribution with differential privacy. In Larson, K. (ed.), Proceedings of the Thirty- Third International Joint Conference on Artificial Intel- ligence. IJCAI-24, pp. 6651-6659. International Joint Conferences on Artificial Intelligence Organization, 8 2024. doi: 10.24963/ijcai URL https: 2024 Main / //doi. wtt.or/g/......... /t144499333/aa12222//33333
Wu, T. Panda, A., Wang, J. T., and Mittal, P. Privacy- preserving in-context learning for large language models. In The Twelfih International Conference on Learning Rep- resentations, 2024b. URL https: openreview. net/
Xie, C., Lin, Z., Backurs, A., Gopi, S., Yu, D., Inan, H. A., Nori, H., Jiang, H., Zhang, H., Lee, Y. T., et al. Differen- tially private synthetic data via foundation model apis 2: Text. ICML, 2024.
Track.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
Xu, J. Beyond goldfish memory: Long-term open-domain conversation. arXiv preprint arXiv:2107.07567, 2021.
Zou, T., Liu, Y., Li, P., Xiong, Y., Zhang, J., Liu, J., Ye, X., Ouyang, Y., and Zhang, Y.Q. Contrastive private data synthesis via weighted multi-plm fusion. arXiv preprint a5X2v:2502.00245, 2025.
Xu, Z., Zhang, Y., Andrew, G., Choquette, C., Kairouz, P., Mcmahan, B., Rosenstock, J., and Zhang, Y. Fed- erated learning of gboard language models with dif- ferential privacy. In Sitaram, S., Beigman Klebanov, B., and Williams, J. D. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pp. Toronto, Canada, July 2023. Association for Computa- tional Linguistics. doi: 10.18653/v122033aaclmm ind-uuusrry. 60. URL https: / /aaclaanni aallanhhlogyy wwtp::ooooo.og.///000 org/2023. acl wnw.trr..yy... 60%.
12

---

Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMs
A. Background on Differential Privacy
We use the standard (e, dddiffferettial privacy (DP) guarantee (Dwork et al., 2006) to measure the privacy risk of an ML algorithm memorizing individual records in the sensitive training data. For simplicity, we present a brief description below, and defer to (Dwork al., 2014; Ponomareva al., 2023) for more details.
Definition A.1 ((c, 5)-DP (Dwork et al., 2006)). A randomized algorithm M satisfies (c,6)-DP if for any two neighboring datasets ID, D' (defined by adding or removing one record from the dataset), and for any S C Range(M), where Range(M) the set of all outcomes of M.
Pr[M(DD) S] S e Pr)M(DD) E S) 6
At the high level. (c, 6)-P provides a formal definition that adding or removing a single record from the dataset should not have a large influence on the algorithm output. This indicates that the algorithm only learns the common knowledge from the entire dataset.
To help readers better understand our paper, we now briefly describe two important facts about (c, 6)-DP. First, one popular approach to DP training is DP-SGD (Song et al., 2013; Bassily et al., 2014; Abadi et al., 2016) or variants such as DP- Adam (Li et al., 2022; Yu et al., 2022), which modifies the standard SGD algorithm by clipping per-sample gradients and adding noise to each gradient updates during training. We use DP-Adam to train LMs throughout this paper. Second, any post-processing of a private algorithm's 's output cannot make it less private (Dwork et al., 2014). In our case, this property means that the synthetic dataset generated by a DP finetuned LM has the same (c, 8)DP guarantee as that of the DP finetuned LMM
B. Supplementary Discussion on Related Work
This paper focuses on generating privacy-preserving text data that resemble a private data source. We discuss more about prior work here in addition to $2.
Synthetic text data generated by DP-finetuned LMs. This is a popular approach: LM is first finetuned on the private data with DP, and then sampled to generate synthetic data (Bommasani et al., 2019; Putta et al., 2022; Mattemn et al., 2022a; Yue et al., 2023; Kurakin et al., 2023; Yu et al., 2024; Wang et al., 2024; Ochs & Habernal, 2024; Carranza et al., 2024). While this paper follows similar approach, we primarily focus on improving the data generation quality from a small O(lleM) LM. By carefully finetuning on the public and private data, the synthetic data generated by our method have significantly better quality than that obtained by the previous DP finetuning approaches for the O(100M)-scale LMs.
Existing DP finetuning-based approaches for synthetic data generation often rely on the strong capability and generalability of billion-scale LLMs, especially when the downstream tasks are the challenging generative tasks opposed to the simpler classification tasks. For instance, (Yu et al., 2024) DP finetune LLaMA-7B to synthesize Chatbot Arena-style short (often one-sentence) human-to muman-to-mmcchne machine instructions. While CTCL also incorporates DP finetuning step on private data, it significantly reduces the computational cost by using a backbone LM with only 140M parameters, making it much more acceptable for real-world resource-constraaned applications.
Synthetic text data that only require LLM API access. Because DP finetuning be expensive and sometimes impossible (e.g.. for non-public models), this line of work explores data synthesis assuming only access to the LLM inference APIs. Simply relying on the high-level knowledge about a private domain to design proper LLM prompts is not enough to generate synthetic data that well represent the actual private domain (Wu et al., 2024a). Using small DP language model to filter the initial synthetic data makes the data more similar to private data, but does not close the distribution gap for training a model for the downstream task in the private domain (Wu et al., 2024a; Zhang et al., 2025).
The Private Evolution (PE) framework, initially developed by Lin al. (2024) for the image domain, and later extended to the text domain in (Xie et al., 2024; Hou et al., 2024), proposes to "refine" (i.e., select good examples from) the current synthetic dataset according to the closeness of each example with respect to the private dataset. A similar idea is also explored by Zhao et al. (2024). The key idea behind PE is to measure closeness using DP nearest neighbor histogram: if an example is close to the private distribution, then it would receive a lot nearest neighbor votes from the private examples. PE starts with an initial dataset generated by the state-of-the of-th-------ttt aht LLMs via API access (using domain knowledge to design LLM prompts). PE then works by iteratively selecting good examples (measured by the DP nearest neighbor histogram) and
13

---

Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMs
using LLMs generate more similar examples. In our experiments, we show that PE performs worse than DP finetuning especially for generative tasks, even when we are only allowed to finetune small (million parameters) LMs.
Unlike prompting grompting-based approaches such as PE, CTCL does not depend on prompting advanced LLMs or external APIs when applied on downstream data. Although prompting step is involved during CTCL's 's pretraining phase in $3.1 (note that this step only needs to be done once), our prompt is only designed for summarizing a public document, without requiring the strong creativity capabilities from advanced LLMs. In practice, for this step, we only need a lightweight 2B-parameter LLM, ensuring that the constructed data can be scaled up to pretraining level. Table I presents the examples of prompts used in existing prompting- based data synthesis approaches as well as the one used in our pretraining data construction.
The size of the synthetic dataset in PE-based approaches is often constrained due to the sample-wise noise introduced by the differential-prig
nearest neighbor (DP-NN) process. In this setup, DP-NN serves as the only mechanism for incorporating the information from the private data into the synthetic dataset. Specifically, DP-NN identifies the nearest neighbors of each private data sample within the synthetic dataset. To preserve privacy, DP noise is added to each synthetic sample. In this process, each synthetic sample acts as a bin, with its count indicating how many private data samples identify it as their nearest neighbor. Noise then applied to these counts to ensure privacy. However, this process requires each bin to contain sufficiently large number of counts; otherwise, the noise overwhelms the signal, making difficult to distinguish between zero and small counts. Consequently, the synthetic dataset size must be limited, an excessive number of bins would make DP-NN ineffective. To overcome this limitation, variation process is often employed, where LLM APIs are prompted to generate additional samples based on the initially selected subset. However, since the private data does not directly influence this generation process, the final synthetic dataset theoretically contains no more information from the private data than the small subset originally selected. A few concurrent work (Hou et al., 2025; Zou et al., 2025) further improved the PE algorithm; while we did not directly compare to their experimental results, a lot of our discussion can still apply to these improved PE methods.
Nagesh et al. (2024) propose an approach that privately learns the probability distribution of keyphrases via kernel density estimation, followed by sampling sequences of keyphrases to seed LLM prompts. To better capture the correlations between the sampled keyphrases, this method requires estimating the distribution of keyphrases at varying lengths. As result, it is hard to scale this method to generating very long documents.
Another line of work explores generating et al., 2024; Wu al., 2024b; Duan et al., 2023). Given a reasonable DP guarantee, these methods can only generate a limited amount of synthetic data, e.g., few for ICI., or few thousands (Amin et al., 2024). By contrast, our method (based on DP finetuning) can generate a much larger dataset of synthetic examples.
few-shot examples for in-context learning (ICL), e.g., (Tang
Text privatization based on word or sentence perturbations. These approaches, e.g., (Feyisetan et al., 2020; Mattern et al., 2022b; Carvalho et al., 2023; Utpala et al., 2023), usually use different DP notion and perform worse than the approaches discussed above. We defer interested readers to Appendix C.11 in (Xie et al., 2024) for more details.
methods for synthesizing data for a specific domain, conditional generation is also explored to improve the diversity and generative quality of models in less privacy sensitive applications. We now discuss a few recent work in large language models. In concurrent work, Gao al. (2025) discussed metadata conditioning for pre-trainig large language models; DeSalvo al. (2025) developed soft prompt as condition improve synthetic data generation.
Conditional generation. In addition to
14

---

Synthesizing Privacy-Pree Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMs
C. Dataset Sizes
Dataset
Train
Valid
Test
PubMed
75,316
14,423
4,453
Chatbot Arena
1 180,000
5,000
3,819
Multi-Session Chat
17,940
3,000
2,505
Yelp 1,939,290
5,000
5,000
OpenReview 8,396
2,798
2,798
Table 7. Sizes of datasets in experiments.
D. Privacy Budget Allocation
Our privacy budget allocation follows the "Post-Generation Resample" algorithm proposed by Yu et al. (2024), which also has the DP-histogram and DP-finetune components. For the -i-histogamm step, we adopt the same configuration cowtrrttioonnndnnnngee Gaussian noise on every histogram bin with a standard deviation 10. The overall privacy budget (c) computed as the composition both steps using the standard DP accounting library"**
To illustrate, we provide the individual e values in the table below for the DP-histogram and DP-finetuning steps on the PubMed dataset (75K training samples), under composed privacy budgets of 4, 2, and 1, and ō As shown below, the DP-histogram step consistently accounts for a small portion of the overall budget-for instance,
0.39 for DP-histogram versus 3.96 for Di-finetuning when the composed c is 4. This trend aligns with the observations in (Yu et al., 2024) and highlights that our CTCL-Topic design achieves strong performance while consuming only a small portion of the privacy budget.
(Composed)
(DP-Hiitogram)
(DP-Finntunc)
0.39
3.96
2
0.39
1.94
0.39
0.9
Table 8. The allocation of provacy budget between DP-Histogram and DP-Finetune.
Additionally, we find that changing the privacy allocation to the DP-histogram step only has very small impact to the DP-finetuning step. For example, in the setting on PubMed, increasing the Gaussian noise in the histogram step from 10 to 20 alters the DP-finctuning marginally- from 3.96 3.99. This corresponds to small change in the noise multiplier for DP-finetuning (from 3.03 to 3.02).
15

---

Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMs
E. Noise Multipliers
The table below gives the noise multipliers used when DP finetuning LMs in our experiments. Following (Yue et al., 2023). we set 6 Vlog N where N is the size of private training set. Given desired (e, 6)-DP guarantee, the noise multipliers are computed using the standard dp_ccccunn package (DP Team, 2022). As pointed out in Appendix we use DP-Adam for DP finetuning and follow the standard Gaussian mechanism to obtain (c, 6)-DP guarantee. Compared to the "Vanilla DP Finetune" approach, the noise multiplier used by our method is slightly larger, because we need to allocate a small portion of the privacy budget to the DP topic histogram (see Appendix D). Besides, GPT2xl -15B has much smaller noise multipliers because we reduce the training batch size from 4096 to 256 to save computational resources. For other non-DP training hyperparameters, see $4.1.3.
c= 00
c
== 2
c
PubMed
Vanilla DP Finetune (BART Baw and GPT22mll) GPT2xu- 115B (reduced hatch size)
3.01
5.49
10.3
0.63
0.77
0.97
Ours
3.03
5.63
11.33
Chatbot Arena
Vanilla DP Finetune (BART Mose and GPT2smat) GPT2xx-1.58 (reduced batch size)
1.47
2.5
4,58
0,56
0.67
0.78
Ours
1.48
2.57
5.08
Chat
Vanilla DP Finetune (BART Base and GPT2 GPT2xu- (reduced batch size)
11.38
21.01
39.41
1.00
1.52
2.61
Ours
11.45
21.47
42.70
Yelp
Vanilla DP Finetune (BART Ruse and GPT2 GPT2x (reduced batch size)
0.63
0.77
0.91
0.69
Ours
0.63
0.77
0.94
OpenReview
Vanilla Finetune (BART Rose and GPT2smatt) GPT2x, -1.5B (reduced batch size)
23.3
42.87
80.05
1.64
2.81
5.11
Ours
23.44
43.72
86.05
Table 9. The noise multipliers used when DP finetuning LMs in experiments.
16

---

Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMs
F. Topic Model's Coverage
We choose Wikipedia as the basis for our topic model because it is large-scale, high-quality, and semantically diverse. Most real-world documents with meaningful content would fall within Wikipedia's 's domain. practice, rare encounter texts that are entirely outside of its scope. To illustrate the coverage of our topic model, the table below presents topics sampled from five diverse datasets used in our experiments. These examples demonstrate that the model captures broad range of universal topics:
Dataset
Sample topic
Sample Topic
PubMed
microscopyyelectron....
ma/mir/gene//...
Chatbot Arena
gameplay/rpg/wii....
cruptions/volcanoo...
Multi-Session Chat
comedian/preseentrrl..
oscar/nominations...
Yelp
restaurant/diner/... grammar/syntax/...
theater/cinemas/... computationnal/tuingg.....
OpenReview
Table 10. Example topics in the datasets of our experiments.
The "out-of-domain" text is also considered and processed in our experiments. Specifically, in our topic model, there an "unclassified" bin for documents that are sufficiently close to any learned topic. The table below reports the number of unclassified samples in each dataset, along with example such texts:
Dataset
Training size
Unclassified samples
Unclassified sample
PubMed
75,316
0
Chatbot Arena
180,000
86
hello. have no idea who am i, maybe you can help?
Multi-Session Chat
17,940
0
Yelp
1,939,290
4 I am unable to provide check number.
Revnee
8,396
0
Table 11. Analysis of unclassified samples in every datasets.
As shown in Table 11, the number of unclassified samples neglectable across all datasets. Furthermore, those that do fall into the unclassified bin typically contain little to no substantive content, confirming the broad coverage of our topic model.
G. Computational Cost and Tradeoff's
Our pretraining pipeline involves three main components:
Data Curation:
Running inference using a 2B LLM on 430M pretraining documents.
Generator Pretraining: Training a 140M conditional LM on 430M (condition, document) pairs.
Topic Modeling: Generating embeddings for 6M Wikipedia documents with 20M document embedding model, followed by a HDBSCAN clustering.
Among these, data curation for pretraining dominates the computational cost due to the scale of LLM inference. This remains lighter than or comparable to the cost of the pretraining of 2B LLM. Note that after we release our pretrained CTCL-Generator and CTCL-Topic, other uses can skip the pretraining step (i.c., do not need to pay the pretraining costs), and directly finetune the model on their private data.
For the synthetic data generation stage, our approach samples from a small 140M model after finetuning on the private data. By contrast, existing PE-based methods often involve significant API costs that are proportionally the sample size. For instance, in the experiments of AUG-PE (Xie et al., 2024), they synthesize 2,000 samples using T 10 evolution iterations, each with L = 4 variations, totaling 80, 000 (2, 000 10 x 4) ChatGPT requests.
17

---

Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMs
H. Pseudo-code
We provide pseudo-code our framework corresponding to Section 3 and Figure
Algorithm 1 A. Pretraining with Public Corpora
CTCL-Topic Development
1: Cluster Wikipedia document embeddings via HDBSCAN.
2: Assign top-10 keywords to each cluster as topics.
TTCe--Generrtor Pretraining
1: Extract aspects (e.g.. keywords, document type) from public corpora such as Slimpajama.
2: Train a 40r-parameter conditional LM on (aspect, document) pairs.
Algorithm 2 B. Learning the Private Domain
Private Topic Histogram Construction
1: Use CTCL-Topic assign topics to each private document.
2: Construct a DP histogram to represent topic-level distribution of the private domain.
DP Finetuning CTCL-Generator
1: Form (keyword, document) training pairs using CTCL-TTpic.
2: Finetune the CTCC-Generator using these pairs with differential privacy.
Algorithm 3 C. Data Synthesis
For each topic, determine the number samples based on DP topic histogram.
2: Use the finetuned CTCL-Generator generate documents conditioned on topic keywords.
18

---

Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMs
I. Discrepancy in Constructed Conditions between Pretraining and Finetuning
In our pretraining stage ($3.1), the condition is extracted for each document from Gemma 2B, while in the learning of private data, the condition is from the topic model. The distinction between pretraining and finetuning conditions intentional and serves two main purposes: enhancing model generalizability and adhering to privacy constraints. During the pretraining stage, using the 2B LLM to extract aspects is for increasing the diversity and flexibility of the aspects the model can handle, enabling stronger controllability at inference time. Otherwise, "keywords" becomes the only aspect in the pretraining.
On the other hand, not also using Gemma 2B in the finetuning time is for privacy consideration. Specifically, if Gemma 2B were applied directly to private data, the extracted conditions would be in free-text format. This raises our concerns about sending the private information to the synthetic data generation process. contrast, our framework ensures differential privacy by using noised, topie-wise histogram instead. This approach preserves key high-level topic information of the private domain while satisfying privacy requirements.
To demonstrate the value of the flexible conditioning during pretraining, consider scenario where has a bit prior knowledge about the downstream domain (e.g., that the documents are dialogues). Our design enables such weak supervision to be incorporated, providing a better initialization for subsequent DP-finetuning. Below are examples of model generations before any finetuning, controlled only by document type, showcasing the better initialization:
Condition
Document Type: News
Keywords: Surfing, World Championship, Young Athletes
Generation
The 15- year-odd, who is member of the World Surfing Association (WSA) and the World Surf League (WSL), won the World Championships for the first time in his career, [...]
Condition 2
Document Type: Chat
Keywords: Surfing, World Championship, Young Athletes
Generation
How do you feel about the current state of surfing in the U.S. right now? There are a lot of great young athletes out there right now, and it's going to be a really exciting year for them. [...]
Condition 3
Document Type: Blog Post
Keywords: Surfing. World Championship, Young Athletes
Generation
I've been fortunate enough have the opportunity to meet of the most talented young athletes in the world, and I'm [...]
Table 12. The flexible design of CTCL-Generator's pretraining also intends to encourage broader and more creative uses.
19

---

Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMS
J. Qualitative Analysis
We provide synthetic data examples in addition to Table 6.
BART-Baas (Downstream Model Performance: 30.5%);
Sample 1: We explored the relationship hetween molecular interaction, NCT-2 and NCT-3 (NCT-1), NCT-4, NCT and NCT-1.5 (NCT), NCT- NCT-6, and NCT[4]. In recent clinical trial, we described an enzyme in NCT-10 10 that enabled novel processes to explore novel approaches for NCT- 2.3 to NCT-IIL NCT was applied to NCT and the NCT-8 was used to explore the implications of NCT NCT in NCT. NCT-11 was used to describe NCT-7 to NCT preparation for NCT preparation. NCT and PCT were compared with NCT-16 with NCT and [..]]
Sample 2: The authors of this paper present a unique method for understanding the mechanism of the effects of the mechanism. It aims investigate whether the mechanism used by the methods involved in the study was used determine the mechanism of treatment, including the mechanism of its regulation. The analysis was conducted on an open-source basis and did not rely on the traditional methods of the primary investigation, such as the original design for the method used, which looked at the mechanisms used to [...]
G..5 (Upper Bound, Downstream Model Performance: 37.7%
Sample 1:
The ability of leptin to induce weight loss, to stimulate ectothermic thermogenesis, and to augment activity of the AMPK system and the AMPK- -dppedennt lipoprotein lipase activity, was examined. Circulating concentrations of leptin were assessed in the femoral adipose fat pad of the lean and obese rats. Serum leptin levels and AMPK mRNA expression were assessed in sera obtained from lean and obese female rats fed diets enriched with fat or carbohydrate. The leptin binding to the leptin receptor [...]
Sample 2: Nuclear stress hormone (NTH) levels increased patients with type 2 diabetes mellitus (T2D) following therapy with metformia [13] or insulin (e.g.. Lipid. insulin- C3G or mnsulin-G3F) [14, 15]. NTH was not increased in patients enrolled to receive metformin or lispro (or in patients taking insulin-C C-based drugs (i.e., lisulin and galantangulin, alemtubusin) [16], suggesting the insulin
treatment itself might cause patients to become insulin sensitive (i.g.. via altered insulin action). [...]
AUG-PE + GPT-3.5 (Downstream Model Performance: 30.3%)
Sample 1: Deep learning, in combination with artificial intelligence, has emerged powerful tool in the field of medical imaging. This study explores the application of deep learning algorithms the identification of various medical conditions. By leveraging neural networks, this approach provides unparalleled accuracy across diffcrent modalities, enhancing our understanding of pathological processes. The integration of deep learning in medical imaging has the potential to revolutionize diagnostic processes [...]
Sample 2: Investigating synaptic plasticity in the context of neurological networks investigation, cohort of patients with neurodegenera- tive disorders was closely monitored. The efficiency of synaptic transmission was evaluated, considering the underlying mechanisms particularly focusing on the composition scaffolding proteins. Insights into the impact of adipocyte signaling on synaptic function were uncovered. The research further characterized the dynamics of synaptic plasticity and illuminated how [...]
Ours (Downstream Model Performance: 35.9%):
Sample 1: To develop a therapeutic formula to reduce rates of morbidity that occur in people with a combination of cardiovascular problems. We used multi-state, multidisciplinary approach to the research of the clinical manifestations of cardiovascular problems with the introduction of a biocontrol. Using multi-choice, multi-prongee' approach evaluated a combination of the three major elements of cardiovascular problems: chronic diseases; vascular disease that occurs in chronic conditions in chronic [...]
Sample 2: This study describes the mechanism of action of different mechanisms to be performed to evaluate the efficacy and quality of control activities at three different levels. A cross-hatch method was used to investigate the efficacy and health outcomes of different mcchanisms in all three levels of intervention. The outcome compared with the outcomes obtained by crosshatch method. A total of 20 control methods used during this study included 7 standard I/A for four trials. All intervention levels [...]
Table 13. Additional synthetic data samples on PubMed under = 4. Randomly Sampled.
20

---

Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMs
K. Full Results of Generative Tasks
PubMed (Medical Paper Abstract)
Setting
BERT
BERT
BERTMMn
BERT
BERT
BERT
BERT Min
BERT
GP12x.-1.5B (Upper Bound)
40.2
36.8
39.7
-..T2x-1.5B-L.oo (Upper Round)
39.4
42.5
34.7
37.7
34.9
37.9
34.9
37.9
Downstream DPFT (No Syn. Data)
44.3
46.0
30.7
34.1
28.9
32.5
26.7
30.4
Private Evolution (PE) (Lin et al., 2024)
29.7
31.8
29.6
31.8
29.7
31.9
29.8
31.9
AUG-PE Mixtral-&xBB (Xie al. 2024)
24.9
27.6
24.5
27.1
AUG-PE GPT-3.5 (Xie et al., 2024)
30.4
32.5
32.5
30.1
32.4
GPT2smn (Yue al., 2023)
38.1
41.6
35.0
37.4
32.0
34.4
26.8
29.3
GPT2 Resample (Yu al., 2024)
39.0
42.4
35.3
37.5
33.0
35.1
27.6
29.1
BART (Yue al., 2023)
40.9
43.9
30.5
32.4
28.9
30.8
26.7
28.5
BART Resample (Yu al., 2024)
41.3
44.2
30.7
32.5
29.0
30.7
26.5
28.0
Ours
41.5
44.6
35,9
38.1
35,4
37.6
34.5
36,7
Chatbot Arena (lluman-to Ma-Mhhine Instructions)
Setting
oc
BERT Min
BERT Smat
BERT Mini
BERT Small
BERT Miii
BERT Small
BERT Staai
BERT Small
G.72a.11.58 (Upper Bound)
26.6
29.4
19.6
21.9
19.4
21.8
19.2
21.6
-1.2xx.-15BBLLokk (Upper Bound)
28.5
31.1
22.9
25
22.8
24.9
22.8
25.0
Downstream DPFT (No Syn. Data)
28.9
31.9
13.3
12.5
11.9
10.9
10.3
9.2
GPT2sma (Yue et al., 2023)
26.1
28.8
18.8
20.7
17.7
19.5
16.0
17.6
GPT2 + Resample (Yu ct al., 2024)
26.8
29.3
18.7
20.0
17.6
18.6
15.9
17.1
BART (Yue et ul., 2023)
21.8
24.1
15.9
16.8
14.9
16.1
13.5
14.5
BART Bose Resample (Yu al., 2024)
23.4
25.6
16.3
15.3
16.7
14.3
15.1
Ours
22.5
24.9
19.6
21.5
21.2
19.2
20.7
MessioSession Chat (Long-Term Term Human-Human Human Conversations)
4
Setting
BERT BERT
BERT
BERT
BERT
BERT
BERT
BERT
G.T (Upper Bound)
33.2
25.3
28.7
23.9
27.0
GPT2xa-1.5B-Loke (Upper Bound)
38.3
40.8
28.4
30.7
28.4
31.1
28.8
31.1
Downstream DPFT (No Syn. Data)
38.8
40.1
21.6
17.7
11.8
15.1
6.7
GPT2smat (Yue et al., 2023)
34.6
19.9
20.2
21.4
15.1
17.3
GPT2sma + Resample (Yu 2024)
18.7
13.4
BART (Yue al., 2023)
34.2
27.8
29.1
23.8
25.0
10.8
11.2
BART Resample (Yu al., 2024)
34.8
37.4
28.1
29.1
24.2
25.1
9.1
9.8
Ours
34.3
36.4
30.3
32.6
29.1
29.7
27.6
29.3
Table 14. Performance of generative tasks evaluated by next-word prediction accuracy of downstream models (BERT and BERT smaller privacy budget (c) corresponds to stricter privacy constraint.
21


Reviews:
[{"review_id": "Q5PZJ8h9zh", "reviewer": "Reviewer_QqU8", "summary": "This paper introduces a new method CTCL to enable efficient synthetic data generation through privately fine-tuning a controlled generation LM. CTCL has two stages, first learning the general topics of private data through DP histogram learning and then privately fine-tunes a language model given the topics as the condition. CTCL only requires a LM of size 140M which is considerably smaller than existing on-device LLMs in the scale of billions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and encouraging feedback that the problem we investigate is important, that our method is well-motivated and matching real-world applications, that our experiments are thorough, and that our paper is well-written!\n\n### Pretraining\nWe would like to clarify that our pretraining *does not* start from random initialization. As noted in Sec. 3.1, we perform *continual pretraining* on top of BART-base [1], a model already pretrained on a general corpus and it possesses basic language understanding abilities. Our continual pretraining is to endow the model with controllability.\n\nWe agree that knowledge distillation could be an alternative to our pretraining, while it still falls within our overall framework, as it also needs properly constructed data in order to achieve controllability. Moreover, it should require more meticulous design choices and more computation due to the large teacher model.\n\nWe would also like to emphasize that the core contribution of our work is the overall CTCL framework. Each module design within the framework remains flexible and open to further optimization and design improvements.\n\n[1] BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension, ACL 2020\n\n\n### Discrepancy in constructed conditions between pretraining and finetuning\n\n**Why not topic model for pretraining?**\nThe distinction between pretraining and finetuning conditions is intentional for two purposes: enhancing model generalizability and adhering to privacy constraints. In pretraining, using conditions from 2B LLM is for the diversity and flexibility of the aspects that the model can handle, enabling stronger controllability. Otherwise, “keywords” would become the only aspect in the pretraining.\n\n**why not Gemma 2B for private data?**\nIf Gemma 2B were applied directly to private data, the extracted conditions would be in free-text format. This raises our concerns about sending the private information to the synthetic data generation process. In contrast, our framework ensures differential privacy by using a noised, topic-wise histogram instead. This preserves meaningful high-level topic information of the private domain while satisfying privacy requirements.\n\nTo demonstrate the value of the flexible conditioning in pretraining, consider one may have a bit prior knowledge about the downstream domain, e.g., knowing the data is dialogues. In our framework, such prior knowledge can be incorporated, providing a better initialization for DP-finetuning. \n\nBelow are the model generations controlled by doc type using the same keywords (*Surfing, World Championship, Young Athletes*), showcasing the better initialization:\n * (Doc Type: News): *A member of WSA and WSL, won the World Championships for the first time in his career. [...]*\n * (Doc Type: Chat): *How do you feel about the current state of surfing in the U.S. right now? \\n There are a lot of great young athletes [...]*\n * (Doc Type: Blog Post): *I've been fortunate enough to have the opportunity to meet the most talented young athletes [...]*\n\nThis flexible design also intends to encourage broader and more creative uses of our model.\n\n### Adapting condition generation to PE methods\nWe agree our condition generation could be potentially adapted by PE, for example, within the variation prompts. However, as noted in the paper, a key limitation of PE methods is not learning fine-grained info from private data. Applying our condition generation into the PE framework might not address this key limitation of theirs.\n\n### Topic Model on Wikipedia and Out-of-Domain Generalization\nWe choose Wikipedia for our topic model because it is large-scale, high-quality, and semantically diverse. In practice, it is rare to encounter texts that are entirely outside of its scope. The table below with sampled topics of our five diverse datasets illustrates that a broad range of universal topics are captured:\n\n| Dataset | Sample topic 1 | Sample Topic 2 |\n|--|--|--|\n| PubMed | microscopy/… | rna/gene/… |\n| Arena | gameplay/rpg/…    | volcano/… |\n| Chat | comedian/…  | oscar/… |\n| Yelp | restaurant/…    | theater/… |\n| OpenReview | grammar/… | computational/… |\n\nThe \"out-of-domain\" text is also considered and processed in our experiments--in our topic model, there is an “unclassified” bin for samples that are not close to any topic. The table below shows the details\n\n| Dataset | Training size | unclassified samples | Example |\n|-|:-:|:-:|--|\n| PubMed | 75,316 | 0 | |\n| Arena | 180,000 | 86 | i have no idea who am i, maybe you can help? |\n| Chat | 17,940 | 0 | |\n| Yelp | 1,939,290 | 4 | I am unable to provide check number. |\n| OpenReview | 8,396 | 0 | |\n\nWe can see the number of unclassified samples is neglectable across all datasets, and those unclassified samples typically contain little to no substantive content, confirming the broad coverage of our topic model.", "claims_and_evidence": "The claims are supported by the evidence under the assumptions that the author made.", "methods_and_evaluation_criteria": "The evaluation criteria makes sense to demonstrate the quality of synthetic data.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiments are thorough to demonstrate the effectiveness of the proposed approach.", "relation_to_broader_scientific_literature": "The paper is related to private synthetic data generation which is an important topic in the age of LLMs. The method targets resource-limited scenarios which match closely with real-world deployment.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and motivated, privately generating high quality synthetic data with a small model is desirable especially considering fine-tuning on the private data on edge devices which have limited resources.\n2. The experiments are thorough with multiple baseline approaches, tasks, and datasets considered. \n\nWeaknesses:\n1. It is unjustified why pre-training from scratch is needed for the LM part. Can one start from a pre-trained smaller language model with a similar number of parameters? Also, would knowledge distillation from an LLM into a smaller LM be enough?\n2. There is an under-explained discrepancy between how condition is generated in the pretraining and fine-tuning stage. In the pre-training stage, the condition is extracted for each document from Gemma 2B, while for private data, the condition is from the topic model. Why not also use the topic model to generate the condition for the pretraining data as well? Or why not use Gemma 2B to extract aspects for the private data and learn those aspects privately?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. The generated conditions seem to be helpful for other methods as well, e.g. for private evolution, the conditions can be used for engineering the prompts for variations. Have the authors considered this as an alternative?\n2. The topic model is trained on Wikipedia, how would it work when the topics from private data are personal and out of the domain of Wikipedia?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XuajlkuSD8", "reviewer": "Reviewer_E1ix", "summary": "The paper proposes a novel framework for generating privacy-preserving synthetic text data called CTCL. The authors claim that previous works mainly utilized fine-tuning the LLMs with differential privacy, which is computationally expensive or relies on extensive prompt engineering, which is time-consuming and does not necessarily result in a good performance. \n\nInstead, CTCL includes several partitions to guarantee better performance and privacy at the same time. First, they use public data (here Wikipedia) to train a generator. Then using DP fine-tuning they adapt the generator to private data. The generator then synthesizes data guided by this DP histogram information of data. Evaluations across various domains demonstrate CTCL's effectiveness, especially under stringent privacy conditions, and its superior scalability compared to existing techniques.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your recognition of the adequate discussion in our paper writing, the accurate identification of limitations in prior approaches, the efficiency of our lightweight final model, and our efforts to reduce reliance on prompt engineering.\n\n### Public Data capturing private domain info.\nWe agree a key intuition behind our design is the information learned from large-scale, diverse public data can benefit downstream learning in private domains. This aligns with the principle behind the pretraining–finetuning paradigm adopted across most LLM research.\n\n### Topic model design choices\nWe choose Wikipedia for our topic model because it is large-scale, high-quality, and semantically diverse. In practice, it is rare to encounter texts that are entirely outside of its scope. The table below with sampled topics of our five diverse datasets illustrates that a broad range of universal topics are captured:\n\n| Dataset | Sample topic 1 | Sample Topic 2 |\n|--|--|--|\n| PubMed | microscopy/… | rna/gene/… |\n| Arena | gameplay/rpg/…    | volcano/… |\n| Chat | comedian/…  | oscar/… |\n| Yelp | restaurant/…    | theater/… |\n| OpenReview | grammar/… | computational/… |\n\nThe \"out-of-domain\" text is also considered and processed in our experiments--in our topic model, there is an “unclassified” bin for samples that are not close to any topic. The table below shows the details\n\n| Dataset | Training size | unclassified samples | Example |\n|-|:-:|:-:|--|\n| PubMed | 75,316 | 0 | |\n| Arena | 180,000 | 86 | i have no idea who am i, maybe you can help? |\n| Chat | 17,940 | 0 | |\n| Yelp | 1,939,290 | 4 | I am unable to provide check number. |\n| OpenReview | 8,396 | 0 | |\n\nWe can see the number of unclassified samples is neglectable across all datasets, and those unclassified samples typically contain little to no substantive content, confirming the broad coverage of our topic model.\n\n### Privacy Budget Allocation\nOur budget allocation follows [1], our \"Post-Generation Resample\" baseline, which also has both DP-histogram and DP-finetune steps. For DP-histogram, we do the same -- adding a Gaussian noise 10 on every histogram item. The overall privacy budget (ε) is computed as a composition of both steps using the standard `dp_accouting` package.\n\nWe also show individual ε values below on PubMed, under composed $ε=4, 2, 1$. DP-histogram only takes a small portion of the overall budget. This aligns with observations in [1] (page 5) and highlights our CTCL-topic design achieves strong performance while consuming only a small portion of the privacy budget.\n\n| ε (Composed) | ε (DP-Histogram) | ε (DP-Finetune) |\n|:-:|:-:|:-:|\n| 4 | 0.39 | 3.96 |\n| 2 | 0.39 | 1.94 |\n| 1 | 0.39 | 0.9 |\n\nWe also find that the impact of changing the allocation is insignificant. For example, in $ε=4$ setting on PubMed, increasing histogram's noise from 10 to 20 only alters the DP-finetuning budget $ε$ marginally—from 3.96 to 3.99. \n\n\n[1] Privacy-Preserving Instructions for Aligning Large Language Models, ICML 2024\n\n### Computational Cost\nOur pretraining has 3 steps:\n1. 2B LLM inference on a 430M documents.\n2. Training a 140M LM on 430M (condition, document) pairs.\n3. Generating embeddings for 6M Wikipedia docs with a 20M embedding model, followed by a HDBSCAN clustering.\n\nAmong these, Step 1 dominates the computation due to the 2B LLM. This remains lighter than or comparable to the computation of pretraining a 2B LLM.\n\nNotebly, we will release our pretrained models, so users don't take the pretraining cost. This corresponds to the pretraining of the LLMs behind the APIs in PE-based approaches.\n\nFor the downstream stage, PE methods often have significant user-side costs. For instance, AUG-PE synthesizes 2,000 samples in T=10 evolution iterations, each with L=4 variations, totaling 80K (2K x 10 x 4) ChatGPT requests. This represents a substantial API cost, whereas our approach applies finetuning on the lightweight model on local devices, which has no significant cost overhead.\n\n### Use of Gemma\nAs shown in our paper Table 1 and mentioned in Sec 3.1, unlike PE’s reliance on strongest LLMs like ChatGPT, our framework only requires basic instruction-following ability of the LLM, since aspects are extracted from existing documents without relying on LLM’s creativity.\n\nNotably, the LLM we use, Gemma-2-2B, is already one of the smallest LLMs. Larger models, such as its 9B and 27B variations, would likely yield even stronger results. We don’t experiment with more LLM choices because the small 2B LLM already has strong performance.\n\nWe would also like to emphasize that the core contribution of our work is the overall CTCL framework. Each module design within the framework remains flexible and open to further design improvements.\n\n### Pseudo-code and Limitations section\nThank you for the suggestions! While ICML does not require a Limitations section, we will include both our pseudo-code and a limitation section into our next paper version for improved clarity.", "claims_and_evidence": "YES", "methods_and_evaluation_criteria": "YES", "theoretical_claims": "Not Applicable", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "* Improving the state-of-the-art DP-synthetic data generation. The synthetic data can later be used in the fine-tuning of models to preserve privacy and avoid the problems of directly using DP-finetuning. \n\n* Instead of directly using prompt engineering to synthesize the data, they utilize the LLM to cluster the data. In this way, the performance is not directly dependent on the prompts or models.", "essential_references_not_discussed": "NO", "other_strengths_and_weaknesses": "STRENGTHS:\n\n* The paper is very well-written. Motivation, prior works, and method intuitions are adequately discussed. \n\n* The paper correctly identifies the shortcomings of the SOTA method (PE) and shows it in the experiments. (The most interesting result is that the PE's performance is primarily independent of the privacy budget).\n\n* The final generator has only 140M parameters, which is efficient in fine-tuning and inference.\n\n* The effort toward **reducing** the reliance on prompt engineering is appreciated.\n\nWEAKNESS:\n\n* There is an assumption that the public data can capture the primary information of the private text. \n\n* The computational cost of the methods is missing. Some components seem very complex and expensive (although they are done once, it is good to understand the complexity and performance trade-off).\n\n* The framework is complex; the authors have explained it in Figure 1, but a pseudo-code description can help the readers understand each step better.", "other_comments_or_suggestions": "* There should be a cost comparison between the CTCL and the prior works. PE only used the model API to generate the synthetic data, but the CTCL pipeline requires training and fine-tuning multiple components.\n\n* The authors have shown that changing the model API can change the performance of the PE method. The same analysis is applicable to the use of Gemma 2 in CTCL.", "questions_for_authors": "* The limitation section is missing.\n\n* How is the method performed for out-of-domain private data?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ofcOKJ0QIY", "reviewer": "Reviewer_hXAS", "summary": "This paper introduces CTCL (Data Synthesis with Controllability and Clustering), a framework to generate privacy-preserving synthetic text data without fine-tuning LLMs or extensive domain-specific prompt engineering. The CTCL framework consists of two primary components: a lightweight 140M-parameter conditional text generator and a universal clustering-based topic model, both pretrained on publicly available datasets. To adapt to private domains, the generator undergoes differential privacy (DP) finetuning to capture fine-grained textual details, while the topic model constructs a DP-protected histogram to represent high-level distributional information. Synthetic data generation then leverages this DP histogram to control topic distribution during sampling. Empirical evaluations demonstrate that CTCL outperforms prior methods, such as Aug-PE, in benchmarks including PubMed and Chatbot Arena, particularly under tight privacy budget.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging remarks that our claims are well-supported by clear experiments and detailed analyses, and our proposed method is reasonable.\n\n### Generator design choices\nWe use a seq2seq generator because its encoder is well-suited for understanding the conditions. For size, the 140M model is to balance the efficiency with generation ability, making it practical for resource-constrained scenarios.\n\nWe agree other design choices might perform better, but we do not extensively explore all designs because our intuitive choice already demonstrates strong downstream performanceas in our experiments. Moreover, same as all pretraining research, due to the cost and scale, it could be super challenging to exhaustively search over all possible configurations.\n\nWe would also like to emphasize that the core contribution of our work is the overall CTCL framework. Each module within the framework remains flexible and open to further design improvements.\n\n\n### Qualitative Analysis\nThank you for the suggestion! We will include more qualitative examples in the next version of the paper in addition to those in Table 6. As noted in Section 4.3.3, we emphasize qualitative results because our primary usage of synthetic data is to support downstream model development. Therefore, the surface form of the synthetic text is less critical. For instance, Table 6 shows that PE-based generation using ChatGPT consistently produces fluent outputs. However, its downstream model performance is only comparable to (and sometimes worse than) that trained on less fluent synthetic data generated by DP-finetuned BART-base.\n\n\n### Topic model design choices\nWe choose Wikipedia for our topic model because it is large-scale, high-quality, and semantically diverse. In practice, it is rare to encounter texts that are entirely outside of its scope. The table below with sampled topics of our five diverse datasets illustrates that a broad range of universal topics are captured:\n\n| Dataset            | Sample topic 1        | Sample Topic 2         |\n|--------------------|-----------------------|------------------------|\n| PubMed             | microscopy/electron/… | rna/mir/gene/…         |\n| Chatbot Arena      | gameplay/rpg/wii/…    | eruptions/volcano/…    |\n| Multi-Session Chat | comedian/presenter/…  | oscar/nominations/…    |\n| Yelp               | restaurant/diner/…    | theater/cinemas/…      |\n| OpenReview         | grammar/syntax/…      | computational/turing/… |\n\nThe \"out-of-domain\" text is also considered and processed in our experiments--in our topic model, there is an “unclassified” bin for samples that are not close to any topic. The table below shows the details\n\n| Dataset | Training size | unclassified samples | Example |\n|-|:-:|:-:|--|\n| PubMed | 75,316 | 0 | |\n| Chatbot Arena | 180,000 | 86 | i have no idea who am i, maybe you can help? |\n| Multi-Session Chat | 17,940 | 0 | |\n| Yelp | 1,939,290 | 4 | I am unable to provide check number. |\n| OpenReview | 8,396 | 0 | |\n\nWe can see the number of unclassified samples is neglectable across all datasets, and those unclassified samples typically contain little to no substantive content, confirming the broad coverage of our topic model.\n\n### Privacy Budget Allocation\nOur budget allocation follows [1], our \"Post-Generation Resample\" baseline, which also has both DP-histogram and DP-finetune steps. For DP-histogram, we adopt the same -- adding a Gaussian noise 10 on every histogram item. The overall privacy budget (ε) is computed as a composition of both steps using the standard `dp_accouting` package.\n\nWe also show individual ε values below on PubMed, under composed ε=4, 2, or 1. DP-histogram only takes a small portion of the overall budget. This aligns with observations in [1] (page 5) and highlights our CTCL-topic design achieves strong performance while consuming only a small portion of the privacy budget.\n\n| ε (Composed) | ε (DP-Histogram) | ε (DP-Finetune) |\n|:-:|:-:|:-:|\n| 4 | 0.39 | 3.96 |\n| 2 | 0.39 | 1.94 |\n| 1 | 0.39 | 0.9 |\n\nWe also find that the impact of changing the allocation is insignificant. For example, in ε=4 setting on PubMed, increasing histogram's noise from 10 to 20 only alters the DP-finetuning budget $ε$ marginally—from 3.96 to 3.99. \n\n\n\n### Nucleus Sampling\nOur choice of nucleus sampling with top-p = 0.95 follows the standard practice introduced in [2], which has been shown to provide a strong balance between diversity and coherence, outperforming alternatives such as top-k or beam search. Also, since this standard hyperparameter setting already yields strong performance, we chose not to exhaustively tune this particular top-p value.\n\n[1] Privacy-Preserving Instructions for Aligning Large Language Models, ICML 2024\n\n[2] The Curious Case of Neural Text Degeneration, ICLR 2020.", "claims_and_evidence": "The claims made in the submission are generally well-supported by clear experiments and detailed analyses. The authors provide extensive experimental comparisons against several established baseline methods across multiple downstream tasks—both generative and classification—and demonstrate improvements, particularly under strict DP budget constraints.", "methods_and_evaluation_criteria": "The proposed methods make sense conceptually. The evaluation criteria follow prior works in DP finetuning and DP data synthesis.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I reviewed the experiment design, especially the validation dataset. I like the inclusion of open-ended tasks in Chatbot Arena, which are usually not present in other LLM DP fine-tuning papers.", "relation_to_broader_scientific_literature": "The contributions of this paper build specifically upon prior work in privacy-preserving data generation and differential privacy (DP) in LLMs. By integrating a lightweight (140M-parameter) conditional text generator with a universal topic model pretrained on public corpora, CTCL efficiently synthesizes high-quality text data under strict DP constraints without extensive prompt engineering or significant computational resources.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strength**\n\nThe approach to synthesizing privacy-preserving data using a lightweight language model and clustering-based topic model is\n\n**Weakness**\n+ More design choices of the conditional generator are not explored.\n+ The evaluation primarily focuses on quantitative benchmarks, with limited qualitative analysis of generated synthetic data. Additional qualitative analysis could help the reader further understand the practical usefulness and readability of the synthetic data.", "other_comments_or_suggestions": "Please see the questions below.", "questions_for_authors": "1. Since the proposed framework requires training a clustering-based topic model and a generator LLM on Wikipedia, how sensitive is it to domain mismatch between Wikipedia topics and the private datasets? Are there failure cases where this approach may struggle from domain mismatch?\n2. Could the authors clarify how the DP budget was allocated between the DP Topic Histogram and DP Finetuning?\n3. For the sample generation (4.1.3), the authors used nucleus sampling with top-p=0.95. How sensitive is the quality of generated data to the choice of sampling parameters?\n4. What’s the rationale for using a 140M BART-like model? I think the paper can be stronger if more design choices for the generator are explored.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mdgc5tbEuW", "reviewer": "Reviewer_pWe8", "summary": "This paper presents CTCL, a framework that synthesizes privacy-preserving data by combining a lightweight 140M parameter generator with a clustering-based topic model. The generator is differentially privately fine-tuned on private data, while the topic model produces a DP topic histogram to capture high-level distributional information. This approach circumvents the computational burden of fine-tuning billion-scale LLMs and the need for extensive prompt engineering, proving effective across diverse domains including sensitive applications and generative tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive and detailed comments! We are encouraged by your feedback that our approach is novel, our experiments and analyses are comprehensive, and that our method makes  significant advancements in privacy-preserving data synthesis.\n\n### Privacy Attacks\n\nAccording to the post-processing property of differential privacy (DP), our synthetic data has the same formal DP guarantees—$\\epsilon=1,2,4$—as our generator, and these privacy budgets are widely accepted in the machine learning community, with $\\epsilon < 10$ considered reasonable and $\\epsilon \\approx 1$ regarded as offering strong privacy protection ([Ponomareva et al. 2018](https://arxiv.org/abs/2303.00654)).\n\nDP already offers provable protection against membership inference and other, typically weaker, attacks such as reconstruction or attribute inversion attacks. This is supported by extensive empirical evidence, e.g., [Steinke et al. 2023](https://arxiv.org/abs/2305.08846), [Andrew et al. 2023](https://arxiv.org/abs/2302.03098), and [Nasr et al. 2023](https://arxiv.org/abs/2302.07956). \n\nFurthermore, recent works such as [Yue et al. 2023](https://arxiv.org/abs/2210.14348) and [Yu et al. 2024](https://arxiv.org/abs/2402.13659)’s experiments on secret sharer-style attacks ([Carlini et al. 2018](https://arxiv.org/abs/1802.08232)) in the DP synthetic data setting further support the privacy protection offered by DP.\n\nAs practical attacks are usually weaker than the worst-case attacks that DP is designed to protect against, based on empirical privacy auditing and attack studies in recent years, we skipped auditing in this paper. That being said, we would be happy to include relavant experiments if you have a specific attack in mind that is beyond our consideration and feasible in our setting!", "claims_and_evidence": "Yes. The submission’s claims are largely supported by a comprehensive set of experiments and analyses. It demonstrates that CTCL outperforms several baselines on both generative and classification tasks under differential privacy constraints, and the scalability studies show clear improvements with larger synthetic datasets and higher privacy budgets.", "methods_and_evaluation_criteria": "Yes. The use of diverse benchmark datasets—ranging from academic medical texts to conversational dialogues and business reviews—ensures that the evaluation reflects real-world applications where privacy is critical. Moreover, evaluation metrics such as next-word prediction accuracy, perplexity, and MAUVE scores, along with thorough ablation studies and scalability experiments, provide robust evidence for the framework's effectiveness.", "theoretical_claims": "Yes. The algorithm is a chain of well-known DP algorithms. By the composition law of DP, everything is correct.", "experimental_designs_or_analyses": "NA", "relation_to_broader_scientific_literature": "CTCL innovatively integrates a lightweight, 140M-parameter conditional generator with a clustering-based topic model to capture both high-level distributional information and fine-grained textual details. This combination not only circumvents the resource constraints associated with billion-scale models but also offers enhanced controllability and scalability, marking a significant advancement in privacy-preserving data synthesis.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "**Strengths:**  \n\n- The novel integration of a lightweight DP-finetuned generator with a clustering-based topic model enables efficient synthesis of high-quality synthetic data by capturing both fine-grained details and high-level distributional patterns.  \n- The approach leverages well-established DP techniques to provide robust privacy guarantees without requiring extensive prompt engineering or resource-intensive fine-tuning of large-scale models.\n\n\n\n**Weakness:**  \n- My main concern is that paper lacks evaluation of the proposed approach against practical privacy attacks that might occur in real-world scenarios. So while the paper shows good utility improvements, its privacy aspects remain insufficiently evaluated.", "other_comments_or_suggestions": "NA", "questions_for_authors": "See weaknesses part. If the authors can show at least one type of attack (either a reconstruction attack or an attribute inversion attack), it would strengthen the paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4BkvTEuLKI", "reviewer": "Reviewer_qpMn", "summary": "This work proposes a synthetic data generation method with a differential privacy guarantee. It leverages a lightweight data generator and a topic model for topic clustering. This method tackles the limitation of the existing DP fine-tuning method, which is expensive and relies on large LLM, and the prompt-based methods, which heavily rely on manual prompts. The extensive experiments on both generative and classification tasks show the advantage of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your encouraging remarks that our method is reasonable, our experimental design is both proper and comprehensive, and that our analyses are well-supported with most claims being appropriately justified.\n\n### Privacy Budget Allocation\nOur budget allocation follows [1], our \"Post-Generation Resample\" baseline, which also has both DP-histogram and DP-finetune steps. For DP-histogram, we adopt the same -- adding a Gaussian noise 10 on every histogram item. The overall privacy budget (ε) is computed as a composition of both steps using the standard `dp_accouting` package.\n\nWe also show individual ε values below on PubMed, under composed $ε=4,2,1$. DP-histogram only takes a small portion of the overall budget. This aligns with observations in [1] (page 5) and highlights our CTCL-topic design achieves strong performance while consuming only a small portion of the privacy budget.\n\n| ε (Composed) | ε (DP-Histogram) | ε (DP-Finetune) |\n|:-:|:-:|:-:|\n| 4 | 0.39 | 3.96 |\n| 2 | 0.39 | 1.94 |\n| 1 | 0.39 | 0.9 |\n\nWe also find that the impact of changing the allocation is insignificant. For example, in $ε=4$ setting on PubMed, increasing histogram's noise from 10 to 20 only alters the DP-finetuning budget $ε$ marginally—from 3.96 to 3.99. \n\n[1] Privacy-Preserving Instructions for Aligning Large Language Models, ICML 2024\n\n\n### Generator design choices\nWe use a seq2seq generator because its encoder is well-suited for understanding the conditions. For size, the 140M model is to balance the efficiency with generation ability, making it practical for resource-constrained scenarios.\n\nWe agree other model selections might perform better, but we do not extensively explore all designs because our intuitive choice already demonstrates strong downstream performanceas in our experiments. Moreover, same as all pretraining research, due to the cost and scale, it could be super challenging to exhaustively search over all possible configurations.\n\nWe would also like to emphasize that the core contribution of our work is the overall CTCL framework. Each module within the framework remains flexible and open to further design improvements.\n\n**140M model can learn 430M data well?** The large dataset is collected based on the common intuition that larger pretraining datasets typically lead to stronger models. The goal of pretraining is not for the open-ended pretraining task, but for a model with strong controllability supporting downstream tasks. Moreover, large data even on comparatively small models is a common practice. For instance, LLaMA-3 with 3B or 8B parameters are trained on 15T tokens, where the data samples also significantly exceeds the number of model parameters.\n\n### Topic model design choices\nWe choose Wikipedia for our topic model because it is large-scale, high-quality, and semantically diverse. In practice, it is rare to encounter texts that are entirely outside of its scope. The table below with sampled topics of our five diverse datasets illustrates that a broad range of universal topics are captured:\n\n| Dataset | Sample topic 1 | Sample Topic 2 |\n|--|--|--|\n| PubMed | microscopy/… | rna/gene/… |\n| Arena | gameplay/rpg/…    | volcano/… |\n| Chat | comedian/…  | oscar/… |\n| Yelp | restaurant/…    | theater/… |\n| OpenReview | grammar/… | computational/… |\n\nThe \"out-of-domain\" text is also considered and processed in our experiments--in our topic model, there is an “unclassified” bin for samples that are not close to any topic. The table below shows the details\n\n| Dataset | Training size | unclassified samples | Example |\n|-|:-:|:-:|--|\n| PubMed | 75,316 | 0 | |\n| Arena | 180,000 | 86 | i have no idea who am i, maybe you can help? |\n| Chat | 17,940 | 0 | |\n| Yelp | 1,939,290 | 4 | I am unable to provide check number. |\n| OpenReview | 8,396 | 0 | |\n\nWe can see the number of unclassified samples is neglectable across all datasets, and those unclassified samples typically contain little to no substantive content, confirming the broad coverage of our topic model.\n\n### Determine the number of clusters\nFirst, the number should not be too small, so as to keep the topic conditions meaningful. Second, it should not be too large, as that would amplify per-item DP-histogram noise in downstream tasks. To strike a balance, we set a threshold that each cluster must contain at least 100 out of 6M Wikipedia articles, resulting in 1,300 clusters.\n\n\n### Quality guarantee of the proposed method\nOur framework introduces a universal topic model and controllability pretraining, which enables the model to capture valuable high-level, topic-wise information in the private domain—beyond the word-level representations learned by the vanilla DP-finetuning baseline. As a result, the data quality in our method should be guaranteed no worse than that of vanilla DP-finetuning. This quality improvement is empirically supported by the performance gains demonstrated in our experiments and further validated by our ablation study in Section 4.3.2.", "claims_and_evidence": "Most claims in the work are properly justified. There are some minor points to be clarified.\n\n> \"Third, PE-based methods need to balance between data quality and synthetic data size (see discussions in §2), while our framework naturally allows for unlimited data samples using the DP finetuned generator, without additional privacy costs during generation.\" \n\nThis sentence mentions the limitation of data quality of PE-based methods, but it needs clarification on the quality guarantee of the proposed method.", "methods_and_evaluation_criteria": "**Method**\n\nThe whole method is reasonable and indeed provides DP guarantees. I have some confusions to be clarified.\n\n1. The privacy budget.\n\nWhile the authors mention in Appendix A, I think a more detailed privacy analysis should be provided. Since CTCL involves two DP processes, the histogram mechanism and DP fine-tuning, and the two processes are sequential, it is crucial to discuss the budget allocation for the two processes. I am also curious if the allocation will influence the utility.\n\n2. The choice of generator and topic model\n\nFrom my perspective, the generator determines the quality of the synthetic data, and the accuracy of the topic model also influences the construction of the histogram. Could the authors provide some intuitions on the choice of the architecture and size of the generator? I guess a larger size of generator can improve the quality but introduce more computation costs for fine-tuning, so the balance is crucial. I am also a little worried if a 140M model can learn the 430M dataset well. Also, how the number of clusters for the topic model is determined, as it is a universal model that needs to accommodate unseen texts?\n\n\n**Evaluation**\n\nThe evaluation is proper.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental designs are proper and comprehensive. The analyses are well supported. \n\nSimilar to my concern in Method, could the author provide more details on the budget allocation of the proposed method in experiments? And what is the influence of the budget allocation?", "relation_to_broader_scientific_literature": "The proposed method handles limitations of existing DP methods including DP fine-tuning on large models and prompt-based methods.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The whole paper is well written and easy to follow.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See above reviews.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The biggest strength of the paper is that it performs better than prior work, as presented in Table 3. However, the main concern is that the method closely resembles technique proposed in [1], where the generator is DP-finetuned on private data and the DP-histogram is used to better align the distribution of the generated samples with the private data. Both methods follow this two stage approach. All Reviewers lean towards accept.
Decision: Accept (poster)