Private Federated Learning using Preference-Optimized Synthetic Data
Authors: Charlie Hou
Mei-Yu Wang
Yige Zhu
Daniel Lazar
Giulia Fanti
Venue: ICML 2025 poster (2025)
Keywords: Differential privacy
large language models
synthetic data
federated learning
preference optimization
reinforcement learning
Primary area: social_aspects->privacy
TL;DR: We cast private on-device learning (under the synthetic data framework) as an LLM preference optimization problem, and greatly improve the state of the art.

Abstract: In practical settings, differentially private federated learning (DP-FL) is the dominant method for training models from private, on-device client data. Recent work has suggested that DP-FL may be enhanced or outperformed by methods that use DP synthetic data  (Wu et al., 2024; Hou et al., 2024). The primary algorithms for generating DP synthetic data for FL applications require careful prompt engineering based on public information and/or iterative private client feedback. Our key insight is that the private client feedback collected by prior DP synthetic data methods (Hou et al., 2024; Xie et al., 2024)  can be viewed as an RL reward. Our algorithm, Policy Optimization for Private Data (POPri) harnesses client feedback using policy optimization algorithms such as Direct Preference Optimization (DPO) to fine-tune LLMs to generate high-quality DP synthetic data. To evaluate POPri, we release LargeFedBench, a new federated text benchmark for uncontaminated LLM evaluations on federated client data. POPri closes the gap in performance between the fully-private and non-private settings by up to 58%, compared to 28% for prior synthetic data methods, and 3% for state-of-the-art DP federated learning methods. The code and data are available at https://github.com/meiyuw/POPri.

Private Federated Learning using Preference-Optimizzed Synthetic Data
Charlie Hou' Mei-Yu Yu Wang" " Yige Zhu"
Daniel Lazar
Giulia Fanti!
Abstract
els. Examples include Google's 's GBoard (Hard et al., 2019; Xu et al., 2023b; Wu al., 2024) and Apple's mobile auto- matic speech recognition system (Paulik et al., 2021). Today, federated learning (FL) is the most widely-used approach in practice for learning on-device models; it trains models locally on user devices and aggregates model updates on a central server (McMahan et al., 2017b). FL protects the pri- vacy of client data in part by adopting differentially private (DP) (Dwork, 2006) optimization techniques, combination we refer to as DP-FL (McMahan et al., 7b; Kairouz et al., 2021b; Nguyen et al., 2022: Xu et al., 2023a).
In practical settings, differentially private feder- ated learning (DP-FL) is the dominant method for training models from private, on on-device client data. Recent work has suggested that DP-FI. may be enhanced or outperformed by methods that use DP synthetic data (Wu et al., 2024; Hou et al., 2024). The primary algorithms for generating DP synthetic data for FL applications require careful prompt engineering based on public information and/or iterative private client feedback. Our key insight is that the private client feedback collected by prior DP synthetic data methods (Hou et al., 2024; Xie al., 2024) can be viewed as an RL reward. Our algorithm, Policy Optimization for Private Data (POPri) harnesses client feedback using policy optimization algorithms such as Di- rect Preference Optimization (DPO) to fine-tune LLMs to generate high-quality DP synthetic data. evaluate POPri, we release new federated text benchmark for uncontaminated LLM evaluations federated client data. POPri closes the gap in performance between the fully- private and non-private settings by up to 58%. compared to 28% for prior synthetic data methods, and 3% for state-of-the-ane DP federated learn- ing methods. The code and data are available https: httthtbt.ttttttgtg//g com/ /emmmmiiinn seeeyyuw/PPPP
With breakthroughs in large language model (LLM) ca- pabilities (Anil et al., 2023; Team et al., 2023; Achiam et al., 2023; Guo et al., 2025) several research teams have used LLMs to better train models on private client data. common strategy applies standard optimization algorithms (e.g.. DP stochastic gradient descent, DP-SGD (Abadi et al., 2016)) to fine-tune models on private client data (Kurakin et al., 2023; Charles et al., 2024). These approaches have an important limitation in the on-device setting: most LLMs today are too large to fit on client devices, let alone train on them (Radford et al., 2019; Touvron et al., 2023).
a
To sidestep the size issue, Wu et al. (2024); Hou et al. (2024) view the problem of learning from distributed, private client data (partially) as DP synthetic data problem. These ap- proaches use LLM- M-sasssseed workflows to generate privacy- preserving synthetic data, similar to client data, at the server; then they train the on-device model the server on the syn- thetic data. This avoids storing the LLM on client devices.
In more detail, Wu et al. (2024) use prior public information about the clients to create LLM- -e.--eeee synthetic data for pretraining. For example, for their Google GBoard virtual keyboard application, they use prompts like "Imagine you are a [GENDER] of age [AGE]. Write some examples of chat messages." " to generate synthetic samples. This prompt was designed entirely using prior qualitative information about the data client devices. However, prior information may not always be available. Moreover, this prompt was not refined based on clients' realized data, which could limit the relevance of the resulting synthetic data.
1. Introduction
Many important machine learning (ML) applications feature sensitive datasets that distributed across client devices (e.g. mobile devices). Such ML models are often hosted on client devices. These on-device models offer privacy, la- tency, and storage benefits relative to centrally-hosted mod-
'Equal contribution 'Department of ECE, Cameeii Mellon University, Pittsburgh, PA "Pittsburgh Supercomputing Center, Pittsburgh, USA 'Coldrays, Tucson, AZ. Correspondence to: Char- lie Hou <hou.charlie2e @ gmail.com>.
PrE-Text (Hou et al., 2024) instead uses Private Evolution (PE) (Lin al., 2023; Xie et al., 2024; Lin al., 2025b) to leam prompts that relevant to client data. PE itera-
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Private Federated Learning using Preference-Optimized Synthetic Data
1. Generate synthetic data from LLM
Common stops
1. Just shake off the leoves.
Private Data
R
bioRxiv Dataset, < 1.0
Privately
40-
Prompt
was blank spaon.
candidate synthetic samples
Accuracy
LLM
DPO
3. Aggregate
35
SFT
private scores
(Noisy) Scores
30
Private Evolution (PE) Update the prompt using private scores
PE+SFT (baseline) Fine-tune with PE synthatic data
POPrI (Ours)
4. Fine tune with Policy Optimization
M
DP-Feshigg DP-FTRL
PE+
POPrI
Algorithm
Figure 1. Left: Private Evolution (PE)-based techniques. Clients generate www-immnniioall statistics which summarize the similarity of the synthetic data to their private samples. These are privately aggregated to refine the synthetic data generation for future iterations. Traditional PE (brown) uses a prompt-based method. POPri (blue) improves a naive fine-tuning method (PE+SFT, purple) by fine-tuning the LLM using policy optimization rather than tue-tuning directly on aggregated client feedback. Right: Next-token prediction accuracy on the bioRxiv dataset at privacy level e = 1. POPri closes the accuracy gap between the fully-private and non-private settings by 58 %, compared to 23% for prior synthetic data methods, and 3% for DP federated learning methods.
tively sends synthetic data samples to clients for feedback; each client privately measures the closeness of synthetic samples to their own data, discarding irrelevant samples. It returns this feedback to the central server, which crafts new prompt based on the most relevant synthetic samples. Finally, an LLM uses the generated synthetic data to fine- tune a downstream model. This method of utilizing LLMs for on-device learning has some shortcomings: (1) it relies entirely on prompting to teach the LLM to generate relevant synthetic data, which may not be as effective as fine-tuning the weights. (2) It discards irrelevant samples, which may themselves contain valuable information, as shown in rein- forcement learning with human feedback (RLHF) (Ouyang et al., 2022).
(2) We create and maintain LargeFedBench, a new uncon- taminated benchmark of federated client data separated by client for the era of LLMs. The datasets in this benchmark consist of: (1) congressional records in English-speaking countries, and (2) abstracts from bioRxiv, collected starting in April 2023. To our knowledge, this is the first dataset that provides researchers with both (a) over 1,000 clients (congressional records contains 134k clients and bioRxiv contains 57k as of August 2024), and (b) regular updates, allowing researchers to easily filter data to avoid contam- inated evaluations (Magar & Schwartz, 2022; Zhou et al., 2023; Yang et al., 2023; Roberts et al., 2023).
(3) We demonstrate the utility of POPri on this new bench- mark set of datasets, as well as two central (i.e., the setting where all data is present on the server and no server-elient communication needed) DP benchmarks from prior work (Yu al., 2023; Xie et al.. 2024). Across all datasets and tasks (we consider next token prediction and text classifi- cation), POPri achieves the best downstream metrics. For example, Figure shows that on our bioRxiv dataset at a privacy level of e = 1.0, POPri outperforms PE-based algorithms by 6 full percentage points, and closes the gap between fully private and non-private baselines by over 58%, compared to 23% for PE. It outperforms DP-FL-based meth- ods by even more. Additional experimental details, results, and ablations are provided in Section 5.
In this paper, we demonstrate how to better utilize LLMS for on-device learning: we propose POPri (Policy Opti- mization for Private Data), an algorithm that reformulates synthetic data-based approaches for private learning as LLM policy optimization problem. In POPri, we directly fine-tune an LLM's 's weights to improve the (DP-noised) sim- ilarity scores between generated synthetic data and private client samples. The fine-tuned LLM is used to generate synthetic data, which used to train a downstream model.
Contributions. In summary. our contributions are:
(1) We propose POPri, a novel method that casts private learning under the synthetic data framework as an LLM policy optimization problem. Prior work in this space relied on PE, which uses client feedback exclusively to generate new prompts (Hou et al., 2024; Xie et al., 2024). We alter this feedback to instead provide client rewards, and sub- sequently exploit recent advances in policy optimization (Rafailov et al., 2023). This recasting allows us to more effectively exploit the capabilities of LLMs for on-device learning problems.
2. Problem Statement and Background
2.1. Problem Statement
We consider a set S of clients, S = (SI..... S. where S, {s(") (3) i E [n], and m; denotes the number of text samples held by client i. We consider the partial participation setting, where only a subset of clients can participate in communication with the server at any point in time (Kairouz et al., 202 la;
(i) ,..., Sm.) denotes the private text data of client
2

---

Private Federated Learning using Preference-Optimized Synthetic Data
McMahan et al., 2017b), which is consistent with practi- cal private on-device learning deployments. We assume L clients participate in each round & < T and denote this set St. We do not assume an a priori upper bound on m,. A central server is given a pre-trained downstream model Ob which it wants to align with the private client data S. We call the aligned downstream model Φ. In the process of learning 2. the server may make use of a pre-trained public LLM V. We observe that T and T are different models in general; we will assume the server has access to the weights of both o and V. The server is subject to two restrictions: (I) client data cannot leave client devices, and (2) the final model must protect user-level differential privacy (DP):
cerns, central servers cannot conduct this private fine-tuning.
An alternative approach is to train models directly on client devices, using server to coordinate information exchange between clients; in DP federated learning (DP-FL) (McMa- han et al., 2017b; Kairouz et al., 2021a), (small) model weights are iteratively sent to clients for on-device DP opti- mization. DP-FL has struggled to keep up with the growing size of LLMs; many LLMs cannot be stored or trained client devices (Collins et al., 2023). Recent work ex- plores how to train LLMs in the DP-FL framework. Pro- posed approaches include training only subsets of param- eters (Charles et al., 2023), as well as memory-efficiem zero order optimization (Zhang et al., 2024; Malladi et al., 2023). However, these methods still require the storage of the entire model on-device, limiting their practicality.
User-level (distributed) differential privacy (DP). We say two datasets S and S' are neighboring if they differ in at most one client's 's data. That is, there exists an i [n] such that for all j A i, Sn- Sy. A randomized mechanism M is (c, 6)-DP if, for any pair of neighboring datasets S, S' that differ by an entire client's data and any possible output set F., it holds that Pr[M(() E E] S e'Pr[M(S') E E] 66. 8. The post-processing property of a DP mechanism ensures that any data-inde inte-indeeendent transformation applied to its output preserves the same DP guarantees. (Dwork, 2006; Dwork & Roth, 2014).
Synthetic data-based approaches. An alternative ap- proach to DP optimization involves generating private syn- thetic data using LLMs, followed by directly fine-tuning downstream models. Synthetic data can be generated on the server side, which bypasses client-side hardware constraints. The post-processing property of DP also implies that DP synthetic data can be used repeatedly without incurring addi- tional privacy loss (Yue et al., 2023a). In the centralized DP setting (where the server trusted to gather all the data, as opposed our private on-device setting), prior studies have shown that training downstream models on DP synthetic text achieves performance comparable to privately training on real data (Yue et al., 2023a; Mattern et al., 2022; Xie et al., 2024). In the private on-device setting, Hou et al. (2024) show that fine-tuning a small model on user-level DP synthetic text data on the server side can actually outper- form DP-FL, with a significant reduction in communication and computation cost. Similarly, Wu et al. (2024) show that pretraining an FL model on private synthetic data can improve the final outcome of DP-FL.
We also evaluate on central DP baselines, so we define central DP below; in that case the final model & should protect central DP:
Central (example-level) differential privacy (DP). We say two datasets (both fully present the server) S {si.....Sw, } and S' = fs!...., 5' are neighboring if they differ in at most one example's 's data. That is, there exists an i E [m] such that for all j + i, 8j = s'y. A randomized mechanism M (c. 8)-DP if, for any pair of neighboring datasets S, S' that differ by one sample and any possible output set E, it holds that Pr(M(S) E E] < ePP((M(S') E E| + 6.
One approach for generating synthetic text data is to fine- tune an LLM (with DP-SGD) on private data (Kurakin et al., 2023; Yu et al., 2024) and then using the LLM to gener- ate synthetic data. However, client hardware constraints render this approach infeasible on-device. Recent works have relied instead on privacy- aware prompt engineering to generate synthetic data (Wu et al., 2024; Xie et al., 2018; Hou et al., 2024). An important framework by Lin et al. (2023; 2025a) called Private Evolution (PE) is the basis for several competitive DP synthetic text algorithms, including Aug-PE (Xie et al., 2024) and PrE-Text (Hou et al., 2024). Roughly, these algorithms use the public LLM V to gener- ate synthetic data, score each synthetic data according to its closeness to the client data, and discard synthetic data with low scores. The surviving synthetic data are used as in in-context examples for to generate synthetic data. In con- current work to ours, Zou et al. extend the PE framework to
Goal.
The server seeks an algorithm to optimize the down- stream performance (in our paper, this is either next token prediction accuracy or text classification accuracy) of & on a test set of private data, subject an (e, 8)DP constraint.
2.2. Related Work
There are two main approaches for learning on private data.
DP optimization-basel approaches. In natural language processing (NLP) tasks with privacy constraints, DP opti- mization algorithms (e.g., DP-SGD (Abadi et al., 201 6)) are often used to fine-tune massively pretrained LLMs on private data (Bommasani & Schofield, 2019; Kurakin et al., 2023; Charles et al., 2024). However, in settings where client data cannot leave client devices due to privacy con-
3

---

Private Federated Learning using Preference-Optimized Synthetic Data
generate synthetic data from multiple pretrained language models (LMss), and present "good" and "bad" responses to the LMs in the next round for in-context learning (Zou et al., 2025). Private Evolution may sacrifice data quality in two ways: First, it uses in-context learning, which is often less effective than fine-tuning (Mosbach et al., 2023). Second, discarding low-score synthetic data may lose useful infor- mation (Ouyang et al., 2022). We address both by turning the DP synthetic generation problem into LLM policy optimization problem.
statistics as the most promising option, as they have lower communication and storage costs, and they make better use of the privacy budget. In departure from PE, we design the low-dimensional statistics collected by POPri to enable building a preference dataset. We ask the server to generate J samples from each of K prompts; each client then scores the K x J samples according to how well they represent the client's data, and the server aggregates the scores for all the synthetic samples. Using these scores, the server can construct "higher scoring response" and "lower scoring response" pair (a "preference pair") for each of the K prompts. The benefit of this new design ties directly to the next question.
3. POPri
The core idea of POPri (Policy Optimization for Private Data) is natural reformulation of private on-device learning from synthetic data as an LLM policy optimization problem, which enables the use of powerful LLM alignment methods like DPO (Rafailov et al., 2023). In this section, we detail the POPri design principles and algorithm. POPri's design is based on two related questions.
2. How should we use client leedback?
Given vector summarizing the quality of synthetic data samples, how should we use it? A few options arise:
(I) In-Context Learning. We could use the highest-scoring synthetic samples as in-context examples to prompt the LLM V. This is the PE approach (Hou et al., 2024; Xie et al., 2024). However, in-context learning typically performs worse than finetuning-based approaches (Mosbach et al., 2023), and we find experimentally that POPri outperforms
1. What client feedback should we collect for fine- tuning? Three natural options arise:
Private Evolution (PE) (Figure 1, Table 1).
(1) DP Data. Clients could directly transmit DP synthetic data samples for fine-tuning, e.g., using method like DP- Prompt (Utpala et al., 2023). DP-Prompt uses an LLM to summarize text at a temperature specified by the desired DP c level. However, DP text cannot be aggregated into a single statistic, which prevents the use of secure aggregation (Bonawitz et al., 2016); this increases the noise needed to reach a given DP guarantee. As such, prior work has shown that DP-Prompt is not competitive with other private on-device learning methods (Hou et al., 2024). We favor eegregation-compalee representations of client data, such as summary statistics or model parameters.
PORM
PE SFT
Evalustion Data
Figure 2. 2-PCA visualization of synthetic data from POPri and PF+SFT, and evaluation data. We see that POPri's synthetic data distribution (Ieft) is much closer to the evaluation data distribu- tion (right) than the PE+SFT synthetic data distribution (middle). Naive fine-tuning with SFT on PE- PE-generated synthetic data does not make best use of client feedback.
(2) DP Model Parameters. A second alternative is to send the parameters of either the LLM V or the downstream model & to the client and train on the private samples with DP-SGD (Abadi et al., 2016). These parameters are compat- ible with secure aggregation (Bonawitz al., 2016), which makes more efficient use of DP budget. However, V/ cannot be sent to clients because of client storage constraints. Send- ing & is the DP-FL. approach, which is one of our baselines.
(2) Supervised Fine-Tuuiing (SFT). One could directly fine- tune the LLM T the highest scoring samples using next- word-prediction loss. This is analogous to the SFT baseline evaluated in the RLHF (Ouyang et al., 2022) and DPO (Rafailov et al., 2023) papers, which showed that RLHF and DPO outperform SFT. The reason is that the highest scoring samples-while better than the low-scoring samples-are not perfect responses to the prompt. The SFT loss trains the LLM to treat high-scoring samples perfect responses, which is misaligned with the LLM's task. Empirically, we see that this approach (PE+SFT) produces synthetic data that is not representative of the private data (Figure 2) and has poor downstream performance (Table 1).
(3) DP Statistics. Finally, we could collect low-dimensional statisties capturing the quality of synthetic data samples. In PE, the server generates K synthetic data samples (Xie et al., 2024; Hou et al., 2024), and each client computes histogram counting how often each of the private samples is closest to one of the K samples. This K -ddimennionnll histogram can be made DP by adding (comparatively) little noise, and is amenable to secure aggregation (Xie et al., 2024; Hou et al., 2024). We view such liwensnnsional
(3) Policy Optimization (PO). Policy optimization-
4

---

Private Federated Learning using Preference-Optimized Synthetic Data
Algorithm 1 POPri
prompt is given in Appendix B. For each of the K prompts, we generate J synthetic samples (by running the prompt independently J times). In total, the server generates K x J synthetic samples, embeds them using a small sentence embedding model I and sends the embeddings to every client in St, i.e. the clients sampled in round t.
I: Input: Clients private data kejn), Number of rounds T, Number of generated samples Neyu. Noise multiplier a, LLM V, embedding model I. base prompt 7, participating clients in each round S "rejected" index e, random prompt generator A(-), number of clients sampled L
2: Output: Synthetic data S1yn,T+1
3:
4: All clients i E [n]] embed private samples, E. = I(S.)
2. Scoring synthetic data using DP client feedback.
5: Server initializes LJM 1 = V
Next, each client in S scores the synthetic samples. Specifically, each client calculates, for each of the K x J synthetic samples, its cosine similarity with each of the client's private samples, averaged over the client's samples (Algorithm 4). The use of cosine similarity differs from PE, which uses a nearest neighbors histogram (Lin et al., 2023; Hou et al., 2024; Xie et al., 202in- cosine similarity is critical to the performance of POPri as we found in our ablations (see Section 5.2). These similarities for every synthetic sample are arranged into vector. We clip this vector to a norm of 1, which caps the contribution of each client (similar to how gradient updates are clipped per client in DP-FL (McMahan et al., 201 Clipping is done primarily for privacy reasons, as we will elaborate later. Clipping also ensures that the contribution of clients with large amounts of data does not overwhelm the contribution of clients with small amounts of data. We then add (O, o²1/L) (where I is the identity matrix of size KJ X J)) noise to the resulting vector to ensure DP (o² controls the (e, o)) Finally, we aggregate scores via secure aggregation (Bonawitz et al., 2016), yielding a DP score for each synthetic sample that reflects its relevance to client data.
6: for t 14+-1 1.. T do
7: Server:
8: Initialize the response vector R==0
9: for k 4- 1... K do
10: Generate prompt Tjk- A(r),
11: Generate responses Haj = V (.(n)). [J]
12:
end for
13:
Send embeddings Esyn. (I(RR)))eEIK)) to
clients in S"
14:
15:
Client S.:
16:
Scores,, + SIMILARITY(E (Eayn,t, E.)
17:
Send Scoress. (N o I/LL)) Server
18:
19:
Server:
20:
Secure aggregate scores: Scores: = J. Eies Scores,, Set P(k,,] the j-th highest score response for prompt nu. according to Scores;
21:
22:
Initialize preference dataset P. =
23:
for k 1 K do
24:
Select positive synthetic sample: P.[k: 1] P.(k, 1]
25:
Select negative synthetic sample: Ptlk; 2] P. [k,
26:
end for
27:
Fine-tune: 1/+1 + DP (Th beE[K) : Pt))
28:
end for
3. LLM Policy Optimization. The key insight of our paper is that by generating J synthetic samples from K prompts and scoring all of them using DP client feedback, we can create a preference dataset where for each of the K prompts, we can assemble a "good sample" and a "bad sample". This design choice allows the usage of power- ful LLM policy optimization algorithms (we choose DPO (Rafailov et al., 2023)) finetune the LLM V. In detail, each of the K prompts have J synthetic samples which are ranked according to the scores we gathered. Then for each of the K prompts, we set the highest scoring sample as the "chosen sample" and the f-th highest scoring sample as the "rejected sample". This resulting preference dataset can then be passed, along with the LLM V, into the DPO preference optimization loss (Rafailov et al., 2023):
29:
Server:
30:
Output final synthetic data Seyn,T+1 from Ur
methods like DPO (Rafailov et al., 2023) instead directly op- timize the LLM to produce higher-scoring samples (where the score can be defined by the user of the algorithm). In other words, they are designed to directly make of the low-dimensional scores we collect from client feedback. Hence, we expect such methods to produce higher quality synthetic data, as evaluated on downstream tasks.
3.1. POPri Algorithm
Pseudocode can be found in Algorithm 1. We highlight the algorithmically new steps (that differ from PE) in blue
[ log (7 (olog( (yL + log( Viar(Vullr)) T (vr(a))
min E. V
Tett Vre(UUrrez (3rlzz)
1. Synthetic sample generation. We generate K prompts (details in Appendix B.2). A prompt is generated by randomly sampling three samples from Ω and prompting LLaMA-3-88 (Touvron et al., 2023) to generate a fourth sample given the first three samples as examples. The exact
where Ver a fixed checkpoint for the LLM (we use the pub- lie checkpoint of the LLM), F is a parameter controlling deviation of v from Met, x is the prompt. Yo is the chosen sample, y/ the rejected sample, (yy is the probability
5

---

Private Federated Learning using Preference-Optimizzd Synthetic Data
Table 1. Accuracy (%, 1) of different algorithms on variety of tasks and datasets (bioRxiv, Congress, PubMed are sext-token-prediecte accuracy. OpenReview is text classification accuracy). The highest accuracy across all methods is in bold. All standard deviation error bars are less than 0.5.
Datasel
Method
Data Type
On-device Model
c
1
0
DP-FedAvg DP-TRR PF
Original
29.0
28.3
Original
29.0
28.2
bioRxiv
Syntheric
DisiiGPT2
41,5
31.0
31.1
27.9
+ SFT
Synthetic
28.6
28.6
POPr (ours)
Synthetic
34,4
34,8
DP-FedAvg DP-FIRL
Original
29.1
29.0
Original
29.1
29.0
Congress
Symthetic
DistiGPP1
35.7
27.3
27.0
PE SFT
Synthetic
27.1
27.1
POPri (ours)
Synthetic
30,6
30.4
PE POPri (ours)
Alama-2-7b-ehat- Synthetic (2000)
27.5
PubMed (Yue al., 2023b)
BERT
47.6
Opt-6.7b, Synthetic (2000)
27.9
Synthetic (2000)
29.4
Openkeeviw (Xie et al., 2024)
PE PE
Llama-2 Mlama-2-bb-chlle 2-Tb-chat-bi. Synthetic (2000)
Opi 6.7h, Synthetic (2000)
RoBERTANece
50.8
37,0
32.1
POPri (ours)
Synthetic (2000)
40.2
of generating y given x for V and 8 is the sigmoid func- tion. The expectation is taken with respect to the empirical distribution (i.e. real samples). The DPO loss will be used to finetune to generate more samples similar to the cho- sen sample and fewer like the rejected sample. To reduce GPU memory use, we use LoRA (Hu et al., 2021) on all the attention matrices and up/down projection matrices with a rank of 4, a = 8. After fine-tuning over the K prompts and preference pairs, we return back to step (2) and generate new synthetic data using the newly fine-tuned V.
and Shakespeare plays. These datasets pose two evaluation challenges: (1) They pre-tokenize inputs a non-invertible way, which prevents researchers from using custom tok- enizers adopted by several LLMs. (2) The datasets may lead to contaminated evaluations. As state-of-the-arr LLMs have been trained on large swaths of the public internet, old public benchmark datasets may be in the training data of many L.MSS (Magar & Schwartz, 2022; Zhou et al., 2023; Yang et al., 2023; Roberts et al., 2023). To our knowledge, one work proposes a benehmark dataset for federated L.MSs (Ye al., 2024). The datasets in this paper have at most 747 clients, which may be insufficient for simulating pro- duction use cases. Further, they do not explicitly avoid contamination.
4. Synthetic data generation for downstream tasks. Us- ing the final version of Y we generate large set of syn- thetic data T+ which is used to fine-tune & into &. & then sent to all the elient devices, where they can perform inference without communicating information the server.
We release LargeFedBench, a benchmark comprising two new datasets, Congressional Speeches and bioRxiv, for ex- periments over federated client data. These datasets (a) allow researchers easily avoid contamination, and (b) pro- vide enough distinct clients to simulate production settings.
Privacy guarantees. Because each client's vector is clipped to 1, and the only information revealed to the server (or any other party) is the aggregated vector, the sensitivity of the algorithm is 1. We add (O, I noise to each client's vector, so the vector given to the server has noise (O, o² o2)). satisfying the Gaussian Mechanism with sensi- tivity 1. To calculate privacy, we can use privacy accoun- tant like OPACUS ACCOUNTANTS. ANALYSIS RDP (Yousefpour et al., 2021), and input T (the number of rounds we run the algorithm, g (the fraction of clients sampled per round), 8, and set o to get the desired value.
Congressional Speeches ("Congress")) is dataset of 134k speeches or debates scraped from congressional or parlia- mentary transcripts in the US UK, and Canada. We treat each speech as a separate client, and samples are created as successive 64-token spans within the speech. bioRxiv² is a dataset of 57k abstracts, each of which we consider a client dataset of strings, scraped from biology papers. Samples are 64-token spans of the abstract. More details on the datasets are included in Appendix F.
4. I Large Bergehh: A Federated Benchmark for LLM Evaluation
A key feature of our datasets is that they are updated every
Seryylavender/ooggresssioooo
Today, the most widely-used evaluation datasets for feder- ated learning of text models come from the work of Reddi et al. (2020); they include text from SteckOverflow posts
#nttpp://huggiingffae.co http://aaaa.ecccaaaaeeeeeteeeeee taayllavedder/iioxxi--bbbtt
6

---

Private Federated Learning using Preference-Optimizzd Synthetic Data
bioRxiv Dataset, E = 1.0
FedAvg
40
POPri
Accuracy
35
30
1,000
5,000
10,000
Clients per round
Figure 3. Next-token prediction accuracy performance of four methods as function of the number of clients sampled per round of 10000. We see that across different client participation scenarios, POPri consistently performs the best.
6 months and sorted by date. Hence, researchers can cas- ily select datasets that were generated after their model's knowledge cutoff date. In this paper, we use data from LargeFedBench published between the dates of April 2023 August 2024 to avoid contamination with the latest LLM evaluate our algorithms with, LLaMA-3-SB (AI@) Meta, which has a knowledge cutoff of March 2023.
choose DistilGPT2 (Sanh et al., 2019) as the downstream on-device language model for the LargeFedBench evalua- tions, which has only 82M parameters, and BERT as the downstream model for the PubMed evaluation to be con- sistent with Xie et al. (2024). For synthetic text generation (using the LLM V) set the maximum sequence length to 64 for the bioRxiv and Congressional Speeches evaluations and 512 for PubMed/OOenRRevie.
5. Experiments
Text classification: We use I.L for the LLM V (Touvron et al., 2023) to ensure our evaluation was not contaminated, as the knowledge cutoff for LLaMA-2- 7b-chat-hf is September 2022 (before the publish date of the ICLR 2023 reviews). For the embedding model we use the 'sentence-t5-xl" sentence transformer (Reimers & Gurevych, 2019a), and use RoBERTapase as the downstream model to be consistent with Xie et al. (2024).
Datasets and tasks. For next token prediction accu- racy, we evaluate POPri on the LargeFedBench datasets (Congress and bioRxiv), as well PubMed (Yu et al., 2023; Xie et al., 2024) used in the evaluation of Private Evolution (Aug-PE) (Xie et al., 2024). PubMed contains abstracts of medical papers published between August 1-7, 2023 (details in Appendix D.2.2). For text classification, we evaluate POPri on Reen consisting of ICLR 2023 reviews published on November 5, 2022 which was used in the evaluation of Private Evolution (Aug-PE) (Xie al., 2024). Note that PubMed and OpenReview are evaluations in the central DP setling, where the entire dataset is present on the server and no server-client communication is needed. To execute POPri on PubMed, we use the central DP version of POPri, detailed in Algorithm 2. For OpenReview, we employ conditional generation (similar PE (Lin et al., 2023; Xie et al., 2024), where the generation is conditioned on being given class to generate. The (central) conditional generation version of POPri is detailed in Algorithm 3.
Metries. We primarily evaluate each method on accuracy (next-token or text classification) of the final downstream on-device model O In some ablations we also measure the distance of the synthetic dataset to the private dataset using the Fréchet Inception Distance (FID) (Heusel et al., 2017). During training, we evaluate the models on the validation dataset and select the checkpoint that achieves the best validation performance as the model that is evaluated the test set.
Baselines. We compare POPri to several baselines: (I) DP-FedAvg (McMahan et al., 2017a) (2) DP-FTRL (Kairouz et al., 2021a) (3) Private Evolution (PrE-Text (Hou et al., 2024) and Aug-PE (Xie et al., 2024)). DP-FedAvg and DP-FTRL directly privately fine-tune the downstream model & on the client data. Private Evolution (PrE-Text and Aug-PE) generates synthetic data on which the down- stream on-device model & is finetuned. Note that on the PubMed and ReenReview dataset, we compare to Aug-PE results obtained with models of similar size to the model we use (7B-8B parameters) and which are not potentially
Models. Next token prediction tasks: We use LLaMA-3-
8B for the LLM V (Grattafiori al., 2024), which has a knowledge cutoff date of March 2023 (AI@ Meta, 2024). For embedding models (used in measuring semantic dis- tance between text samples), we use "all-MiniLM-LL6: sentence transformer (Reimers & Gurevych, 2019b). We
SSimply speaking, score synthetic data generated for each class separately but combine the preference datasets into one to finetune the L.M V.

---

Private Federated Learning using Preference-Optimized Synthetic Data
contaminated (i.e. model was possibly trained on the bench- mark dataset). We also include = 0 (fully private) and = Do (fully non-private) baselines. The 0 baseline for the LargeFedBench evaluations evaluates the public Distil- GPT2 checkpoint on the test sets with no further fine-tuning- The E = oo baseline is the downstream model finetuned directly on the private training set centralized on the server with no noise. The e = 0 baseline for OpenReview is the accuracy obtained by predicting everything to be the most populous class. More details about the setup can be found in Appendices C and D.2.
communication cost comes from each client downloading
K = 1800 sentence embeddings of size 384 resulting in download cost of 700K (700,000) floats, and uploading a histogram of size 1800 resulting in an upload cost of 1800 floats. The client runtime cost comes from calculating a nearest neighbors histogram and the server runtime cost for PE comes mainly from using the LLM Y to generate syn- thetic samples each round. In POPri each client downloads K x J = 1800 10 sentence embeddings for download cost of 7M floats and uploads a vector of size 18,000 for an upload cost of 18,000 floats. The client runtime cost of POPri comes from calculating the cosine similarities, and the server runtime comes from both using to generate synthetic samples and running DPO.
Privacy Analysis. All baselines use privacy guarantee of (f. o)-DP where 5=3x 10-6 and e=1 or <=7 for each of the bioRxiv and Congressional Speeches datasets, For (Nprie
Interpretation. In summary, POPri is much more communication--effcceen and client compute than FedAvg, while using much more server compute. On the other hand, POPri is generally more communication- and computationally-expennn POPri has the best downstream performance among all three methods, as seen in Table 1. Hence, POPri can be a suitable method when (I) server compute is cheap and powerful, and (2) getting the best synthetic data/downstream model quality is important.
PubMed/OpenReview, we set ó < is the number of private samples). We follow the privacy accounting method detailed in Section 3 for POPri. Details for all baselines are in Appendix D.1.
than PE. At the same time,
5.1. Main Results
Table lists the accuracy (next token prediction and text classification) achieved by baseline methods (DP-FedAvg, DP-FTRL, Private Evolution) and POPri. In this table, we assume full participation (no client sampling) for fair com- parison to baselines, some of which do not have client sam- pling versions. We find that POPri outperforms all the base- line algorithms. Furthermore, in the e = setting POPri closes the gap between fully private learning (e = 0) and fully non-private learning (t = oc) by 40-58% depending on the setting, compared PE which closes 1-28%. For all methods tested, the measured accuracy values do not de- pend strongly on This has been observed in prior work on DP synthetic data using LLMs (Xie et al., 2024; Hou et al.,
5.2. Ablations
Cosine similarity vs. Nearest neighbors histogram. Pri- vate Evolution (Lin al., 2023; Hou et al., 2024; Xie et al., 2024) uses a DP nearest neighbors histogram calculation to score the quality of synthetic samples. The DP nearest neighbors histogram sets the score of a particular synthetic sample to the number of private samples that are closest to that particular synthetic sample (under some text em- 2024). POPri outperforms Private Evolution (Aug-PE) even when holding our synthetic sample budget to 2000. Note that synthetic samples are cheap in POPri (we could gener- ate many more) because we have access to the full model, while Xie et al. (2024) only assume access a model API.
bedding function). In POPri, we instead set the score of a particular synthetic sample to the average cosine similarity between that particular synthetic sample and all private sam- ples (under some text embedding function). We find that cosine similarity works much better than a nearest neighbors histogram (Figure 6), possibly because nearest neighbor his- tograms produce sparser scores, often assigning zero to all synthetic samples associated with a given prompt. In this setting, the chosen and rejected samples for preference op- timization end up being essentially random. In contrast, cosine similarity provides denser scoring that allows the construction of meaningful preference pairs for all prompts.
Cost analysis case study. In Table 2 we analyze the per- round communication and computation costs (and per-client, for download/uuloodd/liinn runtime costs) of FedAvg (a representative and cheap method among the DD-FL- methods), PE, and POPri on the bioRxiv dataset experiment with 1000 clients sampled per round.
For FedAvg each round the sampled clients download and upload the downstream model, which in our case is Dis- tilGPT2. This is an 82M (82 million) parameter model leading to a download and upload cost of 82M floats. The client runtime cost comes from local gradient computation, and server runtime is negligible because the server only needs to average model deltas from the clients. For PF, the
Partial client participation. In each round fixed num- ber of clients is subsampled uniformly at random for feed- back generation. Figure 3 shows the next-token prediction accuracy (%) of four algorithms for different numbers of clients per round. POPri consistently outperforms all of the baselines, regardless of the client sampling rate. Moreover, POPri's 's accuracy is not sensitive to the client sampling rate.
8

---

Private Federated Learning using Preference-Optimizzd Synthetic Data
Text embediing daAldata (d-2)
RoundO FID+0.38
Round8 FIDeO 10
Round? FID-O 07
0.35
0.30
0.25
0.20
0.15
0.10
1bt1 15
Rounda FID-0.15
1o FID-0 26
Evaluation Dataset
Medtan th
1211
Round
Figure 4. PCA visualization of POPri synthetic data embeddings over rounds. Right Panels: PCA-2 plots for synthetic data and evaluation data emheddings from the best checkpoint each round for 20 iterations. The orange (round 7) and maroon point clouds represent the round with the lowest FID score and the validation dataset, respectively. Top Left Panel: FID score vs. rounds. Bottom Left Panel: Median distance to the medoid vs rounds. Running POPri for too many rounds appears to cause overfitting.
Method
Download (floats)
Upload (floats)
Client Runtime
Server Runtime
(GPU sec)
(GPU see)
FedAvg
82 million
82 million 1,800
4.8
PE
700,000
0.0027
326.25 13,547.84
POPri
million
18,000
0.01
Reduction factor (FedAvg POPri) Reduction factor (PE/ / POPri)
11.71x
4555x
480.0x
0.100x
0.100x
0.270
0.024x
Table 2. Table setting, Communication and computation cost comparison per round (and per-client for downtosd/uplooad/ciinn runtime cost) across methods the bioRxiv dataset with 1000 clients sampled per round. Download and upload measured in floats; runtimes are measured in GPU seconds (lower is better). "Reduction factor (X / POPri)" is the cost of method X divided by the cost of POPri for the given resource; green reduction, red is an increase. Server runtime for FedAvg is left blank as it is negligible compared other methods. Overall, we view POPri as suitable when server compute is relatively cheap, and improved sample quality is important enough to justify higher on-device communication and computation costs relative PE (Table 1).
Data Distribution Evolution. Synthetic datasets are often generated using language model distinct from the one being aligned (Guo et al., 2024), making the alignment phase inherently off- off-policy training. This is reflected in the synthetic data, where the FID score (relative to a held-out evaluation set) worsens after improving. Figure shows PCA visualizations of synthetic data embeddings across alignment iterations, while the left panels plot the FID score and median distance to the medoid in the PCA space. The data distribution transitions from being initially clustered to (roughly) matching the true data distribution, back to being clustered, likely due to overfitting. Early stopping based on validation metrics can help.
is too dissimilar to client data, then the preference pair uninformative. However, choosing a sample that is too similar to client data (e.g., rank 2) for the rejected sample could lead to incorrect preference pairs due to DP noise swapping rankings. We use the 5th-ranked sample, and justify it experimentally in Appendix E.3.
the model evolves during
6. Conclusion
Private on-device learning important when data is stored edge devices with hardware, storage, and privacy con- straints. We propose POPri, which recasts synthetic data- based approaches for private learning as an LLM policy optimization problem. POPri makes several novel design choices in how it gathers and utilizes client feedback to gen- erate DP synthetic data, which is used to finetune a down- stream on-device model. POPri outperforms DP-FL and synthetic data baselines on variety of tasks, including large-scale LargeFedBench, new federated benchmark we have curated.
How to select rejected samples. Unlike vanilla DPO, we can select the "chosen" and "rejected" sample pair from the J samples for each of the K prompts. We consistently choose the highest-scoring sample (rank 1) as the "chosen" sample, but there are different options for the "rejected" sample. We found that the middle-ranked sample (e.g., 5th-ranked out of J = 10) yields the best results, rather than using the last-ranked sample. the rejected sample
9

---

Private Federated Learning using Preference-Optimizze Synthetic Data
Impact Statement
Security, pp. 308-318. ACM, 2016.
In this paper, we train models satisfying differential pri- vacy guarantees. When using differential privacy tool for protecting user data, is important to communicate to users what the privacy guarantees mean to be able to obtain informed consent. The algorithms in this paper also use LLMs, which were trained on large scale public text data. While this data was public, explicit consent may not have been given for its use in training the models. The algorithms using LLMs in the paper make no claims about the privacy guarantees of data used in the pretraining of the LLMs.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I.,
Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint vrXiv:2333.08774 2023.
AI@ Meta.
Llama 3 model card. 2024. URL amaatta/aaaaaaaaaaaaaaaaaaaaaaaa 11amm3//////m https://gitie CARD ..RD.md. mdd blob
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
While work aims to show how synthetic data can be useful for federated learning, it also poses a number of ethical risks, including the generation of biased or harmful content. In particular, our method (and all variants of private evolution) inherits the biases and undesirable aspects of the public LLM. For example, suppose the public LLM only generates text in English, but some clients' private data is all in Spanish. In these settings, clients would be forced to vote on synthetic samples, even if potentially none of them are relevant to the client. This may cause the client to contribute data reinforcing model that is actively not useful (or even harmful) to the client. contrast, DP-SGD methods do not suffer from this shortcoming, because they do not rely on public LLM. This problem raises an important point- how design DP synthetic data algorithms in which clients can stem the biases or failures of the public LLM, based their own data? This important question is beyond the scope of the current paper.
Boerner, T. J., Deems, S., Furlani, T. R., Knuth, S. I.., and Towns, J. Access: Advancing innovation: Nsf's advanced cyberinfrastructure coordination ecosystem: Services & support. In Practice and Experience in Advanced Re- search Computing 2023: Computing for the Common Good, pp. 173-176. 2023.
Bommasani, R., W. S. and Schofield, X. Towards private synthetic text generation. In NeurIPS 2019 Machine Learning with Guarantees Workshop, 2019.
Bonawitz, K. A., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H. B., Patel, S., Ramage, D., Segal, A., and Seth, K. Practical secure aggregation for feder- ated learning on user-held data. In NIPS Workshop on Private Multi PartiiParty Machine Learning, 2016. URL https:/ /tt////xxiv .org/ abs 16166111000 04482.
Acknowledgments
This work was supported in part by NSF grants CCF- 2338772 and CNS-2148359, as well C3.ai, Bosch, In- tel, and the Sloan Foundation. This work used Bridges-2 GPU (Brown et al., 2021; Buitrago & Nystrom, 2021) at the Pittsburgh Supercomputing Center through allocation CIS240135 and CIS240937 from the Advanced Cyberinfras- tructure Coordination Ecosystem: Services & Support (AC- CESS) program, which is supported by National Science Foundation grants 2138259, 2138286, 2138307, 2137603, and 2138296 (Boerner et al., 2023). The authors acknowl- edge the National Artificial Intelligence Research Resource (NAIRR) Pilot, the Al and Big Data group at the Pittsburgh Supercomputing Center, and NCSA Delta GPU for con- tributing to this research result.
Brown, S. T., Buitrago, P., Hanna, E., Sanielevici, S., Scibek, R., and Nystrom, N. A. Bridges-2: plat- form for rapidly-evolving and data intensive research. In Practice and Experience in Advanced Research Com- puting 2021: Evolution Across All Dimensions, PEARC 221, New York, NY, USA, 2021. Association for Com- puting Machinery. ISBN 9781450382922. 01553437339.3465599 URL 10.1145/343738 34655933.
doi: 10.
Buitrago, P. A. and Nystrom, N. A. Neocortex and bridges-2: A high performance ai+hpe ecosystem for science, discovery, and societal good. Communica- tions in computer and information science, 1327, 2021. doi: 10.1007797883--03--663333 //par wwwp://ww..gobbbiooooooo22//
URL https:
References
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I.. Talwar, K., and Zhang, L. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference Computer and Communications
Charles, Z., Mitchell, N., Pillutla, K., Reneer, M., and Gar- rett, Z. Towards federated foundation models: Scalable dataset pipelines for group-structured learning. arXiv preprint arXiv:2307.094 2023.
10

---

Private Federated Learning using Preference-Optimizee Synthetic Data
Charles, Z.., Ganesh, A., McKenna, R., McMahan, H. B., Mitchell, N. E., Pillutla, K., and Rush, J. K. Fine-tuning large language models with user-level differential privacy. In ICML 2024 Workshop on Theoretical Foundations of Foundation Models, 2024.
Kairouz, P., McMahan, B., Song, S., Thakkar, O., Thakurta, A., and Xu, Z. Practical and private (deep) learning without sampling shuffling. In ICML, 2021a.
Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al. Advances and open problems in federated learning. Foundations and Trends@ in Machine Learning, 14(1-2):1-210, 2021b.
Collins, L., Wu, S., Oh, S., and Sim, K. C. Profit: Bench- marking personalization and robustness trade-off in fed- erated prompt tuning, arXiv preprint arXiv:2310.04627, 2023.
Kurakin, A., Ponomareva, N., Syed, U., MacDermed, L., and Terzis, A. Harnessing large-language mod- els to generate private synthetic text. arXiv preprint arXiv:2306.01688 2023.
Dwork, C. Differential privacy. In International colloquium on automata. languages, and programming, pp. 1-12. Springer, 2006.
Dwork, C. and Roth, A. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., 9(3-4):21 1-407, August 2014. ISSN 1551-305X. doi: 10.1561/04000000044, httht:::: www.dooooooo..oooo 1561/040000004
Lin, Z.,, Sekar, V., and Fanti, G. On the privacy properties of gan-generated samples. In AISTATS, pp. 1532-1530. PMLR, 2021.
Lin, Z., Gopi, S., Kulkarni, J., Nori, H., and Yekhanin, S. Differentially private synthetic data via foundation model apis 1: Images. arXiv preprint arXiv:2305. 15560, 2023.
Grattafiori, A.. Dubey, A., and et al. The llama 3 herd of models, 2024. URL https: /atpii///aivv..... org/ abs/
Lin, Z., Baltrusaitis, T., Wang. W., and Yekhanin, S. Dif- ferentially private synthetic data via apis 3: Using sim- ulators instead of foundation model. arXiv preprint arXiv:2502.05505. 2025a.
Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q. Ma, S., Wang, P., Bi, X., et al. Deepseck-rl: In- centivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Lin, Z., Baltrusaitis, T., Wang, W., and Yekhanin, S. Differ- entially private synthetic data via apis 3: Using simula- tors instead of foundation model, 2025b. URL https:
Guo, S., Zhang, B,, Liu, T., Liu, T., Khalman, M., Llinares, F,, Rame, A., Mesnard, T., Zhao, Y., Piot, B., Ferret, J. and Blondel, M. Direct Language Model Align- ment from Online AI Feedback. arXiv e-prints, art. Xxiv:2402.047922. February 2024. doi: 10.48550/arXiv. 2402.04792.
/ wlaaxiivorgg/aaaae.a httl//ab//2222 250.. 05505.
Magar, I. and Schwartz, R. Data contamination: From memorization to exploitation, 2022. URL https:// arxiv. wtt. abs// 2203222232222 08242.
Malladi, S., Gao, T., Nichani, E., Damian, A., Lee, J. D., Chen, D., and Arora, S. Fine-tuning language models with just forward passes. Advances in Neural Information Prucessing Systems, 36:53038-53077 2023.
Hard, A., Rao, K., Mathews, R., Ramaswamy, S., Beau- fays, F., Augenstein, S., Eichner, H., Kiddon, C., and Ramage. D. Federated learning for mobile keyboard org/ abs/
prediction, 2019. URL 1811.03604 03604.
Mattern, J., Jin, Z., Weggenmann, B., Schoelkopf, B., and Sachan, M. Differentially private language models for secure data sharing. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp- 4860-4873, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 10.18653/v1/2022.emmll 333n.323. URL https:// ac aclanthology hww.:/ollo.o.oog////// em0022eemmml1m. 1ww.mlpmmmmnm..mm.........mmm.mm 323.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by two time-scale update rule converge to local nash equilibrium. NeurIPS, 30, 2017.
Hou, C., Shrivastava, A., Zhan, H., Conway, R., Le, T., Sagar, A., Fanti, G., and Lazar, D. Pre-text: Training language models on private federated data in the age of llms, 2024. URL httpp:://arxit org/ abs/ 2406. 02958.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and Learning of Deep Networks from Decentralized Data. In Singh, A. and Zhu, J. (eds.), Proceedings of the 20th Interna- tional Conference on Artificial Intelligence and Statis- tics, volume 54 of Proceedings of Machine Learning
Arcas, B. A. y. Communication-FFficnn
Hu, E. J. Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.

---

Private Federated Learning using Preference-Optimizee Synthetic Data
Research, pp. 1273-1282. PMLR, 20-22 Apr 2017a. URL wtt.htttss:pprcceeddiii mlr wtw.mm.....ar...... press/v54/ saathann7aahht....
Reimers, N. and Gurevych, I. Sentence-bert: Sentence em- beddings using siamese bert-networks. In Proceedings of the 2019 Conference Empirical Methods in Natu- ral Language Processing. Association for Computational I Linguistics, 11 2019a. URL https:/ /ttpx/..xxo.///////////////
McMahan, B., Ramage, D., Talwar, K., and Zhang, L. Learn- ing DP recurrent language models. 2017b.
Mitchell, E. A note on dpo with noisy preferences & re- lationship to ipo, November 25 2023. URL https: //eri cmi tccell 11..1... ai /adi//dpo... pdf. Version 1.1.
Reimers, N. and Gurevych, I. Sentence-BERT: Sentence embeddings using Siamese BERT-networks.. In Inui, K., Jiang, J., Ng, V., and Wan, X. (eds.), Proceed- ings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3982-3992, Hong Kong, China, November 2019b. Association for Computational Lin- guistics. doi: 10. 10.18653//11/D1--1410 URL https: / //eel//ttbbooooyyooo/////////-11
Mosbach, M., Pimentel, T., Ravfogel, S., Klakow, D., and Flazar, Y. Few-shot fine-tuning vs. in-context learn- ing: A fair comparison and evaluation. arXiv preprint 2.Xiv:2305.16338 2023.
Nguyen, J., Malik, K., Zhan, H., Yousefpour, A., Rabbat, M.. Malek, M., and Huba, D. Federated learning with buffered asynchronous aggregation. In International Con- ference Artificial Intelligence and Statistics, pp. 3581- 3607. PMLR, 2022.
Roberts, M., Thakur, H., Herlihy, C., White, C.,, and Dooley, S. Data contamination through the lens of time, 2023, URL
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
Sanh, V., Debut, L., Chaumond, J., and Wolf, T. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
Paulik, M., Seigel, M., Mason, H., Telaar, D., Kluivers, J. van Dalen, R., Lau, C. W., Carlson, L., Granqvist, F, Vandevelde, C., Agarwal, S., Freudiger, J., Byde, A., Bhowmick, A., Kapoor, G., Beaumont, S., Áine Cahill, Hughes, D., Javidbakht, O., Dong, F., Rishi, R., and Hung. S. Federated evaluation and tuning for on-device personalization: System design applications, 2021. URL https: //axiiv///xvvv org/ abs/ 22w...222...0 08503.
Team, G., Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, M., Hauth, A., Millican, K., et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. 2023.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Turc, I., Chang, M.-W., Lee, K., and Toutanova, Well-read students learn better: On the importance of pre-training compact models, 2019. URL https:/ arxiv. arxiv.org/abs. 1908 08962.
Rafailov, R., Sharma, A., Mitchell, E., Manning, D. Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 53728-53741. Cur- ran Associates, Inc., 2023.
Utpala, S., Hooker, S., and Chen, P.-Y. Locally differ- entially private document generation using zero shot prompting. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 8442-8457, Singa- pore. December 2023. Association for Computational Linguistics. doi: 2211666511122233.nnnnng--emm 566. URL https: //a///alaaaatccccao org/20233 f Findings-emnlp.. 566/.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019.
Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konečný, J., Kumar, S., and McMahan, H. B. Adaptive federated optimization. arxiv preprint arXiv:2003.00295, 2020.
Wu, S. Xu, Z., Zhang, Y., Zhang, Y., and Ramage, D. Prompt public large language models to synthesize data for private on-device applications. arXiv preprint arXiv:2404.0m 04360, 2024.
12

---

Private Federated Learning using Preference-Optimizze Synthetic Data
Xie, C., Lin, Z., Backurs, A., Gopi, S., Yu, D., Inan, H. A., Nori, H., Jiang, H., Zhang, H., Lee, Y. T., Li, B., and Yekhanin, S. Differentially private synthetic data via foun- dation model APIs 2: Text. In Forty-first International Conference on Machine Learning, 2024. URL https: //openreview. net forum? 1d=1ND7upglob.
Okazaki, N. (eds.), Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Vol- une I: Long Papers), pp. 1321-1342, Toronto, Canada, July 2023a. Association for Computational Linguistics. doi: 10. 10.186553/112022334-oooogg
URL. https: long. 74.
aclanthology. http://..orr.//oooo wwt.:oggg///// -oogoooo...oooon.......
Xie, L., Lin, K., Wang, S., Wang. E., and Zhou, J. Dif- ferentially private generative adversarial network. arXiv preprint arXiv: 188800007779 2018.
Yue, X., Inan, H. A., Li, X., Kumar, G., McAnaalen, J., Shajari, H., Sun, H., Levitan, D., and Sim, R. Synthetic text generation with differential privacy: A simple and practical recipe, 2023b. URL https::: httpp:s///xxxv.xrr/ abs/ 2210/221000 14348.
Xu, Z., Collins, M., Wang, Y., Panait, L., Oh, S., Augenstein, S., Liu, T., Schroff, F., and McMahan, H. Learning to generate image embeddings with user-level differential privacy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7969- 7980, 2023a.
Zhang, L.,, Li, B., Thekumparampil, K. K., Oh, S., and He, N. Dpzero: Private fine-tuning of language mod- els without backpropagation. In Forty-first International Conference on Machine Learning, 2024.
Xu, Z., Zhang, Y., Andrew, G., Choqueete-Choon C. A., Kairouz, P., McMahan, H. B., Rosenstock, J., and Zhang, Y. Federated learning of ghoard language models with differential privacy, 2023b. URL org/ /bs//bs//20 2305 18465.
Zhou, K., Zhu, Y., Chen, Z., Chen, W., Zhao, W. X., Chen, X., Lin, Y., Wen, J.-R., and Han, J. Don't ' make your Ilm an evaluation benchmark cheater, 2023. URL https: / //arxiv. org/ /abs///b///// abs 222.1..0164...
Zou, T., Liu, Y., Li, P., Xiong, Y., Zhang, J., Liu, J., Ye, X., Ouyang, Y., and Zhang, Y.QQ. Contrastive private data synthesis via weighted multi-plm fusion. arXiv preprint arXiv:2502.00245, 2025.
Yang. S., Chiang, W.-L., Zheng, L., Gonzalez, E., and Stoica, Rethinking benchmark and contamination for language models with rephrased samples, 2023. URL bttps://aarxe org/ abs/ 2211 04850.
Ye, R., Ge, R., Zhu, X., Chai, J., Du, Y., Liu, Y., Wang, Y., and Chen, S. Fedllm-benchh Realistic benchmarks for federated learning of large language models. arXiv preprint Xix::22406. 044454 2024.
Yousefpour, A., Shilov, I., Sablayrolles, A., Testuggine, D., Prasad, K., Malek, M., Nguyen, J., Ghosh, S., Bharadwaj,
A., Zhao, J., Cormode, G., and Mironov, I. Opacus: User-friendly differential privacy library in pytorch. In NeurIPS 2021 Workshop Privacy in Machine Learning, 2021. URL https: ptstts::/opee.eee wppp//pweeeeee.ee////eee. wwt. forum? id Id-EopKEYBoI-
Yu. D., Backurs, A., Gopi, S., Inan, H., Kulkarni, J., Lin, Z., Xie, C., Zhang, H., and Zhang. W. Training private and efficient language models with synthetic data from LLMs. In Socially Responsible Language Modelling Research, 2023. URL https: http:://///ee.eeeerreee www.eewww.eeeeeeeeeeeeeeeeeeeeee net/forunt id=FKwtKzg1Fb.
Yu, D., Kairouz, P., Oh, S., and Xu, Z. Privacy-preserving instructions for aligning large language models, 2024. URL https:ttt://. http/aivvvv.ooo/ www.vv.ooorraaoooooooo wtt..r///..../........2 13659.
Yue, X., Inan, H., Li, X., Kumar, G., McAnallen, J., Sha- jari, H., Sun, H., Levitan, D., and Sim, R. Synthetic text generation with differential privacy: simple and practical recipe. In Rogers, A., Boyd-Graber, J., and
13

---

Private Federated Learning using Preference-Optimizzd Synthetic Data
A. Algorithmic Details
Here we provide the pseudocode for the centralized version POPri and the central conditional generation version of POPri.
Algorithm 2 POPri (central DP, unconditional)
1: Imput: Number of iterations T, Noise multiplier LLM V, embedding model I', base prompt vy. random prompt generator A(-).
"rejected" index private dataset S, number prompts, J number of responses per prompt
2: Output: LLM for generating synthetic data Vr+1
3:
4: Embed all private samples E = T(S)
5: Initialize LLM 1
for <- T do
7: Initialize the response vector R =
8: for k + 1. K do
9: Generate prompt Ti
A(1)).
Generate responses Ray Me (m))). j [J]
end Calculate embeddings Fayn. = (T( (Rst,))))
12:
13:
Scores; +- COVTRALSCORE(F.vn... F) + (O, r²
14:
Set J[k.j] as the j-th highest score response for prompt according to Scores;
15:
Initialize preference dataset P.
16:
for k + I...K K do
Select positive synthetic sample: Pilk, 1] [k 1 18: Select negative synthetic sample: Pi(k, 2 = Polk;
19:
end for
20:
Fine-tune: Me+1 + DPO( (mo F Pt)
21:
end for
22:
Output W++1
Algorithm 3 POPri (central DP, conditional)
I: Input: Number of iterations Noise multiplier a, LLM v, embedding model base prompt 7), conditional (class-specifiede random
prompt generator A(:-)) "rejected" index f, private dataset K number prompts, number responses per prompt, number classes
2: Output: LLM for generating synthetic data V1
3: Embed private samples for each class i 1...B, E. I(((s)(()=i.ees) where F(s) is the class index of sample
Initialize LLM vi= W
5: for + 1 ... 1' do
6: Initialize B response vectors R = 10,...03 = (R()) R(B}}
7: for b 4--... do
8: for 1... K do
Generate prompt n/
A(11 b),
Generate responses k/0)) V (((m)
11:
end for
12:
Calculate embeddings (000 (I [EE(K)JEE()
13:
Scores; + CENTRALSCORE( EShi Es) + oF)
14:
Set j] j] as the j-th highest score response for prompt according to Scores;
15:
Initialize preference dataset p.!)) 8
16:
for & 1...... K do
17:
Select positive synthetic sample: P!" [k; 1] P.lk. 1]
18:
Select negative synthetic sample: p [ ( Set prompt (b) [k.3] = m7
19:
20: end
21: end for
Fine-tune: P: U p(() Ven < DP.(
22:
23:
end for
24:
Output VT+1
Below the similarity scoring function we use for the federated setting.
14

---

Private Federated Learning using Preference-Optimizze Synthetic Data
Algorithm 4 SIMILARITY
I: Input: Set of embeddings of private client data E; {emb(s] )..,. emb(sm;) for E St, embeddings of synthetic data Esyn. total synthetic samples M = K x Scores 00
2: Scores[j] = mi, Ee 2 E, | ||]e,
(epri.e)) for E,
E Faye
3: return Scores/ pivres/mmxx1111 1, ||Scores||]
Below is the similarity scoring function we use for the central DP setting.
Algorithm 5 CENTRALSCORE
1: Input: Embeddings of private data E, embeddings synthetic data Esyn
Scores t- OM
2: Scores[j] (1/1 ( Eemes [egelllle;
(0prise4) for ej E Esyn
3: return Scores
B. Implementation Details of POPri
B.1. Model and Hyperparameters
We choose ILLMA-3-8B the data generator in POPri and we fine-tune iteratively during the course of the algorithm. To fine-tune the LLaMA-3-SB model, we use LoRA fine-tuning with rank 4, = 8, applied to all the projection matrices in LLaMA-3-8B B-B We adapt the AdamW optimizer with cosine learning rate scheduler with the learning rate ranging from 3 10-7 to 8- 10-7. In the Congress and bioRxiv evaluations, the sample set a subset of the c4 dataset (Raffel et al., 2019), which is large scale dataset from 2019, which we use for fair comparison with Private Evolution (PrE-Text), though we not know their exact initial sample set because they did not release it. For the PubMed evaluation, the sample set ? set of 2000 samples generated using the PubMed generation prompt Table 16 of the Aug-PE paper. generated by L.L.MA---BB-Instten
(which has knowledge cutoff of March 2023), for comparison with Aug-PE (Xie et al., 2024). For each iteration, we fine-tune the models for 2 epochs and select the best checkpoint with the lowest FID score relative to the validation dataset. This checkpoint is used for synthetic data generation and as the starting point for the next iteration. The batch size is set to 24.
In each round we generate 18000 synthetic data samples for the clients to evaluate. This accomplished with 1800 prompts, each generating 10 samples for clients to rank. We select the 1st and Sth ranked sample for a given prompt for the "selected" and "rejected" data samples in the DPO preference dataset. We describe the experiments regarding which rank use for constructing the preference dataset in detail in Appendix Section E.3. To test the scaling relation with the number of clients per round and the total number of clients participating in the training, we set up the parameters and privacy budget shown in Table 3. The 'all-Mini
sentence transformer model is used as the embedding model in POPri. We note that we adopt "sentence-t5- -base" sentence transformer for PubMed during the step of fine-tuning BERT small, which follows the setting in AUG-PE. We ensure POPri follows privacy guarantee of (c, 6)-PP = (1, 3 x 10-6) or (7, x 10 6) for both the bioRxiv and the Congressional Speeches datasets and run with 20 iterations for DP-FedAvg, DP-FTRL, PrE-Text for comparison. For AUG-PE, we set 0)-P) (1, 2.72 10- 6) or (4, 2.72 10 6). PubMed experiments are run with 10 iterations.
In terms of models for downstream tasks:
For BioRxiv & Congressional Speeches, we fine-tuned the pre-trained DistilIGPT2 for next-token prediction. We set the sequence length 64, number of generated synthetic data 1,000,000, the batch size 160, the learning rate as - and the number of epochs 80.
For PubMed, to compare with (Yue et al., 2023b). we follow their procedure to leverage pre- pre-trained BERT small (Ture et al., 2019). We set the max sequence length 512, number of generated synthetic data as 2000, batch size 32, learning rate as 3e-4, the weight decay 0.01, and the number of epochs 10. To compare with (Xie et al., 2024), we set up the (c, 8DDP value and hypterparameter according to their choice. For example, they set ó = Npeie-log(A (N following (Yue et al., 2023b). To achieve 5 (1,4), we use noise multiplier a 3.87) for 10 iterations under DP
15

---

Private Federated Learning using Preference-Optimizze Synthetic Data
List of 6 diverse original text samples:
Original Text Text Sample 1
The observations showed that the objject is four million times more massive than the sun and is the size of astronomicall
(AU), span equal Earth's distance from the sun. Sgr density least trillion times greater than any known cosmic object.
Original Text Sample 2
In response to the general question, they need to study self- self-pratection away from their marital bagzage. hey need to learn about home security, mobile security, the nature of crime, de-escalation, the escape tactics, awareness, and and on. When
Original Text Sample
Under the Patriot Act of 2001, the government significantly expanded its authority in regardss to electronic surveillance (Henderson, 2002). One of the chief complaints that the government can investigate anything that is considered "significant" problam here that there
Driginal Text Sample
The life history edvance program shall be Tunded from any of the following: monies provided the general fund; amounts in the presidential family partnership fund; or monies provided by the revolving fund.
Original Text Sample As you meet with employers this summer, get in touch with the team....
Figure 5. The synthetic data generation prompt for POPri. The black text marks the input prompt, and the brown text after "Original Text Sample 4" is generated. The generated text between "Original Text Sample 4" and "Original Text Sample 5" is collected and used as a synthetic sample.
on all PubMed data. Note that our noise multiplier values are slightly different than (Xie et al., 2024) due to different methods for calculating differential privacy.
B.2. Prompt Design
To compare with other data generator methods, adopt the prompts used in the baseline models against which compare. We generate the synthetic data using approach similar to that in PrE-Text (Hou et al., 2024). Figure 5 shows example of the prompt we use for prompting LLaMA-3-BI BB for generating synthetic data. For bioRxiv/Congress, we randomly take text samples from the c4 (Raffel et al., 2019) dataset as our examples in the prompt. For PubMed, while running POPri, we still adopt the prompt shown in Figure 5 but reduce the number of examples to two in order to accommodate longer sequence lengths, randomly sampling generated abstracts from LLaMA-3-88 For OpenReview, we prompt the model directly to generate paper reviews (similarly to (Xie et al., 2024)).
C. Implementation Details of Baseline Models
In this section we provide implementation details for the baseline algorithms. We use two DP-FL baselines: DP-FedAvg and DP-FTRL. For the PE baseline, we implement PrE-Text (Hou et al., 2024) for the evaluations on the bioRxiv and Congressional Speeches datasets. For the PE baselines on the PubMed dataset we directly compare against the Aug-PE results from Xie al. (2024).
C.1. DP-FedAvg
We employ the FedAvg federated optimization algorithm (McMahan et al., 2017b) to fully fine-tune DistilGPT2, avoiding linear probing due to its poor performance in DP language models (Lin et al., 2021). Our training configuration includes a batch size of 2, a sequence length of 64, and 20 rounds for Table I and 50 rounds for Figure 3, and either full partial client participation. For differential privacy (DP), noise (McMahan et al., 2017b). We evaluate the model using next-token prediction accuracy across various numbers of training epochs on the clients. We tune the learning rate within the range [0.001, 1. 0. and the clipping threshold between [0.01, 0.1, 1.0], selecting the model with the best performance on the evaluation set for reporting. The noise is scaled to ensure privacy guarantee of (c, 6DP where = 3-10- and = {1,7), representing two distinct privacy regimes. The noise multipliers are a (19.3, 3.35) when considering all the data, and the settings for partial participation experiments
utilize secure aggregation (Bonawitz et al., 2016) and introduce Gaussian
16

---

Private Federated Learning using Preference-Optimized Synthetic Data
Table 3. Experiment privacy budget settings.
Total # of clients
# of clients per round
01,c==7 01..c=1
d2".c=7
02".e=1
10000
1000
19.5
10000
5000
15.5
30.8
10000
10000
30.6
30.8
72000
72000
3.35
19.3
3.35
19.5
133000
133000
3.35
19.3
3.35
19.5
For DP-FodAvg, PrE-Text, POPr.
For DP-FTRL
are shown Table
C.2. DP-FTRL
We also use the DP variant of heroow--Th--Reulaiiiee----- (DP-FTRL) algorithm (Kairouz et al., 2021a) to fully fine-tune DistilGPT2. The hyperparameter settings are similar to DP-FedAvg other than the noise multipliers. The noise multipliers are {19.5, 3.35} when considering all the data, and the settings for partial participation experiments are shown in Table 3.
C.3. PrE-Text
We follow similar settings as Hou et (2024) with some modifications. The privacy budget similar DP-FedAvg and POPri, with privacy guarantee of (e, 6)-DP where 8 6=3 =3-10-6 and = [11,7] with {19.3, 3.35) for full participation and partial participation in Table 3. We set the thresholds H =00.1626, T= = 20, and Nagn 1024. We adopt the "all-MiniLMM L6-v2" sentence transformer model for text embedding generation.
D. Experimental Details
D.I. Privacy Accounting
The precise privacy settings we use and their corresponding values, as calculated by their corresponding privacy budget computation methods. are reported in Table De-FedAvg (McMahan al., 2017b) and Private Evolution (PrE-Text) (Hou et al., 2024) both use the Gaussian mechanism, and thus similar computations. In both cases, accountant of the Opacus library (Yousefpour et al., 2021). For DP-FedAvg, we calculate privacy by inputting the number of rounds, the client sampling ratio, setting the noise multiplier to be the product of and the clipping threshold, choosing 6 < S/ and setting a for the desired Private Evolution (PrE-Text) (Hou et al., 2024) also uses the Gaussian mechanism, so we use the same accounting except the noise multiplier is the product of a and the maximum number of samples per client. For DP-FTRL, we follow the privacy accounting methods from their implementation. For Private Evolution (Aug-PE) (Xie et al., 2024), we report their reported e directly.
the privacy
D.2. Evaluation Details for Different Datasets
1.2.1. LARGEFEDBENCI EVALUATION
For the bioRxiv and Congressional Speeches datasets, we use the PrE-Text version of Private Evolution because the PrE-Text evaluation focused on datasets with samples with max sequence length of 64.
D.2.2. PUBMED AND OPENREVIEW EVALUATION
For PubMed and OpenReview, Private Evolution baseline compares to Aug-PE, which has already been evaluated on PubMed and OpenReview (Xie et al., 2024). Note that PubMed and ReenReview was used by Xie et al. (2024) to evaluate central DP algorithms. In the central DP setting, there are no clients; all private data is held at the server and the goal is to release model with DP guarantees. The notion of neighboring dataset in central DP is a centrally held dataset that is the same except for a single data sample. To compare our algorithm directly with results reported for Private Evolution (Aug-PH) (Xie et al., 2024), we replicate the central DP setting for this dataset by having one PubMed abstract per client and
17

---

Private Federated Learning using Preference-Optimizze Synthetic Data
sampling all clients every iteration (or "round", in case).
E. Ablation Studies
E.1. Cosine similarity vs Nearest neighbors histogram
In this section we perform an ablation justifying the choice of cosine similarity as a scoring function over the nearest neighbor histogram employed by Private Evolution. We find that using cosine similarity works much better than nearest neighbors histogram for our use case, because nearest neighbors histogram is too sparse to ensure the construction of meaningful preference pairs for POPri.
bioRxiv sigma=3. 35 (eps=7) full participation of 72000 clients every round
0.7
1.0
0.6
Recovery Rate
0.8
FID Score
0.6
0.6
04
0.4
02
0.2
0.1
9 10 Round
16
16
17
8
12
13
16
17
Round
Figure Left: scores POPri using NN histogram scoring vs. POPri using cosine similarity. Right: After the client feedback stage, we measure the percentage of the time the non-note non-noised and non-clipped score (nearest neighbor histogram scoring or cosine similarity rejected sample. cosine similarity, this "recovery rate" much higher (nearly 100%)
scoring) of chosen sample is higher than in nearest neighbors histogram. Interpretation. Nearest neighhors histogram much sparser than cosine similarity, often assigning zero to synthetic samples associated with given prompt POPri. leads to preference pairs often being completely noisy. Cosine similarity provides denser scoring that allows the construction of meaningful preference pairs for all prompts.
FID Score over Rounds
0.40
0.35
FID Score
0.30
0.25
0.20
DPO
cDPO (label smoothing--.ck 0.05)
0.15
cDPO (label smoothing-0.02
12
Round
Figure this experiment, we investigate whether use label rejected samples. To do this, used the third-ranked sample as the rejected sample, and evaluated different settings for conservative DPO (cDPO) (Mitchell, 2023). used bioRxiv dataset experiment setting. set eps=7, learning rale Se-7. We find that by tuning the level of conservative-ness may be able to improve slightly on vanilla DPO.
alignment methods could allow the use of higher-
E.2. A Alignment methods
We also experiment with a noise-resistant (or robust) DPO method, conservative DPO (cDPO) (Mitchell, 2023), to see if by using it we can select a higher ranked rejected sample (recall we use the 5th ranked, and higher ranked samples would introduce more noise into the preference pairs). In Figure we find that it can help slightly when choosing a higher rejected sample ranking.
18

---

Private Federated Learning using Preference-Optimizzed Synthetic Data
E.3. Rejected sample selection
We construct the DPO preference data via client feedback by generating ten samples from the same prompt and then picking the "selected" and the "rejected" samples. The samples with the highest scores among the ten examples picked the "selected" sample in the DPO preference dataset. We experiment which rank should be utilized as the "rejected" sample in the DPO preference dataset. In Fig 8 we further explore the effects by examining the "rejected" and "selected" sample FID scores as a function of round. In the left panel where the "selected" sample FID values are shown, their magnitude and trends behave similarly before they reach the best results (marked by colored dashed vertical lines). For the "rejected" sample FID shown the right panel, the 5th rank "rejected" samples yield the lowest FID score and therefore smaller gap between the preference sample pairs. However, we also find that higher rank does not always yield better results. This may result from the boundary between the "rejected" and "selected" samples becoming undistinguishable for rank 5th due DP noise. We therefore select 5th rank samples our "rejected" DPO preference samples.
0.6
0.6
Rejected Sample FID Scare
Selected Semple Score
0.5
0.5
0.2
0.2
0.1
0.1
6
0
5
7 8 9
Round
Figure Ablation study for selecting rejected samples in the preference data. Here generate samples for each prompt and select Nth ranked data as the rejected sample, where is 5, 7, or 10. The vertical lines indicate the round at which the best wowdd-rrdedictoon accuracy was achieved for each choice of rank. Note that the model that produces the lowest overall FID (not the lowest selected sample FID or the lowest rejected sample FID) is the best synthetic data generation model, since on the final round all generated samples are utilized to form
synthetic dataset. hypothesize that round corresponds the highest accuracy for the rank model because after that point, the selected sample FID is higher than the rejected sample FID, which would mean the preference dataset has become mis- aligned with objective generating good synthetic data.
Congressional Speeches
Token Distribution Across Speaches (Max 1000 Takens)
Taken Distribution Across Speeches (Max 1000 Tokens)
60000
25000
50000
20000
40000
muquaccy
19000
20000
10000
20000
5000
10000
400
500
600 700
800
1000
500
Number Tokens
Figure The distribution of how many tokens each client's dataset for the bioRxiv Congressional Speeches datasets.
19

---

Private Federated Learning using Preference-Optimizze Synthetic Data
Table 4. Dataset details.
# Train Samples
Validation Samples
Test Samples
Max Soquence Length
Average # samples per client
bioRxiv
72000
2000
1584
6.6 2.6
Congressional Speeches
133000
4200
1547
64
5.0 + 16.3
PubMed
75316
14423
4453
512
Figure 10. A t-SNE clustering of the Congressional Speeches dataset. US data is colored in purple, UK data is colored in orange, and Canada data colored green. find the three datasets form distinct clusters and also distinct sub-clusters.
F. Datasets
bioRxiv. This dataset consists of abstracts from bioRxiv papers with appropriate copyright permission from April 2023 to August 2024. This was done by using the bioRxiv public API to retrieve the abstracts of the paper with permitted licenses (i.e. 'CC BY NC ND', 'CC BY ND', 'CC BY NC', 'CC BY', 'CCO'). This dataset consists of 72k abstracts (clients), each of which split into chunks of 64 tokens form samples.
Congressional Speeches. This dataset consists of speeches from US, UK and Canada congresioonal/prriimmnnaay transcripts from April 2023 August 2024. All speeches published under a permissive license which allows for third-party use (as detailed in the dataset cards). There are 134k speeches (clients) in total, and 1930 unique speakers. We collected this dataset by using public APIs to retrieve data from each country's 's official amngressional/aatiammenn website. Then we sanitized the data by removing (1) boilerplate procedural language, (2) sentences with more than 30% of the characters not being letters, and (3) some written notation that does not correspond to spoken words. We split each speech into chunks of 64 tokens each. We believe that this dataset a major contribution because spoken language may be more resistant contamination (especially for the UK and Canada parliamentary debates). Because they are more conversational and have large degree of improvisation (many debates off-the-cuff), they less likely to be generated by LLMs. Because Congressional Speeches contains diverse collection of speeches across speakers and also countries, the dataset forms many distinct clusters, reflecting the diversity of the dataset (Figure 10).
library
We will update the dataset periodically with the latest data to allow future researchers to test their algorithms or ideas against an uncontaminated dataset.
20


Reviews:
[{"review_id": "3khWcFCpGb", "reviewer": "Reviewer_Yas7", "summary": "This paper proposes a method for differentially private federated learning of language data, which finetunes a pretrained LLM with synthetic data generated according to client preferences. The method is lightweight in terms of client computation and client-server communication, achieves guarantees of differential privacy, and experimentally outperforms baselines. Ablation studies are also included.\n\n## Update after rebuttal\nDiscussion with the authors helped clarify a few points (choice of DP parameter $\\delta$, relative communication cost of different algorithms) and I will maintain my recommendation of acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The technical novelty of the paper is somewhat incremental. To my understanding, the algorithm is essentially an extension of Private Evolution (PE), but clients provide feedback on synthetic data through preference ranking instead of by choosing the best samples. Still, the performance of the proposed algorithm is convincing enough that I think this is a minor weakness.**\n\nThank you for your comment. You are correct that POPri is an extension of PE. However, we believe that re-interpreting PE as a preference optimization problem is not obvious a priori. \n\nPE was initially presented as an alternative to fine-tuning synthetic data models. In [1], the authors argue that PE can be used to generate DP synthetic data when fine-tuning a model is computationally impractical, but one has access to a foundation model API. Their key insight was that in this setting, one can obtain high-quality DP synthetic data by converting the synthetic data generation problem from a DP optimization problem to an iterative private selection problem, which is much lower-dimensional than private optimization. In particular, the private selection problem aims to select a single item from a set with the highest reward (according to a dataset-specific score function), subject to a DP constraint. Since [1] did not need to fine-tune the base foundation model, they were able to get away with using extremely coarse-grained feedback from the clients—namely, private selection data.\n\nOur paper adds important nuance to this understanding of private synthetic data by arguing that even fine-tuning problems benefit from asking clients to solve a private selection problem. How? By not viewing clients’ private selections as “selections” at all, but as preferences. Indeed, this perspective motivates several design choices in POPri (for example, generating J samples from each of K random prompts to enable the construction of a preference dataset) which are essential for making our method outperform PE. As you observed, our empirical results improve significantly as a result of this interpretation.\n\n[1] Lin et al, “Differentially private synthetic data via foundation model APIs 1: Images”, ICLR 2023\n\n**Can you explain the choice of $\\delta$ given in the paragraph starting on line 376, column 1? I'm curious whether the gap between the proposed algorithm and baselines might grow or shrink depending on the choice of $\\delta$. Did you ever perform any evaluations with different choices of $\\delta$?**\n\nThis is a good question. While we did not directly conduct ablations with different values of delta (due to computational limits and time constraints for the rebuttal), we did run experiments for different values of epsilon. Because the (epsilon,delta) values are computed using a privacy accounting mechanism, we can equivalently interpret the results from (epsilon=7, delta=1e-6) (Appendix E, Figure 6) as a smaller epsilon with a larger delta. For example, under the same noise and number of rounds, the result from (Appendix E, Figure 6) is also (epsilon=1, delta=0.45). Under this interpretation, we can directly compare Figures 3 and 6, as keeping epsilon constant and varying delta. Since the results change minimally across plots, our intuition is that the results are not very sensitive to delta, even across large differences. We will add a full ablation in the final version where we vary delta explicitly to verify this intuition. \n\n**Can you elaborate on the \"Cost analysis\" in Section 5.1? I'm confused about why POPri communicates 10x more embeddings than PE methods. Is this because of a fundamental difference in the algorithms, or is the number of communicated embeddings a parameter that you can freely choose? In the latter case, is it possible that the improved performance of PoPri compared to PE methods is caused by the larger number of communicated embeddings? Did you ever compare against PE methods while controlling for the communication cost? I want to clarify that this is not a major issue, as it seems the current experiments control for the privacy budget instead of the communication budget, which is reasonable. Still, I want to understand where this 10x difference comes from.**\n\nThank you for this great suggestion. We have not run experiments in which we control the communication cost. The number of communicated embeddings is a number we can freely choose (K x J), and we can choose K and J. The reason the communication is 10x is because we set K to be the same as N_syn in PE so that the POPri preference dataset had the same size as the PE iterative synthetic dataset, and set J to be 10 because that number worked well among different choices we made (in early experiments we had tried 2, 4, 6, 8, 10). \n\nWe thank the reviewer for their time in helping improve our paper!", "claims_and_evidence": "All of the claims in the paper are backed up by theoretical and experimental evidence. I did not find any problematic claims.", "methods_and_evaluation_criteria": "The evaluation methods make sense for the problem at hand. The authors introduced a dataset of federated language data, with special attention paid to avoid any data contamination, which brings rigor to the experiments and is a contribution in its own right.", "theoretical_claims": "There are no proofs in the paper. The only theory-adjacent claims are the ones related to differential privacy guarantees, but these guarantees are essentially handled by the Opacus library.", "experimental_designs_or_analyses": "I am very satisfied with the experimental design of the paper. The experiments compare against a number of relevant baselines, report error bars, carefully handle data contamination, and run interesting ablation studies.", "relation_to_broader_scientific_literature": "This paper contributes to the practice of federated learning with provable guarantees of privacy, particularly in the context of training language models. The proposed method appears very practical for use with edge devices (e.g. mobile phones), since the computation and communication costs for the clients is very small compared with classic FL methods. This line of work has important societal implications, since large tech companies can leverage existing infrastructure to immediately implement and deploy these kinds of algorithms with user devices and user data.", "essential_references_not_discussed": "I am not aware of any essential references not discussed.", "other_strengths_and_weaknesses": "Strengths\n1. The proposed algorithm outperforms baselines pretty convincingly. The gap is not huge (between 1 and 2 absolute percentage points improvement to next-token prediction accuracy), but the error bars are non-overlapping, so I think that the improvement is reliable.\n2. The experimental design is rigorous. The authors report error bars, and they introduce their own dataset to eliminate the possibility of data contamination. I also appreciate the ablation studies, which demonstrate robustness to the number of participating clients and probe the design choices of the proposed algorithm. In particular, I appreciate the ablation study in Figure 4, which uncovers a potential weakness of the proposed algorithm.\n3. The paper is very well written. All of the technical ideas are well communicated, all of the baselines and potential variations of the problem setup are explained well, and the novelty of the proposed work is clearly distinguished. The authors are also very transparent about the possible limitations of their work, which I appreciate.\n\nWeaknesses\n1. The technical novelty of the paper is somewhat incremental. To my understanding, the algorithm is essentially an extension of Private Evolution (PE), but clients provide feedback on synthetic data through preference ranking instead of by choosing the best samples. Still, the performance of the proposed algorithm is convincing enough that I think this is a minor weakness.", "other_comments_or_suggestions": "One small formatting suggestion. For the equation on line 240, column 2, I recommend to use \\left( and \\right) for parentheses, so that the parantheses are not smaller than the expressions inside of them.", "questions_for_authors": "1. Can you explain the choice of $\\delta$ given in the paragraph starting on line 376, column 1? I'm curious whether the gap between the proposed algorithm and baselines might grow or shrink depending on the choice of $\\delta$. Did you ever perform any evaluations with different choices of $\\delta$?\n2. Can you elaborate on the \"Cost analysis\" in Section 5.1? I'm confused about why POPri communicates 10x more embeddings than PE methods. Is this because of a fundamental difference in the algorithms, or is the number of communicated embeddings a parameter that you can freely choose? In the latter case, is it possible that the improved performance of PoPri compared to PE methods is caused by the larger number of communicated embeddings? Did you ever compare against PE methods while controlling for the communication cost? I want to clarify that this is not a major issue, as it seems the current experiments control for the privacy budget instead of the communication budget, which is reasonable. Still, I want to understand where this 10x difference comes from.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w8EVJQTKS7", "reviewer": "Reviewer_rhem", "summary": "In the paper, the authors present a novel approach to improving the utility of differentially private federated learning (DP-FL) by leveraging preference-optimized synthetic data generated through large language models (LLMs). The proposed method aggregates client feedback into preference pairs, and then fine-tunes an LLM to generate high-fidelity synthetic data while maintaining differential privacy (DP) guarantees.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Notations in Algorithm 2 are confusing. What’s e_pri? Also in the line 2 of the Algorithm 2, the expression of calculating histogram is hard to understand, and in the paper, there are no descriptions about the details of this line.**\n\nFirst, we would like to apologize for an oversight; Algorithm 2 is not the one actually used in our experiments. In POPri the clients actually score each synthetic sample based on the average cosine similarity between the synthetic embedding and their private data embeddings. The experimental results reported in the paper use this cosine similarity scoring method. Updated pseudocode for Algorithm 2 is below (which explains what e_pri is):\n\n>  \n> 1. **Input:** Embeddings of private client data $E_i$ for $i \\in S^t$, embeddings of synthetic data $E_{syn}$, total synthetic samples $M = K \\times J$  \n>    Scores $\\leftarrow 0^M$  \n> 2. $\\text{Scores}[j] = \\frac{1}{|E_i|} \\sum_{e_{pri} \\in E_i} \\frac{\\langle e_{pri}, e_j \\rangle}{\\|e_{pri}\\| \\|e_j\\|}$   for $e_j \\in E_{syn}$  \n> 3. **return** $Scores/\\|Scores\\|$ if $\\|Scores\\| > 1$ else Scores \n\n\nWe switched from nearest-neighbor (NN) to cosine similarity primarily performed better. As shown in this experiment on the bioRxiv dataset with full participation (https://imgur.com/a/pg1H5iq), using cosine similarity results in a significantly lower FID score compared to the NN histogram. This improvement is because NN histograms produce sparser scores—often assigning zero to all synthetic samples associated with a given prompt in POPri. The chosen and rejected samples end up being essentially random. By contrast, cosine similarity provides more informative and continuous scoring. \n\n[1] Lin et al, “Differentially private synthetic data via foundation model APIs 1: Images”, ICLR 2023\n\n\n**From figure 4, the authors only visualize when the evaluation dataset only has one cluster. How about multiple clusters? Can the proposed method can handle data with multiple clusters?**\n\nThank you for the question. Figure 4 shows the bioRxiv dataset, which is only one of our benchmark datasets. Our Congressional Speeches dataset, which contains congressional speeches from the United States, Canada, and the United Kingdom, contains multiple clusters by country: (see t-SNE plot https://imgur.com/a/IVKgbDL). Given that POPri performs the best on Congressional Speeches as well, our experiments do suggest that POPri can effectively handle data with multiple clusters.\n\n**However, the details about early stopping are missing.**\nIn Appendix B.1, we explain our early stopping criterion. Note that our FedAvg and DP-FTRL baselines also engage in the same form of early stopping (using validation set loss) for fair comparison, and PE (and PE+SFT) did not benefit from stopping early in our experiments. \n\n\n**I am just curious about whether such DPO method will work when taking rank 2 samples as the rejected samples.**\n\nThank you for the great suggestion! Dr. DPO is a strong work tackling the pairwise noise issue in DPO, and we will discuss it in the paper. We are working on an experiment comparing against cDPO (a version of DPO robust to label noise, https://ericmitchell.ai/cdpo.pdf), and will update it here. We also already compare against IPO (Gheshlaghi Azar et al 2023, https://arxiv.org/abs/2310.12036) in Table 3 of the appendix, which also prevents overfitting to incorrect labels (see https://ericmitchell.ai/cdpo.pdf), and found that it performed worse than DPO.\n\n**In figure 3, why the accuracy suddenly drops when number of clients per round reaches 10^4**\n\nIn federated learning, past work has shown that sampling more clients per round can reduce generalization performance (see Figure 3 in https://arxiv.org/abs/2106.07820, NeurIPS 2021). The authors of that work believe that the “number of clients per round” parameter is similar to batch size in centralized training, where larger batch size can lead to worse generalization (https://arxiv.org/abs/1609.04836, ICLR 2017). \n\n**Regarding baselines: the authors only includes the work DP Model Parameters type baselines, DP-FedAvg and DP-RFRL are works in 2016 and 2017, and the authors should include more recent works such as methods in [2], [3]**\n\nLoRA (used in [2]) methods generally are designed to improve the efficiency of learning, not the final model quality. LoRA (and other parameter-efficient fine-tuning variants) generally do not outperform full fine-tuning. In [2], their method performs worse than FedAvg (Full) in Table 3. [3] is out of scope for our paper because they study a setting where each client can have a separate model, whereas our setting aims to train a single model. Hence, we do not believe that either of these papers should be included as baselines. Moreover, we do not know of any algorithms that outperform DP-FTRL in the setting we are considering. \n\nWe thank the reviewer for their time in helping improve our paper, and hope that our responses addressed the reviewer’s concerns!", "claims_and_evidence": "The claims mentioned in the paper are all supported either theoretical or experimentally.", "methods_and_evaluation_criteria": "Regarding the methods, my concerns are that: \n\n1. Notations in Algorithm 2 are confusing. What’s $e_{pri}$? Also in the line 2 of the Algorithm 2, the expression of calculating histogram is hard to understand, and in the paper, there are no descriptions about the details of this line.\n\n2. From figure 4, the authors only visualize when the evaluation dataset only has one cluster. How about multiple clusters? Can the proposed method can handle data with multiple clusters? \n\n3. In section 5.2, the paragraph about the data distribution evolution, the authors mentions that early stopping can help avoids the overfitting. However, the details about early stopping are missing.\n\n4. Regarding DPO:  the authors found that directly use rank 2 response as the rejected samples is not as well as the middle-ranked samples because of DP noise swapping ranking. In the preference learning literature, there are some existing works aiming to handle such pairwise noise, for example, the Distributionally robustifying DPO (Dr. DPO)[1]. I am just curious about whether such DPO method will work when taking rank 2 samples as the rejected samples. \n\n[1] Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization. https://arxiv.org/pdf/2407.07880", "theoretical_claims": "The authors provide the privacy guarantee of the proposed method.", "experimental_designs_or_analyses": "1. Similar to Point 2, can the proposed method handle the data with multiple clusters? \n\n2. In figure 3, why the accuracy suddenly drops when number of clients per round reaches $10^4$ and the total number of clients is larger than $10^5$ in Congressional Speeches dataset? \n\n3. Regarding baselines: the authors only includes the work DP Model Parameters type baselines, DP-FedAvg and DP-RFRL are works in 2016 and 2017, and the authors should include more recent works such as methods in [2], [3]\n\n[2] Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. 2024. Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12903–12913, Miami, Florida, USA. Association for Computational Linguistics.\n\n[3] FDLoRA: Personalized Federated Learning of Large Language\nModel via Dual LoRA Tuning, https://arxiv.org/pdf/2406.07925", "relation_to_broader_scientific_literature": "1. Replacing heuristic PE with preference optimization for higher-quality DP synthetic data.\n\n2. Introducing LargeFedBench , a scalable, uncontaminated benchmark addressing gaps in prior datasets.", "essential_references_not_discussed": "DP Model Parameters type methods: \n\n[2] Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. 2024. Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12903–12913, Miami, Florida, USA. Association for Computational Linguistics.\n\n[3] FDLoRA: Personalized Federated Learning of Large Language\nModel via Dual LoRA Tuning, https://arxiv.org/pdf/2406.07925", "other_strengths_and_weaknesses": "Strengths\n\n1. The authors provide a comprehensive related work.\n\n2. The authors give a comprehensive ablation study.", "other_comments_or_suggestions": "Typo: in the past paragraph of section 2.1, “Goal” -> “Goal.”", "questions_for_authors": "See point 1, 2, and 3 in Methods And Evaluation Criteria part", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "do3E3TssWb", "reviewer": "Reviewer_xvTm", "summary": "The authors introduce a client-level differentially private (DP) federated learning algorithm that leverages synthetic data generation assisted by large language models (LLMs). Unlike previous approaches that rely only on prompting for synthetic data generation, their proposed POPri algorithm fine-tunes the LLM's weights using Direct Preference Optimization (DPO). Through extensive experiments, they demonstrate the effectiveness of POPri, showing improved performance compared to existing methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Are the authors considering trusted server or un-trusted server model? Also, why do the authors add noise both on client and server sides (as in line 17 and 20 in Algorithm 1)?**\n\nThank you for pointing out the typo; we should only add noise in line 17. We consider the server to be untrusted. Our reported (epsilon, delta) are with respect to an adversary that can see only aggregated updates at every round during training, similar to past work on FL[1,2]. We will clarify. \n\n[1] Kairouz et al, 2021 “Practical and Private (Deep) Learning without Sampling or Shuffling” https://arxiv.org/abs/2103.00039 \n\n[2] McMahan et al, 2017 “Learning Differentially Private Recurrent Language Models” https://arxiv.org/abs/1710.06963 \n\n**I don't see any privatization technique used in Algorithm 2? what do e_pri and e_j represent in Algorithm 2? Is E_i just one embedding or a collection of embeddings?**\n\nFirst, we would like to apologize for an oversight; Algorithm 2 is not the one actually used in our experiments. In POPri the clients actually score each synthetic sample based on the average cosine similarity between the synthetic embedding and their private data embeddings. The experimental results reported in the paper use this cosine similarity scoring method. Updated pseudocode for Algorithm 2 is below:\n\n>  \n> 1. **Input:** Embeddings of private client data $E_i$ for $i \\in S^t$, embeddings of synthetic data $E_{syn}$, total synthetic samples $M = K \\times J$  \n>    Scores $\\leftarrow 0^M$  \n> 2. $\\text{Scores}[j] = \\frac{1}{|E_i|} \\sum_{e_{pri} \\in E_i} \\frac{\\langle e_{pri}, e_j \\rangle}{\\|e_{pri}\\| \\|e_j\\|}$   for $e_j \\in E_{syn}$  \n> 3. **return** $Scores/\\|Scores\\|$ if $\\|Scores\\| > 1$ else Scores \n\nWe switched from nearest-neighbor (NN) to cosine similarity primarily performed better. As shown in this experiment on the bioRxiv dataset with full participation (https://imgur.com/a/pg1H5iq), using cosine similarity results in a significantly lower FID score compared to the NN histogram. This improvement is because NN histograms produce sparser scores—often assigning zero to all synthetic samples associated with a given prompt in POPri. The chosen and rejected samples end up being essentially random. By contrast, cosine similarity provides more informative and continuous scoring. \n\n\nOn privatization: we use the gaussian mechanism (line 17) where clients add noise. Their outputs are securely aggregated [3] before being given to the server. This gives the user-level distributed DP guarantee mentioned earlier.\n\n\n[3] Bonawitz et al, “Practical Secure Aggregation for Privacy-Preserving Machine Learning” https://eprint.iacr.org/2017/281.pdf \n\n**However, the approach appears to be a straightforward extension of private evolution algorithms**\n\nThank you for your comment. You are correct that POPri is an extension of PE. However, we believe that properly re-interpreting PE as a preference optimization problem is not obvious a priori. In order to do so, we had to make careful design decisions such as (1) what client feedback we should collect and (2) how we should use the client feedback. This included carefully organizing the synthetic samples and their feedback to ensure that preference datasets could be built, and experimenting/reasoning over what LLM fine-tuning method to use (DPO, SFT, IPO). \n\n**I believe it is essential to include a discussion on the overall computational complexity of the proposed method (both client and server sides) relative to existing approaches.**\n\nThis is a valid concern. The computational cost of POPri is indeed significantly larger than PE on the server side. Below, we provide a table on the server and client compute cost breakdowns when we sample 1000 clients per round on the bioRxiv dataset:\n\n| Method   | Client Runtime in GPU seconds | Server Runtime in GPU seconds     |\n|-|---|--|\n| PrE-Text | 0.0027 sec                 | 326.25 sec                      |\n| POPri    | 0.01 sec                   | 13,547.84 sec |\n| FedAvg   | 4.8 sec                    | -                               |\n\nThe client compute cost of POPri and PrE-Text are both small compared to FedAvg. However, POPri has the highest server compute cost. POPri is suitable in cases where (1) we care a lot about the downstream model quality, (2) the client computation and/or communication is expensive, and (3) server computation is relatively cheap and powerful. We will add this explanation and data to the main paper.\n\n**In Line 20 of Algorithm 1, why does the server aggregate the scores from all clients instead of the clients in S^t**\n\nThis is a typo, thank you for noticing. The server aggregates scores only from clients in S^t. We will fix it!", "claims_and_evidence": "Some of the claims are sort of confusing to me, especially on the part of privacy guarantees. In particular, I have the following questions:\n1. The threat model is unclear to me. Are the authors considering trusted server or un-trusted server model? Also, why do the authors add noise both on client and server sides (as in line 17 and 20 in Algorithm 1)?\n2. The DP nearest neighbors algorithm shown in Algorithm 2, is it a differential private algorithm since I don't see any privatization technique used in Algorithm 2?\n3.  what do $e_{pri}$ and $e_j$ represent in Algorithm 2? Is $E_i$ just one embedding or a collection of embeddings?", "methods_and_evaluation_criteria": "The proposed methods seem reasonable to me. However, the approach appears to be a straightforward extension of private evolution algorithms where POPri utilizes privatized clients responses to fine-tune the LLMs responsible for data generation. This may limit the paper's overall contribution. Additionally, I am concerned about the computational complexity of this method, as fine-tuning is likely to be significantly more resource-intensive than prompting.", "theoretical_claims": "There are no theoretical claims in the paper but I suggest the authors giving a clear argument on the privacy analysis.", "experimental_designs_or_analyses": "The experimental design makes sense to me. However, the performance improvement is not particularly surprising, given that the proposed method is significantly more resource-intensive than the baselines. Compared to DP-FedAvg and DP-FTRL, POPri benefits from LLM-assisted synthetic data generation. Meanwhile, compared to PE, POPri chooses to fine-tune the LLMs instead of prompting, which is more computationally demanding.\n\nThat said, the paper does not clearly address the cost-utility tradeoff of this approach. While it covers communication and computation complexity on the client side, most of the computation occurs on the server side. I believe it is essential to include a discussion on the overall computational complexity of the proposed method (both client and server sides) relative to existing approaches.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "In Line 20 of Algorithm 1, why does the server aggregate the scores from all clients instead of the clients in $\\mathcal{S}^t$?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tkbf01BavL", "reviewer": "Reviewer_W8ui", "summary": "The paper introduces ​POPri, a novel method for private on-device learning that leverages DP synthetic data generated via LLMs. They use ​Direct Preference Optimization (DPO) to fine-tune LLMs for generating high-quality synthetic data. POPri outperforms existing methods, in terms of next-token prediction accuracy, particularly on the newly introduced ​LargeFedBench benchmark.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**It presents very limited downstream task evaluations…It would be beneficial to see how POPri performs on other types of data**\n\nOn more tasks: Thank you for this comment—we are working on an evaluation of a text classification task. Due to the tight deadline of the rebuttal preparation period, we will provide the results in the coming days as the results finish running.\n\nOn other modalities: We completely agree that our evaluation does not allow one to generalize to other modalities. Focusing on a single modality is common in recent DP synthetic data papers, including several works that have won oral or spotlight awards at recent top ML conferences [1-5]. We will clarify in the introduction and abstract that our paper covers only the text modality. \n\n[1] Lin et al, “Differentially private synthetic data via foundation model APIs 1: Images”, ICLR 2023  \n[2] Xie, C. et al, Differentially private synthetic data via foundation model APIs 2: Text, ICML 2024  (Spotlight)  \n[3] Hou, C. et al, Pre-text: Training language models on private federated data in the age  \nof LLMs, ICML 2024 (Oral)   \n[4] Qian et al. \"Synthcity: a benchmark framework for diverse use cases of tabular synthetic data.\" NeurIPS 2023   \n[5] Dockhorn et al. \"Differentially Private Diffusion Models.\" Transactions on Machine Learning Research 2022.   \n[6] Lin et al, \"Differentially Private Synthetic Data via APIs 3: Using Simulators Instead of Foundation Model.\" 2025   \n\n\n**Why was DPO chosen over other preference optimization methods like RLHF?**\n\nThank you for your comment. We compared against IPO (Identity Preference Optimization) [7] in our paper which is a common mainstream alternative method to DPO in Table 3 of the appendix. We found that IPO did not perform as well as DPO.\n\n[7] A General Theoretical Paradigm to Understand Learning from Human Preferences\nGheshlaghi Azar et al., 2023, https://arxiv.org/abs/2310.12036 \n\n**While the paper briefly mentions the ethical implications of using LLMs trained on public data, it does not delve deeply into the potential risks of synthetic data generation, such as the possibility of generating biased or harmful content.**\nThank you for noting this—we will expand our Impact Statement to discuss each of these important points. \n\n**Can you break down the specific cost savings in terms of client upload/download communication and computation, and how these savings are achieved?**\n\nThank you for the comment–our description of the communication and computation cost savings was not clear. First, we describe the communication savings. In our experiment setting, DP-FL clients have to upload and download a DistilGPT2 model, which consists of 82M floats. Meanwhile, POPri communicates K x J = 1800 x 10 = 18000 text embeddings of size 384 for a total of 7M floats to download. POPri clients communicate back a vector of size 18000 (one score per text) to the server, resulting in 18K floats to upload. We have\n\n\n| Method   | Download (floats) | Upload (floats) |\n|--|--|--|\n| DP-FL    | 82M        | 82M       |\n| POPri    | 7M        | 18,000           |\n| Reduction Factor (DP-FL / POPri) | **~11.7×**         | **~4555.6×**        |\n\nNext, we provide a cost savings calculation in terms of compute. Experiment setting: we sample 1000 clients per round on the bioRxiv dataset:\n\n| Method   | Client Runtime (V100 GPU) in GPU seconds | Server Runtime (A100 GPU) in GPU seconds     |\n|--|--|---|\n| DP-FL   | 4.8 sec                    | -                               |\n| POPri    | 0.01 sec                   | 13,547.84 sec |\n\nThe client compute cost is small for POPri (about 500x smaller) than DP-FL. Meanwhile, the server side compute cost is small for FedAvg, and large for POPri. POPri is suitable in cases where (1) we care a lot about the downstream model quality, (2) the client computation and/or communication is expensive, and (3) server computation is relatively cheap and powerful. We will include this analysis in the paper.\n\n**In the context of POPri, how does the choice of the rejected sample (e.g., middle-ranked vs. last-ranked) impact the quality of the synthetic data generated? What is the rationale behind selecting the middle-ranked sample as the rejected sample?**\n\nThis choice was a heuristic based on our experiments, discussed partially in Appendix E.4. Choosing very low-ranked samples (namely, rank 10) was bad–during preference optimization, the model was learning to distinguish between a good sample and a very bad sample, which distorted its performance. We needed to select more challenging preference pairs for DPO. However, if we had chosen rank 3 as the negative sample (for instance), the DP noise would be more likely to swap the ordering of the first and second-ranked samples. Hence, as an empirically-motivated compromise, we chose the 5th-ranked sample, which performed the best in our ablation.", "claims_and_evidence": "See weaknesses and questions.", "methods_and_evaluation_criteria": "See weaknesses and questions.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "See weaknesses and questions.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "**Strengths**\n\n1.  The authors introduce ​LargeFedBench, a new federated benchmark for LLM evaluation, which includes ​Congressional Speeches and ​bioRxiv datasets. This benchmark is designed to avoid contamination and provides a large number of clients, making it a valuable resource for future research.\n\n2. The paper introduces a novel approach to private on-device learning by framing it as an LLM preference optimization problem. This is a significant departure from traditional DP-FL and synthetic data methods, which rely on prompt engineering or in-context learning.\n\n**Weakness**\n\n1. It presents very limited downstream task evaluations. We can hardly judge if it works for a broader scope of downstream tasks. \n\n2.  The experiments are primarily focused on text data (e.g., bioRxiv, Congressional Speeches). It would be beneficial to see how POPri performs on other types of data, such as images or structured data, to assess its generalizability.\n\n3. This paper uses DPO to improve synthetic data generation, but it does not compare it with other possible methods, such as RLHF. \n\n4. While the paper briefly mentions the ethical implications of using LLMs trained on public data, it does not delve deeply into the potential risks of synthetic data generation, such as the possibility of generating biased or harmful content.", "other_comments_or_suggestions": "Please do not change the template format.", "questions_for_authors": "1. The paper uses DPO to fine-tune the LLM. Why was DPO chosen over other preference optimization methods like RLHF? What are the specific advantages of DPO in the context of generating DP synthetic data?\n\n2. The paper claims that POPri has significant communication and computation cost advantages over DP-FL. Can you break down the specific cost savings in terms of client upload/download communication and computation, and how these savings are achieved?\n\n3. In the context of POPri, how does the choice of the rejected sample (e.g., middle-ranked vs. last-ranked) impact the quality of the synthetic data generated? What is the rationale behind selecting the middle-ranked sample as the rejected sample?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review.", "ethics_expertise_needed": ["Other expertise"], "ethical_review_concerns": "This submission significantly changed the original ICML template."}}]

Author rebuttal:

Meta-review: This paper proposes a method called POPri, for federated learning with differential privacy guarantees in the context of language models. In addition, this paper also creates a new benchmark for evaluating federated learning in the era of large language models. Most reviewers are positive about this paper, but reviewer W8ui was concerned about the limited downstream task evaluation and the low accuracy. Therefore I recommend a weak acceptance of this paper.
Decision: Accept (poster)