ICML 2025

Private Federated Learning using Preference-Optimized Synthetic Data
Charlie Hou * 1 Mei-Yu Wang * 2 Yige Zhu * Daniel Lazar 3 Giulia Fanti 1
Abstract
In practical settings, differentially private feder-
ated learning (DP-FL) is the dominant method
for training models from private, on-device client
data. Recent work has suggested that DP-FL may
be enhanced or outperformed by methods that use
DP synthetic data (Wu et al., 2024; Hou et al.,
2024). The primary algorithms for generating DP
synthetic data for FL applications require careful
prompt engineering based on public information
and/or iterative private client feedback. Our key
insight is that the private client feedback collected
by prior DP synthetic data methods (Hou et al.,
2024; Xie et al., 2024) can be viewed as an RL
reward. Our algorithm, Policy Optimization for
Private Data (POPri) harnesses client feedback
using policy optimization algorithms such as Di-
rect Preference Optimization (DPO) to fine-tune
LLMs to generate high-quality DP synthetic data.
To evaluate POPri, we release LargeFedBench, a
new federated text benchmark for uncontaminated
LLM evaluations on federated client data. POPri
closes the gap in performance between the fully-
private and non-private settings by up to 58%,
compared to 28% for prior synthetic data methods,
and 3% for state-of-the-art DP federated learn-
ing methods. The code and data are available at
https://github.com/meiyuw/POPri.
1. Introduction
Many important machine learning (ML) applications feature
sensitive datasets that are distributed across client devices
(e.g. mobile devices). Such ML models are often hosted
on client devices. These on-device models offer privacy, la-
tency, and storage benefits relative to centrally-hosted mod-
*Equal contribution
1Department of ECE, Carnegie Mellon
University, Pittsburgh, PA 2Pittsburgh Supercomputing Center,
Pittsburgh, USA 3Coldrays, Tucson, AZ. Correspondence to: Char-
lie Hou <hou.charlie2@gmail.com>.
Proceedings of the 42 nd...