NeurIPS 2025

Diffusion Federated Dataset
Seok-Ju Hahn1
Argonne National Laboratory
hahns@anl.gov
Junghye Lee2
Seoul National University
junghye@snu.ac.kr
Abstract
Diffusion models have demonstrated decent generation quality, yet their deploy-
ment in federated learning scenarios remains challenging. Due to data heterogeneity
and a large number of parameters, conventional parameter averaging schemes of-
ten fail to achieve stable collaborative training of diffusion models. We reframe
collaborative synthetic data generation as a cooperative sampling procedure from
a mixture of decentralized distributions, each encoded by a pre-trained local dif-
fusion model. This leverages the connection between diffusion and energy-based
models, which readily supports compositional generation thereof. Consequently,
we can directly obtain refined synthetic dataset, optionally with differential privacy
guarantee, even without exchanging diffusion model parameters. Our framework
reduces communication overhead while maintaining the generation quality, realized
through an unadjusted Langevin algorithm with a convergence guarantee.
1
Introduction
Federated learning (FL [1]) enables clients (i.e., data owners) to collaboratively train a statistical
model by exchanging locally updated parameters with a central server over iterative communication
rounds, thereby preserving data privacy. While this model-centric FL paradigm is well-established,
sharing public data can substantially enhance FL performance by mitigating statistical heterogeneity
arising from non-independent and identically distributed (non-IID) local data distributions [1–4]. For
instance, public or synthetic datasets can homogenize disparate local distributions and serve as direct
signals for server-side pretraining. This facilitates client-side transfer learning or data augmentation,
both improve the overall utility of FL.
While these data-centric FL scheme offers clear advantages over purely model-centric approaches,
there remain challen...