Diffusion Federated Dataset
Authors: Seok-Ju Hahn
Junghye Lee
Venue: NeurIPS 2025 poster ()
Keywords: federated learning
diffusion model
collaborative machine learning
synthetic data
cooperative inference
Primary area: infrastructure
TL;DR: A cooperative inference framework of pre-trained diffusion models for collaborative synthetic dataset generation

Abstract: Diffusion models have demonstrated decent generation quality, yet their deployment in federated learning scenarios remains challenging. Due to data heterogeneity and a large number of parameters, conventional parameter averaging schemes often fail to achieve stable collaborative training of diffusion models. We reframe collaborative synthetic data generation as a cooperative sampling procedure from a mixture of decentralized distributions, each encoded by a pre-trained local diffusion model. This leverages the connection between diffusion and energy-based models, which readily supports compositional generation thereof. Consequently, we can directly obtain refined synthetic dataset, optionally with differential privacy guarantee, even without exchanging diffusion model parameters. Our framework reduces communication overhead while maintaining the generation quality, realized through an unadjusted Langevin algorithm with a convergence guarantee.

Diffusion Federated Dataset
Seok-Ju Hahn'
Junghye Lee² Seoul National University junghye@snu. ac .rr
Argonne National Laboratory hahns@anl .gov
Abstract
Diffusion models have demonstrated decent generation quality, yet their deploy- ment in federated learning scenarios remains challenging. Due to data heterogeneity and large number of parameters, conventional parameter averaging schemes of- ten fail to achieve stable collaborative training of diffusion models. We reframe collaborative synthetic data generation as cooperative sampling procedure from a mixture of decentralized distributions, each encoded by pre-trained local dif- fusion model. This leverages the connection between diffusion and energy- basedd models, which readily supports compositional generation thereof. Consequently, directly obtain refined synthetic dataset, optionally with differential privacy guarantee, even without exchanging diffusion model parameters. Our framework reduces communication overhead while maintaining the generation quality, realized through an unadjusted Langevin algorithm with a convergence guarantee.
Introduction
Federated learning (FL [1]) enables clients (i.e., data owners) to collaboratively train statistical model by exchanging locally updated parameters with a central server over iterative communication rounds, thereby preserving data privacy. While this model-centric FL paradigm is well-established, sharing public data can substantially enhance FL performance by mitigating statistical heterogeneity arising from non-independent and identically distributed mno-IIID) local data distributions [1-4]. For instance, public or synthetic datasets can homogenize disparate local distributions and serve as direct signals for server-side pretraining. This facilitates client-side transfer learning or data augmentation, both improve the overall utility of FL.
While these data-centric FL scheme offers clear advantages over purely model-centric approaches, there remain challenges. First, curating public datasets is often infeasible in real-world FL sys- tem. Although generation of synthetic data via the collaborative training of generative models is viable alternative, it is challenging in FL settings. For example, generative adversarial networks (GANs [5]) suffer from training instabilities and suboptimal sample quality, which are exacerbated in FL by statistical heterogeneity [6-8]. Even advanced diffusion models [9-12] incur substantial computational and communication overheads due their large parameter sizes and fine- ane-grained
optimization requirements. Thus, effective synthetic data generation methods for FL constitute critical yet underexplored research area. Specifically cross-silo FL settings, clients often have limited number of samples (e.g., hospitals or enterprises with small datasets). In this sample-limited condition, generating synthetic data becomes critical as can complement scarce and disparate local dataset with high-fidelity synthetic samples. Thereby, this directly addresses the statistical heterogeneity problem federated environments.
In this work, we redefine federated synthetic data generation as a collaborative sampling process from mixture of heterogeneous and inaccessible local distributions. By modeling each local distribution with client-side diffusion model, we enable efficient compositional sampling from them, leveraging
Code is available at:
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

energy-based interpretations of diffusion models [13] and the mixture- saaere--f-eexpee of-experts paradigm [14]. The sampling is embarrassingly simple, through the unadjusted Langevin algorithm (ULA [15, 16]). Building on these, we introduce DfD (Diffusion-federaate Dataset), a cooperative inference framework that generates synthetic data through sampling directly from a mixture distribution, eschewing traditional model averaging. DfD advances federated synthetic data generation as follows:
We propose novel view on federated synthetic data generation as cooperative sampling from individually trained diffusion models, without necessitating the exchange of model parameters.
Through energy-basee based parameterization and compatibility of ULA with the diffusion reverse process, we refine the connections between diffusion models and energy- models (EBMs [17]). We also derive the optimal step size and -on-syyptooii distributional convergence for DfD.
We empirically validate fidelity and utility of synthetic dataset from DfD under non-IID conditions, optionally with formal privacy guarantees, addressing key needs in cross-silo FL scenarios.
Related Works
Synthetic Data in FL. FL often struggles with slow convergence when the client's local private data sets differ significantly, a common challenge known as statistical heterogeneity or the non-IID problem [1, 4]. This issue is critical in that the central server cannot directly access or adjust these heterogeneous local datasets to align their disparate optimization trajectories. Most prior work has addressed this through model-centric approaches, such as local update regularization [18-21], modified central aggregation schemes [22-27], or personalization [28-30].
While effective, complementary data-centric perspective still remains underexplored. These include sharing additional server-side public data [2, 31-33], using indiscernible auxiliary representations [34- 40], or leveraging a generative model to obtain plausible synthetic data [41-50]. These provide clients with a proxy for global distribution, which directly mitigates the non-III problem and improves convergence [5]]. Notably, studied in [2], sharing only small portion of public data can significantly boost FL performance, though acquiring such data is nontrivial in practice.
Hence, synthetic data is widely used with generative models in e.g., healthcare [52-57]. However, current synthetic data generation methods in FL, including real-world applications, mostly resort to GANs [58] (optionally with privacy guarantee [59-62]), which suffer from subpar generation quality and optimization instability due to their adversarial training scheme (e.g., mode collapse [6-8)).
Diffusion Models in FL. Diffusion models [11, 12], such as Denoising Diffusion Probabilistic Models (DDPMs [9]), have offered a superior generation quality training stability, compared to other generative models, e.g., GANs. [10]. Although promising, their adoption in FL is challenging and sometimes even prohibitive due to high computational costs and large model sizes. Thus, current methods suffer from significant communication overhead [63], poor scalability to high-resolution data [64], and even require retraining of local models [65] or data sharing [66] due to non-IID problem. In addition, the inherent loss design of diffusion models, which depend multiple time-steps, also requires frequent parameter exchanges during training. making them difficult to adopt in FL [66, 67].
Our framework detours by directly generating samples from an inaccessible mixture of heterogeneous local distributions, encoded by locally-trainct diffusion models. This is rooted in exploiting the connection of diffusion models energy-based models (EBMs [17]), which estimate unnormalized probability densities through their gradients with respect to inputs (i.c., scores [68]). This intriguing connection enables easy compositional sampling, which can be viewed as sampling from mixture- of-experts [14], even without accessing model parameters. As a result, DfD offers an efficient and scalable solution for adopting diffusion models in federated synthetic data generation.
Preliminaries
3.1 Diffusion Models
Diffusion models aim to encode data distribution (a) by learning transition from noise-perturbed data {a::){= where 27 ~ (0,, I2), into its clean original counterpart, 3p ~ Passu(r)) = g())) through paired forward and backward processes. Specifically, Gaussian diffusion defines a Markov

---

chain joint distribution ((To.....y) = ((x)) [ [ [zt- where the forward process is de- fined by incrementally adding Gaussian noise over E [T] as (xx NNxx; 1,, 5,Ia). Note that d is the data dimension, 0 < 8, S 1 and 0₁ = 1 - 3, are noise constants. The reverse process, typically parameterized by a deep network with 0, approximates po po(2 in order to progressively denoise from the Gaussian noise æ' into the original data Zo. With sufficienlly small 3r, each transition of reverse process approximately follows Gaussian [11]. This allows:
Pa xx,) NN(2
((1t-11 (I--1:
1
B
cZ, t) B.Ia)
(1)
a4
Ppp (2);,))
where 0g, 3, are some transformations B,, Vt E [T), following the configurations of [9] (see also Appendix C.I).
Eventually, the parameterized deep network needs to predict c0( t) as mapping 6g R [T] Rd. Note that diffusion models ensure the analytic conversion from the original the perturbed data at any timestep E [T] [9]:
a + v1 - ā,e, e~ NoO,,
(2)
Using this property, optimize with composite loss [9] as [(0) Σt= C(O, where
C( e- [||c (z, 1)
By minimizing this objective, diffusion models are capable of generating high-quality samples by constructing Mo (z,. t) from their prediction eø c0 t) and progressively denoising from Ty ~ (Oa,, Ia) to 2g over T 1, ..., 1, using Eq. (1). We refer to Appendix A for detailed derivations.
3.2 Energy-based Interpretation of Diffusion Models
Diffusion models have an intriguing connection with EBMs [12, 13). EBMs [17] define an unnormal- ized probability density
(xz) exp (~)/o(x))
(4)
Ze
where EBMs forgo modeling of the normalizing constant Ze = Sacx exp (x)(fff((z)) dæ. We
define So Rd R energy function with parameter 0 E RP, scale factor À IR+ and V (o _o as a score. Note that we have d < p if we choose deep networks, which are typically overparameterized.
The abstention modeling normalizing constant prevents exact likelihood computation. To address the issues that arise from this design, denoising score matching [69] has been proposed to minimize the Fisher divergence between the model's score and that of a noise-perturbed data distribution, i.e., ((xa) = SaEx (x a (z) (d)adaa (x) dæz Note here that o is a noise variance and the perturbation is x + oe, (O,,, La). Building on these, the denoising score matching objective is:
given
(( ) = Eg, <a (x) (p)((( [|[ x. log g( (x) Væ, log (o(xx))1
(5)
This is equivalent (up to a constant) to:
[|e o oAV So(o)))
o² C(, a) Eu, (e)
(6)
Interestingly, the objective of diffusion models in Eq. (3) aligns with the scaled objective above [13]. with following connection (along with replacing a into oy):
Van, log (x = -XV M., fo( E0 50( t) at
Note that this connection to EBMs can be further concretized for diffusion models by a specific choice of the energy-based parameterization introduced in following Section 3.4. It should also be noted that this explicit connection allows using a sampler for diffusion models, e.g., ULA. We defer all detailed derivations in this section to Appendix B.
3

---

3.3 Federated Synthetic Data Generation by Sampling from a Mixture Distributions
The ULA follows a discretized Langevin diffusion process [15] and enables sampling from target distribution p(z) with its score V log p(z). by iteratively updating from 2T ~ ( Ia) using:
-t- at 7/ V æ, logp( 22 (Oa, Ia),
~
(8)
where ?7 2 is a step size, and it ensures Zg p(æ) [70]. We denote the notation of decreasing timesteps as = ..... 1 for the compatibility with the diffusion reverse process.
In FL setup, have K clients each having private dataset D;. Then, target distribution is naturally defined as mixture of local distributions: p (ax) Σ (:P:(æ), where p: (a: represents unknown local distribution of D; from i-th client and wi 2 0 is mixing coefficient satisfying Σ Wi = 1 (e.g., W2 = 1/K K if uniform weighting). To generate samples from the mixture of local distributions, what we need to estimate the global score Va log (egp"(a) (x:) defined as follows:
W; (z)
V loggo (a)) = Σn WA V logpo, (x),
ii Σ wy exp(-Xfo, (fo
where Wi derived from po, (a) x exp exp (=) due to Eq. (4). Note that directly supports embarrassingly parallel computation across clients, aligning well with FL settings. In detail, the estimation of the global score v logp"(2)) ) is available as long as we have both local scores Va log po. (x) and ii) energies (unnormalized density values) exp(- (a)) of each client.
However, diffusion models do explicitly provide Sa, (a)) in its inherent design. This can be easily addressed using energy-based parameterization described in the following section.
3.4 Energy- based Parameterization of Diffusion Models
To implement ULA to directly sample from mixture of local distributions, should estimate the energies Po, (x) & exp(-Afe. (x)) for w; in Eq. (9). Since diffusion models lack explicit density function, prior arts proposed to approximate them by defining fo(z) using energy- -based &z
parameterization trick [13, 71]. (We refer to Section of [13] for details on other tricks)
Definition 3.1 (energy-based & parameterization [13]). The energy function of diffusion model is approximated as (z,, t) = 2 co( t) 12, where (2x, t) is a prediction of diffusion model.
Having this energy function, we can now define scores of diffusion models and obtain the global score in Eq. (9), accordingly. Unfortunately, this parameterization requires a modification in training of diffusion models, and this often yields subpar generation quality [13, 71].
4 Proposed Method
4.1 Refined Energy-based Parameterization
To detour the modification training, we start from the notion of well-trained diffusion models. Definition 4.1 (Well-trained diffusion model). A diffusion model is well-trained if, through mini- mization of the objective in Eq. (3), its noise prediction satisfies Eg (xr t) RS e ~ NO,,,, La).
Remark 4.2. Note that this captures the empirical observation that sufficiently trained diffusion models accurately predict added noise. In addition, thanks to E = (x) Votan)/ ag from Eq. (2), well-trained diffusion model readily satisfies that V c V VI-
With these, we can now approximate the score of well-trained diffusion models follows:
V log po (o t) - AV a, f
Xea (2, T)T Vzz c0(
(10)
A
Eo (x) t),
where the first equality is a direct result from Fq. (4), the second equality due to Definition 3.1, and the last approximation is from Definition 4.1. Notably, this matches Eq. (7) if o, /I ā./2. To

---

summarize, the refined energy- energy-based & reparameterization provides unnormalized density and score well-traines diffusion models as:
po (xr cxp (- (2
A
V Va, log, =- -I- co(
(11)
for all timesteps E (T). With these, no modification to the training of diffusion models required.
4.2 DfD: Cooperative Diffusion Models Inference Framework for Synthetic Dataset
Preparation of diffusion models
Iterative refinement via cooperative inference
K,..,K Client
Server repeats .,..,
Ml
F
Fgglog
2n
(0
to [3]
0 2 31 f 7
8
0
Eq. (9)
(11
Private Dataset
Ditfusion Model Training
Eo,
Eq. (11)
0)
Init' lattoo of synthetic data
Eax (xt)
Server
Client 1,.,, K
DF
E0: (x
Random Init.
Synthetic Samples x JJ E [V]
Diffusion Model Inference
Figure 1: Overview of DÉD. A Clients independently train diffusion models to be well-trained with
Eq. (3). B The server randomly initializes synthetic dataset per Eq. (12). C The server requests
inference on synthetic dataset all clients, receives predictions c0, (az (), Vi
[K].] E [N]. transforms
into energies
and scores
using Eq. (11), composes
into global scores using Eq. (9), and refines synthetic dataset using ULA Eq. (8) over 7 steps.
Our proposed framework, DfD, generates synthetic data samples directly from mixture of local distributions encoded by local diffusion models, independently trained on private and non-IID client datasets. An overview of the framework is provided in Figure 1 and the overall procedure of DfD (for the case of unconditional generation) is described in Algorithm 1.
A key innovation of DfD is its ability to leverage locally trained diffusion models, avoiding repetitive local updates along with the exchange model parameters. This achieved by exchanging the predictions of well-trained diffusion models instead, and the models prepared by each client before cooperative inference begins. These predictions are iteratively collected and transformed into energies and scores at the server, to construct a global score in Eq. (9).
Preparation diffusion models. Each client trains its own diffusion model on its private dataset D, by minimizing Eq. (3), to obtain a well-trained model as in Definition 4.1. The model can unconditional, predicting 60, (ær, (), or conditional on label y (e.g., attributes or classes), predicting cp, (æ,, y. l). Note that the dimension of predictions is equal to that of inputs, which significantly smaller than the model parameter size. In addition, local pre-training can oceur asynchronously, and clients may optionally apply differential privacy (DP) mechanisms [72].

---

Algorithm 1 DfD: Cooperative Diffusion Models Inference Framework for Synthetic Dataset
1: Require: number of clients K, synthetic dataset size N, communication rounds T
2: Procedure:
3: All clients E [K] prepare well-trained diffusion model 0, E R² using Dj with Eq. (3).
4: Server initializes NV samples in Eq. (12) have Dj.
5: for = T", 1 the server
6:
Requests inference to all clients in parallel on D
7:
Receives predictions {Eg, (a( t) E Rd E [K].j E [N]].
8: Transforms predictions into energies and scores with Eq. (11).
9: Computes global scores for all samples with Eq. (9).
10: Updates synthetic dataset into D; using ULA in Eq. (8).
11: end for
12:
Return: Dō
B Initialization of synthetic data. The central server randomly initializes N synthetic data samples
Dj-- = { 1 (or Dj- = {(07)) ) as:
ST ~ (0, Ia),
(if conditional)
y Categorical (C-11c),
(12)
where Cs the number of conditions (e.g., classes, attributes) encoded by labels. The synthetic dataset size N is determined based on communication constraints, where N can be set much smaller than the required parameter size of diffusion models, e.g., N < max, dim().).
Iterative refinement via cooperative inference. For each communication round T, .... the central server sends the current synthetic dataset to all clients and requests predictions from their diffusion models. With these predictions, the server computes energies and scores of each client using Eq. (11). The server then constructs global scores using Eq. (9) and refines the server-side synthetic dataset using ULA, as in Eq. (8). Note that it can be extended to the conditional case by simply incorporating y ( in this step. At the end, the server obtains refined synthetic dataset, Dī-
4.3 Theoretical Analysis
The ULA is the main workhorse of DfD as it relies on energy-based parameterization to sample from mixture of local distributions using global scores in Eq. (9). Hence, must carefully select the step size, denoted by Me. to ensure that the DfD correctly settles at the target mixture distribution. We theoretically derive the step size guidance in two steps: @ verification of the compatibility of ULA with diffusion reverse process, and (6) analysis of non-asymptotic convergence behavior of ULA to the target distribution in KL divergence [70]. We defer all proofs in Appendix C.
@ Compatibility of ULA with diffusion reverse process. DfD resort to diffusion models as main components. Thus, we begin with the successful diffusion reverse process and transplant its key success factor into the ULA to ensure compatibility. Interestingly, we find that non-expanse eoneeaxansseeeess w.r.t. norm is inherently encoded in the diffusion reverse process, and perceive it as key factor.
Lemma 4.3 con-exxanssveeess of diffusion reverse process). The diffusion reverse process in Eq. (1) preserves the squared n--norm of resulting iterates to be non-expansive, i.e., 3[]|,,111] 112] < |||
Next, we proved that this property can be similarly induced for ULA under following conditions.
This gives an explicit guidance for the choice of scale factor in Eq. (4), which is used for the construction of energies and scores in Eq. (9).
Lemma 4.4 (Non-cxpansivene
condition of ULA). ULA satisfies the non non-exxansiveness w.r.t. [[]][
squared fz-norm as [B[[ [|]| [ < E []] for well-trained diffusion models with energy-based &
paraneterization, if and only if n E 0, and À = 2.
b Non-asymptotic convergence of ULA. Though previous work heuristically adopted the naive resemblance of ULA with the diffusion reverse process to set the step size (i.e., simply setting ÿ) = 3 while ignoring the scaling factor Ja [71], this approach has no theoretical justification. Thus, we theoretically derive a ULA step size and the convergence guarantee toward a target mixture distribution under KL divergence, with acceptable assumptions provided in Appendix C.

---

Theorem 4.5 (Convergence guarantee of DfD). Let Pt be the evolving distribution of 0t E RR from
ULA and Pt be the mixture of distributions encoded by diffusion models. For é > 18dsnr (1-
and E (0, v 3/6)), the iterates wr-t ~ Pr-t guarantee (KL(PT-1 || PT- < after 2
1-37-t log Derl (pr) steps with a step size - S (mnn 18dc (1d((1-ar-1)
p p(1-ār-1)
2
Note that this ensures DfD can sample from mixture of inaccessible and heterogeneous distributions finite number steps, without access to the local dataset D, and the local model parameters O,,
4.4 Privacy Guarantee
Definition 4.6 ((e, 8)-DP [72]. A mechanism M satisfies (e,6)-DP if, for any two neighbor- ing datasets D and D' differing in one record, and for any output set S, PD(M(D) e S] s Pr[M((D)) E S] + 6, where é > 0 is the privacy budget and 2 is the failure probability.
The communicated signals in DfD are client predictions ep, (x,", 1) from diffusion model trained on private dataset D; In FL., typically use DP mechanism to protect sensitive information. Intriguingly, DÉD can inherit DP guarantee as long as each client i already trained its own diffusion model 0, achieve (e,, 6,)- e.g., using DP-SGD [73].
Theorem 4.7 (DP guarantee of DíD). Assume all client datasets D2 are disjoint. If each client i [K] trains a diffusion model 0; for (ci, 6:)-DP given G; 0 and 6; 2 the synthetic dataset Di generated by DfD compositely satisfies (max; G,, max; DD)
Proof. As the server processes differentially private local predictions, the post-processing property of DP [74] also ensures that subsequent steps (i.e., global score computation. ULA updates) preserve
DP. The parallel composition theorem [75] provides a composite DP guarantee across clients with
disjoint datasets with each other, terms of the maximum privacy budget and fallure probability. O
Experimental Results
5.1
Sctup
Datasets. We use three benchmark datasets: MNIST [76], CIFAR-10 [77]. and CelebA [78], after resizing all inputs to have spatial dimension of 32x32. As each dataset has separate train test folds, we use the train fold to split into client datasets, and set the test fold aside for server-side evaluation. We distribute the train fold of each dataset into K 10 clients with three different non-IlI) conditions: i) Dirichlet distribution-based non-IID [79] for MNIST, ii) power-law distribution-base non-IID [21] for 100 and iii) pathological non-IID [1] for CelebA.
To further simulate a convincing scenario in which a synthetic dataset should be procured (i.e., setrings), we randomly sample local dataset have size of 300 on average, following the sample size configurations of the curated benchmark for the cross-silo FL setting [80].
Baselines. We compare with FL methods for generative models: FedGAN [42], FedDiffuse [63] and PRISM [67]. All clients are taking 10K steps in total for T 1000 rounds: E 10 local updates for all comparison methods, and = 10 x 1, 1,000- = 10, 000 local updates for DfD as it requires no update during communication rounds. The mini-batch size is set B 32, and the learning rates are tuned for all methods, and set to c(1 ) for c > 0, 2 for DfD.
Evaluation Metrics. We evaluate both fidelity and utility of the generated synthetic dataset. To evaluate the fidelity of synthetic data, we use the widely-used metrics for generative modeling: Fréchet Inception Distance (FID [81], Precision & Recall (P&R [82]), and Density Coverage (D&C [83]). To evaluate utility, we use an accuracy evaluated from classifier trained at the central server using labeled synthetic dataset. We defer the specific experimental setup Appendix
7

---

Table 1: Results on synthetic dataset quality.
FID +
P1
R
D
C1
MNIST
FedGAN [42]
34.8418
0.4189
0.1240
0.1144
0.1378
FedDiffuse [63]
49.5704
0,1842
0.7610
0.1
0.3428
PRISM [67]
36.7945
0,4223
0.1386
0.1639
0.1481
DfD
37.7354
0.6224
0.3437
0.1816
0.3937
CIFAR-10
FedGAN [42]
145,5668
0.6866
0.0221
0.4800
0.1221
FedDiffuse [63]
78.3845
0,4142
0.2119
0.3731
0.2958
PRISM [67]
330.8488
0.0875
0.0077
0.0334
0.0368
DfD
59.9761
0.5153
0.2492
0.3521
0.3590
CelebA
FedGAN [42]
18,4
0.3469
0,4210
0.1349
0,1929
FedDiffuse [63]
33.3323
0,2986
0.5176
0.2318
0.2793
PRISM [67]
200.1870
0.1479
0.1809
0.0684
0.0769
DfD
29.1832
0.3734
0,4143
0.2229
0.2370
Table 2: Results on synthetic dataset utility.
LogReg
MLP
CNV
MNIST
FedGAN [42]
71.7
72.4
73.6
FedDiffuse [63]
78.2
77.5
78.8
PRISM [67]
41.4
45.3
DfD
78.5
78.1
78.9
CIFAR-10
FedGAN [42]
19.8
21.1
24.3
FedDiffuse [63]
29.2
31.3
33.0
PRISM [67]
11.5
12.9
13.2
DfD
28.3
32.4
34.1
CelebA
FedGAN [42]
42.1
43.4
45.8
FedDiffuse [63]
55.2
58.1
58.2
PRISM [67]
12.2
11.3
13.4
DfD
57.3
56.5
59.3
5.2 Results
Quality and Utility. Table 1 summarizes the quality-based results, i.e., FID, Precision (P), Recall (R), Density (D) and Coverage (D). Our method outperforms other FL methods for generative modeling in synthetic data fidelity. We provide generation results of each method in Figure 3. Table summarizes the test accuracies as synthetic data utility. Following [84], we train three server-side classifiers on each generated synthetic dataset: logistic regression (LogReg), multi-layered perceptron (MLP), and convolution neural network (CNN). We evaluate each classifier on separate test fold held in the central server. As proxy of raw local data samples inaccessible in FL settings, synthetic dataset from DfD have been shown to offer better utility compared to existing basclines.
Efficiency. The communication costs differ in DfD compared to other methods. Table 3 summarizes the communication target and computation budget required generate N samples. During FL, DfD is faster in computation as it only conducts inferences on samples (i.e., N forward passes for N samples), whereas other methods require both backward and forward passes N E times to update parameters. Additionally, DfD exchanges predictions of which size is N x d, where the dimension far smaller than the size of the model parameters (i.e., < p. Thus, can significantly reduce communication N x d < p) costs by setting reasonable number of samples, N.

---

Table Comparison on communication cost computation complexity.
Communication
Computation
FedGAN [42]
O(N x É x p)
FedDiffuse [63]
0: R
(forward & backward E times)
PRISM [67]
DfD
E0, (al) .t)], ,t)),- 5=1
E RRd
O(N x p) (single forward pass)
0
/
/
/
/
/
/
)
1
/
/
2
3
2
2
7
2
2
7
2
2
2
2
/
3
3
3
3
3
3
3
3
3
3
3
3
3
3
S
3
3 41
4
4
J 4
4
1
4
4
41
4
4
21
4
7
4
7
0
6
6
77
7
7
7
7
7
7
7
7
7
7 7 2
7 7
7
8
8
8
7
2
8
Figure 2: Differentially private synthetic dataset for MNIST from DfD under (e = 10, 8 = 10 --DP.
Thanks to Theorem 4.7, DfD readily satisfies DP. Following [85], we let each client train its diffusion model with DP-SGD [73], under shard-partition partiitioned non-III setting [1] for K = 10 clients: each client has samples from only 2 out of 10 classes from MNIST dataset, i.c., digits 0 and for client O, digits 1 and to for client 1, I.... and digits 9 and 0 for client 9. To achieve (max; E,, max; 8,) -DP for the resulting synthetic dataset, we set e; 10 and 6; = 10-5 5 for applying DP is detrimental to sample quality as expected, and subtle tuning of step size is required to obtain discernible samples. Thereby, improving the quality of ULA sampling from differentially
Privacy.
E [] clients. We found that
private diffusion models is promising future direction for DfD in practice.
6 Limitation and Discussion
DfD gives clients great flcxibility under the assumption of credible participation, such as cross-silo FL settings. In cross-device FL settings, where massive and unreliable clients [4] participate, DfD may fail, so we only consider cross-silo FL settings where a moderate number of credible clients participate. This could be relaxed by allowing partial participation through approximation of a global score at the central server [86].
Currently, the server ends up having synthetic dataset at last, not generative model. Thus, by training server-side amortized sampler [87-89] to emulate the collaborative sampling process, we can additionally generate samples even after the collaboration. Moreover, the communication cost can be further reduced by adapting advanced samplers [90-93] or by using model compression techniques, which we leave for future work.

---

0
/
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
3
3 4 5 6 7
4
4
4
4
4
7
4 6 6 3 8
5
5
5
5
5
5
6
6
6
6
6
6
6
7
7
7
7 8
8
8
8
8
9
9
9
9
O
O
O
0
0
6
O
0
0 /
/
2
1
/
/
/ 2
/ à
2
a
2 3
2
2
3
3
4
4
4 5
4
4
4
4
5
5
5
5
5
5
5
5
6
6
6
6
6
7
7
7
7
7
7
7
7
8
8
8
8
3
8
8
9
9
9
9
9
9
3 R
o
R
R
3
R
8
7
9
R
7
7
3
2
8
4
Q
0
3
3
9
9
R
R
7
7
8
3
C
9
3
0
3
0
3
3
9
3
R
R
R
R
7
7
3
7
J
8
3
5
0
0
6
0
0
0
1
/
/
1
2
2
2
2
2
2
2
2
2 2
3
3
8
5
4
4
4
4
4
4
4
S
5
5
5
5
5
5
5 5
6
6
6
6
6
7
7
7
7
7
7
7 8
8
8
8
9
9
9
Figure 3: Visualization of synthetic dataset generated under data-limited non-lID setting, Each row corresponds to FedGAN [42], FedDiffuse [63], PRSIM [67], and DED. Each column corresponds
to CIFAR-10 [77], MNIST [76], and CelebA [78].
The success of DfD hinges the faithful, authorized training of local diffusion models by participating clients. However, when local diffusion models are overlit or even memorize samples, this would introduce biased collapsed sampling, resulting in catastrophic generation results. Therefore, careful training is required (e.g., earlystopping, weight decay) to acquire literally well-trained diffusion models. To guarantee trustworthy training, DfD requires credible consortium of clients. Alternatively, we can use cryptographic tools, such as knowlege proofs, to certify verified pre-training [94].
Lastly, thanks the advancement in diffusion models, we expect DfD to be extended to other modalities than images [95, 96]. We believe the directions discussed thus far could improve the scalability and practicality of DFD in future works.
10

---

7 Conclusion
We propose collaborative synthetic data generation framework, DfD, that leverages an energy-based connection for cooperative inference of diffusion models. DfD offers improvements in generation quality, communication efficiency, and easy privacy guarantees with theoretically grounded design. Given wide implications of synthetic data in federated settings, we look forward to exploring extensions of DED to diverse modalities and data-intensive domains as a trustworthy framework.

---

Broader Impact
The DfD framework enables federated synthetic data generation with privacy guarantees, promoting secure data sharing in privacy seevacy-sensttive domains. produces high-quality synthetic data that preserves statistical properties, improving collaborative research and training of models while complying with e.g., GDPR [97] and HIPAA [98]. However, synthetic datasets can be possibly misused for malicious purposes, a robust accounting protocol is required for ethical deployment.
Acknowledgements
This research was conducted when Seok-Ju was at Scoul National University, supported by the National Research Foundation of Korea (NRF) Grant funded by the Korea Government under Grant No. 55-20025-00516770 The computational resource was supported by the high-Peeformance Computing Support Project, funded by the Government of the Republic of Korea (Ministry of Science and ICT) under Grant No. G2025-0146.
References
[1] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficee learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282, PMLR, 2017.
[2] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv: 180600058, 2018.
[3] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the conver- gence of fedavg non-iid data. arXiv preprint arXiv:1907.0211 2019.
[4] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings. et al. Advances and open problems in federated learning. Foundations and trends@ in machine learning, 14(1-2):1 2200, 2021.
[5] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[6] Farzan Farnia and Asuman Ozdaglar. Do gans always have nash equilibria? In International Conference on Machine Learning, pages 3029-3039. PMLR, 2020.
[7] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223. PMLR, 2017.
[8] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning, pages 3481-3490. PMLR, 2018
[9] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851 6851, 2020.
[10] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[1]] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learming, pages 2256-2265. pmlr, 2015.
[12] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
12

---

[13] Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Sussman Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and meme. In International conference on machine learning, pages 8489-8510. PMIR, 2023.
[14] Robert A Jacobs. Bias/variance analyses of mixtures-of-expert architectures. Neural compu- tation, 9()):39--38,, 1997.
[15] Gareth O Roberts and Richard L Tweedie. Exponential convergence of langevin distributions and their discrete approximations, 1996.
[16] Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy of meme-based maximum likelihood learning of energy-based models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5272-5280, 2020.
[17] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
[18] Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprini arXiv:2003.00295, 2020.
[19] Divyansh Jhunjhunwala, Shiqiang Wang. and Gauri Joshi. Fedexp: Speeding up federated averaging via extrapolation. arXiv preprint arXiv:2301 1.6604, 2023.
[20] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference machine learning, pages 5132-5143. PMLR, 2020.
[21] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machi learning and systems, 2:429-450, 2020.
[22] Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In International conference on machine learning, pages 7252-7261. PMLR, 2019.
[23] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khaza- eni. Federated learning with matched averaging. arXiv preprint arXiv:2002.066440, 2020.
[24] Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. Fair resource allocation in federated learning. arXiv preprint arXiv: 0ttp//955./997, 2019.
[25] Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. In International Conference on Machine Learning, pages 4615-462. 4625. PMLR, 2019.
[26] Scok-Ju Hahn, Gi-Soo Kim, and Junghye Lee, Pursuing overall welfare in federated learning through sequential decision making. In Proceedings of the 41st International Conference Machine Learning, pages 17246-17278, 2024.
[27] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv: / 1909.06335, 2019.
[28] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta-learning approach. arXiv preprint 2xXiv:2002.07948 2020.
[29] Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared representations for personalized federated learning. In International conference on machine learning, pages 2089-2099. PMLR, 2021.
[30] Yue Wu, Shuaicheng Zhang, Wenchao Yu, Yanchi Liu, Quanquan Gu, Dawei Zhou, Haifeng Chen, and Wei Cheng. Personalized federated learning under mixture of distributions. In International Conference on Machine Learning, pages 777800-77879, PMLR, 2023.
13

---

[31] Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint 1rXiv:1910.03581, 2019.
[32] Neel Guha, Ameet Talwalkar, and Virginia Smith. One-shot federated learning. arXiv preprint
WrXiv:/902.11177 2019.
[33] Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, Mchdi Bennis, and Seong- Lyun Kim. Communication-fffiiee on-device machine learning: Federated distillation and augmentation under non-iid private data. arXiv preprint arXiv:1811.11477 2018.
[34] Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xin He, Bo Han, and Xiaowen Chu. Vir- tual homogeneity learning: Defending against data heterogeneity in federated learning. In International Conference on Machine Learning, pages 21111-21132 PMLR, 2022.
[35] Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng. No fear of heterogeneity: Classifier calibration for federated learning with non-iid data. Advances in Neural Information Processing Systems, 34:5972-5984, 2021.
[36] Zhiqin Yang, Yonggang Zhang, Yu Zheng, Xinmei Tian, Hao Peng. Tongliang Liu, and Bo Han. Fedfed: Feature distillation against data heterogeneity in federated learning. Advances in Neural Information Processing Systems, 36, 2024.
[37] Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for het- erogeneous federated learning. In International conference machine learning, pages 12878-12889. PMLR, 2021.
[38] Yuanhao Xiong. Ruochen Wang, Minhao Cheng, Felix Yu, and Cho-Jui Hsich. Feddm: Iterative distribution matching for communication-effiiii federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16323-16332, 2023.
[39] Jack Goetz and Ambuj Tewari. Federated learning via synthetic data. arXiv preprint wrXiv:2008.04483 2020.
[40] Yanlin Zhou, George Pu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. Distilled one-shot federated learning. arXiv preprint arXiv:2009. 07999, 2020.
[41] Corentin Hardy, Erwan Le Merrer, and Bruno Sericola. Md-gam Multi Multi-diseriminator gener- ative adversarial networks for distributed datasets. In 2019 IEEE international parallel and distributed processing symposium (IPDPS), pages 866-877. IEFE, 2019.
[42] Mohammad Rasouli, Tao Sun, and Ram Rajagopal. Fedgan: Federated generative adversarial networks for distributed data. arXiv preprint arXiv:2006.07222, 2020.
[43] Wei Li, Jinlin Chen, Zhenyu Wang. Zhidong Shen, Chao Ma, and Xiaohui Cui. Ifl-gan: Improved federated learning generative adversarial network with maximum mean discrep- ancy model aggregation. JEEE Transactions Neural Networks and Learning Systems, 34(12):10502-109 -05000021515,,,.. 2022.
[44] Zijian Li, Jiawei Shao, Yuyi Mao, Jessie Hui Wang, and Jun Zhang. Federated learning with gan-based data synthesis for non-iid clients. In International Workshop Trustworthy Federated Learning, pages 17-32. Springer, 2022.
[45] Huancheng Chen and Haris Vikalo. Federated learning in non-iid settings aided by differentially private synthetie data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5026-5035, 2023.
[46] Clare Elizabeth Heinbaugh, Emilio Liccai and Huajic Shao. Data-free one-shot federated learning under very high statistical heterogeneity. The Eleventh International Conference on Learning Representations, 2022.
[47] Huancheng Chen and Haris Vikalo. Federated learning in non-iid settings aided by differentially private synthetic data. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5027-5036, 2023.
14

---

[48] Bangzhou Xin, Yangyang Geng. Teng Hu, Sheng Chen, Wei Yang, Shaowei Wang, and Liusheng Huang. Federated synthetic data generation with differential privacy. Neurocomput- ing, 468:1-10, 2022.
[49] Charlie Hou, Mei-Yu Wang, Yige Zhu, Daniel Lazar, and Giulia Fanti. Private federated learning using preference-ppimmized synthetic data. arXiv preprint arXiv:2504. 16438, 2025.
[50] Shanshan Wu, Zheng Xu, Yanxiang Zhang, Yuanbo Zhang, and Daniel Ramage. Prompt public large language models to synthesize data for private on-device applications. arXiv preprint arXiv:2404.44300 2024.
[51] Bo Li, Yasin Esfandiari, Mikkel N Schmidt, Tommy S Alstrøm, and Sebastian U Stich. Syn- thetic data shuffling accelerates the convergence of federated learning under data heterogeneity. arXiv preprint arXiv:2306.112: 2023.
[52] Jacky Chung- -aa Wu, Hsuan- Yu, Tsung- Hung Tsai, and Henry Horng- Lu. Dy- namically synthetic images for federated learning of medical images. Computer Methods and Programs in Biomedicine, 242: 107845, 2023.
[53] Qi Chang, Zhennan Yan, Mu Zhou, Hui Qu, Xiaoxiao He, Han Zhang. Lohendran Baskaran, Subhi Al' Aref. Hongsheng Li, Shaoting Zhang, al. Mining multi-center heterogeneous medical data with distributed synthetic learning. Nature communications, 14(1):5510, 2023.
[54] Wei Zhu and Jiebo Luo. Federated medical image analysis with virtual sample synthesis. In International Conference on Medical Image Computing and Computer-Asiisted Intervention, pages 728-738. Springer, 2022.
[55] Jinbao Wang. Guoyang Xie, Yawen Huang, Jiayi Lyu, Feng Zheng, Yefeng Zheng, and Yaochu Jin. Fedmed-gam gan: Federated domain translation unsupervised cross-modality brain image synthesis. Neurocomputingg 546:126283 2023.
[56] Daiqing Li, Amlan Kar, Nishant Ravikumar, Alejandro F Frangi, and Sanja Fidler. Feder- ated simulation for medical imaging. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part 23, pages 159-168. Springer, 2020.
[57] Shuai Li, Liang Hu, Chengyu Sun, Juncheng Hu, and Hongtu Li. Federated edge learning for medical image augmentation. Applied Intelligence, 55(1):56, 2025.
[58] Claire Little, Mark Elliot, and Richard Allmendinger. Federated learning for generating synthetic data: a scoping review. International Journal of Population Data Science, 8(1), 2023.
[59] Bangzhou Xin, Wei Yang, Yangyang Geng, Sheng Chen, Shaowei Wang, and Liusheng Huang. Private fl-gan: Differential privacy synthetic data generation based on federated learning. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages
IEEE, 2020.
[60] Sean Augenstein, H Brendan McMahan, Daniel Ramage, Swaroop Ramaswamy, Peter Kairouz, Mingging Chen, Rajiv Mathews, al. Generative models for effective ml on private, decen- tralized datasets. arXiv preprint 191.06791, 2019.
[61] Dingfan Chen, Tribhuvanesh Orekondy, and Mario Fritz. Gs-wgan: A gradient-sanitized approach for learning differentially private generators. Advances in Neural Information Processing Systems, 33:12673-12684 2020.
[62] Monik Raj Behera, Sudhir Upadhyay, Suresh Shetty, Sudha Priyadarshini, Palka Patel, and Ker Farn Lee. Fedsyn: Synthetic data generation using federated learning. arXiv preprint arXiv:2203.0500 059311, 2022.
[63] Matthijs de Goede, Bart Cox, and Jérémie Decouchant. Training diffusion models with federated learning. arXiv preprint arXiv:2406. / /2575, 2024.
[64] Jayneel Vora, Nader Bouacida, Aditya Krishnan, and Prasant Mohapatra. Feddm: Enhancing communication efficiency and handling data heterogeneity in federated diffusion models. arXiv preprint arXiv:2407. 14730, 2024.
15

---

[65] Zihao Peng, Xijun Wang. Shengbo Chen, Hong Rao, Cong Shen, and Jinpeng Jiang. Feder- ated learning for diffusion models. IEEE Transactions on Cognitive Communications and Networking, 2025.
[66] Fiona Victoria Stanley Jothiraj and Afra Mashhadi. Phoenix: federated generative diffusion model. In Companion Proceedings of the ACM Web Conference 2024, pages 1568-1577, 2024.
[67] Kyeongkook Seo. Dong-Jun Han, and Jaejun Yoo. Prism: Privacy-preserving improved stochastic masking for federated generative models. arXiv preprint arXiv:2503.0808: 2025.
[68] Yang Song and Diederik Kingma. How to train your energy-based models. arXiv preprint arXiv:2101.0 003208,...88.... 2021.
[69] Pascal Vincent. computation, 23(7):1661 2011.
connection between score matching and denoising autoencoders. Neural
[70] Santosh Vempala and Andre Wibisono. Rapid convergence of the unadjusted langevin al- gorithm: Isoperimetry suffices. Advances in neural information processing systems, 32, 2019.
[71] Tim Salimans and Jonathan Ho. Should ebms model the energy or the score? In Energy Based Models Workshop-ICLA 2021, 2021.
[72] Cynthia Dwork.
Differential privacy. International colloquium on automata, languages, and programming, pages 1-12. Springer, 2006.
[73] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-318, 2016.
[74] Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3 9(3-4)221--077, 2014.
[75] Frank McSherry. Privacy integrated queries: an extensible platform for crivacy-preserving data analysis. In Proceedings of the 2009 ACM SIGMOD International Conference on Management of data, pages 19-30, 2009.
[76] Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist// 1998.
[77] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 10 (canadian institute for advanced research).
[78] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[79] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv: (tt::::9900063335 2019.
[80] Jean Ogier du Terrail, Samy-Safwan Ayed, Edwige Cyffers, Felix Grimberg, Chaoyang He, Regis Loeb, Paul Mangold, Tanguy Marchand, Othmane Marfoq, Erum Mushtaq, et al. Flamby: Datasets and henchmarks for cross-silo federated learning in realistic healthcare settings. Advances in Neural Information Processing Systems, 35:5315-5334, 2022.
[81] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[82] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019.
16

---

[83] Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metries for generative models. In International conference on muchine learning, pages 7176-7185. PMLR, 2020.
[84] Tianshi Cao, Alex Bie, Arash Vahdat, Sanja Fidler, and Karsten Kreis. Don't generate me: Training differentially private generative models with sinkhorn divergence. Advances in Neural Information Processing Systems, 0el:12480-12492 2021.
[85] Tim Dockhorn, Tianshi Cao, Arash Vahdat, and Karsten Kreis. Differentially private diffusion models. arXiv preprint
2022.
[86] Wei Deng, Qian Zhang, Yi-An Ma, Zhao Song, and Guang Lin. convergence federated averaging langevin dynamics. arXiv preprini arXiv:2 2021.
[87] David Duvenaud, Jacob Kelly, Kevin Swersky, Milad Hashemi, Mohammad Norouzi, and Will Grathwohl. No meme for me: Amortized samplers for fast and stable training of energy-based models. International Conference on Learning Representations (ICLR), 2021.
[88] Yaxuan Zhu, Jianwen Xie, Yingnian Wu, and Ruiqi Gao. Learning energy anergy-basse based models by cooperative diffusion recovery likelihood. arXiv preprint urXiv:2309.05/53, 2023.
[89] Jiali Cui and Tian Han. Learning energy-based model via dual-meme teaching. Advances in Neural Information Processing Systems, 36:28861-28872, 2023.
[90] Gareth O Roberts and Richard Tweedie. Exponential convergence of langevin distributions and their discrete approximations. 1996.
[91] Simon Duane, Anthony D Kennedy, Brian Pendleton, and Duncan Roweth. Hybrid monte carlo. Physics letters B, 195(2):216-222, 1987.
[92] Radford M Neal. Monte carlo implementation. Bayesian learning for neural networks, pages 55-98, 1996.
[93] Tejas Jayashankar, J Jon Ryu, and Gregory Wornell. Score-of-mixture training: Training one-step generative models made simple. arXiv preprint arXiv:2502.09660 2025.
[94] Sanjam Garg, Aarushi Goel, Somesh Jha, Saeed Mahloujifar, Mohammad Mahmoody, Guru- Vamsi Policharla, and Mingyuan Wang. Experimenting with kero-knoweedge proofs of training. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 1880-1894, 2023.
[95] Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. Tabddpm: Mod- elling tabular data with diffusion models. In International Conference Machine Learning. pages 17564-1755 1759.. PMLR, 2023.
[96] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffussin models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10684-10695, 2022.
[97] Paul Voigt and Axel Von dem Bussche. The eu general data protection regulation (gdpr). practical guide, 1st ed., Cham: Springer International Publishing, 2017.
[98] Accountability Act. Health insurance portability and accountability act of 1996. Public law, 104:191, 1996.
[99] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:761 1-7623, 2020.
[100] Xiaosong Ma, Jie Zhang, Song Guo, and Wenchao Xu. Layer-wised model aggregation for personalized federated learning. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 10092-10101, 2022.
17

---

[101] Seok-Ju Hahn, Minwoo Jeong, and Junghye Lee. Connecting low-loss subspace for personal- ized federated learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 505-515, 2022.
[102] Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. Advances in Neural Information Processing Systems, 32, 2019.
[103] Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. Advances in Neural Information Processing Systems, 33:6637-66411 2020.
[104] Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, et al. Opacus: User-friendly differential privacy library in pytorch. arXiv preprint (Eti::2009112996 2021.
[105] Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. Data-free knowledge distillation for het- erogeneous federated learning. In International conference machine learning, pages 12878-12889. PMLR, 2021.
[106] Ali Reza Ghavamipour, Fatih Turkmen, Rui Wang, and Kaitai Liang. Federated synthetic data generation with stronger security guarantees. In Proceedings of the 28th ACM Symposium Access Control Models and Technologies, pages 31-42, 2023.
[107] Zhendong Wang, Yifan Jiang, Huangjie Zheng, Peihao Wang, Pengcheng He, Zhangyang Wang, Weizhu Chen, Mingyuan Zhou, al. Patch diffusion: Faster and more data-efficient training of diffusion models. Advances in neural information processing systems, 36:72137- 72154. 2023.
[108] Xinlong He, Yang Xu, Sicong Zhang, Weida Xu, and Jiale Yan. Enhance membership inference attacks in federated learning. Computers & Security, 136:103535, 2024.
[109] Georg Pichler, Marco Romanelli, Leonardo Rey Vega, and Pablo Piantanida. Perfectly accurate membership inference by a dishonest central server in federated learning. IEEE Transactions Dependable and Secure Computing, 21(():4290-42) 4296, 2023.
[110] Djalil Chafai and Joseph Lehec. Logarithmic sobolev inequalities essentials. Accessed on, page 4, 2024.
[111] Hong-Bin Chen, Sinho Chewi, and Jonathan Miles-Weed Dimension-fre freee log-sobolev inequal- ities for mixture distributions. Journal of Functional Analysis, 281 2021.
[112] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-an diffusion models. https: / //gi thub. www.hhh..ooooooeeeeeeemooe.oeeee com com/huggingface! diffusers, 2022.
[113] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito. Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035, Curran Associates, Inc., 2019.
[114] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[115] Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study. In 2022 IEFE 38th international conference on data engineering (ICDE), pages 965-978. IEFE, 2022.
18

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]-
Justification: We clearly mentioned our contribution (i.c., federated synthetic data generation through cooperative inference, without exchanging model parameters in traditional federated learning) and scope (i.e., cross-silo FL. setting) in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification: We secure a separate section to discuss limitation of our work and its potential remedy at the end.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "LLimitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
19

---

Answer: [Yes].
Justification:
We provided separate sections for detailed derivations and proofs in appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes].
Justification: We provided code link and separate sections for experimental details and configurations in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
20

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes].
Justification: We only used publicly available datasets in our experiments, and we provided self-contained code implementations link.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
Stapic/gguidd//ooeuuue www.lmisssnooooccyyy...... for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ httppppcccppbppccccpeeeeeeclllet ggwuides////eeeeeeoeo /CodeSubmi ssi asionPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification: We provided separate sections for experimental details and configurations in appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes].
Justification:
We faithfully report all results with reproducible details.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes].
Justification: We provided separate sections for experimental details and configurations in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: :tttpp:://euure http///eerripppeeeee hwtpp//bbbbcc/icccccuu.....eeeed
Answer: [Yes].
Justification:
We sincerely understood and followed research ethics, as in NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification:
We described broader impacts statement the end of the manuscript.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
We do not release any data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
Justification: We clearly described and cited related assets (data, models, previous findings).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswi www.wwtttt......eee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
23

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes].
Justification: We only used publicly available datasets in experiments, and provided code implementations as link.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification:
We do not use crowdsourcing or experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: Our work did require IRB approval and not involve any related risks.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
24

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA].
Justification:
We did LLMs except for polishing some sentences and table formatting.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy (https : /tteuripp..c/ooofeeeeeeeeeeeeeee cc/Conffrrences/2225555 for what should or should not be described.
25

---

Appendix
Table of Contents
A Derivation of Gaussian Diffusion Models
26
B Connection of Energy Based Models and Diffusion Models
28
C Proofs
29
C.I. Proof of Lemma 4.3
29
C.2 Proof of Lemma 4,4
30
C,3 Proof of Theorem 4,5
32
D Experimental Details
38
A Derivation of Gaussian Diffusion Models
Diffusion models are a class generative models that aim to learn data distribution (aura()) = (v)) by learning to transform random Gaussian noise into original data through iterative denoising process. other words, the underlying Markov chain from the noise (xT) to the data (xa) defines diffusion models, and they are realized by two main processes: forward process and reverse process.
In the forward process, data æ q(zo) is gradually perturbed over T timesteps by adding Gaussian noise
((x,<=;-1) N(m: v 3,0, BeTa);
where Be E (0, 1) controls the noise schedule, until øT (O2,,, Ia). Other constants satisfy ar 1 8g and a, = I (ty. Thus, the forward process models g(x1,,.., ær (2o) = II g(2x)) (x)-1)
To reiterate, as in Eq. (2), diffusion models have useful property that enables calculation of anytime marginal distribution a closed form:
q(æ; (x)) = N(xx v Vārzo, (1 - az)
By training parameterized deep network, diffusion models can denoise from noise to the data by approximating the true posterior Pe (xx)) ~ q zu), through the reverse process as defined in Eq. (1). Thus, the reverse process models po Po .... ... p(ær) IT (o (x,).
With these two paired processes, diffusion models maximize the lower bound of likeliio defined as:
po pa (oo ØTT
log log (xro) 2 E Byg(e (rl()) log q( FT x0))
log (oggo))
DKL. (g(01 27 (20) || (m ar (z)))
= llog togpo(2to)
Σ (g( mo) po (2:-)
where the decomposition due to the Markov property of both forward and reverse processes.
From this, we can maximize the lower bound of log-likelihood by minimizing the sum of KL. divergence terms instead:
(x pe (0(xx-11x)))
26

---

Since both q(<;-110 To) and pe (It-1 (z,) are Gaussian, the KL divergence simplifies to mean squared error between the true noise e and the estimated noise c t). Hence, we have
(k po(
a (o0 [|le EØ (2 1)
(t)(
where o, is weight that typically treated equal 01 = aT 1 [9] for all time-dependent loss L(O,7 t), which was defined in Eq. (3).
After the training is completed by optimizing the above composite loss 2()) = Et= C(O, t), we can draw samples through ancestral sampling: starting from ar (O,,, Ia) using (3t L), due
to the connection 2t-1 = (0 t) + V Bic, <~ <~N(0, Ia) from the reverse process. Note here that Po (3r, 1) is computed from the estimated noise eo (æt, l).
27

---

B Connection of Energy Based Models and Diffusion Models
EBMs and diffusion models share a profound theoretical connection through denoising score matching in Eq. (5). This connection not only provides an alternative interpretation of diffusion models but also enables ULA Eq. (8).
Again, EBMs model an unnormalized probability density of the form of:
(a(x) exp(-Afo(z))
Zo
where fe: I2 + R is the energy function parameterized by IP, À R+ is a scale factor, and Ze JeEx exp(- dæ is the normalizing constant, which is typically intractable in practice.
Due to the intractable property of the normalizing constant Ze, direct computation the likelihood is challenging. This necessitates alternative training methods using score, defined as follows. The score is the gradient of the log-des density as:
V m po (a:) (ovx))
To train EBMs, we use denoising score matching objective [69] in Eq. (5) to minimize the Fisher divergence between the score of model's distribution and the score of noise-perturbed data distribution:
/
q(<a)
To (Zo x Pdata (x) dæ,
xEx
where the perturbation is realized as:
Lig ae,
NO,, La),
and is the noise scale. Hence, the DSM objective is:
L(0.0) Er, mug( xe VI, log q( (x)) Vos log pa (xx) )
Rewriting g(x) NN a, Ia), we can explicitly have that
Va, log (x) (xor x) OE.
0²
By substituting the EBM score as Va, log logro( = -V a. (o(x))), the objective becomes equiva- lent (up constant) Eq. (6) as:
L σ) l0)) [lle 2o
Diffusion models, defined in Section A, optimize a similar objective. To reiterate, the objective of diffusion models is given as:
((,, = Emg.c [||c ca 1112
From this, we can easily draw an analogy with DSM objective, by replacing the with time- dependent noise scale de, with the score interpretation as:
V no, log logpo((aa, = - AV no, So( = co(2), t)
This connection shows that the noise prediction co (a, in diffusion models directly corresponds to the score of an implicit EBM, if scaled by the noise level 0N, Thus, diffusion models can be viewed learning EBMs implicitly where the score approximated by the noise prediction network parameterized by This energy-based interpretation allows for alternative sampling methods in diffusion models, such as ULA in Eq. (8). Note that for the compositional generation, this requires the energy-based parameterization tricks of diffusion models, discussed in Section 3.3 and Section 3.4.
28

---

C
Proofs
C.1 Proof of Lemma 4.3
Proof. In this proof, we need show EE/ || - < 0 from diffusion reverse process in
Eq. (1). First, following [9], equivalently define for variance schedule constants 3t, / [7] that other constants are defined as follows.
1 a+-1 ā II ar, 3.= Br 04
04 = Bis
Recall that the reverse process in Eq. (1) can be written as
1
It-1 Mo( t) Biz
xt
co( + Biz, =~N(0s, Ia).
/ak
VI ai
From this, have from the law of total expectation that
E
111-1111 11
-E
[1 (ze, t)]
3,d
For []] t)]]]] we have
E
[1 (2e, 1
23,
E
[|
[12]
E
(ar,
(2t, ())]
E
[[lEo (wt, ()
at
VI
āt
1
2
{8
E3
11
E [xx, E)]
+
E [||=|12]
Definition 4.1 at
v1
αt
2,9
B3 d
|||0
[2
VI and
at
VT- 04
[|
[2]]
233d +
1 α4
where we used E Thus, we have
[||||]]] d for any Noaa. Ia) and E
[ax, c)]
vI and from Eq. (2).
(--1) -1)
23,d
3dd
E Dr 112 ||00
[z [1
3,d
on
an -xa)
From Eq. (2), have
(1||011191 = a.E [|| 112]
+ (1 - a
Zo and are independent and E [|| [ d.
29

---

Using this, we derive
1
11
-
|| [2
1
23,d
5.d
1
(ā
+(1 ā,)d)
+ 3id
a2
a
ar - āx)
1
2/
+
+
B
d
04
an
ā()
1
28
83
1 Ū(-1
31
04
at
ar
at)
1 at
2/
1 á $1 d
04
a
a
B
28
- u2-1 1
3r d
at
Q1
ar(1 - at)
1 - σt
3
a1-1 Br
+
ar(1 - -(r)
1 - dt
Br
(-(1-ah +Br arol1 (z-1)),
(1 āt)
where the first inequality is due to | S vd boo, Vo E Rd, along with typical assumption in diffusion models that | = 1 as inputs are normalized into [-1, [9]]]]] [9].
Rearranging, we have
E
1
|
Br
(-(1 (11-00 (1)=))) + x(-1))
ac(1
(-11- B.) + a0 + + ad=-1))
ar(1
āt)
B1
(-as + + aāt-1))
ar(l āt)
31
(-a: + a + ā. ā.)
ar(1 - āt)
where the second last and the third last equalities are due to the definition of ā, and 3, each.
We finally have E [||=;-118 [ S 0. thus E [ < E [[]=:1]]]
C.2
Proof of Lemma 4.4
C.2.1 Proofs
Proof. In this proof, we need to show EE [||æ;-1 12 | S from ULA update in Eq. (8).
With the energy-based & parameterization in Eq. (11), denote from ULA update that
Xne Ar = 24-1 - 2t vI ai
co (ate; 1) + v 21 2, ~ (Oa,, Ia).
With this, we have that
| - 11 = ||00 + Ax || 2( Axs) + || 1
Now, taking expectations over z; and a0, l2zo, have that
Exz
[iz, || |
Em
Ez, 2(xxe Axr) ||Ax
(A1)
2EE /mo ((x,, Ez, |
E mo EZz Ar, []]]
30

---

Let us demystify the inner expectation first. Since Ez, [21] = O, we have that
AnR
Ex, (Amr)
< ()-
v1 at
Next, for Ez,
|I| A
[)). we have that
X 2
2\n / (co (a,,, t), 21) + 2m ||
|1
xz t)ll
1 α
v1 On
1²12²
2AT v 2T
|00 1)112
(Eo 1), B2, (2))) + 2np Ez 111211
1 ōi
v1 -1
X nt
IEg ( 1)112 2nddd
1 az
where [1] = d for 2, ~ (N,,,
To sum up, for the inner expectation of Eq. (A1), we have that
2 (a,, co(; + 1 ār
2 ( t)l12 + 2ndd.
Ez, 112 |
V1 ā
Going on for the outer expectation, we have that
Egg, (00
Pzz,
||x;-
112
||ac, [|ac.112]
2
A2 Ez, mg (1) t) + 2n,d.
Em, Ima
[x ca t)} + 1 ā,
VI
Ot
Since it is for well-trained diffusion models, we have by using Eq. (2) that
Emg 1000
Ex.
|
-||xx; [
2A17:
Ite 2t Vā,z0
}
Emy
V1
drt
V1 Ctt
(A2)
2
2
Dt va
Ex
+ 2mdd.
1 āi
v1 Cxe
2
From this, the first conditional expectation becomes that
( ( Ft
Vo,30
mm
Dti vI Õi
Bailas ( xt
Ex, |mo
|| 112
von (2ees 20)
VI Oe
VI - Cce
where the former term inside the expectation is that
d
d Σ En, 1a00 [c2:)]]
Ex, (ano ||00|||]] =
Esg, Jao
E(x))
(xxt)
Exe Jaza
Variaae [200] + (E62ze, 1200))
i=1
Σ
(1 āe) + (Vārzo..) (1 (1-aa)d ar 12001
and the second term inside the expectation is that
d
Ex, (mo Kox, xo)| = E Imo Σ L2,
Σ
Fe L. mmo [x),1]]
i=1
d
Lv,i (Vā = VāL Σ
2
Vārl
31

---

Taken together, we have for the first conditional expectation that
Be mm ((a 2t,
1e--vash Vāreo -T-ai 0t
|
at (11 at)d ãe |00ll2 var
vāu
v1 -a,d
VT -
Next, for the second conditional expectation term in Eq. (A2), we have that
Eg (00
Reg
[|| d,
due to Eq. (2) and is for well-trained diffusion models.
Putting all together, the original expectation in Eq. (A2) becomes that
E mz
Ez,
| 22
2\TR:
2²ri.
v1 ard +
dl 21d
vI āt
az
22m
22 +
+ m/d.
0t
Since we want to guarantee this term to non-increasing for the non-expansiveness w.r.t. 1.2 norm as in Lemma 4.3, we need to have that
2 + mi S 0
(-21-
Due to d > 0, v 2 and À > 0, we have that
1²11
2(A 2(A-1) 12
2X +
+250
(1 (1-
To ensure / 0, we should have 1. From max; / -- 0z we can conservatively set
2(A - 1)
S
X2
As (A) =
2(A-1) X² has its maximum in A A2 when g(2) = 1,2 we have E [0, when A = 2.
C.3 Proof of Theorem 4.5
In this section, we present materials related to the proof of Theorem 4.5. For the convergence analysis, adapt the assumptions and result of [70]. First, we introduce the essential definitions, then we provide the technical lemmas and present proof of the main theorem. Note that these proofs demonstrate the exponential convergence of ULA under the minimal isoperimetric condition (i.e. the Log-Sobolev inequality). without the need for strict and often impractical assumptions such as log-concavity boundedness of higher derivatives [70].
C.3.1 Definitions
Definition C.1 (Kullback-Leible (KL) divergence). The Kullback-Leibler (KL) divergence of p with respect to is defined as
p(x) ((x))
DKL 11 - p(x) log
dæ.
Definition C.2 (Log-Soboles Inequality (LSI)). A probability distribution p satisfies the log-Soboley inequality with constant > if for all smooth function g Idd - IR with E, [g2) < oc, and
[g log g21 1E log Ep < ES [.l1%
32

---

C.3.2 Technical Lemmas
In this section, we introduce the essential lemmas and corollaries required to prove the main theorem, i... Theorem 4.5. Note that we omit the proofs of adapted lemmas and refer to the original paper cited, i.e. Lemma C.3, Lemma C.7, and Lemma C.8 (for the adapted intermediate result).
Lemma C.3 (Strong convexity and LSI; Corollary 5.11 of [1]0]. Let μ be probability measure on I of the form du = exp(-h(xx))) the LSI with constant 7,
If h satisfies h()) 2 yla some y > 0 then μ satisfies
Corollary C.4 (LSI of well-trained diffusion models with non-expansiveness guarantee). A well- trained difftusion model with L.2 norm-driven anergy-based reparameterization as in Eq. (11) and Lemma 4.4 satisfies LSI with constant 2
a-a.
Proof. For a well-trained diffusion model, we have po(x;, () x 1 12) from Eq. (11). With the property of well-trained diffusion models stated in Remark 4.2, have that
22
E x. 1)
v., c000), 1)13 = Va, (Aco( V co(2x, t)) -Va, A 2
Va.
(vT-ac co(
A
V co (z VI az VT-
āi ls-
Due to Lemma 4.4 and Lemma C.3, the LSI constant y of well-trained diffusion models with L2 norm-driven energy-basee based reparameterization that guarantees non-expansiveness w.r.t. L2 norm is given as 7 = T-o, since À 2.
Corollary C.5 (Lipschitz smoothness of an energy function of well-trained diffusion models with non-expansiveness guarantee). A well-trained diffusion model with L2 norm-driven energy-based reparameterization is I-a,"
2 Lipschitz smooth.
Proof. It is directly implied from Corollary C.4.
Assumption C.6 (Bounded dissimilarity). The pairwise chi-squared divergence between two different local distributions is uniformly bounded by x, suPişjj[K]] X² (p; ll Pÿ) < K oc.
Lemma C.7 (LSI constant of a mixture of distributions; Theorem I of [111]). Denote mixture of distributions p² := Ex= for w; 2 0, ΣK w/ = where each pt satisfies the LSI with "ht If Assumption C.6 holds, then px also satisfies LSI with a constant of
min 7/
3(++)((1 logg] x))
Lemma C.8 (One-step contraction of ULA; Lemma 3 of [70]). Let x, ~ Pr be the output iterate one-step ULA. In one step, ULA can sample from a distribution pt = po(;, encoded by a single well-trained diffusion model, satisfying
((32-) 3 - DKL (5. p:) + 6dcTt.
DKL (pt+1 || Pt+1) S exp
(A3)
with step size < Th <
where L, is Lipschitz smoothness constant, 7/t is LSI constant, p 2 Le
Ste
1
and 0 < <<<<
Proof. Consider the continuous interpolation 7r. where mpl with
Pr-o- Pt, Pr-m. D
(A4)
Denote the LSI constant of a distribution encoded by well-trained diffusion models as 7 and the Lipschitz smoothness constant as L,, For all - E [0, m.) we can directly adapt the intermediate result of Lemma of [70] as
DEL (ūr pp) s 3%r DEL. (pr pr) + DRL (po || po) + 2dr L3 2dt L2. dr
47
33

---

Denote
4
A,
DKL (po po) + 2d0 ; +2dTLI L.
Yr
and introduce the integrating factor as
3 (M(T) exp 2 % 7ods)
We wish to integrate over 7 to T = Tjt, thus we have T S Tt. Further assume that for any
[0, m7] we have
"tt S 77 1 L, 1 L
(A5)
Then, upper bound as
4n Li
A,
DKL. (5o ll po) + 2dh Li + 2d Li == Ass
(A6)
7
as it becomes irrelevant to T. Then, we can rewrite the inequality as
dr (u(T) DKL (fr || px)) S p(T)
Integrating this inequality from == 0 TJt we have that
(()T) DKL (pre ll Pm.) - DRL (go Po)
Ae u(T)dT = Ae 1/d exp ("
Jn Yods dr
Rearranging, we have that
3 n
DEL (pr, ll Pm.) < exp
7y
DKL (5o ll po)
2
3 m
0 /
7rd7) /-"' exp 2 (
(3
A exp
rods)
dr
n
I exp
Yt
dr
DKL (0o
po)
3
n
( exp
A exp
2
it
Jo
dr)
dr
LiT dr
exp
DKL (pu
Po)
3
2
3
-1).
A, exp
717
exp
LiTe
2
3L,
2
where the second inequality is due to Eq. (A5).
Using the inequality that e 1 + 2c for 0 <
2 Lem - 1 (which holds due to the assumption
that 7/2 I +.. < अे. along with Eq. (A6) we have that
DKL. (p,, Pl))
3
DKL. (pn Po) As exp (
3 (-2101)
< exp
2 7
2nj
3
(1 1+ 8n8 1.
exp
2 7e7
DEL (Po 1l po)
+ exp
(Admi L + 4dmi Li))
3
(1-
877
< exp
2 7e7c
DKL (po ll po) + (Adn) L3 - 4dnî Li))
7/.
34

---

where the last inequality is due to exp
(-2 2 70m) - 1.
Using the assumption that T) S STe
we have
T.
8mp L
85 14 +1+ 85 S exp 7887
Thus, the inequality above becomes that
3
DKL. (Pm. || Pv.) S exp
2 wh
exp (852 DRL. (Do po) + 4dr Li + Addi L
3
exp (852 DEL (Do po) + 4dn; I (n I + 1)
exp
7
2
exxp (185²
-1)) DEL (0o po) + 4dm L +1)
As 11e
So S 21 14
for p 2 1, we have n,L, S 1 and n me S
(8x2
DKL (pm, || Pm.) S exp
2 700
DKL (po po) + 4dn L3 (neLi + 1)
S exp
85²
DEL (po |1 po) + 6dsm.
2
Finally, replacing with Eq. (A4), we finally have that
DKL (p++1 |l Pc+1) S exp ((x2- 2
7elc DKL (pr ll pc) + 6dsn.
(A7)
Lemma C.9 (Convergence of ULA in KL divergence). Let Pt po(:, be the probability distri-
bution defined by a single well-trained diffusion model and let B. = -52 > Assume that the
iterates of Pr are generated by the Unadjusted Langevin Algorithm (ULA) in Eq. (8), and that
(PL po) < co. Then, all 2 O, we have
DKL(P: || pt) I exp ( (BB,tyeooo) (PL |l Po) + 9dcm
(A8)
Hence, for any 8 2 18ds it suffices to run ULA for B,y
DRL (po |l Po)
log
B
8
steps with step size
B 57h
Te S min
18ds
1 {:
for p and LSI constant Tb in order to guarantee DKL (p: Pc)
Proof. From Lemma C.8, recursively applying Eq. (A7) gives
- B. [-1
[-1
(p pt) < exp
Yolls
DKL Po) + 6ds Σ exepp
- B Σ
Ys
==0
s=r+1
Since have 77
170 and 70 ... Nh, can bound that
Malls 2 t7:700
Yolls 2 (t-r- 1)7
35

---

Because B, > O, we get:
DEL(Ft || pt) S exp(-BBl7eno)) (RI |l po) + 6ds Σ T1t expl -
The remaining sum is for geometric series, thus
(on-Bo)))))) exp(- Be - exp( - (ontt)
expl- (1)1))
< 1
3
Sxp(-Ba,700
2
where the last inequality uses
2c
3 T S - - 0<c= B 55 <
from Lemma C.8.
Thus,
DKL || Pt) 3 exp( (xp(-B,tyyon)) (KL Po) + 9dcn:
B
To ensure (ft p+) S 5, it suffices to assume:
9dsn:
S hyp(-Bttyeny (KL |l po) S 2.
Bo
which hold when
2 DKL (Po po)
ne v 18ds
and t t B3yo00
log
5
C.3.3 Proof of Theorem 4.5
Denote Pti = po. (- as a distribution encoded by a locally-trained diffusion model of client i. For the mixture of distribution pf = Σ W.Pa. it is trivial that the energy function of p is Li = - Lipschit smooth since each local distribution is Lipschitz smooth due to Corollary C.5.
From the result of Lemma C.7, the LSI constant of the mixture is y = minselk] local distribution has LSI constant "it T-a," we can further refine 3(++))++Ho((++x) As each
2
3(1 (1)(1 + (1 + log(1 + x))
Denote k 3(1+x](1+og(] +x))
2 set the LSI constant as 72
R
I-ac"
From the result of Lemma C.9, finally have that
9dsm)
(kr(ft || p() I exp ex (-Bel7in no) DKT. (%6 p5) + B n10
BeRmpo
9ds (1 kx.
exp
(KI ll 16) + 1
B.Rojo
with step size
1o 57t (L;)++I
B sh(l 5.)P 18ds(1 - -a))
Me S min
min
18dç
2P
In practice, however, we cannot directly quantify R. Thus, we instead manually adjust a constant /3
ç 6.
Further denote := B.Rmo
36

---

Finally, we have that
(-1 ut
(rt |1 pī) + 9dsmi (1 - at)
DKL 11 pt) s exp
with step size
vd
p(1 044)
7) S min
18ds ā()
2
for any 5 2 (1-a)) in / 1-a: log
Poollpollaee po)
steps. Finally, replacing 0 - T and
t T for the compatibility with ULA reaches the theorem statement.
37

---

D Experimental Details
Specification. We conduct all experiments in a single server with Intel Xcon Gold 6226R CPU (@ 2.90GHz) and single NVIDIA@ Ampere@ A100 GPU (w/ 40GB VRAM). For the implementation of diffusion models, we resort to diffusers 12] library using PyTorch [113].
Simulation of Statistical Heterogeneity. For the faithful evaluation of practical FL setting, we simulate non-lID data split to K 10 clients for all benchmark datasets.
For MNIST dataset, we use Dirichlet distribution with concentration parameter a = 0.1, following the setting of [79].
Sample Composition (MNIST)
300
Classes
250
Cass
150
Class
Class
50
2008 Cass 7
125 Cass B
Class
Figure A1: Non-III) local distributions of MNIST dataset
For CIFAR-10 dataset, we follow the setting of [21] using log-normal distribution with location=0 and scale=2.
Sample Composition (CIFAR-10)
Classes
Class
3B Class 2
200
Cass
205 Casss
8008 Cass
Class
Cleet 10
Figure A2: Non-IID local distributions of CIFAR-10 dataset
For CelebA dataset, which has 40 different attributes, we first construct classes by combining gender (male/female), smiling (0/1), and eyeglasses (0/1) attributes, i.e., 8 classes as result. We randomly distribute samples to clients that they have only three distinct classes.
Sample Composition (CelebA)
Classes
Cass 2
Class
Class
200
A02
2081 Clss 7
Figure A3: Non-lID local distributions of CelebA dataset
38

---

Model and Training Hyperparameters. for experiments Table Al.
We summarize detailed configurations of models used
Table A1: Model and Training Configurations.
MNIST
CIFAR-10
CelebA
Model Configuration
Spatial dimension
32 x 32
Attention resolution
8
Base channels
128
Channel multipliers
1, 1, 44.77MB
1, 2, 2,2
Model size
136.38MB
136.38MB
Base architecture
DDPM [9]
Scheduling scheme
linear scheduling [9]
Training Configuration
Optimizer
Adam [114]
Learning rate
2 x 10-4
39

---

40


Reviews:
[{"review_id": "d77qXB4Q1w", "reviewer": "Reviewer_d6e9", "summary": "The core idea of ​​this paper is to regard the problem of federated synthetic data generation as a task of \"collaborative sampling from a target distribution mixed from all client local distributions\". Instead of exchanging or aggregating model parameters, the server and each client repeatedly exchange noisy prediction scores (energy-based scores) on a small number of samples, and use the unadjusted Langevin algorithm (ULA) to iteratively sample on the global mixed distribution, thereby efficiently and theoretically provably generating high-quality synthetic data in a non-IID environment, while greatly reducing communication overhead and naturally supporting differential privacy.", "questions": "1, If the data differences between clients are too large, will COODI still work?\n\n2, COODI uses the mean aggregation result. If it is attacked by malicious attacks, how should COODI respond?\n\n3, At the beginning of the iteration, when the samples are still almost pure noise, the gradient given by the diffusion model is very weak, resulting in a small update amount for each step of ULA, low information content, and the lowest sampling efficiency. How does COODI overcome this problem?\n\n4, If the quality of the diffusion model on each client is poor, is it possible for COODI to improve the quality of the server-side model?", "limitations": "1, Each client needs to have a well-trained diffusion model\n\n2, Each client needs to be able to respond in time, which is difficult to promote to large-scale cross-device\n\n3, Although the communication burden is reduced, multiple ULA updates must still be performed on the server side in each round, and noise prediction must be run on the client side. The overall computational cost is still not negligible when the number of nodes is large.\n\n4, The performance differences of each client diffusion model will directly affect the accuracy of the hybrid scoring. If a node model is too underfit or has too large a deviation, it will drag down the global generation quality.\n\n5, The sampling efficiency of ULA is sensitive to the step size. If the step size is too small, the convergence is slow, and if the step size is too large, it may deviate from the target distribution. The tuning cost cannot be ignored.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The quality of this paper is acceptable. The proposed COODI aims to significantly reduce communication overhead and provides theoretical support. I think the clarity of this paper still has room for improvement, especially for some beginners and laymen, it may not be intuitive enough and difficult to understand, but the main method part of the paper is clearly explained. The importance of this paper is moderate, because COODI requires a fully fitted diffusion model to be trained locally, which may bring huge costs under conditions of large data or multiple nodes; when the quality of data from different clients varies greatly, the usability of COODI still needs to be considered; the initial information entropy of the diffusion model is low, and the use of COODI at this time may cause a waste of bandwidth. In addition, the simple weighted average of the energy of each client is also a problem. The originality of this paper is average, and the idea of ​​using a diffusion model to generate synthetic data from GAN is not completely novel.", "was_revised": false, "final_justification": "", "rebuttal": "Dear  **reviewer d6e9**,  \nWe sincerely appreciate the reviewer’s time and effort, and we have carefully addressed each question in detail.\n\n----------\n### 1) Case of extreme data heterogeneity \nWe appreciate the reviewer’s insightful question regarding `CooDi`’s performance under extreme data heterogeneity.   \n- In cases of significant data heterogeneity across clients, the ULA may require additional iterations to effectively mix disparate local density distributions, potentially impacting sample fidelity in extreme scenarios.\n- However, such extreme heterogeneity is rare in practical FL settings, where client data distributions typically exhibit **bounded dissimilarity** rather than arbitrary differences (cf. Assumption 7 in [1], Assumption 3 in [2], Assumption 1 in [3]). This ensures that local distributions, while varied, remain within reasonable bounds.\n- Theoretically, `CooDi` assumes **bounded pairwise chi-squared divergences between local distributions** (Lemma C.6 and Theorem 1 in [4]), aligning with the _bounded dissimilarity_ principle expressed in probabilistic terms. \n  - Theorem 4.6 further indicates that large distributional differences necessitate smaller ULA step sizes, increasing the number of iterations required for convergence.\n  - Empirically, `CooDi` demonstrates robust performance even under heterogeneous data settings, as evidenced by our results (lines 252–253, Tables 1 and 2). These findings confirm that CooDi maintains acceptable sample quality and convergence in practical FL scenarios.\n\n### 2) Possible attack and defense scenario\nWe appreciate the reviewer’s concern regarding the potential vulnerability of `CooDi`’s mean aggregation to malicious attacks. \n- As we specified the target setting as **cross-silo FL setting** (line 56; lines 278-281), `CooDi` assumes **a moderate number of reliable, and known, and addressable clients** [5].\n  - Consequently, our initial design did not prioritize malicious attack scenarios, as cross-silo settings typically involve trusted participants.\n- Nevertheless, we acknowledge that mean aggregation can be susceptible to adversarial manipulations, such as scaling attacks where a malicious client submits inflated or deflated scores. \n  - `CooDi` can successfully mitigate such risks through its theoretically grounded design. Specifically, Lemma 4.4 ensures non-expansive updates in the $\\ell_2$-norm, driven by **theoretically-guaranteed step sizes** (Lemma 4.5, Theorem 4.6). \n  - This property allows the central server to detect anomalous updates by simply comparing the norm of a client’s scores against those of others, effectively limiting the impact of scaling attacks.\n- To further enhance robustness, we plan to explore advanced aggregation techniques, e.g., trimmed mean or median-based methods as part of our future work, to address a broader range of practical adversarial scenarios. \n\n### 3) Concern on early sampling stage\nWe thank the reviewer for highlighting the challenge of sampling efficiency in `CooDi`’s early ULA iterations, where noisy samples can lead to weak gradients and smaller updates. \n- To address this, `CooDi` employs a **theoretically derived optimal step size** (Theorem 4.6), which ensures **non-asymptotic convergence** in practice. \n  - Our empirical results (e.g., Tables 1 and 2) are also based on this derived step size, demonstrating robust performance despite early-stage noise.\n- To further enhance convergence speed and sampling efficiency, we propose adopting advanced sampling methods, e.g., the Metropolis-adjusted Langevin Algorithm (MALA [6]) or Hamiltonian Monte Carlo (HMC [7]) in Limitations & Discussion section. (lines 282–284) \n  - We believe these methods could easily reduce the impact of weak gradients and accelerate convergence, but require major changes in design of `CooDi`. \n- We plan to investigate these approaches in future work.\n\n### 4) Effects of local diffusion models on the quality\nWe thank the reviewer for emphasizing the importance of local diffusion model quality in `CooDi`’s collaborative framework. \n- As noted in Remark 4.2, Eq. (9) and (10), and Lemma 4.3, the success of `CooDi` relies heavily on **well-trained** local diffusion models to ensure high-fidelity synthetic data generation. \n  - To support this in practice, we propose an add-on protocol, such as zero-knowledge proof (line 280), to verify that clients faithfully train their local models, enhancing the reliability of the collaborative ULA.\n- Additionally, we clarify that `CooDi` **does not produce a server-side model**, as illustrated in Figure 1, which provides an overview of our federated synthetic data generation process. \n- Instead, `CooDi` aggregates local score estimates to generate server-side synthetic data without centralizing model parameters. \n  - We will update the paper to further emphasize the importance of local model quality and the absence of a server-side model for clarity.\n----------\nWe are grateful to **reviewer d6e9** for constructive feedback.\n\n[1] A Field Guide to Federated Optimization (Wang et al., 2021)  \n[2] On the Convergence of FedAvg on Non-IID Data (Li et al., 2019)  \n[3] Federated Optimization in Heterogeneous Networks (Li et al., 2018)  \n[4] Dimension-free log-sobolev inequalities for mixture distributions (Chen, Chewi and Niles-Weed, 2021)  \n[5] Advances and Open Problems in Federated Learning (Kairouz et al., 2019)  \n[6] Comments on “Representations of knowledge in complex systems” (Besag, 1994)  \n[7] Monte Carlo implementation (Neal, 1996)"}, {"review_id": "F0ZG18zDyL", "reviewer": "Reviewer_oT3q", "summary": "Synthetic data generation is an interesting task for federated learning (FL). The typical FL approach is for many data holders to compute gradients w.r.t to some model and send them to a centralized server that will aggregate them and send back model updates. As generative models such as LLMs and diffusion models become larger, the computation required to compute local gradients and the communication required to send model parameter updates back and forth become intractable. This work develops the method CooDI which is a FL procedure for sampling data from a mixture of local diffusion data without communicating model parameters or gradients. CooDI consists of two initial steps: each party trains a local diffusion model and the server generates an initial synthetic dataset of random noise. Then these steps are repeated: the server sends the synthetic data to the data holders, the data holders run inference with the local diffusion models, the data holders send the diffusion steps to the server, the server converts the diffusion steps to a score (as in grad logp), then the server weights and combines all the local scores for each record, then the server takes a ULA step with each synthetic record according to the aggregated score. The authors prove several results about this CooDI procedure including a required range for the ULA step size and a convergence result. The authors also evaluate CooDI experimentally and show that it performs well compared to several benchmarks.", "questions": "Seeing as the first step of CooDI is for each party to locally train a diffusion model, a simple baseline would be for each party to sample some records from their local model independently and share them with the other parties. This would lead to a synthetic dataset from an equally weighted mixture of each diffusion model. How is the model implied by CooDI different from this simple mixture? How does CooDI compare to the naive mixture experimentally?\n\nWhat happens if some of the local diffusion models are not “well trained” would CooDI handle this issue gracefully?\n\nDP diffusion model training is not trivial, how much does performance degrade when each client trains a DP diffusion model?\n\nThe table in the supplementary material doesn’t seem to include the baseline methods, were the hyperparameters of the baselines tuned?", "limitations": "CooDI alleviated the communication burden but it still assumes each party has the ability to train a local diffusion model which is still computationally demanding. This is mentioned in the paper but I believe it should be highlighted more. Also, the authors claim that CooDI complies with HIPPA, I believe this is a legal claim outside the scope of a ML conference and should not be included.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe authors seem to have advanced the energy-based parameterization of diffusion models by showing that no modification to the training procedure is required if the model is “well-trained”. This is a bit outside my area of expertise but seems to be a meaningful result.\n\nThe empirical results show that CooDI meaningfully improves on the baseline methods across most metrics and several datasets. \n\nWeaknesses:\n\nThe manuscript demands a lot of readers. Readers would benefit from some degree of familiarity with diffusion models, energy-based models, the recently discovered connection between energy-based models and diffusion models, federated learning, and a bit of differential privacy. I believe more could be done to make the manuscript more clear to a general ML audience (see the questions section for particular things that seem unclear to me).\n\nThe approximation for the score of a diffusion model assumes that the model is “well trained”. It is not clear from the paper if this is a reasonable assumption and no experiments are provided showing that diffusion models typically meet this standard.", "was_revised": true, "final_justification": "The authors helped clarify for me the motivation for COODI and I think that if the writing is improved in the manuscript it would merit a score of 4", "rebuttal": "Dear  **reviewer oT3q**,   \nWe are grateful for the reviewer's insightful comments and efforts for fully ruminating our work. We've also tried our best to address raised concerns below:\n\n----------\n### 1) Clarification on the 'well-trained' assumption & Failure case \n- We intended the term 'well-trained' to be close to 'pre-trained'. That being said, as long as each local diffusion model is (literally) _well-trained_ on the local dataset with sufficiently minimized loss, we are ready to operate `CooDI`.\n  - Therefore, we humbly believe that it is not only reasonable but also practical assumption, and several previous works on the composition of diffusion models accredit this [1-3]; they used pre-(or well-)trained diffusion models for combined sampling of multiple concepts.\n- If it's not met, `CooDI` may fail to generate high-fidelity samples due to low-quality signals. That's why we restrict our setting to a cross-silo setting (line 56), in which moderate number of relatively reliable clients collaborate [4].\n  - To encourage explicit compliance with the 'well-trained' condition in practice, we propose adopting _zero-knowledge proofs_ (line 280), to check and ensure each client faithfully completes the training of local diffusion models.\n- Based on the comment, we will further clarify these points in the revised manuscript.\n\n### 2) Comparison with the naive mixture\n- The proposed alternative (i.e., sampling from each local distribution independently and collecting the results) corresponds to an _explicit_ sampling scheme from a mixture distribution: i.e., $p^\\star (\\boldsymbol{x})=\\sum\\nolimits_{i=1}^K w_i p_i (\\boldsymbol{x})$. \n  - This scheme is more appropriate _when if local distributions $p_i(\\boldsymbol{x})$ are entirely disjoint_.     \n  - Note that this idea is partially aligned with `FedSSyn` [5] - but this method distributes the aggregated samples again to augment local dataset, and then conduct FedAvg. \n- However, in typical FL setup, it is unrealistic to assume such extreme separation among local distributions. It is because FL typically assumes local distributions may be heterogeneous, but with the _bounded_ dissimilarity, rather than arbitrary. (cf., Assumption 7 of [6] Assumption 3 of [7], Assumption 1 of [8])\n  - That is why our theoretical analysis (i.e.,  Lemma C.6; Theorem 1 of [9]) is also based on the assumption that the pairwise chi-squared divergences between local distributions are bounded. This assumption aligns with and captures the \"bounded dissimilarity\" assumption in probabilistic language. \n  - Accordingly, we didn't include the locally-sample-then-aggregate strategy into baseline, as it doesn't go well with federated settings.   \n- That being said, under the realistic assumption for FL, there often exists $\\boldsymbol{x}$ for which multiple local distributions assign non-negligible probability, which necessitates advanced sampling strategy.\n  - ULA is a principled and effective tactic, as it explicitly mixes local score functions as in Eq. (8), with inherent stochasticity for an exploration from mode to mode via the noise term in Eq. (7).\n  - However, the explicit mixture sampling might dominate certain $\\boldsymbol{x}$ with high joint probability, it often results in _low diversity_ and fails to adapt dynamically to the relative importance of each local distribution.\n- In summary, `CooDI` implicitly samples from a mixture of local distributions (with bounded dissimilarity) via ULA. This process naturally adapts the mixture weights during sampling, guided by the energy-based parameterization proposed in Definition 3.1.\n- We will also update discussion on this in the revised manuscript. We thank the reviewer for pointing out this aspect.\n\n### 3) DP diffusion model training\n- We agree that the DP training is non-trivial and directly impacts the quality of the synthetic dataset per se.\n- While subsection 4.4 (lines 232–246) demonstrates that achieving DP is theoretically straightforward, the resulting dataset quality under $(\\epsilon=10,\\delta=10^{-5})$-DP --- though expected --- seems unsatisfactory, compared to non-DP case (cf., Figure A1 on MNIST dataset).\n  - Empirically, we observed that the dynamics of collaborative ULA are affected unfavorably when local diffusion model training is constrained by DP-SGD, even if the quality of locally generated samples with DP is acceptable. \n- Thus, we plan to tone down or even move the privacy part back and further investigate the way of enhancing the compatibility of `CooDI` with DP guarantee. We think it is practically important direction and we are eager to explore this as part of our future work.\n  - Also, we hope this will give us more space to revise the manuscript so that it is clearer and more accessible to general audiences in ML fields. \n- We thank the reviewer for giving us the opportunity to improve presentation better for readers.\n\n### 4) Others\n- We wholeheartedly thank the reviewer for granular feedback on many other details in our draft.\n- We tuned all the hyperparameters of the baseline methods according to the reported numbers in the papers or in the official implementations.\n- According to the reviewer's comment, we will emphasize in the revised manuscript that each client should have enough computational resources to train a diffusion model.\n  - We originally thought this was reasonable without note, since we were targeting a cross-silo FL setting, where each client is typically organization or datacenter, with sufficient resources and data samples. (cf., Table 1 of [4]) \n- We will amend the statement on the compliance with HIPAA.\n  - The original claim of HIPAA compliance in the broader impact statement was to highlight `CooDI`’s potential to support privacy-preserving data sharing in domains governed by privacy regulations (e.g., GDPR and HIPAA), rather than asserting formal legal compliance. Especially in cross-silo FL setting, the data sharing is prohibited due to constraints by confidentiality or legal constraints [4], as `CooDi` targeted for cross-silo FL setting, we considered this perspective.\n\n----------\nWe would like to sincerely, deeply thank **reviewer oT3q** for all the valuable comments in this review.  \nPlease kindly let us know if any further clarification is needed.\n\n\n[1] Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC (Du et al., 2023)  \n[2] Compositional Visual Generation with Composable Diffusion Models (Liu et al., 2022)   \n[3] How Diffusion Models Learn to Factorize and Compose (Liang et al., 2024)  \n[4] Advances and Open Problems in Federated Learning (Kairouz et al., 2019)  \n[5] Synthetic data shuffling accelerates the convergence of federated learning under data heterogeneity (Li et al., 2023)  \n[6] A Field Guide to Federated Optimization (Wang et al., 2021)  \n[7] On the Convergence of FedAvg on Non-IID Data (Li et al., 2019)  \n[8] Federated Optimization in Heterogeneous Networks (Li et al., 2018)  \n[9] Dimension-free log-sobolev inequalities for mixture distributions (Chen, Chewi and Niles-Weed, 2021)"}, {"review_id": "aE6ZB05t9E", "reviewer": "Reviewer_u5Vy", "summary": "This paper presents CooDI, a diffusion model-based federated learning method. The method requires all clients to first train local diffusion models independently. Subsequently, the server initializes multiple noise vectors with corresponding prompts. During the denoising process, the server distributes current-step samples to all clients and collects their denoising results for these samples. The server then aggregates outputs from all clients and completes the current-step denoising based on the aggregated results. After T iterations, the server obtains a complete synthetic dataset. Experiments on MNIST, CIFAR-10, and CelebA datasets validate the proposed method's performance.", "questions": "See weaknesses.", "limitations": "The primary limitations of the proposed method lie in its computational and communication overhead.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The introduction of Energy-based Diffusion Models (EBMs) perspective into federated learning demonstrates notable innovation.\n2. The paper provides formal proofs for both privacy preservation and convergence in diffusion model-based federated learning, establishing theoretical soundness.\n\nWeaknesses:\n1. A fundamental contradiction exists in the methodology: While Line 30 explicitly states that diffusion models are generally unsuitable for federated learning due to their massive parameter size (which increases computational and communication overhead), the proposed method not only employs diffusion models but further exacerbates these issues by requiring client-side fine-tuning and multiple iterations. This represents a significant methodological paradox that undermines the approach's practicality.\n2. The experimental comparison is limited to only MD-GAN and PRISM methods, failing to include other state-of-the-art diffusion model-based federated learning approaches. This limited baseline comparison substantially weakens the experimental validity.\n3. Several critical experimental details remain unspecified: The exact architecture of client-side diffusion models is unclear. The claim of using \"DDPM\" is problematic as it refers to a sampling scheduler rather than a specific model architecture. \n4. The evaluation solely focuses on synthetic dataset quality, which deviates from standard federated learning benchmarks that typically assess downstream tasks (e.g., image classification). Moreover, the inherent randomness in diffusion generation processes suggests potential high variance in experimental results, yet no variance analysis is provided.", "was_revised": true, "final_justification": "Although I still believe that the proposed method has certain practical limitations, the paper demonstrates sufficient novelty and theoretical contribution. Therefore, I have decided to raise my score.", "rebuttal": "Dear  **reviewer u5Vy**,  \nWe appreciate your thoughtful reviews. We have addressed each of your concerns in detail below.  \n\n----------\n### 1) Practicality of  `CooDI`  \nWe understand the concern, though we believe there may be a misinterpretation. Below, we clarify the relevant aspects:  \n- i) `CooDI` does **not require the exchange of diffusion model parameters.** \n  - Training a diffusion model through repetitive parameter exchange, as is done in traditional FL, is indeed unsuitable due to the computational and communication overhead. (line 30)\n  - Rather, `CooDI` requires **exchange of synthetic data, not parameters of diffusion models.** (line 6 of Algorithm 1; line 196; lines 179-180; Figure 1-B & C) \n  - That being said, the end goal of `CooDI` is not to obtain a diffusion model. Instead, `CooDI` aims to obtain a global synthetic dataset. \n  - The synthetic data is **far lighter in size than the parameters, as provided in Table 3**, and emphasized in the **Efficiency** subsection. (line 271)\n  - That's why we framed `CooDI` as a _data-centric_ approach in Abstract (line 1) and Introduction (line 24), since `CooDI` can detour repetitive and burdensome parameter exchanges. (line 81; lines 179-180)\n- ii) `CooDI` does **not require client-side fine-tuning for multiple iterations.**\n  - We designed the local training to be required _only once_, right before the communication begins. (lines 184-189; line 3 of Algorithm 1)\n  - `CooDI` **only requires local inference** w.r.t. the collaborative synthetic dataset; no local fine-tuning is required. (lines 195-200; lines 6-7 in Algorithm 1; Figure 1-C)\n- Thus, `CooDI` **does not exacerbate communication/computation issues**, but **rather improves efficiency** compared to parameter exchange-based FL methods for a diffusion model because it **does not necessitate repetitive local fine-tuning nor multiple parameter exchanges**.\n\n### 2) Comparison with SotA federated diffusion approach\nWe thank the reviewer for pointing out concern on experimental baselines.  \n- Originally, we regard `CooDI` as a _federated synthetic data generation_ framework, not a federated diffusion modeling method. (lines 34-38; lines 45-50)\n  - That's why we focused on previous works on **synthetic data generation methods under federated settings**, including `MD-GAN` and `PRISM`.\n- However, we respectfully agree with the reviewer's comment and acknowledge that the federated diffusion model can also be considered an additional baseline, since it can also generate synthetic data at the end.\n  - Thus, we provide additional comparison results on MNIST dataset with state-of-the-art federated diffusion modeling method, i.e., `FedDiffuse` [1], with shards partitioning setting for non-IID and $K=10$ clients in the table below. \n|  | FID | LogReg (Acc.) | MLP (Acc.) | CNN (Acc.) |\n|:--|:--:|:--:|:--:|:--:|\n| `FedDiffuse` | 58.4 | 81.2 | 81.1 | 82.9 |\n| `CooDI` | **4.8730** | **92.8** | **91.2** | **93.9** |    \n  -  Note that we exploited `full` strategy with best reported hyperparameter in [1]. `CooDI` consistently outperforms in both synthetic data quality and utility, even under practical non-IID setting.\n\n### 3) Experimental Details\n- We mentioned (line 259) and readily provided the exact architecture of client-side diffusion models (**Table A1** in the technical appendix), with the source code attached in the supplemental material. \n- To clarify again, we adopted the U-Net architecture [2,3], which is typically used in diffusion modeling and is described in the appendix, using `diffusers` package.\n- Due to the space limit, we also separately spared _Experimental Details_ section in the technical appendix that contains several other experimental details.\n\n### 4) Evaluation of Synthetic Dataset Utility\n- We've already provided **image classification results on a synthetic dataset in Table 2** as one of our main results, in order to evaluate the utility of synthetic dataset for a downstream task.\n- We appreciate the reviewer’s concern on variance due to the stochastic nature of diffusion models. While we couldn't report variance across multiple random seeds due to resource constraints during the submission, we note that the selected evaluation metrics (FID, Precision & Recall, Density & Coverage) are designed to capture quality and distributional differences over multiple generated images, and are commonly used to benchmark generative model robustness.\n- In our experiments, we observed consistent performance and reasonable stability. Still, we acknowledge that a full variance analysis (e.g., over multiple seeds) would strengthen the evaluation, and we plan to include such results in camera ready version if accepted.\n\n----------\nAgain, we are grateful for the time and effort that **reviewer u5Vy** dedicated to this review.\n\n[1] Training diffusion models with federated learning (de Goede, Cox and Decouchant, 2024)  \n[2] U-Net: Convolutional networks for biomedical image segmentation (Ronneberger, Fischer and Brox, 2015)\n[3] Denoising Diffusion Probabilistic Models (Ho, Jain and Abbeel, 2020)"}, {"review_id": "kCqmfPpu1Y", "reviewer": "Reviewer_1TqA", "summary": "This paper proposes COODI, a novel framework for federated synthetic data generation based on cooperative inference of diffusion models. Rather than aggregating model parameters during training, COODI reframes federated generation as a sampling problem over a mixture of locally trained diffusion models. Each client independently trains a conditional diffusion model on its private data, and the server later coordinates the sampling process by requesting noise predictions from clients and aggregating their outputs into a global mixture score. The sampling follows an Unadjusted Langevin Algorithm (ULA) guided by these aggregated scores. \n\nIn addition, the authors provide a theoretical analysis connecting diffusion models to energy-based models (EBMs), derive convergence guarantees under statistical heterogeneity, and demonstrate strong empirical results on multiple image datasets. COODI achieves high data fidelity, utility, and privacy preservation, offering a communication-efficient solution suitable for cross-silo federated settings.", "questions": "Please refer to Weakness for details.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strenths:\n\n1. The paper introduces a new perspective on federated synthetic data generation by reframing the problem from a model-centric training paradigm to a data-centric cooperative inference strategy. Rather than performing expensive and communication-intensive federated training of generative models, COODI proposes a collaborative sampling framework wherein clients locally train their own diffusion models and participate only in the inference stage. The central server aggregates their outputs to perform mixture-based Langevin sampling. This strategy enables a more modular and communication-efficient system design, particularly well-suited for cross-silo settings where each participant retains full control of its local model and data.\n\n2. I believe that a key strength of the paper lies in its rigorous theoretical foundation. The authors establish a clear connection between diffusion models and energy-based models (EBMs), leveraging this relationship to implement Unadjusted Langevin Algorithm (ULA) sampling without requiring access to model parameters or normalized probability densities. This interpretation allows the system to operate within a well-understood probabilistic framework. Furthermore, the authors derive non-asymptotic convergence guarantees under statistical heterogeneity, which is particularly important in the federated setting where client distributions may differ significantly. These results lend both credibility and depth to the proposed method.\n\n3.  The experimental section is extensive and well-executed. The authors evaluate COODI across a wide range of datasets, including CIFAR-10, CelebA, MNIST, and etc. The results show clear improvements in generation quality and in the utility of synthetic data when used for downstream classification tasks. \n\nWeaknesses\n1. The current framework appears to require all clients to synchronously participate in each sampling step by providing noise predictions for the server to compute the global mixture score. This design might limit applicability in more dynamic or large-scale cross-device federated learning scenarios, where clients are often intermittently available. While this constraint seems acceptable in cross-silo settings, could the authors clarify whether asynchronous or partially-participating variants of COODI are feasible, or whether such extensions have been explored?\n\t\n2. The paper emphasizes the generation of synthetic data through collaborative sampling, but it is somewhat unclear what the final deliverable is. Does COODI aim to produce a reusable global generative model (e.g., via score distillation), or are the synthetic samples themselves intended to be directly used for downstream tasks? An explicit clarification of the intended output and deployment scenario would help contextualize the practical usage of the method.", "was_revised": true, "final_justification": "After reading the authors’ rebuttal and  clarifications, I think most of my main concerns have been addressed—especially around how CooDI might be improved and what the final outputs are meant to be. The authors were transparent about the current limitations and shared some good ideas for future extensions. That said, I feel that the need for full client synchronization is still a real drawback and limits the method’s practical use right now. Overall, I see this as a valuable step forward for federated dataset distillation in cross-silo setups.", "rebuttal": "Dear **reviewer 1TqA**,  \nWe deeply appreciate the reviewer's accurate summary of our work as well as the recognition of the significance and strengths of `CooDI`.  \nIn the following sections, we address the points identified as weaknesses.\n\n---\n### 1) On the feasibility of improved `CooDI` \nWe appreciate the reviewer's feedback and agree that the suggested improvements are valid.  \n- As stated in the contribution (lines 55-56) and limitation (line 280) sections, `CooDI` is designed for a cross-silo FL setting, due to its synchronization requirement from all clients.\n  - However, we believe that `CooDI` can be improved in terms of the scalability, as we have conceptualized in lines 280-282. \n- To the best of our knowledge, no such extensions (e.g., asynchronous variant or ULA with partially observed _scores_) are currently available. Instead, a paper proposed ULA with partially observed _parameters_ under FL setting [1].\n  - In line with this, specifically for the partial participation setting, `CooDI` can be modified to construct an unbiased estimator of the full mixture score based on the observed local scores from partially available clients at each round, similarly to [1]. \n- We are open and eager to extend our proposed framework in this direction as future work.\n\n### 2) Final deliverables from `CooDI`\nWe thank the reviewer for highlighting the ambiguous part of our manuscript. We will revise our manuscript accordingly to ensure clear delivery.  \n- Originally, we intended that the _synthetic dataset themselves_ are worth to obtain, as it can be explicitly used to train a server-side global model for downstream task. \n  - That's why we designed an experiment to evaluate the _utility_ perspective of the synthetic dataset. (lines 258-259)\n- We also believe that a synthetic dataset on a credible central server would benefit the following:  \n  - i) incoming clients in the FL system;    \n  - ii) clients with data deficiency or imbalance;  \n  - iii) the server to manage data quality or protocols (e.g., data standardization) by using the synthetic dataset as a proxy for the collection of local datasets.  \n- Alternatively, similar to the score distillation approach mentioned by the reviewer, we are also open to improving `CooDI` to produce a reusable, server-side generative model, based on securing amortized sampler at the server (lines 285-287; e.g., [2]). \n---\nAgain, we sincerely thank the **reviewer 1TqA** for insightful comments.\n\n[1] On Convergence of Federated Averaging Langevin Dynamics (Deng et al., 2021)  \n[2] Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood (Zhu et al., 2023)"}]

Author rebuttal:

Meta-review: All reviewers acknowledged that this paper innovatively formulate the task of federated synthetic data generation as sampling from a mixture of decentralized distributions at client sides, and I also found this problem-reframing innovation really exciting. In this way, the underlying problem setting becomes identical to “compositional generation with energy-based diffusion models” [Du et al, 2023] (i.e. [13] in the submission) and the sampling procedure can be directly implemented using the unadjusted Langevin algorithm proposed in [Du et al, 2023].

All reviewers are positive to recommend this paper. But from the wording of Reviewer 1TqA, there might exist some misunderstandings, for example:

> The authors establish a clear connection between diffusion models and energy-based models (EBMs), leveraging this relationship to implement Unadjusted Langevin Algorithm (ULA) sampling without requiring access to model parameters or normalized probability densities. This interpretation allows the system to operate within a well-understood probabilistic framework.

In my understanding, the abovementioned contributions seem rephrased from [Du et al, 2023]. If so, the authors are requested to clearly state that the connection between diffusion models and energy-based models is rephrased from the reference and clearly cite the literature in the Preliminary subsections to avoid misunderstanding.

Another common concern is the practicability of this approach. Although the reformulation of the problem from a new perspective provides a new approach for federated synthetic data generation, this approach requires each client to train a“well trained” local diffusion model and involve frequent client-server communications. It would be more convincing if the authors can find a practical scenario that fits this problem setting.
Decision: Accept (poster)