One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
Authors: Mingzhao Yang
Shangchao Su
Bin Li
Xiangyang Xue
Venue: ICML 2025 poster (2025)
Keywords: Federated Learning
Diffusion Model
Primary area: deep_learning->everything_else

Abstract: In recent years, One-shot Federated Learning (OSFL) methods based on Diffusion Models (DMs) have garnered increasing attention due to their remarkable performance. However, most of these methods require the deployment of foundation models on client devices, which significantly raises the computational requirements and reduces their adaptability to heterogeneous client models. In this paper, we propose FedLMG, a heterogeneous one-shot Federated learning method with Local Model-Guided diffusion models. In our method, clients do not need access to any foundation models but only train and upload their local models, which is consistent with traditional FL methods. On the clients, we employ classification loss and batch normalization loss to capture the broad category features and detailed contextual features of the client distributions. On the server, based on the uploaded client models, we utilize backpropagation to guide the server’s DM in generating synthetic datasets that comply with the client distributions, which are then used to train the aggregated model. By using the local models as a medium to transfer client knowledge, our method significantly reduces the computational requirements on client devices and effectively adapts to scenarios with heterogeneous clients. Extensive quantitation and visualization experiments on three large-scale real-world datasets, along with theoretical analysis, demonstrate that the synthetic datasets generated by FedLMG exhibit comparable quality and diversity to the client datasets, which leads to an aggregated model that outperforms all compared methods and even the performance ceiling, further elucidating the significant potential of utilizing DMs in FL.

One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
Mingzhao Yang'
Shangchao Su
Bin Li
Xiangyang Xue
Abstract
1. Introduction
In recent years, One-shot Federated Learning (OSFL) methods based on Diffusion Models (DMs) have garnered increasing attention due to their remarkable performance. However, most of these methods require the deployment of foun- dation models client devices, which signifi- cantly raises the computational requirements and reduces their adaptability to heterogeneous client models. In this paper, we propose FedLMG, a het- erogeneous one-shot Federated learning method with Local Model-Guided diffusion models. In our method, clients do not need access any foun- dation models but only train and upload their local models, which is consistent with traditional FI. methods. On the clients, we employ classification loss and batch normalization loss to capture the broad category features and detailed contextual features of the client distributions. On the server, based on the uploaded client models, we utilize hackpropagation to guide the server's DM in gen- erating synthetic datasets that comply with the client distributions, which are then used to train the aggregated model. By using the local mod- cls as medium to transfer client knowledge, our method significantly reduces the computational requirements on client devices and effectively adapts to scenarios with heterogeneous clients. Extensive quantitation and visualization exper- iments on three large-scale real-world datasets, along with theoretical analysis, demonstrate that the synthetic datasets generated by FedLMG ex- hibit comparable quality and diversity to the client datasets, which leads an aggregated model that outperforms all compared methods and even the performance ceiling, further elucidating the sig- nificant potential of utilizing DMs in FI..
Federated learning (FL) (Mammen, 2021) has gained in- creasing attention recently. Standard FL relies on frequent communication between the server and clients. With the growing adoption of AI models by individual users, the application scenarios of FL have expanded significantly, in- cluding mobile photo album categorization and autonomous driving (Nguyen et al., 2022; Fantauzzo et al., 2022). How- ever, in these scenarios, the substantial communication cost associated with FL is often impractical for individual users. As result, one-shot FL (OSFL) has emerged tion (Li et al., 2021; Zhou et al., 2020). OSFL aims to establish the aggregated model within a single communi- cation round. Currently, mainstream OSFL methods can be categorized into four types: 1) Methods using the auxil- iary dataset (Guha et al., 2019; Li et al., 2020a; Lin et al., 2020). Methods training generators (Zhang al., 2022: Heinbaugh et al., 2022). 3) Methods transferring auxiliary information (Zhou et al., 2020; Su et al., 2023). 4) Methods based on DMs (Yang et al., 2024a;b; Zhang et al., 2023a).
solu-
However, existing methods are hard to apply in real-world scenarios due to the following reasons: 1) Collecting public datasets that comply with all client distribution is imprac- tical, owing to privacy concerns and data diversity issues. 2) Due to the limited computational power and data of the clients, training generators on realistic client images is chal- lenging. So most OSFL methods can only be applied to small-scale toy datasets, such as MNIST and CIFARIO. 3) The transmission of auxiliary information incurs communi- cation cost, and extracting the auxiliary information on the clients also incurs additional computation cost, further re- stricting the practicality of OSFL.. Current OSFL methods that leverage DMs necessitate the deployment of founda- tion models the clients (Yang al., 2024a; Zhang et al., 2023a). such as CLIP (Radford et al., 2021) and BLIP (Li et al., 2023), or directly involve client training of diffusion models (Yang et al., 2024b), leading to significant commu- nication and computational costs. comparison to widely used traditional FL methods, which only require local train- ing of client models, these additional burdens imposed by OSFL methods significantly increase the strain client resources, thereby limiting their practical applicability.
Shhnnhaai Key Laboratory of Intelligent Information Process- ing, School of Computer Science, Fudan University. Correspon- dence to: Bin Li <libin @ edu.en..
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
To reduce the computational requirements on client devices and alleviate the burden clients, we propose MGG, a heterogeneous one-shot Federated learning method with Local Model-Guided diffusion models. Our method in- volves using the locally trained client models to guide the DMs in generating the synthetic dataset that complies with different client distributions. Specifically, FedLMG consists of three steps: Local Training, Image Generation, and Model Aggregation. Firstly, based on the theoretical analyses of the client's local distribution and the server's conditional distribution, the clients independently train the client models on their private data and upload them to the server. Subse- quently, assisted by the received client models, the server generates realistic images that comply with different client distributions based on DMs. After obtaining the synthetic dataset, we introduce three strategies to obtain the aggre- gated model: fine-tuning, multi-teacher distillation, and specific-teacher distillation. Through these strategies, we achieve the model aggregation in a single round of commu- nication, without accessing any client data or any additional information transferring compared with traditional FL.
We conduct thorough theoretical analyses, demonstrat- ing that under the assistance of client models, the KI. divergence between the data distribution of the DM on the server and the local data distribution is bounded.
We conduct sufficient quantitation and visualization experiments to demonstrate that the proposed method outperforms other compared methods and can even sur- pass the performance ceiling of centralized training in some cases, further evidencing the enormous potential of utilizing DM OSFL.
2. Related Work
2.1. One-shot Federated Learning
In the standard FL (McMahan et al., 2017), there are multi- ple rounds of communication between the server and clients. To reduce the high communication cost, OSFL entails clients training their local models to convergence first, fol- lowed by aggregation on the server. Existing OSFL methods can be broadly categorized into three main types. 1) Meth- ods based public auxiliary dataset. (Guha et al., 2019) utilizes unlabeled public data the server for model distil- lation. Similarly, FedKT (Li et al., 2020a) and FedDF (Lin et al., 2020) employ an auxiliary dataset for knowledge transfer on the server. 2) Methods based on generators. DENSE (Zhang et al., 2022) employs an ensemble of client models as a discriminator to train generator for generat- ing pseudo samples, which is used to train the aggregated model. To address very high statistical heterogeneity, FedC- VAE (Heinbaugh et al., 2022) trains a conditional variational autoencoder (CVAH) on the client side and sends the de- coders to the server to generate data. 3) Methods based on sharing auxiliary information. DOSFL (Zhou et al., 2020) performs data distillation on the client, and the distilled pseudo samples are uploaded to the server for global model training. MAEcho (Su et al., 2023) shares the orthogonal projection matrices of client features to the server to opti- mize global model parameters. 4) Methods based on DMs. We provide a detailed introduction to such methods in the next section. Although a large number of OSFL methods have been proposed, their practicality significantly limited due the reasons mentioned in the Introduction.
To validate the performance of our method, we conduct extensive quantitation and visualization experiments on three large-scale real image datasets: DomainNet (Peng et al., 2019), OpenImage (Kuznetsova et al., 2020) and NICO+ (Zhang et al., 2023c). Sufficient quantitation ex- periments under various client scenarios demonstrate that our method outperforms all compared methods in a single communication round, and in some cases even surpasses the ceiling performance of centralized training, strongly un- derscoring the potential of DMs and providing convincing evidence for our aforementioned ideas. Visualization exper- iments also illustrate that our method generates synthetic datasets that comply with both the specific categories and the personalized client distributions, with comparable qual- ity and diversity to the original client dataset. Moreover, we conduct thorough discussions on communication cost, computational cost, and privacy concern,, further enhancing the practicality of the proposed method.
In summary, this paper makes the following contributions:
We propose FedLMG, a novel OSFL method, to achieve real-world OSFL without utilizing any foun- dation models on the clients, ensuring no additional communicational or computational burden compared to traditional FL methods, thereby significantly expand- ing the practicality of OSFL.
2.2. FL with Diffusion Models
The DM introduced by (Sohl-Dickstein et al., 2015). (Ho et al., 2020) proposes the fundamental framework of the DM. Following this, a series of sampling techniques emerge (Song et al., 2020a; I Liu et al., 2022; Song et al., 2020b), lead- ing to the success of DMs in generation (Kingma et al., 2021; Wang et al., 2018). Subsequently, Stable Diffusion (Rom- bach et al., 2022) provides a series of powerful DM capable of generating images complying with the most common
We propose using the locally trained client models as medium to transfer client knowledge to the server, guiding the diffusion model through classification loss and BN loss to capture the client's 's category and contex- tual features, thereby generating high-quality synthetic datasets that comply with client distributions.
2

---

One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
Clients
Server
Fine-tuming
Aggregated
Client 1
Model
Client Models
Multi-tcacher Distillation
Train
Assist I
Client 2
Stable Diffusion
Aggregated
Model
Train
Upload
Diffusion Denoising
Client K
Client Models
Generate
Specific-teacher Distillation
Train
W
Aggregated Model
Client Model
Local Data
Synthetic Dataset
Client Models
Step I: Local Training
Step 2: Image Generation
Step Model Aggregation
Figure 1. Overview of FedLMG. Our method consists of three steps: Local Training, Image Generation, and Model Aggregation. Firstly, cach client independently trains their models using its private data and uploads them to the server. Assisted by these client models, our method leverages the powerful DM to obtain the synthetic dataset that complies with different client distributions. Based on the synthetic dataset, three strategies are provided to obtain the aggregated model.
real-world distributions. One major advantage of DMs lies their powerful conditional generation capability. By using images (Saharia al., 2022a; Wang et al., 2022; Zhang et al., 2023b), text (Nichol et al., 2022; Saharia et al., 2022b; Kim et al., 2022; Preechakul et al., 2022), other models (Dhari- wal & Nichol, 2021), initial noise (Mao et al., 2023) as condition, DMs can accordingly generate high-quality im- ages, inspiring the development of DM-based FL methods.
the practicality of our method and enabling our method to address the scenario of heterogeneous client models.
3. Method
In this section, we introduce the proposed method in detail. Firstly, we provide essential preliminaries regarding the diffusion process and our problem setting. Then, as stated in Figure I, we detail the three steps of the proposed method: Local Training, Image Generation, and Model Aggregation.
Currently, only a small number of studies have focused on the significant potential of DMs in FL. Most DM-based based the idea
FL methods (Yang al., 2024a;b) of uploading distribution information of the client data to the server, which can be traced back to (Li et al., 2007). In FedDISC (Yang et al., 2024a), Stable Diffusion is intro- duced into sumissuper FL for the first time, achiev- ing remarkable results within just single communication round. However, it requires using the CLIP image branch as the backbone for classification, which limits its flexibility. FGL (Zhang al., 2023a) employ BLIPv2 (Li et al., 2023) on the clients to extract text prompts of client images, which are then sent to the server for image generation using DMs. FedDEO (Yang et al., 2024b) trains a description of the local distribution using DMs on the clients. Phoenix (Stan- ley Jothiraj Mashhadi, 2024) introduced FL into DMs, proposing distributed method for training DMs. All the aforementioned methods involve substantial computation and communication costs on the clients. Unlike these meth- ods, we do not require deploying any foundational models on the clients, which significantly reduces communication and computational costs on the clients, further enhancing
3.1. Preliminaries
Problem Setting and Notations. Consider that we have clients. Taking client l as an example, this client has a private dataset 2 {xx,, yı) y:)),,, and client model Fe. The server needs to aggregate the client models {Foo ) to obtain an aggregated model Fe, that adapts various client distributions. The overall objective of our method is:
K Σ EXDD, [ea[Fo,; x)]
1
min weRd
(1)
K
where Eg the local objective function for the k-th client, Fe, is the parameters of the aggregated model. From this objective function, it is evident that our goal is to train aggregated model that adapts all client distributions and exhibits excellent classification performance data from each client. We evaluate the performance in the subsequent experimental section according to this objective.
the
Diffusion Process. The DM cg samples initial noise x
3

---

One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
from standard Gaussian Distribution (0, and itera- tively denoises it, resulting in a realistic image Xo, where T denotes the maximum timestep. (Dhariwal & Nichol, 2021) proposes the classifier- classifierguuddane,, wherein the gradi- ents backpropagated through the classifier Fo, are used to modify the predicted noise of the DM eg Ex(x,,) t) at each timestep. The loss function utilized for generating gradi- ents typically involves the cross-entropy loss Lcg between the given class label y and the output of classifier Fo, (x1). Under the guidance of the classifier Fe., the sampling pro- cess at each timestep has two steps: Firstly, for any given timestep t E {O.... { the predicted noise is modified by the gradients of Fe, according to the following equation:
model lies in its ability to generate data that comply with al- most any data distribution with proper guidance. Therefore, regarding the data distribution pa px(x) of the client's 's local dataset De and the data distribution Peo (x) that the DMs eg can generate, we can make the following assumption:
Assumption 1 There exists > 0 such that the Kullback- Leibler divergence from py p&(x) to Peo (x) is bounded above by A:
KL (x) (x))) < A
(4)
In this assumption, we assume that there is some overlap between (x(x) and Peg (x), which is considered reasonable. We don't rigidly demand that Peg (x) fully encompasses pk (xxx). Even if the clients specialize in certain professional domains such as medical images, given the widespread ap- plication of DMs across various domains (Kazerouni et al., 2023; Wang et al., 2024), the assumption can be satisfied by replacing the DM being used. Based on Assumption 1, we have the following theorem regarding the relationship be- tween the conditional distribution Pea (x (0x) and the client local data distribution pa (x):
E(x,, (1y) : co (Xe, f1y) - v1 - at Vx, po, (u)x))
(2)
where Vx, log po, (u/x²) is the gradient of the classifier For with respect to the input (xe, v). Afterwards, utilizing the modified é (x.,4y). the sample for the next time step
x is obtained:
- = -vaa-1 Xx VI -a--V-aareee -a,a((xx tly)
Vat
Theorem 1 There exists X > 0, for the local data distri- bution px(x) and the conditional distribution Pe, (x(0x) of the DM 6g conditioned on the client model Fo, trained on client k, we have:
- 01-1 11-a-11 - of E(xe: (ly) + o18t (3)
where at, 0+-1 and or are pre-defined parameters, e is the Gaussian noise randomly sampled at each timestep. Ac- cording to the diffusion process described above, for the randomly sampled initial noise Xx ~ [00 Z), after T iter- ations, the diffusion model can denoise the initial noise into high-quality realistic images Xo under the guidance of the classifier Fe..
(x(p(()))l)x((((1=))) < A + E(log (0x))
-f pa (x) log Pro (0xlx)ds (5)
For a detailed proof, please refer to the appendix. From Eq. 5, we can observe that the KL divergence between the conditional distribution Pe, (x(0x), which is also the distribution of the synthetic data, and the distribution of client's local data (xx(x) is bounded above. This upper bound consists of three components: A, [E(log Peo (0n)), and - J Pa (x) lt. Pt. (0x|x)dx. À is the same as in Eq. 4. B(log Pr,, (0x)) is a constant independent of the sample x. Pc.. (0& [x)dx is the negative log- likeeiioldoodd between the client model Fe, and the client distribution p& (x). Minimizing this negative log- likeliood is equiva- lent to minimizing the cross-entropy loss LCV. This implies that during the local training, we need to train client models using the cross-entropy loss to minimize this upper bound of the KL divergence. Consequently, the conditional distribu- tion of the synthetic dataset Pe,, (x(0x) can closely approx- imate the client's local data distribution ps (x). Therefore, for client k. we utilize its privacy dataset D and train the client model Fe, using the following cross-entropy loss function:
3.2. Local Training
The first step of our method is local training. To generate data on the server that complies with the client distribution, we need to transmit client information to the server. In fed- crated learning, it's common to send locally trained client models to the server. Therefore, for the local data distribu- tion p& (x) of the private dataset Dk for client k; we leverage the information from the locally trained client model Fo, on the server and obtain the conditional distribution Peo (x 0x) based on the data distribution of the diffusion model Peo (x). We aim for the synthetic dataset sampled from the condi- tional distribution Pe., [x (0)) to closely comply with the client local data distribution (x). Therefore, considering the relationship between (x) and Peo (x(0x) is essential. We conduct comprehensive theoretical analyses regarding the relationship between these two distributions.
J p& (x) Po.
Firstly, we need to analyze the relationship between the unconditional data distribution of diffusion model Pe, (x) and the client local data distribution py (x). As stated in the Introduction, our motivation for utilizing the diffusion
e (Fo, = (x.))
(6)

---

One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
After multiple rounds of training, the client models are sent to the server to guide the generation process on the server. It's worth noting that in FedLMG, there are no requirements for the used model structures, which further enhances the practicality of our method.
Subsequently, based X we compute the loss function and gradient to modify c0(x,, Although in the ini- tial time steps, Xo, may appear blurry, the noise level in comparison to x, is noticeably reduced. This decreases the demand for the client models' robustness of noise, mitigat- ing the need for clients to specifically train for classifying noised samples and further enhancing the practicality of our method.
3.3. Image Generation
After receiving the locally trained client models Fox, & 1,..., K uploaded by the clients, these client models serve as cues for the DM, generating the synthetic dataset com- plies with different client distributions. Firstly, we elaborate on how the client models assist DM in generation. our problem setting, generated images must possess accurate categories and comply with specified client distributions, introducing novel demands to sampling, and necessitating consideration of additional image attributes such as style, color, background, etc. Relying solely on classification re- sults falls significantly short of achieving these demands since the classification results mainly provide information on categories.
Finally, the overall loss function L employed in the condi- tional generation is as follows:
C [(x,, y, 0x) = LCE (Ro.)) v) + WBN CEB 0%)
(9)
where WRN is the weight of BN Loss. After performing gradient backpropagation according to this loss function, we use Eqs. 2 and 3 to guide the generation process through the client model Fe. and its accompanying BN statistics, enabling conditional generation.
For any given time step & E (o,..., T). we modify the predicted noise of the DM based on Eq. 9:
To provide detailed context information about the client dis- tributions, we utilize the statistics of each batch normaliza- tion (BN) layer of client models Fe.: mean μ and variance σ. In other words, we need to consider the conditional gen- eration process p(x;-1|x; y: (1 { where Fy. and i.e respectively denote the means and the vari- ances of all BN layers within Fe., and La represents the number of BN layers within Fe.. Therefore, during modi- fying the predicted noise of the DM co(x,, tly) at each time step f, we compute gradients by summing the cross-entropy loss LCE and the BN Loss LEN to incorporate the addi- tional distribution details embedded within the statistics of the BN layers into the diffusion process. The computation of LAN as follows:
? (xx, fly) == co (x,, fly) - v1 ã Xx, y, 0z) (10)
Subsequently, based on Eq. 3, we compute x using the modified é tly), leading the realistic image X, after T iterations. During the generation, since we specify the category y and the classifier Fe., the generated image Xy is automatically labeled. We define Xy as X and include along with its label ve in the synthetic dataset X. After undergoing multiple iterations of generation, we obtain the synthetic dataset X- = (()),,)))
L2
3.4. Model Aggregation
Based on the synthetic dataset X, we proceed to obtain the aggregated model. We employ the idea of distillation to achieve model aggregation and introduce three strategies to obtain the aggregated model: Fine-tuning, Multi-teacher Distillation, and Specific-teacher Distillation.
LBN
(x) -Ma,l + ||oo((x Ok) - (,,l)) 1=1
(7)
Fine-tuning. As all samples xt, 1,..,,N,k
1,..., K in the synthetic dataset X have their labels y². this strategy refers directly training aggregated model using the cross-entropy loss LCF. The specific loss function used during the aggregation process is as follows:
where (x, 0x) and (x, 0x) denote the mean and vari- ance of the output feature from the 1-th BN layer after feed- ing the sample x into the client model Fo..
Furthermore, since local models Fe, are simply trained on the client and are not accustomed to the noised input x,, traditional classifier-giidanc struggles to provide accurate guidance through the computed gradient Vx, logPe, Po. (y) xx)) To address this challenge, at any time step i, we utilize the predicted noise E Ex(,, tly) to predict xo according to the following equation:
Lago (x) v²t) =CCE (x)) (
(11)
A Although this strategy is relatively simple, since the perfor- mance ceiling of centralized training also involves training the aggregated model on the client data, want empha- size that this strategy is closest to the centralized training, directly reflecting the quality and diversity difference be- tween the synthetic dataset and the original client dataset.
X1
VI cre(xe, tly)
Multi-teacher Distillation. The second strategy is multi- teacher distillation. The synthetic dataset X serves as a
(8)
5

---

One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
Table 1. Performance of different methods OpenImage, DomainNet, Unique NICO++ and Common NICO++ under feature distribution skew, where italicized text represents the ceiling performance used solely as reference, and bold text signifies the optimal performance excluding the ceiling performance.
Opealmage client3
DomainNet quickdraw
clientD
client1
cliem12
client4
client5
Avg
clipurt
infograph
puinting
real
sketch
Avg
Cedling
49.88
50.56
57.89
59.96
66.53
51.38
56.03
47.48
19.64
45,24
12.31
59.79
42.35
36.89
FedAvg
41.46
50.36
52.61
62.10
50.17
51.18
37,96
12.55
34.41
5.93
51.33
32.37
29,09
FedDF
44.96
16.15
59.69
58.69
63,45
46.63
53.26
38.09
13.68
35.48
7.32
53.83
34,69
30.52
FedProx
44.99
48.83
49.25
56.68
61.23
46.07
51.18
38.24
12.46
37.29
6,26
54.88
35.76
30.82
FedDyn
46.08
52.44
54.67
62.84
47.73
$1.78
40.12
14.77
36.59
7.73
54.85
34.81
Prompts Only
30.41
30.25
42.92
43.48
50.75
33.43
38.54
31.8
11.61
31.14
4.13
61.53
31.44
28.61
FedDISC
47.42
49.65
54.73
53.41
60.74
52.81
53.13
43.89
14.84
38.38
8.35
56.19
36.82
33.08
48.21
49.16
54.98
55.47
63.14
49.32
53.38
41.81
15.30
40.67
8.79
57.58
39.54
33.95
FedLMG FT
48,99
51.66
55.59
52,80
62.41
58.86
55.05
44.25
17.51
38.74
9.43
57.31
38.44
34.28
RdILMG.SD
47.60
55,20
61.54
61.83
67.07
59.90
58.86
46.23
18:42
42.85
10.24
58.52
39.13
35.90
Fedl.MG_MD
44.70
53.08
58.67
60.13
64.06
58.06
56.45
47.21
18.49
40.37
10.02
59.67
40.19
35.99
Unique
NICO++
Common NICO+
clientO
elientl
client2
clicnt3
client4
client5
Avg
auromn
dim
grass
outdoor
rock
water
Avg
Celling
79.16
81.51
76.04
72.97
79.16
79.29
78.07
62.66
54.07
64,89
63.04
61.08
54.63
60.06
FedAvg
67.31
74.75
73.07
67.87
69.39
52.51
40.45
57.21
51.59
49.31
43.56
FodDF
69.79
78.90
69.53
66.01
74,86
70.80
71.65
50.44
39.62
57.42
52.91
51.61
44.76
49.46
FedProx
70.46
75.3
70.87
67.67
72.84
71.51
71.44
53.49
42.41
58.84
53.08
53.67
45.42
51.15
FedDyn
71.23
74.98
69.68
68.13
73.63
70.61
71.37
$4.38
43,20
57.56
$2.63
$2.86
46.76
51.23
Prompts Only
69.79
69,14
69.32
59.89
67.70
66.60
67.07
50.51
38.10
54.53
49.39
49.12
41.58
47.21
ReIDISC
74.32
73:47
71.25
66.79
75,28
70.06
71.86
56.82
51.43
59.45
56.17
52.32
45.64
53.64
74.62
79.43
71.26
68.65
76.37
74.31
74.11
57.25
49.35
61.81
58.42
54,29
47.62
54.79
Fedl MG FT
75.13
73.30
70.31
68.88
73,60
72.51
72.29
54.63
49.21
58.13
54.75
54.64
47.03
53.07
FedLMG.9
77.34
79.94
75.01
71.87
76.69
74.92
75.96
61.49
51.47
65.28
60.03
59.57
51.14
58.16
FedL.GGMM
74.66
75.78
71.05
69.58
74.34
72.11
72.92
54.42
47.83
59.85
53.94
52.96
45.15
52.36
medium of knowledge distillation, utilizing all client clas- sifiers Fe,, as teachers to distill their knowledge into the aggregated model. The loss function is as follows:
4. Experiments
4.1. Experimental Settings
Datasets. We conduct experiments on three large-scale
Logg(E:; (o, v/²) = (F
real-world image datasets: OpenImage (Kuznetsova et al., 2020), DomainNet (Peng et al., 2019) and NICO+ (Zhang et al., 2023c). DomainNet comprises six domains: clipart, infograph, painting, quickdraw, real, and sketch. Each do- main has 345 categories. Following the partition in (Yang et al., 2024a;b), according to the hierarchy of categories provided by OpenImage, OpenImage is partitioned into 20 supercategories, with each supercategory comprising 6 fine- grained subclasses, serving as the six data domains for each category. NICO+ involves 60 categories, with each cate- gory having six common domains shared across categories and six unique domains specific to each category. These two scenarios are respectively referred to as the Unique NICO+ and Common NICO+ datasets. For instance, in Common NICO++, both the Cat and Dog categories en- compass 6 data domains: autumn, dim, grass, ouidoor, rock, and water. In Unique NICO++, the Cat class comprises 6 unique data domains:
K
For (x7)
2x
+ +WMDKL(Fo, (x
K
(12)
where Fe, and Fes, are the aggregated model and client clas- sifiers, WMD is the weight of distillation loss. This strategy maximally leverages the knowledge from all client clas- sifiers. However, in cases of substantial variations among clients or under the label distribution skew, the teachers from different clients may provide wrong guidance, impacting the performance of the aggregated model.
Specific- Distillation. The third strategy is specific- teacher distillation. Given that & with its client ID k we can use the specific teacher model Fo, to achieve model aggregation. The loss function is as follows:
Eating, In Cave, In Mud, Jumping, Maine Cat, and Walking, while the Dog class comprises 6 distinct data domains: Lying, Pug Dog, Rumning, Sticking Out Tongue, Teddy Dog, and Wearing Clothes.
Logo .tt) = LCE(FFF (x+).
+ WWSDDT (FF, (x+))
(13)
The meanings of each parameter are the same as mentioned earlier. When there are significant differences between clients or under the label distribution skew, this strategy ensures accurate guidance and the stable performance of the aggregated model.
Compared Methods. We primarily compare 3 strategies of MG: Fine-Tuning (FedLMG_FF) Multi-teacher Dis- tillation TTeLLMGMMD and Specific-teacher Distillation (FedLMG SD) against 3 kinds of methods: 1) Ceiling, The
6

---

One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
Clipart
Infograph
Painting
Quickdraw
Real
Skotch
Table 3. Impact of the numher of generated images.
cliport
infograph
pointing
quickcdraw
real
sketch
average
DomainNet Apple
3450
40.77
15.95
35.66
8.51
55.81
37.1
32.1
10350
34.28
Clipart
Infogruph
Painting
Quickkraa
Real
Sketch
17250
46.03
18.61
40.07
10.7
59.27
40.72
35.9
ods primarily demonstrate results on smaller datasets like
Domain Backpack
Hed
Chair
Couch
Desk
Shelf
Wardrohe
CIFAR-I0. Therefore, these methods utilized here.
4.2. Main Results
OpenImage Furniture
Firstly, we conduct experiments assess our method un- der feature distribution skew. From the results in Table highlight several observations: 1) Compared all used methods, FedLMMG consistently demonstrates superior per- formance across all datasets, which effectively demonstrates the performance of our method on large-scale realistic datasets. 2) Compared to Ceiling, in multiple data domains, our method exhibits superior performance, which confirms that the vast knowledge of diffusion models can effectively assist in the training of the aggregated model, resulting in the performance surpassing the traditional performance ceil- ing of centralized training. 3) Compared to Prompts Only, FedLMG shows promising performance on most clients, which emphasizes the necessity of assistance from the client models. Images generated solely based on text prompts have a distribution that too broad and cannot comply with the local image distribution of the clients. 4) Compared
Autumn
Dim
Grass
Outdoor
Rock
Water
Common NICO Bus
Lying
Pug Dog
Running
Tongue
Teddy Dog
Clothes
Unique NICO+ Dog
Figure Visualization generated samples on different datasets.
Table 2. Impact of different losses.
BN Loss
CLS Loss
clipact
infogruph
painting
quickdraw
real
sketch
overage
31.8
11.61
31.14
4.13
61.51
31:44
28.40
32.61
31:11
to other ceffuusionbbased methods, FedLMG demonstrates performance advantage on most clients. This indicates that our method can extract more precise information about the client distribution from the client models, guiding the DM to generate higher-quality synthetic datasets. 5) The comparison of three model aggregation strategies shows that FedLMG.SD achieves superior performance on most clients, further confirming the ability of our method to gener- ate data that complies with different distributions, enabling the specific teachers to provide more accurate guidance. Additionally, on DomainNet, the reason FedLMG MG_M per- forms better on more clients is that the distribution within the same data domain more complex, and there is more overlap between domains, allowing teachers from other clients contribute valuable information.
44.25
17.51
38.74
0.43
57.31
38.44
34.28
performance ceiling of traditional FL methods central- ized training, involving the uploading of all client local data for the training of the aggregated model. 2) Traditional FL methods with multiple rounds of communications: Fe- dAvg (McMahan et al., 2017), FedDF (Lin et al., 2020), FedProx (Li et al., 2020b), FedDyn (Acar et al. 2021). of them have 20 rounds of communications. Follow- ing standard experimental settings, each round involves one ImageNet
epoch of training each client. And as the additional public data for distillation in FedDF, 3) Diffusion based OSFL methods: FedDISC (Yang et al., 2024a), FGL (Zhang et al., 2023a) and Prompts Only. Although FedDISC is designed for semi-supervised FL see- narios, we remove pseudo-labeling process of FedDISC and directly utilize the labels client images. An- other point notice is the Prompts Only, where the server does not the client models uploaded from clients at all but only uses the text prompts of category names in the server image generation. It is important to note that, we also compare our method with DENSE (Zhang et al., 2022) and FedCVAE (Heinbaugh et al., 2022). However, due to the reasons highlighted the Introduction, these meth-
To further validate the generating ability of our method, we present visualization results in Figure 2, illustrating that FedLMG successfully generates images that possess accu- rate semantic information and exhibit competitive quality with the original client datasets.
4.3. Ablation Experiments
Ablation experiments about the loss functions, Table and Figure 3 demonstrate the impact of different losses. As

---

One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
Table 4. Comparison about the communication
Parameters requiring communication (M)
Ceiling
FodAvg Download 222.11
FedDISC
FGL Download
FddLMG Download
Uplond 270.95
Download
Total 270.95
Upoad 233.8
Total 455,91
Upload
Download
Tocal
Upload
Total 470.08
Upload 11.69
Total
4.23
427.62
431.8
0.345
469.73
0
11.69
Quickdraw BN Loss
Prompts Only
Painting BN Loss
Table 5. Comparison the client computation cost.
FedAvg
FedDISC
FGL
FedLMG
FLOPS (G)
72.8
334.75
227.34
3.64
-rrunss Classification Loss
Prompts Only
Fiolin Classification Loss
shown in Table 2, the inclusion of BN loss and classification loss introduces category and contextual information, lead- ing to the generation of synthetic datasets that better com- ply with client distributions, thereby training an aggregated model with improved performance. Figure illustrates that classification loss provides more precise category informa- tion, while BN loss introduces more detailed contextual information, preserving the stylistic alignment between the synthetic dataset and the client data.
Figure 3. Visualization about the impact of different losses.
Ablation experiments about the number of generated images. The number of images in the synthetic dataset directly affects the performance of the aggregated model. In most of our experiments, ensure fair comparison with the Ceiling baseline in terms of the number of training images, maintain the total number of images across all clients equal to that in the synthetic dataset. However, to further investigate the impact of the number of generated images on the performance of the aggregated model, we conduct related ablation experiments. Using each client's local model, we generated 10, 30, and 50 images per class for the categories present on the client. Therefore, for Do- mainNet with 345 categories, the total number of images per client N in the synthetic dataset is 3450, 10350, and 17250 respectively. The results are presented in Table 3. As shown, increasing the number of generated images leads to improved performance. Moreover, we observe that as the dataset size grows, the performance gain does not saturate, further demonstrating the diversity of the synthetic dataset.
4.4. Discussions and Limitations
Communication Cost. We thoroughly discuss the commu- nication cost of the proposed method. Since the communi- cation cost of FedAvg, FedDF, FedProx, and FedDyn essentially the same, their results not repeated. Prompts Only does not involve any communication between the client and the server. The number of iterations and the used model structures follow the default experimental set- tings. The comparison results of the upload and download communication costs between FedLMG and other methods are shown in Table 4. From the results, it is evident that because there is foundation model used on the clients, FedLMG does not involve any download communication cost, resulting in the lowest communication cost.
Computation Cost. The computation cost include the com- putation cost on the client and the server. Regarding the server computation cost, on one hand, as same as other diffusion-basee OSFL methods, FedLMG involves gener- ating synthetic datasets and training the aggregated model on the server, leading to similar server computation cost. Mcanwhile, FedLMG uses local models for conditional gen- eration rather than incorporating additional input conditions, which perform repeated noise prediction at each timestep. FedLMG can reduce the server computation cost required for generation. On the other hand, in FL, although generat- ing data requires more computation on the server, the server, as the center of the FL., typically has sufficient computa- tional power. Therefore, lower client computation cost are relatively more advantageous for practicality (Kairouz et al., 2021). Regarding client computation cost, we conduct thor- ough quantitative experiments in Table Since FedAvg,
Ablation experiments about the number of clients We conduct ablation experiments on the number of clients on the Common NICO+ and Unique NICO+ datasets under label distribution skew. We set the number of clients to 6, 30, 60, and 180. The experimental results are presented in be observed that, with the well-trained client
Table 6. models, the quality of the synthetic dataset and the perfor- mance of the trained aggregated model are not significantly affected by the increase in the number of clients. What's 's more, our method is relatively independent for each client, showing strong adaptability to the increase in the number of clients. These results underscore the practicality of our method in scenarios with large number of clients.
8

---

One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
Table 6. Results of the ablation experiments the numher of clients.
The Numher of
Unique NICO+
Common NICO+
Clients
client0
client
client2
client3
client4
client5
Avg
autumn
dim
grass
outdoor
rock
water
Avg
6
75.13
73.30
70.31
68.88
73.60
72.51
72.28
54,63
49.21
58,13
54.75
54,64
47.03
53.07
30
74.24
74.32
71.63
68.97
72.53
72.10
72.29
53.23
50.47
57.07
55,66
54.09
46,49
52.84
60
74.52
73.64
70.39
69.54
73.65
71.05
72.13
54.27
49.18
57.99
54,44
53.16
48.90
52.99
180
75.64
74.06
70.89
68.67
72.15
71.94
72.23
53.26
49.45
56.55
53.86
53.19
46.73
52.17
Client Dataset
The Synthetic Datasct
Table 7. Impact of adding noise to the uploaded parameters.
Noise Weight
clipart
infograph
painting
quickdraw
sketch
average
0
44.25
17.51
38.74
9.43
57.31
38.44
34.28
33.91
30
42.82
16.73
37.63
8.67
56.51
37
53.273
100
40.86
16.04
35.28
8.19
55.68
35.14
31.86
PD
user privacy information from client models is more chal- lenging. To further demonstrate FedLMG's performance in privacy protection, conduct sufficient quantitation and visualization experiments. We select some categories from OpenImage that may contain privacy-sensitive infor- mation, such as human faces, vehicle registration plates, and notebooks. We train client models on these categories and generate synthetic datasets. The visualization results are shown in Figure 4. It can be observed that the synthetic datasets only share similar styles and accurate semanties with the original client datasets. It is almost impossible to extract specific models, which aim to learn the classification boundary.
TEAT
DENERGES SE
STADEE 7378 MY
7 6219
IEE
51 19
information from the client
Moreover, since our method only involves uploading lo- cal models, consistent with traditional FL., most existing privacy-preservve techniques traditional FL can be di- rectly applied to method. To further validate this, add noise to the uploaded model parameters to ensure dif- ferential privacy, which is commonly used in FL (Wei et al., 2020), and evaluate its impact on model performance, We add different weights of noises to the uploaded model pa- rameters and conduct experiments. The results are presented in Table 7. As shown in this table, our method can effec- tively accommodate crivacy-protecting techniques in FL. By adding noise the uploaded parameters, we can protect client privacy and achieve differential privacy.
Figure 4. Visualization of privacy-sensitive categories.
FedDF, FedProx, and FedDyn have similar client computa- tion costs, they are not compared separately. Ceiling and Prompts Only do not involve any client computation, so they are not included in the comparison. The number of iterations and the model structures used follow the default experimental settings. The quantified results demonstrate that FedLMG has a significant advantage in client compu- tation cost due to not involving any foundation model on the clients and only requiring training of client models a single round of iteration, demonstrating its practicality.
5. Conclusion
Privacy Issues. Transmitting client models is the most com- mon practice in FL. Since our method only requires upload- ing the client model once, it offers significant advantage in privacy protection compared to other traditional FL meth- ods. Compared to other OSFL methods, where either the trained generative model (Zhang et al., 2022; Heinbaugh et al., 2022) or direct descriptions of client images are up- loaded (Yang al., 2024a; Zhang et al., 2023a), extracting
In this paper, we propose FedLMG. Compared to existing OSFL methods, eliminate the need for auxiliary datasets and generator training, making it effortlessly applicable in real-world scenarios. Comprehensive experiments three large-scale datasets demonstrate that the proposed FedLMG outperforms all compared methods and even surpasses the performance ceiling centralized training in some cases, underscoring the potential of applying DMs in OSFL.

---

One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
Impact Statement
Howard, A., Sandler, M., Chu, G., Chen, L.-C., Chen, B., Tan, M., Wang, W. Zhu, Y., Pang, R., Vasudevan, V., et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1314-1324, 2019.
Given that our method utilizes a pre-trained diffusion model, there is a possibility of generating sensitive or private infor- mation. However, the Stable Diffusion model we rely on has been equipped with robust safety-checking mechanism designed to minimize such risks. We also conduct sufficient experiments to demonstrate the performance of our method in privacy protection. As result, we feel that our method does not raise additional significant concerns or potential broader impacts that warrant specific attention or further discussion here.
Hsu, T.-M. H.. Qi, H., and Brown, M. Measuring the effects of non-identical data distribution for federated visual clas- sification. arXiv preprint arXiv: 1e0e:999993335 2019.
Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., et al. Advances and open problems in federated learning. Foundations and Trends@9 in Machine Learning, 14(1-2):1-210, 2021.
Acknowledgements
This work was supported in part by the National Key R&D Program of China (No.20212D01 12803), the Na- tional Natural Science Foundation of China (No.62166661) the Shanghai Research and Innovation Functional Program (No. (N..17772200000) and the Program for Professor of Spe- cial Appointment (Eastern Scholar) at Shanghai Institutions of Higher Learning.
Kazerouni, A., Aghdam, E. K., Heidari, M., Azad, R., Fayyaz, M., Hacihaliloglu, I., and Merhof, D. Diffusion models in medical imaging: A comprehensive survey. Medical image analysis, 88: 38:102846, 2023.
Kim, G., Kwon, T., and Ye, J. C. Diffusionclip: Text- guided diffusion models for robust image manipulation. Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 2426-24262422 2435, 2022.
References
Acar, D. A. E., Zhao, Y., Matas, R., Mattina, M., What- mough, P., and Saligrama, V. Federated learning based on dynamic regularization. In International Conference on Learning Representations, 2021.
Kingma, D., Salimans, T., Poole, B., and Ho, J. Varia- tional diffusion models. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, vol- ume 34, pp. 21696-211111 21707, 2021.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8799 8794, 2021.
Kuznetsova, A., Rom, H., Alldrin. N., Uijlings. J., Krasin, I., Pont-TTuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al. The open images dataset v4: Uni- fied image classification, object detection, and visual re- lationship detection at scale. International Journal of Computer Vision 128(7):1956-199 1981, 2020.
Fantauzzo, L., Fani, E., Caldarola, D., Tavera, A., Cer- melli, F., Ciccone. M., and Caputo, B. Feddrive: Gen- eralizing federated learning to semantic segmentation in autonomous driving. 2022 IEEE/RSI International Conference on Intelligent Robots and Systems (IROS), pp- 11504-11511. IEEE, 2022.
Li, B., Chi, M., Fan, J., and Xue, X. Support cluster machine. In Proceedings of the 24th international conference on Machine learning, pp. 505-512, 2007.
Guha, N.. Talwalkar, A., and Smith, V. One-shot federated 2019.
learning. arXiv preprint
Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-imen -maaa pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19744 PMLR, 2023.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEFE conference on computer vision and puttern recognition, pp. 770-778, 2016.
Li, Q. He, B., and Song, D. Practical one-shot fed- erated learning for cross-silo setting. arXiv preprint arXiv:2010.01017, 2020a.
Heinbaugh, C. E., Luz-Ricca, E., and Shao, H. Data-free one-shot federated learning under very high statistical heterogeneity. In The Eleventh International Conference on Learning Representations, 2022.
Li, Q. He, B., and Song, D. Practical one-shot federated learning for cross-silo setting. In Proceedings of the Thirtieth International Joint Conference on Artificial In- telligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pp. 1484-149., ijcai.org. 2021.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in Neural Information Process- ing Systems, 33:6680-6851, 2020.
10

---

One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
Li, T., Sahu, A. K., Talwalkar, A., and Smith, V. Feder- ated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 37(3):50-60, 2020b.
al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Lin, T., Kong, L.,, Stich, S. U., and Jaggi, M. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33: 2351-2363, 2020.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Liu, I.,, Ren, Y., Lin, 7.,, and Zhao, Pseudo numerical methods for diffusion models on manifolds. In Interna- tional Conference on Learning Representations, 2022.
Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D., and Norouzi. M. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022a.
Mammen, P. M. Federated learning: Opportunities and challenges. arXiv preprint arXiv:2101.05422 2021.
Saharia, C., Chan, W., Saxena, S., Li, I.., Whang, J., Denton, E. L.., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image dif- fusion models with deep language understanding. Ad- vances Neural Information Processing Systems, 35: 36479-36499, 2022b.
Mao, J., Wang, X., and Aizawa, K. Guided image syn- thesis via initial image editing in diffusion model. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 5329,, 2023.
McMahan, B., Moore, E., Ramage, D., Hampson, S., and learning of deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pp. 1273-1288 PMLR, 2017.
Arcas, B. A. Communication--fiin
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510 45200, 2018.
Nguyen, A., Do, T., Tran, M., Nguyen, B. X., Duong, C.,, Phan, T., Tjiputra, E., and Tran, Q. D. Deep federated learning for autonomous driving. In 2022 IEEE Intelligent Vehicles Symposium (IV), pp. 1824-1833 1830. IEEE, 2022.
Schuhmann, C., Kaczmarczyk, R., Komatsuzaki, A., Katta, A., Vencu, R., Beaumont, R., Jitsev, J., Coombes, T., and Mullis, C. Laion-400m: Open dataset of tip-filtered 400 million image-text pairs. In NeurIPS Workshop Datacen- tric AI. Jülich Supercomputing Center, 2021.
Nichol, A. Q. Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., Mcgrew, B., Sutskever, I., and Chen, M. Glide: Towards photorealistic image generation and edit- ing with text-guided diffusion models. In International Conference on Machine Learning, pp. 16784-16804. PMLR, 2022.
Schuhmann, C.,, Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5h: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25278-25294 2022.
Peng, X., Bai, Q., Xia, X., Huang, Z., Saenko, K., and Wang, B. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1444 1015,1115,222 2019.
Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for large-scale image recognition. arXiv preprint arXiv:1409. 14t..1449...5... 2014.
Preechakul, K., Chatthee, N., Wizadwongsa, S., and Suwa- janakorn, S. Diffusson autoencoders: Toward a meaning- ful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10619- 10629, 2022.
J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International Conference on Machine Learning, pp. 2256-2265. PMIR, 2015.
Qiu, C., Li, X., Mummadi, C. K., Ganesh, M. R., Li, Z., Peng, L., and Lin, W.-Y. Text-driven prompt generation for vision- language models in federated learning. In International Workshop on Federated Learning the Age of Foundation Models in Conjunction with NeurIPS 2023, 2023.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In International Conference on Learning Representations, 2020a.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020b.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J.,

---

One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
Stanley Jothiraj, F. V. and Mashhadi, A. Phoenix: A feder- ated generative diffusion model. In Companion Proceed- ings of the ACM Web Conference 2024, pp. 1568-1577, 2024.
Zhang, Qi, X., and Zhao, B. Federated genera- tive learning with foundation models. arXiv preprint arXiv:2306. 16064, 2023a.
Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEFE//CV International Conference Computer Vision, pp. 9936-3847, 2023b.
Su. S., Li, B., and Xue, X. One-shot federated learning without server-side training. Neural Networks, 164:203- 215, 2023.
Su, S., Yang, M., Li, B., and Xue, X. Federated adaptive prompt tuning for multi-domain collaborative learning. In Proceedings of the AAAI Conference on Artificial In- telligence, volume 38, pp. 15117-15125, 2024.
Zhang. X., Zhou, X., Lin, M., and Sun, Shufflenet: extremely efficient convolutional neural network for mo- bile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6848 6856.6855. 2018.
Wang, T., Zhang, T., Zhang, B., Ouyang, H., Chen, D., Chen, Q. and Wen, F. Pretraining is all you need for image- to-image translation. arXiv preprint arXiv:2205.12955. 2022.
Zhang, X., He, Y., Xu, R., Yu, H., Shen, Z., and Cui, P. Nico++: Towards better benchmarking for domain gener- alization. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16036- 16047, 2023c.
Wang, T.-C., Liu, M.-Y., Zhu, J.-Y., Tao, A., Kautz, J., and Catanzaro, High-resolution image synthesis and semantic manipulation with conditional gans. In Pro- ceedings of the IEEE conference computer vision and pattern recognition, pp. 8798-8807, 2018.
Zhou, Y., Pu, G., Ma. X., Li, X., and Wu, D. Distilled one- shot federated learning, arXiv preprint arXiv:2009.07999, 2020.
Wang, X., Zhu, Z., Huang, G., Chen, X., Zhu, J., and Lu, J. Drivedreamer: Towards real-world-dre world models for autonomous driving. In European Conference on Computer Vision, pp. 55-72. Springer, 2024.
Wei, K., Li, Ding, M., Ma, C., Yang, H. H., Farokhi, F,, Jin, S., Quek, T. Q., and Poor, H. V. Federated learning with differential privacy: Algorithms and performance analysis. IEEE transactions on information forensscs and security, 15:3454-3469, 2020.
Yang, F.-E,, Wang, C.-Y,, and Wang, Y.-C. F. Effi cient model personalization in federated learning via client-specific prompt generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19159-19168, 2023.
Yang. M. Su, S., Li, B., and Xue. X. Exploring one-shot semi-supervised federated learning with pre-trained dif- fusion models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 16325-16333, 2024a.
Yang, M., Su, S., Li, B., and Xue, X. Feddeo: Description- enhanced one-shot federated learning with diffusion mod- els. In Proceedings of the 32nd ACM International Con- ference Multimedia, pp. 6666-6675, 2024b.
Zhang, J., Chen, C., Li, B., Lyu, L., Wu, S., Ding, S., Shen, C., and Wu, C. Dense: Data-free one-shot federated learning. Advances in Neural Information Processing Systems, 35:21414-2144 2022.
12

---

One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
In the appendix, provide substantial amount of content that couldn't be included in the main text due space limitations, which mainly be divided into three sections: Method Details. We further elaborate the details of the proposed method, including proofs of theoretical analyses and pseudo code of the proposed method. 2) Experimental Setting Details. More detailed information about the datasets and client partitions are provided, along with thorough implementation details. 3) Supplementary Experiments. We further demonstrate the performance of the proposed method through supplementary quantitation and visualization experiments, such as ablation experiments regarding the number of clients and the use different diffusion models, quantitation and visualization experiments concerning privacy, and more visualization experiments of synthetic datasets, etc.
A. Method Details
Our method focuses on one-shot federated learning (OSFL), involving using the client models trained by the clients to guide the DMs in generating the synthetic dataset that complies with different client distributions. In the main text, we delve into the relationship between the synthetic data distribution and the client distribution and provide theoretical analyses. this section, we detail our theoretical analyses and provide the pseudocode of the proposed method in Algorithm 1.
A.1. Proofss
In this section, we provide a proof of the theoretical analyses the main text, regrading the upper bound of the KL divergence between the distributions of the synthetic dataset and the client datasets. Firstly, in section 3.1 of the main text, we introduce an assumption: for the data distribution of client & pk (x) and the data distribution of the DMs Pea (x), have:
Assumption There exists À 0 such that the Kullback- divergence from (x(x)) to Peo (x) is bounded above by A:
(x) <
(14)
Based this assumption, considering the KI. divergence between the distributions of the original client dataset pı: and the synthetic dataset Peo (x(02), which is the conditional distribution of the DMs conditioned on the trained discriptions 0g. we have:
Theorem There exists X > O, for the local data distribution p& (x) and the conditional distribution Peo (x(0x) of the DM Eg conditioned the client model Fex, trained on client k. we have:
(L(p(())lle < À B(log Peo (0n)) / ps (x() log peo (Ox (x)xx
(15)
Proof. Firstly, based on the definition of KL divergence, we have:
(I(x(x))ll - px(x) log Peo (x(Ox) dx pr. (x)
(16)
Based on the Bayes' theorem, we have:
Peo (x(xx)) Pe. (0,(Ox|))pee
(17)
Pe, (0%)
From Eq. 16 and Eq. 17, we have:
13

---

One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
K (x)p(())lle -
Pa (x) log Peo (xxx0 (xo(x px(x)
-
dx
/ pk (xx) log
Peo 10x x Pee (x) dx
Ps (x)Peo (0x)
/ pa (x(xx) log
(x( (0x) dx
Pro 10x x)Pro (x)
/ px(x)) log
Pa (x() dx / px(x)) log Peo (0x|xx
Peo (0x) dx
Peo (x)
Peo (0x) [X(pp(x)]lpee (x)) + / px(xx) log
dx
(18)
Peo (0x x)
From Eq. 14 and Eq. 18, we have:
(x) (x(0+)) < x / (x(x) log
Peo Peo(Ou) (0x) Peo (0x (x)
dx
(19)
where defined in Assumption Next, we focus on the integral term within Eq. 19:
f px(())log log Peo (Ox) dx Peo Pep(0x) (0x|x)
/ px(x) log Pco (0z)dx -
pa (x) log Pes (0xlx)dx
Ellog Peo (0x)) pa log Peo (0x x)dx
(20)
Therefore, from Eq. 19 and Eq. 20, have
KL(px(x)Il (x(0s)) < A + B(log (On)) / ps (x( log Peo (0ulx)ds
(21)
With Theorem proven, we thoroughly demonstrate that under the assistance of client models, the conditional distribution of the server's DM sufficiently close to the local distribution of different clients. Moreover, the more comprehensive the client models training, the higher overlap between the non-conditional distribution of the diffusion model and the client local distributions, and the smaller the KL divergence between the conditional distribution and the local distributions. This provides a theoretical foundation for us to generate synthetic datasets that match different client distributions on the server.
B. Experimental Setting Details
In this section, we detail the experimental settings of the proposed method that couldn't be elaborated on in the main text due the space limitations, primarily comprising three parts: 1) Dataset Details. 2) Client Partition Details. 3) Implementation Details.
B.1. Dataset Details
Our experiments are conducted on three datasets: DomainNet (Peng et al., 2019), OpenImage (Kuznetsova al., 2020) and NICO+ (Zhang al., 2023c). The example images of each dataset are presented in figure 5. As mentioned in the main text, this figure clearly illustrates the emphases on the partition of data domains across different datasets is different. DomainNet primarily focuses on image style, OpenImage concentrates on fine-grained subcategories within each supercategory, Common NICO+ prioritizes image backgrounds, and Unique NICO+ places its emphasis on specific object attributes. The datasets we employ comprehensively simulate various types of differences that may exist among clients, thereby further enhancing the practicality of the proposed method.
14

---

One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
Algorithm 1 FedLMG: heterogeneous one-shot Federated learning method with Local model-Guided diffusion models
Input: The client models (Fo,, ) -= A pre-trained diffusion model eg. Output: An aggregated model Fo, adapted to the data distributions of client datasets D {x,, 30) Us E (1,...,C).
1: create empty synthetic dataset X = 0
2: for domain label A = I,...,
3: for number of synthetic images i ,....,N NN do
4: randomly select category y supported by the client model Fe..
5: randomly sample the initial noise x from (0,
6: for M,.... ,......0e.m do
7: use Eg to compute Eo (XCF, Tly)
Xm f X vI 1o (x 1 .ll)
8:
Var
9:
C(ST)) y, 0x)= < LBN (x..T 0k)
10:
XT f- X7 Vxp [(x y, Ow).
end for
XT f- x9
13:
for t T, ....., do
14:
use e0 compute eo (xt, (ly) N0. - (x /ay compute L(x,, 0x) by Eq. (6).
15:
16:
17:
compute è (xe, (ly) by Eq. (8).
18:
compute x,-1 by Eq. (2).
19:
end for
20: (x) vf) +- (Xo. 3)
21:
add (x) uft) to synthetic dataset X
22: end for
23: end for
use X {(*) , yf 1 k 1.... ..... to train an aggregated model Fe, by fine-tuning, distillation or specific-teacher distillation described in the main text. 25: return the aggregated model Fe,
B.2. Client Partition Details
B.2.1. CLIENT PARTITION.
The client partitioning primarily aimed at reflecting the non-IID of data across various clients. federated learning, there are primarily two types of non-IlI data: feature distribution skew and label distribution skew (Kairouz et al., 2021). We address these two scenarios separately in our client partition. For feature distribution skew, we conduct experiments on all four datasets. For each dataset, we allocate the six data domains of all categories to six clients, with each client possessing data from one unique domain for all categories. Regarding label distribution skew, experiments are conducted on Common NICO+ and Unique NICO+ datasets. Each 10 categories of the total 60 categories is grouped, resulting in six clients. Each client owns all data from 10 categories. As mentioned in the main text, there is no data overlap between clients in all partitions. Therefore, our partitioning maximizes the degree of non-IID among client data and considers various non-IID scenarios.
B.2.2. NUMBER OF IMAGES.
The number of images each client is important in experimental setting, since we need to compare the performance of the proposed method with Ceiling, the performance ceiling of centralized training, involving the direct comparison between the synthetic dataset and original client dataset. Considering the cost of generation, we set the number of images generated with the assistance of each client model to 30 in most experiments except the ablation experiments about the number of images. The total number of images in each category of the synthetic dataset is 180. To ensure the fairness in comparing the synthetic dataset with the original client dataset, the maximum number of images for each category each client local
15

---

One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
DomainNet
Cat
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
OpenImage Furniture
Chair
Desk
Couch
Wardrobe
Bed
Shelf'
Common NICO+ Cat
Autumn
Dim
Grass
Outdoor
Rock
Water
Unique NICO+ Cat
Eating
In Cave
In Mud
Jumping
Maine Cat
Walking
Figure 5. Example images of used datasets.
Table 8. Client partition the OpenImage dataset.
Supenenepery
Raked Gloeds
Ried
Buildiin
Camivore
Clothing
Drink
Foni
Furniture
Hlemo spplisnce
Human hody
Preseet
Woodpecker
Convenience
Shorts
Apple
Clair
Wisting Mactine
Hauus Eye
Clieat
Bagel
Parrot
House
Leopurd
Dress
Cockmal
Lemon
Desk
Towster
Skall
Cheast
Mollin
Magpia
Tower
Fox
Swimwers
Coliee
Rarana
Couch
Ohen
Haman Mouth
Eagle
Buiking
Rneail
Paloon
Ciotle
Tir
Rnl
Stove
Chiend
Crissaant
Spartow
Skyscnger
Otter
Shin
Wine
Pineopple
Shelf
Mechanical I'nn
Human Foor
Superctegery
Vihicle
Supplies
Equipment
Try
Client
Spatula
Ambilance
Drum
Fen
Magle
Dinosaur
Pootball
Doll
Foiato
Knile
Clicatl
Sponn
Cant
Guitar
Poster
Willow
Lizud
Tenas Ball
Ballonn
Carrol
Axe
Clieat
Fork
Rmscoli
Cheen
Knile
Van
Pinto
Whieboard
Lay
lonoise
Golf
Flying Disc
Cabbage
Tanogun
Cheme
Whink
Inck
Vinlin
lios
Common Sunllwer
Rugby Bail
Kile
Bell Pepper
Shoigin
Cliead
Bourd
Car
Aecordion
Houseplan
Sea Turtle
Volleybull
Teddy
Pumplin
Dugger
dataset is also set to 30, as same as the image number for each category in each data domain of the synthetic dataset.
B.2.3. NUMBER OF CLIENTS.
Regarding the number of clients, it's worth noticing that in method, each client is entirely independent of the other clients. When the total number of images of the synthetic dataset is consistent, increasing the number of clients do not introduce interference between clients and affect the performance of the proposed method. Therefore, in the majority of our experiments, we set the number clients to 6.
Nevertheless, we still conduct experiments related to the number of clients to demonstrate the practicality of the proposed method for large number of clients, Following the commonly used Dirichlet distribution in partitioning non-IID clients (Hsu al., 2019), in the feature distribution skew scenario, for the domains the Common NICO+ and Unique NICO+ datasets, sample 5, 10, and 30 clients per domain from Dirichlet(a 1.0) according to the categories. Consequently, the total number of clients changes from the original 6 to 30, 60, and 180. The clients simultaneously exhibit feature distribution skew and label distribution skew. The number of images in each category of the client local dataset remains 30. To ensure fairness in comparison across different numbers of clients, the total number of images in each category of the synthetic dataset remains 180, shifting the number of the generated images guided by each client model from 30 to 6, 3, and For example, when the number of clients is 180, each model trained on each client guides the generation of image the
16

---

One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
Table 9. Results of the ablation experiments on the used diffusion model.
The Used
Unique NICO++
Common NICO+
Diffusion Model
client0
clicnt
client2
client3
client4
client5
Avg
autumn
dim
grass
outdoor
rock
water
Avg
SD-L3
77.34
79.94
75.01
71.87
76.69
74.92
75.96
61.49
51.47
65.28
60.03
59.57
51.14
58.16
SD-2.1
78,56
80.43
75.80
72.79
76.57
74.21
76.39
60.13
52.01
66.03
59.60
59.88
52.46
58,35
LDM
75.50
77.11
75.34
70.14
72.37
72.25
73.79
58.74
50.42
65.78
58.3
55.85
50.85
56.65
Table 10. Comparison of the FID between datasets.
FID between Different Datasets
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
Synthetic
Real
284.39
228.40 44
221.01
297.37
131.48
195.74
147.94
synthetic dataset on the server.
B.3. Implementation Details
In our experiments, we mainly use ResNet-18 (He et al., 2016) as the model structure of the aggregated model. In the experiment with heterogeneous models, the structures of the client models are MobileNetV3 (Howard et al., 2019), ResNet18 (He et al., 2016), ResNet34 (He et al., 2016), MobileNetV2 (Sandler et al., 2018), VGGI6 (Simonyan & Zisserman, 2014), and ShuffleNet (Zhang et al., 2018) 8), and the structure of the aggregated model is ResNet50 (He al., 2016). The pre-trained DM we mainly used is Stable-difission-v.. from the FlaggingFace model repository, which includes corresponding CLIP text encoder used in our method to extract text features fe for the name of each category We also use Stable muable-iffusion-v2.. from the HuggingFace Face model repository and the pre-trained Latent Diffusion Model (Rombach et al., 2022) from Github. The dibble-diffusion--1.33 and Stable-diffusion diffusion-v2.1 are pre-trained on the LAION-5B dataset (Schuhmann et al., 2022) and the Latent Diffusion Model is pre-trained on the LAION-400M dataset (Schuhmann et al., 2021). Both datasets are large-scale image-text paired datasets, covering wide range of image distributions encountered in daily life to satisfy Assumption conducted with four NVIDIA GeForce RTX 3090 GPUs. Regarding specific hyperparameters, the
1. All experiments weight À in the loss function is set to 0.2. The relevant hyperparameters for the diffusion generation process are set to their default values. The number of inference steps is 50, and the guidance scale of the generation is 3.
C. Supplementary Experiments
We primarily conduct supplementary experiments targeting three aspects that are not detailed the main text because of the space limitation: 1) Ablation Experiments. Experiments regarding the used pre-trained diffusion models, the heterogencous client models and the experiments under the label distribution skew. 2) Privacy-related Supplementary Experiments. Experiments regarding the discussions in the main text, including experiments on privacy. 3) More Visualization Experiments. Experiments to further illustrate the quality and diversity of the synthetic dataset.
C.1. Ablation Experiments.
As stated in the main text, to further demonstrate the performance of the proposed method, we conduct sufficient ablation experiments. We discuss the two components of the loss function: the BN loss and the cross entropy loss, as well as the role of initial noise editing. Additionally, we discuss the impact of various hyperparameters used in our method, including the number of images generated by the server, the number of clients, and the specific employed DM. Due to space constraints in the main text, we provide some of the ablation experiments here.
C.I.1. EXPERIMENTS WITH DIFFERENT DIFFUSION MODEL
Since the synthetic datasets are used train the aggregated model, the DM used for generating these synthetic datasets is important in our method. However, this does not mean that our method is heavily dependent a specific DM. Firstly, as stated in Assumption 1, we only need partial overlap between the distributions of the DM and the client distributions. This can be easily achieved with DM pre-trained on large-scale image datasets like LAION-5B (Schuhmann et al., 2022). Even clients are concentrated in specialized fields, such as medical images, it is entirely feasible to firstly train the specialized DM on the server. Secondly, compared other diffusion- based OSFL methods (Yang et al., 2024a; Zhang al., 2023a) or
17

---

One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
Table II. Performance of different methods NICO+ with heterogencous models under feature distribution skew.
Unique NICO+ client3
Common NICO+
clientO
clientl
client2
client4
client5
Avg
autumn
dim
outdoor
rock
water
Avg
Ceiling
83.07
84.89
84.11
80.85
84.76
84.47
83.69
70.25
60,98
70.28
68.45
68.01
59.97
66.33
FedDF
71.09
70.57
67.96
66.67
62.11
52.34
65.12
49.97
43.37
60.05
55.12
53.75
51.17
52.24
Prompts Only
76.56
76,43
77.08
70.05
75.78
71.77
74.61
66.68
50.52
67.42
59.73
62.50
52.42
59.88
FodDISC
80.15
77.53
77.18
72.92
76.55
73.85
76.36
65,93
52.78
68.58
60.35
64.09
54,20
60.98
FGL
79,47
78.76
78.73
71.88
77.86
70.59
76.21
65,31
54.38
70.29
62.67
60.28
53,08
61.01
FedLMG_FT
81.77
76.56
79.68
75,03
76.82
77.24
77.85
65.06
57.33
69.47
63.64
66.01
57.34
63.14
FedLMG_SD
81.38
79.68
82.03
76.63
79.42
74.31
78.91
64.62
55.51
66.42
61.89
62.40
55.26
61.01
FedLMG MD
81.64
81.91
82.55
81.89
82.94
75.48
81.07
68.19
57.55
71.46
66.28
67.18
58.82
64.91
Table 12. Performance of different methods on NICO++ under label distribution skew.
Unique NICO++
Common NICO+
clientO
clientl
client2
client3
elient4
clientS
Avg
autumn
dim
grass
outdoor
rock
water
Avg
Ceiling
74.02
78.90
79.68
74.47
77.34
77.47
76.98
50.24
54.36
63.35
64.82
61.99
65.09
FelAvg
34,96
58.98
38
63.41
45.44
59.76
50.16
18.23
27.79
36.32
52.42
37.96
39.24
35.33
FelDF
$1.85
$2.34
55.85
52.47
54.42
59.24
54.36
31.40
32.22
43.73
45.19
36.01
43.08
38.61
FedProx
54.55
60.51
54.05
58.34
55.69
57.78
56.82
37.31
35.95
42.78
48.92
41.07
47.53
42.26
FydDyn
$5,29
59.71
56.68
61.74
48.99
61,31
57.29
36.83
37.85
45.21
51.38
42.74
44,36
43.06
Prompis Only
67.38
71.88
67.70
64.19
63.41
63.28
66.31
38.64
45.55
53.08
54.72
50.19
59.91
50.35
FedDISC
71.89
73.20
70.51
70.02
75.62
69.82
71.84
50.75
51.64
60.79
58.33
55.41
57.28
55.70
FGL
69.51
74.59
71.36
69.41
69.65
71.42
70.99
45.34
51.41
60.44
59.65
58.87
62.33
56.34
FedLMG_FT
73.30
71.48
68.97
69.71
72.91
70.31
58.98
46.53
60.93
57.45
53.92
54.32
55.36
FedLMG_SD SD
67.77
76.04
73.95
70.44
76.56
68.35
72.19
49.27
52.63
57.52
59.84
65.11
64.98
58.25
federated learning methods based on foundation models (Su et al., 2024: Yang et al., 2023; Qiu et al., 2023), our method does not utilize the foundation models on the clients. Therefore, there is no requirement for the foundation model to the client data, significantly reducing the dependency on the foundation model.
To substantiate this claim, we conduct ablation experiments on three commonly used diffusion models, Stable-diffusion--.... ttable-diffusion-v2.. and Latent Diffussion Model (LDM). The experimental results are provided in Table B.2.1. From the table, it is evident that: 1) Our method can train high-performance aggregated models with different diffusion models. 2) Although the performance with LDM consistently surpasses other traditional FI. methods, there is performance gap compared to using Stable Diffusion. This is mainly because the LDM is pre-trained on LAION-400M (Schuhmann et al., 2021), and the difference in data scale results in a less extensive distribution and limited generative capability. This indirectly supports Theorem 1. 3) The best training results are achieved using Stable-diffusion-v22 Despite both Stable-diffusion--2. and Stable-diffusion generative quality due its parameters and improved denoising capability. These results indicate that our method is limited to the specific diffusion model, enhancing its practicality.
..5 being pre-trained on LAION-5B (Schuhmann al., 2022), Stable-diffusion-Ve v2.1 has better
C. 1.2. EXPERIMENTS ON HETEROGENEOUS CLIENT MODELS
In addition to data heterogeneity. we also discuss the performance of FedLMG when there is heterogeneity in model structures across clients. Since FedAvg, FedProx, and FedDyn all require averaging the model parameters uploaded by clients and do not support heterogeneous model aggregation, they are not included in the comparison. The results are shown in the table B.3. From the table, we highlight the following two observations: 1) Because FedLMG does not restrict the specific model structure used by clients, these results also demonstrate excellent performance of our method on heterogeneous models, consistently surpassing all comparison methods and exceeding the performance ceiling of centralized training on some clients, further enhancing the practicality of FedL.MG. 2) Notably, unlike the scenario with homogeneous client models, FedLMG MD achieves better performance on most clients with heterogencous models. The primary reason that due to differences in model structures, the classifiers uploaded by clients vary significantly in performance. As a result, teachers from other clients might provide more accurate information despite the differences among clients, leading to better performances.
18

---

One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
DomainNet
Angel
DomainNet
Bathtub
DomainNet
Book
DomainNet Bridge
RUGPUE
DomainNet
Car
Figure 6. Supplementary visualization of the synthetic dataset DomainNet.
C.1.3. EXPERIMENTS UNDER LABEL DISTRIBUTION SKEW
To validate the performance of FedLMG under label distribution skew, we conduct experiments Unique NICO++ and Common NICO++. Due to the label distribution skew among clients, the teachers from different clients cannot provide effective information, and therefore FedLMG MD is not included this comparison. The results are presented in Table B.3. As shown in the table, similar to the experiments conducted under feature distribution skew, our method also outperforms all comparison methods under label distribution skew, with performance on some clients even surpassing the centralized training upper bound. This further demonstrates the adaptability of our method various non-IID client data scenarios.
C.2. Privacy-related Supplementary Experiments.
In the main text, we conduct sufficient discussions and visualization experiments demonstrate the privacy protection performance of our method. Here, further address privacy issues, including comparing the FID (Fréchet Inception Distance) between the synthetic dataset and the client datasets, as well presenting
visualization experiment results.
To illustrate the effectiveness of our method in preserving privacy, we assess the Fréchet Inception Distance (FID) between the synthetic dataset and the client local datasets, focusing on categories containing potentially sensitive information, such as the Face in DomainNet. Specifically, we divide the dataset of the Real domain into two non-overlapping parts, referred to as Real-A and Real_B, to represent the FID between datasets without privacy leakage but with the same style. We compute the FID between Real A and datasets from other domains, as well the Real domain of the synthetic dataset. Additionally, we gather photos of the same individual with different styles from the internet, partitioning them into two groups and calculating the FID between these groups establish threshold for potential privacy leakage. Through various experiments, determine that this FID threshold is approximately 80, shown in Table 10. Analysis of the table reveals that the FID between Real A and the synthetic dataset falls within an optimal Lange-not too low, which could imply the presence of identical images leading to privacy concerns, nor too high, indicating significant style discrepancies the generated images. These quantitative findings affirm that with the assistance of client models, the server can produce a synthetic dataset that aligns with client distributions while safeguarding sriviiy-sensitive information.
19

---

One-Shot Heterogeneous Federated Learning with Local Model-Guided Diffusion Models
Spatula
Spoon
Fork
Knife
Whisk Cutting Board
Openlmage Kitchen
Utensil
Autumn
Dim
Grass
Outdoor
Rock
Water
Common NICO+
Hclicopter
Autumn
Dim
Grass
Outdoor
Rock
Water
Common
NICO+
Train
Black
Nearby Tower
On Fire
On Ground
Pink
With Rainbow
Unique
+ICO+
Hot Air Balloon
Aside People Bell Tent
Blue
Military Mongolia Yurt Yellow
Unique
NICO+
Tent
Figure 7. Supplementary visualization of the synthetic dataset on OpenImage and NICO++,
C.3. More Visualization Experiments.
Similar to the main text, conduct more visualization experiments to illustrate the quality and diversity of the synthetic dataset. The experimental results are presented in Figure 6 and 7. These visualizations further demonstrate that the generated synthetic dataset complies with various client distributions in style, subcategory, or background, with accurate semantic information. The synthetic dataset has comparable diversity and quality with the original client datasets, directly contributing to the performance of the trained aggregated model.
20


Reviews:
[{"review_id": "FvOdL8RZ33", "reviewer": "Reviewer_vCBA", "summary": "In response to the increasing demand for efficient One-Shot Federated Learning (OSFL) solutions, this paper introduces FedLMG, a novel OSFL method leveraging Local Model-Guided diffusion models. Unlike existing OSFL methods that rely on foundation models deployed on client devices—causing significant computational overhead—FedLMG allows clients to train and upload only their local models, maintaining the lightweight nature of traditional Federated Learning (FL).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your review and valuable comments and provide detailed responses to the key concerns:\n\n>**(Privacy Concerns)** *Claims And Evidence #1 & Weaknesses \"general quantitative metrics for privacy of FL, such as Gradient Leakage (GL) or Differential privacy (DP).\"*\n\nRegarding GL, as discussed in [1] and [2], such attacks occur during the sharing of gradients , where attackers infer client data by analyzing gradients. However, our method does not involve sharing gradients, mitigating the risk of GL.\n\nRegarding DP, because our method only involves uploading local models, aligning with traditional FL, most DP preserving methods in FL can be directly applied to our method. To further validate this, we incorporated the method proposed in [3] to ensure DP and assessed its impact on performance. The following table demonstrate that the DP preserving methods remain applicable to our method without significantly compromising its effectiveness.\n\n|noise level $\\epsilon$ |\tclipart\t|infograph|\tpainting|\tquickdraw|\treal|\tsketch|\taverage|\n| :----: |:----: |:----: |:----: |:----: |:----: |:----: |:----: |\n|0|\t**44.25**|\t**17.51**|\t**38.74**\t|**9.43**\t|**57.31**|\t**38.44**\t|**34.28**|\n|20|\t43.53|\t17.06|\t38.49|\t9.13|\t57.08\t|38.15\t|33.91|\n|50|42.82|\t16.73|\t37.63\t|8.67|\t56.51\t|37.25\t|33.27|\n|100|\t40.86\t|16.04|\t35.28|\t8.19\t|55.68\t|35.14|\t31.86|\n\n>**(Server Computation Cost)** *Claims And Evidence #2 & Methods And Evaluation Criteria #2: \"... much more burdens to the server.\"*\n\nIn FL, the server is generally designed to have sufficient resources to handle the aggregation of client models, but clients often exhibit significant heterogeneity, necessitating constraints on computation costs[5]. Our method adheres to this principle by reducing client burdens.\n\nAdditionally, the following table presents a comparision of the server computation costs with other DM-based FL methods. Our method employs local models rather than additional prompts for guiding generation, eliminating the need for compositional diffusion, imposing a lower server computation cost.\n\n| |FedDISC| FGL\t|FedDEO|\tOSCAR|\tFedLMG|\n| :----: |:----: |:----: |:----: |:----: |:----: |\n|flops (T)\t|135.71\t|102.83|\t101.78|\t67.85\t|**38.87**|\n\n>**(Additional Ablation Experiments)** *Experimental Designs Or Analyses #1: \"More ablation experiments are needed to prove the effectiveness of the proposed method.\"*\n\nWe appreciate the reviewer’s suggestion. The following table show that increasing the number of  generated images leads to an improvement in the performance. Moreover, we observe that the improvement does not saturate as the dataset size increases, further demonstrating the diversity of the synthetic dataset.\n\n|the number of generated images| \tclipart\t|infograph\t|painting|\tquickdraw|\treal|\tsketch\t|average|\n| :----: |:----: |:----: |:----: |:----: |:----: |:----: |:----: |\n|10|\t40.77|\t15.95\t|35.66|\t8.51|\t55.81|\t37.1|\t32.3|\n|30\t|44.25\t|17.51\t|38.74\t|9.43|\t57.31\t|38.44|\t34.28|\n|50\t|**46.03**|\t**18.61**\t|**40.07**|\t**10.7**\t|**59.27**\t|**40.72**\t|**35.9**|\n\n>**(Multimodality)** *Methods And Evaluation Criteria #1: \"... how the proposed method will perform on non-image and other modality datasets.\"*\n\nSimilar to many FL studies[5], we select image as the primary modality in our paper. However, our method is not restricted to the image modality. By utilizing DMs for other modalities such as [6, 7], our method can be seamlessly adapted to other modalities.\n\n>**(Dataset Heterogeneity)** *Experimental Designs Or Analyses #2: \"The impact of the heterogeneity of datasets in the paper should be mentioned.\"*\n\nDataset heterogeneity in FL primarily manifests as feature distribution skew and label distribution skew [8]. In Tables 1 and 10, we demonstrate the impact of both types of heterogeneities. Therefore, we respectfully disagree with this concern.\n\n>**(Contribution of the Paper)** *Relation To Broader Scientific Literature: \"The key contribution of the paper is in the knowledge aggregation part...\"*\n\nAs stated in the Introduction and acknowledged by Reviewer #A93d and #5oFb, a key advantage of our method is eliminating the need for foundation model inference on clients. We believe this characteristic significantly enhances the practicality of diffusion-based FL methods and represents a meaningful contribution to the field.\n\n\n\t[1] Deep leakage from gradients, NIPS 2019.\n\t[2] Understanding Deep Gradient Leakage via Inversion Influence Functions, NIPS 2023\n\t[3] Federated Learning with Differential Privacy: Algorithms and Performance Analysis, TIFS 2020.\n\t[4] FedDEO: Description-Enhanced One-Shot Federated Learning with Diffusion Models, MM 2024.\n\t[5] A survey on federated learning: challenges and applications, IJMLC 2023.\n\t[6] DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models, ICLR 2023.\n\t[7] DiffWave: A Versatile Diffusion Model for Audio Synthesis, ICLR 2021.\n\t[8] Federated Learning on Non-IID Data Silos: An Experimental Study, ICDE 2022.", "claims_and_evidence": "1. The privacy issue of the proposed approach remain questioned. Although the paper visualized the synthetic data, the limited discussions do not convince that the proposed method preserves user privacy. The proposed method highly relies on the generated synthetic dataset to present server-side model aggregation, which is naturally a negative part of diffusion-based FL that contradicts the privacy nature of FL.\n2. The paper also claims that using the proposed method is computationally efficient. However, neither training the stable diffusion model to generate synthetic data nor the multi-teacher distillation process for knowledge aggregation is efficient. Compared with traditional FL, this raises much more burdens to the server. Also, does the proposed aggregation method via distillation introduces instability due to wrong teacher selection?", "methods_and_evaluation_criteria": "1. The evaluations and discussions in the paper are only based on the image dataset. It is questioning how the proposed method will perform on non-image and other modality datasets, especially the privacy issues of generated data in the other modalities.\n2. The computation costs comparison in the paper includes the communication cost and client computation costs. However, the cost of model aggregation on the server is not mentioned, which should be less-important but still necessary metric of the algorithm.", "theoretical_claims": "No issues on proofs.", "experimental_designs_or_analyses": "1. More ablation experiments are needed to prove the effectiveness of the proposed method, such as hyper-parameter studies, the size of synthetic data, the teacher selection constraints during distillation, etc. \n2. The impact of the heterogeneity of datasets in the paper should be mentioned. \n3. In Table 3, comparing the client computation costs between FedAvg and FedLMG to show that FedLMG is even more efficient than the traditional FedAvg does not make sense without simultaneously providing the convergence speed over communication rounds.", "relation_to_broader_scientific_literature": "The key contribution of the paper is in the knowledge aggregation part -- leveraging a synthetic dataset as the aggregated knowledge of all clients instead of leveraging a unified model as the aggregated knowledge. In terms of federated learning, this might be new. \n\nHowever, the data synthesis associated with multi-teacher distillation is not new in domain generalization and knowledge distillation fields. The paper itself does not seem contribute significantly to the broader ML community.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "1. Privacy discussions of the synthetic dataset are based on selected visualization results and FID scores. However, FID is not an official metrics for privacy protecting performance. How does it perform in terms of general quantitative metrics for privacy of FL, such as Gradient Leakage or Differential privacy, if applicable?", "other_comments_or_suggestions": "Typo \"Table 4.2\" -> Table 4 in section 4.3.", "questions_for_authors": "The aggregated information from clients on server is represented as a synthetic dataset instead of an aggregated model like traditional FL.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q7z1n65tR6", "reviewer": "Reviewer_A93d", "summary": "This paper introduces FedLMG, a novel approach for One-shot Federated Learning (OSFL) designed to establish an aggregated model within a single communication round. Specifically, FedLMG leverages fully-trained client models as classifier guidance to facilitate diffusion generation at the server. The generated images can represent the data distributions of the clients and are subsequently used for training an aggregated model. Experimental results demonstrate that FedLMG achieves superior performance compared to conventional Federated Learning (FL) and alternative diffusion-based OSFL methods, and sometimes even outperforms centralized training, which typically serves as the upper bound for FL.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive comments of our work and address each of your concerns as follows:\n\n>**(Server Cost)** *Claims #1 & Questions #3: \"… DMs on the server, which can be an additional computation cost\"*\n\nIn FL, the server is generally designed to have sufficient resources to handle the aggregation of client models, but clients often exhibit significant heterogeneity, necessitating constraints on computation costs[1]. Our method adheres to this principle by reducing client burdens.\n\nAdditionally, the following table presents a comparision of the server computation costs with other DM-based FL methods. Our method employs local models rather than additional prompts for guiding generation, eliminating the need for compositional diffusion, imposing a lower server computation cost.\n\n| |FedDISC| FGL|FedDEO|OSCAR|FedLMG|\n| :----: |:----: |:----: |:----: |:----: |:----: |\n|flops (T)\t|135.71|102.83|101.78|67.85|**38.87**|\n\n>**(Uncertain Clients)** *Claims #2: \"… we do not know if the clients’ data are in specialized domains\"*\n\nThe setting that the specific client tasks is uncertain for the server is Many-Task FL[2], where clients have diverse tasks simultaneously. However, this setting is not common. Most FL research presumes that the client task is known[1,3]. Therefore, we consider that our task setting is reasonable in real-world applications.\n\n>**(Applicability)** *Claims #2 & Weaknesses #2 & Questions #2: \"… may not have the data and computation resources to train a specialized DM.\"*\n\nCurrently, pre-trained DMs have been widely applied across various domains[4,5,6]. Based on the above discussion, we posit that the server know the overall task and can select an appropriate DM. Even where the data distribution is challenging, a pre-trained DM from a similar domain can be fine-tuned on the server. We assert that our method possesses practicality in diverse application scenarios.\n\n>**(References)** *Essential References: \"... similar idea of using guidance for DM generation in OSFL are missing …\"*\n\nWe appreciate your suggestion. We incorporate the relevant works into our compared methods. As shown in the table below, our method achieves comparable performance without the employment of any foundational models on the clients, further demonstrating the performance of our method.\n\n||client0|\tclient1|\tclient2|\tclient3|\tclient4|\tclient5|\taverage|\n| :----: |:----: |:----: |:----: |:----: |:----: |:----: |:----: |\n|FedLMG_FT|\t48.99|\t51.66|\t55.59|\t52.80|\t62.41|\t58.86|\t55.05|\n|FedLMG_SD|\t47.60|\t**55.20**|\t**61.54**|\t61.83|\t67.07|\t**59.90**|\t**58.86**|\n|FedLMG_MD|\t44.70|\t53.08|\t58.67|\t60.13|\t64.06|\t58.06|\t56.45|\t\t\t\t\t\t\t\t\t\n|FedDEO\t|\t**51.08**|\t52.53|\t61.22|\t**62.18**|\t67.31|\t56.68|\t58.50|\n|OSCAR\t|\t50.89|\t53.51|\t60.05|\t61.98|\t**68.76**|\t56.52|\t58.61|\n\n>**(BN Loss)** *Weaknesses #1: \"… whether the proposed BN loss is the most suitable choice\"*\n\nAs noted by Reviewer #xTsg, BN Loss has been widely utilized[7]. Although there is no prior precedent for employing BN Loss in the DMs, the general method of designing task-specific loss functions and guiding the diffusion process is well established[8]. While we acknowledge that BN Loss might not be the optimal choice in all circumstances, its effectiveness has been clearly validated by the ablation experiments presented in Table 4.\n\n>**(Privacy)** *Questions #1: \"… applying noisy SGD during client model training offer stronger privacy protection\"*\n\nWe appreciate your suggestion. Since our method only involves uploading local models, aligning with traditional FL, most privacy preserving methods in FL can be directly applied to our method. To validate this, we adopt [8] to ensure differential privacy and evaluate its impact on model performance. The experimental results shown in the table below indicate that traditional FL privacy protection measures remain effective within our framework without significantly degrading model performance.\n\n|noise level $\\epsilon$ |\tclipart\t|infograph|\tpainting|\tquickdraw|\treal|\tsketch|\taverage|\n| :----: |:----: |:----: |:----: |:----: |:----: |:----: |:----: |\n|0|\t**44.25**|\t**17.51**|\t**38.74**\t|**9.43**\t|**57.31**|\t**38.44**\t|**34.28**|\n|20|\t43.53|\t17.06|\t38.49|\t9.13|\t57.08\t|38.15\t|33.91|\n|50|42.82|\t16.73|\t37.63\t|8.67|\t56.51\t|37.25\t|33.273|\n|100|\t40.86\t|16.04|\t35.28|\t8.19\t|55.68\t|35.14|\t31.86|\n\n\t[1] A survey on federated learning: challenges and applications, IJMLC 2023.\n\t[2] Many-Task Federated Learning: A New Problem Setting and A Simple Baseline, CVPR 2023.\n\t[3] A survey on federated learning systems: Vision, hype and reality for data privacy and protection, TKDE 2021.\n\t[4] DMs in medical imaging: A comprehensive survey, MIA 2023.\n\t[5] DiffuSeq: Sequence to Sequence Text Generation with DMs, ICLR 2023.\n\t[6] DiffWave: A Versatile DM for Audio Synthesis, ICLR 2021.\n\t[7] Dreaming to distill: Data-free knowledge transfer via deepinversion, CVPR 2020.\n\t[8] Federated Learning with Differential Privacy: Algorithms and Performance Analysis, TIFS 2020.", "claims_and_evidence": "In Line 99 (in the left column), the paper states “we propose FedLMG, a novel OSFL method, to\nachieve real-world OSFL without utilizing any foundation models on the clients, ensuring no additional\ncommunicational or computational burden compared to traditional FL methods.” However, FedLMG needs image generation with diffusion models on the server, which can be an additional computation cost because traditional Federated Learning does not require this step. \n\nIn Line 214 (in the left column), the paper mentions “Even if the clients specialize in certain professional domains, like medical images, it’s entirely viable to train specialized diffusion models on the server. Hence, this assumption is entirely reasonable, considering a comprehensive assessment of practical scenarios.” \nHowever, within the context of Federated Learning, we do not know if the clients’ data are in specialized domains, and the server may not have the data and computation resources to train a specialized diffusion model. Therefore, rather than characterizing the aforementioned assumption as \"entirely reasonable,\" it would be more accurate to consider it as a potential limitation of the proposed method.", "methods_and_evaluation_criteria": "The proposed method, FedLMG, involves transmitting fully trained client models to a central server, where they are utilized to synthesize images for establishing an aggregated model. As the trained client model can be regarded as a compressed representation of its data distribution, the proposed method makes sense and is aligned with the setting of One-shot Federated Learning.", "theoretical_claims": "In Theorem 1, the paper claims that the KL divergence between the client’s data distribution and the conditional distribution of the synthetic data is upper-bounded. Moreover, minimizing the cross-entropy loss within a client reduces the upper bound of the KL divergence. I’ve checked the proof in Appendix A, and the result seems to be correct.", "experimental_designs_or_analyses": "The experiments are conducted under standard Federated Learning settings, where heterogeneous class and style distributions are distributed across different clients. The performance metric is the accuracy of the aggregated model evaluated across all clients. In addition, the paper presents an ablation study on the classification and BN losses of the proposed method. Overall, the experimental design and analysis are considered valid.", "relation_to_broader_scientific_literature": "This paper applies an existing idea of classifier-guidance diffusion models to the domain of One-shot Federated Learning (OSFL). The main contribution resides in the connection between these two distinct areas and the introduction of a BN loss to further improve the OSFL performance. Compared to existing OSFL methods such as FGL and FedDISC, a key advantage of the proposed FedLMG is that it eliminates the need for foundation model inference on clients, which may have limited computational resources.", "essential_references_not_discussed": "Some related papers that follow the similar idea of using guidance for diffusion model generation in OSFL are missing, such as FedDEO [1], OSCAR [2], and FedCADO [3]. \n\n[1] 2024 ACM MM FedDEO: Description-Enhanced One-Shot Federated Learning with Diffusion Models \n\n[2] 2025 arXiv One-Shot Federated Learning with Classifier-Free Diffusion Models \n\n[3] 2023 arXiv One-Shot Federated Learning with Classifier-Guided Diffusion Models", "other_strengths_and_weaknesses": "Strengths \n- The paper is well-written and easy to follow. The proposed FedLMG also demonstrates promising performance in experimental evaluations.\n- The paper addresses privacy concerns related to regenerating client data distributions using guided diffusion models and demonstrates, through visualization, that specific privacy-sensitive information may remain concealed and preserved.\n\nWeaknesses \n- The main idea of the paper lies in improving the use of diffusion models in OSFL by leveraging trained client classifiers as guidance and introducing the BN loss. However, the paper does not thoroughly discuss prior work on guided diffusion models [1][2], making it unclear whether the proposed BN loss is the most suitable choice for the OSFL setting or if existing methods could be equally applicable.\n- A potential limitation arises from the requirement for some degree of overlap between the client data distributions and the diffusion model’s data distribution, as determined by the parameter $\\lambda$ in Equation 4. Specifically, when clients have highly specialized data distributions, such as medical images, the diffusion model may struggle to reconstruct these distributions accurately due to a large $\\lambda$ value. Although the paper suggests that a specialized diffusion model could be trained on the server to address this issue, doing so may not be feasible in practice due to limited data or computational resources.\n\n\n[1] 2024 NeurIPS TFG: Unified Training-Free Guidance for Diffusion Models \n\n[2] 2023 CVPR Universal Guidance for Diffusion Models", "other_comments_or_suggestions": "There are several typos from Line 178 and Line 188 (in the right column). \n- Eq. 15 should be Eq. 5. \n- Eq. 14 should be Eq. 4. \n- It should be $p_k(x)$ instead of $p_n(x)$\n\nIn Appendix A, the proof is cut by Algorithm 1, making it a little hard to read.", "questions_for_authors": "Besides the weakness in the above sections, please also check the questions below: \n\n1. Although the paper provides some visualizations suggesting that privacy-sensitive information may not be revealed by the guided diffusion generation, would applying noisy SGD (or related techniques) during client model training offer stronger privacy protection with formal differential privacy guarantees? How would this impact the performance of FedLMG? \n\n2. How does the different degree of $\\lambda$ impact the performance of FedLMG? For example, if clients hold data such as medical or aerial images, the diffusion model might struggle to reconstruct such distributions accurately. In this case, could FedAvg potentially perform better since it is indirectly trained on client data? Alternatively, how could FedLMG be adapted to handle such scenarios effectively? \n\n3. Compared to FedAvg, which performs model aggregation through a simple averaging process, FedLMG requires significantly more computational resources on the server side due to image generation via diffusion models. What is the computational cost associated with this image-generation process? In Table 3, only the client-side cost is considered. How might the results change if the server-side cost is also taken into account?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uch81pJn5E", "reviewer": "Reviewer_5oFb", "summary": "This paper introduces FedLMG, a novel One-Shot Federated Learning (OSFL) method addressing limitations of diffusion model-based OSFL. FedLMG leverages locally trained client models to guide a server-side diffusion model in generating synthetic datasets tailored to individual client distributions. This approach eliminates the need for foundation models on clients, reducing computational burden and enhancing adaptability to heterogeneous clients. Extensive experiments on multiple datasets demonstrate FedLMG's superior performance over existing methods, even surpassing centralized training in some scenarios. Theoretical analysis and visualizations confirm the high quality and diversity of the generated synthetic data and the method's effectiveness in capturing client-specific distributions, highlighting the potential of diffusion models in practical OSFL.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our work. Below, we provide detailed responses to your concerns:\n\n>**(Server Cost)** *Questions #3\"... server computing costs.\"*\n\nIn FL, the server is generally designed to have sufficient resources to handle the model aggregation, but clients often exhibit significant heterogeneity, necessitating constraints on costs[1]. Our method adheres to this principle by reducing client burdens.\n\nAdditionally, the following table presents a comparision of the server computation costs with other DM-based FL methods. Our method employs local models rather than additional prompts for guiding generation, eliminating the need for compositional diffusion, imposing a lower server computation cost.\n\n||FedDISC| FGL|FedDEO|OSCAR|FedLMG|\n|:----:|:----:|:----:|:----:|:----:|:----:|\n|flops (T)|135.71|102.83|101.78|67.85|**38.87**|\n\n>**(Additional Ablation Experiments)** *Questions #4 \"… vary with dataset size.\"*\n\nWe appreciate the reviewer’s suggestion. The following table show that increasing the number of  generated images leads to an improvement in the performance. Moreover, we observe that the improvement does not saturate as the dataset size increases, further demonstrating the diversity of the synthetic dataset.\n\n|the number of generated images|clipart|infograph|painting|quickdraw|real|sketch|average|\n|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n|10|40.77|15.95|35.66|8.51|55.81|37.1|32.3|\n|30|44.25|17.51|38.74|9.43|57.31|38.44|34.28|\n|50|46.03|18.61|40.07|10.7|59.27|40.72|35.9|\n\n>**(Experimental Analysis)** *Weaknesses #2 \"FedLMG_SD should give the best result ... no relevant analysis.\"*\n\nWe would like to clarify that we include a dedicated analysis in line 937. We argue that due to the varying architectures of client models, some clients with more complex model structures demonstrate superior performance, allowing them to provide more accurate knowledge than the specific teacher in FedLMG_SD.\n\n>**(Privacy-Related Experiments)**: *Weaknesses #3 \"… the threshold chosen is subjective.\"*\n\nWe appreciate your suggestion, to further verify our method’s effectiveness in privacy protection, we employe [2] to ensure differential privacy and evaluate its impact on model performance. The results in the table below indicate that since our method only involves uploading local models, aligning with traditional FL, most privacy preserving methods in FL can be directly applied to our method without significantly degrading model effectiveness.\n\n|noise level $\\epsilon$|clipart\t|infograph|painting|quickdraw|real|sketch|average|\n|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n|0|**44.25**|**17.51**|**38.74**|**9.43**|**57.31**|**38.44**|**34.28**|\n|20|43.53|17.06|38.49|9.13|57.08|38.15|33.91|\n|50|42.82|16.73|37.63|8.67|56.51|37.25|33.273|\n|100|40.86|16.04|35.28|8.19|55.68|35.14|31.86|\n\n\n>**(Distribution Similarity)** *Questions #1 \"Boundedness lacks quantitative validation ... client data distributions differ significantly...\"*\n\nQuantitatively evaluating the similarity between the default distribution of a DM and client distributions is challenging, particularly given the large-scale pre-training datasets of DM. However, with the recent advancements, pre-trained DMs tailored for various domains[3,4] have become increasingly available. We believe that servers can select appropriate DMs based on the target application. Even where the data distribution is challenging, a pre-trained DM from a similar domain can be fine-tuned on the server. Thus, We assert that our method possesses practicality in diverse application scenarios.\n\n>**(BN Loss)** *Weaknesses #1 \"... BN loss lacks a strong theoretical basis.\"*\n\nAs noted by Reviewer #xTsg, BN Loss has been widely applied[5]. Since BN Loss compares statistics, it provides an intuitive guidance and is adopted in studies such as [5,6] without additional theoretical analysis. We plan to further explore its theoretical underpinnings in future to strengthen our method’s theoretical foundation.\n\n>**(Effectiveness of Guidance)** *Questions #2\"... lacks quantitative analysis of the guidance effectiveness.\"*\n\nWe would like to clarify that we provide quantitative analyses of the effectiveness of guidance. The Prompts Only in Table 1 represent results where no local model guidance was used, whereas FedLMG denotes results with guidance. We believe that the comparison between these settings sufficiently demonstrates the effectiveness of the guidance in our method.\n\n\t[1] A survey on federated learning: challenges and applications, IJMLC 2023.\n\t[2] Federated Learning with Differential Privacy: Algorithms and Performance Analysis, TIFS 2020.\n\t[3] Diffusion probabilistic models for 3d point cloud generation, CVPR 2021.\n\t[4] DiffuSeq: Sequence to Sequence Text Generation with DMs, ICLR 2023.\n\t[5] Dreaming to distill: Data-free knowledge transfer via deepinversion, CVPR 2020.\n\t[6] Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation? NIPS 2024.", "claims_and_evidence": "The paper's central claim - the effectiveness and superiority of FedLMG for OSFL - is strongly supported by comprehensive evidence. Extensive quantitative experiments across diverse datasets (Table 1) convincingly demonstrate FedLMG's outperformance against various baselines, including traditional FL and other diffusion-based OSFL methods. The claim of surpassing centralized training ceilings is also empirically supported. Ablation studies (Table 4, Figure 3, Appendix C.1) provide evidence for the roles of BN loss and classification loss. Theoretical analysis (Theorem 1, Appendix A.1) offers a formal justification for the method's ability to generate client-aligned data. Visualizations (Figures 2, 4, 7, 8) qualitatively support the high quality and diversity of synthetic datasets and privacy-preserving nature.", "methods_and_evaluation_criteria": "The paper proposes FedLMG, a novel method for one-shot federated learning utilizing diffusion models guided by locally trained client models. The methodology, encompassing local client training, guided synthetic data generation, and three aggregation strategies, is well-suited for addressing OSFL challenges, particularly in heterogeneous settings. The evaluation is comprehensive, employing large-scale datasets: OpenImage, DomainNet, and NICO++. Benchmarking against strong baselines, including traditional FL methods, diffusion-based OSFL methods, and a centralized training ceiling, provides a robust comparative analysis. Classification accuracy serves as a relevant and standard metric for evaluating model performance in image classification tasks within federated learning.", "theoretical_claims": "The paper presents one main theoretical claim in Theorem 1, which is formally proven in Appendix A.1. I have carefully examined the provided proof of Theorem 1 and found it to be mathematically sound and logically consistent. The proof correctly demonstrates that, under Assumption 1 (bounded KL divergence between the diffusion model's unconditional distribution and client data distribution), the KL divergence between the synthetic dataset distribution and the client's local data distribution is indeed bounded.", "experimental_designs_or_analyses": "The experimental designs are robust and effectively validate FedLMG. The core experiments (Table 1) comprehensively assess performance under feature distribution skew across diverse datasets, using appropriate baselines and metrics (accuracy). Ablation studies systematically dissect the contributions of key components like BN loss and diffusion model choices (Table 4, Appendix C.1), strengthening mechanistic understanding. The exploration of heterogeneous client models and label distribution skew (Appendix C.1) broadens the evaluation scope. Privacy experiments employing FID and visualizations (Figure 4, Appendix C.2) directly address privacy concerns. Visualization of synthetic data (Figures 2, 7, 8) provides qualitative validation of data quality and diversity.", "relation_to_broader_scientific_literature": "FedLMG makes significant contributions to the intersection of Federated Learning (FL) and Diffusion Models (DMs). It addresses limitations of existing DM-based One-Shot FL (OSFL) methods (FedDISC, FGL) by eliminating the need for foundation models on clients. Unlike methods relying on public auxiliary data, FedLMG cleverly utilizes locally trained client models to guide DM-based data generation, a novel approach compared to generator-based OSFL and auxiliary information transfer methods. The distillation-based aggregation strategies build upon knowledge distillation in FL but introduce specific adaptations for OSFL with synthetic data.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths: \n1.FedLMG presents a new approach to OSFL by innovatively using locally trained client models to guide diffusion-based synthetic data generation.\n2.The paper provides theoretical justification for the method, adding rigor and confidence to the empirical findings.\n3. The method significantly enhances the practicality of OSFL by eliminating the need for foundation models on resource-constrained clients and effectively addressing heterogeneous client scenarios.\n\nWeaknesses:\n1.Although the paper provides a theory about KL divergence boundary, the specific selection of BN loss as a guiding mechanism lacks a strong theoretical basis.\n2.Of the three polymerization strategies, FedLMG_SD (i.e. distillation using the synthetic sample and its corresponding client model) should theoretically give the best result, but in the experiment shown in Table 9 in the appendix, FedLMG_SD lags behind FedLMG_MD by a large margin. There is no relevant analysis in this paper.\n3.Privacy assessments that rely on FID thresholds are not rigorous or convincing enough. The FID is not a specific privacy indicator, and the threshold chosen is subjective.", "other_comments_or_suggestions": "In this paper, a reference error occurred in Ablation Experiments in 4.3, that is, Table4.2 should be Table 4.", "questions_for_authors": "1.Assumption 1's \"boundedness\" of KL divergence is overly broad and lacks quantitative validation, leading to a fragile theoretical foundation. If client data distributions differ significantly from the diffusion model's default distribution, FedLMG performance may substantially degrade. \n2.The paper lacks quantitative analysis of the guidance signal effectiveness in image generation. The quality and effectiveness of client model guidance are unevaluated, obscuring FedLMG's working mechanism.\n3.While the client side computation is reduced, server-side data generation can become a bottleneck for large-scale federated scenarios, limiting the actual application scale. The paper lacks a detailed breakdown of server-side computing costs and how they vary with dataset size.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x5WeKNPl7t", "reviewer": "Reviewer_xTsg", "summary": "This paper addresses the important and interesting problem of one-shot federated learning (OSFL), aiming to reduce the communication round of FL to 1. With the help of pretrained Classifier-Guided Diffusion Models, this paper proposes to generate local clients' data distribution in the server side with the guidance of the locally update models. The generated data are further used to train an aggregated global model.\n\n## update after rebuttal\n\nAfter two iteractions with the Authors, my questions has been solved.\n\nI would recommend **Accept** after the necessary revisions, including more accurate explanations to the theretical results and proper citations for the borrowed contents. This paper addresses the interesting and important topic of one-shot federated learning by proposing a effective and reasonable method of utilizing classifier-guided DM. Even though it adds more computational requirements to the server comparing with standard federated learning method, the benefits from single communication round worth the cost. After all, if your computaional ability can not even run a diffusion model, you are not qualified as a server.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your recognition of our work and your valuable feedback. Below, we provide detailed responses to the key concerns you raised:\n\n>**(Essential References Not Discussed)** *\"BN loss has been widely used, for example, [Yin'2020], while the related citations are missing.\"*\n\nThank you for pointing this out. In the revised version of our manuscript, we will incorporate relevant references[1,2,3] to ensure a more comprehensive discussion of BN Loss and its applications.\n\n>**(Generality of BN Loss)** *Weaknesses #1 \"BN loss only works with models with BN layers, while it does not work on others such as transformers or models for other data modalities.\"*\n\nWe refer to the loss function in our study as BN Loss primarily because BN has been widely adopted, as demonstrated in prior works [1,2,3]. However, the core mechanism of BN Loss involves leveraging statistics, such as mean and variance, which are also present in other normalization layers such as Layer Normalization [4]. Notably, Layer Normalization has been extensively used in transformer, including models like CLIP [5]. Therefore, our method is not strictly limited to models with BN layers but can be extended to transformers and other architectures employing various normalization layers.\n\n>**(Further Explanation of Theorem 1)** *Weaknesses #1 \"The explanation of theorem 1 is not enough.\"*\n\n>* *\"Why is $\\mathbb{E}(\\log p\\_{\\epsilon\\_\\theta}(\\boldsymbol{\\theta}\\_k))$ a constant?\"*\n\nThis term is solely dependent on the parameters of the diffusion model, which remain fixed during our method. Consequently, $\\mathbb{E}(\\log p_{\\epsilon_\\theta}(\\boldsymbol{\\theta}\\_k))$ can be considered a constant.\n\n>* *\"What does $p\\_{\\epsilon_\\theta}(\\boldsymbol{\\theta}\\_k)$ represent?\"*\n\nThis term represents the default data distribution of the diffusion model. After extensive pretraining, $p\\_{\\epsilon\\_\\theta}(\\boldsymbol{\\theta}\\_k)$ approximates the data distribution of the diffusion model’s pre-train dataset.\n\n>* *\"Why is that minimizing the negative log-likelihood is equivalent to minimizing the cross-entropy loss?\"*\n\nThe negative log-likelihood and the cross-entropy loss are formally equivalent. When the target distribution is a one-hot distribution, maximizing the likelihood corresponds to minimizing the cross-entropy loss. This equivalence is widely utilized in deep learning, particularly in the loss functions of softmax-based classification tasks[6].\n\n>**(Typos)** *Other Comments Or Suggestions*\n\nWe appreciate your meticulous attention to detail in identifying typographical errors. These will be corrected in the revised manuscript. Additionally, we will conduct a thorough review to ensure the clarity and accuracy of our presentation.\n\nOnce again, thank you for your insightful review and constructive feedback. We look forward to any further comments you may have.\n\n\t[1] Dreaming to distill: Data-free knowledge transfer via deepinversion, CVPR 2020.\n\t[2] Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation? NIPS 2024.\n\t[3] Source-Free Domain Adaptation for Semantic Segmentation, CVPR 2021.\n\t[4] Layer Normalization, NIPS 2016.\n\t[5] Learning transferable visual models from natural language supervision, ICML 2021.\n\t[6] Machine learning: a probabilistic perspective, MIT press 2012.", "claims_and_evidence": "Clear and Convincing.", "methods_and_evaluation_criteria": "Make sense.", "theoretical_claims": "I have checked the proofs.\nHowever, the proof for Theorem 1 is not enough, specifically the explanation of the last two two terms in equation 5. See the questions for authors below.", "experimental_designs_or_analyses": "Experimental designs and analyses totally make sense.", "relation_to_broader_scientific_literature": "The proposed method effectively addressed the one-shot federated learning problem, with experimental results showing significant performance improvements. The proposed method is intersting and totally makes sense, providing remarkable contribution to soloving the one-shot federated learning problem.", "essential_references_not_discussed": "BN loss has been widely used, for example, [Yin'2020], while the related citations are missing.\n\n[Yin'2020] Yin, Hongxu, et al. \"Dreaming to distill: Data-free knowledge transfer via deepinversion.\" CVPR 2020.", "other_strengths_and_weaknesses": "Strength:\n\n- This paper addresses the important and interesting setting, one-shot FL, which restricts communication round of FL to only 1.\n\n- The proposed method generates synthetic images in the server side for aggregated model training, which makes sense considering computational ability.\n\n- Even though this paper does not propose novel techniques, the creative combination of existing techniques to address the important and cutting-edge problem makes this paper interesting.\n\n- Experiments are adequate and solid, especially the datasets are realistic and large-scale. Performance improvements are significant.\n\n- I like the Privacy Issues part, which explains why the proposed method will not recover the original client data using meaningful experimental results, i.e. avoiding raising privacy concerns. It should also be highlighted in the abstract and introduction, avoiding confusion in the beginning.\n\nWeaknesses:\n\n- The explanation of theorem 1 is not enough. Refer to questions to authors below.\n\n- BN loss only works with models with BN layers, while it does not work on others such as transformers or models for other data modalities.", "other_comments_or_suggestions": "Typos.\n\n- Theorem 1: error in right column of line 187.\n- Eq 7, error in s (not bold)\n- Eq 8, $\\hat{\\boldsymbol{s}}_{0,t}$ \n  \n  or \n\n  $\\hat{\\boldsymbol{s}}_{0}$?\n- Right column of line 373, table 4.2?\n- In line 437, it is claimed that the local model training only takes 1 iteration. I guess you mean a single round of optimization.", "questions_for_authors": "Theorem 1: the explanation of the last two terms are not clear enough.\n\n1. Why is $\\mathbb{E}(\\log p_{\\epsilon_\\theta}(\\boldsymbol{\\theta}_k))$ a constant? \n\n    What does $p_{\\epsilon_\\theta}(\\boldsymbol{\\theta}_k)$ represent?\n\n2. Why is that minimizing the negative log-likelihood is equivalent to minimizing the cross-entropy loss?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of One-shot Federated Learning by leveraging Local Model-Guided diffusion models. 

Although the underlying techniques are not new, most of the reviewers think the proposed method is a good combination of existing techniques and the performance is promising. 

Given all the reviews, I recommend weak accept of the paper and I encourage the authors to include the discussions and experiments in the response.
Decision: Accept (poster)