FedDEO: Description-Enhanced One-Shot Federated Learning with Diffusion Models

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 10:49:23

Abstract: In recent years, the attention towards One-Shot Federated Learning (OSFL) has been driven by its capacity to minimize communication. With the development of the diffusion model (DM), several methods employ the DM for OSFL, utilizing model parameters, image features, or textual prompts as mediums to transfer the local client knowledge to the server. However, these mediums often require public datasets or the uniform feature extractor, significantly limiting their practicality. In this paper, we propose FedDEO, a Description-Enhanced One-Shot Federated Learning Method with DMs, offering a novel exploration of utilizing the DM in OSFL. The core idea of our method involves training local descriptions on the clients, serving as the medium to transfer the knowledge of the distributed clients to the server. Firstly, we train local descriptions on the client data to capture the characteristics of client distributions, which are then uploaded to the server. On the server, the descriptions are used as conditions to guide the DM in generating synthetic datasets that comply with the distributions of various clients, enabling the training of the aggregated model. Theoretical analyses and sufficient quantitation and visualization experiments on three large-scale real-world datasets demonstrate that through the training of local descriptions, the server is capable of generating synthetic datasets with high quality and diversity. Consequently, with advantages in communication and privacy protection, the aggregated model outperforms compared FL or diffusion-based OSFL methods and, on some clients, outperforms the performance ceiling of centralized training.

Main paper: 
 introduction: 
 Federated Learning (FL) [22] has attracted increasing attention due to its capability to enable collaborative training across multiple clients while ensuring the protection of user data within local environments. In contrast to centralized training that involves the direct uploading of client data to the server, FL fundamentally operates as a distinctive form of knowledge transfer. In FL, all participating clients transfer knowledge in their local data to the aggregated model without sharing the raw local data. Previous extensive works have aimed to enhance the efficiency of this knowledge transfer, leading to the development of One-Shot Federated Learning (OSFL).In OSFL, clients are tasked with using some mediums to transfer the information about the local distributions to the server within a single communication round. The traditional OSFL methods primarily utilize two kinds of mediums. Firstly, model parameters serve as the primary medium. Parameters of generative models trained on clients are uploaded to the server for generating pseudosamples to train the aggregated model [8, 43] . Additionally, distilled client data [47] can also serve as the medium and be uploaded to the server for the training of the aggregated model. Indeed, the practical deployment of these mediums is challenging due to the difficulty in training generative models and the privacy concerns and communication associated with uploading data.In recent years, the development of diffusion models (DMs) has brought new opportunities for OSFL. There are powerful pretrained DMs that learn vast knowledge from large-scale datasets, allowing them to generate data complying with most of common distributions as long as suitable condition is provided. Moreover, impressively, due to the knowledge from the pre-trained DMs, the trained aggregated model has the potential to outperform the performance ceiling in traditional FL methods, which involves uploading all client data to the server for the centralized training of the aggregated model. This brings significant changes for OSFL. A small amount of descriptions of the client data distribution are sufficient to serve as the medium for transferring client knowledge and guiding the conditional generation on the server. Some works has already begun to explore how to describe the distribution of client data. FedDISC [41] utilizes client image features. FGL [44] uses textual descriptions of the client data generated by BLIPv2 [14] . FedCADO [42] employs local classifiers trained on clients. All these methods have demonstrated the significant potential of DMs in OSFL. But FedDISC requires the sharing of pre-trained feature extractors between the server and clients, limiting its practicality and flexibility. The text descriptions uploaded by FGL may involve more direct privacy risks, and the classifiers uploaded by FedCADO typically only provide relatively vague guidance. We aim to find a more flexible, practical, accurate, and privacyprotecting description of client data distribution to serve as the medium for transferring knowledge from the clients. A natural solution arises: training the local descriptions of the client data and utilizing them for guiding the image generation on the server.Based on this idea, in this paper, we introduce a Description-Enhanced One-shot Federated learning method with diffusion models, FedDEO, where we employ a learnable vector as the description for the client distribution to be the medium of client knowledge. In brief, our method consists of two main components: client description training and server image generation. On the clients, we employ the noise-predicting ability of the DMs to train the descriptions of the client data and embeds personalized distribution information into the descriptions, which are subsequently uploaded to the server. On the server, we utilize the descriptions uploaded by each client to guide the server's pre-trained DM in generating a synthetic dataset that complies with the clients' distributions, and use the synthetic dataset to train the aggregated model. Training the aggregated model based on synthetic datasets means we can flexibly select the model structure. The guidance provided by descriptions trained by pre-trained DMs is more accurate, improving the quality of diffusion generation. These descriptions are merely vectors, which significantly reduces the communication of the clients, enhancing the practicality of the method. Due to the randomness of the diffusion generation process, it is almost impossible to directly or indirectly obtain sensitive client privacy information through the descriptions or the generated samples. The description serves as the medium that perfectly meets our requirements for flexibility, practicality, accuracy, and privacy protection.To demonstrate the validity of FedDEO, on one hand, we conduct thorough theoretical analyses. We prove the boundedness of the Kullback-Leibler (KL) divergence between the distribution of the synthetic data and the distribution of client local data. This provides theoretical assurance for the server's generation of synthetic data that complies with the client distributions. On the other hand, to further validate the performance of our method, we conduct extensive quantitation and visualization experiments on three largescale real-world datasets, DomainNet [25] , OpenImage [13] , and NICO++ [46] . The visualization experiments demonstrate that our method is capable of generating a synthetic dataset that complies with the distribution of each client's data. Moreover, the quality and diversity of the synthetic dataset are comparable with the original client datasets. In our quantitation experiments, we explore scenarios with varying numbers of clients, skews in feature distribution and label distribution among clients, as well as the utilization of different pre-trained DMs. These quantitation experiments also demonstrate that the aggregated model trained by FedDEO outperforms other compared methods, and notably, on NICO++, it outperforms the ceiling performance of traditional FL frameworks that involve uploading all client data to the server. Additionally, we conduct comprehensive quantitation and visualization experiments to demonstrate FedDEO's performance in terms of privacy protection, communication, and computational efficiency. All these findings demonstrate the performance of our method.In summary, the contributions of this paper are as follows:• We propose FedDEO, realizing one-shot federated learning in realistic scenarios and further exploring the potential of utilizing diffusion models in federated learning.  
related work 2.1 diffusion model: 
 The DM is first introduced in [33] . The overall framework of current DMs occurs in [9] . Subsequent sampling methods, including DDIM [34] and PNDM [19] , further improved the quality and efficiency of DMs' generation. [12] and [3] demonstrate excellent generation results on real images. The stable diffusion based on LDM [29] , which pre-trains on large-scale datasets, has ignited a trend in the AIGC (Artificial Intelligence Generated Content). One notable characteristic of stable diffusion is its conditional generation capability. Given suitable conditions as guidance, such as image [31, 37, 38, 45] , text [11, 24, 26, 32] or the graient of loss function [3, 5, 39] , stable diffusion can generate images complying with almost any distribution we encounter in our daily lives. The generated images exhibit remarkable quality and diversity. There are also some methods [6, 7, 23, 30] that fine-tune the input conditions of stable diffusion using specific datasets. These methods enable the conditions to learn the distribution of the used dataset and generate synthetic data that conforms to the specific distribution. Additionally, DMs possess the ability for compositional generation [4, 20] , allowing them to handle scenarios where multiple conditions are simultaneously provided as guidance. These lead us to consider the application of these powerful pre-trained DMs in federated learning. If we can obtain guidance from clients to guide the pre-trained DM on the server, we can generate data that comply with client distributions and address the challenges of OSFL.The high communication cost has been one of the major challenges faced by the classic FedAvg [22] algorithm in federated learning since its proposal. Some efforts [1, 10, 15] have addressed the noniid problem from the perspective of optimizing algorithms to improve communication efficiency. Others [2, 16, 21] have focused on enhancing the practicality of FedAvg in non-iid scenarios from a personalized perspective. However, these efforts still encounter high communication costs. Recently, some works [8, 35, 41, 43, 47] have focused on one-shot federated learning (OSFL), considering federated learning within the constraints of a single communication round. This setting differs from standard federated learning in that OSFL allows clients to train their local models to converge in a single round and then send them to the server for aggregation. The essence of OSFL is that all clients transfer their local knowledge to the server for aggregation through some medium. One category of work uses model parameters as the medium, such as DENSE [43] , which collects classifiers trained by clients and uses them to train a generator on the server, followed by using the generator to produce pseudo-samples for knowledge distillation. FedCVAE [8] involves training conditional-VAE on clients and sending the decoders to the server, where the server generates pseudo-samples through the decoders. Another category of work uses distilled datasets as the medium; DOSFL [47] distills privacy data on clients and sends the distilled data to the server for aggregation. Additionally, recent efforts have combined pretrained DMs with OSFL. FedDISC [41] uses the data features as the medium, leveraging client data features as conditions for a pretrained DM to generate pseudo-samples. Similarly, FGL [44] uses text prompts as the medium, extracting text prompts of the local data using BLIPv2 [14] and sending them to the server as conditions for the DM. FedCADO [42] utilizes classifiers trained on clients as a medium, using the classifier-guided diffusion to generate the synthetic dataset and train the aggregated model. In contrast to these works, we train descriptions on clients as the medium. Compared to FedDISC, FedCADO and FGL, local descriptions can better capture information about the distribution of client local data and provide suitable guidance to the server to generate pseudo-samples that better comply with the client's data distribution. 
method: 
 In this section, we elaborate on our method in two parts, including Client Description Training and Server Image Generation, followed by the theoretical analyses about the distributions of the synthetic data and the client local data. The overall framework of FedDEO is illustrated in Figure 1 and the pseudocode of FedDEO is provided in the supplementary materials.Notations and Objectives. In this paper, we aim to address the standard one-shot federated learning setting. Assuming we have 𝑁 client datasets D 𝑛 , 𝑛 = 1, . . . , 𝑁 , and collectively, these datasets encompass a total of 𝑀 categories. The objective of OSFL is to obtain a global aggregated model w 𝑔 within a single communication round, minimizing the global objective function:EQUATIONwhere 𝑦 ∈ C 𝑛 , and C 𝑛 represents the set of categories owned by each client and is a subset of {1, ..., 𝑀 }. From this objective function, it is evident that our goal is to train an aggregated model that adapts to all client distributions and exhibits excellent classification performance on the data from each client. We also assess the model performance in subsequent experimental sections according to this objective.An Assumption about Diffusion Model. The pre-trained DM is a key component of our method. As stated in the Introduction, our motivation for using pre-trained DM lies in its ability to generate synthetic data that complies with the client distributions. This is because the data used in the pre-training process of these DMs covers almost all common distributions. This motivation implies an assumption: the DMs we use have been sufficiently pre-trained to cover the data distribution of the clients. Therefore, regarding the data distribution 𝑝 𝑛 (x) of the client's local dataset D 𝑛 and the data distribution 𝑝 𝜖 𝜃 (x) that the DMs 𝜖 𝜃 can generate, we can make the following assumption:Assumption 1 There exists 𝜆 > 0 such that the Kullback-Leibler divergence from 𝑝 𝑛 (x) to 𝑝 𝜖 𝜃 (x) is bounded above by 𝜆: It's evident that this assumption is flexible. We don't strictly restrict the DM's distribution 𝑝 𝜖 𝜃 (x) to entirely cover the client distributions 𝑝 𝑛 (x). Instead, we only require some overlap of the distributions, as totally non-overlap distributions is unreasonable. Even if the clients focus on some professional fields, such as medical images, it is entirely feasible to train the specialized DMs on the server. Therefore, this is a completely reasonable assumption made based on a comprehensive consideration of practical scenarios, and it is also the motivation of our method.EQUATIONFirstly, the server distributes the pre-trained DM 𝜖 𝜃 to the clients. Based on 𝜖 𝜃 , the clients train the local descriptions capturing the characteristics of the client's data distribution. We first initialize the local descriptions. Then, we fix the parameters of the pre-trained DMs and train the local descriptions on the client's local data.Description Initialization. For each category 𝑐 within the client 𝑛, we define a vector d 𝑛,𝑐 , 𝑐 ∈ C 𝑛 as the description of the distribution of this category. To ease the training process and consequently lessen the computation on clients, we initialize each description using the text features 𝑓 𝑐 of the name of category 𝑐 extracted by CLIP [28] , which is sent by the server. This initialization ensures that the descriptions possess the capability to guide the DM in generating images of the correct category from the beginning.Image Noising. During training, for each sample x 0 ∈ 𝐷 𝑛 , we start by sampling a random Gaussian noise 𝜖 from the standard Gaussian distribution N (0, I). We randomly sample a timestep 𝑡 from {0, ...,𝑇 }, where 𝑇 is the maximum timestep defined by the pre-trained DM. Based on 𝑡, we adjust the intensity of the sampled Gaussian noise and obtain the noised sample x 𝑡 as follows:EQUATIONwhere 𝑎 𝑡 is the variance schedule defined by the pre-trained DM and changes with the timestep 𝑡.Description Training. After obtaining x 𝑡 , we use the description d 𝑛,𝑐 as a condition for 𝜖 𝜃 to predict the noise 𝜖 within x 𝑡 and employ Mean Squared Error (MSE) Loss to compute the difference between the predicted noise 𝜖 𝜃 (x 𝑡 , 𝑡 |d 𝑛,𝑐 )) and the actual added noise 𝜖. During this process, we fix all parameters of the pre-trained DM 𝝐 𝜃 and solely use the backpropagation to train the description d 𝑛,𝑐 . Therefore, the loss function used during the training of the description is as follows:EQUATIONAfter 𝑆 epochs of training, the description can capture the characteristic of the client distributions and become proficient in guiding the pre-trained DM to effectively denoise the noise-added images. Therefore, during server image generation, with these uploaded descriptions, the DM can denoise the randomly sampled initial noise as x 𝑇 and generates high-quality synthetic images that comply with the client distributions guided by the trained descriptions.Upon receiving the local descriptions {d 𝑛,𝑐 } from client 𝑛, the server utilizes these local descriptions to guide the pre-trained DM in generating samples that comply with the data distribution of the client 𝑛 and trains the aggregated model.Image Generation. Firstly, we sample the random initial noise x 𝑇 from N (0, I) as the start for denoising. Through multiple iterations of the timestep 𝑡 = {𝑇 , ..., 0}, we denoise x 𝑇 to obtain the realistic sample x 0 . Specifically, to further ensure that the generated images possess accurate semantic information for the specified category, we use compositional diffusion using the text feature 𝑓 𝑐 of the specified category 𝑐 and the local description d 𝑛,𝑐 capturing the personalized distribution of category 𝑐 on client 𝑛. At each timestep 𝑡, we employ both 𝑓 𝑐 and d 𝑛,𝑐 as conditions for the compositional DM. These two conditions are separately inputted into the DM for noise prediction. And the predicted noises are accumulated, resulting in the final predicted noise as follows:EQUATIONAfter obtaining ε𝜃 (x 𝑡 |𝑓 𝑐 , d 𝑛,𝑐 ), we denoise the current timestep's sample x 𝑡 to x 𝑡 -1 according to the following formula:EQUATIONwhere 𝛼 𝑡 and 𝜎 𝑡 are pre-defined by the pre-trained DM, and 𝜺 𝑡 is random noise sampled from N (0, I) at each timestep 𝑡. After multiple iterations, the randomly sampled initial noise x 𝑇 is denoised into the realistic image x 0 . We can define x 0 as x𝑛,𝑐 𝑖 and incorporate it into the synthetic dataset { x𝑛,𝑐 𝑖 }, 𝑖 = {1, ..., 𝑅}, where 𝑛 and 𝑐 is the client index and the category. And 𝑅 represents the number of images generated for each category of each client, which is set to 30 in most of our experiments.Aggregated Model Training. After multiple generations, we obtain the synthetic dataset { x𝑛,𝑐 𝑖 }. Each image in { x𝑛,𝑐 𝑖 } has its pseudo-label 𝑐, thus allowing us to directly train the aggregated model w 𝑔 using cross-entropy loss:EQUATIONAccording to the loss function in Eq. 7, we train until convergence to obtain the final aggregated model. A question arises: How is the performance of the aggregated model we trained? Has it learned the local knowledge from the clients? We conduct the subsequent theoretical analysis as well as extensive quantitation and visualization experiments to answer this question.From Eq. 7, it can be seen that the performance of the trained aggregated model is entirely determined by the quality of the synthetic dataset. As mentioned in the Introduction, the performance of the aggregated model trained by FedDEO has the potential to surpass the performance ceiling of centralized training, involving uploading all client local data into the server and train the aggregated model. Therefore, we use the client local datasets to evaluate the quality of the synthetic dataset. The comparison between the distribution of synthetic data and the distribution of client local data is necessary. Based on Assumption 1, we have the following theorem:Theorem 1 For the distribution of client data 𝑝 𝑛 (x) and the conditional distribution 𝑝 𝜖 𝜃 (x|d) of the DM 𝜖 𝜃 conditioned the description d trained on the clients, we have: The proof of Theorem 1 is detailed in the supplementary materials. From it, we can observe that the KL divergence between the conditional distribution 𝑝 𝜖 𝜃 (x|d), which is also the distribution of the synthetic data, and the distribution of client's local data 𝑝 𝑛 (x) is bounded above. Furthermore, we can split this upper bound into three terms: 𝜆, E(log 𝑝 𝜖 𝜃 (d)), and -∫ 𝑝 𝑛 (x) log 𝑝 𝜖 𝜃 (d|x)𝑑x. E(log 𝑝 𝜖 𝜃 (d)) is a constant independent of the sample x. 𝜆 is defined in Assumption 1 and represents the upper bound of the KL divergence between 𝑝 𝑛 (x) and the unconditional distribution of the DM 𝑝 𝜖 𝜃 (x), which means the overlap between the distribution of the client local data and the pre-trained data of the DM. Meanwhile, ∫ 𝑝 𝑛 (x) log 𝑝 𝜖 𝜃 (d|x)𝑑x is the log-likelihood between the description d and the client distribution, representing the information of the client distribution contained by the trained description, which has been maximized during the training process of the descriptions.From the above analysis, we have two conclusions: 1) the quality of the synthetic dataset is related to the used pre-trained DM, which aligns well with the intuition. 2) locally trained descriptions can indeed learn information about the client local distribution and effectively guide the DM to generate high-quality synthetic datasets. These conclusions regarding the quality of the synthetic dataset, and the performance of the aggregated model are further demonstrated in the subsequent experimental section. 
experiments 4.1 experimental settings: 
 Dataset. We conduct experiments on three datasets: Domain-Net [25] , OpenImage [13] and NICO++ [46] . We use DomainNet to simulate style differences within the same category, OpenImage to simulate subcategories' differences within super-categories, and NICO++ to simulate differences in background and specific object attributes. All datasets consist of large-scale real-world images with the resolution of 224x224 pixels. DomainNet comprises six domains: clipart, infograph, painting, quickdraw, real, and sketch. Each domain has 345 categories. Following the partition in FedDISC [41] , we select 20 super-categories of OpenImage with 6 subcategories in each super-category according to the hierarchy of categories provided by OpenImage. NICO++ involves 60 categories, with each category having six common domains shared across categories (autumn, dim, grass, outdoor, rock, water) and six unique domains specific to each category. These two scenarios are respectively referred to as the Unique NICO++ (NICO++_U) and Common NICO++ (NICO++_C) datasets. It is worth noticing that despite each data domain having its own textual description, only the category names are used as textual information for all generations, which is more practical. For more detailed information regarding the dataset, please refer to the supplementary materials.Client Partition. To validate FedDEO under two scenarios of non-IID data distributions: Feature Distribution Skew and Label Distribution Skew, we partition the clients differently. To simulate Feature Distribution Skew, we set up six clients for each dataset. Each client possesses the same set of categories but with completely different data domains. To simulate Label Distribution Skew, we divide the 60 categories of Common NICO++ and Unique NICO++ into six clients. Each client possesses all six data domains of 10 different categories. For all client partitions, there is no overlap between the data of different clients. Due to space limitations, for detailed client partition, including the number of images in each client dataset and experiments related to the total number of clients, please refer to the supplementary materials.Compared Methods. We compare our method with nine other methods, which can be divided into three major categories: 1) Ceiling. The performance ceiling of traditional FL methods is centralized training, involving the uploading of all client local data for the training of the aggregated model. 2) Traditional FL methods with multiple rounds of communications: FedAvg [22] , FedDF [18] , FedProx [15] , FedDyn [1] . All of them have 20 rounds of communications. Following standard experimental settings, each round involves one epoch of training on each client. And we use Ima-geNet as the additional public data for distillation in FedDF. 3) Diffusion-based OSFL methods: FedDISC [41] , FedCADO [42] , FGL [44] and Prompts Only. Although FedDISC is designed for semi-supervised FL scenarios, we remove the pseudo labeling process of FedDISC and directly utilize the true labels of client images. Another point to notice is the Prompts Only, where the server does not use the uploaded local descriptions from clients at all but only uses the text prompts of category names in the server image generation. It's worth noticing that some mentioned OSFL methods such as DENSE [43] and FedCVAE [8] is not used as the compared methods because of the difficulty in training generative models until convergence in the high-resolution realistic image scenarios we selected. For more details about our experimental settings and implementation, please refer to the supplementary materials.In Table 1 and Table 2 , we present the performance of our method under two non-IID scenarios, feature distribution skew and label distribution skew. Several observations can be made:• Our method demonstrates significantly superior performance on all used datasets, surpassing other compared methods and even the performance ceiling on OpenImage and NICO++, demonstrating the potential of utilizing DMs in FL. • The reason for the better performance on OpenImage and NICO++ lies in the fact that these two datasets primarily consist of realistic images, and their distributions are closer to the distributions of the used DM. This also corroborates our theoretical analysis and the better performance on the real domain of DomainNet. The visualization experiments in Figure 2 , Figure 3 , and other visual experiments in the supplementary materials clearly demonstrate that FedDEO is capable of generating synthetic dataset that complies with different client distributions when there are differences in style, subcategory, or background among clients, while being semantically correct. The synthetic datasets exhibit high quality and diversity, which is comparable to the client local datasets, underscoring the superior performance of FedDEO.To further demonstrate the performance of the proposed method, we conduct sufficient ablation experiments, thoroughly discussing the impacts of hyperparameters and other settings in our method, including the number of images in the synthetic dataset, the number of training epochs for the local descriptions, the used pre-trained DMs, the number of the clients, and more. Due to space limitations, we present partial experimental results and discussions here. For more ablation experiments and detailed experimental settings, please refer to the supplementary materials. The Number of Images. The number of images in the synthetic dataset is one of the key factors influencing the performance of our method. The total number of images in the dataset is determined by the number of images generated under the guidance of each local description, defined as 𝑅 in the method section. Due to time limitations, we conduct relevant ablation experiments on the first 90 categories of DomainNet, and the experimental results are presented in Table 3. From the table, we can observe that as 𝑅 increases, there is indeed a noticeable improvement in the performance of the trained aggregated model. Additionally, it is worth noticing that the rate of performance improvement does not significantly slow down when the number of images increases from 30 to 50, proving the diversity of the synthetic dataset.The Epochs for Training Descriptions. The number of training epochs for local descriptions, defined as 𝑆 in the method section, directly influence the characteristics of the client distributions captured by the local descriptions, thereby influencing the quality of the synthetic dataset and the upper bound mentioned in Theorem 1. Due to time limitations, we conduct relevant ablation experiments on the first 90 categories of DomainNet, and the experimental results are presented in Table 4 . From the table, it can be observed that with the increase of 𝑆, the performance of the aggregated model shows a stable improvement. Additionally, when the descriptions are trained for only one epoch, they barely learn the stylistic information of the client distribution, hence only showing good performance on the real domain. This indicates the crucial role of the local descriptions in guiding the server image generation.To more comprehensively demonstrate both the practicality and the limitations of the proposed method, we conduct discussions regarding three issues here: communication costs, computation costs, and the privacy concerns.Communication Costs. The communication costs include both upload communication and download communication. We discuss these two parts separately. In the table. Prompts Only does not involve any communication costs, so it is not included. FGL is also not included in the comparison because the length of the text prompts generated in FGL is highly random. From table 5, It can be observed that FedDEO, with the least upload communication costs, is capable of training the aggregated model with better performance. This indicates the locally trained descriptions can more accurately convey client knowledge and efficiently guide the image generation on the server.Autumn Dim Grass Outdoor Rock Water Unique NICO++ Ship Cargo Ship Civil Military With Flag With Spray With Steam Common NICO++ Horse Autumn Dim Grass Outdoor Rock Water Unique NICO++ Dog Pug Dog Lying Running Tongue Teddy Dog ClothesRegarding the download communication, it is undeniable that FedDEO does not have a significant advantage due to the need for clients to download DM to train local descriptions. However, we must point out that, on one hand, most FL methods based on base models, including diffusion-based OSFL methods such as Fed-DISC, FGL, and various federated fine-tuning methods [27, 36, 40] , require downloading the base model to the clients, resulting in additional download communication. On the other hand, some existing works [17] focus on the use of DM on mobile devices. FedDEO is fully compatible with these methods and can effectively reduce the download communication. Therefore, the download communication of FedDEO is entirely acceptable.Computation Costs. The computation costs include the computation costs on the client and the server. Since FedDEO fixes all parameters of the DM and only trains the local descriptions, the client computation costs of FedDEO is comparable to various federated fine-tuning methods [27, 36, 40] , making it entirely acceptable. Additionally, with the same number of generated images, the computation cost on the server is identical to other diffusionbased OSFL methods. Therefore, although the computation cost poses some limitations, it does not significantly compromise the practicality of our method.Privacy Concerns. Essentially, the local descriptions trained on the clients is a kind of model parameters, which is widely used in FL methods. Additionally, due to the low upload communication of Fed-DEO, where privacy leakage mainly occurs, FedDEO exhibits lower risk compared to other FL methods. To further validate FedDEO's performance in privacy protection, we conducted sufficient quantitative and visual experiments. We select some categories from OpenImage and DomainNet that may contain privacy-sensitive information, such as faces, license plates, books, etc. We train descriptions separately on images of these categories and generate synthetic datasets. The visualization results are shown in Figure 4 . It can be observed that the synthetic datasets only share similar styles and identical semantics with the original client datasets. It is almost impossible to extract specific privacy-sensitive information from the descriptions, which aiming to characterize the overall distribution. Furthermore, we conduct more quantitation and visualization experiments, including overall comparisons between the synthetic dataset and client local datasets, as well as whether encryption of descriptions can be achieved through noise addition to the uploaded descriptions, etc. Due to space limitations, please refer to the supplementary materials for more detailed experimental settings and results about privacy issues. 
conclusions: 
 In  
 
 Appendix:  


Decision: FedDEO is a description-enhanced one-shot federated learning method leveraging diffusion models. The technique uses local descriptions to guide the generation of synthetic datasets, improving model performance, communication efficiency, as well as privacy protection.

Reviewers noted several strengths, including the novel idea of using local descriptions for synthetic data generation, thorough theoretical analysis, and comprehensive experiments. However, concerns were raised about the fairness of experimental comparisons, potential overfitting, and the contribution of the method to the federated learning community.

Addressing these concerns and providing more detailed comparisons and evaluations could significantly enhance the paper's impact.

Reviews: review: paper_summary: paper_summary: This paper proposed a new one-shot federated learning by leveraging the pre-trained diffusion model. It involves a local descriptions
training step to capture the characteristics of client distributions, and a data synthesis step that utilizes the local descriptions in the server. Experiments demonstrate that the proposed description-enhanced generation can promote the model performance significantly.
strength_weakness: strength_weakness: 1. The proposed framework is simple yet effecitive, non-trivil improvement can be obtained compared with other baselines
2. Theoretical and experimental results seem to be promising
limitations: limitations: 1. the data synthesis based approaches highly rely on the assumption that the generative model have been trained on the data with the same or similar semantic meaning, which may not be versatile enough for all kinds of FL tasks.
2. If the pre-trained diffusion models could be utilized, why not compare those methods that leverage the well pre-trained models on large scale data, such as ResNet and ViT with PEFT techniques.
3. Will the style/domain of image a kind of sensitive information of local client? 
4. While the diffusion model could generate natural images/objects, what if the target classification tasks are not natural images/objects
paper_summary: paper_summary: This manuscript proposes FedDEO, a description-enhanced one-shot federated learning (OSFL) method leveraging diffusion models (DMs). This method addresses the limitations of traditional OSFL methods, which often require public datasets or uniform feature extractors, by introducing local descriptions as a new medium for knowledge transfer. These descriptions guide the generation of synthetic datasets that adhere to the clients' data distributions, improving model performance, communication efficiency, and privacy protection. Experiments have been conducted on three datasets.
strength_weakness: strength_weakness: 1. The idea of using local descriptions to guide the generation of synthetic datasets is novel and makes sense.
2. Theoretical analyses have been thoroughly conducted, demonstrating the bounded KL divergence between synthetic and client data distributions. The quality and diversity of the synthetic datasets are comparable with the original client datasets.
3. The manuscript is generally well-written and organized, making the complex methodology understandable. Many experiments and method details are demonstrated in the main text and supplementary materials.
limitations: limitations: 1. The proposed FedDEO outperforms even the performance ceiling of centralized training. Please provide an analysis of whether it is overfitting.
2. Diffusion-based OSFL baseline methods used in this manuscript, such as FedDISC, FGL, and FedCADO, are still in the preprint stage on arXiv and have not been formally published. To enhance the credibility and robustness of the evaluations, the manuscript could benefit from including comparisons with more established and formally published baselines in diffusion-based federated learning. This would provide a more comprehensive and reliable assessment of the proposed method's performance.
paper_summary: paper_summary: In this paper, the authors propose FedDEO, a Description-Enhanced One-Shot Federated Learning Method with DMs, offering a novel exploration of utilizing the DM in OSFL. The core idea of their method involves training local descriptions on the clients, serving as the medium to transfer the knowledge of the distributed clients to the server.
strength_weakness: strength_weakness: The authors applied DM to federated learning and achieved better performance than the centralized training.
limitations: limitations: 1. I think the experimental comparison is unfair. The authors claim that they used a very adequately pre-trained diffusion model, but that would lead to the introduction of very much additional knowledge. It can be assumed that the performance improvement is not due to the federated paradigm proposed by the authors.
2. I think the authors' proposed method benefits from existing methods, such as DM, and does not contribute usefully to the federated learning community.