FedGMKD: An Efficient Prototype Federated Learning Framework through Knowledge Distillation and Discrepancy-Aware Aggregation

Authors: jianqiao  zhang
caifeng  shan
jungong  han
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 10:50:08

Abstract: Federated Learning (FL) faces significant challenges due to data heterogeneity across distributed clients. To address this, we propose FedGMKD, a novel framework that combines knowledge distillation and differential aggregation for efficient prototype-based personalized FL without the need for public datasets or server-side generative models. FedGMKD introduces Cluster Knowledge Fusion, utilizing Gaussian Mixture Models to generate prototype features and soft predictions on the client side, enabling effective knowledge distillation while preserving data privacy. Additionally, we implement a Discrepancy-Aware Aggregation Technique that weights client contributions based on data quality and quantity, enhancing the global model's generalization across diverse client distributions. Theoretical analysis confirms the convergence of FedGMKD. Extensive experiments on benchmark datasets, including SVHN, CIFAR-10, and CIFAR-100, demonstrate that FedGMKD outperforms state-of-the-art methods, significantly improving both local and global accuracy in non-IID data settings.

Main paper: 
 introduction: 
 Federated Learning (FL) is a transformative approach to collaborative machine learning that allows multiple participants to train a shared model while maintaining data privacy by keeping datasets decentralized. This architecture mitigates privacy risks, particularly in sensitive domains such as healthcare and finance [3, 5, 6, 14, 23, 26] . FL enables clients to contribute to the global model without transferring raw data, reducing communication overhead while ensuring privacy. However, the Non-IID (non-independent and identically distributed) nature of client data poses challenges for model convergence and consistency. Heterogeneous data leads to divergence in model updates, slowing convergence and degrading performance across clients [14, 15, 23] .Personalized Federated Learning (pFL) has emerged as a solution to tailor models to individual client data distributions, improving local accuracy while maintaining global robustness [7, 8, 37] . knowledge distillation by introducing the concept of "hyper-knowledge", which involves sending the mean representations of both feature and soft predictions from clients to the server for global aggregation. While FedHKD significantly enhances client-side personalized model performance without relying on public datasets [4] , it focuses mainly on improving the client models. It does not introduce substantial advancements in hyper-knowledge extraction techniques, nor does it achieve dual improvement for both client and server performance in theory, as knowledge distillation predominantly benefits the client-side models. Building upon FedHKD, our research addresses these limitations by not only improving the knowledge distillation process by prototype feature and soft predictions extraction but also benefiting the server-side models by using DAT. This ensures simultaneous improvement in both global model performance and client models. 
problem formulation: 
 In the context of FL, we consider a scenario involving n distinct clients, each holding a private dataset {D 1 , D 2 , . . . , D n }, where each dataset reflects a potentially unique subset of the overall class distributions. This setup illustrates the fundamental Non-IID data challenge in FL, where the heterogeneity of each client's dataset D i complicates the task of learning a generalized global model. Each client optimizes a local model w i by minimizing a local loss function F i , which is typically expressed as the empirical risk over its dataset D i :EQUATIONwhere ℓ denotes the loss function measuring the prediction error on an instance x.To combine these local models into a global model, the FedAvg algorithm, introduced by McMahan et al. [26] , applies a weighted averaging scheme based on the size of each client's dataset:EQUATIONwhere W r represents the parameters of the global model after the r-th aggregation round, w r i denotes the parameters of the i-th client's model in the r-th round, and N is the total number of samples across all clients, N = n i=1 |D i |. Our objective with FedGMKD is to tailor personalized models w i for each client i that not only achieve high accuracy on locally specific data but also enhance a global model W that excels across diverse client distributions. This task becomes particularly challenging in the presence of heterogeneous data, as naive aggregation approaches tend to diminish the effectiveness of the global model. FedHKD [4] has demonstrated significant success in addressing this issue by using hyper-knowledge distillation, which improves both global and personalized models through the exchange of mean representations and soft predictions. Inspired by this approach, FedGMKD enhances model performance on both the client and server sides by introducing Cluster Knowledge Fusion (CKF) based on Gaussian Mixture Models (GMM) for prototype feature and soft predictions generation and Discrepancy-Aware Aggregation Technique (DAT) based on data quality. This dual mechanism not only improves local model performance through better feature representation but also refines the global model by leveraging high-quality aggregated knowledge across clients, thereby effectively addressing the challenges posed by non-IID data distributions in federated learning.In Federated Learning (FL), traditional Knowledge Distillation (KD) techniques often rely on transferring knowledge from a complex central model (teacher) to simpler client models (students). This process typically requires a public dataset to generate soft labels representing the teacher's predictions in FL. Meanwhile, this approach raises significant privacy concerns and faces challenges, especially in Non-IID data environments, where the heterogeneity of client data exacerbates these issues, reducing the effectiveness of traditional KD methods in FL. To address these challenges and inspired by hyper-knowledge of FedHKD [4] , we propose a Clustered Knowledge Fusion (CKF) approach based on Gaussian Mixture Models (GMM). CKF clusters client updates according to data similarity, generating prototype features and soft predictions for each class, which are then aggregated at the server. These aggregated results form a synthetic dataset that serves as the foundation for knowledge distillation, enabling efficient knowledge transfer between the server and clients. Unlike traditional KD approaches, CKF not only eliminates the reliance on public datasets but also leverages client-specific representations, effectively addressing the Non-IID data problem and improving overall FL performance.In the context of federated image classification, each client i processes raw image data x i through a feature extraction function F ϕi , producing a feature representation h i , and a classifier function C ωi which maps h i into a soft prediction vector z i :EQUATIONwhere F θi (•) and C ψi (•) represent the feature extractor and classification function for client i, respectively. The feature vector h i encodes the latent features of the input data, and z i represents the soft prediction vector, indicating the probability distribution over the classes.When the features and soft predictions have been extracted, GMM are employed to cluster client updates based on data similarity prior to aggregation.The responsibility γ m (x j i ) indicates the probability that the data point x j i (which can be a feature vector or a soft prediction vector) belongs to the m-th Gaussian component. It is calculated as:EQUATIONHere, π m represents the mixture coefficient of the m-th Gaussian component, whereM m=1 π m = 1. The term N (x j i ; ¯m, Σ m )denotes the Gaussian probability density function evaluated at x j i , with ¯m being the mean vector and Σ m being the covariance matrix of the m-th Gaussian component. The denominator sums over all M Gaussian components, normalizing the responsibility values so that they sum to 1 across all components. These responsibility values indicate the contribution of each Gaussian component to the data point x j i , and they are later used to compute the prototype features and soft predictions by weighting the means and predictions of each Gaussian component. Using the responsibility values, the prototype features and soft predictions for class j in the local dataset of client i are calculated as follows:EQUATIONwhere ĥj i denotes the prototype feature for class j at client i, synthesized from the cluster knowledge, and qj i represents the prototype soft prediction for class j at client i, calculated based on the responsibility values. Here, ¯mj represents the mean vector of the m-th Gaussian component for class j, and z mj represents the soft prediction vector corresponding to the m-th Gaussian component.CKF is derived by integrating these GMM-based prototype features and their corresponding prototype soft predictions. For class j in the local dataset of client i, CKF is defined by combining the prototype features and their prototype soft prediction:EQUATIONIf there are j classes, then the full CKF of client i is:EQUATIONA flow diagram illustrating the computation of CKF is provided in Supplementary Section A.1.The aggregation phase in FedGMKD amalgamates CKF from each client to construct a global representation of CKF for a given class j at each iteration r + 1. For the given class j, the global CKF represented by the aggregated prototype features H r+1 j and soft predictions Q r+1 j . These are calculated as follows:EQUATIONwhere w ′ i is the weight corresponding to client i's contribution, reflecting both the volume and the quality of the data contributed by client i.In traditional FL approaches such as FedAvg, the server aggregates models by averaging the parameters submitted by each client. This method assumes that data across different clients are identically and independently distributed (IID). However, tin real-world applications, this assumption often fails due to Non-IID data distributions, leading to sub-optimal global models when client data varies significantly in distribution and relevance. Most existing methods primarily weight client contributions based on data volume, neglecting the quality of the data, which can further exacerbate this issue.To address these challenges, FedGMKD introduces a Discrepancy-Aware Aggregation method that evaluates both the volume and the quality of data each client contributes. This is achieved by quantifying how well the local data aligns with the global data distribution. The aggregation process is refined by incorporating a measure of the discrepancy between local model prototype predictions and the aggregated global prototype predictions, thereby enhancing the robustness and accuracy of the federated model.To begin, the initial weights for each client and class are calculated based on the proportion of samples for that class across all clients:EQUATIONwhere N j i is the number of samples of class j in client i's local dataset. This ensures that the initial weight w init i,j reflects the proportion of class j samples that client i contributes to the global dataset for that class.After this, the final aggregation weights are adjusted based on the discrepancies between the local and global data distributions. This discrepancy is quantified using the Kullback-Leibler (KL) divergence and is balanced by the initial weight w init i,j for each category j:EQUATIONwhere w init i,j is the initial weight for class j on client i, reflecting the proportion of class j samples contributed by client i to the global dataset. d j i represents the KL divergence between the client and server distributions for class j, and a and b are adjustment parameters that control the sensitivity of the weight updates based on the discrepancies. The KL divergence d j i between the client i's local distribution and the server's global distribution for class j is calculated as follows:EQUATIONwhere qj i and Qj are the predicted probabilities for class j in the local client i and the global server distributions, respectively. This KL divergence measures the discrepancy between the local and global predictions specifically for class j, allowing the model to adjust the aggregation weights based on this class-specific difference.For each class j, the global CKF is computed by aggregating the local CKF from all clients using the discrepancy-aware weights w ′ i . These global CKF values, represented by H r+1 j and Q r+1 j , encapsulate the collective knowledge from all clients for class j. After calculating the global CKF for each class j, the complete global CKF, denoted as G r+1 , is constructed by taking the union of the global CKF for all classes:EQUATIONThis complete global CKF G r+1 serves as the updated global knowledge representation, which is used to guide future iterations of model updates. A flow diagram illustrating the computation of DAT is provided in Supplementary Section A.2.After the server completes the aggregation, the updated global CKF G r+1 is sent to the clients selected for the next FL round to aid in their local training. For client i, with dataset D i , the local training objective integrates the empirical risk with regularization terms designed to align the local model with the global CKF. The loss function for client i is defined as:EQUATION.where |D i | denotes the number of samples in the dataset owned by client i, ℓ(C ψi , y k ) denotes the cross-entropy loss function, ∥•∥ 2 2 denotes the Euclidean norm, and λ and γ are hyper-parameters. Note that H r+1 y k represents the global prototype feature for class j, and Q r+1 j denotes the global soft predictions for class j at iteration r + 1. The term C ψi (H r+1 j ) represents the local classifier's predictions on the global prototype feature H r+1 j and T is the temperature of KD.The loss function consists of three terms: the empirical risk formed using predictions and ground-truth labels, and two regularization terms that utilize the global CKF. The first term is the empirical risk, represented by the cross-entropy loss function. This term encourages the local model to perform well on its own data. The second term, known as the feature alignment loss, aligns the local feature extractor with the global CKF by minimizing the squared Euclidean distance between the local feature representations F θi (x k ) and the globally aggregated CKF features H r+1 y k . This regularization encourages the local feature extractor to produce similar feature representations to the global prototype features for each corresponding class, improving consistency between the local and global models. The third term, called the knowledge alignment loss, ensures predictive consistency across federated learning by minimizing the discrepancy between the local classifier's predictions on the global prototype features and the global soft predictions for each class j. Specifically, by using Euclidean distance, which is non-negative and convex, these terms effectively regularize the local models to be more consistent with the global CKF.FedGMKD integrates CKF and DAT to enhance both local and global model performance. The framework operates iteratively, as outlined in Algorithm 1. The server initializes the global model W 0 = (F 0 , C 0 ), where F 0 and C 0 denote the parameters of the global feature extractor and classifier, respectively. The global CKF G 0 is also initialized, comprising global prototype features and soft predictions for each class. At each global epoch, the server transmits the global model W r-1 and global CKF G r-1 to the selected clients. Clients update their local models by minimizing a composite loss function comprising three components: (1) the empirical risk, representing the cross-entropy between predicted and ground truth labels; (2) the feature alignment loss, measured by the Euclidean distance between local and global prototype features; and (3) the knowledge alignment loss, measured by the Euclidean distance between local classifier outputs on global prototype features and the global soft predictions. Upon completing local training, clients compute local CKF and transmit these, along with their updated models, back to the server. The server aggregates the received CKF using DAT, which adjusts client contributions based on data volume and quality. The global CKF and model are updated accordingly. This process repeats across multiple federated learning rounds. Server selects i clients for participation.Server broadcasts W r-1 and G r-1 to the selected clients.for each selected client i do6: Client i initializes local model w r-1 i from W r-1 . 7: if r == 1 then 8:Client i updates w r i using Equation 1.Client i computes initial CKF K r i using Equations 3, 4,Client i updates w r i using Equation 13, incorporating G r-1 .12:Client i computes CKF K r i using Equations 3, 4, 5, 6, and 7, and computes divergence d r i for each class using Equation 11. Client i sends w r i , K r i , and d r i (if r > 1) back to the server. Server computes weights for each class of each client using Equation 10.Server updates global CKF G r+1 using Equations 8 and 12 based on the computed weights.Server updates global model W r+1 using weighted averaging of the models.end ifServer sends W r+1 , G r+1 (if r > 1) back to client i. Given the Non-IID nature of data across clients in FedGMKD, we establish two theorems to describe the framework's convergence under well-defined mathematical assumptions.Under Assumptions 1-5 (A.6.1), for any client i, after R global communication rounds, the expected global loss function is bounded as:EQUATIONThe detailed proof is provided in A.6.3.Under Assumptions 1-5 (A.6.1), for any client i, after R global communication rounds, the convergence rate of the global loss function F (W) is bounded as follows:EQUATIONwhere F (W) is the global loss function, F * represents the lower bound of F (W), and C 1 and C 2 are constants that depend on variance σ 2 , Lipschitz constant L, learning rate η, and the number of local steps.The detailed proof is provided in A.6.3. 
datasets: 
 We evaluate FedGMKD on three widely used FL benchmark datasets, selected to cover a range of task complexities, demonstrating the method's robustness and scalability.SVHN [30] : The Street View House Numbers (SVHN) dataset consists of over 600,000 labeled digit images from street scenes. It presents varied imaging conditions, such as lighting and background differences, making it useful for testing under non-IID scenarios. Despite these variations, SVHN is considered relatively simple due to its large dataset size and clear digit images.: CIFAR-10 comprises 60,000 32x32 color images across 10 classes, with 6,000 images per class. It is a standard benchmark for image classification in both centralized and federated learning, providing moderate complexity due to the diverse nature of the images.CIFAR-100 [17] : Similar to CIFAR-10 but with 100 classes, CIFAR-100 contains 60,000 images with 600 images per class. It poses a more challenging task, given the finer granularity and increased class variability, making it especially difficult in federated learning with non-IID data.For our experiments, we adopt the ResNet18 architecture [10] , which has consistently demonstrated superior performance across a wide range of learning tasks, outperforming traditional Convolutional Neural Networks (CNNs). ResNet18 incorporates residual connections to effectively address the vanishing gradient issue, enabling the training of deeper networks while maintaining computational efficiency.To establish a comprehensive comparison, our evaluation includes a diverse set of baselines, encompassing both well-established methods and recent advances in federated learning. These baselines include: FedAvg [26] , the foundational algorithm in federated learning; FedProx [22] , addressing data heterogeneity; MOON [19] , focusing on model personalization; FedMD [18] and FedGen [40] , which utilize public datasets and generative models, respectively, to enhance performance under Non-IID conditions; FedProto [36] and PFL [13] , which employ prototype and clustering learning methods to handle data disparities in federated learning; and FjORD [12] , which introduces an ordered dropout mechanism to enable fair and accurate federated learning under heterogeneous target distributions.The models were implemented and run using PyTorch [31] with 2 NVIDIA A100 GPUs. The Adam optimizer [16] was used for model training in all experiments. The learning rate was initialized to 0.001 and decreased every 10 iterations with a decay factor of 0.5, while the hyper-parameter in Adam was set to 0.5. The number of global communication rounds was set to 50, and the number of local epochs was set to 3. The size of a data batch was set to 64, and the participating rate of clients was set to 1. For all datasets (SVHN [30] , CIFAR-10, and CIFAR-100 [17] ), the latent dimension of data representation was set to 32.Hyper-parameters: For the FedProx [22] algorithm, the hyper-parameter µ prox was set to 0.5. For the MOON [19] algorithm, the proximal term's hyper-parameter µ moon was set to 1. In FedGen [40] , a Multi-Layer Perception (MLP)-based architecture with a hidden dimension of 512 was employed for the generative model. Latent, noise, and input/output dimensions were tailored to each dataset, and the generative model was trained for 5 epochs per global round. The ratio of the generative batch-size to the training batch-size was 0.5 (generative batch-size set to 32). Parameters α generative and β generative were initialized at 10 with a decay factor of 0.98 per global round. FedMD [18] used a regularization hyper-parameter λ md of 0.05, and the public dataset size matched the clients' local training dataset size. FedProto [36] had a regularization parameter λ proto set to 0.05. For FPL [13] , the regularization parameter λ FPL was set to 0.1, with the number of prototypes per class fixed at 10. For FjORD [12] , the unique dropout rate δ was set to 0.5, and the temperature for knowledge distillation T fjord was set to 0.7, following the original paper's settings. Finally, in our proposed FedGMKD, hyper-parameters λ and γ were set to 0.6. The clustering centers were dynamically adjusted between 2-7 based on the data distribution in each category, with the parameters a and b for differential aggregation both set to 0.2, and the temperature T for knowledge distillation was set to 0.6.Table In terms of computational efficiency, FedGMKD incurs a modest increase in training time compared to FedAvg but remains competitive given its substantial accuracy improvements. Compared to Fed-Prox, FedGMKD shows comparable or slightly better training efficiency while maintaining superior accuracy. FedGMKD is also significantly more efficient than FPL, achieving better accuracy with reduced computational overhead. FjORD shows efficiency similar to FedProto but requires more time than FedGMKD on SVHN and CIFAR10. Overall, FedGMKD strikes an optimal balance between accuracy and computational requirements.The additional computational burden of FedGMKD is justified by the substantial gains in local and global accuracy through CKF and DAT. These improvements suggest that FedGMKD offers a highly effective and efficient solution for real-world federated learning scenarios, particularly in the presence of Non-IID data distributions. 
conclusion: 
 This paper presents FedGMKD, a novel federated learning framework that addresses data heterogeneity without requiring public datasets and complex server-side models. Through Cluster Knowledge Fusion (CKF) and Discrepancy-Aware Aggregation Technique (DAT), FedGMKD achieves superior local and global accuracy across various benchmark datasets, outperforming methods like FedAvg, FedProto, and FPL. Meanwhile, theoretical convergence guarantees and experimental results validate its effectiveness. While FedGMKD introduces moderate computational overhead, the accuracy gains justify this cost. Future work will focus on improving computational efficiency and scalability to better handle large, complex datasets, enhancing its applicability to broader federated learning scenarios. 
 
 Appendix: A As shown in Figure 2 , the computation of DAT begins by calculating an initial weight for each client's CKF. Each client's initial weight w init i,j for class j is derived based on the proportion of samples for that class relative to the total samples across all clients, reflecting each client's contribution in terms of data volume. This initial aggregation step combines CKF from all clients to generate preliminary The flow diagram in Figure 3 illustrates the iterative process of CKF computation and model aggregation DAT within the FedGMKD framework. This visualization helps clarify the operational phases of training local models, extracting CKF, and DAT aggregating updates at the server, leading to a refined global model.In Figure 3 , the FedGMKD process is visualized as follows: During Global Round 1, clients (e.g., Client i and Client n) train their local models using their private data and extract CKF from these models. The local CKF and model updates are then sent to the server. Upon receiving updates from all participating clients, the server employs the DAT to aggregate the local CKF and model updates, refining both the global CKF and the global model.In subsequent rounds (e.g., Global Round 2 through Global Round T ), the server broadcasts the updated global CKF and global model to the clients. Each client integrates the global CKF into its local training process, updating its model based on both the global CKF and its own local data. After training, the clients extract new CKF values from their updated models and send these back to the server for further DAT aggregation. This iterative process continues across multiple global rounds, progressively refining both the global model and CKF to account for the diversity and heterogeneity of client data, thereby improving the personalization and generalization of the federated learning model.By leveraging CKF at each iteration, the FedGMKD framework ensures that the global model retains a balanced understanding of the local client data distributions while adapting to the variations in the feature and label spaces across different clients. This process addresses the challenges posed by non-IID data, leading to a more robust and well-personalized federated learning system. To evaluate the robustness of FedGMKD in addressing data heterogeneity, we conducted experiments on the CIFAR-10 and SVHN datasets under varying degrees of Non-IID conditions, controlled by the Dirichlet distribution parameter β. A smaller β indicates greater heterogeneity in the data distribution, with clients typically possessing only one or two classes in their local datasets; a larger β suggests a more homogeneous distribution of data across clients. Specifically, when β = 0.2, the local datasets exhibit high heterogeneity, whereas β = 5 produces nearly homogeneous datasets.For these experiments, we utilized datasets from Torchvision, where the global test dataset was directly obtained by loading the standard test sets for CIFAR-10 and SVHN without any sampling. To create the local datasets, we partitioned each dataset into n clients (where n denotes the number of clients) using a Dirichlet distribution with parameter β. Each client's dataset was subsequently divided into a local training set and a local test set in a 75%/25% split. The results of these experiments, which compare the performance of various methods under β = 0.2 and β = 5 settings for both CIFAR-10 and SVHN, are summarized in Table 2 .The results presented in Table 2 demonstrate that FedGMKD consistently surpasses other federated learning techniques, especially under challenging heterogeneous data settings. Under the high heterogeneity scenario (β = 0.2), FedGMKD exhibits notable improvements. On CIFAR-10, it achieves the highest local accuracy of 74.34%, representing an increase of 3.4% over FedAvg, 1.93% over FedProto, and 2.26% over FPL. Moreover, FedGMKD attains the best global accuracy of 44.79%, outperforming FedProto and FPL by 7.43% and 2.72%, respectively. These significant improvements emphasize the ability of CKF and DAT to handle highly heterogeneous client data while capturing essential features. When the data is more homogeneous (β = 5), FedGMKD maintains its superior performance. For instance, it achieves the highest global accuracy of 53.84% on CIFAR-10, outperforming FedAvg by 1.12% and FPL by 0.19%, showcasing its effective model alignment even in near-homogeneous settings.On the SVHN dataset, FedGMKD also demonstrates its robustness across varying heterogeneity levels. Under the high heterogeneity setting (β = 0.2), it achieves a local accuracy of 87.29%, outperforming FedAvg by 1.33% and FedProto by 0.71%. In terms of global accuracy, FedGMKD These results underscore the effectiveness of FedGMKD in addressing both extreme heterogeneity and more homogeneous environments. Its ability to consistently improve both local and global accuracy across different datasets highlights the robustness of the CKF and DAT techniques. By leveraging these mechanisms, FedGMKD captures high-quality local model representations while optimizing the aggregation process, ensuring better personalization and improved global performance in federated learning.In this section, we perform an ablation study to assess the contributions of the CKF and DAT within our FedGMKD framework. We compare the performance of FedGMKD with its variants-FedGMKD using only CKF and FedGMKD using only DAT-and a baseline method, FedAvg, on CIFAR-10 and SVHN datasets under different data heterogeneity settings, controlled by the Dirichlet distribution parameter β. The results are presented in Table 3 . While ResNet-50 offers higher theoretical representational power, practical considerations such as communication efficiency and the number of training rounds needed to achieve convergence limit its effectiveness in federated learning. FedGMKD with ResNet-50 still outperforms other schemes, demonstrating the robustness of our framework. However, the results suggest that balancing model complexity with communication and computational constraints is crucial in federated settings. Future work may explore strategies to mitigate these challenges, such as adaptive aggregation techniques and personalized model training strategies, to further improve the performance of deeper models like ResNet-50 in federated learning.This section explores the effect of varying the regularization coefficients λ and γ within the FedGMKD framework on the CIFAR-10 dataset with 10 clients over 50 epochs. Table 5 summarizes the local and global accuracy for different combinations of λ and γ, while also providing baseline comparisons with FedAvg, FedProto, and FPL. FedGMKD consistently outperforms the baseline methods FedAvg, FedProto, and FPL across all configurations, particularly at the optimal setting (λ = 0.06, γ = 0.06), where it achieves a local accuracy improvement of 6.03% and a global accuracy improvement of 3.36% over FedAvg. This indicates that FedGMKD's CKF and DAT mechanisms effectively handle Non-IID data and improve both local and global model performance.The sensitivity analysis indicates that while FedGMKD performs well across different settings, finetuning λ and γ is crucial for maximizing performance. The robustness of FedGMKD is highlighted by its stable accuracy across various configurations, though the results suggest that balancing the feature alignment (λ) and prediction alignment (γ) terms is key to optimizing performance. This balance allows FedGMKD to maintain consistency between local and global models and handle diverse data distributions effectively.In this section, we present a comparison between an approach that utilizes the hyper-knowledge concept from FedHKD-specifically averaging features and soft predictions-and our proposed FedGMKD framework. Both methods incorporate the Discrepancy-Aware Aggregation Technique (DAT) across CIFAR-10, SVHN, and CIFAR-100 datasets with varying numbers of clients. Table 6 summarizes the local and global accuracy results for both methods, demonstrating the differences in their effectiveness when applied to non-IID federated learning scenarios.As shown in Table 6 , FedGMKD consistently outperforms the method based on hyper-knowledge averaging with DAT across all datasets and client configurations. The GMM-based approach in FedGMKD allows for the extraction of more precise prototype features and soft predictions by clustering data points according to their similarity. This clustering technique captures the inherent structure and diversity within the data better than the simple averaging of features and soft predictions derived from hyper-knowledge. By leveraging these GMM-derived prototypes, FedGMKD provides a richer and more representative set of features for aggregation, leading to superior performance in both local and global metrics. This alignment ensures that the global model benefits from a more accurate and relevant integration of client data, particularly in highly heterogeneous settings. The close performance in global accuracy between the two methods in some configurations shows that DAT effectively mitigates client discrepancies; however, the advantage of using GMMs for prototype generation becomes evident as FedGMKD achieves consistently higher global accuracy, demonstrating the effectiveness of its integrated CKF and DAT mechanisms.To further assess the adaptability of FedGMKD across different data modalities, we conducted experiments on the IMDB (Internet Movie Database) dataset, a prominent benchmark in natural language processing (NLP) used for sentiment analysis tasks. The IMDB dataset comprises a large collection of movie reviews, presenting a significant challenge for federated learning due to the inherent diversity of client data distributions. Sentiment analysis in particular requires models to generalize effectively across these distributed datasets.The experimental setup mirrored the structure of previous evaluations, utilizing 10 In particular, the results highlight the effectiveness of CKF and DAT in adapting to text-based tasks. While the average computation time for FedGMKD is slightly higher due to the complexity of BERT, the gains in both local and global accuracy justify the computational overhead. These findings underscore the versatility of FedGMKD, demonstrating that it can generalize well to NLP tasks and effectively handle different data modalities, further validating its broader applicability across federated learning scenarios.In this section, we present an ablation study comparing the t-SNE visualizations of feature representations obtained using different federated learning methods: FedAvg, FedProto, FPL, and FedGMKD. These experiments were conducted with 10 clients and a data heterogeneity parameter (β) of 0.5 on the CIFAR-10 dataset. The primary aim is to evaluate how effectively each method learns to separate feature representations by class, as visualized using t-SNE. In the FedAvg method (Figure 4a ), feature representations are widely dispersed with significant overlap between classes. Although FedAvg captures some class-specific features, the resulting feature clusters are neither distinct nor compact, indicating weak separation between the classes. This is expected, as FedAvg primarily averages model parameters without specific focus on class differentiation or feature compactness.The FedProto method (Figure 4b ), while not explicitly clustering features, introduces prototypebased learning where class prototypes guide the learning process. The t-SNE visualization shows slightly more distinct feature distributions compared to FedAvg, reflecting an improvement in feature separation. However, since FedProto emphasizes prototyping over clustering, some overlap between class features still persists, suggesting room for more effective inter-class separation.FPL (Figure 4c ) shows clustering behavior very similar to FedAvg, with large overlaps between clusters and less distinct boundaries. The feature separation is minimal, and the clusters are not as compact as those in FedProto. This indicates that FPL struggles to produce clear, discriminative class-specific features, making its clustering quality less ideal compared to FedProto.FedGMKD (Figure 4d ) demonstrates the most compact and well-separated feature representations. The t-SNE visualization shows that FedGMKD's CKF successfully generates class-specific prototypes that contribute to tight and distinct feature clusters. This clear separation highlights how CKF, along with the DAT, enhances feature learning across clients, leading to superior class separation and making FedGMKD the most effective method in terms of feature representation quality.To prove the convergence of FedGMKD, we introduce the following assumptions:Assumption 1 (Lipschitz Continuity of the Local Loss Function): The local loss function L(D i , w r i ) at each client i is L-smooth, meaning that its gradient is Lipschitz continuous with a constant L > 0. Formally, for any w a i , w b i , we have:EQUATION) This ensures that the local loss function is smooth, and the gradient changes gradually for any two parameter vectors.Assumption 2 (Unbiased Gradient and Bounded Variance): The gradient estimation at each client i is unbiased with bounded variance. Specifically, for any w r i , the expectation of the local gradient equals the true global gradient at the global parameter w R , and the variance of the gradient is bounded by a constant σ 2 :EQUATIONEQUATION)Assumption 3 (Bounded Gradient Norm): The gradient of the local loss function L(D i , w r i ) is bounded. There exists a constant G > 0 such that:EQUATIONAssumption 4 (Bounded Discrepancy in KL Divergence): The Kullback-Leibler (KL) divergence between the local model's predictions and the global model's predictions for each class is bounded. For any client i and class j, the KL divergence d j i between the local soft prediction qj i and the global soft prediction Qj satisfies:EQUATION) This ensures that the discrepancy between local and global models is controlled.The Gaussian Mixture Model (GMM)based clustering used in Cluster Knowledge Fusion (CKF) converges to a stable clustering of client updates. Let M be the number of Gaussian components (to avoid conflict with global round notation), and let γ m (x j i ) represent the responsibility of the m-th Gaussian component for a data point x j i . After a sufficient number of iterations, we assume:EQUATIONA.6.2 Lemma Lemma 1Lemma 1: Under Assumptions 1-5 A.6.1, after R local training epochs at global round r + 1, the local loss function can be bounded as follows:EQUATION)where L is the Lipschitz constant from Assumption 1, σ 2 is the variance bound from Assumption 2, G is the bound on the gradient norm from Assumption 3, and η is the learning rate.In each communication round r, client i performs R local updates based on stochastic gradient descent (SGD) on its local dataset D i . The local training objective for client i is defined as: ( 23)EQUATION.First, we use the Lipschitz continuity of the gradient (Assumption 1). This assumption implies that the loss function L(D i , w) is L-smooth. Formally:EQUATION)We set w 1 = w r+j+1 i and w 2 = w r+j i . Substituting these into the Lipschitz continuity condition:EQUATIONThe SGD update rule is given by:EQUATIONwhere η is the learning rate. Substituting this into the previous equation:EQUATION)To analyze the expected change in the loss, we take the expectation on both sides with respect to the randomness of the gradient:EQUATION)According to Assumption 2 (Unbiased Gradient and Bounded Variance), the gradient is an unbiased estimator of the true gradient, and its variance is bounded:EQUATION)We decompose the expected squared gradient norm as follows:EQUATIONThus:EQUATION)To understand the cumulative effect of R local updates, we sum the inequalities over j = 0 to j = R -1:EQUATIONFrom Assumption 3 (Bounded Gradient Norm), ∥∇L(D i , w r+j i )∥≤ G for all j. Substituting this bound:EQUATIONSubstituting this result into the inequality:EQUATION)Lemma 2Lemma 2: Under Assumptions 1-5 A.6.1, the global loss function after one global aggregation step at round r + 1 can be bounded as:EQUATIONwhere F (W) is the global loss function, F i (w i ) is the local loss function for client i, w ′ i are the discrepancy-aware weights, and σ 2 is the variance bound from Assumption 2.The global model after aggregation at round r + 1 is computed as:EQUATIONwhere w ′ i are the discrepancy-aware weights based on each client's data quality and quantity, and w r+R i represents the model parameters after R local updates for client i.We express the global loss function at time r + 1 in terms of the aggregated client models:EQUATIONwhere F i (w r+R i ) is the local loss function value for client i after R local updates.From Lemma 1, we have:EQUATIONwhere ρ represents the cosine similarity between the gradients and L, σ 2 , and G are constants derived from the assumptions.Taking the expectation over all clients and substituting the bound from Lemma 1:EQUATION)Substituting the inequality from Lemma 1 into this expression:EQUATION)Recall that the global loss function at time r is:EQUATIONSubstituting this into the inequality:EQUATION)According to Assumption 2 (Unbiased Gradient and Bounded Variance), we can further refine the variance term:EQUATIONsince the weights w ′ i sum to 1. Therefore:EQUATION)The factor ρ represents the alignment of the gradient directions across clients. In practice, ρ can vary, but for worst-case analysis, we set ρ = 1 to provide a conservative bound. Substituting ρ = 1:EQUATION)We rearrange the expression to factor out the learning rate η and the Lipschitz constant L:EQUATIONA.6.3 Theorem Theorem 1: FedGMKD Convergence Under Assumptions 1-5 A.6.1, for any client i, after R global communication rounds, the expected global loss function is bounded as:EQUATION)where F (W) is the global loss function, F * is the lower bound of the global objective function, η is the learning rate, R is the number of global updates, L is the Lipschitz constant from Assumption 1, σ 2 is the variance bound on the gradient from Assumption 2, and G is the upper bound on the gradient norm from Assumption 3.As R → ∞, the global loss function converges to a neighborhood around a stationary point, with the size of the neighborhood bounded by:EQUATION)Proof:The global model update after aggregation at round r + 1 is defined as:EQUATIONwhere w ′ i are the discrepancy-aware weights, and w r+R i are the parameters of client i after R local updates.The global loss function after aggregation is:EQUATIONwhere F i (w r+R i ) denotes the local loss function of client i after R local updates.From Lemma 2, we know:EQUATION)Let ∆ r denote the change in the global loss function:EQUATIONSubstituting the result from Lemma 2:EQUATION)This inequality indicates the amount by which the global loss function decreases in expectation after one round of global aggregation.We sum the inequality ∆ r ≥ . . . over r = 1 to R:EQUATIONSubstituting the inequality for ∆ r :EQUATION)Rearrange the inequality to isolate the sum of the expected gradient norms:EQUATION)To find the average over R rounds, divide both sides by ηR 1 -Lη 2 R:EQUATION)Assuming that F (W) is bounded below by F * , such that:EQUATIONwe substitute this bound into the inequality:EQUATION)As R → ∞, the term F (W 1 )-F * ηR(1-Lη 2 )R tends to zero. Therefore:EQUATION)Since η is chosen such that 1 -Lη 2 > 0, this shows that the global loss function converges to a neighborhood around a stationary point with the size of the neighborhood given by:EQUATION)As R → ∞, the term C1 R → 0. This indicates that the average squared gradient norm converges to C 2 .Given that C 2 is bounded, the behavior of F (W) over R rounds implies:EQUATION)This shows that the global loss function F (W) converges at a rate proportional to 1 R , plus an asymptotic constant C 2 that defines the neighborhood around the stationary point.As R → ∞, the term C1 R diminishes, and the global model converges to a neighborhood around the stationary point defined by C 2 .Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [NA] Justification: This research does not involve human subjects. 


Decision: The paper addresses the issue of data heterogeneity in FL with an eye toward robustness, by introducing  Cluster Knowledge Fusion and Differential Aggregation Techniques.  With the exception of one low score, the other three are generally in agreement and positive about the novelty of the approach.

Reviews: review: paper_summary: paper_summary: The authors proposed FedGMKD, a federated learning framework designed to handle data heterogeneity across distributed clients. FedGMKD introduces Cluster Knowledge Fusion (CKF), which uses Gaussian Mixture Model (GMM) clustering to generate prototype features and soft predictions, facilitating knowledge distillation without public datasets. Additionally, the Differential Aggregation Technique (DAT) tailors the aggregation process to distinct feature distributions, optimizing both efficiency and performance. Experiments demonstrate that FedGMKD significantly improves both personalized and global model performance, achieving state-of-the-art results in heterogeneous data scenarios compared to traditional federated learning methods.
strength_weakness: strength_weakness: •	While the evaluation demonstrates the potential of FedGMKD, it would be more convincing if it included a broader range of neural network architectures, particularly larger and more complex models like Transformers or ResNet-50.
•	The paper does not provide sufficient details on how FedGMKD could be adapted to datasets with different characteristics from those tested (SVHN, CIFAR-10, CIFAR-100).
•	Overall, I’d like to see a discussion about what could be the good impacts/applications based on the technique proposed in the real world.
questions: questions: I’d like the authors to answer my above three questions in the rebuttal.
limitations: limitations: The authors discussed the limitations in section 4.
paper_summary: paper_summary: This paper introduces FedGMKD to tackle data heterogeneity by CKF and DAT. Specifically, to get a prototype of each class at each client, CKF is proposed by GMM and aggregates prototypes of the same class from different clients via discrepancy-aware weight.
strength_weakness: strength_weakness: - This paper lacks the motivation and insights to present this method. I can’t catch the idea of FedGMKD tackling the stragglers which is a problem in pFL presented by authors.
- The contribution of this paper is limited.
- The organization, writing, and presentation should be improved in this paper to illustrate the challenges and ideas more clearly. The quality of the figures should be improved.
- This paper lacks some necessary citations in the convergence analysis part.
- The computation and communication overhead should be discussed especially with the prototype-based FL methods like FedProto.
questions: questions: - How to calculate the  Discrepancy if some clients lack some class, e.g., Client 1 has no prototype of class 1. This is also important in a real FL setting.
- The detailed FL setting is missing. ‘’The participating rate of clients was set to 1“ is not real in the FL setting.
limitations: limitations: See weakness and questions.
paper_summary: paper_summary: The authors introduce a novel federated learning algorithm aimed at addressing the challenge of data heterogeneity among distributed clients. The key innovation of this work is the integration of Cluster Knowledge Fusion (CKF) and the Differential Aggregation Technique (DAT). CKF employs Gaussian Mixture Model (GMM) clustering to generate prototype features and soft predictions, facilitating effective local model training without the reliance on public datasets. DAT further refines the aggregation process by accounting for the distinct feature distributions across categories, thereby enhancing both efficiency and performance. Comprehensive experiments on benchmark datasets (SVHN, CIFAR-10, and CIFAR-100) reveal that FedGMKD delivers superior results in both personalized and global model accuracy compared to traditional federated learning methods. The theoretical analysis included in the paper substantiates the convergence and robustness of the proposed framework.
strength_weakness: strength_weakness: 1、The experiments conducted in the paper primarily use a narrow range of hyperparameter settings. While the results are strong, exploring a wider range of hyperparameters, such as the α and β in local loss function, would provide a more comprehensive understanding of the framework's robustness and sensitivity. 
2、While FedGMKD uses Gaussian Mixture Models (GMM) for obtaining prototype features, the paper does not provide sufficient justification for choosing GMM over other clustering methods.
questions: questions: 1、Can the proposed technique generalize to a wider range of hyperparameter settings, such as α and β in local loss function, to assess the robustness and sensitivity of FedGMKD?
2、Can the authors elaborate the reasons you choose Gaussian Mixture Models (GMM) for obtaining prototype features over other clustering methods?
limitations: limitations: The authors have discussed the limitations in the section of conclusion.
paper_summary: paper_summary: FedGMKD addresses data heterogeneity using a dual-enhancement approach through Cluster Knowledge Fusion (CKF) and Differential Aggregation Technique (DAT). This method effectively enhances both local and global model performance without relying on public datasets or complex server-side models.
strength_weakness: strength_weakness: The motivation behind using Knowledge Distillation (KD) is to tackle both data and model heterogeneity. However, this paper falls short by not comparing its performance against architectures like FjORD [1], which handle model heterogeneity effectively. Moreover, restricting comparisons to vision datasets alone limits the broader applicability and merit of this work.

[1] Horvath et al., "FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout," 35th Conference on Neural Information Processing Systems (NeurIPS’21).
questions: questions: 1. See weaknesses
2. line 129 space is missing jin
3. The assumption should be inside the main paper for the convergence analysis.
limitations: limitations: The paper addressed limitations.

Comments: title: Response to the Rebuttal
comment: I think these two concepts may be mixed in FL. These papers also can be recognized to sample data from Dirichlet distribution, e.g., a $\alpha$ to control the heterogeneity degree, even use LDA.
In the context of Federated Learning (FL), both the Dirichlet distribution and Latent Dirichlet Allocation (LDA) can be used to model data heterogeneity across clients, but they serve different purposes and are recognized differently within the field.

### Dirichlet Distribution in Federated Learning
The Dirichlet distribution is widely recognized and commonly used in Federated Learning to simulate non-IID (non-Independent and Identically Distributed) data among clients. It is frequently used to generate client-specific data distributions, where the parameter \$\alpha\$ controls the degree of heterogeneity. A smaller \$\alpha\$ value results in more skewed distributions, where each client’s data is concentrated on a few classes, leading to higher heterogeneity. This approach is particularly useful in benchmarking and evaluating the robustness of federated learning algorithms under varying degrees of data heterogeneity.

One widely cited work utilizing the Dirichlet distribution for this purpose is McMahan et al.'s "Communication-Efficient Learning of Deep Networks from Decentralized Data" (2017), which introduced the concept of Federated Averaging (FedAvg). In this work, the authors used the Dirichlet distribution to simulate non-IID data distributions across clients, which has since become a standard practice in FL research.

### Latent Dirichlet Allocation (LDA) in Federated Learning
LDA, although related to the Dirichlet distribution, is less commonly used in Federated Learning. LDA is a generative model primarily used for topic modeling in natural language processing. It assumes that each document (or client data in the context of FL) is a mixture of several topics, and these topics are distributed according to a Dirichlet distribution. While LDA can theoretically be applied to FL to model complex data distributions across clients, it is not as widely adopted because of its complexity and the fact that it is more suited to scenarios where topics or latent structures are being modeled, rather than straightforward class distributions.

### Recognition and Use in FL
In the field of Federated Learning, the Dirichlet distribution is more recognized and widely used than LDA. Researchers and practitioners often prefer the Dirichlet distribution for its ease of use and direct applicability to generating client-specific data distributions with controlled heterogeneity. This simplicity allows for more straightforward comparisons across different FL algorithms and facilitates reproducibility in experiments.

In contrast, LDA’s complexity and its focus on modeling latent structures make it less suited for the typical data partitioning tasks encountered in Federated Learning. While LDA could provide a more nuanced understanding of client data in some scenarios, it is generally not the standard approach in FL research.

In summary, while both Dirichlet distribution and LDA have their places in probabilistic modeling, the Dirichlet distribution is more widely recognized and utilized in Federated Learning due to its practicality and direct application in modeling data heterogeneity across clients.

References:
- McMahan et al., "Communication-Efficient Learning of Deep Networks from Decentralized Data," 2017.
- Zhao et al., "Federated Learning with Non-IID Data," 2018.
What's the difference between LDA and Dirichlet distribution?
Thank you for your detailed feedback. I would like to clarify the distinction between Latent Dirichlet Allocation (LDA) and the Dirichlet distribution in the context of federated learning (FL). The Dirichlet distribution is widely recognized and commonly used in FL to generate non-IID data across clients, simulating data heterogeneity, which is critical in FL studies. LDA, on the other hand, is not a standard approach for this purpose. Regarding the cited papers, there seems to be some misunderstanding: for example, "FedFA" employs a Dirichlet distribution (Dir(α=0.1)) rather than LDA. Additionally, only three of the seven papers you previously mentioned actually use LDA distributions, indicating that LDA is not universally adopted for testing data heterogeneity in FL research.

Due to objective constraints, we were unable to compare our method with the papers you mentioned under identical experimental settings. It is important to note that the specific methods you highlighted were only introduced on the final day of our discussion, despite multiple rounds of review and rebuttal where these references were not mentioned. Additionally, most of these methods do not have official code repositories, making it difficult to accurately reproduce their results within the limited timeframe.

Not only did we not make these comparisons, but the seven papers you mentioned also did not compare performance under the real-world scenarios you are concerned with. As your descriptions indicate, the experimental settings across these papers vary significantly, further proving that there is no universally accepted experimental setup in this field. While you emphasize that their settings are more challenging, this conclusion seems to be based solely on the use of LDA in some cases. Out of the seven papers, only two really used random client selection at a rate of 0.1, and one of these focused on comparing the number of epochs rather than accuracy. Furthermore, you have completely overlooked my previous point regarding the significant impact of parameters such as the number of epochs and local iterations on experimental outcomes, which is clearly demonstrated in our results. Therefore, I do not agree with your assertion that their experiments are more challenging simply due to the occasional use of LDA, as this claim lacks sufficient evidence.

Additionally, I’m unclear as to why we should compare our approach specifically with the algorithms you mentioned. Is it solely because they also address data heterogeneity? The methods we chose to compare against, apart from the baseline FedAvg, are also well-recognized solutions to data heterogeneity, published in top-tier conferences and widely accepted in the field. For instance, FedProto and FPL are notable examples, with FPL being a 2023 CVPR paper that, importantly, also employs clustering techniques. Wouldn't this make our comparisons even more meaningful?

You also frequently emphasize that our method shows lower accuracy than random initialization, which I assume is based on the expectation of 10% accuracy for a ten-class problem. However, this 10% is only applicable in scenarios where the dataset is infinitely large and uniformly distributed. In our case, during the initial round, the accuracy was only around 4% because only 0.1 of the clients participated, and the data was highly heterogeneous. This context is critical to understanding the results and accurately interpreting the effectiveness of our approach.

Lastly, I find it puzzling that, without direct comparisons, there is an assumption that our method performs worse than the algorithms you mentioned. As I previously explained, due to various constraints, we were unable to conduct experiments to definitively prove that our approach is superior. However, how can you be certain of this conclusion? In fact, a straightforward comparison shows that, in scenarios where all clients participate, our method outperforms FedFA on the CIFAR10 dataset, achieving at least 61.78% accuracy compared to FedFA’s 60.40%, and with fewer epochs. This indicates that our approach is not only competitive but potentially more efficient. Moreover, I would be interested to know if there are any theoretical grounds that could substantiate the claim that our method is inherently less effective than those you mentioned. From our perspective, the results do not suggest such a conclusion.
At first, you claim this paper is focusing on heterogeneity, which has also been tested by many works using LDA distribution. This is very relevant to your work. 

Secondly
1. **FedFA: Federated Learning With Feature Anchors to Align Features and Classifiers for Heterogeneous Data**\
This paper does not use an LDA distribution at all.(we set $\alpha$ of Dirichlet distribution Dir($\alpha$) as 0.1 at page 8.)
2.  **Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning**\
This paper **does not use LDA distribution** (this paper claim that use LDA at page 6 "Different Non-IID Partition Methods") and **does not address the issue of randomly selecting 100 clients** ( They random select 10 clients 0.1 participation at page 22"the server randomly selects 10 clients out of 100 clients each round" ). Instead, it conducts experiments with 10 and 100 clients separately, without employing the random client selection method we used. " 
3.  **Federated Learning with Data-Agnostic Distribution Fusion**\
This paper uses 50 fixed clients ( Fig. 8 compares the test accuracy of the global model for a different numbers of involved clients. When the number of clients increases from 20 to 100, the accuracy of FedFusion decreases much slower than that of the baselines at page 8)
4. **FedFed: Feature Distillation against Data Heterogeneity in Federated Learning**\
but it discusses client numbers of 20 and 100 without addressing the random selection of 0.1 of the clients per round. (server samples a subset of clients at page 18.)

Lastly, I'm not asking to compare these methods in different settings. You absolutely should compare under the same setting. I am only concerned about the true performance under a real scenario, e.g., cross-devices that seldom clients can be online all the time.  Because many works test and get pretty good performance in a worse setting than your work which just gets lower ACC than random initialization. I just want to justify what is the performance of this paper and other methods in a real scenario. e.g. FedGMKD can get the same performance using fewer communication rounds or can get better performance with the same communications with these methods. What's the ability boundary of the FL method is very important to practical application.
Thank you for your feedback. I would like to start by highlighting that the issue of data distribution heterogeneity, particularly through the use of LDA distributions, is a new concern raised in your latest review. This aspect was not previously discussed in earlier feedback, making it an important point to address. Additionally, while you provided several references to support your critique, it's important to note that only a few of these works actually utilize LDA distributions, and even then, the settings differ significantly from our approach. Below are specific clarifications for each of the papers you mentioned:


1. **[FedFA: Federated Learning With Feature Anchors to Align Features and Classifiers for Heterogeneous Data]** 
This paper does not use an LDA distribution at all. While it does involve a 0.1 client participation ratio, the client selection is not random but follows a specific strategy. Additionally, the local update is set to 5, and the number of epochs is 200, both of which differ from our settings. It is also important to note that this paper was published on June 1, 2024, which is after our submission date. This raises a question about the appropriateness of including this reference in the review and whether it suggests an attempt to unfairly reject our submission.

2. **[Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learning]**  
This paper does not use LDA distribution and does not address the issue of randomly selecting 100 clients. Instead, it conducts experiments with 10 and 100 clients separately, without employing the random client selection method we used.

3. **[CLIENT2VEC: IMPROVING FEDERATED LEARNING BY DISTRIBUTION SHIFTS AWARE CLIENT INDEXING]**  
While this paper does use LDA distribution, its experimental settings are markedly different from ours. The technique focuses on complementing existing algorithms rather than implementing an independent solution. The training process involves 100 epochs, which also differs from our setup.

4. **[Federated Learning with Data-Agnostic Distribution Fusion]**  
This paper uses 50 fixed clients and does not employ LDA or traditional Dirichlet distributions. All parameters, including the training settings, differ from ours, and the number of epochs is not specified—training continues until convergence.

5. **[FIND YOUR OPTIMAL ASSIGNMENTS ON-THE-FLY: A HOLISTIC FRAMEWORK FOR CLUSTERED FEDERATED LEARNING]**  
Although this paper uses an LDA distribution, the parameter is set to 1.0, and all 100 clients participate in each round, contrasting with our approach of randomly selecting 0.1 of the clients. The training model and number of epochs (200) are also different from those in our study.

6. **[FedFed: Feature Distillation against Data Heterogeneity in Federated Learning]**  
This paper does involve LDA distribution experiments, but it discusses client numbers of 20 and 100 without addressing the random selection of 0.1 of the clients per round. Additionally, the metric used is target accuracy for training termination, with different epoch settings and other parameters.

7. **[FedCiR: Client-Invariant Representation Learning for Federated Non-IID Features]**  
This paper uses a different dataset, with local epochs set to 20 and global epochs to 300. Clients are not randomly selected at a 0.1 ratio per round but instead follow a different client selection strategy.

In summary, the issue of data heterogeneity, especially with LDA distributions, is a novel point of concern in this review cycle. The references provided, while relevant in some aspects, do not entirely align with our approach as only a subset uses LDA distributions, and even then, under different conditions. Moreover, it's important to emphasize that the number of global and local epochs significantly affects model performance. In addition, the client participation ratio and the method of client selection in each round are critical factors that can drastically alter the outcomes of federated learning models. The discrepancies in these settings across the referenced works and our study further underline the distinctiveness of our experimental setup and results.

Furthermore, there is no unified standard for client participation settings across different studies, and even the papers you listed do not compare consistently against each other. Given our limited time and the lack of provided code from these works, we were unable to conduct direct experimental comparisons. However, our primary focus is not on how to handle varying client participation but rather on addressing the core issue of data heterogeneity among clients. When evaluating under the scenario where all clients participate, our results outperform theirs, even with fewer epochs. This gives us confidence that if we were to align the client participation settings with theirs, our results would not be inferior.

I hope these clarifications help in better understanding our work.
Many works focus on the setting of 100 clients with 0.1 participation under a more challenging non-iid distribution than your worst performance setting, e.g., LDA distribution with  $\alpha=0.1$ [R1-R7]. These works also have some improvements without a public dataset which is claimed a key contribution of this paper. 

Reference:\
[R1] Tailin Zhou, et al. FedFA: Federated Learning With Feature Anchors to Align Features and Classifiers for Heterogeneous Data. In IEEE Transactions on Mobile Computing, 2024. \
[R2] Zhenheng Tang, et al. Virtual homogeneity learning: Defending against data heterogeneity in federated learning. In ICML, 2022.\
[R3] Yongxin Guo, et al. Client2Vec: Improving Federated Learning by Distribution Shifts Aware Client Indexing. In arXiv: 2405.16233. \
[R4] Jianhui, Duan, et al. Federated learning with data-agnostic distribution fusion. In CVPR, 2023.\
[R5] Yongxin, et al. Find Your Optimal Assignments On-the-fly: A Holistic Framework for Clustered Federated Learning. In arXivarXiv:2310.05397.\
[R6] Zhiqin Yang, et al. FedFed: Feature distillation against data heterogeneity in federated learning. In NeurIPS, 2023.\
[R7] Zijian Li, et al. FedCiR: Client-Invariant Representation Learning for Federated Non-IID Features. In IEEE Transactions on Mobile Computing, 2024.
Thanks for your response. My problems have been addressed. I will still keep my score.
Thank you for your detailed feedback and for recognizing the effort we put into conducting additional experiments. We appreciate the opportunity to clarify and address the concerns you’ve raised.
 
Regarding the observed gap between the performance of the FL model and a randomly initialized model when PR=0.1, we acknowledge that the accuracy observed in our experiments under these conditions might appear concerning. However, it is important to consider the broader significance of federated learning (FL). The primary goal of FL is to enable the sharing of data to train better models while preserving privacy, particularly under challenging conditions such as high non-IID data distributions. Given the experimental setting we chose 100 clients with a participation rate of 0.1 and a high Non-IID distribution. Therefore, it is understandable that the performance would be lower compared to a scenario with IID data distribution or centralized training.
 
 
This observed performance does NOT suggest that FL methods have lost their effectiveness. On the contrary, it highlights the inherent difficulties in such scenarios, where maintaining model accuracy is particularly challenging. It is well-known that FL methods perform better within the same IID data distribution, but evaluating under these conditions was not the focus of our research. Our primary goal was to address the non-IID problem in FL without relying on public datasets, which is central to the contribution of our work. We want to emphasize this point, as it appears there is still some misunderstanding about our contributions and the problem we intended to solve in this paper. It is both surprising and unfortunate that this has not been fully recognised.

Furthermore, in response to your concern about the practicality and effectiveness of the comparative algorithms we used, I noticed that, similar to your previous response to my rebuttal, the issue of comparative experiment choosing is another new issue being raised. While you are eligible to introduce new point at each round, it is uncommon to do so. Moreover, as before, your comments are quite general and lack reference to any specific paper that performs better than the methods we included in the paper.  Actually, the algorithms we chosen are well-established in the FL community, widely recognized for their practicality and effectiveness in FL experiments.
I would like to emphasize that our experiments were conducted with a consistent experimental setup across all methods. This includes not only 100 clients, PR = 0.1 and $\alpha = 0.5\$, but also the same epoch, iteration, data distribution, learning rate and batch size. By maintaining these parameters across all experiments, we ensured that the comparisons were fair and that the performance differences observed are attributable to the algorithms themselves rather than discrepancies in the experimental setup. It is also important to note that increasing the number of epochs typically leads to a significant improvement in the accuracy of FedGMKD, as well as other algorithms tested in our study. However, in the interest of fairness and practicality, we adhered to the standard practices observed in the three referenced papers, which set the epoch to 50. "Global Convergence of Federated Learning for Mixed Regression" (NeurIPS 2022) and "Robust Federated Learning With Noisy and Heterogeneous Clients" (CVPR 2022) highlight the importance of setting an appropriate number of epochs to balance model convergence with the computational limits of clients. Similarly, "Exploring User-level Gradient Inversion with a Diffusion Prior" (NeurIPS 2023 Workshop) reflects the need to balance accuracy with resource constraints. Therefore, setting the epoch count to 50 aligns with common practices in federated learning, providing a fair and realistic assessment of the algorithms under typical FL constraints.
 
Finally, we want to emphasize the significant contributions of our method, FedGMKD, to the field of federated learning. Our approach offers robust privacy protection and effectively tackles the Non-IID problem, improving both local and global accuracy. This is achieved through a novel combination of clustering and knowledge distillation techniques, without the need for public data. Additionally, FedGMKD ensures communication efficiency by interacting only with prototype features and soft predictions, making it a practical and scalable solution for real-world federated learning scenarios.
Thanks to the extra experiments. I know that must spend lots of time. In these experimental results, we can see that Local Accuracy (PR=0.1) and Global Accuracy (PR=0.1) in SVHN dataset is about under 10.0. However, a random initialized model can get 10.0 ACC in a 10-class problem. So does this mean that all of these methods have lost their effectiveness? At the same time, there are lots of methods that gain higher ACC at this setting (100 clients, PR=0.1 with $\alpha=0.5$ SVHN and CIFAR100). At the same time, they also have no public dataset. In summary, I am still concerned about the practicability and effectiveness of these methods compared to other methods. I will keep my score.
Dear Reviewer 8XP3

We have conducted new experiments based on your new concerns and obtained corresponding results and discussions. Have these results and discussions resolved your concerns? If these results and discussions address your concerns, can you reconsider rating my paper?
Thank you for recognizing the enhancements made to our manuscript and the additional experiments provided. We greatly appreciate your updated evaluation and insightful comments, which have undoubtedly helped refine our paper.
We sincerely appreciate your acknowledgment of our efforts to address all concerns raised, and thank you for your constructive feedback throughout the review process.
Thank you for the thorough rebuttal and for addressing my concerns. I appreciate the additional experiments with the IMBD dataset, which demonstrate the broader applicability of FedGMKD beyond vision datasets. The comparative analysis with FjORD, despite the challenges in reproducing it, was also insightful and showed the strengths of your approach in addressing data heterogeneity. The corrections and clarifications you plan to include in the final version further improve the paper's quality. Based on this, I have updated my score accordingly.
Thank you for your feedback. In our initial response, we have justified setting the client participation rate to 1 as standard practice in federated learning (FL) to establish a baseline. Although active and passive client selection are critical areas in FL, they were not within the initial scope of our study, which focused on addressing the Non-IID problem without public data.

However, we were also interested in assessing our model's performance in scenarios where not all clients are accessible. To address this, we have conducted experiments based on the SVHN and CIFAR100 datasets, varying client participation rates to simulate real-world conditions of intermittent client availability. For a fair comparison, we benchmarked our model against several existing models under the same conditions.
## Experimental Design
- **Total Clients:** 100 
- **Participation Rates (PR):** 0.1, 0.2, 0.5 (corresponding to 10%, 20%, and 50% client participation)
- **Data Heterogeneity:** $\alpha = 0.5\$.
## Results and Analysis
| **Scheme**   | **Local Accuracy (PR=0.1)** | **Local Accuracy (PR=0.2)** | **Local Accuracy (PR=0.5)** | **Global Accuracy (PR=0.1)** | **Global Accuracy (PR=0.2)** | **Global Accuracy (PR=0.5)** |
|--------------|-----------------------------|-----------------------------|-----------------------------|------------------------------|------------------------------|------------------------------|
| **Dataset:** SVHN | **Clients (Total):** 100 | | | | | |
| FedGMKD      | 9.22                        | 18.17                       | 45.01                       | 8.76                         | 16.73                        | 44.32                         |
| FedAvg       | 8.82                        | 17.81                       | 43.85                       | 8.53                         | 15.91                        | 40.78                        |
| FedProx      | 8.96                        | 17.99                       | 44.50                        | 8.64                         | 15.94                        | 41.10                         |
| FedMD        | 8.99                        | 17.94                       | 44.67                       | 8.63                         | 16.01                        | 42.02                        |
| FedProto     | 9.02                        | 18.05                       | 44.83                       | 8.67                         | 16.08                        | 41.55                        |
| FPL          | 8.83                        | 18.12                       | 44.23                       | 8.64                         | 16.05                        | 42.64                        |
| **Dataset:** CIFAR100 | **Clients (Total):** 100 | | | | | |
| FedGMKD      | 2.85                        | 5.11                        | 13.58                       | 2.68                         | 4.73                         | 12.83                        |
| FedAvg       | 2.39                        | 4.77                        | 10.70                        | 2.18                         | 4.52                         | 9.77                         |
| FedProx      | 2.44                        | 4.92                        | 11.35                       | 2.35                         | 4.23                         | 10.14                        |
| FedMD        | 2.51                        | 4.96                        | 11.62                       | 2.42                         | 4.41                         | 11.27                        |
| FedProto     | 2.49                        | 4.93                        | 11.45                       | 2.37                         | 4.58                         | 11.46                         |
| FPL          | 2.55                        | 4.99                        | 11.57                       | 2.53                         | 4.54                         | 11.43                        |


These results demonstrate that FedGMKD consistently achieves the highest accuracies across all participation levels, compared to other schemes. For instance, on the SVHN dataset, FedGMKD's performance peaks at PR=0.5 with local and global accuracies of 45.01% and 44.3%, respectively. Similarly, for CIFAR100, it leads with a local accuracy of 13.58% and a global accuracy of 12.83% at PR=0.5.

The robust performance of FedGMKD across different client participation rates confirms its effectiveness in real-world settings, addressing reviewers' concern about its performance when not all clients are accessible. The adaptability and robustness of FedGMKD make it a promising solution for practical deployment in diverse federated learning applications, ensuring reliable performance even under fluctuating client participation conditions.
Thanks for your response. My problems have been addressed partially. However, I still have concerns about whether this method will remain effective when all clients are not accessible at any time. So I will keep my score
The authers have solved all my concerns.