FedGMKD: An Efficient Prototype Federated Learning Framework through Knowledge Distillation and Discrepancy-Aware Aggregation
Authors: Jianqiao Zhang
Caifeng Shan
Jungong Han
Venue: NeurIPS 2024 poster (2024)
Keywords: Federated Learning
Clustering
Aggragation method
Knowledge Distillation
Prototype
Heterogeneous Data Environments
Primary area: machine_vision

Abstract: Federated Learning (FL) faces significant challenges due to data heterogeneity across distributed clients. To address this, we propose FedGMKD, a novel framework that combines knowledge distillation and differential aggregation for efficient prototype-based personalized FL without the need for public datasets or server-side generative models. FedGMKD introduces Cluster Knowledge Fusion, utilizing Gaussian Mixture Models to generate prototype features and soft predictions on the client side, enabling effective knowledge distillation while preserving data privacy. Additionally, we implement a Discrepancy-Aware Aggregation Technique that weights client contributions based on data quality and quantity, enhancing the global model's generalization across diverse client distributions. Theoretical analysis confirms the convergence of FedGMKD. Extensive experiments on benchmark datasets, including SVHN, CIFAR-10, and CIFAR-100, demonstrate that FedGMKD outperforms state-of-the-art methods, significantly improving both local and global accuracy in non-IID data settings.

FedGMKD: An Efficient Prototype Federated Learning Framework through Knowledge Distillation and Discrepancy-Aware Aggregation
Jianqiao Zhang
Caifeng Shan"
Department of Computer Science Aberystwyth University Aberystwyth, UK j jiz200aber. ac. uk
School Intelligence Science and Technology
Nanjing University
Nanjing, China caifeng watgsgsaaannnllammm com
Jungong Han"
Department Automation
Tsinghua University
Beijing, China jungonghan770@maill com
Abstract
Federated Learning (FL.) faces significant challenges due data heterogeneity across distributed clients. To address this, we propose FedGMKD, a novel frame- work that combines knowledge distillation and differential aggregation for efficient prototype-based personalized FL without the need for public datasets or server-side generative models. FedGMKD introduces Cluster Knowledge Fusion, utilizing Gaussian Mixture Models to generate prototype features and soft predictions on the client side, enabling effective knowledge distillation while preserving data privacy. Additionally, we implement a Discrepancy-Awart Aggregation Technique that weights client contributions based on data quality and quantity, enhancing the global model's generalization across diverse client distributions. Theoretical analysis confirms the convergence FedGMKD. Extensive experiments on bench- mark datasets, including SVHN, CIFAR-10 10, and CIFAR-100, demonstrate that FedGMKD outperforms state-of-the-ae methods, significantly improving both local and global accuracy in Non-IID data settings.
Introduction
Federated Learning (FL) is a transformative approach to collaborative machine learning that allows multiple participants to train a shared model while maintaining data privacy by keeping datasets decentralized. This architecture mitigates privacy risks, particularly in sensitive domains such as healthcare and 3inance[3] 5 6 14
23 26 FL enables clients contribute the global model without transferring raw data, reducing communication overhead while ensuring privacy. However, the Non-IID (non-ines mon-inddeeenent and identically distributed) nature of client data poses challenges for model convergence and consistency. Heterogeneous data leads to divergence in model updates,
slowing convergence and degrading performance across clients 14 15 23
Personalized Federated Learning (pFL) has emerged as a solution to tailor models to individual client data distributions, improving local accuracy while maintaining global robustness 7 g 37
Joint corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

However, traditional pFL approaches relying on knowledge distillation (KD) often require public datasets, raising privacy concerns and complicating impleeenation I9 Straggler inefficiencies in traditional aggregation methods also delay convergence 25 highlighting the need for more effective
PFI. frameworks 29 40.
To address these challenges, we propose FedGMKD, novel framework that integrates Cluster Knowledge Fusion (CKF) and a Discrepancy-Aware Aggregation Technique (DAT) enhance both local and global model performance. Our approach introduces several key innovations: first, by employing Gaussian Mixture Models (GMM), we generate prototype features and soft predictions for each class at the client side, which are aggregated at the server without the need for public datasets, thus preserving data privacy and effectively addressing the Non-IID problem. Second, DAT adjusts the aggregation process by weighting prototype features and soft predictions based on both the quality and the quantity, rather than the quantity, of client data, thus enhancing the global model's ability to generalize across diverse client data distributions. Furthermore, we conduct a rigorous theoretical analysis of the convergence and convergence rate of FedGMKD, providing strong guarantees for its effectiveness. Finally, experimental results on multiple benchmark datasets demonstrate that FedGMKD not only achieves state-of-the-art accuracy in local and global models but also significantly improves efficiency, making it highly effective in addressing the heterogeneity challenge while maintaining computational feasibility.
2 Related Work
2.1 Heterogeneity in Federated Learning
FL techniques have evolved significantly to address the challenges posed by the non-IlD nature of data across diverse clients. Methods such as FedProx [22], Scaffold [15], and FedAlign [27] have substantially advanced the reduction of client-side bias by refining local updates to achieve better alignment with the global model. Nevertheless, these methods often rely on simplified assumptions about the underlying data distributions, which may prove inadequate in handling extreme variations in data heterogeneity. such as severe data imbalance and non-overlapping feature spaces. On the
server side, solutions like FedOpt [32] and Agnostic Federated Learning (28] attempt to dynamically adjust aggregation strategies based on client contributions. However, despite these innovations, they continue to face significant challenges with scalability and often incur high computational costs, which limit their applicability in large-scale, highly diverse networks (20) 33 38). Moreover, approaches such as FedMix [39]. which introduce synthetic data generation to approximate IID conditions, still struggle to accurately replicate the complex and highly variable data distributions encountered in real-world scenarios, particularly in non-IID environments.
2.2 Personalized Federated Learning
Personalization in federated learning has emerged as crucial strategy to cater to diverse client- specific data characteristics. Strategies range from adapting global models for local refinement (c.g., FedPer 2 and Per-FedAvg (8) to fully personalized approaches like MOCHA [34,, pFedMe [35]. and Ditto 21]. These personalized models often perform well on local data but can diverge significantly from the global model, leading to challenges in maintaining cohesive learning strategy across the network. Additionally, the resource demands for training individualized models can be prohibitive, especially in scenarios with limited computational and bandwidth resources 7 25
2.3 Knowledge Distillation and Prototype Learning in FL
Knowledge distillation and prototype learning have made notable contributions to federated learning by enabling more efficient model training and facilitating knowledge transfer across distributed networks. KD-based approaches, such as FedMD [8]] and FedDF [24). allow for the compression of model knowledge into more efficient representations, thereby reducing communication costs and enhancing scalability. However, these methods necessitate meticulous tuning of distillation parameters to achieve a balance between model complexity and performance [I]. FedProto (36) introduces prototype learning by aggregating class prototypes from clients to improve generalization across heterogeneous datasets. However, while FedProto improves local validation accuracy, it shows
minimal improvement in global performance. As extension of FedProto, FedHKD [4] integrates

---

knowledge distillation by introducing the concept of "hyper-kmon -nowee-enowwedgee", which involves sending the mean representations of both feature and soft predictions from clients the server for global aggregation. While FedHKID significantly enhances client-side personalized model performance without relying on public datasets I it focuses mainly on improving the client models. It does not introduce substantial advancements in hyper-knowledde extraction techniques, nor does it achieve dual improvement for both client and server performance in theory, as knowledge distillation predominantly benefits the client-side models. Building upon FedHKD, our research addresses these limitations by not only improving the knowledge distillation process by prototype feature and soft predictions extraction but also benefiting the server-side models by using DAT. This ensures simultaneous improvement both global model performance and client models.
Methodology
3.1 Problem Formulation
In the context of FL., we consider a scenario involving distinct clients, each holding a private where each dataset reflects a potentially unique subset of the overall class distributions. This setup illustrates the fundamental Non-IID data challenge in FL, where the heterogeneity of each client's dataset D, complicates the task of learning generalized global model. Fach client optimizes local model w, by minimizing a local loss function F;, which is typically expressed as the empirical risk over its dataset D₁:
dataset ... }.
(w:))=
1 D
Σ ((w,,
ZED:
where & denotes the loss function measuring the prediction error on an instance x.
To combine these local models into a global model, the FedAvg algorithm, introduced by McMahan et al. (26). applies weighted averaging scheme based on the size of each client's dataset:
W
D.W.
N
2=1
where W represents the parameters of the global model after the r-th aggregation round, w. denotes the parameters of the i-th client's 's model in the r-th round, and N is the total number of samples across all clients, N = Ei= (Dil. Our objective with FedGMKD is to tailor personalized models w; for each client i that not only achieve high accuracy locally specific data but also enhance a global model W that excels across diverse client distributions. This task becomes particularly challenging in the presence of heterogeneous data, as naive aggregation approaches tend to diminish the effectiveness of the global model. FedHKD 4 has demonstrated significant success in addressing this issue by using seyper-knowlegge distillation, which improves both global and personalized models through the exchange of mean representations and soft predictions. Inspired by this approach, FedGMKD enhances model performance on both the client and server sides by introducing Cluster Knowledge Fusion (CKF) based on Gaussian Mixture Models (GMM) for prototype feature and soft predictions generation and Discrepancy-Awart Aggregation Technique (DAT) based on data quality. This dual mechanism not only improves local model performance through better feature representation but also refines the global model by leveraging high-qu high-quality aggregated knowledge across clients, thereby effectively addressing the challenges posed by non-lID data distributions in federated learning.
3.2 Utilizing Cluster Knowledge Fusion
In Federated Learning (FL), traditional Knowledge Distillation (KD) techniques often rely on transferring knowledge from a complex central model (teacher) to simpler client models (students). This process typically requires a public dataset to generate soft labels representing the teacher's predictions in FL. Meanwhile, this approach raises significant privacy concerns and faces challenges, especially Non-IID data environments, where the heterogeneity of client data exacerbates these issues, reducing the effectiveness of traditional KD methods in FL. To address these challenges and inspired by enper-knowledge of FedHKD 4. we propose a Clustered Knowledge Fusion (CKF) approach based on Gaussian Mixture Models (GMM). CKF clusters client updates according to data
3

---

similarity, generating prototype features and soft predictions for each class, which are then aggregated the server. These aggregated results form a synthetic dataset that serves as the foundation for knowledge distillation, enabling efficient knowledge transfer between the server and clients. Unlike traditional KD approaches, CKF not only eliminates the reliance on public datasets but also leverages client-specific representations, effectively addressing the Non-IID data problem and improving overall FL performance.
In the context of federated image classification, each client processes raw image data e; through a feature extraction function Fs., producing feature representation h;, and a classifier function C.s, which maps h; into a soft prediction vector 2;:
h4 Fo, (2s), 2 = Softmax (Co. (l4)):
where (.) and (.) represent the feature extractor and classification function for client i, respectively. The feature vector h4 encodes the latent features of the input data, and 2 represents the soft prediction vector, indicating the probability distribution over the classes.
When the features and soft predictions have been extracted, GMM are employed to cluster client updates based on data similarity prior to aggregation.
The responsibility 7/m (x() indicates the probability that the data point x (which can be feature vector or soft prediction vector) belongs to the m-th Gaussian component. It is calculated as:
Tm [(x;;; M - Ax ).)
m. mm)
7 fx()x
Here, Tm represents the mixture coefficient of the m-th Gaussian component, where ΣM Tm
The term (x{;; m. Em) denotes the Gaussian probability density function evaluated at x/. with being the mean vector and Em being the covariance matrix of the m-th Gaussian component. The denominator sums over all M Gaussian components, normalizing the responsibility values so that they sum across all components. These responsibility values indicate the contribution of each Gaussian component to the data point x/. and they are later used to compute the prototype features and soft predictions by weighting the means and predictions of each Gaussian component. Using the responsibility values, the prototype features and soft predictions for class j in the local dataset of client i are calculated as follows:
M
M Σ 17
(h
( my:
H.
Ymm 12mys
(5)
where i denotes the prototype feature for class j at client i, synthesized from the cluster knowledge, and q/ represents the prototype soft prediction for class j at client i, calculated based on the responsi- bility values. Here, m, represents the mean vector of the m-th Gaussian component for class j. and Zm, represents the soft prediction vector corresponding to the m-th Gaussian component.
CKF is derived by integrating these GMM-based prototype features and their corresponding prototype soft predictions. For class j in the local dataset client i, CKF is defined by combining the prototype features and their prototype soft prediction:
Ki- g))
(6)
If there are j classes, then the full CKF of client i is:
K; = U (M.
(7)
j=1
A flow diagram illustrating the computation of CKF is provided in Supplementary Section

---

3.3 Discrepancy-Aware Aggregation Technique
The aggregation phase in FedGMKD amalgamates CKF from each client to construct global representation of CKF for given class j at each iteration r + 1. For the given class j. the global CKF represented by the aggregated prototype features H2+1 and soft predictions Q,+². These are calculated follows:
HJ
Σ
97th
Σ w/ 9.
(8)
where w/ is the weight corresponding to client i's contribution, reflecting both the volume and the quality of the data contributed by client i.
In traditional FL approaches such as FedAvg, the server aggregates models by averaging the parame- ters submitted by each client. This method assumes that data across different clients are identically and independently distributed (IID). However, tin real-world applications, this assumption often fails due to Non-IID data distributions, leading sub-optimal global models when client data varies sig- nificantly in distribution and relevance. Most existing methods primarily weight client contributions based on data volume, neglecting the quality of the data, which can further exacerbate this issue.
To address these challenges, FedGMKD introduces a Discrepancy- evaluates both the volume and the quality of data each client contributes. This is achieved by quantifying how well the local data aligns with the global data distribution. The aggregation process refined by incorporating a measure of the discrepancy between local model prototype predictions and the aggregated global prototype predictions, thereby enhancing the robustness and accuracy of the federated model.
Aggregation method that
To begin, the initial weights for each client and class are calculated based on the proportion of samples for that class across all clients:
N
init
NO
where N/ is the number of samples of class in client i's local dataset. This ensures that the initial weight w.y reflects the proportion of class j samples that client contributes to the global dataset for that class.
After this, the final aggregation weights are adjusted based on the discrepancies between the local and global data distributions. This discrepancy is quantified using the Kullback- (KL) divergence and is balanced by the initial weight whi for each category j:
ReLU
(Wi
a de +
w
(10)
Σ ReLUU wiij - a d3 +
where with is the initial weight for class j on client i, reflecting the proportion of class j samples contributed by client to the global dataset. df represents the KL divergence between the client and server distributions for class j. and a and b are adjustment parameters that control the sensitivity of the weight updates based on the discrepancies.
The KL divergence de between the client i's local distribution and the server's global distribution for class j is calculated as follows:
d. = DKL ④ (1 á. log
(11)
where j/ and Q are the predicted probabilities for class j the local client and the global server distributions, respectively. This KL divergence measures the discrepancy between the local and
5

---

global predictions specifically for class j. allowing the model to adjust the aggregation weights based on this class-specific difference.
For each class j. the global CKF is computed by aggregating the local CKF from all clients using the discrepancy-aware weights w. These global CKF values, represented by H and Q encapsulate the collective knowledge from all clients for class j. After calculating the global CKF for
each class j. the complete global CKF, denoted as G is constructed by taking the union of the global CKF for all classes:
G
U ( 07 f=1
(12)
This complete global CKF GT+1 serves the updated global knowledge representation, which is used to guide future iterations of model updates. A flow diagram illustrating the computation of DAT provided in |Supplementary Section
3.4 Local Training Objective Function
After the server completes the aggregation, the updated global CKF G is sent to the clients selected for the next FL round aid their local training. For client with dataset Dj. the local training objective integrates the empirical risk with regularization terms designed to align the local model with the global CKF. The loss function for client defined as:
L(D,, wi)
Σ (E.,
(CC,, (Fe, (xx))
Dil
(13)
a 7 TI
HH)+1)
H
r++
Σ
(xx)
yk 11
|Di|
T
where [D;] denotes the number of samples in the dataset owned by client i, ((Co. 9k) denotes the cross-entropy loss function, 11.112 denotes the Euclidean norm, and À and are hyper-parameters. Note that H*+1 represents the global prototype feature for class j, and Q denotes the global soft predictions for class j at iteration + The term Cv. (H₃+1) represents the local classifier's predictions on the global prototype feature H H and T is the temperature of KD.
The loss function consists of three terms: the empirical risk formed using predictions and ground-truth labels, and two regularization terms that utilize the global CKF The first term is the empirical risk, represented by the cross-entropy loss function. This term encourages the local model perform well on its own data. The second term, known as the feature alignment loss, aligns the local feature extractor with the global CKF by minimizing the squared Euclidean distance between the local feature representations Fo, (Ix) and the globally aggregated CKF features H This regularization encourages the local feature extractor to produce similar feature representations the global prototype features for each corresponding class, improving consistency between the local and global models. The third term, called the knowledge alignment loss, ensures predictive consistency across federated learning by minimizing the discrepancy between the local classifier's predictions on the global prototype features and the global soft predictions for each class j. Specifically, by using Euclidean distance, which is nen-negative and convex, these terms effectively regularize the local models to be more consistent with the global CKF.
3.5 Overall Framework of FedGMKD
FedGMKD integrates CKF and DAT to enhance both local and global model performance. The framework operates iteratively, as outlined in Algorithm [ The server initializes the global model W = (F", C*), where fo and C denote the parameters of the global feature extractor and classifier, respectively. The global CKF Go is also initialized, comprising global prototype features and soft predictions for each class. At each global epoch, the server transmits the global model W and

---

global CKF -*- to the selected clients. Clients update their local models by minimizing a composite loss function comprising three components: (1) the empirical risk. representing the cross-entropy between predicted and ground truth labels; (2) the feature alignment loss, measured by the Euclidean distance between local and global prototype features; and (3) the knowledge alignment loss, measured by the Euclidean distance between local classifier outputs global prototype features and the global soft predictions. Upon completing local training, clients compute local CKF and transmit these, along with their updated models, back to the server. The server aggregates the received CKF using DAT, which adjusts client contributions based on data volume and quality. The global CKF and model are updated accordingly. This process repeats across multiple federated learning rounds.
Algorithm 1 FedGMKD
Require:
Distributed datasets across clients, D == (D1, D2...., D J client participation rate μr; hyper-parameters À and temperature T;; number of global epochs Rt.
Ensure: Updated global model WRR-1 and personalized
local
models
[wB.++ W2 R,+1 ..... w. 1
1: Server initializes the global model 8 and global CKF GD for each class.
2: for
Rr do
3: Server selects i clients for participation.
4: Server broadcasts W and G" the selected clients.
5:
for each selected client i do
6:
Client initializes local model w from W
7:
if r == then
8:
Client updates W. using Equation
9:
Client computes initial CKF KY using Equations 3
5
6
and
else
11:
Client updates w. using Equation 13 incorporating G
12:
Client computes CKF K? using Equations 5 and and computes divergence
de for each class using Equation I1
13:
end Client i sends w.,, KJ, and di (if 1) back the server. end for
14:
15:
16:
i = then
17:
Server averages CKFs and models from clients to update global CKF Gr+1 and glohal
model W using Equations g and 12
18:
else
Server computes weights for each class of each client using Equation I0
20:
Server updates global CKF G++1 using Equations g and 12 based on the computed
weights.
21:
Server updates global model W using weighted averaging of the models.
end if
23:
Server sends W G (if 1) back to client i.
end for
25:
return Updated global model W R,++1 and local models (wfrtt1 .2 R2,
..... B, R.tt)..
3.6 Convergence Analysis
Given the Non-IID nature of data across clients in FedGMKD, we establish two theorems to describe the framework's convergence under well-defined mathematical assumptions.
Theorem 1: FedGMKD Convergence
Under Assumptions 1-5 (A.6. 1j, for any client i, after R global communication rounds, the expected
global loss function is bounded as:
F(W!) F-
LnRG
W (
n12 0²
(14)
2

---

The detailed proof is provided in A.6.3
Theorem 2: FedGMKD Convergence Rate
Under Assumptions 1-5 (A.6.I), J, for any client after R global communication rounds, the conver- gence rate of the global loss function F( (W) is bounded as follows:
Ci R
F(W") F
+ C2.
(15)
where ((W) is the global loss function, 1 represents the lower bound F(W), and C₁ and C2 are constants that depend on variance o², Lipschitz constant L, learning rate and the number of local steps.
The detailed proof is provided in A.6.3
Experiments
4.1
Datasets
We evaluate FedGMKD on three widely used FL benchmark datasets, selected to cover a range of task complexities, demonstrating the method's robustness and scalability.
The Street View House Numbers (SVHN) dataset consists of over 600,000 labeled digit images from street scenes. It presents varied imaging conditions, such as lighting and background differences, making useful for testing under non-IID scenarios. Despite these variations, SVHN is considered relatively simple due to its large dataset size and clear digit images.
SVHN 30
CIFAR-10 comprises 60,000 32x32 color images across 10 classes, with 6,000 images per class. It is a standard benchmark for image classification in both centralized and federated learning, providing moderate complexity due the diverse nature of the images.
CIFAR-10 17:
CIFAR-100 17 Similar to CIFAR-10 but with 100 classes, CIFAR-100 contains 60,000 images with 600 images per class. It poses more challenging task, given the finer granularity and increased class variability, making especially difficult in federated learning with non-IID data.
4.2 Models
For our experiments, adopt the ResNet18 architecture D0,, which has consistently demonstrated superior performance across a wide range of learning tasks, outperforming traditional Convolutional Neural Networks (CNNs). ResNet18 incorporates residual connections to effectively address the vanishing gradient issue, enabling the training of deeper networks while maintaining computational efficiency.
4.3 Baselines
To establish a comprehensive comparison, our evaluation includes diverse set of baselines, encom- passing both well-established methods and recent advances in federated learning. These baselines include: FedAvg (266), the foundational algorithm in federated learning; FedProx (22). addressing data heterogeneity; MOON [19], focusing model personalization; FedMD (18) and FedGen [40], which utilize public datasets and generative models, respectively, to enhance performance under Non-IID conditions; FedProto [36] and PFL [13,, which employ prototype and clustering learning methods to handle data disparities in federated learning; and FjORD [12,, which introduces an ordered dropout mechanism to enable fair and accurate federated learning under heterogeneous target distributions.
4.4 Experimental Setting
The models were implemented and run using PyTorch 31 with 2 NVIDIA A100 GPUs. The Adam optimizer [16] was used for model training experiments. The learning rate was initialized to 0.001 and decreased every 10 iterations with decay factor of 0.5, while the hyper-parameter in

---

Adam was set to 0.5. The number of global communication rounds was set to 50, and the number of local epochs was set 3. The size of a data batch was set to 64, and the participating rate of clients was set to 1. For all datasets (SVHN 30) CIFAR-10) 10, and CIFAR-I00 100 [7 the latent dimension of data representation was set to 32.
Hyper-parameters: For the FedProx [22] algorithm, the hyper-paramedee Marrx was set to 0,5, For the MOON [9] algorithm, the proximal term's hyper-parameter Mmmoon was to 1. In FedGen [40], Multi-Layer Perception (MLP)-based architecture with a hidden dimension of 512 was employed for the generative model. Latent, noise, and input/output dimensions were tailored each dataset, and the generative model was trained for 5 epochs per global round. The ratio of the generative batch-size the training batch-size was 0.5 (generative batch- match-size set to 32). Parameters Oggnnerative and ggenerative were initialized at 10 with a decay factor of 0.98 per global round. FedMD [8] used regularization hyper-parameter And of 0.05, and the public dataset size matched the clients' local training dataset size. FedProto [6 had regularization parameter Apruto set to 0.05. For FPL. [33, the regularization parameter ÄFPL was set to 0.1, with the number of prototypes per class fixed at 10. For FjORD [2], the unique dropout rate 8 was set to 0.5, and the temperature for knowledge distillation Tijord set to 0.7, following the original paper's s settings. Finally, in our proposed FedGMKD, hyper-parameters and were set to 0.6. The clustering centers were dynamically adjusted between 2-7 based on the data distribution in each category, with the parameters a and for differential aggregation both set to 0.2, and the temperature T for knowledge distillation was set to 0.6.
4.5 Results
Table 1: Results on data partitions generated from Dirichlet distribution with the concentration
parameter 3 = 0.5. The number of clients is 10, 20, and 50; the clients utilize 10%, 20%, and 50% of the datasets. A single client's averaged wall-clock time per round measured across 2 A100 GPUs in parallel manner. The reported local and global accuracies are the averages of the last 5 rounds.
Dataset
Scheme
Local Acc
Global Ace
Avg Time (S) Pub Data
10
20
50
10
20
50
SVHN
FedAvg
84.29
85.20
85.67
81.98
87.32
89.72
168.44
No
FedProx
85.25
86.38
86.08
81.71
87.40
88.74
229.17
No
Moon
84.11
85.43
85.43
81.95
86.90
88.97
358.14
No
FedGen
85.18
85.10
84.96
81.96
86.02
88.52
205.37
No
FedMD
85.45
85.90
86.31
82.04
87.30
89.91
611.33
Yes
FedProto
85.58
86.44
86.85
81.34
86.97
89.79
346.13
No
FPL
85.37
86.02
85.87
79.81
85.64
88.76
522.83
No
FjORD
85.13
85.97
86.21
81.56
85.09
89.36
380.74
No
FedGMKD
86.26
87.43
87.16
82.64
87.78
90.17
312.52
No
CIFARIO
FedAvg
55.75
58.76
61.51
46.62
52.61
51.53
98.94
No
FedProx
57.46
58.91
62.94
47.97
53.13
56.04
126.56
No
Moon
58.61
59.12
62.42
46.89
50.16
57.29
221.19
No
FedGen
59.46
60.17
61.03
48.09
51.55
52.62
122.35
No
FedMD
60.15
62.05
63.37
48.32
53.73
57.69
410.19
Yes
FedProto
59.77
62.85
64.98
48.97
50.88
57.12
229.40
No
FPL
60.95
62.74
64.49
47.19
52.04
58.35
295.97
No
FjORD
59.62
63.36
63.61
49.18
53.22
58.74
252.34
No
FedGMKD
61.78
64.04
65.69
49.78
55.16
60.31
251.55
No
CIFAR100
FedAvg
15.39
17.10
21.09
14.51
18.98
22.21
97.02
No
FedProx
16.45
17.56
21.91
16.06
19.67
23.35
120.36
No
Moon
15.46
18.03
21.25
15.19
18.16
21.37
201.91
No
FedGen
14.08
17.05
19.54
14.88
19.05
23.16
148.58
No
FedMD
13.25
19.03
21.93
15.96
19.20
23.75
482.76
Yes
FedProto
15.70
18.63
22.50
15.38
17.13
18.72
206.12
FPL
15.93
18.24
21.96
15.37
18.19
21.59
373.09
No
FjORD
15.94
19.91
22.60
16.93
21.45
22.86
226.73
No
FedGMKD
17.16
20.96
23.57
16.97
21.56
24.63
275.60
No

---

The proposed method, FedGMKD. consistently demonstrates superior performance across the bench- mark datasets, both in terms of local and global accuracy, often outperforming other federated learning methods. On the SVHN dataset, FedGMKD significantly improves both local and global test accuracies over FedAvg across different client counts. Compared to FedProto, FedGMKD exhibits notable improvements, with local accuracy gains ranging from 0.31 % to 0.99%, and global accuracy improvements ranging from 0.38% to 1.3%. When compared to FPL, FedGMKD demonstrates even more substantial improvements, with local accuracy gains of 0.89% to 1.41%, and global accuracy gains of 1.41% to 2.83%. FjORD performs competitively in this dataset, showing results second only to FedGMKD, particularly in global accuracy. However, FedGMKD still outperforms FjORD by up to 1.46% in local accuracy and 2.69% in global accuracy, demonstrating its advantage in Non-IID environments.
CIFAR-10 10, FedGMKD achieves substantial improvements over FedAvg, with local accuracy increases of 4. 18% to 6.03% and global accuracy gains of 2.55% to 8.78%. FedGMKD also surpasses FedProto and FPI., showing gains of up to 2.01 % local accuracy and 4. 4.28% in global accuracy compared to FedProto, and up 1.3% in local accuracy and 3.12% in global accuracy compared to FPL. In this case, while FjORD also performs well, FedGMKD remains superior, particularly in terms of global accuracy, further showcasing the robustness of the proposed approach in handling Non-IID data. Meanwhile, on CIFAR- 100, FedGMKD achieves significant gains over FedAvg, FedProx, and FPL in both local and global accuracies across all client counts. FedGMKD shows local accuracy improvements of 1.23% to 2.72% compared to FPL, and global accuracy gains of 1.6% to 3.37%. FjORD's performance in CIFAR- -IFAR-100, while strong in some settings, is still notably behind FedGMKD, particularly as the number of clients increases, where FedGMKD's clustering-hased approach becomes increasingly effective.
In terms of computational efficiency, FedGMKD incurs a modest increase training time compared to FedAvg but remains competitive given its substantial accuracy improvements. Compared to Fed- Prox, FedGMKD shows comparable or slightly better training efficiency while maintaining superior accuracy. FedGMKD is also significantly more efficient than FPL, achieving better accuracy with reduced computational overhead. FjORD shows efficiency similar to FedProto but requires more time than FrdGMKD on SVHN and CIFARIO. Overall, FedGMKD strikes optimal balance between accuracy and computational requirements. The additional computational burden of FedGMKD is justified by the substantial gains in local and global accuracy through CKF and DAT. These improve- ments suggest that FedGMKD offers a highly effective and efficient solution for real-world federated learning scenarios, particularly in the presence of NID-IID data distributions.
5
Conclusion
This paper presents FedGMKD, a novel federated learning framework that addresses data heterogene- ity without requiring public datasets and complex server-side models. Through Cluster Knowledge Fusion (CKF) and Discrepancy-
Aggregation Technique (DAT), FedGMKD achieves superior local and global accuracy across various benchmark datasets, outperforming methods like FedAvg, FedProto, and FPL. Meanwhile, theoretical convergence guarantees and experimental results validate its effectiveness. While FedGMKD introduces moderate computational overhead, the accuracy gains justify this cost. Future work will focus on improving computational efficiency and scalability to better handle large, complex datasets, enhancing its applicability to broader federated learning scenarios.
10

---

References
[1] Dmitry Afonin et al. Towards federated learning with adaptive knowledge sharing. arXiv preprint arXiv:2110.11549 2021.
[2] Mridul Arivazhagan, Ananda Theertha Aggarwal, et al. Federated learning with personalization layers. arXiv preprint arXiv:1912.008// 2019.
[3] Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzocchi, H Brendan McMahan, et al. To- wards federated learning at scale: System design. In Proceedings of the 2nd SysML Conference, 2019.
[4] Huancheng Chen, Haris Vikalo, et al. The best of both worlds: Accurate global and personalized models through federated learning with data-free hypee-knowleddee distillation. arXiv preprint 230.99230, 2023.
[5] Jingkun Chen, Changrui Chen, Wenjian Huang. Jianguo Zhang, Kurt Debattista, and Jungong Han. Dynamic contrastive learning guided by class confidence and confusion degree for medical image segmentation. Pattern Recognition, 145: 145:109881, 2024.
[6] Jingkun Chen, Jianguo Zhang. Kurt Debattista, and Jungong Han. Semi-supervised unpaired medical image segmentation through task-affinity consistency. IEEE Transactions on Medical Imaging, 42(3):594 605, 2023.
[7] Yu Deng, Mohammad Mehdi Kamani, al. Adaptive personalized federated learning, arXiv preprint arXiv:2003. 13461, 2020.
[8] Alireza Fallah et al. Personalized federated learning with theoretical guarantees: A model- agnostic meta-learning approach. Advances in Neural Information Processing Systems, 33:3557- 3568, 2020.
[9] Jiangtao Gou, Baosheng Yu, et al. Knowledge distillation: A survey. International Journal of Computer Vision, 129:1789-1819, 2021.
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[11] Geoffrey Hinton. Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint (tt:::55000053.1/ 2015.
[12] Samuel Horvath, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos Venieris, and Nicholas Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems, 1111288772289, 2021.
[13] Wenke Huang, Mang Ye, Zekun Shi, He Li, and Bo Du. Rethinking federated learning with domain shift: A prototype view. In 2023 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 16312- IEEE, 2023.
[14] Peter Kairouz, H Brendan McMahan, Brendan Avent, et al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14(1-2):1-214 2021.
[15] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. International Conference on Machine Learning, pages 5132-5143, 2020.
[16] Diederík P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
[17] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[18] Daheng Li, Ryo Murakami, Jiayuan Liu, et Fedmd: Heterogenous federated learning via model distillation. Advances in Neural Information Processing Systems, 2019.
11

---

[19] Qinkai Li, Bingsheng He, and Dawn Song. Model contrasvee federated learning, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[20] Qitian Li, Hao Sun, et al. Sync-sim: A scalable method for learning via synthetic data generation. IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pages $132-5137, 2020.
federated
[21] Tian Li, Anit Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. International Conference on Machine Learning, 2021.
[22] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith, Fedprox: Federated optimiza- tion in heterogeneous networks. arXiv preprint arXiv:
2020.
[23] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. A review of federated learning: Challenges and opportunities. IEEE Signal Processing Magazine, 2020.
[24] Tao Lin. Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion federated learning. arXiv preprint orXiv:200.00222,, 2020.
[25] Yishay Mansour, Daniel Hsu, et al. Three approaches for personalization with applications to federated learning, arXiv preprint arXiv:2002.10619, 2020.
[26] Brendan McMahan, Eider Moore, Daniel Ramage, and Seth Hampson. Communication--ffiiienn learning of deep networks from decentralized data. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, pages 1273-1282, 2017.
[27] Matias Mendieta, Shiori Sagawa, Akito Makino, et al. Local adaptation for alignment in federated learning, In Conference Neural Information Processing Systems, 2022.
[28] Mehryar Mohri, Ananda Theertha Suresh, and Nathan Srebro. Agnostic federated learning. In International Conference on Machine Learning, 2019.
[29] Pablo Mora et al. Knowledge distillation for federated learning with personalized client models. arXiv preprint arXiv:2210.14666, 2022.
[30] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Read- ing digits in natural images with unsupervised feature learning. In Proceedings of the 24th International Conference on Neural Information Processing Systems (NIPS), 2011.
[31] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32. pages 8024-8035, 2019.
[32] Sashank J Reddi, Zachary Charles, Manzil Zaheer, et al. Adaptive federated optimization. International Conference on Learning Representations, 2020.
[33] Felix Sautler, Simon Wiedemann, Klaus-Robert Müller, and Wojeiech Samek. Robust and communication--ffiiieee federated learning from non-iid data. IEEE transactions on neural networks and learning systems, 2019.
[34] Virginia Smith, Chao-Kar Chiang, Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learning. Advances in Neural Information Processing Systems, 2017.
[35] Dinh T al. Personalized federated learning: A meta-learning approach. arXiv preprint arXiv:2002.07948, 2020.
[36] Renzhong Tan, Qingyang Ang, Qianru Sun, and Fan Guo. Fedproto: Federated learning with prototypical representations. International Conference on Learning Representations, 2022.
[37] Renzhong Tan et al. Towards personalized federated learning. International Conference on Learning Representations, 2022.
12

---

[38] Jianyu Wang et al. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in Neural Information Processing Systems, 2020.
[39] Jae Hong Yoon, Young Gil Joo, and Sung Ju Lee, Fedmix: Approximation of mixup under mean augmented federated learning. arXiv preprini arXiv:2101.064480, 2021.
[40] Wencong Zhu al. Data-free knowledge distillation for heterogeneous federated learning. arXiv preprint arXiv: 10056, 2021.
13

---

A Supplementary
A.1 Flow Diagram Demonstrating the Computation of CKF
The flow diagram in Figure Iillustrates the process of computing Cluster Knowledge Fusion (CKF) within the Federated Learning (FL) framework. This process integrates feature extraction, Gaussian Mixture Model (GMM) clustering, and the synthesis of prototype features and soft predictions from clients. Following this, aggregation oceurs at the server using the Diserepancy-Awess Technique (DAT), facilitating the alignment of global knowledge with local client distributions.
Aggregation
CKI cfclintr
LE ICT
NETIORE
irglnn
diplan
Clicst
hind
daunniing
DAT
Aggragsted CKF
Server iggrepation
airpl-ase
Daxa
Chasköo
Sodl predictions
CMN drstiing
Proketpp
bonte
CXT
CKFafeleet
CIT whtwree
Figure Flow diagram demonstrating the computation of Cluster Knowledge Fusion (CKF) in Federated Learning. The diagram highlights the steps involved in extracting features, generating soft predictions, and performing GMM clustering to compute prototype features and predictions, followed by the aggregation of CKF at the server.
As shown in Figure [ the computation of CKF begins at each participating client after local training. Each elient, first fine-tunes its local model consisting of a feature extractor Fe, (-) and classifier Co,() The CKF computation for class occurs in three major steps: the feature extractor processes raw input data generate latent data representations for each class (c.g., bird, airplane, horse). These data representations capture the essential features of the input data at the client level. Using the generated data representations, GMM clustering is applied to compute the prototype features for each class. Concurrently, the classifier generates soft predictions for these data representations, which are also clustered using GMM derive prototype soft predictions for each class. After obtaining the prototype features and soft predictions for each class, they are aggregated to form the CKF The CKF encapsulates the most representative features and soft predictions for each class, thereby summarizing the local client knowledge in form that is robust and suitable for global aggregation at the server. In this way, CKF computation reduces the redundancy inherent in the client data while maintaining critical data characteristics, ensuring that local model updates are aligned with class slas-speccfii feature distributions. The CKF values are then transmitted to the server, where the DAT employed to merge the CKF across clients, resulting in a robust global CKF that reflects the diverse data distributions across the FL network.
A.2 Flow Diagram Demonstrating the Computation of DAT
The flow diagram in Figure Dillustailsss the process of computing the Discrepancy- Aggregation Technique (DAT) within the Federated Learning (FL) framework. This technique enhances the aggregation of Cluster Knowledge Fusion (CKF) from clients by incorporating both data quantity and quality, ensuring that the global CKF a robust representation aligned with diverse local client distributions.
As shown in Figure 2 the computation of DAT begins by calculating an initial weight for each client's CKF Each client's 's initial weight w for class j is derived based on the proportion of samples for that class relative to the total samples across all clients, reflecting each client's contribution in terms of data volume. This initial aggregation step combines CKF from all clients to generate preliminary
14

---

CKF of Clietss
Aguregated Soft
Calculation of Discrepancy of
Weights Aggregation
Prodictions by ui]
Client 1
bird
bird
Global
Initial weight wil
airplane
airplane
airplane
Client
horse
Figure 2: Flow diagram demonstrating the computation of Discrepancy-Awar Aggregation Technique (DAT) in Federated Learning. The diagram details the steps involved in computing initial weights, aggregating soft predictions, calculating discrepancies, and performing the final aggregation of CKF the server.
aggregated soft predictions. Following this, the server evaluates the aligmment between each client's local data distribution and the preliminary global soft predictions by calculating the Dullback-Leibbee (KL.) divergence d' for each client i and class j. This discrepancy measure serves quantify the quality of each client's CKF, indicating how closely each client's data aligns with the global distribution. Based on the calculated discrepancies, the server adjusts the initial weights, producing the final aggregation weights w for each client. These final weights balance the data volume and data quality, allowing clients with more aligned distributions to have greater influence in the aggregation process. Finally, the server aggregates the CKFs from all clients using the discrepancy-awe -waae weights w/. resulting robust global CKF for each class. This global CKF incorporates the essential knowledge from all clients while accounting for variations in data quality, effectively addressing the challenges posed by Non-IID data distributions in FL.
A.3 Visualization of FedGMKD
The flow diagram in Figure 3 illustrates the iterative process of CKF computation and model aggregation DAT within the FedGMKD framework. This visualization helps clarify the operational phases of training local models, extracting CKF, and DAT aggregating updates at the server, leading to refined global model.
In Figure 3 the FedGMKD process is visualized follows: During Global Round 1, clients (c.g.. Client i and Client n) train their local models using their private data and extract CKF from these then sent to the server. Upon receiving updates from all participating clients, the server employs the DAT to aggregate the local CKF and model updates, refining both the global CKF and the global model.
models. The local CKF and model updates
In subsequent rounds (e.g., Global Round through Global Round T), the server broadcasts updated global CKF and global model to the clients. Each client integrates the global CKF into its local training process, updating its model based on both the global CKF and its own local data. After training, the clients extract new CKF values from their updated models and send these back to the server for further DAT aggregation. This iterative process continues across multiple global rounds, progressively refining both the global model and CKF to account for the diversity and heterogeneity of client data, thereby improving the personalization and generalization of the federated learning model.
By leveraging CKF at each iteration, the FedGMKD framework ensures that the global model retains a balanced understanding of the local client data distributions while adapting to the variations in the feature and label spaces across different clients. This process addresses the challenges posed by mDD data, leading to more robust and pell-personalized federated learning system.
15

---

Globel I: model
Olibel round Trinn loal modkl by glotoi CKF
ciictal model by globsl
Extrat CKF local
Extrat CKP local
Caantidga
DAT
DAT
Server for aperecation
DAT
Global raund
Gicbal TTrm by plobal
Glibel romnd: by global
Intract CSC Usiuz local mad0el and data
Extrec CK. Using local modkl and chis
Exttact CKP Using loca. mockl and data
Cheat
local CKF local model updates
2lobal CKF+ model
Figure 3: Visualization of the FedGMKD framework. Each client trains local model and extracts CKF using its local data. The server aggregates the CKF and model updates using Discrepancy-Awar Aggregation Technique (DAT) to improve the global CKF and model. This process iterates over multiple global rounds.
A.4 Ablation Study
A.4.1 Handling Data Heterogeneity
To evaluate the robustness of FedGMKD in addressing data heterogeneity, conducted experiments on the CIFAR-10 10 and SVHN datasets under varying degrees Non-IID conditions, controlled by the Dirichlet distribution parameter B. A smaller 3 indicates greater heterogeneity in the data distribution, with clients typically possessing only one or two classes their local datasets; a larger suggests a more homogeneous distribution of data across clients. Specifically, when / 0.2, the local datasets exhibit high heterogeneity, whereas 3 = 5 produces nearly homogeneous datasets.
For these experiments, we utilized datasets from Torchvision, where the global test dataset was directly obtained by loading the standard test sets for CIFAR-10 and SVHN without any sampling. To create the local datasets, partitioned each dataset into clients (where denotes the number of clients) using Dirichlet distribution with parameter 3, Each client's ''s dataset was subsequently divided into a local training set and local test set in 75%/25% split. The results of these experiments, which compare the performance of various methods under 3 = 0.2 and 3 = 5 settings for both CIFAR- 10 and SVHN, summarized in Table 2
The results presented in Table Sablee@dooe demonstrate that FedGMKD consistently surpasses other federated learning techniques, especially under challenging heterogeneous data settings. Under the high heterogeneity scenario (3 = 0.2), FedGMKD exhibits notable improvements. On CIFAR-10, achieves the highest local accuracy of 74.34%, representing an increase of 3.4% over FedAvg, 1.93% over FedProto, and 2.26% over FPL. Moreover, FedGMKD attains the best global accuracy of 44.79%. outperforming FedProto and FPL by 7.43% and 2.72%. respectively. These significant improvements emphasize the ability of CKF and DAT to handle highly heterogeneous client data while capturing essential features. When the data is more homogeneous (3 5), FedGMKD maintains its superior performance. For instance, achieves the highest global accuracy of 53.84% on CIFAR-10, outperforming FedAvg by 1.12% and FPL by 0.19%, showcasing its effective model alignment even in near- haaa-hooggeeouus settings.
On the SVHN dataset, FedGMKD also demonstrates its robustness across varying heterogeneity levels. Under the high heterogeneity setting (3 0.2), it achieves a local accuracy 87. 27.29%, outperforming FedAvg by 1.33% and FedProto by 0.7 17.. In terms of global accuracy, FedGMKD
16

---

Table 2: Performance of different schemes CIFAR-10 and SVHN datasets under various data heterogeneity settings controlled by Dirichlet distribution parameter J.
Scheme
CIFAR-10
SVHN
Local Ace
Global Ace
Local Ace
Global Acc
8 0.2
B =5
8 = 0.2
/ = 5
8 = 0.2
3 == 5
B = 0.2
3 = 5
FedAvg
70.94%
43.85%
44.78%
52.72%
85.96%
79.55%
55.46%
86.28%
FedProx
71.15%
43.72%
44.59%
52.56%
86.08%
80.01%
56.36%
86.41%
FedMD
71.56%
45.91%
43.84%
52.95%
87.31%
79.74%
72.89%
86.29%
FedGen
71.32%
42.08%
40.64%
51.61%
85.45%
78.71%
64.42%
86.01%
Moon
71.65%
44.77%
43.43%
52.11%
85.66%
78.91%
66.65%
86.37%
FedProto
72.41%
46.12%
37.36%
52.81%
86.58%
79.84%
69.31%
86.81%
FPL 72.08%
47.11%
42.07%
53.65%
85,49%
79.82%
67.72%
86.02%
FedGMKD
74.34%
46.81%
44.79%
53.84%
87.29%
80.41 %
70.11%
86.94%
shows substantial improvement of 14.65% over FedAvg, with global accuracy of 70.11% Under the more homogeneous setting (3 = 5), FedGMKD continues to deliver top performance, achieving local accuracy of 80.41% % and global accuracy of 86.94% outperforming FedProto by 0.57% in local accuracy and by 0.13% global accuracy.
These results underscore the effectiveness of FedGMKD in addressing both extreme heterogeneity and more homogeneous environments. Its ability to consistently improve both local and global accuracy across different datasets highlights the robustness of the CKF and DAT techniques. By leveraging these mechanisms, FedGMKD captures high-quality local model representations while optimizing the aggregation process, ensuring better personalization and improved global performance in federated learning.
A.4.2 Evaluation of Component Effectiveness in FedGMKD
In this section, we perform an ablation study to assess the contributions of the CKF and DAT within our FedGMKD framework. We compare the performance of FedGMKD with its Buriants-FedGMe using only CKF and FedGMKD using only SVHN datasets under different data heterogeneity settings, controlled by the Dirichlet distribution parameter The results are presented in Table
and a baseline method, FedAvg, CIFAR-10 and
Table 3: Performance of different schemes on CIFAR-10 10 and SVHN datasets under various data heterogeneity settings controlled by Dirichlet distribution parameter 3.
Scheme
CIFAR-10
SVHN
Local Acc
Global Acc
Local Acc
Global Acc
B 0.2
B 5
3 ===00 0.2
B = 5
B = 0.2
B
3 0.2
B 5
FedAvg
70.94
43.85
44.84
52.72
85.96
79.55
55.46
86.28
FedGMKD (Only CKF)
73.74
46.06
41.66
53.07
86.86
80.18
69.02
86.35
FedGMKD (Only DAT)
73.39
42.42
45.65
52.33
86.41
79.84
69.93
86.45
FedGMKD
74.34
46.81
44.79
53.84
87.29
80.41
70.11
86.84
Table shows that FedGMKD, incorporating both CKF and DAT, consistently outperforms baseline FedAvg method under various data heterogeneity conditions. On the CIFAR-10 10 dataset with high heterogeneity (3 = 0.2), FedGMKD achieves an improvement of approximately 3.4% in local accuracy over FedAvg, while the variants utilizing only CKF and only DAT show improvements of 2.8% and 2.45%, respectively. Similarly, on the SVHN dataset under the same high heterogeneity, FedGMKD achieves 1.33% increase in local accuracy and a substantial 14.65% increase in global accuracy over FedAvg. The CKF-only and DAT-only variants also show significant improvements over FedAvg.
For the CIFAR-10 10 dataset with high heterogeneity (8 = 0.2), FedGMKD using only CKF achieves a local accuracy of 73.74% while FedGMKD using only DAT reaches global accuracy of 45.65%. These results indicate that CKF plays crucial role in improving local model accuracy by effectively leveraging client-specific data characteristics, whereas DAT ensures that local improvements are
17

---

properly aggregated into the global model. On the SVHN dataset, CKF alone results in a local accuracy of 86.86% and global accuracy 69.02%, outperforming the DAT-only variant in local accuracy but not in global accuracy. This suggests that CKF is particularly beneficial in highly non-III settings where local data characteristics are vital for model performance.
Nevertheless, the full integration of both CKF and DAT in FedGMKD consistently yields the highest performance across all metries and settings. For example, on the SVHN dataset with 0.2. FedGMKD achieves a local accuracy of 87 37.29% and a global accuracy of 70. 1%, outperforming both the CKF-only and onT- variants. This demonstrates that CKF and DAT work synergistically to enhance both local and global model performance, making FedGMKD robust and effective in diverse data environments.
A.4.3 Impact of Model Complexity
In addition to our previous experiments, we conducted an ablation study to assess the impact model complexity by using the ResNet-50 architecture in comparison with ResNet-18 on the CIFAR-10 dataset. The goal was to explore how a deeper network affects local and global model performance the performance of FedGMKD and other
in the context of federated learning. baseline methods using both ResNet- and ResNet-50.
Table 4: Comparison of performance for various schemes on CIFAR-10 using ResNet-18 and ResNet-50 architectures.
Scheme
Acc (ResNet-18)
Acc (ResNet-50)
Local Ace
Global Acc
Local Acc
Global Acc
FedAvg
61.78
49.78
41.69
49.58
FedProx
64.04
55.16
43.25
49.67
FedMD
62.05
53.73
43.34
49.85
FedGen
60.17
51.55
42.81
48.99
FedProto
62.85
50.88
43.35
49,98
Moon
62.74
52.04
42.05
48.52
FPL
62.74
52.04
43.71
49.78
FedGMKD
65.69
60.31
46.27
50.48
As shown in Table 4 FedGMKD consistently outperforms other federated learning methods under both ResNet-18 and ResNet-50 configurations. However, there are noticeable differences in per- formance between the two architectures. For ResNet-50, while FedGMKD achieves the highest local and global accuracies among all methods, the performance does not exceed that of ResNet- 18. Specifically, FedGMKD achieves local accuracy of 46.27% and a global accuracy of 50.48% with ResNet-50, compared to 65.69% and 60,31 %, respectively, with ResNet-18. The primary reason for the reduced performance of ResNet-50 in comparison to ResNet-18 lies in the increased model com- plexity and the corresponding challenges in federated learning environments. Deeper networks like ResNet-50 require more training epochs and greater communication bandwidth between the server and clients due to the larger number of parameters. The federated learning framework, particularly under constrained network conditions and limited communication rounds, may not able to fully exploit the representational power of ResNet-50. Additionally, deeper models tend to require more iterations to converge, and given the fixed number global communication rounds, ResNet-50 might not have had sufficient time to fully optimize in this federated setting.
While ResNet-50 offers higher theoretical representational power, practical considerations such as communication efficiency and the number of training rounds needed to achieve convergence limit its effectiveness in federated learning. FedGMKD with ResNet-50 still outperforms other schemes, demonstrating the robustness of our framework. However, the results suggest that balancing model complexity with communication and computational constraints is crucial in federated settings. Future work may explore strategies to mitigate these challenges, such as adaptive aggregation techniques and personalized model training strategies, to further improve the performance deeper models like ResNet-50 in federated learning.
18

---

A.4.4 Hyperparameter Exploration in FedGMKD
This section explores the effect of varying the regularization coefficients À and within the FedGMKD framework on the CIFAR-IU dataset with 10 clients over 50 epochs. and global accuracy for different combinations of À and 7, while also providing baseline comparisons with FedAvg, FedProto, and FPL.
the local
Table FedGMKD performance with varying and y values on CIFAR-10 dataset (10 clients, 50 epochs).
Scheme
?
Local Acc (%)
Global Ace ( (%)
FedGMKD
0.06
0
60.14
48.17
0.06
0.02
60.32
49.44
0.06
0.04
60.99
49.48
0.06
0.06
61.78
49.98
0.06
0.08
60.27
48.64
0.06
0.10
60.14
49.97
0.06
60.73
48.72
0.02
0.06
60.33
48.21
0.04
0.06
61.01
49.47
0.08
0.06
60.93
49.69
0.10
0.06
59.86
47.52
FedAvg
55.75
46.62
FedProto
59.77
48.97
FPL
60.95
47.19
As shown in Table FedGMKD demonstrates strong performance across range of and y values. The highest local and global accuracies are achieved with À = 0.06 and 0.06, resulting in a local accuracy of 61.78% and a global accuracy of 49.98%.
FedGMKD consistently outperforms the baseline methods FedAvg, FedProto, and FPL across all configurations, particularly at the optimal setting accuracy improvement of 6.03% and a global accuracy improvement of 3.36% over FedAvg. This indicates that FedGMKD's 's CKF and DAT mechanisms effectively handle Non-IID data and improve both local and global model performance.
0.06, 7 0.06), where it achieves local
The sensitivity analysis indicates that while FedGMKD performs well across different settings, fine- tuning and is crucial for maximizing performance. The robustness of FedGMKD is highlighted by its stable accuracy across various configurations, though the results suggest that balancing the feature alignment (A) and prediction alignment (7) terms is key to optimizing performance. This balance allows FedGMKD maintain consistency between local and global models and handle diverse data distributions effectively.
A.4.5 Comparison Between Hyper-Knowledge Averaging with DAT and FedGMKD
In this section, we present a comparison between an approach that utilizes the syper-knowledge concept from BeddiKKD-peciifclllll averaging features and soft predictions our proposed FedGMKD framework. Both methods incorporate Discrepancy-Awe (DAT) across CIFAR-10, SVHN, and CIFAR-100 datasets with varying numbers of clients. Table summarizes the local and global accuracy results for both methods, demonstrating the differences in their effecttieness when applied to non-lID federated learning scenarios.
Aggregation Technique
As shown in Table FedGMKD consistently outperforms the method based on hyper-knowledge averaging with DAT across all datasets and client configurations. The GMM--based approach in FedGMKD allows for the extraction of more precise prototype features and soft predictions by clustering data points according to their similarity. This clustering technique captures the inherent structure and diversity within the data better than simple averaging of features and soft predictions derived from syper-knowledge By leveraging these GMM-derived prototypes, FedGMKD provides a richer and more representative set of features for aggregation, leading to superior performance both local and global metrics.
19

---

Table 6: Comparison of Hyper-Knowledge Averaging with DAT and FedGMKD CIFAR- 100 SVHN, and CIFAR-100 datasets with B = 0.5.
Dataset
Clients
FedGMKD Acc (%)
HK with DAT Acc (%)
Local
Global
Local
Global
CIFAR-10
10
61.78
49.78
60.33
48.96
64.04
55. 16
62.31
52.67
50
65.69
60.31
64.23
58.64
SVHN
10
86.26
82.64
85.91
81.45
20
87.43
87.78
86.75
87.69
50
87.16
90.17
86.98
89.78
CIFAR-100
10
17.16
16.97
16.12
16.09
20
20.96
21.56
19.71
20.69
50
23.57
24.63
22.44
23.82
Furthermore, while both methods use the discrepancy-awart aggregation technique (DAT) weight client contributions, FedGMKD's s clustering- based feature extraction aligns more closely with client data distributions. This alignment ensures that the global model benefits from more accurate and rel- evant integration of client data, particularly in highly heterogeneous settings. The close performance in global accuracy between the two methods in some configurations shows that DAT effectively mitigates client discrepancies; however, the advantage of using GMMs for prototype generation becomes evident as FedGMKD achieves consistently higher global accuracy, demonstrating the effectiveness of its integrated CKF and DAT mechanisms.
A.4.6 Evaluation on IMDB Dataset
To further assess the adaptability of FedGMKD across different data modalities, we conducted experiments on the IMDB (Internet Movie Database) dataset, a prominent benchmark in natural language processing (NLP) used for sentiment analysis tasks. The IMDB dataset comprises large collection of movie reviews, presenting a significant challenge for federated learning due to the inherent diversity of client data distributions. Sentiment analysis in particular requires models to generalize effectively across these distributed datasets.
The experimental setup mirrored the structure of previous evaluations, utilizing 10 clients over the course of 50 training epochs. As the task shifted from computer vision to NLP, the model architecture was updated from ResNet- 18 to BERT (Bidirectional Encoder Representations from Transformers), model widely recognized for its effectiveness text-based tasks like sentiment analysis. BERT's ability to capture contextual embeddings makes it particularly well-suited for handling the complexity of the IMDB dataset. Evaluation metrics included local accuracy, global accuracy, and the average computation time per client.
Table 7: Performance of different schemes on IMDB dataset using BERT model (10 clients, 50 epochs).
Scheme
Local Acc (%)
Global Acc (%)
Avg Time (s)
FedAvg 83.71
50.52
411.95
FedProx
83.75
48.50
438.52
FedMD
83,87
48.29
700.73
FedGen
83,54
49.16
471.35
FedProto
84.13
49.72
586.77
FPL
83.96
50.12
665.29
FedGMKD
85.11
51.58
677.79
Table 7 shows that FedGMKD consistently outperforms other federated learning algorithms in terms of both local and global accuracy on the IMDB dataset. FedGMKD achieves the highest local accuracy of 85.11% and the highest global accuracy of 51.58%, indicating its robustness in handling
20

---

non-vision datasets. In particular, the results highlight the effectiveness of CKP and DAT in adapting text-based tasks. While the average computation time for FedGMKD slightly higher due the complexity of BERT, the gains in both local and global accuracy justify the computational overhead. These findings underscore the versatility of FedGMKD, demonstrating that can generalize well to NLP tasks and effectively handle different data modalities, further validating its broader applicability across federated learning scenarios.
A.5 Feature Representations
In this section, we present an ablation study comparing the t-SNE visualizations of feature representa- tions obtained using different federated learning methods: FedAvg, FedProto, FPL, and FedGMKD. These experiments were conducted with 10 clients and a data heterogeneity parameter (3) of 0.5 on the CIFAR- 10 dataset. The primary aim is to evaluate how effectively each method learns to separate feature representations by class, as visualized using t-SNE.
LAA
Vie
(a) FedAvg
(b) FedProto
(c) FPL
(d) FedGMKD
Figure 4: Qualitative comparison of t-SNE visualization among FedAvg, FedProto, FPL and FedGMKD. Compared with other methods, the feature distribution of the FedGMKD is more compact within each category, and more discriminative across classes.
In the FedAvg method (Figure 4a), feature representations are widely dispersed with significant overlap between classes. Although FedAvg captures some class-specife features, the resulting feature clusters are neither distinct nor compact, indicating weak separation between the classes. This is expected, as FedAvg primarily averages model parameters without specific focus class differentiation or feature compactness.
The FedProto method (Figure 4b), while not explicitly clustering features, introduces prototype- based learning where class prototypes guide the learning process. The t-SNE visualization shows slightly more distinct feature distributions compared to FedAvg, reflecting improvement in feature separation. However, since FedProto emphasizes prototyping over clustering, some overlap between class features still persists, suggesting room for more effective inter-class separation.
21

---

FPL (Figure 4p) shows clustering behavior very similar to FedAvg, with large overlaps between clusters and less distinct boundaries. The feature separation is minimal, and the clusters are not as compact as those in FedProto. This indicates that FPI. struggles to produce clear, discriminative class-specific features, making its clustering quality less ideal compared to FedProto.
FedGMKD (Figure 4)) demonstrates the most compact and well-separated feature representations. The t-SNE visualization shows that FedGMKDDss CKF successfully generates class-specific prototypes that contribute to tight and distinct feature clusters, This clear separation highlights how CKF, along with the DAT, enhances feature learning across clients, leading to superior class separation and making FedGMKD the most effective method in terms of feature representation quality.
A.6
Convergence Analysis of FedGMKD
A.6.1 Assumptions
To prove the convergence of FedGMKD, we introduce the following assumptions:
Assumption (Lipschitz Continuity of the Local Loss Function): The local loss function L(D,, w() at each client is L-smooth, meaning that its gradient is Lipschitz continuous with a constant > 0. Formally, for any w w we have:
IVL(D; w() VL(D, wills L||w; - -
(16)
This ensures that the local loss function is smooth, and the gradient changes gradually for any two parameter vectors.
Assumption 2 (Unbiased Gradient and Bounded Variance): The gradient estimation at each client is unbiased with bounded variance. Specifically, for any w/, the expectation of the local gradient equals the true global gradient at the global parameter w and the variance of the gradient bounded by constant ²²:
L w!)] = VL(W
(17)
and
LD, w() VL(W [l" < 0².
(18)
Assumption 3 (Bounded Gradient Norm): The gradient of the local loss function L(D,, w!) is bounded. There exists constant G' > 0 such that:
I[,, w!llS Vi,r.
(19)
Assumption 4 (Bounded Discrepancy in KL Divergence): The Kullback-Leible (KL) divergence between the local model's predictions and the global model's s predictions for each class is bounded. For any client and class j. the KL divergence de between the local soft prediction of and the global soft prediction 2 satisfies:
= DRI (áf II 11),) D
(20)
This ensures that the discrepancy between local and global models is controlled.
Assumption (Cluster Knowledge Fusion Convergence): The Gaussian Mixture Model (GMM)- based clustering used in Cluster Knowledge Fusion (CKF) converges stable clustering of client updates. Let M be the number of Gaussian components (to avoid conflict with global round notation),
and let 7m (x() represent the responsibility of the m-th Gaussian component for a data point x/. After sufficient number of iterations, we assume:
M
M
Σ
Y'm (x)) =1 and 14
Σ
Fim (h2) m.-
(21)
A.6.2 Lemma
Lemma 1
Lemma 1: Under Assumptions A.6.1 after R local training epochs at global round r + 1, the local loss function can be bounded as follows:
RG
11002
E[L,D,, w + +*)]+ < L(D,, w() V
2 +
(22)
22

---

where L is the Lipschitz constant from Assumption 1, o² is the variance bound from Assumption 2, is the bound on the gradient norm from Assumption and rj the learning rate.
Proof:
In each communication round client i performs R local updates based on stochastic gradient descent (SGD) on its local dataset D. The local training objective for client is defined as:
L(D,, ws)
Σ
E(C,, (Fo, (2%)), 9kx
Diil
(ED.
(23)
C
(H
gr+1
Σ
Fo. (xx)
H yk
7 n
[D:]
T
First, we use the Lipschitz continuity of the gradient (Assumption 1). This assumption implies that the loss function L(D;, w) is L-smooth. Formally:
L L(Di, wi) & L(D,, w2) + VL(D,, w2) W2) + 2 /w: w2
(24)
We set W1 - w/ w/ and W2 w +i Substituting these into the Lipschitz continuity condition:
L(D,, w; ) LD; w; +)) + L(D,,, w/ (w; r+tt1 -w; w,+ + Iw +J+1
W
(25)
The SGD update rule is given by:
w - w/ - w/t),,
(26)
where 1) is the learning rate. Substituting this into the previous equation:
L(D;, w/ w < L(D:,, w VII w/ |1 Ln²
IVL(D, 21
(27)
2
To analyze the expected change in the loss, we take the expectation on both sides with respect the randomness of the gradient:
[(D,, w [ & &E [[D. -n ( w
Ln² E [VI w | 2
(28)
According to Assumption 2 (Unbiased Gradient and Bounded Variance), the gradient is an unbiased estimator of the true gradient, and its variance bounded:
E LD,, + VL(W)]P] o
(29)
We decompose the expected squared gradient norm as follows:
E [IVL
[P]] = IIV,,D, +/ Var (VL(D,, w/th))
(30)
Thus:
E |VV((,, + < [[V,(D, w. t
(31)
To understand the cumulative effect of R local updates, we sum the inequalities over j 0 to
j = R-11 1:
R
Lnd R-1 V[ w |.
E[L(D,, ([L(DD,,/+R)) < L(D;, w!) BII
w/t)))
1=0
1=0
(32)
23

---

From Assumption 3 (Bounded Gradient Norm), [VV( wا G for all j. Substituting this
bound:
R-1
LIIV
R(G +
(33)
==0
Substituting this result into the inequality:
7 VL, w L +
BIL(D;, W[ x] S L(Di, w()
(34)
Lemma 2
Lemma 2: Under Assumptions 1-5 A.6.1
the global loss function after one global aggregation step
round 1 can be bounded
Ln
W [IVF + La
E[F(Wr+ S F(W") nR
(35)
where F(W) is the global loss function, (ww::) is the local loss function for client i, w/ are the discrepancy-aware weights, and d² the variance bound from Assumption 2.
Proof:
The global model after aggregation at round + 1 is computed as:
W
w/ w +
(36)
where w/ are the diserepancy-aware weights based on each client's data quality and quantity, and w represents the model parameters after /t local updates for client i.
We express the global loss function at time + in terms of the aggregated client models:
F(W Σ
(37)
where RR the local loss function value for client after R local updates.
From Lemma 1, we have:
[[((W++)) & Fi(w)) RRRE [[VF + Lap R(G + +
(38)
2
where P represents the cosine similarity between the gradients and L, o and G are constants derived from the assumptions.
Taking the expectation over all clients and substituting the bound from Lemma 1:
B[F +1)
Σ W [F(w R]]
(39)
Substituting the inequality from Lemma into this expression:
Lnd R(G² +02)
B[F( < m F((w)) RRpE VV
(40)
2
Recall that the global loss function at time r is:
F(W") = Σ w.Fl(w)).
(41)
1=1
24

---

Substituting this into the inequality:
B[F(W*th) F(W") nRp [IVE + LI i= wi(G² +
(42)
2
According to Assumption 2 (Unbiased Gradient and Bounded Variance), can further refine the variance term:
u o²
(43)
since the weights vd sum to 1. Therefore:
n1
D1 + Lij
E[ s F(W") RR > wBB VI
(44)
The factor p represents the alignment of the gradient directions across clients. In practice, p can vary, but for worst-case analysis, set p 1 to provide conservative bound. Substituting p 1:
Lijo
B[F(WT S F(W)) nR
w [[VF)(())]]
(45)
We rearrange the expression to factor out the learning rate 7) and the Lipschitz constant L:
Lnj
n
+ Liio0²
)|P((rt)) F(W") - n/R
(46)
A.6.3 Theorem
Theorem 1: FedGMKD Convergence
Under Assumptions 1-5 for any client i, after Rt global communication rounds, the expected global loss function bounded as:
w/ [IVE F(W') F + LMRG²
(47)
R2
where F(W) is the global loss function, F is the lower bound of the global objective function, is the learning rate, R is the number of global updates, L is the Lipschitz constant from Assumption 1, o² is the variance bound on the gradient from Assumption 2, and G is the upper bound on the gradient norm from Assumption 3.
As R + co, the global loss function converges to a neighborhood around a stationary point, with the size of the neighborhood bounded by:
LnRG²
0² + 2
(48)
Proof:
The global model update after aggregation at round r + is defined as:
W w +
(49)
where ii are the discrepancy-aware weights, and w + are the parameters of client after It local updates. The global loss function after aggregation is:
n
F(Wr+1)= Σ
(50)
25

---

where F + R) denotes the local loss function of client after R local updates.
From Lemma 2, we know:
Ln
Inpo
B F " F(W") -nR [1-
B VI
[1 +
(51)
Let A, denote the change in the global loss function:
A = F(W") B[P(W-)11
(52)
Substituting the result from Lemma 2:
(1
L
Lnpo²
A, 2 nlt
w [VP((W))))
(53)
2
2
This inequality indicates the amount by which the global loss function decreases in expectation after one round of global aggregation.
We sum the inequality A, 2 ... over r = 1 to R:
Σ A. F(W") -[(F(WRt)) -W)--
(54)
Substituting the inequality for A,:
Lnj
R
LIPo²R
- - D(F(EE(FHHHHH (
nR
Σ
W/ [[((1)
(55)
2
Rearrange the inequality to isolate the sum of the expected gradient norms:
Ln w E VF < -
LIpo²R
nR 1
E[F(WR+I))
(56)
2
2
To find the average over rounds, divide both sides by nR 1 -
In
R:
2
R
(VP((W)) F(W1) [E (W
Lnoz
(57)
nR 1 - Im
R
2
In
Assuming that F(W) is bounded below by F", such that:
F(WR+1) 2 F*,
(58)
we substitute this bound into the inequality:
F(W!)
F
Lnp?
w/ [IVE
(59)
nR
1
La
2
Ln
2
2
F tends zero. Therefore:
As R + co, the term
nR(1 - jR
lim Ro
1
ΣΣ EH [VF(()))))
(60)
2
Lg
Since 7) chosen such that La > O. this shows that the global loss function converges to
2
neighborhood around stationary point with the size of the neighborhood given by:
02 + LnRGa
(61)
26

---

Theorem 2: FedGMKD Convergence Rate
Under Assumptions 1-5 A.6.1 for any client i, after R glohall communication rounds, the convergence rate the global loss function F(W) is bounded as follows:
F(W") F
R C2,
(62)
where F(W) is the global loss function, F represents the lower bound F(W), and C1 and C2 are constants that depend on variance o², Lipschitz constant learning rate 77, and the number of local steps.
As R + co, the global loss function converges to neighborhood around stationary point.
Proof:
To establish the convergence rate of F(W), we begin with Lemma
Ly 2
Lapo0
E [[F(W+))) S -
w V(I
(63)
Define the decrease in the global loss function after one global round as:
D F(W")
[[[(("")))
(64)
Substituting the result from Lemma 2:
Ln 2
Ln7²0² 2
A 2n n
(65)
This equation indicates that the decrease the global loss function F(W) after each round is proportional the squared gradient norm and controlled by the variance and Lipschitz constant.
We sum A, over R:
Σ A, AA,=((W)) F[
(66)
Substituting the inequality for A;:
Lıj
LN2o2R
F(W') - 1)
(67)
This inequality now relates the total decrease in the global loss function to the sum of the squared gradients over all rounds.
We rearrange the equation to isolate the sum of the expected gradient norms:
(1
Ln
w [/B]]///(W))) < F(W') F" + LITo²R
(68)
2
Dividing by
Lg
R We divide both sides of the inequality by - Lg L
R to find the
average squared gradient norm over Rt rounds:
R
F(W1)
F
Lno²
(69)
R
Lu
2
Lg
2
Define C
((W*)-1 F-F" 1(1-
and
C2
Lng² im
Substituting these constants:
2(1
C₁
+
Cz.
(70)
R
R
27

---

As R - oo, the term Cz.
C1
+ 0. This indicates that the average squared gradient norm converges to R
Given that C2 is bounded, the behavior of F(W) over R rounds implies:
Ci R
F(W F
C2.
(71)
This shows that the global loss function F F(W) converges at rate proportional to 1 asymptotic constant C2 that defines the neighborhood around the stationary point. R. plus an
As R + 00, the term C₁ diminishes, and the global model converges to neighborhood around the stationary point defined by C2.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly outline the contributions of FedGMKD, including the introduction of Cluster Knowledge Fusion (CKF) and Differential Aggrega- tion Technique (DAT). These claims are supported by theoretical analyses and extensive experimental results detailed in Sections and 3.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors? Answer: [Yes]
Justification: Section 4 and supplementary A.4.3 discuss potential limitations, such as the computational overhead associated with CKF and DAT, and the scalability of FedGMKD in extremely large federated networks.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The theoretical results are fully detailed in Section 3.6, with all necessary assumptions and complete proofs provided in the supplementary material A.6.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Section 4 provides comprehensive details on the experimental setup, including datasets, models, baselines, and hyperparameters. Additional details and scripts are provided in the supplementary material ensure reproducibility.
Justification:
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We have provided code and data in the supplementary material, along with detailed instructions for reproducing the experiments.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Section 4.4 specifies all relevant training and testing details, including data splits, hyperparameters, and optimizer settings.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The paper does not report enror bars or statistical significance tests. The primary focus was on comparing the performance metries (e.g., accuracy) across different methods without including statistical analysis of the results.
29

---

8. Experiments Compute Resourees
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Section 4 specifies all relevant training and testing details, including data splits, hyperparameters, and optimizer settings.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://p///aaaap...oooo/o//////a
Answer: [Yes]
The research adheres to the NeurIPS Code of Ethics, ensuring data privacy and ethical conduct throughout the study.
Justification:
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Section discusses the broader impacts of the proposed method, highlighting potential positive applications in privacy-presed preserving machine learning and possible negative impacts, such computational overhead.
Justification:
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This research does not involve the release of high-risk models or datasets.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets used are properly credited, and their licenses are respected, as detailed in Section and the supplementary material.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets introduced in this research are thoroughly documented, with detailed instructions provided in the supplementary material.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This research does not involve crowdsourcing human subjects.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
30

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This research does not involve human subjects.
31


Reviews:
[{"review_id": "QITGh4mzQa", "reviewer": "Reviewer_KnUX", "summary": "The authors proposed FedGMKD, a federated learning framework designed to handle data heterogeneity across distributed clients. FedGMKD introduces Cluster Knowledge Fusion (CKF), which uses Gaussian Mixture Model (GMM) clustering to generate prototype features and soft predictions, facilitating knowledge distillation without public datasets. Additionally, the Differential Aggregation Technique (DAT) tailors the aggregation process to distinct feature distributions, optimizing both efficiency and performance. Experiments demonstrate that FedGMKD significantly improves both personalized and global model performance, achieving state-of-the-art results in heterogeneous data scenarios compared to traditional federated learning methods.", "questions": "I’d like the authors to answer my above three questions in the rebuttal.", "limitations": "The authors discussed the limitations in section 4.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "•\tThe proposed method addresses the critical issue of data heterogeneity in federated learning, a key factor for enhancing the applicability and robustness of federated learning in practical scenarios.\n•\tThe introduction of Cluster Knowledge Fusion (CKF) and Differential Aggregation Technique (DAT) offers a novel and effective approach. These techniques improve local training and aggregation processes without the need for public datasets, thereby significantly enhancing both personalized and global model performance.\n•\tThe experiments are meticulously designed and executed using benchmark datasets (SVHN, CIFAR-10, and CIFAR-100), demonstrating substantial improvements over existing methods. The results are compelling, underscoring the robustness and efficiency of the FedGMKD framework.\n•\tThe paper is well-structured, providing a clear explanation of complex concepts such as CKF and DAT. The inclusion of diagrams and detailed descriptions aids in comprehending the methodologies and results. The review of existing literature is thorough, establishing a solid context for the contributions.\n•\tThe theoretical analysis delivers strong mathematical guarantees for the convergence and performance of FedGMKD, bolstering the credibility of the research.", "weaknesses": "•\tWhile the evaluation demonstrates the potential of FedGMKD, it would be more convincing if it included a broader range of neural network architectures, particularly larger and more complex models like Transformers or ResNet-50.\n•\tThe paper does not provide sufficient details on how FedGMKD could be adapted to datasets with different characteristics from those tested (SVHN, CIFAR-10, CIFAR-100).\n•\tOverall, I’d like to see a discussion about what could be the good impacts/applications based on the technique proposed in the real world.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback. We appreciate the opportunity to discuss FedGMKD with more complex neural network architectures, its adaptability, and its real-world impacts and applications.\n# 1. FedGMKD with More Complex Neural Network Architectures\n\nTo address your suggestion, we conducted additional experiments using the ResNet-50 model on the CIFAR-10 dataset. Below is a summary of the results compared with our original experiments using ResNet-18:\n\n| Scheme    | Local Acc (ResNet-18) | Global Acc (ResNet-18) | Local Acc (ResNet-50) | Global Acc (ResNet-50) |\n|-----------|-----------------------|------------------------|-----------------------|------------------------|\n| FedAvg    | 61.78                 | 49.78                  | 41.69                 | 49.58                  |\n| FedProx   | 64.04                 | 55.16                  | 43.25                 | 49.67                  |\n| FedMD     | 62.05                 | 53.73                  | 43.34                 | 49.85                  |\n| FedGen    | 60.17                 | 51.55                  | 42.81                 | 48.99                  |\n| FedProto  | 62.85                 | 50.88                  | 43.35                 | 49.98                  |\n| Moon      | 62.74                 | 52.04                  | 42.05                 | 48.52                  |\n| FPL       | 62.74                 | 52.04                  | 43.71                 | 49.78                  |\n| FedGMKD   | 65.69                 | 60.31                  | 46.27                 | 50.48                  |\n\nWhile ResNet-50 theoretically offers higher representation power, practical challenges like increased model complexity, potential overfitting, and greater communication requirements limit its performance in federated settings. FedGMKD with ResNet-50 still outperforms other schemes but does not surpass the results with ResNet-18. These findings highlight the importance of balancing model complexity with the realities of federated learning environments. In future work, we plan to explore advanced aggregation techniques and personalized strategies to mitigate these challenges and fully harness the capabilities of complex models like ResNet-50.\n\n# 2. Adaptability of FedGMKD\n\nFedGMKD is designed with a modular architecture that enhances adaptability to diverse data distributions and experimental settings through its use of Gaussian Mixture Models for clustering and flexible parameter customization.\n## a. Modular Design of FedGMKD\n\n- **Feature Extraction and Clustering:** FedGMKD utilizes GMM for clustering, which can be adapted to various datasets by adjusting the number of Gaussian components according to the dataset's complexity (Section 3.2).\n- **Differential Aggregation Technique (DAT):** DAT weights client contributions based on data quality, making aggregation adaptable to different distributions and class imbalances (Section 3.3).\n\n## b. Handling Diverse Data Distributions\n\n- **Customizable Hyperparameters:** Parameters such as clustering centres, learning rates, and temperature for knowledge distillation are adjustable for different datasets. Our experiments (Section 3.7) showed flexibility for SVHN, CIFAR-10, and CIFAR-100.\n- **Adaptive Clustering:** GMM models data as a mixture of Gaussian distributions, adapting to varying distributions and complexities by adjusting Gaussian components and clustering settings.\n\n## c. Detailed Experimentation with Different Datasets\n\n- **Experimental Settings:** The `option.py` file in FedGMKD's code allows customization of parameters like the number of clients, classes, learning rates, and epochs. \n\n# 3. Real-World Impacts and Applications of FedGMKD\n\nFedGMKD enables impactful applications across healthcare, finance, and smart cities by enhancing data integration, privacy, and communication efficiency.\n\n## a. Healthcare\n\n- **Improved Local and Global Performance:** FedGMKD can significantly improve diagnostic models by integrating diverse medical data from multiple institutions. This results in more accurate and comprehensive predictions, enhancing both local models tailored to specific patient populations and a robust global model.\n- **No Need for Public Data:** Hospitals can train collaborative models without the need for publicly available medical datasets, ensuring compliance with privacy regulations and safeguarding patient data.\n- **High Communication Efficiency:** The efficient communication protocol ensures that even hospitals with limited bandwidth can participate in federated learning, making advanced diagnostic tools accessible across a wide range of medical facilities.\n\n## b. Finance\n\n- **Enhanced Fraud Detection:** FedGMKD improves the performance of fraud detection models by aggregating diverse transaction data from various financial institutions. This collective intelligence helps in identifying complex fraud patterns more accurately.\n- **Privacy Preservation:** Financial institutions can collaborate without exposing sensitive customer information, maintaining data confidentiality and complying with stringent financial regulations.\n- **Efficient Aggregation:** The differential aggregation technique ensures efficient use of communication bandwidth, which is crucial for real-time fraud detection and quick response to fraudulent activities.\n\n## c. Smart Cities\n\n- **Optimized Resource Management:** FedGMKD can enhance traffic management and energy consumption models by integrating data from different city departments, leading to more efficient use of city resources and improved quality of life for residents.\n- **Scalable Solution:** The high communication efficiency allows the deployment of smart city solutions across multiple, distributed sensors and devices without overloading the network, making it scalable for large urban environments.\n- **Handling Heterogeneous Data:** FedGMKD effectively integrates diverse data sources for comprehensive urban planning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NedhF607VP", "reviewer": "Reviewer_8XP3", "summary": "This paper introduces FedGMKD to tackle data heterogeneity by CKF and DAT. Specifically, to get a prototype of each class at each client, CKF is proposed by GMM and aggregates prototypes of the same class from different clients via discrepancy-aware weight.", "questions": "- How to calculate the  Discrepancy if some clients lack some class, e.g., Client 1 has no prototype of class 1. This is also important in a real FL setting.\n- The detailed FL setting is missing. ‘’The participating rate of clients was set to 1“ is not real in the FL setting.", "limitations": "See weakness and questions.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "This paper analyzes convergence and convergence rates mathematically.", "weaknesses": "- This paper lacks the motivation and insights to present this method. I can’t catch the idea of FedGMKD tackling the stragglers which is a problem in pFL presented by authors.\n- The contribution of this paper is limited.\n- The organization, writing, and presentation should be improved in this paper to illustrate the challenges and ideas more clearly. The quality of the figures should be improved.\n- This paper lacks some necessary citations in the convergence analysis part.\n- The computation and communication overhead should be discussed especially with the prototype-based FL methods like FedProto.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We appreciate the opportunity to discuss the weakness and questions. \n# 1.  Motivation and Insights\n## a. Motivation and Insight of FedGMKD\nFedGMKD tackles key challenges in personalized federated learning, particularly the non-IID client data that can lead to suboptimal model performance. Unlike traditional KD-based PFL methods that require public datasets, raising privacy issues and struggling with data heterogeneity, FedGMKD introduces Cluster Knowledge Fusion (CKF) and Differential Aggregation Technique (DAT). CKF uses Gaussian Mixture Models to create prototype features and soft predictions without public datasets, maintaining privacy and handling data heterogeneity. DAT improves server aggregation by prioritizing client data quality, allowing high-quality data to have more influence on the global model, thereby enhancing both local and global accuracy.\n## b. Tracking and Handling Stragglers\nAlthough stragglers are not explicitly focused on, DAT mitigates their impact by weighting client contributions based on data quality, ensuring reliable aggregation.\n# 2. Response to Lack of Necessary Citations in the Convergence Analysis Part\nOur convergence analysis relies on standard theoretical principles widely accepted in federated learning, which do not typically require citations. For example, the FedProto presents similar analyses without additional citations. Our analysis aligns with these practices, focusing on mathematical rigor consistent with academic norms.\n# 3. Response to Comment on Computation and Communication Overhead\nTable 1 displays the average times for FedProto and FedGMKD across three datasets. FedProto reduces computational demands and communication overhead by using prototype sharing and only transmitting prototypes. In contrast, FedGMKD optimizes computations through CKF and DAT while sharing prototypes, predictions, and quality assessments to efficiently transmit necessary data. When compared, FedGMKD is more computationally efficient due to its optimized operations and minimizes communication overhead by selectively exchanging essential data.\n# 4. Calculating Discrepancy with Non-IID Data\n## a. Purpose of Discrepancy Calculation\nThe discrepancy is calculated to determine the weights for global aggregation for each class. This process ensures that the contribution of each client to the global model is proportional to the quality and relevance of its data.\n## b. Handling Missing Classes\nIf a client lacks data for certain classes, it will not provide corresponding prototype features and soft predictions.\nThese equations are used by clients to calculate prototype features and soft predictions only for the classes they have.\n   $$\n   \\\\hat{h}\\_i^{j} = \\\\frac{1}{R} \\\\sum_{r=1}^{R} p(\\\\hat{h}\\_i^{j} \\\\mid \\\\theta)\\\\bar{r}^j\n   $$\n   $$\n   \\\\hat{q}\\_i^{j} = \\\\frac{1}{R} \\\\sum\\_{r=1}^{R} p(z\\_i^{j} \\\\mid \\\\theta)z\\_r^j\n   $$\nHere, $\\hat{h}\\_i^{j} $ and $\\\\hat{q}\\_i^{j}$ represent the prototype feature and the soft prediction vector for class $j$ at client $i$, synthesized from the cluster knowledge, where $R$ is the number of clusters. So if clients miss some classes, they will not calculate $\\hat{h}\\_i^{j} $ and $\\\\hat{q}\\_i^{j}$.\n## c. KL Divergence for Missing Classes\nThis equation is used to compute the KL divergence for the classes present in the client data. If a client lacks a class, the KL divergence for that class is effectively zero.\n  $$\n   D\\_{KL}(Q\\_i^j \\\\parallel Q\\_{\\\\text{global}}^j) = 0 \\\\quad \\\\text{if client } i \\\\text{ lacks class } j\n  $$\nThen, the weights for a client's contribution to each class are calculated, excluding those for which the client lacks data. Therefore, clients without data do not contribute to the global aggregation for those classes, while clients with data for the missing classes have more weight, ensuring that the aggregation is dominated by contributions from clients with relevant data, improving the robustness and accuracy of the global model.\n# 5. Detailed Federated Learning (FL) Setting and client participation\n## a. FL setting\nThe paper has provided the details of the federated learning (FL) setting: \n- Datasets (Lines 222-236): SVHN, CIFAR-10, CIFAR-100.\n- Non-IID Partitioning (Lines 449-454): Dirichlet distribution\n- Model Architecture (Lines 238-243): ResNet18\n- Baselines (Lines 245-251): Includes FedAvg, FedProx, Moon, FedGen, FedMD, FedProto, and FPL.\n- Implementation (Lines 253-259): PyTorch on NVIDIA A100 GPUs, using Adam optimizer, with specific hyperparameters.\n## b. Client Participation\nSetting the client participation rate to 1 is common in federated learning to provide a controlled benchmark for evaluating algorithm performance. This approach allows researchers to assess the theoretical limits and maximum potential of federated learning algorithms under ideal conditions without the complexities introduced by client dropout or partial participation. By using full participation as a baseline, studies can more accurately compare the efficiency and accuracy of different algorithms when all clients contribute to the training process.\n\nIn practice, this setting is widely adopted across various federated learning studies:\n\n- Tan et al. (2022). \"FedProto: Federated Prototype Learning Across Heterogeneous Clients.\" *arXiv:2105.00243*.\n\n- Mendieta et al. (2022). \"Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning.\" In *CVPR*.\n\n- Karimireddy et al. (2020). \"SCAFFOLD: Stochastic Controlled Averaging for Federated Learning.\" In *ICML*, PMLR, 119:5132-5143.\n\n- Huang et al. (2023). \"Rethinking Federated Learning with Domain Shift: A Prototype View.\" In *CVPR*.\n\n- McMahan et al. (2017). \"Communication-Efficient Learning of Deep Networks from Decentralized Data.\" In *AISTATS*.\n\n- Chen et al. (2023). \"The Best of Both Worlds: Accurate Global and Personalized Models Through Federated Learning with Data-Free Hyper-Knowledge Distillation.\" In *ICLR*. *arXiv:2301.08968*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JunnOESqD6", "reviewer": "Reviewer_EYJh", "summary": "The authors introduce a novel federated learning algorithm aimed at addressing the challenge of data heterogeneity among distributed clients. The key innovation of this work is the integration of Cluster Knowledge Fusion (CKF) and the Differential Aggregation Technique (DAT). CKF employs Gaussian Mixture Model (GMM) clustering to generate prototype features and soft predictions, facilitating effective local model training without the reliance on public datasets. DAT further refines the aggregation process by accounting for the distinct feature distributions across categories, thereby enhancing both efficiency and performance. Comprehensive experiments on benchmark datasets (SVHN, CIFAR-10, and CIFAR-100) reveal that FedGMKD delivers superior results in both personalized and global model accuracy compared to traditional federated learning methods. The theoretical analysis included in the paper substantiates the convergence and robustness of the proposed framework.", "questions": "1、Can the proposed technique generalize to a wider range of hyperparameter settings, such as α and β in local loss function, to assess the robustness and sensitivity of FedGMKD?\n2、Can the authors elaborate the reasons you choose Gaussian Mixture Models (GMM) for obtaining prototype features over other clustering methods?", "limitations": "The authors have discussed the limitations in the section of conclusion.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1、The paper presents an innovative approach to addressing data heterogeneity in federated learning through the integration of Cluster Knowledge Fusion (CKF) and the Differential Aggregation Technique (DAT). The use of Gaussian Mixture Model (GMM) clustering in CKF to generate prototype features and soft predictions marks a significant advancement in handling diverse data distributions without relying on public datasets.\n2、The authors provide a comprehensive theoretical analysis of the proposed methods, including mathematical guarantees for convergence and performance. This rigorous approach enhances the credibility and robustness of the framework.\n3、The experimental evaluation is thorough and extensive, utilizing well-known benchmark datasets such as SVHN, CIFAR-10, and CIFAR-100. The results consistently indicate that FedGMKD surpasses existing federated learning methods in both local and global accuracy, demonstrating the practical effectiveness of the proposed techniques.\n4、By using prototype features instead of true data for knowledge distillation, FedGMKD ensures privacy preservation. This approach eliminates the need for public datasets and mitigates privacy concerns, making it particularly relevant for applications in sensitive domains where data privacy is paramount. This enhances the framework's suitability for real-world federated learning applications.", "weaknesses": "1、The experiments conducted in the paper primarily use a narrow range of hyperparameter settings. While the results are strong, exploring a wider range of hyperparameters, such as the α and β in local loss function, would provide a more comprehensive understanding of the framework's robustness and sensitivity. \n2、While FedGMKD uses Gaussian Mixture Models (GMM) for obtaining prototype features, the paper does not provide sufficient justification for choosing GMM over other clustering methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback. We appreciate the opportunity to address the concern regarding the exploration of a wider range of hyperparameters to demonstrate FedGMKD's robustness& sensitivity and discuss the justification of using GMM in FedGMKD.\n# 1. Hyperparameter Exploration in FedGMKD\n## a. Exploration of Hyperparameters  $\\gamma\\$ and $\\lambda\\$ \nThe provided ablation study systematically varies the hyperparameters 𝛾 and λ within the FedGMKD framework using the CIFAR-10 dataset with 10 clients over 50 epochs. Below is a summary of the results:\n| Scheme   | Clients | Epochs |  $\\gamma\\$  | $\\lambda\\$  | Local Acc (%) | Global Acc (%) |\n|----------|---------|--------|-------------|------------|---------------|----------------|\n| FedGMKD  | 10      | 50     | 0.06        | 0          | 60.14         | 48.17          |\n| FedGMKD  | 10      | 50     | 0.06        | 0.02       | 60.32         | 49.44          |\n| FedGMKD  | 10      | 50     | 0.06        | 0.04       | 60.99         | 49.48          |\n| FedGMKD  | 10      | 50     | 0.06        | 0.06       | 61.78         | 49.98          |\n| FedGMKD  | 10      | 50     | 0.06        | 0.08       | 60.27         | 48.64          |\n| FedGMKD  | 10      | 50     | 0.06        | 0.1        | 60.14         | 49.97          |\n| FedGMKD  | 10      | 50     | 0           | 0.06       | 60.73         | 48.72          |\n| FedGMKD  | 10      | 50     | 0.02        | 0.06       | 60.33         | 48.21          |\n| FedGMKD  | 10      | 50     | 0.04        | 0.06       | 61.01         | 49.47          |\n| FedGMKD  | 10      | 50     | 0.08        | 0.06       | 60.93         | 49.69          |\n| FedGMKD  | 10      | 50     | 0.1         | 0.06       | 59.86         | 47.52          |\n| FedAvg   | 10      | 50     | 0           | 0          | 55.75         | 46.62          |\n| FedProto | 10      | 50     | 0.05        | 0          | 59.77         | 48.97          |\n| FPL      | 10      | 50     | 0           | 0          | 60.95         | 47.19          |\n\n## b. Robustness of FedGMKD\n### 1. Consistency Across Hyperparameter Values\n- **High Local and Global Accuracies:** FedGMKD maintains high accuracy across different $\\gamma\\$ and $\\lambda\\$ values, indicating robustness.\n- **Optimal Performance Stability:** With $\\gamma\\$ = 0.06 and  $\\lambda\\$ = 0.06, FedGMKD achieves its peak performance, highlighting adaptability.\n### 2. Comparative Performance\n- **Superior Performance:** FedGMKD outperforms FedAvg, FedProto, and FPL, underscoring its robustness and efficiency.\n## c. Sensitivity of FedGMKD\n###  Adaptive Tuning Insights\n- **Parameter Optimization:** While FedGMKD operates effectively across various settings, fine-tuning $\\gamma\\$ and $\\lambda\\$ is essential to leverage its full potential and adapt to specific datasets.\n\n#  2. The Justification of Using Gaussian Mixture Models (GMM) in FedGMKD\nBelow, we provide a detailed justification for selecting GMM over other clustering methods, supported by relevant literature and content.\n## a. Handling Data Heterogeneity\n- **Robustness in Diverse Environments:** GMM effectively handles heterogeneous data distributions, crucial for federated learning. The study \"Personalized Federated Learning under Mixture of Distributions\" highlights GMM's robustness in diverse client data environments.  \n- **Managing Non-IID Data:** GMM efficiently manages non-IID data distributions, enhancing learning efficiency in federated contexts, as shown in \"An Efficient Framework for Clustered Federated Learning.\" \n## b. Flexibility and Adaptability\n- **Effective Utilization of Task Similarities:** GMM leverages unknown task similarities, offering theoretical guarantees for convergence, as discussed in \"Robust Unsupervised Multi-task and Transfer Learning on Gaussian Mixture Models.\" This adaptability is advantageous for FedGMKD, improving model generalization.  \n- **Transfer Learning Capabilities:** GMM enhances clustering performance through transfer learning, as seen in \"A General Transfer Learning-based Gaussian Mixture Model for Clustering,\"  enabling FedGMKD to handle limited data per client.\n## c. Privacy and Outlier Management\n- **Data Privacy and Heterogeneity:** GMM addresses data privacy and heterogeneity, demonstrated in \"Federated Learning for Misbehavior Detection with Variational Autoencoders and Gaussian Mixture Models,\"  making it suitable for federated learning applications.  \n- **Robustness Against Outliers:** GMM's probabilistic nature allows it to handle outliers effectively, as evidenced in \"Robust Unsupervised Multi-task and Transfer Learning on Gaussian Mixture Models,\"  making it an ideal choice for FedGMKD.\n\n## references\n- Smith, V., Chiang, C.-K., Sanjabi, M., & Talwalkar, A. (2017). *\"Personalized Federated Learning under Mixture of Distributions.\"* arXiv:1705.10467\n- Zhang, Y., & Yang, Q. (2017). *\"Robust Unsupervised Multi-task and Transfer Learning on Gaussian Mixture Models.\"* arXiv:1711.05995\n- Zhao, Y., Liu, P., Cheng, J., Chen, M., & Chen, L. (2020). *\"Federated Learning for Misbehavior Detection with Variational Autoencoders and Gaussian Mixture Models.\"* IEEE Transactions on Intelligent Transportation Systems. DOI:10.1109/TITS.2020.3006572\n- Sattler, F., Wiedemann, S., Müller, K.-R., & Samek, W. (2020). *\"An Efficient Framework for Clustered Federated Learning.\"* arXiv:2004.03337\n- Luo, P., Ding, X., & Zhao, X. (2019). *\"A General Transfer Learning-based Gaussian Mixture Model for Clustering.\"* Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI-19). DOI:10.24963/ijcai.2019/243", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6ZiUFzvNTI", "reviewer": "Reviewer_nWER", "summary": "FedGMKD addresses data heterogeneity using a dual-enhancement approach through Cluster Knowledge Fusion (CKF) and Differential Aggregation Technique (DAT). This method effectively enhances both local and global model performance without relying on public datasets or complex server-side models.", "questions": "1. See weaknesses\n2. line 129 space is missing jin\n3. The assumption should be inside the main paper for the convergence analysis.", "limitations": "The paper addressed limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strength of FedGMKD lies in its innovative use of Gaussian Mixture Model (GMM) clustering to generate prototype features and soft predictions for each category, which are aggregated server-side to preserve privacy and address non-IID data challenges effectively. Additionally, the Differential Aggregation Technique (DAT) optimizes the aggregation process by weighting prototype features and soft predictions based on the quality of each client's data per category, enhancing global model performance robustly.", "weaknesses": "The motivation behind using Knowledge Distillation (KD) is to tackle both data and model heterogeneity. However, this paper falls short by not comparing its performance against architectures like FjORD [1], which handle model heterogeneity effectively. Moreover, restricting comparisons to vision datasets alone limits the broader applicability and merit of this work.\n\n[1] Horvath et al., \"FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout,\" 35th Conference on Neural Information Processing Systems (NeurIPS’21).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# 1. Broader Applicability and Merit of FedGMKD\n\nThank you for your insightful feedback regarding the scope of our experiments and the potential broader applicability of FedGMKD. We appreciate the opportunity to clarify and expand upon our work by exploring its performance on datasets beyond the computer vision domain.\n\n## a. Choice of IMBD Dataset\n\nTo address the reviewer's concerns, we conducted additional experiments using the IMBD (Internet Movie Database) dataset, a well-known resource in natural language processing for sentiment analysis. This text-based dataset, comprising a vast number of movie reviews, allowed us to test FedGMKD's adaptability across different data modalities. IMBD's popularity in NLP research makes it an ideal benchmark for evaluating federated learning algorithms, particularly as sentiment analysis challenges models to generalize effectively across diverse client data distributions.\n\n## b. Experimental Setup\n\nIn alignment with the paper's experimental setup, we utilized 10 clients and conducted the experiment over 50 epochs. Due to the transition from computer vision to NLP, the model used was changed from ResNet-18 to BERT, which is well-suited for text-based tasks like sentiment analysis. BERT's capability to capture contextual word embeddings made it an appropriate choice for handling the IMBD dataset. The main metrics for evaluation were local accuracy, global accuracy, and average computation time per client.\n\n## c. Results and Analysis\n\n| Scheme   | Dataset | Clients | Local Acc | Global Acc | Avg Time (S) |\n|----------|---------|---------|-----------|------------|--------------|\n| FedGMKD  | IMBD    | 10      | 85.11     | 51.58      | 677.79       |\n| FedAvg   | IMBD    | 10      | 83.71     | 50.52      | 411.95       |\n| FedProx  | IMBD    | 10      | 83.75     | 48.50      | 438.52       |\n| FedMD    | IMBD    | 10      | 83.87     | 48.29      | 700.73       |\n| FedGen   | IMBD    | 10      | 83.54     | 49.16      | 471.35       |\n| FedProto | IMBD    | 10      | 84.13     | 49.72      | 586.77       |\n| FPL      | IMBD    | 10      | 83.96     | 50.12      | 665.29       |\n\nThe results demonstrate that FedGMKD consistently outperforms other federated learning algorithms in both local and global accuracy metrics on the IMBD dataset. This suggests that the core mechanisms of Cluster Knowledge Fusion (CKF) and Differential Aggregation Technique (DAT) are effective not only for image-based datasets but also for text-based datasets like IMBD. The high local accuracy of 85.11 and global accuracy of 51.58 highlight the framework’s robustness and adaptability to different data types, further validating its broader applicability beyond just vision datasets.\n# 2. Response to typo of line 129\nWe apologize for the typo on line 129. We have carefully reviewed the manuscript and have fixed all typos and mistakes, including the missing space in \"jin.\" The final version of the manuscript reflects these corrections.\n# 3. Response to Assumptions \nThank for your highlighting, the final version of my paper will include assumptions in main paper.\n# 4. Response to Comparison with FjORD\nThank you for highlighting the importance of comparing FedGMKD with architectures like FjORD, which address model heterogeneity. Here, we clarify why FjORD was not initially chosen for comparison and provide an analysis of the experimental results by using FjORD and FedGMKD.\n\n## a. Reason for Not Initially Choosing FjORD\nBoth FjORD and FedGMKD aim to address distinct challenges in federated learning. FjORD primarily focuses on model heterogeneity by employing dropout techniques that dynamically adjust model capacity, allowing models to efficiently adapt to diverse client computational resources. This makes FjORD effective in handling varying hardware environments. Conversely, FedGMKD targets data heterogeneity through CKF and DAT, aiming to enhance model performance by aligning client data distributions and improving aggregation efficiency. These differing focuses led us to initially compare FedGMKD with frameworks that similarly prioritize data heterogeneity, rather than FjORD’s emphasis on model heterogeneity.\n## b. Experimental Comparison and Analysis\nIn the absence of official code for the FjORD method, we combined various reproduced code implementations to create an experimental version for comparison. This allowed us to assess FjORD's performance under the same conditions used for FedGMKD. The results are summarized below:\n| Datasets | Scheme  | Local Acc 10 | Local Acc 20 | Local Acc 50 | Global Acc 10 | Global Acc 20 | Global Acc 50 |\n|----------|---------|--------------|--------------|--------------|---------------|---------------|---------------|\n| CIFAR10  | FjORD   | 59.62        | 63.36        | 63.61        | 49.18         | 53.22         | 58.74         |\n|          | FedGMKD | 61.78        | 64.04        | 65.69        | 49.78         | 55.16         | 60.31         |\n| SVHN     | FjORD   | 85.13        | 85.97        | 86.21        | 81.56         | 85.09         | 89.36         |\n|          | FedGMKD | 86.26        | 87.43        | 87.16        | 82.64         | 87.78         | 90.17         |\n| CIFAR100 | FjORD   | 15.94        | 19.91        | 22.60        | 16.93         | 21.45         | 22.86         |\n|          | FedGMKD | 17.16        | 20.96        | 23.57        | 16.97         | 21.56         | 24.63         |\n\nThe comparison results indicate that while FjORD effectively manages model heterogeneity through dropout techniques, FedGMKD provides superior performance in scenarios where data heterogeneity is the primary concern. The ability of FedGMKD to adapt to diverse client data distributions and enhance aggregation strategies makes it a more effective solution in the contexts tested. These results underscore FedGMKD's capability to address federated learning challenges beyond model heterogeneity, reinforcing its broader applicability and effectiveness.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate all the reviewers for their constructive and valuable feedback. We are pleased that our work has been recognized for its innovative approach to addressing data heterogeneity in federated learning through the integration of Cluster Knowledge Fusion (CKF) and Differential Aggregation Technique (DAT) (EYJh, KnUX, nWER). The reviewers have noted our paper's theoretical rigor, with comprehensive mathematical guarantees for convergence and performance (EYJh, 8XP3). We are encouraged by the recognition of the thorough and extensive experimental evaluation using benchmark datasets such as SVHN, CIFAR-10, and CIFAR-100, which demonstrate substantial improvements over existing methods (EYJh, KnUX). Additionally, the use of Gaussian Mixture Model (GMM) clustering for privacy-preserving prototype generation and the clear explanation of complex concepts have been highlighted as strengths of our manuscript (EYJh, KnUX, nWER). The feedback underscores the applicability and robustness of FedGMKD in practical federated learning scenarios, particularly in privacy-sensitive domains (EYJh, nWER).

We have addressed the reviewers’ comments and concerns in individual responses to each reviewer. The reviews allowed us to strengthen our manuscript, and the changes made are summarized below:

### Reviewer KnUX
- Conducted additional experiments using ResNet-50 on the CIFAR-10 dataset to explore the performance of FedGMKD with more complex neural network architectures.
- Expanded discussion on FedGMKD's adaptability to diverse data distributions and its real-world applications in healthcare, finance, and smart cities.

### Reviewer 8XP3
- Elaborated on the motivation and insights of FedGMKD, including its strategies for handling non-IID data without relying on public datasets.
- Addressed the necessity of citations in the convergence analysis, aligning with standard practices in federated learning literature.
- Clarified the computational and communication overhead by comparing FedGMKD and FedProto.
- Provided a detailed explanation of the federated learning setting and client participation strategy, with references to relevant studies.
- Explained the methodology for calculating discrepancy with non-IID data and how FedGMKD handles missing data classes in client datasets.


### Reviewer EYJh
- Conducted a comprehensive ablation study on the hyperparameters $\gamma\$ and $\lambda\$ to demonstrate FedGMKD's robustness and sensitivity.
- Justified the use of Gaussian Mixture Models (GMM) in FedGMKD, supported by relevant literature on handling data heterogeneity and privacy concerns.

### Reviewer nWER
- Performed additional experiments using the IMBD dataset to evaluate FedGMKD's performance in natural language processing tasks, demonstrating its adaptability beyond computer vision.
- Solved typo problem in the paper.
- Included assumptions in the main paper.
- Compared FedGMKD with FjORD to assess its effectiveness in addressing model heterogeneity and provided a detailed analysis of the results.

Meta-review: The paper addresses the issue of data heterogeneity in FL with an eye toward robustness, by introducing  Cluster Knowledge Fusion and Differential Aggregation Techniques.  With the exception of one low score, the other three are generally in agreement and positive about the novelty of the approach.
Decision: Accept (poster)