Interaction-Aware Gaussian Weighting for Clustered Federated Learning
Authors: Alessandro Licciardi
Davide Leo
Eros Fanì
Barbara Caputo
Marco Ciccone
Venue: ICML 2025 poster (2025)
Keywords: Federated Learning
Clustered Federated Learning
Primary area: general_machine_learning->everything_else
TL;DR: We propose FedGWC a new clustering algorithm for Federated Learning and a new clustering metric tailored for heterogeneous FL

Abstract: Federated Learning (FL) emerged as a decentralized paradigm to train models while preserving privacy. However, conventional FL struggles with data heterogeneity and class imbalance, which degrade model performance.
Clustered FL balances personalization and decentralized training by grouping clients with analogous data distributions, enabling improved accuracy while adhering to privacy constraints. This approach effectively mitigates the adverse impact of heterogeneity in FL.
In this work, we propose a novel clustering method for FL, **FedGWC** (Federated Gaussian Weighting Clustering), which groups clients based on their data distribution, allowing training of a more robust and personalized model on the identified clusters. **FedGWC** identifies homogeneous clusters by transforming individual empirical losses to model client interactions with a Gaussian reward mechanism. Additionally, we introduce the *Wasserstein Adjusted Score*, a new clustering metric for FL to evaluate cluster cohesion with respect to the individual class distribution. Our experiments on benchmark datasets show that **FedGWC** outperforms existing FL algorithms in cluster quality and classification accuracy, validating the efficacy of our approach.

Interaction-Aware Gaussian Weighting for Clustered Federated Learning
Alessandro Licciardi 12
Davide Leo *3
Eros Fani
Barbara Caputo 3 Marco Ciccone'
Abstract
tional machine learning frameworks, FL enables collabora- tive training between multiple clients without requiring data transfer, making it particularly attractive in privacy-sensitive domains (Bonawitz et al., 2019). FL was introduced pri- marily to address two major challenges in decentralized scenarios: ensuring privacy (Kairouz et al., 2021) and re- ducing communication overhead (Hamer et al., 2020; Asad et al., 2020). In particular. FL algorithms must guarantee communication efficiency, to reduce the burden associated with the exchange of model updates between clients and the central server, while maintaining privacy, as clients should not expose their private data during the training process.
Federated Learning (FL) emerged as decentral- ized paradigm to train models while preserving privacy. However, conventional FL struggles with data heterogeneity and class imbalance, which degrade model performance. Clustered FL bal- ances personalization and decentralized training by grouping clients with analogous data distribu- tions, enabling improved accuracy while adhering to privacy constraints. This approach effectively mitigates the adverse impact of heterogeneity in FL. this work, we propose a novel clustered FI. method, FedCWC (Federated Gaussian Weighting Clustering), which groups clients based on their data distribution, allowing training of more ro- bust and personalized model on the identified clus- ters. FedGNC identifies homogeneous clusters by transforming individual empirical losses to model client interactions with Gaussian reward mecha- nism. Additionally, we introduce the Wasserstein Adjusted Score, a new clustering metric for FL. to evaluate cluster cohesion with respect the individual class distribution. Our experiments on benchmark datasets show that FedGWC outper- forms existing FI. algorithms in cluster quality and classification accuracy, validating the efficacy of our approach. Code is available at https: / //github. wwwp///wwww////////oooo/o////eee FFeGGR
A core challenge in federated learning is data heterogeneity (I.i et al., 2020). This manifests in two key ways: through imbalances in data quantity and class distribution both within and across clients, and through the non-independent and non-identical distribution (non-IID) of data across the federation. In this scenario, a single global model often fails to generalize due to clients contributing with updates from skewed distributions, leading to degraded performance (Zhao et al., 2018; Caldarola et al., 2022) compared to the centralized counterpart. Furthermore, noisy or corrupted data from some clients can further complicate the learning process (Cao et al., 2020; Zhang al., 2022), while non-IlD data often results in unstable convergence and conflicting gradient updates (Hsieh et al., 2020; Zhao et al., 201 8). Despite the introduction of various techniques to mitigate these issues - ranging from regularization methods (Li et al., 2020) and optimizer modifications based on momentum (Mendieta al., 2022) control variates (Karimireddy et al., 2020b), to model-level strategies such as feature dis- tillation (Yang et al., 2023), and deeper investigations into solution landscape properties like achieving flatter minima (Lee & Yoon, 2024; Fan et al., 2024) or preventing dimen- sional collapse (Shi et al., 2023) - data heterogeneity persists as a critical problem.
1. Introduction
Federated Learning (FL) (McMahan et al., 2017) has emerged as a promising paradigm for training models on decentralized data while preserving privacy. Unlike tradi-
'Equal contribution 'Department of Mathematical Sciences, Polytechnic University of Turin, Italy "Istituto Nazionale di Fisica Nuelcare (INFN), Sezione di Torino, Turin, Italy 'Department of Computing and Control Engineering, Polytechnic University of Turin, Italy "Basque Center for Applied Mathematics (BCAM), Bilbao, Spain VVector Institute. Toronto, Ontario, Canada. Work started when the author was the Polytechnic University of Turin. Correspondence to: Alessandro Licciardi <alessan- dro. frociiiciardi @ it..
In this work, address the fundamental challenges of data heterogeneity and class imbalance in federated learn- ing through novel clustering based approach. We pro- pose F FedGWC (Federated Gaussian Weighting Clustering), a method that groups clients with similar data distributions into clusters, enabling the training of personalized feder- ated models for each group. Our key insight is that clients' data distributions can be inferred by analyzing their empir-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Interaction-Aware Gaussian Weighting for Clustered Federated Learning
ical loss functions, rather than relying on model updates as most existing approaches do. Inspired by (Cho et al., 2022), we hypothesize that clients with similar data distribu- tions will exhibit similar loss landscapes. FedGWC imple- ments this insight using a Gaussian reward mechanism to form homogeneous clusters based on an interaction matrix, which encodes the pairwise similarity of clients' data dis- tributions. This clustering is achieved efficiently by having clients communicate only their empirical losses to the server at each communication round. Our method transforms these loss values to estimate the similarity between each client's s data distribution and the global distribution, using Gaussian weights as statistical estimators. Each cluster then trains its own specialized federated model, leveraging the shared data characteristics within that group. This approach pre- serves the knowledge-sharing benefits of federated learning while reducing the negative effects of statistical heterogene- ity, such as client drift (Karimireddy et al., 2020b). We develop a comprehensive mathematical framework for this approach and rigorously prove the convergence properties of the Gaussian weights estimators.
2. Related Work
FL with Heterogeneous Data. Handling data heterogene- ity, especially class imbalance, remains a critical challenge in FL. FedProx(Li et al., 2020) was one of the first at- tempts to address heterogeneity by introducing a proximal term that constrains local model updates close to the global model. FedMD (Li & Wang, 2019) focuses on heterogeneity in model architectures, allowing collaborative training be- tween clients with different neural network structures using model distillation. Methods such as SCAFFOLD (Karim- ireddy et al., 2020b) and Mime (Karimireddy et al., 2020a) have also been proposed to reduce client drift by using con- trol variates during the optimization process, which helps mitigate the effects of non-lll) data. Furthermore, strategies such as biased client selection (Cho et al., 2022) based on ranking local losses of clients and normalization of updates in FecNova (Wang et al., 2021) have been developed to specifically address class imbalance in federated networks, leading more equitable global model performance.
Clustered FL.. Clustering has proven to be an effective strategy in FL for handling client heterogeneity and im- proving personalization (Huang et al., 2022; Duan et al., 2021; Briggs et al., 2020; Caldarola et al., 2021; Ye et al., 2023). Clustered FL. (Sattler et al., 2020) is one of the first methods proposed group clients with similar data distributions to train specialized models rather than rely- ing on a single global one. Nevertheless, from practical perspective, this method exhibits pronounced sensitivity to hyper-parameter tuning, especially concerning the gradient norms threshold, which is intricately linked to the dataset. This sensitivity can result in significant issues of either ex- cessive under-splitting or over-splitting Additionally, as client sampling is independent of the clustering, there may be privacy concerns due to the potential for updating cluster models with the gradient of single client. An extension of this is the efficient framework for Clustered FL proposed by (Ghosh et al., 2020), which strikes a balance between model accuracy and communication efficiency. Multi-Center FL (Long et al., 2023) builds on this concept by dynamically adjusting client clusters achieve better personalization, however a-priori knowledge on the number of clusters is needed. Similarly, IFCA (Ghosh et al., 2020) addresses client heterogeneity by predefining a fixed number of clus- ters and alternately estimating the cluster identities of the users by optimizing model parameters for the user clusters via gradient descent. However, imposes a significant com- putational burden, as the server communicates all cluster models to each client, which must evaluate every model locally to select the best fit based on loss minimization. This approach not only increases communication overhead but also introduces inefficiencies, as each client must test all models, making it less scalable in larger networks. Re- cent studies on clustered FL investigate sophisticated client
To evaluate our method, we introduce the Wasserstein Adjusted Score, a new clustering metric tailored for assessing cluster cohesion in class-imbalances FL scenarios. Through extensive experiments on both benchmark (Caldas et al., 2018) and large-scale datasets (Hsu et al., 2020), we demonstrate that FedGWC outperforms existing clustered FL algorithms in terms of both accuracy and clustering quality. Furthermore, FedGWC can be integrated with any robust FL aggregation algorithm provide additional resilience against data heterogeneity.
Contributions.
We propose FedGWC, an efficient federated learning framework that clusters clients based on their data distri- butions, enabling personalized models that better handle heterogeneity.
We provide a rigorous mathematical framework to mo- tivate the algorithm, proving its convergence properties and providing theoretical guarantees for our clustering approach.
We introduce novel clustering metric specifically de- signed to evaluate cluster quality in the presence of class imbalance.
We demonstrate through extensive experiments that 1) FedGWC outperforms existing clustered FL approaches in terms of both clustering quality and model performance, 2) our method successfully handles both class and domain imbalance scenarios, and 3) the framework can be effec- tively integrated with any FL aggregation algorithm.
2

---

Interaction-Aware Gaussian Weighting for Clustered Federated Learning
grouping strategies. For instance, (Carrillo et al., 2024) employs consensus-based optimization dynamics, where iterative interactions and calculations across all client rep- resentations, viewed as particles, are typically required to guide them towards a group consensus on cluster mem- bership; this process can be computationally demanding, especially as the number of clients and the complexity of their models increase. Another approach (Bao et al., 2023) involves training pairwise diseriminators between all, significant number of, potential client pairs to estimate their data distribution similarities. This approach may result in quadratic increase in the number of auxiliary models for the exclusive aim of clustering, thus imposing a substantial com- putational burden, particularly within federated networks with a vast number of clients.
di distribution distribution Average less process
150
125
1.00
0.75
050
025
0.00
Iterations
Figure I. Illustration of the Gaussian reward mechanism for two clients from C0 (Dirichlet a = 0.05. 10 sampled clients per round and S 8 local iterations). The dashed line represents the average loss process m" with the blue region indicating the confidence interval m + + at fixed f, = 1,...,& 8. The green curve corresponds to an in in-distribution client, whose loss remains within the confidence region, resulting in a high Gaussian reward. The red line represents an out-of of-distribution client, whose loss lies outside the confidence region, resulting in lower reward. between client distributions. For elarity, we first present these concepts considering the federation as a single cluster, focusing on the fundamental mechanisms that enable client grouping. Finally, Subsection 4.3 introduces the complete algorithmic framework, including cluster indices and the iterative structure that allows FedGWC to partition clients into increasingly homogeneous groups.
Compared to previous approaches, the key advantage of the proposed algorithm, FedGWC, lies in its ability to effectively identify clusters of clients with similar levels of heterogene- ity and class distribution through simple transformations of the individual empirical loss process. This is achieved without imposing significant communication overhead or requiring additional computational resources. Further de- tails on the computational and communication overhead are provided in Appendix D. Additionally, FedGWC can be seamlessly integrated with any aggregation method, en- hancing its robustness and performance when dealing with heterogencous scenarios.
4.1. Gaussian Weighting Mechanism
To assess how closely the local data of each client aligns with the global distribution, we introduce the Gaussian Weights 7k. statistical estimators that capture the closeness of each clients' distribution to the main distribution of the cluster. A weight near zero suggests that the client's dis- tribution is far from the main distribution. We graphically represent the idea of the Gaussian rewards in Figure 1.
3. Problem Formulation
Consider a standard FL scenario (McMahan al., 2017)
with K clients and central server. FL typically ad- dresses the following optimization problem minsee L())= = minøee 2k=1 Th Lx (0), where Lx(-) represents the loss function of client k, n4 is the number of training samples on client k, n =k=1 nig is the total number of samples, and è denotes the model's parameters space. At each com- munication round E [T]. a subset Pe of clients is selected to participate in training. Each participating client performs S iterations, updating its local parameters using stochas- tic optimizer, e.g. Stochastic Gradient Descent (SGD). In clustered FL, the objective is to partition clients into non- overlapping groups ((1) ..... ((na) based on similarities in their data distributions, with each group having its own model, (1),...,0p ((ma)-
n
The fundamental principle of FedCWC is to group clients based on the similarity of their empirical losses, which are used to assign a reward between and 1 to each client at each local iteration. A high reward indicates that client's s loss is close to the cluster's 's mean loss, while lower reward reflects greater divergence. Gaussian weights estimate the expected value of these rewards, quantifying the closeness between each client's s distribution and the global one.
Every communication round I, each sampled client l: E Pe communicates the server the empirical loss process 1/20 C(0"."), for s = 1,..., S, alongside the updated model ot+ The server computes the rewards as
4. FedGWC Algorithm
In this section, we present the core components of FedGWC in progressive manner. First, Subsection 4.1 introduces the Gaussian Weighting mechanism, a statistical method that estimates how well each client's 's data distribution aligns with the overall federation. Subsection 4.2 then explains how we model and detect clusters by analyzing interactions
1. rk =cxpp exp (- (1.. - -o-mmm--p m/.)²2
E 10.
(1)
2(ot.1)3
where mis 1/IP. EkEr, 1h"" is the average loss pro- cess w.r.t. the local iterations s S, and (a
3

---

Interaction-Aware Gaussian Weighting for Clustered Federated Learning
1/(PP)) - E)E a4 m/.0)² is the sample variance. The rewards rk= 1.a + 1 as the distance between 1/= and the av- erage process m²** decreases, while r f. - 0 as the distance increases. In Eq. 1, during any local iteration s, a Gaussian kernel centered on the mean loss and with spread the sample variance assesses the clients' proximity to the confidence interval's center, indicating their probability of sharing the same learning process distribution. Since the values of the rewards can suffer from stochastic oscillations, we reduce the noise in the estimate by averaging the rewards over the S local iterations, thus obtaining w 1/S > EE(s rik" f.* for each sampled client k. Instead of using single sample, such as the last value of the loss as done in (Cho et al., 2022), we opted for averaging across iterations to provide a more stable estimate. However, the averaged rewards wi depend on the sampled set of clients Pe and on the current round. Hence, we introduce the Gaussian weights *ff to keep track over time of these rewards. The Gaussian weights are com- puted via a running average of the instant rewards wf. In particular, for each client &o the weight 7 is initialized to O to avoid biases in the expectation estimate, and when it is randomly selected is update according to
where {art is the same sequence used update the weights, and Pa is the subset of clients sampled in round t.
Intuitively, in the long run, since wi measures the proximity of the loss process of client k to the average loss process of clients in Pe at round f, we are estimating the expected perception of client k by client j with Pi.. i.e. a larger value indicates a higher degree of similarity between the loss profiles, whereas smaller values indicate a lower degree of similarity. For example, if Pkş is close to 1, it suggests that on average, whenever k and j have been simultaneously sampled prior to round t. w was high, meaning that the two clients are well-represented within the same distribution.
To effectively extract the information embedded in P. we introduce the concept of unbiased perception vectors (UPV). For any pair of clients k.j E [K], the UPV vi E RK-2 represents the k-th row of P. excluding the k-th and j- th entries. Recalling the construction of Pt, where each row indicates how a client is perceived to share the same distribution as other clients in the federation, the UPV og captures the collective perception of client k by all other clients, excluding both itself and client j. This exclusion why we refer to Vg as unbiased.
++ = - (1-ax)) + aqwi 7
(2)
The UPVs encode information about the relationships be- tween clients, which can be exploited for clustering. How- ever, the UPVs cannot be directly used as their entries are only aligned when considered in pairs. Instead, we con- struct the affinity matrix W by transforming the information encoded by the UPV' through an RBF kernel, as this choice allows to effectively model the affinity between clients: two clients are considered affine if similarly perceived by oth- ers. This relation is encoded by the entries of W E R R which we define as:
for a sequence of coefficients a E (0, 1) for any & [T]. The weight definition in Eq.2 is closely related to the Robbins-Monro stochastic approximation method (Robbins & Monro, 1951). If a client is not participating in the train- ing, its weight is not updated. FedGWC mitigates biases in the estimation of rewards by employing two mechanisms: (I) uniform random sampling method for clients, with a dynamic adjustment process to prioritize clients that are infrequently sampled, thus ensuring equitable participation across time periods; and (2) when a client is not sampled in round, its weight and contribution to the reward estimate remain unchanged.
Wky = K(ook = exp (-81le2- -ock--ff ())) (4)
4.2. Modeling Interactions with Gaussian Weights Interaction Matrix. Gaussian weights are scalar quanti- ties that offer an absolute measure of the alignment between a client's data distribution and the global distribution. Al- though these weights indicate the conformity of each client's distribution individually, they do not consider the interrela- tions among the distributions of different clients. Therefore, we propose to encode these interactions in an interaction matrix P' E R x K whose element Píş estimates the simi- larity between the k-th and the j-th client data distribution. The interaction matrix is initialized to the null matrix, i.e. Pł for every couple j,j E [K].
The spread of the RBF kernel is controlled by single hyper- parameter ,3 >> 0: changes in this value provide different clustering outcomes, as shown in the sensitivity analysis in Appendix G.
Clustering. The affinity matrix W, designed to be sym- metric, highlights features that capture similarities between clients' distributions. Clustering is performed by the server using the rows of W as feature vectors, as they contain the relevant information. We apply the spectral clustering algo- rithm (Ng et al., 2001) to W due to its effectiveness in de- tecting non-convex relationships embedded within the client affinities. Symmetrizing the interaction matrix P into the affinity matrix W is fundamental for spectral clustering as it refines inter-client relationship representation. It models interactions, reducing biases, and emphasizing reliable simi- larities. This improves robustness to noise, allowing spectral
Specificallyy, we define the update rule for the matrix pt as follows:
p4+1 (1 (1-a8 (1-ar) + tawh,
(k, P, x Pt
(3)
(k, $ x
4

---

Interaction-Awary Gaussian Weighting for Clustered Federated Learning
clustering to effectively detect the distributional structure un- derlying the clients' network (Von Luxburg, 2007). During the iterative training process, the server determines whether to perform clustering by checking the convergence of the matrix Pt. Convergence is numerically verified when the mean squared error (MSE) between consecutive updates is less than a small threshold e > 0. Algorithm 2 summarizes the clustering procedure. If the MSE is below e, the server computes the matrix W and performs spectral clustering over W with number of clusters n E (2...., Mmmox) J. For each clustering outcome, the Davies-Boulde (DB) score (Davies& & Bouldin, 1979) is computed: DB larger than one means that clusters are not well separated, while if it is smaller than one, the clusters are well separated, a detailed description of the clustering metrics is provided in Appendix B. We denote by nhe the optimal number of clusters detected by F FedGWC. If min.,=,....Po... DB, > 1, we do not split the current cluster. Hence, the optimal number of clusters is net is one. In the other case, the optimal number of clus- ters is ne E arg min,,...................... ensures proper control over the over- werr-plittingg phenomenon, a common issue in hierarchical clustering algorithms in FL which can undermine key principles of FL by creating degen- erate clusters with very few clients. Finally, each cluster (()) ..... C(na)) an FL. aggregation algorithm is trained sep- arately, resulting in models 0 (1)...., (pa))) personalized for each cluster.
Algorithm 1 FedGWC
1: Input: K, T, S, 04, e, IPel.
2: Output: C(L). ,...., C(Na) and 011),..., ((Na)
Initialize Nd 6-1
4: Initialize pu + OK*K
(1)
5: Initialize MSE (1) t-1 1
6: for 1 = .,...,, T - do
7: A <- 0 for each iterations it counts the number of
new clusters that are detected
8:
for ... N₂ do
Server samples P(()) E C((n) and sends the current
9:
cluster model (n)
Each client k E P!(n) locally updates 0% and % then sends them to the server
10:
w f Gauss an Reward rds (1, p(()), ). Eq. 1
11:
0+ FL.Aggregator((01 P(1))
12:
(n)
13:
P( + Update Matrix (Pt
(n) w 01; P(()),
DB,-- This requirement
according to Eq. 3
14:
Update MSE(+) (n)
15:
if MSE < then (n)
16:
Perform FedGW Cluster p1+1 (n) Amaaz 1C)
, providing Ved sub-clusters on
17:
Update the number of new clusters AN AN n2d -
4.3. FedGWC Implementation
18:
Cluster server splits P(-11 filtering rows and columns according to the new clusters
In the previous sections, we have detailed the splitting al- gorithm within the individual clusters. In this section, we present the full FedGWC procedure (Algorithm 1), intro- ducing the complete notation with indices for the distinct clusters. We denote the clustering index as n, and the total number of clusters Ne..
19:
Re-initialize MSE for new clusters to end if
20:
21:
end for
22: Update the total number of clusters.N <- N/. + AN
The interaction matrix P(i is initialized to the null matrix Oxxk. and the total number of clusters Nº. as no clus- ters have been formed yet, and MSE are initialized to 1, in order to ensure stability in early updates, allowing a gradual decrease. At each communication round t, and for cluster ((*), where
23: end for
I,..., N', the cluster server in- dependently samples the participating clients p!")) C c(m). Each client is E (p(") receives the current cluster model P.,.. After performing local updates, each client sends its updated model + and empirical loss 1 back to the cluster server. The server aggregates these updates to form the new cluster model (t))) computes the Gaussian rewards wt for the sampled clients, and updates the interaction matrix P and MSE according to Eq. 3. If ] threshold the server of the cluster performs clustering to determine whether to split cluster ((s) into ne sub-groups,
tioned into sub-matrices by filtering its columns and rows according to the newly formed clusters, with the MSE for these sub-matrices reinitialized to 1. This process results in a distinct model (()) for each cluster ((x). When the final iteration T is reached we are left with NT clusters with personalized models (() for n = 1,..., NT.
Thanks the Gaussian Weights, and the iterative spectral clustering on the affinity matrices, our algorithm, FedGWC, is able to autonomously detect groups of clients that display similar levels of heterogeneity. The clusters formed are more uniform, i.e. the class distributions within each group are more similar. These results are supported by experimental evaluations, discussed in Section 7.3.
(n) (n) is lower than
as outlined in A Algorithm 2. The matrix pt+1 is then parti-
5

---

Interaction-Aware Gaussian Weighting for Clustered Federated Learning
5. Theoretical Results and Derivation of FedGWC
to the individual theoretical reward pe for each client k
1, ,.... K, i.e.,
In this section, provide formal derivation of the algo- rithm, discussing the mathematical properties of Gaussian Weights and outlining the structured formalism and rationale underlying FedGWC.
B
Pk - 0G.
(6)
Proposition 5.3 shows that Gaussian weights reduce the vari- ance of the estimate, thus decreasing the error and enabling the construction of confidence interval for pk-
The stochastic process induced by the optimization algo- rithm in the local update step, allows the evolution of the empirical loss to be modeled using random variables within probabilistic framework. We denote random variables with capital letters (e.g., X), and their realizations with lowercase letters (e.g., z).
Proposition 5.3. The variance of the weights F22 is smaller than the variance of of the theoretical rewards R
Complete proofs of Theorems 5.1, 5.2, and Proposition 5.3 are provided in Appendix A. Additionally, Appendix A includes further analysis of FedGWC. Specifically, Propo- sition A.4 demonstrates that the entries of the interaction matrix P are bounded, while Theorem A.5 establishes a sufficient condition for conserving the sampling rate during the recursive procedure.
The observed loss process th is the outcome of a stochastic process L and the rewards f computed according to Eq. 1, are samples from a random reward 12 supported in (0, 1), whose expectation E [[ is lower for out-of- distribution clients and higher for in-distribution ones. To estimate the expected reward [[ we introduce the r.v. 2 =1/5E. R which is an estimator less affected by noisy fluctuations in the empirical loss. Due to the linear- ity of the expectation operator, the expected reward [R] [[rk"") for the k-th client at round t, local iteration s equals the expected Gaussian reward [[[2]] that, to simplify the no- tation, we denote by Mg. Ps is the theoretical value that we aim to estimate by designing our Gaussian weights TE
6. Wasserstein Adjusted Score
In the previous Section we observed that when clustering clients according to different heterogeneity levels, the out- come must be evaluated using a metric that assesses the cohesion of individual distributions. In this Section, we introduce a novel metric to evaluate the performance of clustering algorithms in FL.. This metric, derived from the Wasserstein distance (Kantorovich, 1942), quantifies the cohesion of client groups based on their class distribution similarities.
appropriately, as it encodes the ideal reward quantify the closeness of the distribution of each client k to the main distribution. Note that the process is stationary by construc- tion. Therefore, it does not depend on ł but differs between clients, as it reaches a higher value for in-distribution clients and a lower for out-of- distriibiit clients.
We propose a general method for adapting clustering metrics to account for class imbalances. This adjustment is particu- larly relevant when the underlying class distributions across clients are skewed. The formal derivation and mathematical details of the proposed metric are provided in Appendix B, We now provide high-level overview of our new metric.
To rigorously motivate the construction of our algorithm and the reliability of the weights, introduce the follow- ing theoretical results. Theorems 5, and 5.2 demonstrate that the weights converge to a finite value and, more impor- tantly, that this limit serves as an unbiased estimator of the theoretical reward μ4-- The first theorem provides a strong convergence result, showing that, with suitable choices of the sequence {ar}: Jt, the expectation of the Gaussian weights T converges l4 in 1,2 and almost surely. In addition, Theorem 5.2 extends this to the case of constant ag proving that the weights still converge and remain unbiased estima- tors of the rewards as t + CC.
Consider a generic clustering metric s, e.g. Davies-Bouldin score (Davies & Bouldin, 1979) or the Silhouette score (Rousseeuw, 1987). Let C denote the total number of classes, and ạ the empirical frequency of the i-th class in the k-tth client's local training set. Following theoretical rea- sonings, as shown in Appendix B, the empirical frequency vector for client k, denoted by (()) is ordered according to the rank statistic of the class frequencies, ie. ot, 2 x =((41) for any i 1,..., metric is defined as the standard clustering metrie s com- puted on the ranked frequency vectors atoy- Specifically, the distance between two clients j and k results in
Theorem 5.1. Let { be a sequence of positive real values, and (Tt) [e the sequence of Gaussian weights. If fas 1 12 (N), then r converges in L?? Further- more, for 1 + X.,
.,. C' 1. The class-adjusted clustering
pig a.s.
(5)
1/2
Theorem 5.2. Let a E 10,1) be a fixed constant, then in the limit & - co, the expectation of the weights converges
k
12) [1()
C
(1)
6

---

Interaction-Awary Gaussian Weighting for Clustered Federated Learning
Table I. FL. haselines in heterogeneous scenarios. Clustering base- lines use FedAvg as aggregation mechanism. We emphasize the fact that FedGWC and CFL automatically detect the number of clusters, unlike IFCA and FeSEM which require tuning the num- ber clusters. higher WAS, denoted by 1. and lower WADB, denoted by 4. indicate better clustering outcomes
Balanced accuracy
FedGWC
IFCA
reSem
FedAig
Automatic
Clustering rounds
FL method
C
Cluster Selection
Acc
WAST
WADB L
0
2000
4000
6000
8000
10000 12000 14000
16000
18000 20000
Communication raunds
Clusterod
IPCA
5
47.5 3.
-0.8 #:0
5.2 4:551 -0.3 1:1
F
FeSem
53.4 1:13
38.4
Cifar 100
CFL
1
41.6 1
Figure 2. Balanced accuracy on C0far for FedGWC (blue curve) with FedAvg aggregation compared the clustered FL baselines. FedGWC detects two splits demonstrating sigmificant improve- ments in accuracy when clustering is performed, leading also to faster and more stable convergence than baseline algorithms.
FedGWC
4
534
0.1
2.4 1.
Classic
FedAvg RedAvigM
11.61:
FL
1
41.5+
RedP nox
41.81 1.
Clastered FT.
TPCA
5
76.7 1:06
0.3 R.
0.5 a.
Fe5om
75.6
0.0 0.
25.6 3:6:::7. 7:
Femniet
CFL,
76.0 a:11
FedGWC
76.1
-022 1:1
18.0 18.0
Classic
FadAvg RedAvigH
76.62
This modification ensures that the clustering evaluation is sensitive to the distributional characteristics of the class imbalance. As we show in Appendix B, this adjustment is mathematically equivalent to assessing the dispersion between the empirical class probability distributions of dif- ferent clients using the Wasserstein distance, also known as the Seatbooovic--Rubensiee metric (Kantorovich, 1942). This equivalence highlights the theoretical soundness of us- ing ranked class frequencies to better capture variations in class distributions when evaluating clustering outcomes in FL.
FL
83.3
FedFrox
75
Algorithm performance is evaluated by averaging the accu- racy achieved the local test sets across the federation, enabling comparison between FL aggregation and clus- tered FL approaches (refer to Appendix F for additional insights). When assessing clustering baselines, we also use the Wasserstein's 's Adjusted Silhouette Score (WAS) and Wasserstein's Adjusted Davies-Bouldin Score (WADB) to quantify the distributional cohesion among clients, an evalu- ation performed posteriori. For detection tasks in visual domains (Section 7.3), we compute the Rand Index (Rand, 1971), clustering metric that compares the obtained clus- tering with a ground truth labeling. Further details on the chosen metrics are provided in Appendices E and B.
7. Experiments
In this section, we present the experimental results on widely used FL benehmark datasets (Caldas al., 2018) includ- ing real-world datasets (Hsu al., 2020), comparing the performance of FedGWC with other baselines from the liter- ature, including standard FL algorithms and clustering meth- ods. A detailed description of the implementation settings, datasets and models used for the evaluation are reported in Appendix F. In Section 7.1, we evaluate our method, FedGWC, against various clustering algorithms, including CFL (Sattler et al., 2020), FeSEM (Long et al., 2023), and IFCA (Ghosh et al., 2020), and standard FL. aggregations FedAvg (McMahan et al., 2017), FedAvgM (Asad et al., 2020), FairAvg (Michieli & Ozay. 2021) and FedP rox (Li et al., 2020), showing also that how our approach is orthogonal to conventional FL aggregation methods.
7.1. FedGWC in heterogeneous settings
In this section, we analyze the effectiveness of FedGWC in mitigating the impact of data heterogeneity compared to standard aggregation methods and other clustered FL algo- rithms. We conduct experiments on Cifar100 (Krizhevsky et al., 2009) with 100 clients and Femnist (LeCun, 1998) with 400 clients, controlling heterogeneity through Dirich- let parameter set to 0.5 for Cifar100 and 0.01 for Femnist, reflecting a realistic class imbalance across clients. Imple- mentation details are provided in Appendix F.
We compare FedGWC against clustered FL baselines (IFCA, F FESEM, CFL) using FedAvg aggregation, as well as stan- dard FL algorithms (FedAvg, FedAvgM, FedProx) For algorithms requiring a predefined number of clusters (IFCA, F FeSEM), we report the best result among 2, 3, 4, and 5 clus- ters, with full tuning details in Appendix I. While IFCA achieves competitive results, its high communication over- head- requiriiggingg each client to evaluate models from every cluster in each round- makes it impractical for cross-device
In Section 7.2, we underscore that FedGWC has the capa- bility surpass FL methods in real-world and large-scale scenarios (Hsu al., 2020).
Finally, in Section 7.3, we propose analyses on class and domain imbalance, showing that our algorithm successfully detects clients belonging to separate distributions. Further experiments are presented in Appendix J. Each client has its own local train and test sets.
7

---

Interaction-Aware Gaussian Weighting for Clustered Federated Learning
FL., serving as an upper bound in our study. FeSEM is more efficient than IFCA but lacks adaptability due its fixed cluster count. Meanwhile, CFL requires extensive hyper- parameter tuning and often produces overly fine-grained clusters or fails to form clusters altogether. In contrast, FedGWC requires only one hyperparameter and provides a more practical clustering strategy for cross-device FL.
uralist (Van Horn et al., 2018), respectively considering the partitions Landmarks-Users, 160K, and (Naturalist-Uset 120K, proposed in (Hsu et al., 2020). Both datasets exhibit high data heterogeneity and involve large number of clients approximately 800 for Landmarks and 2,700 for iNatural- ist. To simulate a realistic cross-device scenario, we set 10 participating clients per round. For this experiment, we com- pare FedGWC against clustered FL baselines (IFCA, CFL) and standard FL aggregation methods (FedAvg, FedAvgM, F FedPror rox, Fai rAvg), The number of clusters for IFCA is tuned between 2 and 5. Due to its high computational cost in large-scale settings, FeSEM was not included in this anal- ysis. Table reports the results: FedcwC with FedAvg aggregation achieves 57.4% accuracy on Landmarks, sig- nificantly outperforming all standard FL methods. In this scenario, FedGWC detects 5 clusters with the best hyperpa- rameter setting (B = 0.5), while IFCA identifies 3 clusters. Similarly, on iNaturalist, FedGWC consistently surpasses FL baselines, reaching an average accuracy of 47.8% with 3 = 0.5 (automatically detecting a partition with 4 clusters). Results in Table 2 remark that, when dealing with realis- tic complex decentralized scenarios, standard aggregation methods are not able to mitigate the effects of heterogeneity across the federation, while, on the other hand, clustered FL provides more efficient solution.
Table 1 presents a comparative analysis of these algorithms in terms of balanced accuracy, WAS, and WADB, using FedAvg as the aggregation method. Higher WAS values indicate better clustering, while lower WADB values sug- gest better cohesion. On Femnist, clustering- based methods perform worse than standard FL aggregation, but as we move to the more complex and realistic Cifar100 scenario, becomes evident that clustered FL is necessary to ad- dress heterogeneity. In this case, FedGWC achieves the best performance in both classification accuracy and clustering quality, with the latter directly influencing the former. The need for clustering grows with increasing heterogeneity, as seen in Table I: standard FL approaches struggle when trained on a single heterogeneous cluster, whereas clustered FL effectively mitigates the heterogeneity effect. This is particularly relevant for Cifar which has a larger num- ber of classes and three-channel images, whereas Femnist consists of grayscale images from only 47 classes.
In Table 1, we present a comparative analysis of these algo- rithms with respect to balanced accuracy, WAS, and WADB, employing FedAvg as the aggregation strategy. Recall that higher the value of WAS the better the clustering outcome, as, for WADB, lower value suggests a better cohesion between clusters. Further details on the metrics used are provided in Appendix E.
7.3. Clustering analysis of FedGWC In this section, we investigate the underlying mechanisms behind FedGNC 's clustering in heterogeneous scenarios on Cifar100. Further experiments on Cifarl0 are presented in Appendix H.
FedGWC detects different client class distributions We explore how the algorithm identifies and groups clients based on the non-IID nature of their data distributions, rep- resented by the Dirichlet concentration parameter a. For the Cifar100 dataset, we apply a similar splitting approach, obtaining the following partitions: (1) 90 clients with = 0 and 10 clients with and 10 clients with a = 1000; and (3) 40 clients with a = 1000, 30 clients with = 0.05, and 30 clients with a = 0. We evaluate the outcome of this clustering exper- iment by means of WAS and WADB. Results in Table show that FedGWC detects clusters groups clients according to the level of heterogeneity of the group.
Notably, both FedGWC and CFL automatically determine the optimal number of clusters based on data heterogeneity, offering a more scalable solution for large-scale cross-devies FL. In contrast CFL, FedGWC consistently produced a reasonable number of clusters, even when using the opti- mal hyperparameters for CFL, which resulted in no splits, thereby achieving performance equivalent to FedAvg.
1000; (2) 90 clients with a 0.5
We observe in Figure 2 that FedCWC exhibits a signifi- cant improvement in accuracy on C0 precisely at the rounds where clustering occurs.
As detailed in Table 9 in J, is orthogonal to any FL. aggregation algorithm, i.e. any FL method can be easily embedded in FedGWC. Our method consistently boosted the performance of FL algorithms for the more heterogencous settings of Cifar100 and Femnist.
FedGWC detects different visual client domains. Here, we focus on scenarios with nearly uniform class imbalance (high values) but with different visual domains investi- gate how FedGWC forms clusters in such settings. We incor- porated various artificial domains (non-perturbed, noisy, and blurred images) C0 dataset under homogeneous condi- tions (a = 100.00). Our results demonstrate that F FedGwC effectively clustered clients according to these distinct do-
7.2. FedGWC in Large Scale and Real World Scenarios We evaluate FedGWC on two large-scale, real-world datasets: Google Landmarks (Weyand et al., 2020) and iNat-
8

---

Interaction-Aware Gaussian Weighting for Clustered Federated Learning
Dataset
FodGBC
CPT.
TPCA
FodAvg
FodAvgM
FodProx
Fa FAirAvg
Google Landmarks
$7.4 10.3
40.5 10.22
49.4 10.3
40.5 102
36.4 L13
40.2 1006
39.0 103
iNaturalist
47.8 =0.2
45.3 +0.1
45.8
20.6
45.3 #0.1
37.7 +14
44.9 #02
45.1 #02
Table 2. Comparison of test accuracy large scale FL datasets Google Landmarks and iNaturalist, between FedGWC and FL baselines all clustered FL algorithms use F FedAvg aggregation. FedGWC outperforms both clustered and standard FL methods detecting and 4 clusters Landmarks and iNaturalist, respectively.
FL methods, FedGWC groups clients by data distributions with flexibility and robustness, simply using the informa- tion encoded by the individual empirical loss. FedGWC successfully detects homogeneous clusters, as proved by our proposed novel Wasserstein Adjusted Score. FedGWC detects splits by removing out-of-distribution clients, thus simplifying the learning task within clusters without in- creasing communication overhead or computational cost. Empirical evaluations show that separately training classi- cal FL algorithms the homogeneous clusters detected by FedGWC consistently improves the performance. Addi- tionally, FedGWC excels over other clustering techniques in grouping clients uniformly with respect to class imbalance and heterogeneity levels, which is erucial to mitigate the ef- feet of non-IIDness FL. Finally, clustering on different class unbalanced and domain unbalanced scenarios, which are correctly detected by FedGWC (see Section 7.3), suggests that FedGWC can also be applied to anomaly client detec- tion and to enhance robustness against malicious attacks in future research.
Table Clustering with three different splits on Cifar100 datasets. FedGWC has superior clustering quality across different splits (homogeneous 1000, heterogeneous a 0.05, extremely
heterogeneous 0.
Clustering method
Dataset (Hom, Het, X Het)
c
WASt
WADB
IFCA
5
-0.9
1.8
(10, 0, 90)
Pešem
5
-0.8 0:
2.6 06 06
FedCXC
IFCA
5
-0.0 0:00
1.
Cifar100
(10, 90,0)
FoSem
5
0.2
12.0 2.
FedGKC
5
0.4 0.1
2:0
IFCA
5
-0.2
1.0
(40, 30, 30)
Pešem
5
-0.2 + u0
33.2 0.0
FodCKC
0.2
=
Table Clustering performance of redGWC is assessed on feder- ations with clients from varied domains using clean, noisy, and blurred (Clean, Noise, Blur) images from Cifar100 datasets. It utilizes the Rand Index score (Rand, 1971), where value close to I represents a perfect match between clustering and labels. Consis- lently FedGWC accurately distinguishes all visual domains. The ground truth number of clusters is respectively 2, 2, and 3.
Dataset
(Clenn, Noise, Blur)
Clustering method
Automatic
Selection
Cluster
(max = 1.0)
f
FedGüC is an efficient clustering-baseee approach that im- proves personalization while reducing communication and computation costs, enhancing the advance in the field of Federated Learning. Providing efficiency and explainability in clustering decisions, FedGWC enables more interpretable and scalable federated learning. Its efficiency makes it well- suited for IoT, decentralized AI, and sustainable AI applica- tions, particularly in privacy-sensitive domains.
Impact Statement
LEGA RwSee FedeNc
(50,50, 50, dy
0.49
1.
Cifar100
(50, 50)
LEGA
X
0.51 a:
0,5 #:00
FedewC
1.0 as
IFCA
0.33 oo
(40,
7a5cm
FedeNC
0.6 n:
mains. Table 4 presents the Rand-Index scores, which assess clustering quality based on known domain labels. The high Rand-Index scores, often approaching the upper bound of 1, indicate that FedGWC successfully separated clients into distinct clusters corresponding to their respective domains. This anaylsis suggests that FedGWC may be applicable for detecting malicious clients in FL, pinpointing a potential direction for future research.
A Acknowledgements
We acknowledge the CINECA award under the ISCRA initiative for the availability of serg-peeffrmancc comput- ing resources and support. A.L. worked under the aus- pices of Italian National Group of Mathematical Physics (GNFM) of INdAM, and was supported by the Project Piano Nazionale di Ripresa Resilienza Next Generation EU NGNRR-NGEU) from Italian Ministry of University and Re- search (MUR) under Grant DM 1 117/2023. This work was partially supported by the Innovation Grant gAIA within the activities of the National Research Center in High Per- formance Computing, Big Data and Quantum Computing (ICSC) CN00000013 Spoke 1. Funded by the European
8. Conclusions
We propose FedGWC, an efficient clustering algorithm for heterogeneous FL settings addressing the challenge of non- IID data and class imbalance. Unlike existing clustered

---

Interaction-Aware Gaussian Weighting for Clustered Federated Learning
Union under Horizon Europe Programme Grant Agree- ment 101 123000 Act.Al. Views and opinions expressed are however those of the author(s) only and do not nec- essarily reflect those of the European Union or European Research Council Executive Agency (ERCEA). Neither the European Union nor the granting authority can be held re- sponsible for them.
Reaching group consensus in clustered federated learn- ing through consensus- based optimization. Journal of machine learning research, 25(214):1-51, 2024.
Cho, J.J., Wang. J., and Joshi, G. Towards understanding biased client selection in federated learning. In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 10351-10375. PMLR, 2022.
References
Davies, D. L. and Bouldin, D. W. A cluster separation measure. IEEE transactions on pattern analysis and machine intelligence, (2):224 227, 1979.
Asad, M., Moustafa, A., and Ito, T. Fedopt: Towards com- munication efficiency and privacy preservation in feder- ated learning. Applied Sciences, 10(8):2864, 2020.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248- leee, 2009.
Bao, W., Wang, H., Wu, J., and He, Optimizing the collaboration structure in cross-silo federated learning. In International Conference Machine Learning, pp- 1718-1736. PMLR, 2023.
Duan, M., Liu, D., Ji, X., Liu, R., Liang, L., Chen, X., and Tan, Y. Fedgroup: Efficient federated learning via decom- posed similarity -based clustering. In 2021 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communicationss Social Computing & Networking C.PAA/DCCood/SscaaaCoSSSStttttco pp. 228-237. IEEE, 2021.
Bonawitz, K., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H. B.. Patel, S., Ramage, D., Segal, A., and Seth, K. Practical secure aggregation for federated learn- ing on user-held data. arXiv preprint arXiv: 1XXi:1611.00483) 2016.
Bonawitz, K., Salehi, F., Konečný, J., McMahan, B., and Gruteser, M. Federated learning with autotuned communication-effiiiii secure aggregation. In 2019 53rd Asilomar Conference on Signals, Systems, and Com- puters, pp. 1222-1226 IEEE, 2019.
Fallah, A., Mokhtari, A., and Ozdaglar, A. Personalized federated learning with theoretical guarantees: A model- agnostic meta-learning approach. Advances in neural information processing systems, 33:3557-3568, 2020.
Briggs, C., Fan, Z., and Andras, P. Federated learning with hierarchical clustering of local updates to improve train- ing on non-iid data. In 2020 international joint conference neural networks (IJCNN), pp. 1-9. IEEE, 2020.
Fan, Z., Hu, S., Yao, J., Niu, G.. Zhang, Y., Sugiyama, M., and Wang, Y. Locally estimated global perturbations are better than local perturbations for federated sharpness- aware minimization. arXiv preprint arXiv:2405. 18890, 2024.
Caldarola, D., Mancini, M., Galasso, F., Ciccone, M., Rodola, E., and Caputo, B. Cluster-driven graph fed- erated learning over multiple domains. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 2749-2758, June 2021.
Ghosh, A., Chung, J., Yin, D., and Ramchandran, K. An efficient framework for clustered federated learning. Ad- vances Neural Information Processing Systems, 33: 19586-19597, 2020.
Hamer, J. Mohri, M., and Suresh, A. T. Fedboost: communication-efficieee algorithm for federated learning. In International Conference on Machine Learning, pp. 3973 PMLR, 2020.
Caldarola, D., Caputo, B., and Ciccone, M. Improving generalization in federated learning by seeking flat min- ima. In European Conference on Computer Vision, pp. 654 672. Springer, 2022.
Harold, J.. Kushner, G., and Yin, G. Stochastic approxima- tion and recursive algorithm and applications. Application of Mathematics, 35(10), 1997.
Caldas, S., Duddu, S. M. K., Wu, P., Li, T., Konečný, J., McMahan, H. B., Smith, V., and Talwalkar, A. Leaf: A benchmark for federated settings. arXiv preprint vrXiv:182..01091 2018.
Hsieh, K., Phanishayee, A., Mutlu, O., and Gibbons, P. The non-iid data quagmire of decentralized machine learning. International Conference on Machine Learning, pp. 4398. PMLR, 2020.
Cao, X., Fang, M., Liu, J., and Gong, N. Z. Fitrust: Byzantine-robust federated learning via trust bootstrap- ping. arXiv preprint arXiv:20/2.13993 2020.
Hsu, T.-M. H., Qi, H., and Brown, M. Federated visual classification with real-worlk data distribution. Com-
Carrillo, A., Trillos, N. G., Li, S., and Zhu, Y. Fedebo:
10

---

Interaction-Aware Gaussian Weighting for Clustered Federated Learning
puter Vision-ECCV 2020: 16th European Conference, Glasgow; UK, August 23-28, 2020, Proceedings, Part X 16, pp. 76-92. Springer, 2020.
McMahan, H. B., Yu, F,, Richtarik, P., Suresh, A., Bacon, D., al. Federated learning: Strategies for improving communication efficiency. In Proceedings of the 29th Conference on Neural Information Processing Systems (NIPS), Barcelona, Spain, pp. 5-10, 2016.
Huang, G., Chen, X., Ouyang, T., Ma. Q. Chen, L., and Zhang, J. Collaboration in participant-centre federated learning: A game-theoretical perspective. JEEE Transac- 2022.
Mendieta, M., Yang, T., Wang, P.. Lee, M., Ding, Z. and Chen, C. Local learning matters: Rethinking data het- erogeneity in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8397-8406, 2022.
fions on Mobile Computing, (2(11)66311666
Kairouz, P., McMahan, H. B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A. N., Bonawitz, K., Charles, Z., Cormode, G. Cummings, R., et al. Advances and open problems in federated learning. Foundations and trends in machine learning, 14(1-20 (11-2):1-210, 2021.
Michieli, U. and Ozay, M. Are all users treated fairly in federated learning systems? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2-22, 2021.
Kantorovich, L. V. On the translocation of masses. In Dokl. Akad. Nauk. USSR (NS), volume 37, pp. 199-201- 1942.
Ng, A., Jordan, M., and Weiss, Y. On spectral clustering: Analysis and an algorithm. Advances in neural informa- tion processing systems, 14, 2001.
Karimireddy, S. P., Jaggi, M., Kale, S., Mohri, M., Reddi, J., Stich, S. U., and Suresh, A. T. Mime: Mimicking centralized stochastic algorithms in federated learning. arXiv preprint arXiv:2008.00 03606, 2020a.
Rand, W. M. Objective criteria for the evaluation of cluster- ing methods. Journal of the American Statistical associa- tion, 66(336):846-85)) 850, 1971.
Karimireddy, S. P., Kale, S., Mohri, M., Reddi, S., Stich, S., and Suresh, A. T. Scaffold: Stochastic controlled averag- ing for federated learning. In International conference on machine learning. pp. 1133,143, PMLR, 2020b.
Robbins, H. and Monro, S. A stochastic approximation method. The annals of mathematical statistics, pp. 400- 407, 1951.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Roussccuw, J. Silhouettes: a graphical aid to the inter- pretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65, 1987,
LeCun, Y. The mnist database of handwritten digits. http://yann. lecun. com/exdb/nnistt/ 1998.
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient- based learning applied to document recognition. Proceed- ings of the IEEE, 86(1) 1):2278-2324 1998.
Lee, T. and Yoon, S. W. Rethinking the flat minima search- ing in federated learning. In Forty-first International Conference on Machine Learning, 2024.
Sattler, F., Müller, K.-R., and Samek, W. Clustered federated learning: Model-agnostic distributed multitask optimiza- tion under privacy constraints. IEEE transactions on neural networks and learning systems, 3213)3370-337 2020.
Li, D. and Wang, J. Fedmd: Heterogenous feder- ated learning via model distillation. arXiv preprint arXiv:1910.03581 2019.
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 1:429-450. 200004550200000 2020.
Shi, Y., Liang, J., Zhang, W., Xue, C., Tan, V. Y., and Bai, S. Understanding and mitigating dimensional collapse in federated learning. IEEE Transactions on Pattern Analy- sis and Machine Intelligence, (15))299992999 2023.
Long, G., Xie, M., Shen, T., Zhou, T., Wang, X., and Jiang, J. Multi-center federated learning: clients clustering for better personalization. World Wide Web, 26(1 (111-500 2023.
T Dinh, C., Tran, N., and Nguyen. J. Personalized federated learning with moreau envelopes. Advances in neural information processing systems, 33:21394-21444 2020.
Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., and Belongie, S. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778, 2018
McMahan, B., Moore, E., Ramage, D., Hampson, S., and Arcas, B. A. Communication-efficeen learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282, PMLR, 2017.

---

Interaction-Aware Gaussian Weighting for Clustered Federated Learning
Von Luxburg, U. A tutorial spectral clustering. Statistics and computing, 17:395-416, 2007.
Wang, J., Liu, Q., Liang, H., Joshi, G., and Poor, H. V. A novel framework for the analysis and design of heteroge- neous federated learning. IEEE Transactions on Signal Processing, 69:5234-5249, 2021.
Weyand, T., Araujo, A., Cao, B., and Sim, J. Google land- marks dataset v2-a large-scale benchmark for instance- level recognition and retrieval. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 2575-2584, 2020.
Yang. Z., Zhang. Y.. Zheng, Y., Tian, X., Peng, H., Liu, T., and Han, B. Fedfed: Feature distillation against data heterogeneity in federated learning. Advances in Neural Information Processing Systems, 36:60397-604233 2023.
Ye, R., Ni, Z.., Wu, F., Chen, S., and Wang, Y. Personal- ized federated learning with inferred collaboration graphs. In International Conference on Machine Learning, pp- 39801-39817 17. PMLR, 2023.
Zhang, Z., Cao, X., Jia, J., and Gong, N. Z. Fldetector: Defending federated learning against model poisoning attacks via detecting malicious clients. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2545-2555, 2022.
Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, V. Federated learning with non-iid data, arXiv preprint orXiv:1806. 00582, 2018
12

---

Interaction-Aware Gaussian Weighting for Clustered Federated Learning
A. Theoretical Results for FedGWC
This section provides algorithms, in pseudo-code, to describe FedCWC (see Algorithms 2 and 1). Additionally, here we provide the proofs for the convergence results introduced in Section 5, specifically addressing the convergence (Theorems 5.1 and 5.2) and the formal derivation on the variance bound of the Gaussian weights (Proposition 5.3). In addition, we also present sufficient condition, under which is guaranteed that the overall sampling rate of the training algorithm does not increase and remain unchanged during the training process (Theorem A.5).
Theorem A.1. Let {ar be sequence of positive real values, and {T& he the sequence of Gaussian weights. If {ar) E (NN)/( then r converges in L2 Furthermore, for + oc,
r
pak 0.8.
(8)
Proof. At each communication round, compute the samples r from R via a Gaussian transformation of the observed loss in Eq. 1. Notice that, due to the linearity of the expectation operator, E[ puk, that is the true, unknown, expected 8 reward. The observed value for the random variable is given by w - 1/5 -=1 "i which sampled from a distribution centered on Ma. Each client's weight updated according to
(9) Since such an estimator follows a Robbins-Monro algorithm, is proved converge in L In addition, I converges the expectation E(sg) = 14 with probability 1, provided that or satisfies Σ121 |ao| oo, and Eiz1 |a, / < co (Harold et al., 1997).
x+1 = (1 - a + aww k
Theorem A.2. Let E (0, 1) be fixed constant, then in the limit t + oo, the expectation of the weights converges the individual theoretical reward pko for each client i- = 1...., K, i.e.,
ETL pi t 00.
(10)
Proof. Recall that 7k (1-1a) (1-a))/ + awt, where w are samples from D. If we substitute backward the value of of we write
% ++== = -1 a) + ow, + a(1 a)
(11)
By iterating up to the initialization term 7% we get the following formulation:
2+1 (1 a) 11
7K Σ a(1 Ca(1- -al1-aw a)
(12)
Since wf. are independent and identically distributed samples from 2g with expected value pkx then the expectation of the weight at the 1-th communication round would be
E T = E (1
Σ
a(1 a
(13)
-=0
that, due to the linearity of expectation, becomes
E (1 a) + Σ
a(1 T
(14)
7=0
If we compute the limit
lim E[T]] lim (1 a)'
Σ a(1 a)
uk
(15)
and since a E (0,1), the first term tends to zero, and also the geometric series converges. Therefore, the expectation of the weights converges to pk, namely
lim ETT pk
(16)
13

---

Interaction-Aware Gaussian Weighting for Clustered Federated Learning
Proposition A.3. The variance of the weights T is smaller than the variance of of the theoretical rewards R
Proof. From 12. we can show that Var(TT) converges value that depends on and the number of local training iterations S. Indeed
(1 (a- (1-a))h² + a
Var(Ti) Var
a) -
(17)
Σ (1 2 var r=0
1 Σ
2 ))
T=0
since 0% = 1/55. R
If we compute the limit, that exists finite due to the hypothesis (0, 1), we get
a²0
0 2 - a 5 5
a o
lim Var(T)) 100
Σ
0)2²
o
(18)
S
T=0
We further demonstrate that the interaction matrix 1s identified by FedGWC is entry-wise bounded from above, established in the following proposition.
Proposition A.4. The entries of the interaction matrix pt are bounded from above, namely for any / there exists positive finite constant Ce >> O such that
Phj S C2
(19)
And furthemorre
lim C = 1. 100
(20)
Proof. Without loss generality we assume that every client the federation sampled, and we assume that a; = a (0, 1) for any 0. We recall, from Eq.3, that for any couple of clients k,j Pe the entries of the interaction matrix are updated according to
p =1 -1- aw?
(21)
If iterate backward until Po we obtain the following update
Kj p+ = (1- +
Σ a(1 at
(22)
We know that, by constructions, the Gaussian rewards wf < at any time t, therefore the following inequality holds
Pt = (1- P
< <(1 (1-a)) a)(1-a)uut
+ Σ a(1
(23)
7=0
T=0
At any round we can define the constant C₁, as
:= (1-aa)-P +a (1- =1 + (1 -1- a) <0.
(24)
T=0
Moreover, since a E (0, 1), by taking the limit we prove that
lim C. lim (1 - P&, + (1- (1-a)tt1 =1
(25)
4000
1-000
14

---

Interaction-Aware Gaussian Weighting for Clustered Federated Learning
Algorithm 2 F BeeGG,CClusee
1: Input: P. Remax, Mlmar,K(:"" -)
2: Output: cluster labels Ynar and number of clusters neat
3: Extract UPVs uy. of from for any k, j
4: Way < K(v uf)) for any j
5: for = 2.....
6: Vn < Amar do
7: DB, + Davies Bouldin(W,, yn)
8: if min, DB, then
9: nd 1
else
11: net + arg min, DB,,
12: end if
13:
end for
Theorem A.5. (Sufficient Condition for Sample Rate Conservation) Consider Kiin as the minimum number of clients permitted per cluster, i.e the cardinality [C,l 2 Kmin for any given cluster TI = 1,... Bet and E (0, 1] to represent the initial sample rate. There exists a critical threshold n > such that, if Kmin 2 n is met, the total sample size does not increase.
Proof.
Let us denote by Pr. the participation rate relative to the n-th cluster, Lee.
Pn. max
P:
Cr
(26)
because, order maintain privacy of the clients' information need sample least three clients, therefore p" is least 3 over the number of clients belonging the cluster. The total participation rate the end of the clustering process is given by
palobal
That
Kn
(27)
where K2 denotes the number of clients sampled within the n-th cluster. If we focus on the term Kn, recalling Equation 26, we have that
K₁, Pn Pn/Cr.l = max
P:
x |C, = max(p/C,ll.3)
(28)
C.
we write Equation 28, by the means of the positive part function, denoted by (x) = max x), we obtain that
Kn = 3 + max p/C.l - 3} = 3 (p|C,/ -3 3)+
(29)
Observe that we are looking for a threshold value for which pelohal = p. i.e. the participation rate remains the same during the whole training process.
Let us observe that K, = plC,.l = p|Ca| 2 3 |c,l 2 n² = =3/p. p. In fact, if we assume that Kmin 2 n," then the following chain of equalities holds
pelobal
Rel
Kr
Σ
net
pl
Rel
C.l
pK K =
K
K
K
n=1
==1
thus proving that Kmin 2 is a sufficient condition for not increasing the sampling rate during the training process.
15

---

Interaction-Aware Gaussian Weighting for Clustered Federated Learning
B. Theoretical Derivation of the Wasserstein Adjusted Score
To address the lack clustering evaluation metrics suited for FL with distributional heterogeneity and class imbal- ance, we introduced theoretically grounded adjustment to standard metrics, derived from the Wasserstein distance, -ubbissttn metric (Kantorovich, 1942). This metric, integrated with popular scores like Silhouette and enables modular framework for posteriori evaluation, effectively comparing clustering outcomes across federated algorithms. In this paragraph, we show how the proposed clustering metric that accounts for class imbalance can be derived from probabilistic interpretation of clustering.
Definition B.1. Let (M, d) be a metric space, and E [1, oo). The Wasserstein distance between two probability measures I and Q over M is defined as
(FP = inf (EL(PQQ)
B(a,y)) Id(z,, y)111/1
(30)
where I(IP, Q is the set of all the possible couplings of I and Q (see Def. B.2).
Furthermore, we need to introduce the notion of coupling of two probability measures.
Definition B.2. Let (M,d) d) be metric space, and IP, Q two probability measures over M. A coupling 7 of P and Q is joint probability measure M x M such that, for any measurable subset A c M,
SA (Jh r(dr, dy) dd))(d)) P(dz) = P(A), SA (ns 7(dx, dy))P(de) Q(dy) - Q(A).
(31)
Let us recall that the empirical measure over M of sample of observations ("1."" ,,,",, ,NN is defined such that for any measurable set C M
C Σ (A)
P(A) = N 1=1
(32)
where 6z, is the Dirac's measure concentrated on the data point Vi. In particular, we aim to measure the goodness of cluster by taking into account the distance between the empirical frequencies between two clients' class distributions and use that to properly adjust the clustering metric. For the sake of simplicity, we assume that the distance d over M is the -nomm. We obtain the following theoretical result to justify the rationale behind our proposed metric.
Theorem B.3. Let s be an arbitrary clustering score. Then, the class-imbalance adjusted score is exactly the metric s computed with the Wasserstein distance between the empirical measures over each client's class distribution.
Proof. Let us consider two clients; each one has its own sample of observations ( mc) and {yr,.... VC where the i-th position corresponds the frequency of training points of class for each client. We aim to compute the p- Wasserstein distance between the empirical measures IP and Q of the two clients, in particular for any dr. dy > 0
P(dx)
8z. (dx),
N
(33)
N
1
(dyy) =
8y (dy)
In order compute (PPP, (P, Q) need to carefully investigate the set of all possible coupling measures [(P, Q) However, since either IP and Q are concentrated over countable sets, it is possible Eq. 31 are the Dirac's measures over all the possible permutations of x, and y:- In particular, by fixing the ordering of æ.. according to the rank statistic E(i), the coupling set can be written as
that the only possible couplings satisfying
{ (ore))))) i E
T(P. Q)
(34)
16

---

Interaction-Aware Gaussian Weighting for Clustered Federated Learning
where S is the of all possible permutations of C elements. Therefore we could write Eq. 30 follows
min E88
y
Σ S(ens dy)
(35)
MMX
since S is finite, the infimum is minimum. By exploiting the definition of Dirac's distribution and the linearity of the Lebesgue integral, for any π E S, we get
C
JAXM x - y 6 (a (dx, dy) =
Σ
J - o (mm)) (dx, dy)
C
MXM
1=1
1=1
(36)
C
(()) - Vt(A)).
C
Therefore, finding the Wasserstein distance between P' and Q boils down to a combinatorial optimization problem, that is, finding the permutation that solves
WF(P.Q)
min RES
I(t)
Vz(())
(37)
The minimum is achieved when n² that is the permutation providing the ranking statistic, i.e. x²())) V(a), since the smallest value of the sum is given for the smallest fluctuations. Thus we conclude that the p-Wasserstein distance between IP and Q is given by
1/P
(P, Q)) = (
C Σ (
v(y) P
(38)
that is the pairwise distance computed between the class frequency vectors, sorted in order of magnitude, for each client, introduced in Section 6, where we chose p 2.
C. Privacy of FedGWC
In the framework of FedGWC, clients are required to send only the empirical loss vectors ff' to the server (Cho et al., 2022). While concerns might arise regarding the potential leakage of sensitive information from sharing this data, it is important to clarify that the server only needs access aggregated statistics, working on aggregated data. This ensures that spient-specific information remains private. Privacy can be effectively preserved by implementing the Secure Aggregation protocol (Bonawitz et al., 2016), which guarantees that only the aggregated results are shared, preventing the exposure of any raw client data.
D. Communication and Computational Overhead of FedGWC
FedGWC minimizes communication and computational overhead, aligning with the requirements of scalable FL systems (McMahan al., 2016). On the client side, the computational cost remains unchanged compared to the chosen FL aggregation, e.g. FedA, clients only required to communicate their local models and a vector of empirical losses after each round. The size of this loss vector, denoted by S, corresponds to the number of local iterations (i.e. the product of local epochs and the number of batches) and is negligible w.r.t. the size of the model parameter space, |θ], In our experimental setup, S = 8, ensuring that the additional communication overhead from transmitting loss values is negligible in comparison to the transmission of model weights.
All clustering computations, including those based on interaction matrices and Gaussian weighting, are performed exclusively on the server. This design ensures that client devices not burdened with additional computational complexity memory demands. The interaction matrix P used in FedGWC is updated incrementally and involves sparse matrix operations, which significantly reduce both memory usage and computational costs.
These characteristics make FedcWC particularly well-suited for cross-device scenarios involving large federations and numerous communication rounds. Moreover, by operating on scalar loss values rather than high-dimensional model parameters, the clustering process in FedGWC achieves computational efficiency while maintaining effective grouping
17

---

Interaction-Aware Gaussian Weighting for Clustered Federated Learning
of clients. The server-side processing ensures that the method remains scalable, even as the number of clients and communication rounds increases. Consequently, FedGäC meets the fundamental objectives of FL. by minimizing costs while preserving privacy and maintaining high performance.
Computational Costs Analysis In this section, present the analysis of computational costs for FedGWC. We would like to emphasize that all additional computational costs are incurred on the server-side, which in typical FL frameworks possesses abundant resources. Importantly, our approach does not increase the computational burden on the client side beyond the standard cost of the chosen FL aggregation method (c.g., backpropagation for F FedAvg).
The most computationally intensive operation in our clustering procedure is spectral clustering. For an interaction matrix involving K clients, where is the number of eigenvectors considered (i.e., the number of clusters sought) and H is the number of k-means iterations required within spectral clustering, a single execution of spectral clustering has complexity of OmK + K mmH) ) (Von Luxburg, 2007). To determine the best partition of the client federation, we evaluate clustering outcomes for varying number of clusters m, from m = ....., Amax where Amax is the maximum number of clusters admitted. This means spectral clustering is performed Amax - 1 times. The total complexity for this phase is the sum of costs for each m.
Ranax
(O(Km² KmH)) OKKnaax + K Hn
This entire clustering phase is performed only a limited number of times during training (typically three to four times on average in our experiments); this small constant factor is absorbed by the O notation. In our experimental settings, H was 300, and naax (the maximum number of clusters explored) ranged between and 10. The parameters namax and H are fixed for each invocation of the clustering phase.
Beyond clustering, other computational steps on the server at each communication round include:
Computing client weights wf. has cost of (SP: where [Pel is the number of participating clients at round I, and S' is the number of local iterations performed by clients. This cost reflects server-side work proportional to the number of participants and their local effort.
Other server-side computations, for instance, operations like computing global sample means and variances from aggregated client information might contribute an additional O(S) per round.
Thus, the total cost for these operations over T communication rounds is O(T(P: In our setting, PP, was typically 10 and S 8.
Considering that the clustering (costing OKKn + K ( ) occurs only a few times, the overall additional computa- tional overhead introduced by our method can be expressed as:
OKK max + K K +
(39)
By applying properties of Landau notation and considering parameters Fmnax. H, S, and the client sampling rate p p= IP:l/K
as constants or small fixed values (which do not scale with K or T, a common consideration in large-scale FL systems (Bonawitz et al., 2019)), the overall computational cost can be simplified. Substituting [Pr| = =KK into the expression:
OKn + K H + TTKS + TS).
This simplifies with respect to the total number of clients K and the total number of communication rounds T:
O(K) + O(K) O(TK) + O(T) O(TK - T TKK
This analysis demonstrates that FedGWC not only places all computationally intensive operations on the server but also introduces very limited overall computational cost, primarily scaling with TK.
E. Metrics Used for Evaluation
E.1. Silhouette Score
Silhouette Score is a clustering metric that measures the consistency of points within clusters by comparing intra-cluster and nearest-cluster distances (Rousseeuw, 1987). Let us consider a metric space (M, d). For a set of points (f,,...,,N) c M
18

---

Interaction-Aware Gaussian Weighting for Clustered Federated Learning
and clustering labels Ci..... Cna- The Silhouette score of data point x; belonging cluster C, is defined as
b a, m b
(40)
where the values b, and a, represent the average intra-cluster distance and the minimal average outer-cluster distance, i.e.
a, C.l (=)))((m))
2,)
(41)
b min jf
Σ d( EyEC,
Cg
The value of the Silhouette score ranges between -1 and +1, i.e. 8; E [-1, In particular, a Silhouette score close to I indicates well-clustered data points, 0 denotes points near cluster boundaries, and suggests misclassified points. order to evaluate the overall performance of the clustering, a common choice, that the one adopted in this paper, is average the score value for each data point.
E.2. Davies-Bouldin Score
The -evies-BBuuldin Score a clustering metric that evaluates the quality of clustering by measuring the ratio of intra-cluster dispersion to inter-cluster separation (Davies & Bouldin, 1979). Let consider metric space (M, d), set of points .NN C M, and clustering labels C The Davies-Bouldis score is defined the average similarity measure Ray between each cluster C; and its most similar cluster Cy:
DB =
max Rij j#1
(42)
Rel
where Raj is given by the ratio of intra-cluster distance to inter-cluster distance Diy, L.e.
Rag = S S
(43)
Dij
with intra-cluster distance S; defined as
S; = C.l
Σ c(
(44)
ZLEC.
where & denotes the centroid of cluster C, and Dij = d(c,, ey) is the distance between centroids of clusters C2 and Cy. A lower Davies-Bouldin Index indicates better clustering, as reflects well-separated and compact clusters. Conversely, higher DBI suggests that clusters are less distinct and more dispersed.
F.2.1. 1. RAND INDEX
Rand Index is clustering score that measures the outcome of clustering algorithm with respect ground truth clustering label (Rand, 1971). Let us denote by a the number of pairs that have been grouped in the same clusters, while by b the number of pairs that have been grouped in different clusters, then the Rand-Index is defined as
RI b
(45)
2
where N denotes the number of data points. In our experiments we opted for the Rand Index score to evaluate how the algorithm was able to separate clients in groups of the same level of heterogeneity (which was known a priori and used as ground truth). A Rand Index ranges in [0, 1], and value of signifies perfect agreement between the identified clusters and the ground truth.
F. Datasets and implementation details To simulate a realistic FL environment with heterogeneous data distribution, we conduct experiments on Cifar100 (Krizhevsky et al., 2009). As a comparison, we also run experiments on the simpler Cifar10 dataset (Krizhevsky et al., 2009).
19

---

Interaction-Aware Gaussian Weighting for Clustered Federated Learning
1 and Cifar 100 are distributed among K clients using a Dirichlet distribution (by default, we use a 0.05 for Cifar10 and a = 0.5 for Cifar100) create highly imbalanced and heterogeneous settings. By default, use K 100 clients with 500 training and 100 test images. The classification model is a CNN with two convolutional blocks and three dense layers. Additionally, we perform experiments on the Femnist dataset (LeCun, 1998), partitioned among 400 clients using a Dirichlet distribution with a = 0.01. The choice of different values for a is made to ensure comparable levels of heterogeneity through the adjustment of the Dirichlet parameter in relation to the number of classes. In particular, by employing a = 0.5 for Cifar 100 and a = 0.01 for FeMNIST, we preserve a consistent ratio of a/C across the datasets. This approach aids in regulating the level of class imbalance, thereby ensuring comparability in heterogeneity across different datasets. In these experiments, we employ LeNet5 as the classification model (LeCun al., 1998). Local training each client uses SGD with a learning rate of 0.01, weight decay of 4 10-², and batch size 64. The number of local epochs is I, resulting in 7 batch iterations for Cifar10 and Cifar 100 and 8 batch iterations for Femnist. The number of communication rounds is set to 3,000 for Femnist, 10,000 for Cifar and 20,000 for Cifar100, with 10% client participation rate per cluster. For FedGWC we tuned the
3 E {0.1, 0.5, 1, 2),), i.e. the spread of the RBF kernel, and we set the tolerance e to 10-5, constant value at = a equal to the participation rate, iee. 10%. FeSEM's and IFCA's 's number of clusters was tuned between 2,3.4, and 5. Each client has its own local training and test sets. We evaluate classification performance using balanced accuracy, computed per client as the average class-wise recall. The overall federated balanced accuracy is then obtained by averaging client-wise balanced aceuracy, optionally weighted by test set sizes, to account for heterogeneous data distributions.
Large Scale experiments are conducted on Google Landmarks (Weyand et al., 2020) with K = 823 clients and (Van Horn et al., 2018) with K = 2714 clients, as partitioned in (Hsu et al., 2020). For Landmarks and iNaturalist, we always refer to the Landmark- 10K and AaturalisttUseers -10k partition, respectively. The classification model MobileNetV2 architecture (Sandler et al., 2018) with pre-trained weights on ImageNet-IK dataset (Deng et al., 2009) optimized with SGD having learning rate of 0.1. To mimic real world low client availability we employed 10 sampled clients per communication round, with total training of 1000 and 2000 communication rounds, with 7 and 5 batch iterations respectively. For FedGWC we tuned the for iaturalist and to 10-4 for Landmark, constant value 0g a = .1. IFCA's 's number of clusters was tuned between 2,3,4, and 5. Each client has its own local training and test sets. Performance in large scale scenarios are evaluated by averaging the accuracy achieved on the local test sets across the federation.
3 E (0.1, 0.5, 1,2,4) i.e. the spread of the RBF kernel, and we set the tolerance c to 10-2
G. Sensitive Analysis beta value RBF kernel
This section provides sensitivity analysis for the 33 hyper-parameter of the RBF kernel adopted for FedGWC. The results of this tuning are shown in Table 5.
Table 5. A sensitivity analysis on the RBF kernel hyper-paramcter 3 is conducted. We present the balanced accuracy for FedGWC on the C0far10, Cifar100, and Femnist datasets for / E (0.1, 0.5, 1.0, 2.0, 1.0). It is noteworthy that FedGWC demonstrates robustness variations in this hyperparameter.
B
Cifar100
Femnist
Google Landmarks
iNaturalist
0.1
49.9
76.0
55.0
47.5
0.5
53.4
76.0
57.4
47.8
1.0
49.5
76.0
56.0
47.5
2.0
50.9
75.6
57.0
47.2
4.0
52.6
76.1
55.8
47.1
H. Additional Experiments: Visual Domain Detection in Cifar10
In this section we present the result for the domain ablation discussed in Section 7.3 conducted on Cifar10 (Krizhevsky et al., 2009). We explore how the algorithm identifies and groups clients based on the non-lID nature of their data distributions, represented by the Dirichlet concentration parameter a. We apply similar splitting approach, obtaining the following partitions: (1) 90 clients with a = O and 10 clients with a = 100; (2) 90 clients with a = 0.5 and 10 clients with a = 100; and (3) 40 clients with a = 100, 30 clients with a = 0.5, and 30 clients with a = 0. We evaluate the outcome this clustering experiment by means of WAS and WADB. Results in Table 3 show that FedGwC detects clusters groups clients
20

---

Interaction-Aware Gaussian Weighting for Clustered Federated Learning
Table Clustering with three different splits Cifar10. FedGWC has superior clustering quality across different splits (homogeneous
Hom, heterogeneous Het, extremly heterogeneous. X Het)
Clustering method
Dataset
(Hom, Het, X Het)
C
WAS
WADB
IFCA
/
(10, 0, 90)
eeeee
-0.0 0.1
12.0 + 2.0
FedGAC
0.1
0.2
C0
IPCA
(10, 0)
7eSum.
3 -0.0
12.0 +
FedGwC
0.2 tue
0.6 + 1.0
IFCh
2
-0.2 = oue
1.0 = 0.e
(40, 30, 30)
7eScm
0.1 = 0.1
20.6 7.1
FedsWC
0.6
Table Clustering performance of FeeGGC is assessed federations with clients from varied domains using clean, noisy, and blurred (Clean, Noise, Blur) images from Cifar 10 dataset. It utilizes the Rand Index score (Rand, 1971), where a value close to represents a perfect match between clustering and labels. Consistently FedGWC accurately distinguishes all visual domains.
Clustering
Dataset
(Clean, Noise, Blur)
C
Rand
method
IFCA
1
0.5
(50,5000 0)
FeSem
2
0.49 4:02
FedGWC
2
1.0 n.
IFCA
1
0.5 0.
Cifar10
050,0,50)
FeSem
2
0.5 I.
FedGwC
2
1,0 n0
IFCA
1
0.33 a 3.00
(40, 30)
FeSem
3
0.34 0.1
FedGwC
according to the level of heterogeneity of the group.
I. Evaluation of IFCA and FeSEM algorithms with different number of clusters
This section shows the tuning of the number of clusters for the IFCA and FeSEM algorithms, which cannot automatically detect this value. The results of this tuning are shown in Table 8.
J. Further Experiments
In Table we show that FedGWC is orthogonal to FL aggregation, which means that any algorithm including personalized FL algorithms- can be easily embedded in our clustering setting, improving the final performance. We show this in Table 9, where we also include personalization algorithms PFedMe (T Dinh et al., 2020) and Per Fadd-g (Fallah et al., 2020).
Note that FedGWC does not directly compare personalization methods, it would result in unfair comparison, since personalization methods require more local resources and overhead than clustering methods. Personalization methods aim to create models uniquely tailored to each client, a distinct objective from clustering, which instead aims to identify and exploit shared structures within client clusters. Given these fundamental distinctions in their objectives - highly individualized models versus robust cluster level representations - and their differing resource demands, a direct benchmark of their standalone effectiveness would be misleading. Therefore, exploring their potential integration is a more insightful line of investigation. We rather show that applying FedGWC on top personalization algorithms allows to increase model performance. Figure illustrates the clustering results corresponding varying degrees of heterogeneity, as described in Section 7.3. As per FedGwC, the detection of clusters based different levels of heterogeneity in the Cifar10 dataset is achieved. Specifically, an examination of the interaction matrix reveals clear distinction between the two groups. In Figure 6, we show that in class-balanced scenarios with small heterogeneity. like Cifar10 with 100, FedGWC successfully detects one single cluster. Indeed, in homogeneous scenarios such as this one, the model benefits from accessing more data from all the clients.
Figure shows how the MSE converges to a small value as the rounds increase for a C0 experiment.
21

---

Interaction-Aware Gaussian Weighting for Clustered Federated Learning
Root: split
Level 1 Cluster 0-0: split
Level 1 Cluster 0-1: split
Group 0
Group
Group
Group 2
6.04
6:3
0.0
-000
-406
-032
-182
-001
0.00
800
102
035
-2015
-4893 -0003
5.00
3223
0003
0.01
0330
0.225
8.38
-0.0
0.50
0.52
0.54
0.88
338
133
Level 1 Cluster 0-2: split
Level 2 Cluster 0-2-0: no split
Level 2 Cluster 0-2-1: no split
Group
Group
Group
0.33
Group
511
6:30
6.00
-0.00
-0.06 +0.32
8115
-0.02
6.52
Figure 3. Cluster evolution with respect to the recursive splits in FedGWC on Cifar100, projected the spectral emhedded bi-dimensional space. From left to right, top to bottom, we can see that FedGWC splits client into cluster, until certain level of intra-cluster homogeneity is reached
10
smoothed MSE
10
1
2000
4000
6000
0000
10000
round
Figure 4. Interaction matrix convergence: the y-axis MSE in logarithmic scale w.r.t. communication rounds in the r-axis on Cifar10, with Dirichlet parameter 0.05.
22

---

Interaction-Aware Gaussian Weighting for Clustered Federated Learning
Table Performance of for baseline algorithms for clustering in FL FeSKM, and FCA, w.r.t the number of clusters
Clustering
C
Acc
method
46.7 +
44.0 + 1.6
IFCA
Cifar 100
4
45,1 2.6
47.5 3.5
2
43.3 1: 1.3
48.0 + 1.9
FeSem
50,9 +
53.4 1
76.1 0.1
75.9 +
IFCA
4
76.6 0.1
Femnist
76.7 0.6 5
75.6 0.2
3
75.5 0.5
FeSem
75.0 0.1
74.9 0.1
Table FedGwC orthogonal to FL aggregation algorithms, improving their performance in heterogencous scenarios (Cifarl00 with 0.5 and Femnist with 0.01). This shows that FedGWC and clustering are beneficial in this scenarios, also with personalization methods FeMMe (T Dinh et al., 2020) and Per-FedAva (Fallah et al., 2020)
FL method
Cifar100
Femnist
No Clusters
FedGWC
No Clusters
FedGWC
FedAvg
41.6 + 1.3
53.4 + 0.4
76.0 =.1
76.1 + 0.1
FedAvgM
41.5 0.5
50.5 0.3
83.3 0.3
83.3 0.4
FedProx
41.8 :::
49.1 +: 1.0
75.9 0.2
1: 0.2
рFedME
93.4 0.1
93.5 0.1
63.6 0.3
63,9 0
per- Fer-ve
0.1
93.5 0.1
71,2 0.2
72.0 0.2
As Figure 7 illustrates, FedGWC partitions the Cifar100 dataset into clients based on class distributions. Each cluster's s distribution is distinct and non-overlapping. demonstrating the algorithm's efficacy in partitioning data with varying degrees of heterogeneity. In Figure 8, we report the domain detection on Cifar 100, where 40 clients have clean images, 30 have noisy images, and 30 have blurred images. Table 4 shows that FedGWC performs good clustering, effectively separating the different domains.
23

---

Interaction-Aware Gaussian Weighting for Clustered Federated Learning
Interaction Matrix
Scaled Affinity Matrix
Spectral Clustering Results
3.2
Dusar i
1.22
54
18.-
5
3.00
U1
Figure 5. Homogeneous (Cifarl0 a = 100) vs heterogeneous clustering (Cifarl0 a = 0.05). The interaction matrix at convergence and the corresponding scaled affinity matrix are the left. The scatter plot in the 2D plane with spectral embedding is on the right. It is possible to see that the algorithm perfectly separates homogeneous clients (orange) from heterogeneous clients (black)
Interaction Matrix
Scaled Affinity Matrix
Spectral Clustering Results
3.4
1.8
5.4
382
8
3.00
82
0.2
-1.3
Figure 6. Homogeneous case (Cifar10 a = 100). The interaction matrix convergence and the corresponding scaled affinity matrix are on the left. The scatter plot the 2D plane with spectral embedding is on the right. In the homogeneous case where no clustering needed, FedGW does not split the clients.
Cluster 0 vs Cluster 1
Cluster 0 vs Cluster 2
Cluster 0 Cluster 3
Cluster 0
Cluster Cluster
Cluster D
Cluster 1
Cluster 3
Clssse Indexes
Classes Indexes
Classes Indexes
0.03 862
334 Classes Frequencies
882 cas 666 Classes Frequencies
6.0
ais Classes Frequencies
0,00
0,2
Cluster 1 vs Cluster 2
Cluster 1 vs Cluster 3
Cluster 2 vs Cluster 3
Cluster 1 Cluster 2
Cluster Cluster
Cluster 2 Cluster 3
Casses Indexes
Casses Indexes
Classes Indexes
Classes Frequencies
Classes Frequencies
Classes Frequencies
Figure 7. Class distributions among distinct clusters detected by FedGWIC on Cifar100. Specifically, we examine the class distributions for each pair of clusters, demonstrating that (1) the clusters were identified by grouping differing levels of heterogeneity and (2) there is, most cases, an absence of overlapping classes.
24

---

Interaction-Awart Gaussian Weighting for Clustered Federated Learning
Interaction Matrix
Scaled Affinity Matrix
Spectral Clustering Results
Noise
0.6
Blur
0.2
46
24
-200
3. -103
-001
0.38
8.62
0.6
U1
Figure 8. FedGWC in the presence of domain imbalance. Three domains on Cifar100: clean clients (unlabeled), noisy clients (+), and blurred clients (x). Left: is the interaction matrix P convergence from which it is possible see client relations. Center: The affinity matrix W computed with respect to the UPVs extracted from P, and on which FedGW_Cluster FedGWC clusters the clients according to the domain, proved results in Table 4.
www.ccluattc.o www.:cuuuueiinnn is performed. We can see that
25


Reviews:
[{"review_id": "d8ZN4ex81K", "reviewer": "Reviewer_GoFZ", "summary": "This paper proposes FedGWC, a new clustered FL algorithm to tacke data heterogeneity and class imbalance among clients. FedGWC clusters clients based on their empirical losses, using a Gaussian reward mechanism. They also propose a new clustering metric, Wasserstein Adjusted Score, to evaluate cluster cohesion. The proposed algorithm is tested on benchmark datasets with standard partitions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes. The algorithm is tested on benchmarking datasets.", "methods_and_evaluation_criteria": "Generally yes. It is intuitive to use empirical loss as signal for clustering. And the contruction of gamma in Subsection 4.1 is convincing. \n\nHowever, I am a little bit confused about equation (3) in Subsection 4.2. In FL, typicall we want to choose $P_t$ large enough such that the aggregation is not significantly influenced by individual client's update.  I believe each $w_k^t$ measures the similarity from client $j$ to global, and is almost not relevant to client $j$ when the number of selected clients is large. it would be great if the author can show some evidence showing that the P matrix can capture client distribution similarity.", "theoretical_claims": "I briefly look through the statements and don't see any significant issues since they seem standard. However, I did not check the correctness of the proof.", "experimental_designs_or_analyses": "Yes. I believe the experiment part is very solid. For datasets, the proposed algorithm uses both artificial-partitioned dataset and real federated datasets. The partition is also not designed for clustered FL. This is very different from many previous clustered FL papers demonstrating the genereralization of this algorithm. The author also compared the algorithm to important clustered FL baselines. \n\nThe soundness of the experiment can be further improved, if the author can compare to personalized FL baselines that are not restricted to clustered FL.", "relation_to_broader_scientific_literature": "No comments.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "[1] is also a recent work of clustered FL considering both data quantity imbalance and non-IIDness. I suggest the author to discuss the difference between the proposed work and [1]. \n\n[1] Optimizing the Collaboration Structure in Cross-Silo Federated Learning. ICML 2023", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EfFRPM192B", "reviewer": "Reviewer_s2xj", "summary": "This paper focus on the clustered FL method to mitigate the non-iid problem in FL. FedGWC groups clients based on the data distribution. Gaussian reward mechanism is used to form homogeneous clusters. Comprehensive experiments demonstrate this method achieve better performance.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "See weakness", "methods_and_evaluation_criteria": "See weakness", "theoretical_claims": "See weakness", "experimental_designs_or_analyses": "See weakness", "relation_to_broader_scientific_literature": "Not relevant", "essential_references_not_discussed": "The FL works referenced in related work are relatively outdated and new FL works should be added[R1-R4].\n\nReferences:\n\n[R1] Fan, Ziqing, et al. \"Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimization.\" *Forty-first International Conference on Machine Learning*.\n\n[R2] Yang, Zhiqin, et al. \"Fedfed: Feature distillation against data heterogeneity in federated learning.\" *Advances in Neural Information Processing Systems* 36 (2023): 60397-60428.\n\n[R3] Lee, Taehwan, and Sung Whan Yoon. \"Rethinking the flat minima searching in federated learning.\" *Forty-first International Conference on Machine Learning*. 2024.\n\n[R4] Shi, Yujun, et al. \"Understanding and mitigating dimensional collapse in federated learning.\" *IEEE Transactions on Pattern Analysis and Machine Intelligence* 46.5 (2023): 2936-2949.", "other_strengths_and_weaknesses": "- Concerns about the efficiency of this algorithm arise from the large number of training rounds required for it to converge.\n- Why the author do not inquiry the data distribution from clients directly, rather infer by analyzing empirical loss function. Dose this induce privacy concern， if we can get the distribution of local data directly, what is the meaning of Gaussian Weighting Mechanism?\n- Whether the transmission loss is an individual loss for each sample or a loss for each client？\n- Is that the same for $\\theta_{(1)}, \\cdot,\\theta_{(n_{cl})} $, for different groups.\n-  It is unclear about how  to get $m^{t,s}$ in Equation (1).\n- Why the Dirichlet parameter and client number is different for CIFAR-100 and Femnist, it is better to align the FL setting, it should be isolated from dataset selection. To be convincing is not an FL setting picked specifically for each dataset.", "other_comments_or_suggestions": "See above", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z6a7khBIcD", "reviewer": "Reviewer_3mtD", "summary": "This paper introduces FedGWC (Federated Gaussian Weighting Clustering), a clustered federated learning (CFL) framework designed to address data heterogeneity and class imbalance. The key idea behind FedGWC is to group clients into homogeneous clusters based on their data distributions, enabling personalized model training within each cluster. Key contributions of this paper include:\n\n- Gaussian Weighting Mechanism: Clients are clustered by analyzing their empirical loss landscapes. A reward system quantifies alignment between client data distributions and cluster averages. Gaussian weights are introduced to keep track over time of these rewards, computed as a running average of the rewards.\n- Interaction Matrix and Spectral Clustering: Pairwise client similarities (the Gaussian weights) are encoded in an interaction matrix, refined into an affinity matrix using RBF kernels. Spectral clustering partitions clients into groups, dynamically adjusting clusters based on convergence criteria of the interaction matrix.\n- Wasserstein Adjusted Score: A novel metric, Kantorovich–Rubinstein metric, evaluates cluster cohesion under class imbalance, leveraging Wasserstein distance (with standard clustering quality metrics) to assess distributional alignment of ranked class frequencies.\n\nThe authors demonstrate that FedGWC outperforms existing CFL baselines (classical ones: IFCA, Sattler, and a recent one FedSem) and standard federated learning (FL) methods (FedAvg, FedProx) on benchmark datasets (Cifar100, Femnist) and large-scale real-world datasets (Google Landmarks, iNaturalist). Numerical results (Table 1 - 4) showed that FedGWC achieves higher accuracy and better clustering quality, except for the Femnist dataset, effectively handling domain shifts and class imbalance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "The majority of claims are supported by rigorous theoretical analyses and empirical (numerical experiments) evidence.", "methods_and_evaluation_criteria": "- Methods: Gaussian weighting (loss-based similarity), interaction matrix, and spectral clustering are well-suited for clustering clients in CFL.\n- Evaluation: Standard FL benchmarks (Cifar100, Femnist) and large-scale datasets (Landmarks, iNaturalist) are appropriate for evaluating FedGWC's performance.\n- Potential Limitations: the authors could consider more diverse datasets (e.g., NLP) to validate the generalizability of FedGWC.", "theoretical_claims": "Theorem 5.1 and Theorem 5.2 on the convergence of Gaussian weights are checked (proofs in Appendix A).", "experimental_designs_or_analyses": "The experimental design is largely sound for validating FedGWC’s core claims, with appropriate benchmarks and metrics. (See the last paragraph of the \"Summary\" section for details.) However, the authors did not provide a detailed analysis of why FedGWC performed far worse on the Femnist dataset, which is the simplest dataset, compared to the baselines.", "relation_to_broader_scientific_literature": "The paper’s contributions advance clustered federated learning (FL) by addressing key limitations of prior work and successfully integrating insights from optimization, distribution alignment, etc.\n\nP.S. I don't quite understand the exact meaning of \"Broader Scientific Literature\". I assume it refers to the broader context of the paper's contributions (mainly compared to existing methods in literature) in the whole field of federated learning.", "essential_references_not_discussed": "Essential references are well-discussed in the paper.", "other_strengths_and_weaknesses": "Strengths and weaknesses are discussed in previous sections.", "other_comments_or_suggestions": "- Why are the algorithms presented in the appendices rather than included in the main paper?\n- Algorithm name capitalization in the References: e.g. Fldetector -> FLDetector. Use curly braces to enclose such terms in the bib file to preserve the capitalization.", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IxBpco9keC", "reviewer": "Reviewer_vYq3", "summary": "This paper proposes a novel federated learning (FL) method called FedGWC (Federated Gaussian Weighting Clustering), which aims to mitigate the challenges of data heterogeneity and class imbalance in FL by clustering clients based on their data distributions. This method allows for the creation of more homogeneous client clusters, leading to more personalized and robust federated models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "N/A.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "Strengths:\n1. The Gaussian reward mechanism provides a statistical method to determine the similarity between clients based on their empirical loss.\n2. Comprehensive theoretical foundation and convergence guarantees.\n\nWeakness:\n1. Although the appendix includes experiments on additional datasets, the performance of baseline methods on these datasets is not provided.\n2. The empirical loss may fail to fully capture the subtle differences in data distributions across different client datasets.\n3. The authors claim that all clustering computations, including those based on interaction matrices and Gaussian weighting, are performed exclusively on the server. However, a detailed complexity analysis is needed for clarification.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "N/A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents an interesting approach. Even though there was no rebuttal, I feel the results are sufficiently interesting to be accepted.
Decision: Accept (poster)