Enhancing Clustered Federated Learning: Integration of Strategies and Improved Methodologies
Authors: Yongxin Guo
Xiaoying Tang
Tao Lin
Venue: ICLR 2025 Poster (2025)
Keywords: Federated Learning
Clustering
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose a unified framework for clustered FL algorithms and improve the techniques within the framework.

Abstract: Federated Learning (FL) is an evolving distributed machine learning approach that safeguards client privacy by keeping data on edge devices. However, the variation in data among clients poses challenges in training models that excel across all local distributions. Recent studies suggest clustering as a solution to address client heterogeneity in FL by grouping clients with distribution shifts into distinct clusters. Nonetheless, the diverse learning frameworks used in current clustered FL methods create difficulties in integrating these methods, leveraging their advantages, and making further enhancements.  
To this end, this paper conducts a thorough examination of existing clustered FL methods and introduces a four-tier framework, named HCFL, to encompass and extend the existing approaches. Utilizing the HCFL, we identify persistent challenges associated with current clustering methods in each tier and propose an enhanced clustering method called HCFL$^{+}$ to overcome these challenges. Through extensive numerical evaluations, we demonstrate the effectiveness of our clustering framework and the enhanced components. Our code is available at \url{https://github.com/LINs-lab/HCFL}.

Published as a conference paper at ICLR 2025
ENHANCING CLUSTERED FEDERATED LEARNING: IN- TEGRATION OF STRATEGIES AND IMPROVED METHOD-
OLOGIES
Yongxin Guo
Xiaoying Tang- Tao Lin
School of Science and Engineering. The Chinese University of Hong Kong. Shenzhen 518172, China Shenzhen Institute of Artificial Intelligence and Robotics for Society (AIRS), Shenzhen, China
*Guangdong Provincial Key Laboratory of Future Networks of Intelligence, Shenzhen, China "School of Engineering, Westlake University
6 "Research Center for Industries of the Future, Westlake University
ABSTRACT
Federated Learning (FL) is evolving distributed machine learning approach that safeguards client privacy by keeping data on edge devices. However, the variation in data among clients poses challenges in training models that excel across all local distributions. Recent studies suggest clustering as solution to address client heterogeneity in FL by grouping clients with distribution shifts into distinct clusters. Nonetheless, the diverse learning frameworks used in current clustered FL methods create difficulties in integrating these methods, leveraging their advantages, and making further enhancements. To this end, this paper conducts thorough examination of existing clustered FL methods and introduces a four-tier framework, named HCFL, to encompass and extend the existing approaches. Utilizing the HCFL, we identify persistent challenges associated with current clustering methods in each tier and propose enhanced clustering method called HCFL* to overcome these challenges. Through extensive numerical evaluations, we demonstrate the effectiveness of our clustering framework and the enhanced components. Our code is available at https: / / thub. cwm/com////////o LIN--- lab/ HCFL.
INTRODUCTION
Federated Learning (FL) is a privacy-focused distributed machine learning approach. In FL., the server shares the model with clients for local training, and the clients send parameter updates back to the server. The clients will not share their raw data with servers, ensuring privacy. However, the non-iid client data distribution leads to significant performance drops for FL algorithms (McMahan et al., 2016; Li et al., 2018; Karimireddy et al., 2020; 2019). To address data heterogeneity, traditional FL fo- cuses training single global model that performs well across all local distributions (Li et al., 2021; 2018; Tang et al., 2022; Guo et al., 2023a). However, relying solely on a global model may not ad- equately handle the heterogeneous client distributions. As remedy, clustered FL. methods have been proposed to group clients into different clusters based on their local distributions Numerous studies have demonstrated the superiority of clustered FL methods over ingle-model FL approaches (Long et al., 2023; Sattler et al., 2020b; Ghosh et al., 2020; Marfoq et al., 2021; Guo et al., 2023b).
Diverse learning frameworks pose challenges on enhancing the clustered FL. Despite the suc- cess of current clustered FL methods, the use of diverse learning frameworks poses challenges in integrating different algorithms, gathering their advantages, and achieving further improvements. For instance, FedEM (Marfoq et al., 2021) excels in addressing complex mixture distribution scenarios and performs admirably on challenging tasks. However, necessitates a predefined number of clusters, constraining its practicality. In contrast, adaptive clustering techniques such CFL (Sattler et al., 2020b) can autonomously determine the number of clusters. Nonetheless, CFL cannot be
*Corresponding author:
In this study, address the issue of supervised clustered FI., which may differ from the unsupervised clustered FL. examined by Ding al. (2023); Qiao al. (2024).

---

Published as a conference paper at ICLR 2025
Tier Cluster Foomulations
Loamiing rg Parametens for Glaters
masimaz conditional distrbbuiion Plylx)
maximina joint distribution Pxx, y)
madimiz varablle
relationship FixiP
Tier Cluster Weights Calc.fattion
Hard Clustering
Soft Cuustering
Clasert
CLaBr
Corrett Clusters
Cierk
Oiert CCiene
Adaptive Clustering
Fored Cluster Number
G
Add New Cluster
H
Merga/ Perrgaee Clusters
Find Tochniques Muusinn Chonaer Numbers
Clube
Caler
CLABY
Quater
Tier Client Distance Metrics
Cetermine Adjunting Murbben
local ents
parameters
local feature norms
Figure I: Overview of the HCFL. The HCFL encompasses the existing clustered algorithms through the design of four tiers, including cluster f formulations, which maximize conditional distribution, joint distribution, or variable relationships; cluster weights calculation, including soft clustering and hard clustering; adaptive clustering procedure, including using predefined number of clusters, automatically adding merge and remove existing clusters, client distance metrics, including using distance on clients' local gradients, clients' local model parameters, or clients' local feature norms. The four tiers collaborate to a comprehensive clustered FL learning process, as shown in the left part of the figure. For instance, CFL can be described by the A, D. G, and J, while A, E, and F cover FedEM.
clusters,
seamlessly integrated with soft clustering methods like FedEM, thereby limiting its effectiveness in handling complex mixture distribution tasks.
Consolidating existing methods as a solution. To tackle these challenges, we believe there is need to develop holistic learning framework for supervised clustered FL methods, allowing us to seamlessly combine their advantages. In this paper, we introduce HCFL, holistic clustered FL algorithm framework incorporating (1) a unified clustering objective function that handles both soft and hard clustering and (2) a unified, four-tier clustering procedure paradigm (as shown Figure 1).
The HCFL framework enables the flexible combination of existing techniques within each tier, unlocking new benefits beyond the mere recovery traditional methods. For example, FedRC Guo et al. (2023b) demonstrates strong generalization performance but cannot automatically determine the number of clusters, whereas CFL Sattler et al. (2020a) excels at determining the number of clusters but lacks generalization. The HCFL framework allows for the integration of the strengths of both FedRC and CFL, achieving both superior generalization and personalization performance (see Table 1). Additionally, enabling both cluster removal and addition yields comparable- better pereorr--eformccceee than baseline algorithms, with a significantly reduced number of clusters (see Table 1).
Enhancing clustered FL methods by improved methodologies. light of the HCFL, we have identified the remaining challenges within each tier that were previously overlooked by existing clustered FL., as illustrated in Figure 2 in Section 4.1. We then introduce HCFL+ an enhanced algorithm designed to tackle these remaining challenges. Numerical results confirm that HCFL* effectively extends existing methods, achieving superior balance between personalization and generalization while delivering strong performance. We summarize the contribution of this paper as follows:
We introduce HCFL, a holistic framework for clustered FL that encompass the existing methods. The HCFL represents feasible approach in achieving the integration of benefits from existing methods by adjusting the techniques at each tier.
This study focuses on data heterogeneity, While other techniques may enhance privacy and communication efficiency, they not directly related approach.
2

---

Published as a conference paper at ICLR 2025
We identify four remaining challenges within each tier of HCFL, and introduce improved algorithm called HCFL to address these challenges. Extensive experiments on different datasets (CIFARIO, CIFARI00, and Tiny- Imagenn) and various architectures (MobileNet-V2 and ResNet18) demonstrate the effectiveness of our framework and the improved components of HCFL*
RELATED WORKS
In the field of Federated Learning, FedAvg serves as the de-facto algorithm, employing local Stochastic Gradient Descent (local SGD) techniques (MeMahan et al., 2016; Lin et al., 2020) to reduce communication costs and protect client privacy. However, FL faces significant challenges due to distribution shifts among clients, which can hinder the performance FL algorithms (Li et al., 2018: Wang et al., 2020; Karimireddy et al., 2020; Jiang & Lin, 2023; Guo et al., 2021). To address these challenges, researchers have introduced clustered FL algorithms to enhance the performance of FL. algorithms.
Clustered FL groups clients based on their local data distribution, addressing the distribution shift problem. Most methods employ hard clustering with fixed number of clusters, grouping clients by measuring their similarities (Ghosh et al., 2020; Long et al., 2023; Wang et al., 2022b; Stallmann & Wilbik, 2022; Ma al., 2022). However, hard clustering may adequately capture complex relationships between local distributions, and soft clustering paradigms have been proposed to address this issue (Marfoq et al., 2021; Wu al., 2023: Ruan & Joe- be-Wong, 2022: Guo et al., 2023b; Reisser et al., 2021; Ruan & Joe- Noeg 2022). In this paper, we propose a generalized formulation for clustered FI. that encompasses current methods and improves them by addressing issues related to intra-client inconsistency and efficiency.
Another line of research focuses on automatically determining the number of clusters. Current methods utilize hierarchical clustering (Sattler et al., 2020b;a; Zhao et al., 2020; Briggs et al., 2020; Zeng et al., 2023; Duan al., 2021a;b), which measures client dissimilarity using model parameters or local gradient distances. Some papers enhance these distance metrics by employing various techniques, such as eigenvectors (Yan et al., 2023) and local feature norms (Wei & Huang, 2023). FEDCOLLAB (Bao et al., 2023) quantifies client similarity through client discriminators. However, the requirement for discriminators between every client pair in FEDCOLLAB hinders scalability for cross-device scenarios with numerous clients. In this paper, concentrate on cross-device settings, introducing a holistic adaptive clustering framework enabling cluster splitting and merging. We also present enhanced weight updating for soft clustering and finer distance metries for various clustering principles. For further discussions on related works, please refer to Appendix C.
3 HCFL: REVISITING AND EXTENDING CLUSTERED FL METHODS
Current clustered FL methods typically employ diverse learning frameworks. As a result, existing methods often face challenges in gathering the advantages of different algorithms for potential enhancements. To address this issue, as shown in Figure 1, we introduce the HCFL, consisting of four tiers designed to tackle the primary tasks of clustering methods: (1) Cluster Learning and Assignment (tiers and 2): Identify which clients should belong to the same clusters and learns parameters for each cluster. (2) Cluster Number Determinant (tiers and 4): Decide the number of clusters. As a result, the four tiers of the HCFL form a comprehensive learning process (Algorithm I), enabling flexible improvements and the integration of advantages from different algorithms (Table 1).
3.1 TIERS & 2: THE CLUSTER FORMULATIONS AND CLUSTER WEIGHTS CALCULATION
We introduce the first two tiers: Cluster Formulations and Cluster Weights Calculation. Cluster Formulations defines the objective functions of the clustering methods, aiming to learn the underlying distributions of each cluster. Cluster Weights Calculation orthogonally helps find the suitable clusters for each client, whereas hard clustering assigns each client to cluster. while soft clustering allows clients to contribute to multiple clusters. We propose the following optimization framework to encompass these two tiers.
3

---

Published as a conference paper at ICLR 2025
Optimization framework of clustered FL methods. The clustered FL methods can be expressed as dual-variable optimization problem that maximizes L(O, with K clusters and M data sources represented as DI,-- DM:
((), = E. EN log (Et (x Ue:
S.t. Σ Wis W.
(I)
where N - objective function. The parameters to be optimized are clustering weights N [w1:1, WM.K),
-i=1 N₁ N; == [D:], (x Miy) are sampled from D, and LE Lx( Miy; 0x)) the local
and model parameters + = ... [x]
Tier 1: Incorporate existing Cluster Formulations. The existing methods employ clustering to address diverse tasks, which results in the proposal of various formulations. Our Algorithm encompasses the existing clustering formulations by selecting C3 (xi,,, Bay; 0k) follows:
Po. (yo.a (x,.). Most existing methods (Marfoq et al., 2021; Ghosh et al., 2020; Long et al., 2023)
can be recovered using this conditional distribution (likelihood functions).
Po, (X.j) Vt.j). FedGMM (Wu et al., 2022) uses joint probability.
Por (o) (Bt,)) FedRC (Guo et al., 2023b) relies on correlations between variables x and y.
Po, (x.)))h.))
Tier 2: Incorporate existing Cluster Weights Calculation. Various methods employ distinet mechanisms for calculating clustering weights Wirk- The choice of Wirk, with either binary values witk E {0,1} or continuous values with E [0, 1J. characterizes the dynamic clustering procedure.
Hard clustering methods employ binary values Wi;k E 0, 1. these methods, wie is determined using heuristic techniques, such parameter distance (Long et al., 2023; Zeng et al., 2023; Sattler et al., 2020b), local loss function values (Ghosh et al., 2020) or label distributions (Diao et al., 2024).
Soft clustering approaches permit Wi:k E [o, 1], determined by maximizing ((O), ())MMaooq et al., 2021; Guo et al., 2023b; Wu et al., 2023), or by normalizing local loss values(Ruan & Joe- le-Wong, 2022). Soft clustering methods do not assume separated clients' local distributions and can thus handle complex scenarios, such as mixture distributions (Marfoq al., 2021; Wu et al., 2023).
3.2 TIERS 3 & 4: THE ADAPTIVE CLUSTERING PROCEDURE AND DISTANCE METRICS
Tiers 3 and 4 illustrate the techniques for Cluster Number Determination. In detail, the adaptive clustering procedures automatically adjust the number of clusters, while distance metrics control the adaptive clustering procedures, determining whether clusters should split or merge. The HCFL allows for different techniques at each tier, enhancing flexibility in choosing the optimal adaptive clustering methods or converting methods that rely on fixed cluster numbers to adaptive ones.
Tier 3: Adaptive clustering procedures demonstrate how to modify cluster numbers. automat- ically determine the number of clusters, current approaches can be categorized into two orthogonal methods: (1) Splitting clusters to increase the number of clusters (Sattler et al., 2020b;a). (2) Merging clusters to reduce the number of clusters (Zeng et al., 2023). We unify these approaches tier 3.
Tier 4: Client distance metrics dictate when cluster numbers should be adjusted. The client's distances are utilized to determine whether the current number of clusters should be adjusted. For instance, when the distance within cluster is large, the cluster will divide into sub-clusters. Conversely, if the distances between two clusters Existing clustering methods use various metrics such as cosine similarity of local gradients (Sattler al., 2020b), gradients from globally shared network (Zeng et al., 2023), and local feature norms (Wei & Huang, 2023).
small, these
clusters should be merged.
4 HCFL+: TACKLING REMAINING CHALLENGES IN CLUSTERED FL
Section introduces holistic clustering framework with four tiers to encompass existing methods. However, each tier still presents challenges that current methods cannot address. In this section,
II FedGMM (Wu al., 2023), 0₂ is split into [0 Mhul, and it uses LA (x Vi.3 0%) Pon, (Uis xi.)Pr,, (x).) i) to model the joint probability.
4

---

Published as a conference paper at ICLR 2025
Incomsistent Optimal Clustering Weights
Efficiency (Tier
Absence of Adaptiiv Clustering Soft Clustering ifler
CB
Require Methods
Cleen
CB weet
Kmodss
Grained Listance Metric
ASCP
Gradient Similerity
Tires Cowvsaelottee/orrrraatt
CSCP
Ambiguuus Reiationship
Parameter Simiuaity
Figure 2: Remaining challenges in clustered FL methods. We identify four key issues in clustered FL algorithms: (1) inconsistent intra-client clustering weights, (2) efficiency concerns, (3) the absence of adaptive clustering for soft clustering methods, and (4) the lack of fine- fine-grained distance metrics for various clustering principles. Clustering principles ASCP and CSCP differ in their approach follows: ASCP assigns clients with any shifts into diffcrent clusters, while CSCP only assigns clients with concept shifts different clusters."
outline four key remaining challenges in Figure and introduce HCFL tackle them. Due to space constraints, we summarize the improved algorithm in Algorithm 2.
While the HCFL+ employs standard FL training process without introducing extra privacy concerns compared to traditional methods. Existing privacy-precee preserving techniques, like secure aggregation, can be integrated plugins. We plan explore this possibility further the future work.
4.1 REMAINING CHALLENGES OF THE CLUSTERING IN FL
In this subsection, we identify four remaining challenges of the HCFL. We categorize these challenges by tiers in the HCFL, as shown in Figure 2. The details are provided below.
Challenges tier 1: Inconsistent intra-client clustering weights and efficiency concerns. These challenges can be addressed by improving the clustering formulations.
Inconsistent intra-client clustering weights. Existing approaches use the same clustering weights Wi:k for all the samples belonging to client i (Sattler et al., 2020b; Ghosh et al., 2020; Marfoq et al., 2021: Guo al., 2023b). However, they overlook cases where the optimal clustering weights of different samples within the same client can be inconsistent, implying that Wishh A Wiij for certain samples (Xijj++ Ji.ja) and (xx )t.jz). See our example here".
Efffcciecy.. The current clustered FI. methods (Marfoq al., 2021; Long et al., 2023; Guo et al., 2023b) require K-fold higher communication or computation costs, hindering overall algorithm efficiency during deployment.
Challenges on tiers 2 & 3: The absence of adaptive clustering for soft clustering methods. Cur- rent adaptive clustering methods primarily address hard clustering (Sattler et al., 2020b;a; Zeng et al., 2023). Hence, there exists a gap between research and practice, as there is a need to automatically determine the number of clusters for soft clustering methods (Marfoq al.. 2021: Guo al., 2023b).
Challenges on tier 4: Lack of fine-grained distance metrics for various clustering principles. The clustering principles determine which clients should be assigned the same clusters. Existing clustering methods may use different clustering principles, as described by ASCP and CSCP below:
ASCP (Any Shift Type Clustering Principle): clients with any distribution shifts are placed into separate clusters (Marfoq et al., 2021; Wu et al., 2023).
CSCP (Concept Shift Only Clustering Principle): only clients with concept shifts are assigned to separate clusters (Guo et al., 2023b).
As discussed in Section 3.2, client distances determine whether the current number of clusters should be changed, aligning with the role of clustering principles: if the current number of clusters cannot meet the requirements of the clustering principles, the cluster number should be adjusted. Consequently, we advocate for distance metries to be closely tied to distribution shifts, ultimately aligning with clustering principles. Unfortunately, existing distance metrics, such as those based on local gradients local model parameters (Sattler et al., 2020b; Zeng et al., 2023; Long al., 2023;
'In real- world scenarios, clients may have varying local data distributions, but they can still share certain data, like common knowledge location information. While client-specifi data may belong to different clusters, shared data should he in the clusters, leading inconsistent intra-client clustering weights.
5

---

Published as a conference paper at ICLR 2025
Algorithm 1 Algorithm Framework of HCFL.
Require: Number of communication rounds T' initial number clusters K initial parameters Φ. and 6
Ensure: Number of clusters K trained parameters ø², and O
I: for t= 0, T- do
Sample subset clients St, and send @**1 the clients.
3: for Client in S do
4:
Do local SGD by solving (I) for 7 local epochs..
b Tiers and
5:
Upload local model updates 1411 to the server. Si;k
6:
0%*
= 0 Eigs' 8isk
1+1 vk.
7:
Calculate distance matrix D', and Dj. Vk.
b Tier 4
8:
Detect cluster k. need to be split then Split cluster hs into sub-clusters St,1 and S.,2.
D Tier
9:
o
1/8 Exes,, Bisk 0 0 ns Dies., Bis
11:
12:
Update wis for corresponding clients.
13:
if Detect cluster ka need to be deleted then
D. Tier
Delete cluster ha-
Update Wos corresponding clients.
16:
Update k++1 by the current number of clusters.
Yan et al., 2023), cannot establish a clear link to distribution shifts. As result, current methods struggle to satisfy diverse and detailed clustering principles.
4.2 IMPROVE TIERI: INCONSISTENCY AND EFFICIENCY AWARE OBJECTIVE FUNCTIONS
To address tier challenges, specifically, (i) inconsistent intra-client clustering weights and (ii) efficiency, we propose an extension of the objective function (Eq. (1)), which defined as [(Ø, Φ 02,, 52) and includes the parameters Φ, O Q and 52.
Shared feature extractor Φ. and cluster-specific predictors θ 10,,... .O]] Dividing the feature extractor & and the predictors {0,} reduces communication and computation costs since the predictors lightweight architectures, like linear classifier layers.
Sample-wise clustering weights Ω = [W1,1;11 ... NMrkk for enhanced training stage and client-wise clustering weights Ω = [w11 ... @M.K] for testing stage 5. We employ sample-specific clustering weights (Wi,fsk) during training to ensure that data samples from the same clients can contribute to different cluster models, resolving the issue of inconsistent intra-client clustering weights. Furthermore, during testing, when test-time label information is unavailable. utilize client-specific weights (wi.k) for each client and cluster.
Enhance the optimization of wi,j; by regularizing the distance between Wa,k and Wi Motivated by the intuition that "if data from the same clients have similar distributions, the corresponding clustering weights should be similar", we encourage Wisk and Wijjk to be close to each other.
The following objective function is designed to meet our requirements.
C(6. e, 1 MM
log EK wigi La (x(x). Mks)
4 EM Σj
Ek=1
Wiih log
Aa
(2)
s.t. >K Wiitk = Vi,j. ΣK Wes = 1, vi, á = arg ming maxn C(d, -lo.
where Lx(x,,, Vky) is the simplified notation of L (x Blys Φ. 0x). Ai term is extended from (1)
by using the global shared feature extractor & and the sample-wise weights Wi.stt- A focuses on regularizing the difference between the sample-wise clustering weights Wi.jkh and the client- wise clustering weights Wj The y controls the strength of this regularization. We obtain Re by solving (3), where we aim minimize the impact of replacing Wi with Witk-
EEperiments the effectiveness of saiple-wise clustering weights in Figures 3a and 3h.

---

Published as a conference paper at ICLR 2025
Optimization of the proposed objective function. Different from heuristic methods used in most studies to optimize (2) 6, we aim to introduce a more interpretable approach by maximizing the objective functions (Eg.. (2)). Detailed proof refer to Appendix A. Specifically, we can update Wask- Wijj On. and Φ by (7).
Wiih Σ - ii C/ (x1.)121.5)
w/t (.6.))
2A-1 Miiik
0. f), (o..) ER who (x..)19))))
(4)
دز= j=1 78.33k
ItaN
Baak
+ ItaN
ON # =
jniii + (1 1)
(5)
0 0 Σ M Σ
f= ((.
N
Vo, (x(xi.
(6)
801 Ø² N Σ 1= DE= (( (xs,)
(7)
where C1 (xi.j. Wis) the simplified notation of local objective function Lx Lx(Xi., Majs 6', 0g), Niikk and Ji.j are intermediate results for calculating wi,jsk and Wis.. More detailed proofs can be found in Appendix A. j I+uN serves as a hyperparameter to control the strength of the penalty term in Equation (2). We provide the theoretical analysis on linear representation learning case in Appendix B.
4.3 IMPROVE TIERS 2 & 3: ADAPTIVE CLUSTERING FOR SOFT CLUSTERING PARADIGMS
Given the limitations of existing adaptive clustering methods, we have extended the clustering weight update mechanisms to incorporate soft clustering and have verified its effectiveness in Figures 3c and 3d. The overall process is summarized in Algorithms 4 and 5. In Algorithm 4, the clustering weights are adjusted after splitting cluster k into two sub-clusters, denoted by kı and k2. Then set Wi,j,k = Wi wi.j.&/ 2 for all and j. In Algorithm 5, the clustering weights are updated when removing cluster k. For all k + k, we modify Wa.j.k' as Wi.j.h' Enpe Wi.j
We use the hyperparameter p to control cluster splitting. As evidenced in Table 1, a higher results in fewer clusters, signifying enhanced generalization but reduced personalization. In detail, the cluster I will split if the following condition is met:
mDx) mean(Da) 2 p.
(8)
where D₁ is the distance matrix of cluster k. We identify the need for cluster removal when the cluster no longer receives the highest clustering weights from any clients. Additional details about the enhanced adaptive process can be found in Algorithm 2.
4.4 IMPROVE TIER4: FINE-GRAINEL DISTANCE METRIC DESIGN
Due to the page limitations, we include most of the details about the method design and practical implements in Appendix D. As discussed in Section 4.1, various algorithms may group clients into different clusters based on different clustering principles. Therefore, in this section, we design the fol- lowing fine-grained distance metrics for these different clustering principles. In detal, we have D
{ max (de, dir) ED, (2((z,)) w)] Eo, [ch(2.))] d.En, (((lz.)) w)] En, [c
ASCP:
(9)
CSCP,
where
d.=man max [dist (En: [P(2|x, y: 6)] En, [P(z)x, ,dyy= dist (En, [P(z]x; 6) Bo, (P(z)x: +))
where 2 (z. 3) is the simplified notation of local objective function E (z. y; 0&) given extracted feature z, dist is the cos-similarity. The distances above become large only when the following
'IFCA (Ghosh et al.. 2020) sets wigjsky.in = 1, Výj when li..inn = arg ming En, (x1,; Ways &. 0z)). where fix is the local loss function. FeSEM (Long et al., 2023) sets k = arg min, |10, 0 where Oz, O represents the model parameters of cluster & and client respectively.
7

---

Published as a conference paper at ICLR 2025
conditions oceur together: (1) Large values of de indicate concept shifts between clients i and (2) Large dis indicate significant feature and label distribution differences. (3) Large values of ED; [z, y; 0x)) ED; [x y: 0x) indicate incorrect clustering weights with high confidence. The effectiveness of the above distance metrics design is evidenced in Table 2.
Approximation of the distance metrics in practice. When calculating the distance metrics (Equation (9)) in practice, to avoid training extra generative networks and transmitting more data between servers and clients, substitute Wisk for L (z. y; 0x) since Wes is positively corre- lated with (z, y; 0%) (Marfoq et al., 2021; Guo et al., 2023b). Additionally, we approximate ED, [P([[P(1)
(2zzx, x. y/ Φ)] and ED, [P(z|x, y; Φ)] using feature prototypes. The prototypes are defined by
the following equation:
de Dist(P P.. Paj), dis Dis,
(10)
where
N; 1y 1 $),..
N y =C
Pas RR
(11)
Ni,1
Nic
j=1
j=1
Pifa R
g( Φ),
(12)
N
Ni.- = Ef = g( Φ) is the function parameterized by Φ, Dist is function measure the distance between prototypes, which we use the cosine similarity as an example in this paper.
5 NUMERICAL RESULTS
In this section, we evaluate the performance of HCFL+ and other clustered FL methods. Additional experiment results, including hyper-parameter ablation studies, different model architectures, additional scenarios, efficiency comparision, and clusteirng quality illustration can be found in Appendix E.
5.1 DATASETS AND EXPERIMENT SETTINGS
Diverse distribution shifts scenarios. We establish clients with three types of distribution shifts. For label distribution shifts, we employ LDA with a = 1.0, as introduced by (Yoshida et al., 2019; Hsu al., 2019; Reddi et al., 2021). For feature distribution shifts, we adopt the methodology from CIFAR1O-C and CIFAR100-CC creation (Hendrycks & Dietterich, 2019). Regarding concept shift, draw inspiration from (Guo et al., 2023b; Jothimurugesan
al., 2023), and selectively swap labels based on the parameter 3. For example, with B = 0.1 for CIFAR10 10, two labels per concept are swapped, while the remaining eight labels remain unchanged. By default, create three concepts in the experiments. More details about the construction of scenarios are included in Appendix E.
Baselines. We use FedAvg (McMahan et al., 2016) as single-model FL example. We consider the most recently published clustered FL methods as our baselines. For clustered FL with fixed cluster number, we select IFCA (Ghosh et al., 2020), FedEM (Marfoq al., 2021), FeSEM (Long et al., 2023), and FedRC (Guo et al., 2023b). For the adaptive clustering FL methods, we choose CFL (Sattler al., 2020b), ICFL (Yan et al., 2023), and StoCFL (Zeng al., 2023). For the HCFL* variants, we enhance backbones that necessitate a predetermined number of clusters (FeSEM, FedEM, and FedRC) by integrating them with the adaptive soft clustering method and the feature split mechanism of HCFL+ The HCFL + (FedSoft) is devised by substituting the local training component of HCFL + (FedEM) with proximal regularized local updates as outlined in FedSoft (Ruan& & Joe- Dong 2022). Further elaboration can be found in Appendix E.2.
Experiment settings. Unless specifically mentioned, we divide the datasets into 100 clients and exe- cute all algorithms for 200 communication rounds. Additional settings provided in Appendix E.2. We conducted all experiments using MobileNet- V2 (Sandler et al., 2018) and results on ResNet18 defer to Table 8 of Appendix E.
8

---

Published as a conference paper at ICLR 2025
Table 1: Performance of the adaptive clustering methods CIFARIO. CIFARI00, and Iny-Imagenet datasets. For each algorithm, we present the best Validation and Test accuracies averaged over three trials. For clustering methods that require a fixed number of clusters, we set K = 3. The hyperparameters tol,, tolg, a (0), and in adaptive clustering methods govern the balance between personalization and generalization, as well as the cluster number. For instance, lower in StoCFL or lower p in HCFL* indicate improved personalization and reduced gencralization. K" denotes the cluster number in the final training round, where larger K" suggests improved personalization reduced generalization and communication efficiency. We emphasize the best results in bold and the worst results in blue.
Algorithm
CIFARIO,
CO
Tiny Imagenett
Test
KT
Test
KT
Test
KT
Pro-detined K
FedAvg
49.10
=2.15
45.42
3:2-42
26.00 01 11.15
27.87
3:2:12
38.63
200.20
39.07
1:01:4
FeSEM
45.30
20.00
29:01
1:
26.37
10.06
24.50
1:0/28
37.10
30.80
90.00
1:244M
IFCA
34.40 46
25.18
4:2.55
26.90
25.20
4:1.66
38.52
29.92
1:0.30
FodEM
53.64
+:0.05
42.00
FoaRRC
63.63
=2.05
50.41
#60.19
34.56
30.79
37. 6/
#:0.00
38.53
20.18
29.73
1:0.04
Adaptive K
CFL
toll
0.4, tol
1.0
61.55
11.74
$6.88
3:0.10
35.05 140,35
24.84
d:2
37:41
4:1.87
30.25
3:0.35
tal:
0.4,
tnlo
0.8
65.06
=3.51
45.74
1-4 0..0
36.96
+:3.3
22.00
+:1 888
40.36 +:3.55
28.82
+:0.71
tol.
0.2,
lol
0.8
58.92
2:2000
55,02
37.73 73
8.7
51.47
3:00
35.74
3.0.57
$6.41
ICIF.
(0) 0.85
77.50
20.04
57.38
4:101
64.72
+:0,30
1:0.30
(0) 0.58
00.58
=1.07
61.18
60.78
41.49
9-1.11
33.57
d:1
$3.00
#3.67
35.00
#:0.25
StoCHL
0.06
50.70
:1.51
57:55
+0092
15
07
+0.07
$1.40
1:2.1
31.85
+0.00
51.59
+:0.8
7 0:10
70.84
21.08
51.72
3:0.07
56
69.76
1:2.07
9.42 1:0.07
89
67.48
4:1.53
13.03
3:0.67
91
ICCFL (PeSEM)
87.77
+:-11
26
14:34
60.44
#0.86
85.08 =0.11
43.34
460.94
02
16.87
4:2.97
#2.00
22.77
4:1.27
== 0.3
79.31 -3.95
47.62
1:2.00
44.49
11.57
25.03
1:0.5
45.76
10.09
36.05
1:1:35
HCFL (FedEM)
0.05
62:45 21.33
57.73
3:1.70
22
60:36 11.47
22.05
1:1
40
63:41
54.24
1:01/333
33
0.1
84.64 11:47
60.90
1:0.01
62.88 98
10.02
25.17
1:1
34
30.68
10011
57.17
1:0:37
20
0.3
83.67
62.43
3:0.71
30.72
52.13
1:0.18
45.53
38.64
3:0.23
IICFL- (FedRC)
p-
09.16 =0.65
077.37
d60.00
20.88
4:0.64
43.77
+:0.81
4:0.34
P= 0.1
71.67
-0.83
68.64
16
56
82
44.26
110 10
38.82
1:0.77
0.3
69:33
-0.21
69:67
1:1227
30.97
+0.21
96.50
=:0.8
42.00
#0.21
40.65
1:0 0..3
Evaluation metrics. We present the following metrics to evaluate the personalization and gener- alization abilities of the algorithms: (1) Validation Accuracy for evaluating personalization: The average accuracy on local validation datasets that match the distribution of local training sets. (2)
Test Accuracy evaluating generalization: The average accuracy on global shared test datasets.
5.2 RESULTS ON DIVERSE DISTRIBUTION SHIFTS SCENARIOS
In this section, we compare the performance of HCFL+ with other clustered FL methods. We also perform ablation studies to confirm the effectiveness of HCFL+'s proposed components.
HCFL+ achieves comparable performance and better aesionaaizatiiongennraiiiinn
trade-offs.
We highlight some key observations in Table 1. A. HCFL* consistently achieves superior test accuracy, with validation accuracy surpassing that of baseline methods with a similar number of clusters. This demonstrates improved efficiency and better balance between personalization and generalization. B. Soft clustering methods like HCFL+ + (FedEM) and HCFL+ (FedRC) outperform hard clustering methods in test accuracy, showcasing their superior generalization capabilities C. While baseline methods may achieve higher validation accuracy by separating every client into different clusters (namely when the value of KT is close 100), these trained clusters tend to overfit local distributions, resulting in significantly lower test accuracy. As detailed in Table 9, ICFL exhibits lower test accuracy when achieving the best personalization, while its personalization cannot outperform HCFL+ when achieving its best test performance. D. The extended algorithms, namely HCFL. (FeSEM), HCFL+ (FedEM), and HCFI. (FedRC), outperform the original methods that rely fixed cluster numbers significantly. Additionally, these extended algorithms can automatically adjust the number of clusters, making the algorithms more practical, as we illustrated in Figure 4. E. As shown in Table 3, it is possible to enhance the personalized performance of HCFL* + by employing a smaller p. However, this comes at the expense of reduced generalization performance.
Ablation studies on See 4.2. We perform ablation studies on jú, which control the distance between sample-wise weights wijjs and client-wise weights Tis in Figures 3a and 3b. A larger ji signifies a greater difference between ji.j and Wig.- Our results show that HCFI. (FedEM) prefers smaller

---

Published as a conference paper at ICLR 2025
Table 2: Ablation studies Sec 4.4, We conducted experiments on the CIFARIO and CIFARIO0 datasets, showcasing the highest test accuracies, the maximum number of clusters during training (max, K'), ). and the final number of clusters (K" ) or each algorithm while maintaining a fixed value of = 0.3. We used FedRC as the backbone, with 3 clusters identified as ideal by CSCP, ① Mcasuring client distance by gradient similarity; Excluding En, [Ax (z, y: 0s)] and ED, [Cx y; 0x)] from Eq (9); @ Averaging de over label y instead of maximizing it over y-
Algorithm
CIPARIO,
CIFARIO, B
CIFARIOO,
CIPAR100, 0.4
Test Ace
maxy K
X7
Test Acc
ILAN;
R²
Test Are
E3nX, X
K²
Test Acc
max, K'
HCEL"
69.67
1:1:27
4.5
3.0
70 13
:0000
7.0
6.0
36 50
19:25
3.5
3.5
32.22
1:0.0
5.0
4.0
-0
07.83
4:170
9.5
7.0
64.53
10.5
10.0
36.77
9.5
8.5
31.33
1:2.12
11.0
7.5
12.5
10.0
32.75
1:067
08.62
60:04
8.0
6.0
69:47
20015
13.0
8.5
34.65
2::16
8.5
31.61
11:54
11.0
7.5
68.82
+0.50
5.5
3.5
65.74
+0.00
7.0
7.0
35.97
+0.80
4.0
3.5
31.72
+0.59
4.5
4.0
USB
RCA
050 CPWEUCC 2C1 CRAN p41
p48 OPRReCA DANG yd p4
0MRC
PARMANX
(a) HCFL* (FeJEM)
(b) HCFL* (FedRC)
Validation
(d) Test
Figure 3: Ablation studies on Sections 4.2 and 4.3. For Sec 4.2, we evaluated lest accuracies of HCFL* using different backbones (FedEM and FedRC) and varying values of jt, as shown in Figures 3a and 3b. For Sec 4.3, we present the best Val and Test accuracy achieved by HCFL* with either FedEM or FedRC as backbones. "w/ SCWU" indicates the use of soft clustering weight updating mechanisms introduced in Section 4.3. More detailed results can be found Tables and in Appendix E.
Table 3: Performance of HCFL with small p. We set the p 0.01 for HCFL* and use FedEM the backbone algorithm.
Algorithms
CIFARIO0 Val
CIFAR100 Test
Tiny ImageNNet Val
Tiny ImageNNt Test
Baselines (best)
69.76
37.62
67.48
39.73
HCFI (p 0.01)
78
14.33
77.56
21.53
HCFL+ (best in Table 1)
69.25
36.50
63.41
40.65
distance between Wi.j and Wisk- However, HCFL+ (FedRC) prefers larger ju values, highlighting the necessity of different clustering weights among samples within the same clients.
Ablation studies on Sec 4.3. In Figures 3c and 3d, we perform ablation studies on the soft clustering weight updating mechanism (w/ SCWU) introduced in Section 4.3, The term w/o SCWU refers to using the traditional clustering weight updating mechanism as described (Sattler et al., 2020b; Zeng et al., 2023). The results demonstrate that our proposed SCWU consistently achieves better performance in terms of both validation and test accuracies.
Ablation studies on techniques in Sec 4.4. We perform ablation studies to demonstrate the ef- fectiveness of the designed distance metrics in Sec 4.4. The ablation studies include: C Using gradient similarity, as in previous works (Sattler al., 2020b; Yan al., 2023), instead of distance on P(z|x; Φ) and P(z ( yr Φ), as we proposed in Equation 9; @ Remove ED, [Cx (z, y; 0g)] and ID, (z. y; 0x]] in (9); Using mean distances instead of maximum distances (9). The results show that HCFL* consistently achieves the highest test accuracy and produces a number of clusters closer to the ideal number than other ablation studies.
CONCLUSION AND LIMITATIONS
In this paper, we introduce HCFL, a comprehensive clustered FL framework that unifies existing methods while enabling the integration of diverse algorithms to gather the advantages various clustered FL approaches. Additionally, we identify persistent challenges unaddressed by current algorithms and propose HCFL as a solution. The HCFL is flexible and can generate numerous clustered FL methods by altering techniques in each tier. Though we have chosen some typical components and demonstrated their effectiveness, conducting further performance verification with more choices in each tier would be beneficial.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported in part by the funding from Shenzhen Institute of Artificial Intelligence and Robotics for Society, in part by the Shenzhen Key Lab of Crowd Intelligence Empowered Low-Carbon Energy Network (Grant No. ZDSYS202206061 1000000000000000200 100601002). in part by Shenzhen Sta- bility Science Program 2023, and in part by the Guangdong Provincial Key Laboratory of Future Networks of Intelligence (Grant No. 2022B1212010001, This work is also supported in part by the Research Center for Industries of the Future (RCIF) Westlake University, and Westlake Education Foundation.
REFERENCES
Wenxuan Bao, Haohan Wang, Jun Wu, and Jingrui He. Optimizing the collaboration structure in cross-silo federated learning. 2023.
Christopher Briggs, Zhong Fan, and Peter Andras. Federated learning with hierarchical clustering of local updates to improve training on non-iid data. In 2020 International Joint Conference on Neural Networks (IJCNN,, pp. 1-9. IEEE, 2020.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared represen- tations for personalized federated learning. In International Conference on Machine Learning, pp-
PMLR, 2021.
Yiqun Diao, Qinbin Li, and Bingsheng He. Exploiting label skews in federated learning with model concatenation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 11784-11792, 2024.
Shifei Ding. Chao Li, Xiao Xu, Lili Guo, Ling Ding, and Xindong Wu. Horizontal federated density peaks clustering. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Moming Duan, Duo Liu, Xinyuan Ji, Renping Liu, Liang Liang, Xianzhang Chen, and Yujuan Tan. FedGroup: Efficient federated learning via decomposed similarity-based clustering. In 2021 IEEE Inil Conf on Purallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing Communications, Social Computing & Networking CAAABDDCCoou/SSccalCC///ttttttt
pp. 228 237. IEEE, 2021a.
Moming Duan, Duo Liu, Xinyuan Ji, Yu Wu, Liang Liang, Xianzhang Chen, Yujuan Tan, and Ao Ren. Flexible clustered federated learning for client-level data distribution shift. IEEE Transactions on Parallel and Distributed Systems, 33(11):2661-20) 2021b.
Xiuwen Fang and Mang Ye. Robust federated learning with noisy and heterogeneous clients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10072-10081, 2022.
Shaoduo Gan, Akhil Mathur, Anton Isopoussu, Fahim Kawsar, Nadia Berthouze, and Nicholas Lane. FRuDA: Framework for distributed adversarial domain adaptation. IEEE Transactions on Parallel and Distributed Systems, 2021.
Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 33:1 13:19586- 19597, 2020.
Yongxin Guo, Tao Lin, and Xiaoying Tang. Towards federated learning time-evolving heteroge- neous data. arXiv preprint arXiv:21 /2. 13246, 2021.
Yongxin Guo, Xiaoying Tang, and Tao Lin. FedBR: Improving federated learning on heterogeneous data via local learning bias reduction. 2023a.
Yongxin Guo, Xiaoying Tang, and Tao Lin. FedRC: Tackling diverse distribution shifts challenge in federated learning by robust clustering. arXiv preprint arXiv:2301. 12379, 2023b.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: /22611 2019.
11

---

Published as a conference paper at ICLR 2025
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non- con-identical data distribution for federated visual classification. arXiv preprint (rXXi:1909.063300 2019.
Prateek Jain, Praneeth Netrapalli, and Sujay Sanghavi. Low-rank matrix completion using alternating minimization. In Proceedings of the forty-fifth annual ACM symposium on Theory of computing. pp. 665-674, 2013.
Liangze Jiang and Tao Lin. Test-time robust personalization for federated learning. In International Conference on Learning Representations, 2023.
Ellango Jothimurugesan, Kevin Hsieh, Jianyu Wang, Gauri Joshi, and Phillip B Gibbons. Federated learning under distributed concept drift. pp. 5834-583,, 2023.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U. Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. 2019. URL https: /ttp:v//aaivvvv.. wttp://ab//....00077221
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. SCAFFOLD: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pp. 5132-5143. PMLR, 2020.
Shuqi Ke, Chao Huang, and Xin Liu. Quantifying the impact of label noise on federated learning. arXiv preprint arXiv:2211 1.07816, 2022.
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. arXiv preprint arXiv: arXiv:1812.06127, 2018.
Tao Lin, Sebastian U. Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't use large mini-batches, use local sgd. In International Conference on Learning Representations, 2020. URL https:
//openreview net. /ere/fformmtt forum? het.mmmdddd1eee1FFPp..
Guodong Long, Ming Xie, Tao Shen, Tianyi Zhou, Xianzi Wang, and Jing Jiang. Multi-center federated learning: clients clustering for better personalization. World Wide Web, 26(1):481-500 2023.
Jie Ma, Guodong Long, Tianyi Zhou, Jing Jiang, and Chengqi Zhang. On the convergence of clustered federated learning. arXiv preprint vrXiv:2202.06187 2022.
Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Laetitia Kameni, and Richard Vidal. Federated multi-task learning under a mixture of distributions. Advances in Neural Information Processing Systems,
2021.
H. Brendan McMaaan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera Arcas. Communication-efficeenn learning of deep networks from decentralized data, 2016. doi: 10.48550/ ARXIV.1602.05629 URL httpp::/aee /awxivvvvvvv www.eiiiiooooooooooooooooooooo.r aabs/////// / 602. 05629.
Xingchao Peng, Zijun Huang, Yizhe Zhu, and Kate Saenko. Federated adversarial domain adaptation, 2019. URL httph:thp//.cmcc....c wawai.iiii...//aa org/ aass/aa///// / 911.9211200 02054.
Dong Qiao, Chris Ding, and Jicong Fan. Federated spectral clustering via secure similarity recon- struction. Advances in Neural Information Processing Systems, 36, 2024.
Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference on Learning Representations, 2021. URL https: /wtp///ppneeeeeeeeeeee/ee net forum? id=I.kFG11B 305.
Matthias Reisser, Christos Louizos, Efstratios Gavves, and Max Welling. Federated mixture of experts. arXiv preprint 2r.Xiv22107.000 2021.
12

---

Published as a conference paper at ICLR 2025
Yichen Ruan and Carlee Joe- Wong. FedSoft: Soft clustered federated learning with proximal local updating. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. $124-8131, 2022.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bileNerV2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Felix Sattler, Klaus-Rober Müller, and Wojeiech Samek. Clustered federated learning: Model- agnostic distributed multitask optimization under privacy constraints. IEEE transactions on neural networks and learning systems, 3273):371--3722, 10-3722, 2020a.
Felix Sattler, Klaus-Rober Müller, Thomas Wiegand, and Wojeiech Samek. On the byzantine robustness of clustered federated learning. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8861-8865. IEEE, 2020b.
Yan Shen, Jian Du, Han Zhao, Benyu Zhang, Zhanghexuan Ji, and Mingchen Gao. FedMM: Saddle point optimization for federated adversarial domain adaptation. arXiv preprint arXiv:2110.08477, 2021.
Morris Stallmann and Anna Wilbik. Towards federated clustering: A federated fuzzy c-means algorithm (ffem). arXiv preprint arXiv:2201.07310 2022.
Yuwei Sun, Ng Chong, and Ochiai Hideya. Multi-source domain adaptation based on federated knowledge alignment. arXiv preprint arXiv:2203. 2022.
Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xin He, Bo Han, and Xiaowen Chu. Virtual homogeneity learning: Defending against data heterogeneity in federated learning. International Conference on Machine Learning, pp. 21111-21132, 2022.
Isidoros Tziotis, Zebang Shen, Ramtin Pedarsani, Hamed Hassani, and Aryan Mokhtari. Straggler- resilient personalized federated learning. arXiv preprint (xXiv:2206.0077: 2022.
Saeed Vahidian, Mahdi Morafah, Weijia Wang. Vyacheslav Kungurtsev, Chen Chen, Mubarak Shah, and Bill Lin. Efficient distribution similarity identification in clustered federated learning via principal angles between client data subspaces. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 10043-10052, 2023.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47, Cambridge university press, 2018
Bin Wang, Gang Li, Chao Wu, WeiShan Zhang, Jiehan Zhou, and Ye Wei. A framework for self- supervised federated domain adaptation. EURASIP Journal on Wireless Communications and Networking, 2022(I):1-17,, 2022a.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. arXiv preprint arXiv:2002.0644 2020.
Zhiyuan Wang, Hongli Xu, Jianchun Liu, Yang Xu, He Huang, and Yangming Zhao. Accelerating federated learning with cluster construction and hierarchical aggregation. IEEE Transactions on Mobile Computing, 2022b.
Xiao-Xiang Wei and Hua Huang. Edge devices clustering for federated visual classification: A feature norm based framework. IEEE Transactions on Image Processing, 32:995-1010, 2023.
Shanshan Wu, Tian Li, Zachary Charles, Yu Xiao, Ziyu Liu, Zheng Xu, and Virginia Smith. Mot- ley: Benchmarking heterogeneity and personalization in federated learning. arXiv preprint arXiv:2206.09266 2022.
Yue Wu, Shuaicheng Zhang, Wenchao Yu, Yanchi Liu, Quanquan Gu, Dawei Zhou, Haifeng Chen, and Wei Cheng. Personalized federated learning under mixture of distributions. 2023.
Yihan Yan, Xiaojun Tong, and Shen Wang. Clustered federated learning in heterogeneous environ- ment. IEEE Transactions Neural Networks and Learning Systems, 2023.
13

---

Published as a conference paper at ICLR 2025
Naoya Yoshida, Takayuki Nishio, Masahiro Morikura, Koji Yamamoto, and Ryo Yonetani. Hybrid- FL: Cooperative learning mechanism using non-iid data in wireless networks. arXiv preprint arXiv:1905.07210, 2019.
Dun Zeng, Xiangjing Hu, Shiyu Liu, Yue Yu, Qifan Wang, and Zenglin Xu. Stochastic clustered federated learning. arXiv preprint arXiv:2303 2088007700000897, 00897, 2023.
Fengpan Zhao, Yan Huang, Akshita Maradapu Vera Venkata Sai, and Yubao Wu. A cluster-based solution to achieve fairness in federated learning. 2020 IEEE Indl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communi- cations, Social Computing & Networking SPPA/BDCooudSSoccalCCo/SSssaaaa IEEE, 2020.
pp. 875-882.
14

---

Published as a conference paper at ICLR 2025
CONTENTS OF APPENDIX
A Proof of Optimization Steps
15
Theoretical Study on Lincar Representation Case
17
B.1
Definition and Assumptions
17
B.2 Preliminary and Lemmas
18
B.3 Main Results
27
C Related Works
36
D Algorithms
37
E Experiment Results
41
E.1
Datasets and Models
E.2
Baselines and Syper-Parammter Settings
41
E.3 Additional Experiment Results
42
Below are the key takeaways of the appendix:
Appendix A: We provide a proof showing how (4)-7) are derived, demonstrating how they solve the EM-like objective function (2) (3) in practice.
We analyze the HCFL framework in the context of linear representation learning, without assuming an equal number of samples across clusters. Our findings indicate that an imbalance in sample sizes and higher level of drift can slow down convergence.
Appendix B:
Appendix C: A more detailed discussion of related works.
Appendix D: We explain the rationale behind the design of the distance metric and its practical implementation. Specifically, for each pair clients, we compute the distance between their local class prototypes for each class, well as the distance between their feature means. The maximum of these two distances then selected as the final distance metric for the client pair.
Appendix D: A detailed version of the HCFL algorithm is provided.
Appendix E: report the experimental settings used in our study in detail.
Appendix E: Ablation studies on the impact of hyperparameters and algorithmic components show that HCFL- is robust to variations in hyperparameters, and that all proposed components provide individual performance gains.
A PROOF OF OPTIMIZATION STEPS
Theorem A.1. Given objective function L(Φ, 02)
M
N
K
0,6,0,0000 = N
log
Σ
wiij La (x Miyi Φ. 0k)
i=1
j=1
Ai.5
Wi.goh
Wick We.ssh
Wiik log
(13)
and we define 52 (W /Vi, k). then ái is obtained by
arg min
max L(Φ 0, -...
(14)
f
12
15

---

Published as a conference paper at ICLR 2025
Then E-M steps are obtained by maximizing [( O S2).
whitk (X Miji 0%
1+1
(15)
>m=1 w/ Lu (Xi,) Miji 0%
Wf C.k Cx(Xi,)) Miji 0%)
7-+1
(16)
En 1 wiin Lx (x Miys 0%
1 No
Wakk
a1+1
21+1
(17)
N
++1
AN
1-11
1t1 lik
(18)
Sit
1 + WN
+ WN
M
N
0% 0% 17
Mijj
Ve, Vo Cx Viy. 8., 0%)
(19)
(x Yis. 0%
M
N.
Φ 1+1 0¹
17
Cx Piy. Φ¹ 0%)
VoLu(XIS, Beg, Yig, 8., 0%
(20)
Proof.
Let's begin by assuming / is given for each round t, then we will discuss how to compute Consider the objective function [(o,, S,5 52), we have
w Lijsk
1+1
AL(Φ, 02 Owi,fik
1 N
Lx(xi Diys On) K
+ Ao. + + Wish
(21)
Ex= Wi.j:n Ln (X Biyi Q.)
Wiijk
Then define
Wh.ji Lx Xz.jj Vsyi o,,) Σx= Winj:n Ln (x Boj; Q.))
Biskk
K
(22)
and take 0 Sis = 0 we have
Wiik N
puds; = - -i.
(23)
Then we have
1
Phisik
Wiijk
+ uwissh)
(24)
Xi.
N
Because we have Ek=1 With 1, we have
1
+ #1
(25)
Xi.
N
1 + WN
A
(26)
N
Then we have
UN
Wiijk
1 + WN
1 IN
(27)
Then consider to optimize os. we have,
ac(Ø
9,22
00,
M
N
Wiijk
0L (Xij, Biyi 0x)x)
(28)
N
Σx
1
wi Winj (Xij, Vijs Φ 0.)
00
M
N
I+1
Pisk
Vo, Lx(xi,,,55 Φ 0k)
(29)
N
La (Xij. Vijs Φ 0z)
16

---

Published as a conference paper at ICLR 2025
Finally, consider to optimize Φ, we have
aL(Ø, OC 22 S2)
dΦ M
N
Wijih
0Lx (Xij. Miji Φ, 0k)
(30)
2n Wi Ln (Xix. Vigi 0,)
K
aΦ
V V Wiji 0n).
(31)
(xij, Miyi 0,
Because if hard to find a close-form solution to deep neural networks, we use gradient ascent to optimize 0 extractors Φ
e.9.2) = 0 when 0₁ is the parameter of The same method is used for feature
30,
Then the remaining thing is how to decide Wakk. From the formulation of objective function, the Wys decided by
S arg max [(Ø, O. -..
min min
(32)
f2
To solve this problem, firstly, we can transform the definition of L(Φ, 0, to
M
N.
L( C(0,0,2,,00
log
Σ wiss Lx (xi,r Muy Φ. 0s)
1=1
j=1
X
Whisik
M
N.
Wick Wiish
Wiih log
(33)
s.t. wijk Widk: ji,
(34)
Then by removing the constrains, we can always find
max L(Φ, θ 02 > 0,
(35)
Then we have
2 arg min á
max L(Φ,, 0, C(Ø, θ Ω.
(36)
ft
arg min
max L(Φ Q0 Ω L( 0,0,0
(37)
á
arg max L(6, Ω.
(38)
à
Then can obtain the results by directly use the proof in (Guo et al., 2023b) and (Marfoq et al., 2021).
B THEORETICAL STUDY ON LINEAR REPRESENTATION CASE
B. DEFINITION AND ASSUMPTIONS
In this section, we target the effectiveness of split-feature-clasiffer method. Therefore, focus on a case study that clients are solving a linear representation learning problem, similar to the analysis in (Collins et al., 2021; Tziotis et al., 2022), and assume the optimal wi is already given. In detail, assume local data X.j Rd, and & global shared projection onto c-dimensional subspace Rd which is parameterized by matrix B E dxd Besides, for each underlying distribution,
17

---

Published as a conference paper at ICLR 2025
have 0x R", and the labels of each sample X,,j belongs to the distribution k is given by Ri. 0 B æıj + 24, where 0% and B are ground truth parameters, and z4 _N(0,, o²) is to capture the heterogencous between K underlying distributions. Under these assumptions, the global empirical risk is
M
N2
K Wi 0 B
2
min B,0 2N
Vi.j
X
(39)
1=
k=1
The distance that measuring the distance between sub-spaces defined as follows (Jain et al., 2013; Collins et al., 2021; Tziotis al., 2022), Definition B.1. The principal angle distance between the column spaces of B1, B2 E RR dxxe is given by,
dist(B = BT B2
(40)
where B1,1 and B2 orthogonal matrices satisfying span(Bi.) span( (B,) ) and span (B2) = (22))(B2..
Definition B.2 (I]A Sub-Gaussian). For random vector E Rd and fixed matrix A E paxe, Gaussian y A is sub- Sabb-Gassiin with sub- Gaussian norm the vector ATx is called ||Allz-sutt -ubb--uuu-an
A 2 2 all y E RR, iee,, [eexply" ATx)] S exp(II |yy /2).
Assumption 1 (Sub-Gaussian design). The samples X.,j E Rd are i.idd. with mean 0, covariance La. and are La-sub-Gaussian ie., TE (ev"x..) X.. <ellv
(.llv1. 1:/2 for all v E Rd.
Assumption 2 (Underlying distribution diversity). Let min. be the minimum singular value of any matrix W E R &xx with rows being a K K-sized subset of ground-truth perribution-speciify parameters (0....., ,0x}; }. Then min,, > O.
Assumption 3 (Client normalization.. The ground-trute distribution parameters satisfy |0 for all k E [K], and B* has orthogonal columns.
Assumption (Binary weights). We assume the value of Wi is given in {0, 13.
Assumption 3 are widely used assumptions in the analysis of linear representation learning problem (Collins et al., 2021; Tziotis et al., 2022). We introduce Assumption 4 to simplify the analysis, and this assumption match the observation our numerical experiments that the max weights max [s]. usually close to 1 when converge.
B.2 PRELIMINARY AND LEMMAS
We first introduce some important lemmas that we would like to use in the following parts of proof.
Lemma B.3. Given orthogonal matrix B E Raxe, matrix X E RRNxd that each row of X is sub-
Gaussian (Assumption 1), and matrix Si e R xd valued by {0, 1}. Then let dg 120 Alg(Ad) we have
v NNx
1 B"
Ñ
Omin
(N
X
2 22,) B
2
(1 - 54)
(41)
N
N
with probability at least 1 exp ()), and Ñr. = EM Ennn 1m
Proof. Firstly, can rewrite
1 BT ( o X
X
B
Weifk BBT Xi.
Wrisk BT Xi.
BTxi.
(42)
N
N
VNz
(Nnx
N
T
Σ
BT Bixi Xi.i
1
BTT
xi.j
(43)
N
NNk
VN
Wi WiJ =1
18

---

Published as a conference paper at ICLR 2025
where N₂ Em Dj=1 N. 1.. Define Vi.j:
W
X..j B'x,, by Definition B.2 and Assump-
V No
tion I, we can observe that each Vijik is either
B
Bub-Gaussian or Vidik = 0. Then we
Nx
can define V E I. Nexe, and each row of V is Vi.isk that Vi.ji # O. Then we have
NA V VK.
B (2 C x")
X 2 S2
B
(44)
Then based Theorem 4.6.1, Equation (4.22) in (Vershynin, 2018), have
L BT
Nu
O Ominn
2 x²)
④ 24x B
=
N
mmin (V. 2 N (1
5%)
(45)
with probability at least - exp(- (OAV/N//C: - -VV)) for constant Cl.k- We then set
12C 3/2 (lgg(M) and we have
5k-
N2
1 1 exp(- (ouVNN/Cuue-V (1 (. 211==x((221c"
(46)
which finish the proof.
Lemma B.4. Given the objective function defined in Equation (39)
M
N;
min B
Vi.j
wi Wi O B X
(47)
2N
define the matrix form of Xt,. Vag. and Waithe by e R Y Y Sr RN and define Sh E xxd by repeat Sh for d times, can have the following optimization steps solve the above objective function
0%+
1 N
(S) O XX'
(X St.) (O) B¹)
(H (S) o xx)) (Y
(xoxx')( (Y
(48)
B B'
n
( x²) (x 12x) Bo (22 o ox)) (Y 2 0
(49)
N
B++1 B (R*
(50)
where B R' is the QR factorization of B
Proof. We first extend the empirical function via matrices X; E Nidd Y, E RN, and S2 RR: Diy E Nid is repeat Qitt R.N, for d times.
M
K Σ
C(B 0) 2N
Σ
-Y,
(51)
2 M IT. and Sr RRN
Define X R.d X
X Sr E Nxd
ST [
[S2 M. then compute the gradients of B and O..
ØL(B, θ)
M
K
Shik C X.)" (Res o (Y, - x,BO,)) 0f.
(52)
aB
1
( O X") (X 22x BO, - (Ω の x²) (Y ④ lR) 0%.
(53)
N
k=1
AL(B, O)
1 M
Disk C x,,) T (n o (X,-XX,B))
(54)
00,
1
BT
(U) @ x²) (X 22)) BO - -B" (n o x²) (Y o 2x))
(55)
N
19

---

Published as a conference paper at ICLR 2025
Then we can define the following optimization steps based on the above gradients, similar to the analysis in (Collins et al., 2021).
1 N
f2,) B5) ) ( (M) @ x²) (Y o (n)))
k =
B
x')
(56)
B = B'
( x ( 22 B0, (0) ox" (Y の (olx)) 0%
(57)
N
B = B RR (R!+))-
(58)
where B of each optimization step. From Lemma B.3 we know that (1 [B'7 (?7 xT) o Qc) B¹) is
++1 is the QR factorization of B Then the B will be orthogonal matrix at the end
invertible with high probability, which indicates the feasibility of the above optimization steps.
O
Lemma B.5, The optimization step of 0₂ can be expressed by the the following equation
02 = O + Fx + Ge.
(59)
where Fk: G3 E IR are given in Equation (65) and (66).
Proof. From Lemma B.4 we have
-1 02 1 BT ) (X S2x) (x) B)
(11,
(1
B" (n + x²) (Y f2)))
(60)
N
Based on the fact that Y >k= S2 O XB + 2) S2 2 S2 = 22x S2 3 S2 -0 we have
0 (NBT ( o xx) (X f2x) B)
(NB" B" ox) (Y (xox))(Yo0))) (2)
(61)
(NB" (Sy o x²)
(X (1O2))) S4)
S2 1 a @ OX 0 + 2 2
B"
(1) oxx)
Σ
(62)
1 BT
3 x")
(X B
1 (BT
( 2 x²) (x B BTO:))
N
N
B"
-1 [ oxx) (X ((QL)
1
(B" (N) ox') (2 (2o
(63)
N
N
= B + B² (0 xxx) (X 22x) B
N (B" 3 ox") (X (x) B
1
BN (n) ox)
N (B" (N) ox') (X 2. BB"
N BT (u) ox (x) BB) (+ (T (n ox) (z (2)
(64)
20

---

Published as a conference paper at ICLR 2025
Then define
1 N
Fk
Bd o X 2 B
( (D) x x 02 B
N
BJ (S
1
TT
(22 x (xos f2x) BB BBTBBO:)
(65)
N
1 BT
oxx)
( (B" (o o x²)) (Z o S2x) fn)))
G
(66)
N
Corollary B.6. Define O F, G by the matrices that each row 0x, F&, Gk. respectively, we have
@ BB G
(67)
Proof. We can easily obtain this result by Lemma B.5.
Lemma B.7. Define
H 1 Ba (52g oOxO x²)
1
x (ts)
(l-BBB")
BB") B
(68)
VN
N₂ S Sgg and 8 C 202 1iii//oog (o)(MM) for constant C, we have Vinue N
8Nr
Hall22 s
dist (B.B)).
(69)
VeN
k
k= N2
Hn02 <
8² |
80
dist (B.B*)
(70)
KN2
with probability at least exp(-1111c log(M)).
Proof. Because have
M
N:
T
H
Wiisth
BT xis)
Whisi BTT (I--BB") BBB)) xii)
(71)
VN
(
(111 BTJT La (ls-BBB]) Xis
BT
3"xis) X1.)
(72)
VN
W
BTXT)
1
x
L1 (1-BB") B",
(73)
VN
N
X (I. BB²) B
(74)
N
No
where X E N.xx xd with rows the concatenation of x that wiisk 1. Here we define
N₂ DM Ejn -.m Then directly use Lemma of (Collins et al., 2021), and define
8 = c 2/2 (log(A)) for constant C, we have
Viiin No
8N₂ VeN
H 5
dist BB B-)
(75)
21

---

Published as a conference paper at ICLR 2025
with probability at least - exp(- 111c (o)( Then have
K
K
H
H
2 10
(76)
1==1
S K
1 11
H 12
(77)
N=1
K
N2
k=
8² @10011
dist (B.B))
(78)
KN2
Then the proof finished.
Lemma B.8. Given defined in Corollary B.6, define C M)
we have
Vuiin No.
F < 5
10% dist B, BB
(79)
(1- 8)
| 5 - 5)v
1 11 dist (B.B*)
(80)
with probability at least 1 exp(-111c² log(M)).
Proof. From Lemma B.3, we have
[GB (o) OX) B
-112
N2
S
(81)
N
8)2
Then we consider to bound F3 first, by Lemma B.7 we have
-1
F 12
B" (nf o x²)
S2xt B
|
10
(82)
2
82 dist ( (B.B*) |10112
(83)
(1 8)2
for 8 C 3/2 M)
with probability at least 1 - exp(-1111 (og))). Then consider to
ming N₂
bound F, and have
F
Fill2
(84)
-1112
T
(o) o
(x B
H 0 11
(85)
1
K
N2
<
HHO
(86)
(1 8)2
k=1
N2
1
H
11
(87)
(1 8)2K
82
5
|0'1|2 dist?
B
(88)
(1 8)2K
with probability at least
11pp-111ee2 log( M)). The last equation comes from Lemma B.7.
22

---

Published as a conference paper at ICLR 2025
Lemma B.9. Given G3 defined by
G2 = ( ( o x") (X
-1
"(N
N
B (s2 o x²) (2 o Sui) )
(89)
c (og(M)
and defined in Corollary B.6, Define
we have
V Vming No
8
|Gx
(90)
1
|G|| S K (1-
(91)
with probability at least 1 - exp(-110c² log(M)).
Proof. We can rewrite G5 by
Gk (N B" (
1
-1
M N.
B
B xi.
(92)
j=1
(NB" BT (N x X 24) B
B Xi.i
(93)
N
WiAA
BBBT N (Ω x") (x 24) B
(x)
-
(BTX722)
(94)
where X E R Ngxd with rows the concatenation of x.j that Wi.jsh Here we define N₂
EM Ex=
j=1 Imges Then directly use Lemma A.7 of (Tziotis et al., 2022), and define =
3/2 Mlg(M)
we have
mine N₂
BT XT 2/
0088.
(95)
with probability at least - 1exxp(1113c (og(M)),
Then consider Gg, we have
||&x|l2 S
N (
TT
(0 o x²) (x
(x0 S2x 2x)
B
1 B
X
2
(96)
N
5
(97)
8
Then consider G, we have
K
8
|
2
Σ | S K
(98)
(1
8)
Lemma B.10. Define 8 = C 3/2 Mo
we huve
Vmine No.
1001l2 s v
1- 5
dist (B.
+ I 1- 0
02,
(99)
|B0 - B'0211 5 <ve+
Ve+
dist (B.
+
8
(100)
1 1-8
1 8
1 Σ
(2 x²) (Z 2 14 0%
2 Vd+c
8
8
Vee+
dist [B, (B.B*)
+
1 1
0²
(101)
VN
1 1-
1-
with probability at least 1 exp( - -10 + ))) exp( 1100= log(M)) for some constant C1.
23

---

Published as a conference paper at ICLR 2025
Proof. Define
qk = (12 о x
(x o S2.) B0: - (N o x²) (Y o ozx).
(102)
and Q R with rows the concatenation of qk- With the fact that Y
Ex= Sg ④
XB XB* 0 Z have
qs (Sg o x
X f2 B0,-
(S @ x²) Ut).
(103)
K
XB 0 + +2))
= (S) @
X S2g)
B0 (0) @ x*)
.t @ Σ S2n @
(104)
==1
(Sg o
X 22
BO, (S) 0 x")
(2. 0 O 0 + z))
(105)
(SS o x²) X 02.
B0: (0) x
(X 12 B
(N @ x") Z o 12x)
(106)
= (Sg o x X o 12s)
[B0 B²0:² (S2 @ x ZZ 2 22x)
(107)
Then we would like to consider the Bd. B*0%) first. From Lemma B.5, we have
0 = B + F& + Ge.
(108)
Therefore we have
10: -B R
BB" + BF + BG. 8 0
(109)
< [BBB" - Ia) B 0i + +| || + |BG
(110)
< dist
(B, BB) |10 + |Full2 ||Guell
(111)
s Vedist
(B, + 8)- 1
dist (B.B))
+ 1 1-8
1-
8
(112)
= =(vo 1 1-8
8 disl (B,B*)
+ 1 8
0²
(113)
with probability at least 1expp-110e2 log(M)): (M), and 5 -0 fog(As)
/ming No.
Then we consider to bound og. and we have
||0x1|2 = B B o + F2 + G:
(114)
< 1100112 + | + |
(115)
8
S V + 1 8
dist
B B.)
+ 1 - 8
(116)
with probability at least 1 - e1-eep(-1101) 110c (M))
Then we consider to bound (5% O x
(Z @ I2x), and we have
N Σ
n ④ x
Z ( =
(117)
Wiisk X Zx
(118)
where rows of X2 E Nexd and 2. E S. are x and z subject to Wi.jth 1. Then let 8d-1, denote the unit spheres in d and e dimensions, and Na, denote the - of cardinality
24

---

Published as a conference paper at ICLR 2025
9d and 9h, respectively. Then by Equation 4.13 of (Vershynin, 2018), we have
K
x 0
2
max PENAYEN,
pd
X
x
2
y,
(119)
2
K
NR
=2
max PENayENN
Zk
(Xi.) P) (0s,
(120)
2.
Notice that for any fixed p. y, the random variables " (xij. p) y) are i.d. zero-mean sub- exponentials with the norm most Ci 10xll for some constant Then consider the event
K 3
110ull2 S v 8 1-j dist (B.B*)
1- 5 0²
(121)
1=
which holds with probability at least exp(-105x (M). Then use the Bernstein's 's inequality we have
Nx zk N
(xiss () 28 E
8²
- Eexpl C,N min(
01 (Vo+ I-o dist (B + I-8 122
21
8
}).
(122)
03 Vo+ 1-8 1 disl (B. + 4 02
Setting
0² Vi ( 1 dist (B, BB. -
8 C2
(123)
we have
K
N
Pr(
2k
(Xi.j. ()
N
1=
C20² Vd+c
T- disl BB + -
Ve+
1 8)
VN
I exxp(-CCCad) S exp(-110(d +c),
(124)
for Co large enough. Taking the union bound over all points p. y the Na, have
K Pr(
OT
OT
2o2
Vd+c
Vc+
dist
B, BB,
8 x0) E)
VN
S exp( 110(d + c))
+ c))
(125)
Removing the conditional on E, we have
Pr
K Σ
X
2
> 2C2 2 vd+c
Ve+
1-8 T- dist dist (B.B*)
+ 1-30²)
8
N
VN
k=1
expp(115(d + c)) + Pr(Ee)= < 10p(-105(d c)) exp(-105c' log(M)))
(126)
25

---

Published as a conference paper at ICLR 2025
Define 8 C 2// Mlog(M) we have Vinu No.
Lemma B.11.
K 1 Σ
k1 о x
(x (BOs. B²0:) - -oo (Bor. B²0²) 0
N
S scVd+c +c
(127)
VN
where
8
8²
Vc
dist?
BB,B*) B*)
B,
8-8
(1-8)²
8)2
8
(1 8)2 0² dist (B.B') + vc 5 0²
28
52
c+
1)Ve +02
8--00²
(1- 8)2
(1-
0 (128)
with probability al least exp(- (100p + c)) exp(- log(M)).
Proof. Define
Ns (BO+ 1B0x-B4 (B0k-BB:1))
qk N (N) o xx") (x o (2x)
(BO: -B
(129)
N
No N
8 &R [B - B²)O))
[B0 - 0
(130)
N
where rows of X₁ E R Nexx are Xi,j subject to wigg = Then we would like to define the event
K
n
Ax n Ba).
(131)
k=1
8
A
1101112 S Ve+
1 1 dist BB,
+
0²
(132)
1 1-6
Bg BB, - B.0% S (ve+ v
8
dist (B,B*)
8
+
(133)
1-8
1
happens with the probability at least - ex-pppp(1ll 105 + +c) -
log(M) by Lemma B.10.
Define ge B0,
0% we have
Σ
2. N
ΣV Σ (xx,).x))x,. of gx0%)
(134)
Let Sd-1, 1 10-1 denote the unit spheres in and e dimensions and Na, the 1 -etts cardinality gd and gh, respectively. By Equation 4.13 in (Vershynin, 2018), we have
N X1. Bk
Σ 0
2 max PENUYENN
Dd
0
B y,
(135)
N
2
K
No
2
max
((xi, (p) xi.)(Ok..)) (p. gx) (0x, y))
(136)
N
PeNayEN2
Then for any p.y, the inner products (x..+-Kh) (p. xi.) sub-gaussians with norm most C2, holds we have N xx B1) (p, xi.)) (fx. y) is sub-exponential with norm at most
Ci l|gilla and C2 1l12 C2. respectively for some constants C₁ C2. Then under the condition that &
C3
N N
=
C3
Ve
1 - 5
6
+
(1-8)3
62
dist (B.B")
C3
8
252
N
c+ (a²
+ +1)ve
1
+
(1 (1- 8)2
o² disl [B, B))
i
82
Ve
0² +
(1 0
(137)
26

---

Published as a conference paper at ICLR 2025
The same thing can be observed for (p. Bx) ().. Besides, can observe that
E (.k.s.gx(pp x1,) (0x.y) - (p.ga)(O...)) = =.
(138)
Then we are dealing with N zero-mean, sub-exponential random variables. Using Bernstein's inequality we have
Nk
Pr
Xi. Bb xxa) gx) 2 2 5
kj
22)))
exp
C4N min
(139)
VCsld+e)e VN
Setting
for constant Cs that satisfy Cs S d+c. and taking union bound over all p.y.
we have
K Pr
2 Cs(c+
Σ 0
E
VN
< gute exp (-C + c)) s cc).
(140)
Then by removing the conditional &, have
K
SC <c
o
(141)
VN
with probability at least exp(-100(d c)) - -ee(pp11000 -10c log M)).
B.3 MAIN RESULTS
Theorem B.12. Under Assumption 1- 4, when have N
K² d+c
and ming No. >
c (140 Py lo min ( o2 min) for some constant C, have
Eg
57
1
-1/2
dist (B) s (ist(B",B")
1 Cmin
Craa 1 (1-
Cmax
200
2
1
-1/2
100 Cmax)
2 Cmax
(142)
with the probability at least - exp( (d + c)) - exp((90c² -000 log(M)). Here Nu
N: I E Wijjkr Eg 1 dist (Bo, B²), Crain
n ming No g2 min, En. and Cmaax
MK maxt Nx 02 min, Eo.
Proof. From the optimization steps we have
K
B 4+1 = B'
(
o
x²)
( o (Y (Y (02)T, (143)
N
B
( x7) (xos)) (B'0, BOO
[
X
ZZ 3 24 (O)T
(144)
k
B' - n
( o x²) (x x (B'0% B.
N
k=1
No
(B'et - B 0 ; (0
N
K
K
No (B'Ot -B:07) (0
n o x² (Z 3 24) (0))
(145)
N
N
27

---

Published as a conference paper at ICLR 2025
Multiplying both sides by (Bi),, we have
(B1) (B1)"B* +1 B''
=BB")
K
-n
sh o x*) (Xo (B'ei B²0:)
N
1=1
No
(B'0) - o² 000))(01)) (01)))
N
K
Nx (BBI) 0: - (Bi) 0%) (0))²
k=
+
(B1)"* T 5/% o x") (Z (Z (0))
(146)
(B1)²B
Ie
Σ N(
"
E ( @ x²x (2 s2x) (01))
N
N k-1
K
(B
Σ
((
x²) (x 24)
(B'0 - BBo0) No (B'0) B-0i)) (0L)²
(147)
N
Because we have B++1_ B (R+t)-) .multiplynng both sides by (R++)-- we have
dist (B't1, B¹)
dist (Bi, B.)
Ie Te-n) I (1
k=1
Σ (Bi)
(2 oxx) (2 2x)
||(
K
(BL)
7
Σ [(1h 22% o xx)) (x0sx)) s2x) (B'0% B'O%) -No (B'O; B'oi)) (0t)T)
N
[R
1|2
(148)
Then we can define
A dist (B', B.)
--- E N
(149)
A2
1² (nf の x² 12 22 (01))
(150)
N
k
As N B
Σ
(2% o x*) (x) (B'O) B²OO)
t=]
No B'o - B 0: (00) (02) ))12.
(151)
Then the inequality become
dist r 3B') s (A1 As + +R)
'12
(152)
28

---

Published as a conference paper at ICLR 2025
For the following parts of the proof, consider the following events hold
K &
Ax BB).
(153)
k=1
8 | 11 dist (B'.B') || S VK 5
&
FF 11 1
0²
(154)
(1 (1-8)vl (1-6)vK
(1 - d)
& =
1
12f 3 x²)
[ 2 (0))
N
vd+c
(BB I 8 g00))
(8o+
disl
(155)
VN
1-6 0
=
( o x²) (x
(BO, O²0²) No (BO', BOoi)) (0))7
vd+c VN
I C2
c.
(156)
(157)
where
8
8
A
10%112 S Vo+
1-8 d
dist (B
+
1-000
0²
(158)
B&
Bo B
Ve+
6
dist (B',B*)
+ 1-8
8
0
(159)
1 1-8
Vo 1 (1 01-
8
8²
dist?
(B,
c+
(o²
1)ve
8
28
0²
dist (B, (:) + Vc
6
0²
+
8
8
(1 (1-8)-
5)2
(1 5)2
(160)
which hold with probability at least 1 - exp(- c)) - exp(e (og()) for some constants C₁,C2 C2 by Lemma B.8, B.9, 1. 1.11. Then we consider to bound A1, A2, A3, respectively. Then we consider to bound A₁ first, and we have
Amax A (00)T0') = 0 1
(B") + + G +
(161)
< 22 1/ + 2 F 1| + 2
G 2
(162)
20
+ 2K 8
2 2 2 + (1 - K)2K
|
12 dist?
B'BB"
(1 8)2 01
(163)
28
8
2 (1 K)2K
||0" 1/ + 2K
(164)
8)2
28
2K (1 -)
22
maax, + 2K
(165)
(1 8)2
29

---

Published as a conference paper at ICLR 2025
Then when rj is small enough, it's simple to promise that I 2/2 E O.O0 is positive definite.
Define diagonal matrix W E IRX* and Wa Ns. Then we have
L. N
O²2)
[
(166)
< Mmnin (wW))")))
(167)
1 (amin (W) oin
(B"(B") Tain ((WF' F¹) Gmin (WG''"
(BF")"((""""" + Tmax ((WF)FG) + Tmax (IWG))"\)>)"))) (168)
2m (mmax
< 1
mink No
Jiin 0 (0')azin (IB')B++++ +1
maxe NE
omin (F')
N
N
maxx Nu
+1
(G))
minn (G')
21/
maxe Ni
Fmax (F*))T00)) + Tmax ((G))T)))
+
N
N
max N
+ 2m
Tmax FF )o (G'),
(169)
N
S 1 n/K
min& NA
min aiin (BB") + 2m
max; NA
(| 11, + ||G |
N
maxe No 2m
maxx No
max: No.
+
| ||l|| +
|F"1 +
|
(170)
N
N
N
1
mina No
ounn aai (IB")TB) + 2n
max NA
([[F*]] | + || [|,)),) |
N
+
maxi Nu
([[F")] |[G')1))
(171)
30

---

Published as a conference paper at ICLR 2025
Then under condition 3,, Ez, Ex, Eg, we have
I
N
mink No
σ2 minn o min (BB")TB')
<
n
N
+ 2n
maxy No
6
10 dist (B'.B") + VK
8
0²)
|
N
(1 5) V
K
1 0
+1
maxk No
|001|2 dist
B (B',
VK K
8
(172)
(1 8-88
ming NR
0 min, o min (BB')
1
n
N
maxk N
8Ko²
+
2m
8 max,+
1 1-8 mmax,
N
maxk N
8²
+1
(20 max,. + -2Ko"),
(173)
N
8)2
ming N
maxk N₂
6
20 2ō + [002
< 1 - njKK
2 min, Eo +nk
(174)
N
N
ming N
max; No
< 1 - n
2
Eo +
min,+ Fr
N
min,
5
N
KKK
max NE
+
-² min, Eg.
(175)
20
N
where Ep 1 dist
[ (B'
02
(BB.) The Equation (174) holds when N₁. is large
enough that makes the following equation holds
Eo 20(1 + 02)²
S 25
<
max
o min
(176)
-
N2
where
Smax. mmin.
and the equation holds when N₂ satisfy
c1 + log
min Ñu 2 Co
min
K
(177)
Eg
σ m
Then consider A1, will have
A < dist B BB.BB.)
(1-(
((
ming No
max No
)) KKoii Foo)
(178)
N
N
The consider A2, because
B
1, and Eg holds, we have
A2 < 11102
Vd+c
Ve+ 1- T-Jdish dist (B,B*) + 1 0²
(179)
VN
21 Vd+c VN
Ve+
(180)
10
31

---

Published as a conference paper at ICLR 2025
Similarly, for Ag, we have
vd+e
As $ n2
E,
(181)
VN
Vd+c
((VC-
1 2
disl B BB) + 1100 +
1
vc
(o² +111
22
(182)
VN
V10
20
Combining Equation (152), (178), (180), and (182), and choose C' = max{C;, + C2), we have
dist B BB) S disl BB,B')
ming No
maxx No
))
(1 (n-nk
minnn Eo
N
4
N
Vd+c VN
Ve++ 1
>|(((()))))
vTO
vd+c
(o² + +1) (2ve+ }) ||<
c
(183)
VN
Then the remaining thing is to bound (R++1)- Firstly we define
K
St
2
o
x²)
(X o 12x) (B'O BB0;) (02)*²
(184)
k=
K
E =
( + x T
(Z 3 26 (0)
(185)
k=1
Then we have
B+ B
7 st
+
" E
(186)
N
N
Because (R++)TR***
(B"-1) and we have
"
+ (B)) + " (IB')'E (E
N
N
n² (S'
No (EE) + (S)) + n² N2 "E''E,,
n²
+ No
(187)
I. "
(B')''' + (S + " N (B')EE + (B
N
1
n" N2 (E' + (
n N2 E''
N2 S
(188)
By Weyl's inequality, we have
min
(R++))
2
N Amax
"
(B''TSS*' + B'))"B')
7
7 Amax
No Amax (E'")TS² + (S))
(189)
N
Then we can define
R₁ Amax
(B''''s+ + Bs)
(190)
N
R2 Amax
(B'')'E' + (E') B¹)
(191)
N
R3 Amax
((E')T'S + (S')' E²)
(192)
32

---

Published as a conference paper at ICLR 2025
Then we have
o min (R+ 2 Rı R2 - R3.
(193)
Then we consider to bound R1, R2, and R3, respectively. Consider R1 first, and we have
R1
2n max
P SBBB')'S'p
(194)
N
|pll==1
2n
max p" PB())T"
Σ 2 o x²) (xo R.) (B'O B:0;) (0:)T) p. (195)
N
[PPll 11
2m
max P" (B')"
2 + x*) (X f2)) (B'0 BO
N
lipll, -1
No (B'0%
B 8'07) ((L)) P
2n max P (B)) 1p
Σ No (B'8) B-ei) (0))² p.
(196)
Under the condition Es. we have
R1
S CnC2
B' 2 VN
cc+c
c+
2n
max
p" (B')"
Σ Nu (B'0% BOoi) (0))²)
p. (197)
N
llp
2=
Cn
Vd+c
(e++
1
dist (B,BB) +
1
+
vc
(o²
VN
V10
400
20
K Σ
+ N
2m
max ||
p² PBB)TT
Nk
(B'O% - -B:0)) (01)² P.
(198)
k=1
C2 vd+
(0² (0
(vo+ -11aa 2
< 4n
VN
v1O
K
27
max
p"
Σ
Nu (B'0 - -B:OE) (01)T)
(199)
N
The remaining thing is to bound
2g
N
"BPT(B') (E# No (B'O; - -B.0;) (01)²) P. and we have
2n T
(B')T
No
(''0 - B.OO)) (0h)T) P
N
1=
2m tr N
Σ Nu (B'0% -B:0:) (0')'pp"(
(200)
2n Ir
Σ
K
No
(B'o - B²OO)) (BB')BB0 + +F + Gb) T
pp"
(201)
N
1=
Then can define
T
27 N
Σ No (B'0) -B:0;) (B')
(202)
T3
21 tr N
Σ No (B'0) B'0i)) (F (B')T
(203)
21, T3- N
Σ No (B'O' - BB0;:) (G Pp (B")""
(204)
33

---

Published as a conference paper at ICLR 2025
Consider 7i first, have
K
Ti =
2n
N tr
N (B'(B))'B'3 + B + BBG; - B.0;)
A=
(0) (B
B pp (Dp")(""))
(205)
K
2n tr
(B(BITT
NB (B)" (B")
= N
2m
Σ No (0')) (B") B'pp"""
N tr
20
Σ No (B'G)(03))) (B')"''p""B")" (B))
+
(206)
N
= 2ntr
(BB) ((F)) + (G')²) wB'
(207)
max Nk
s 2n
[ + G 1)) |0||,
(208)
N
Because & holds, have
Ti S 21
max No
8
|001
+ K
5
0² 1 1001l2)
(209)
N
(1
8)VK
1 1
max No
8
2n K
72
max.+
max.+
(210)
N
1
1-8 8
π max; N
Ego
(211)
5
N
Then consider T2, we have
K
T2
21 N
tr
Σ No
(B' - B-0;) (F (F
(212)
1=
K
2m tr N
No (B'(B') + B'F ( +BG
B=OO) - (FF) pp (B'IT
(213)
(B') (F (G)) W
2nnr
(214)
max No.
[|F* F' || G G
2m
(215)
N
2n
max No.
82
22
max.*
8²
/ V mmmxxxx0))
(216)
(1-
81-
πlK max Nk
Eno2 min,
(217)
100
N
34

---

Published as a conference paper at ICLR 2025
Finally, consider T3, have
K
211 N
Nk (B'0% - -BrO)) pp
T3 =
tr
(218)
2m Ir N
K
No (B"(B")BBO + B + B'G - B-0i) (i))"pp"(BB)
(219)
k=1
=2ntr
(B")"B' ((
G)) Wpp"].
(220)
s 2n maxa NA
(IG'
2 F
+
|
F [[G])1)).
(221)
N
S 2n max; N&
8²
Kot
8²
/ K m
(222)
N
51-
(1- 8)2
nKK maxe No
Fno min.*
(223)
100 N
Combining Equation (211), (217), and (223), we have
) ((0+ +1) (Vo+ 1 vIo))
11nK
max; N
R1 W 4n
C2
+
End min,. Enomin...
(224)
50
N
Then consider R2, because condition &g holds, we have
R₂
2n max
EBB'
(225)
N p
2m
max pll,"
p" PiBIT Σ 2 C x²) (Z Scc (06)T
P.
(226)
N
1 K
2 3 x² (Z + 24 (1/)
< 2m
Σ
(227)
N
200 2
Vd+c
(o++ 11)
(228)
VN
10
Then consider Rg, when condition Ea, E,, &s, and & hold, we have
1/2
Rs = N
max p" (E')"S'p
(229)
||pll==1
2n²2
2102 Voo2Vd+c de (vet 1
10 10)) (cs vd+c +1) (0² +1) (1²2+1)
(ve+
)
< N
VN
vN
(230)
2 N3/2
(02 +12 (vc+
1 3
S Sc
(231)
v10
The last equation holds when N satisfy N 2 R²
Then combine Equation (193), (199), (228).
2tc
and (231), because N 2 (vē + 1)(0² + 1) holds, we have
2
min RR* +1 2 - Rı - R2 - R3,
(232)
vd
(ve+
1 -1
)=0²² 11nK max; N₂
1 - 8nc
Ego min..
VN
V10
50
N
(233)
35

---

Published as a conference paper at ICLR 2025
Combining Equation (183) and (233), we have
dist (Bi+1 BB) S dist BB'BB* B.)
1 1-
minx Ni
max; No
))
o Painn
Fio + np
vd+c VN
(eet {
N
N
Omio (R 1
v vd+
(0 (o* +1) (2vo+ 5) Omin FRia(R (R
1)
n
(234)
VN
When max No
satisfy
max No
200 vd 1)(v++1)²
C
(235)
VN
K KEgo min,+
we will have
max N N
o min
R
*++
2
2 2n
J minu, Eu
(236)
ne Vd+c
maxx N N
Ve+
nk
min, Eo
(237)
VN
VIO
200
Then we have
dist
B +1 B
S dist (B',BB)
ming Nx N
57
max; N₁
- nk
- Flo
200
N
-1/2
max; N
n
2
Ea
max; N
(1-
maxe No
-1/2
n
min, Eo
omin Es
(238)
100
N
N
Remark B.13. Different from previous studies that assuming clusters to have the same number of data points (Collins el al., 2021; Tziotis et al., 2022), we analysed the impact of imbalanced sampled numbers among clusters in Theorem B.12. We can see that HCFL*'s convergence is strongly influenced by maxs N₁ and ming NÇ. In other words, HCFL+ converges faster as ming Ny. max. Nr. increases. This suggests that HCFL+ performs better when the number of samples is evenly distributed among all K underlying clusters. In contrary, if the number of samples is extremely imbalanced, the HCFL* may hard to converge to optimum.
C RELATED WORKS
Federated Learning. As the de-facto algorithm in FL, FedAvg employs local SGD (McMahan et al., 2016; Lin et al., 2020) to reduce communication costs and protect client privacy. However, distribution shifts among clients pose a significant challenge in FL and hinder the performance of FL algorithms (Li et al., 2018; Wang et al., 2020; Karimireddy et al., 2020; Jiang & Lin, 2023; Guo al., 2021). Traditional FL. methods primarily aim improve the convergence speed of global models and incorporate bias reduction techniques (Tang et al., 2022; Guo et al., 2023a; Li et al., 2021; 2018). At the same time, some studies investigate feature distribution shifts using domain generalization techniques (Peng et al., 2019; Wang et al., 2022a; Shen et al., 2021; Sun et al., 2022; Gan et al., 2021). However, single-mode data distributions, especially when dealing with concept shifts (Ke et al., 2022; Guo et al., 2023b; Jothimurugesan et al., 2023). To tackle these challenges, clustered FL algorithms are introduced to enhance FI. algorithm performance.
approaches are inadequate for handling heterogeneous
36

---

Published as a conference paper at ICLR 2025
Clustered FL with fixed cluster numbers. Clustered FL groups clients based on their local data distribution, tackling the distribution shift problem. Most methods employ hard clustering with a fixed number of clusters, grouping clients by various similarity metrics, such as local loss values (Ghosh et al., 2020), local model parameter differences (Long et al., 2023), communication time/local calculation time (Wang et al., 2022b), and fuzzy e-Means (Stallmann & Wilbik, 2022). However, hard clustering may not capture complex relationships between local distributions adequately, and soft clustering paradigms have been proposed address this issue. For instance, FedEM (Marfoq et al., 20 employs AmpectationMMxxmmizatin techniques to maximize likelihood functions. FedG- MMMcttepwu2022persoaaiieee suggests using joint distributions instead of conditional distributions. FedRC FedRC(Guo et al., 2023b) introduces Robust Clustering, assigning clients with concept shifts to different clusters to enhance model generalization. FedSoft (Ruan & Joe- Abe-Wong, 2022) calculates weights based on the distances between clients' local model parameters and cluster model param- eters, with smaller distances indicating larger weights for that cluster. In this paper, we propose a generalized formulation for clustered FL that encompasses the current methods and improves them by addressing issues related to intra-client inconsistency and efficiency.
Clustered FL with adaptive clustering numbers. Another line of research focuses on automat- ically determining the number of clusters. Current methods utilize hierarchical clustering, which measures client dissimilarity using model parameters or local gradient distances. Most current meth- ods modify cluster numbers by splitting them when client distances within clusters are large (Sattler et al., 2020b;a; Zhao et al., 2020; Briggs et al., 2020; Duan et al., 202 a;)]; Recently, StoCFL (Zeng al., 2023) suggests initially setting cluster numbers equal to the client count and merging clusters with small distances. In addition to model parameter distances, some papers employ alternative distance metrics for improved performance. For instance, (Yan et al., 2023) employ principal eigenvectors of model parameters. (Vahidian et al., 2023) use truncated singular value decomposition (SVD) to obtain a reduced set of principal vectors for distance measurement. Meanwhile, (Wei & Huang, 2023) focus on the distance of normalized local features. FEDCOLLAB (Bao et al., 2023) focuses on cross-silo scenarios with a limited number of clients and quantifies client similarity by training client discriminators. However, the need for discriminators between every pair of clients in FEDCOLLAB makes it challenging to expand to cross-device scenarios with numerous clients. In this paper, we concentrate on cross-device settings, introducing a holistic adaptive clustering framework enabling cluster splitting and merging. We also present enhanced weight updating for soft clustering and finer distance metries for various clustering principles.
D ALGORIHTM
Details of the HCFL-. In Algorithm 2, we present a concise summary of the comprehensive
algorithm that integrates all the enhanced components of HCFL+ as introduced in Section 4.
Specifically, during each communication round, the algorithm performs the following steps: (1)
Randomly selects a subset of clients. (2) Calculates prototypes using Equations (11) and (12). (3)
Performs local updates using Algorithm 3. (4) The server aggregates local updates. updates cluster model parameters, and computes client distance metries using Equation (9) for each cluster k. (5)
Identifies kmar as the cluster with the highest average distance. (6) Checks if the maximum distance within kmazr significantly exceeds the average distance in this cluster. (7) If the following condition is met, splits the clusters using Algorithm 4.
max (D.....) mean (D)
1imee) p.
(239)
(8) Mark and remove the empty clusters no clients will assign large clustering weights to using Algorithm 5.
Intuitions on the distance metrics design. From the objective function (Eq. (2)), we should assign higher clustering weights wi.jsk to clusters with greater La maximize the objective function. Because the ultimate goals of the clustering algorithms are solving the objective functions, we analyse the C. check the key factors influencing the value of Cg, and the relationships between these factors and the clustering principles.
We use the following algorithms as examples. For FedEM (Marfoq et al., 2021) and IFCA (Ghosh et al., 2020), Lx y, Q 0z) Po.o, (y)x); For FedRC (Guo et al., 2023b), La (x, y, Φi 0x)
37

---

Published as a conference paper at ICLR 2025
(x.0,(x.)) (x.9)
Pa.o, (x]Po.e, (o)" Defining z = g(x; ()) as the local features extracted by Φ. assuming a x - 2 ->
Probabilistic Graphical Model (with x and y being independent given z). we obtain:
Plyxx ; Φ 0c) 1
P(alx.v:))
(FodEM, IRCA)
P(x.x: 0c)
(FeIRC)
(z. y. 0. (2)P(z))x
P(z|,
Then we aim to give the following explanations of the three terms P(z|x; P(z|x; Φ), and Eg (z, yy 0z), which align with the terms considered in Sec 4.4.
P(z|x; () for feature and label shifts. Feature shifts introduce significant distances in Additionally, x with different values generally exhibit substantial distances the feature space. Without this, classifiers cannot distinguish samples with different labels. Hence, employ P(z|x; x: Φ) to assess both feature and label shifts.
P[zz|, x, y: <)) for concept shifts. Concept shifts signify alerted x y correlations. Hence, samples with concept shifts but have the same y should exhibit significant difference in P((z y; Φ)
(z, y; 0s) for the quality of clustering. The (z, 0x) is defined using features z = g(;; () instead of data in (x, y, Φ, 0x). This term evaluates if features can be correctly assigned to clusters given the current θ otherwise, the objectives in (2) cannot be achieved.
Finally, we propose the following distance metric:
{
max {de, dey En, [2, y: 0n) Eng Ea(a, y: d,E.,, [En(2, vi 0x) En, [u(z, vi 04)
ASCP
D
(240)
CSCP
where dist the cos-similarity in this paper, and
da=max {dist (ED, [P(z)x, yi 6) ED, [P(z|x, yi o)1)}
(241)
y
dr= = dist (ED, [P(z]x; d) En, [P(z|x; 6)))
(242)
The distances above become large only when the following conditions occur together: (1) Large values of de indicate concept shifts between clients and j: (2) Large dif indicate significant feature and label distribution differences. (2) Large values of IED, [u y: 0x) ED, [Lu(2, y; 0x) indicate incorrect clustering weights with high confidence.
Approximation of the distance metrics in practice. When calculating the distance metrics (Equation (9)) in practice, to avoid training extra generative networks and transmitting more data between servers and clients, we substitute Wi:k for E. (z, y; 0x) since Wisk positively corre- lated with (x y 0x) (Marfoq et al., 2021; Guo et al., 2023b). Additionally, we approximate ED, [P y Φ) and ED, [P(z yi 6)] using feature prototypes. The prototypes are defined by the following equation:
de = Dist(PPii,PPar c,
Pist
(243)
where
N
N
Pas d
1y, 1g(x $).""
1
1 ccg(_)(()
(244)
Ni,1
Noc
N.
Pifs Edd
No Σ g(x,,,))
(245)
No Nr.- = ビ =r ((xig Φ) is the function parameterized by d, Dist is function to measure the distance between prototypes, which we use the cosine similarity as example in this paper.
38

---

Published as a conference paper at ICLR 2025
Algorithm 2 Algorithm Framework of HCFL*
Require: Local datasets DI...., Dx. number of local iterations T. number of communication rounds T. number of clients chosen each round S, initial number clusters K number classes C, and hyper-parameter p.
Ensure: Trained global feature extractor s², final number of clusters K", and cluster-specifie predictors θ = -0T
0 kT)
1: Initialize Φ 0 1 xo)
2: for .,.... - do
Choose subset clients St, where [8*| S.
4:
for chosen client St do
5:
Calculate client prototypes P by Equation (11)- (12). Fet1, _ o² ,0,, f Local updates by Algorithm 3. Send P J/+1, and w 6T, 07 Vk K' to the server.
6:
7:
jilatett + -F(+1
Φ Eiest Eiesi N.Ø..
0 Eiest Eiest N.OT, Vk S K'.
Fet [Ftt]] 1... "..,,F :. where Ft
a. t- 12,
Flatest ck,
E, Flatest
12:
Initialize Cj = 0. wk S K"
13:
for all client do
G4 f arg max, Wes-
15:
C. + C, Ui.
16:
R'
17:
for k < K' do
18:
if C is empty then
19:
R' + R' k.
20:
Get the cluster-specific distance matrix D₁: E R R155 Ak S K by Equation (9). Kmin arg max; max (D)).
21:
22:
if max(DD Rmmnn) mean D kminn then
23:
Split Chemen into two clusters Chimen.1 1 and Chmeni
24:
0.+1 Exech kmend
Σiect
kmin. N,OT K.msii
25:
Add cluster and update Fg by server side of Algorithm kt K* L
26:
27:
else
28:
Kt+1 c- K*.
29:
for cluster ho R2 do
30:
Remove cluster hyr and update Fg by server side of Algorithm 5.
31:
KT+1 Kt+1 1.
32:
Send 2²+11 (+1 [0] ,..,, 0 1 and information about add/remove cluster clients.
Algorithm 3 Local Updates of HCFL+
Require: Number of local iterations current number of clusters K*, number of classes C, local dataset D;.
global feature extractor ø', cluster-specific predictors O = [0 xx
Ensure: Trained feature extractor Φ predictors O 10,,,.. xTx,) 22/
(axt), and
F!+1 = [F .... F.411 where Ft+1 = [r++] F.H
1: Update "ijjk
1+1 7/ 2.
w at+1 by Equations (4)-(5), Vj S N S K'.
Locko
b Tier
2: for 1,..., T do
b Tier 1
3: Update 07,, by Equation (6), vk < K*.
4: Update oi Equation (7).
5: F(11 + Σn4 7i ikk
39

---

Published as a conference paper at ICLR 2025
Algorithm 4 Cluster Adding of HCFL*
Require: kmin. set of clients C. the corresponding Ol for each client i E C m and Fg.
Ensure: New F7+1, and predictor of the new cluster 0* K++11
I: Server Side:
0++1
Exech Eiect Eiect mmini22
N.
K'+1
3: Add Fg,K(41 t- Fu.kmin to Jt+1
Add Flatesi 5- platest
s,hmin Flatest Vi.
5: Client Side:
6: WasKit1 + Wh.sikmmnn /2. Vj < No.
7: Wefikmin Wi.jikmmn
/2, Vj N..
WaK++1 Wiskm in
Wehmiin f- Wiikm
Algorithm 5 Cluster Removing of HCFL*
Require: The cluster needs to be removed kr. and F!+1
I: Server Side;
2: Remove F(+ from F
q.Rr
3: Remove platest from Folatest Vi.
4: Client Side:
5: Wiiikk Enge, Wiijin" Vj & No, ls o kr.
Wi.sik
6: Wis Enar, win'
Wilk
Vk & kr.
7: Remove "oiiii. Pesik, Wefak Whi Vj $ N..
40

---

Published as a conference paper at ICLR 2025
E EXPERIMENT RESULTS
E. DATASETS AND MODELS
Diverse distribution shifts scenarios. Similar to previous work (Guo et al., 2023b), the diverse distribution shift scenario construct clients with three types of distribution shifts with each other:
Label Distribution Shifts: We use the idea introduced (Yoshida et al., 2019; Hsu et al., 2019; Reddi et al., 2021), where we leverage the Latent Dirichlet Allocation (LDA) with a 1.0. We split datasets to 100 clients by default.
Feature Distribution Shifts: We utilize the idea of constructing CIFARIO-C, CIFARI00-C and ImageNet-C (Hendrycks & Dietterich, 2019). In detail, we apply random augmentations to client samples, selecting from 20 types, including 'Original', 'Gaussian Noise", 'Shot Noise', "Impulse Noise", 'Defocus Blur', "Glass Blur'. "Motion Blur', "Zoom Blur', 'Snow', 'Frost', 'Fog'. "Brightness' 'Contrast', 'Elastic'. 'Pixelate', 'JPEG', "Speckle Noise", 'Gaussian Blur', 'Spatter', and 'Saturate'. Augmentation types remain consistent within each client.
Concept Shifts: For label y S Cs, it becomes y. (1 + y)%C3, and (2 + y)%C2 across concepts, where Cg = [C 3]. and C is the number of classes.
Noisy label scenarios. We follow the methodology of previous works (Fang & Ye, 2022: Ke et al., 2022) to construct noisy label scenarios. Our approach involves two types of noisy labels: symmetric flip and pair flip. Symmetric flip entails randomly flipping the original class label to any wrong class label with equal probability. Pair flip involves flipping the original class label only to very similar wrong category. We use the parameter X control the noisy rate, where X 0.1 indicates that 10% of the data have wrong labels.
E.2 BASELINES AND HYPER- PARAMETER SETTINGS
Detailed implementations and pyper-parrmeter settings for all the algorithms Unless special mentioned, we split each dataset to 100 clients with 3 concepts. The learning rates are chosen in (0.03, 0.06, 1.0), and we report the best results for each algorithm. We run the algorithms for 200 communication rounds and set the number of local epochs to 1. The experiments are conducted on single NVIDIA RTX 3090 GPUs.
Detailed implementations and hyper-paramete settings baseline algorithms. The details of the settings and hyper- parameters we used for the baseline methods a summarized below. We exclude the algorithms that do not require additional tlper-paramenn here.
CFL (Sattler et al., 2020b). We use the public code provided by (Marfoq et al., 2021) for the CFL algorithm. The affect the results of the algorithm in Table 1.
tol, and tolg are tuned, and we report how the saprr-prrameters
ICFL (Yan et al., 2023). Follow the same setting as the original paper, we set the hyper- parameter a * (0) to (0.85, 0.98), and E = 4.0.
stoCFL (Zeng et al., 2023). We choose r = {0, 0.05, 0.1, 0.15) to control the trade-off between personalization and generalization as suggested by the original paper. In addition, we choose À 0.5, which always achieve the best performance as reported in the original paper.
HCFL+ (FedRC) (Guo et al., 2023b). We set á = 0.4, and choose p = (0.05, 0.1, 0.33. The distance between clients are calculated by Equation (9).
HCFL+ (FedEM) (Marfoq et al., 2021). We set 0.4, and choose {0.05, 0.1, 0.3). The distance between clients are calculated by Equation (9).
HCFL+ + (FeSEM) (Long et al., 2023). We choose = {0.05, 0.3). Follow the original paper, use hard clustering paradigms that does not require the sxper-parameter j. The model splitting process is the same as (Sattler et al., 2020b) that designed for hard clustering paradigms. The distance between clients are calculated by Equation (9).
41

---

Published as a conference paper at ICLR 2025
Table 4: Performance of algorithms noisy data scenarios. We evaluated the performance of algorithms using the CIFARIO dataset split into 100 clients. For each algorithm, we report the best test accuracy for all 200 communication rounds.
CIFARIO (MobileNerV2)
Algorithm
Pairflip, X 0.1
Pairflip.x = 0.2
Symflip, X 0.2
Symflip, 0.4
FedAvg
54.75 +1.45
52.35
+1.65
52.60
10.50
41.80
#0.50
FeSEM
32.60 +1.30
35.25
+2.95
32.40
=2.80
29.70
#0.01
IFCA
24.95 +7.05
20.55
+4.65
30.35
+2.05
36.05 #4.45
FedEM
64.40 #1.10
57.55
+2.95
53.00
#1.90
43.10
#0.20
FedRC
67.90 #1.00
59.95
#1.05
55.25
#2.05
42.00
#0.40
HCFL +
66.70 #0.40
62.70
+0.30
59.95
#115
47.20
#0.20
60
25
Number of Clusters
Number of Clusters
20
M
ww
30
10
0.3
25
150
175
200
100 Communication Rounds
Communication Rounds
(a) HCFL* + (FeSEM)
(b) HCFL + (FedEM)
Figure 4: Number of clusters in HCFL communication rounds. We illustrate changes in cluster numbers across communication rounds for various values using the CIFAR-10 dataset in experiments.
E.3 ADDITIONAL EXPERIMENT RESULTS
Results on noisy data scenarios In Table 4, we show the performance of clustered FL. emthods on noisy data scenarios. Results show that HCFL+ consistently outperform other methods by large margin.
Additional results on diverse distribution shift scenarios. In Table we show the performance of algorithms with 3 = 0.4. Results show HCFL always achieve the best test accuracy, and achieve a good local-global balance.
Ablation studies on only using feature extractor-ckassifer split mechanism. In Table 10, we present a new ablation study HCFL', only using the feature-extractor classifier split method top of the naive HCFL framework (Using backbone algorithms for Tiers & 2, and CFL for Tiers 3 & 4). The results indicate HCFL' outperforming the original algorithms, notably on CIFAR100
Number of clusters in HCFL+ over communication rounds. Figure 4 shows how the number of clusters evolves across communication rounds for different P values using the CIFAR- 10 dataset in our experiments. It is evident that HCFL* allows the algorithm to automatically determine the optimal number of clusters during training.
Efficiency comparison. In Table 1,, we compare the efficiency of HCFL+ with baseline methods. Notably, HCFL (FedEM) demonstrates superior performance while using a significantly reduced number of clusters, resulting in lower simulation time and memory usage.
Evvluattnn the HCFI. without the split is challenging due to the 200+ million trainable parameters, but this he reduced 10 million with the split.
42

---

Published as a conference paper at ICLR 2025
Table Performance of the adaptive clustering methods. We evaluated algorithm performance CIFARIO
CIFARI00 datasets, employing 100 clients. For each algorithm. we present highest validation test
accuracies across 200 communication rounds, and the final number of clusters during training denoted as K All experiments utilized MobileNet-V2 (Sandler et al., 2018).
CIFARIO, 0.4
CIFARIOO, 0.4
Algorithm
Val
Test
KT
Val
Test
K-
FedAvg
48.16
1.664
49.93
+0080
3.0
22.77
+0.01
24.62
+0.55
3.0
FeSEM
46.08
+4.54
35.99
+44.9
3.0
23.56
+1.52
22.31
+1.08
3.0
IFCA
36.15 +3.45
24.79
+1.18
27.72
+0.82
21.37
+1.33
3.0
FedEM
60.26
+1.10
54.44
+0.04
3.0
25.80
+0.20
22.88
+0.19
3.0
FedRC
57.99
+0.29
56.75
+-0.8
3.0
30.94
+0.88
31.63
+0020
3.0
CFL
tol, = 0.4, tol, 1.6
61.86
#5.29
51.15
+0.82
6.0
34.11
#6.35
21.04
#2.21
5.0
tol, 0.4, tolp 0.8
60.42
#0.31
41.59
#2.14
8.0
36.23
#3.58
16.03
#2.69
6.0
tol, 0.2, tolg
49.14
#6.11
49.88
+4.21
3.0
34.20
#7.13
26.42
+0.73
ICFL
a" (0) = 0.85
77.73
+0.47
52.03
+0.10
100.0
49.71
+0.55
28.55
#0.03
100.0
a" a0(0) 0.98
63.69
13.58
54.02
11.11
81.5
45.72
11.10
28.45
10.82
70.0
StoCFL
7 0.00
48.55
10.95
51.25
11.16
1.5
24.50
10.03
25.70
11.51
1.0
0.05
57.84
12.26
50.42
+0.97
20.5
26.24
11.46
26.60
11.17
4.0
0.10
72.91
+2.25
47.84
+2.60
59.0
67.67
+1668
9.89
+0045
86.0
0.15
77.19
+2.31
41.49
+0.97
92.0
70.13
+0.27
7.77
+.023
94.0
HCFL+ (FeSEM)
P 0.1
85.30
#1.05
45.20
+0.28
47.0
38.61
14.14
18.29
#2238
A 0.3
80.34
1.33
48.25
12.72
20.5
44.65
10.35
21.73
11.27
12.0
HCFL+ (FedEM)
0.05
80.31
11.60
53.62
14.36
18.5
62.19
#1.54
21.15
#0.88
44.5
P 00.1
82.89
#0.92
56.27
+1.08
26.5
+0.06
21.29
#0.87
31.5
P = 0.3
80.72
11.90
55.77
11.93
10.0
49.84
16.85
28.62
10.78
11.0
HCCF+ (FedRC)
P 0.05
68.48
#0.25
66.77
+0.28
9.5
38.75
#0.98
30.45
#0.07
10.0
P 0.1
68.56
13.56
65.75
15.40
6.0
40.30
11.19
30.23
10.85
11.0
0.3
70.86
10.31
70.13
10.42
5.5
39.62
10.34
32.22
10.20
5.0
Table 6: Ablation studies on techniques in Sec 4.2. We evaluated algorithm performance on CIFARIO and
CIFAR100 datasets, showcasing the top Validation and Test accuracies for each. We kept = 0.3 consistent across all algorithms and varied já to adjust the penalty term's 's strength in the objective function. The best results each block are highlighted.
Algorithm
CIFARI0,
CIFARIO,
CHFARTO0, 0.2
CIEART0L
Test
Test
Test
HCFL (FedEM)
83.67
62.43
=0.71
80.72
55.77
50.72
+2.97
#0.18
28.62
60.48
#:2000
#1.17
0.4
79.52
+0.11
53.33
#2.97
76.50
+-0.54
48.07
-2.06
44.85
+0.48
28.3
+-0.1
41.52
+0.08
22.83
+0.42
HCFL (FedRC)
A
+0.35
=1.00
#1.13
89.91
20.36
68.77
#1.06
69.53
#:0.21
68.54
=1.08
39.38
201.00
35.95
#0050
39.77
2:2
31.52
2:0.0
0.4
69.33
+0.21
69.67
=1.27
70.86
+-0.51
70.13
+0042
39.97
+0.21
36.50
+0.28
39.62
+0.34
32.22
+0.20
Performance of algorithms under various heterogeneous settings. In Table 12, we show that HCFL* consistently outperforms the baseline algorithms across all settings, demonstrating its reliable performance improvement.
Clustering quality illustration. In Table 13, we present the clustering results to evaluate the clustering quality. It is evident that HCFL* successfully identifies relatively optimal clustering by: (1) assigning all samples with the same label to the same cluster; (2) avoiding the creation of an excessive number of clusters, observed in stoCFL and ICFL.; and (3) achieving higher validation and test accuracy compared to the baseline methods.
43

---

Published as a conference paper at ICLR 2025
Table 7: Ablation studies on techniques in Sec 4.3, We evaluated algorithm performance on CIFARIO and CIFARI00 datasets, displaying their highest Validation and Test accuracies. We kept consistent at 0.3 for all algorithms. "w/ SCWU" denotes the of soft clustering weight updating mechanisms designed in Section 4.3.
Algorithm
CIPARIO, $ 0.2
CIFARIO, 9 0.d
CIFAR100, 0.2
CIFARIO0, 3 0.4
Test
HCFL" (PedEM)
SCWU
83
+0.72
62.43
10.71
80.72
$1.00
06.77
=1.00
50.72
=2.97
32.13
10.15
40.84
1: 85
28,
1:078 27.40
SCWO
63.84
-2.12
49.77
47.91
HCFL" (PadRC)
w/ SCWU
69.33
#0.1
69.67
=1.27
70.86
#0.31
70.13
#0.42
39.97
=:022
36.50
36.50
39.62
#0.34
32.22
=0.20
SCWU
68.83
+1.24
32.08
Table 8: Performance of algorithms with R8snet18, We evaluated algorithm performance on CIFARIO datasets
with 9 = 0.2, displaying their highest Validation and Test accuracies. All algorithms utilize ResNet18 and run for 200 communication rounds.
Algorithm
Val
Test
CFL
tol, 0.4, tol, = 0.6
63.07
17.42
53.65
12.33
tol 0.4, tolp 0.8
61.14
+1.87
54.87
#1.32
ICFL
a" (0) 0.85
80.46
10.9
45.28
16.5
a" a²(0) = 0.98
82.34
+0.28
44.08
+0.40
StoCFL
== 0.1
57.41
1.669
48.95
11.5
T 0.15
66.54
#1.05
47.77
10.14
HCFL+ (FeSEM)
P = 0.05
86.90
10.20
50.34
15.99
= 0.1
85.55
10.24
49.38
16.15
HCFL+ (FedEM)
P 0.05
83.88
+0.25
58.92
#1.11
P = 0.1
83.83
#0.01
60.27
#3.11
HCFL (FedRC)
0.05
67.72
1.30
64.13
-033
P 0.1
67.51
+0.24
63.15
+0.78
Table 9: details about the ICFL results.
ICFL
CIFAR100 (Val)
CIFAR100 (Test)
Round on Best Val
52.73
6.40
Round on Best Test
29.22
32.77
Table 10: Ablation studied on only using Feature classftorrclassiffer split mechnism. HCFL' uses the feature-extractor classifier split mechanism and the optimization steps outlined in Fq (4)- (7).
Algorithms
CIFARIO (Val)
CIFARIO (Test)
CIFAR100 (Val)
CIFAR 100 (Test)
FedEM
66.49
$3.64
29.75
24.18
FedRC
63.65
59.41
34.56
37.62
FedSoft
83.62
34.70
73.58
4.60
HCFL/ (FedEM)
61.24
59,83
42.44
28.13
HCCFL' (FedRC)
63.58
63,57
37.46
37.27
HCFL' (FedSoft)
85.76
52.20
76.70
13.47
HCFI. (FedEM)
83.67
62.43
$0.72
32.13
HCFL+ (FedRC)
69.33
69.67
39,97
36.50
IICFL* + (FedSoft)
85,92
51.23
76.28
12.50
44

---

Published as a conference paper at ICLR 2025
Table 11: Efficiency comparison. The CIFAR-10 dataset is divided among 100 clients, with each client holding data from 2 classes. The hyperparameters are chosen based on the best-performing configuration from Table 1.
Algorithms
Memory (M)
Simulation Time (s/it)
Final Cluster Number
Val
Test
(c()) 0.85)
4190
StoCFL (7 0.15)
2834
178.53
86.92
20.00
HCFL (FedEM. 0.1)
2564
Table 12: Performance Under Various Heterogeneous Settings. evaluated algorithm's performance on the CIFAR-10 dataset, presenting the top Validation and Test accuracies for each setting. Results are reported with / = 0.8. Additionally, C = 2 and C' indicate that each client contains data from 2 and 4 classes, respectively.
8 0.9
C = 2
C' = 4
Algorithm
Val
Test
Val
Test
Val
Test
ICFL
72.94
24.98
83.96
46.00
72.34
72.30
stoCFL
60.0
11.97
86.92
20.00
70.38
32.60
HCFL (FedEM)
73.68
28.63
95.14
49.00
89.80
62.30
Table Clustering quality illustration. present the clustering results HCFL" the C = setting,
where each client is assigned data from two classes. Specifically, HCFL* generates 5 clusters in this setting, and we report the number of samples from each class assigned to clusters through 5.
(FFL. == 0.1) 0.1) Cluster 1
Cluster
Cluster 3
Cluster
Cluster 5
Class
4483
0
0
0
Class
0
4502
0
Class 3
0
0
4464
0
0
Class
0
4536
0
Class 5
0
4498
0
0
0
Class
0
0
4483
Class 7
0
0
4491
0
Class
4517
0
0
0
Class 9
0
0
4509
0
Class 10
0
0
0
4517
45


Reviews:
[{"review_id": "U9TFmtTzfN", "reviewer": "Reviewer_hh5r", "summary": "This manuscript proposes a holistic federated learning framework to enhance classification performance by grouping clients into clusters. The framework comprehensively integrates the hard and soft partitional clustering, and clustering with and without automatic cluster number determination. Comphrehensive theoretical and empirical evidence have been provided to illustrate the effectiveness of the proposed method. Moreover, the paper is generally well-written and easy to follow.", "questions": "See the weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A holistic FL framework incorporating different clustered FL methods for more comprehensive FL. The research problem is important, and this work contributes to fixing the shortcomings of the existing related works.\n2. Comprehensive experimental evaluation has been conducted to illustrate the effectiveness of the proposed method.\n3. The paper is well written with a clear demonstration of the motivations and problems for solving.", "weaknesses": "1. This work focuses on using clustering techniques to enhance the classification accuracy of FL. The difference between this type of research and the fully unsupervised federated clustering should be discussed to avoid potential misunderstandings.\n2. The efficiency issue is listed as one of the challenges in Section 4.1. But only the final number of clusters is reported accordingly. More discussions about the time and space complexity of this work, or even corresponding evaluation results are preferable.\n3. The source code is not opened in the current version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VK8sXrawkx", "reviewer": "Reviewer_NXiS", "summary": "The contribution formalizes a general pipeline/framework for clustered federated learning, on the one hand via a cost function, on the other hand via algorithmic choices (as it is not easy to include the objective of an optimum clustering number into costs in a meaningful way). This leads to an improved variant, where the data per client can belong to different clusters, and extensions of at the moment rare soft clustering schemes. The benefit is evaluated in both, comparative studies and ablation studies. A comparably long appendix addresses how to compute an EM scheme based on the costs, how to design data, what happens for linear parts, and some more insight.", "questions": "What is the overall objective of FL in a learning theoretical sense, i.e. how exactly would the generalization error which is targeted be expressed?\nWhat would be naturally occurring drift/shift in  such scenarios and how does this match with the shift modelled in experiments?\nWhat are results if the clustering itself is evaluated (eg having ground truth on the data distributions)?\nHow personalized are the models? And how does this scale with the required number of data as regards valid generalization?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The article addresses the important problem of efficient federated learning in the presence of data shift. \nIt provides a very detailed experimental analysis. \nIt also takes some effort to substantiate the observations with theoretical insight. \nMoreover, the authors promise to release the code open source.", "weaknesses": "The proposal left me a bit puzzled, as the specific contribution is somewhat unclear. On the one hand, the contribution promises a general framework/principle how to model FL with clustering. Here, the costs are rather obvious, as is the four-tier modeling, given the existing work how to model clustering; hence I am not sure what exactly is the contribution, is it the specific way of implementation, or specific guarantees which can be given? In how far is this modeling surprising/challenging and what exactly is the contribution, please specify (eg is it the better implementation? It would help if you could either provide specific benefits which arise from there which would not have been possible without this abstraction, or to provide examples where it is not obvious that the method falls under this common framework. \n\nThe improved version allows the individual assignment of data of one client. Here an according EM scheme is derived (a bit lengthy but straightforward), soft clustering is considered (which seems also straightforward given the existing work on soft clustering and its algorithms). Personally, I find the definition of a new way to measure distances w.r.t. drift most interesting, albeit very shortly presented. Here references to existing technologies how to deal with drift are missing (such as decomposition of sets of data with drift into some where the drift is homogeneous, e.g. moment trees and Kolmogorov trees). I suggest to have a closer look at the (exhaustive) literature on learning with drift in the incremental setup. \n\nI find the presentation suboptimal as the main part of the work reads almost trivial in wide parts, whereas some important insight seems to be hidden in the appendix. It would help if the main take aways of the appendix would be highlighted. Moreover (as already said before), please more clearly highlight why the holistic framework is not trivial, and benefitial with specific non-trivial results/examples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D8FnuNokQg", "reviewer": "Reviewer_KWQ4", "summary": "This paper introduces HCFL, a holistic framework for Clustered Federated Learning (CFL) that integrates existing methods. HCFL+ builds on this by addressing key challenges in HCFL, improving the effectiveness. Extensive experiments show the effectiveness of the proposed method.", "questions": "Please see weaknesses. Generally, I think this is a solid work. I will adjust the score based on author response.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem is well motivated and interesting. The algorithm proposed is novel, with solid theoretical analysis and extensive experimental studies. The paper is well structured and written.", "weaknesses": "For clustered based FL algorithm, there is a recent work [1] to conduct clustering based on the inferred label distributions. The authors are suggested to discuss about this clustering strategy.\n\nCould the authors provide more experiments on a wider range of beta (e.g., from 0.1 to 1.0), to show the effectiveness on different levels of data heterogeneity?\n\nAlso, there are setups such as C=2,C=3 in [1]. Such settings are also suggested to be studied. If the proposed algorithm can perform well across various data heterogeneity partitions, the paper can be stronger.\n\n[1] Diao, Yiqun, Qinbin Li, and Bingsheng He. \"Exploiting Label Skews in Federated Learning with Model Concatenation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 10. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)