# Increasing the Accuracy of Federated Learning on Non-IID Scenarios using Client Clustering

Lucas Airam Castro de Souza, Gustavo Franco Camilo, Miguel Elias Mitre Campista, Luís Henrique Maciel Kosmalski Costa

► To cite this version:

Lucas Airam Castro de Souza, Gustavo Franco Camilo, Miguel Elias Mitre Campista, Luís Henrique Maciel Kosmalski Costa. Increasing the Accuracy of Federated Learning on Non-IID Scenarios using Client Clustering. LATINCOM 2024 - IEEE Latin-American Conference on Communications, Nov 2024, Medellin, Colombia. hal-04739253

HAL Id: hal-04739253

https://hal.science/hal-04739253v1

Submitted on 16 Oct 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Increasing the Accuracy of Federated Learning on Non-IID Scenarios using Client Clustering

Lucas Airam C. de Souza<fnref n="1" />,², Gustavo F. Camilo<fnref n="1" />, Miguel Elias M. Campista<fnref n="1" />, and Luís Henrique M. K. Costa<fnref n="1" />

Universidade Federal do Rio de Janeiro (UFRJ), Brazil<fnref n="1" />

²École Polytechnique, INRIA Saclay

*Abstract*—Federated Learning (FL) depends on the data distribution among the clients. FL struggles in scenarios with heterogeneous data collected by different clients, which reduces the model's accuracy. Hence, we propose a client clustering system to mitigate the convergence obstacles of federated learning in non-Independent and Identically Distributed (IID) scenarios. Our proposal identifies the clusters using a testing neural network model sent to clients as a probe. After a few training epochs, each client returns to the server a vector containing the last-layer weights of the obtained model. Thus, we save communication and computation resources compared with other clustering proposals that apply iterative methods or use the entire neural network model. We also evaluate three clustering algorithms: K-Means, DBSCAN, and OPTICS. The DBSCAN algorithm demonstrates better results, correctly identifying the clients' clusters in IID and non-IID data distributions. Finally, the results show that our system has a better classification performance than FedAVG, increasing its accuracy by approximately 16% on non-IID scenarios.

*Keywords*—Federated Learning, Non-IID, Clustering.

## I. INTRODUCTION

Federated learning (FL) [1] has become popular among researchers and industry due to the possibility of training machine learning models while preserving users' data privacy [2], [3]. After the change in data processing regulations in several countries, for instance, the California Consumer Privacy Act (CCPA) in the USA or The importance of federated learning research and adoption increased with the introduction of the General Data Protection Regulation (GDPR) in Europe.

The algorithm most commonly used for creating federated models is FedAVG (Federated Averaging), which was introduced in Google's FL proposal. FedAVG uses a client-server architecture in which the global server shares a model, and the clients return it to the server after a few local training epochs. This procedure prevents the clients from sharing their data and increases the system's privacy. Nevertheless, the non-Independent and Identically Distributed (non-IID) distribution of the clients' training data reduces the model performance when trained with FedAVG [4]. Zhao et al. [5] analyze the behavior of data heterogeneity in neural network weights trained with SGD compared with the same model trained with FedAVG. During the training, the weights generated by these two approaches diverge, indicating a model performance degradation.

This paper proposes a client clustering system<fnref n="1" /> to increase the efficiency of federated learning in scenarios where clients have non-IID datasets. Clients are divided into clusters where the data is similar. The proposal uses clients' last-layer neural network weights as input to perform clustering. The last-layer neural network weights maintain statistical relationships with the clients' private data without revealing them [6]. Thus, we maintain the same privacy model as FedAVG. Each cluster trains a personalized model with independent hyperparameters and parameters, allowing high classification performance on specific tasks such as image processing for object detection.

We analyze different clustering models and show the advantage of adopting the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) [7] algorithm for client clustering. DBSCAN can identify homogeneous and heterogeneous data distributions from the client neural network weights, outperforming the Ordering Points to Identify the Clustering Structure (OPTICS) [8] and K-Means [9] algorithms. Furthermore, the results show that the proposed approach performance has FedAVG as a lower bound, outperforming it when clients have heterogeneous data. For non-IID datasets, the FedAVG algorithm converges with low accuracy after 100 global epochs. Nonetheless, the current proposal achieves better results across all clusters in 10 global epochs, providing high classification performance. We summarize our contributions as follows:

* **High classification performance with non-IID data:** We propose a system that achieves high accuracy results under non-IID data distributions. Our proposal obtains these results by assigning clients to a cluster in which the data distribution approximates the IID condition.

* **Low-communication requirement:** The proposal uses clients' last-layer neural network weights as the input to perform client clustering. Thus, we avoid the communication overhead of recursive clustering proposals. Also, we reduce the amount of data used to cluster clients compared to a proposal that sends a whole model.

* **Privacy-preservation:** Our proposal holds the same privacy model and assumptions as FedAVG, given that the system shares only the model weights.

<footnote n="1">¹https://github.com/GTA-UFRJ-team/Hierachical-Federated-Learning.</footnote>
* **Prototype:** We implement a proof-of-concept of our proposed system. We also present a practical evaluation of FedAVG under non-IID data distributions and compare it with our proposal.

This paper is organized as follows. Section II reviews the state-of-the-art performance optimization of federated learning systems when clients hold non-IID data. Section III details the proposed system. Section IV presents the development of the prototype system, the environment settings, and the analysis of the obtained results. Finally, Section V concludes this work and discusses future directions.

## II. RELATED WORK

The global convergence time in federated learning is attributed to three main factors: communication latency, processing capacity, and data representation. The first two are related to the user device, while the third depends on the data distribution collected and stored by the client. Nevertheless, FedAVG ignores these relevant factors by adopting a standard random client selection [1]. Therefore, several research groups present proposals to reduce the convergence time of FL and improve the performance of the generated model. The main proposed solutions include client selection and personalized models. On the one hand, client selection typically relies on device characteristics or clients' data. The goal is to decide the best subset of clients for the current training epoch, increasing the average model performance. On the other hand, model personalization means the creation of models specialized in clients' characteristics, consequently improving local model performance.

### A. Client Selection for Efficient Training

Luo et al. [10] and Lai et al. [11] propose client selection schemes that optimize the time to convergence in federated learning environments. The authors argue that selection based only on the representativeness of the data decreases the total number of epochs for model convergence. However, when clients with higher computational latency participate, the delay between epochs increases significantly. Nevertheless, a selection based only on processing capacity may incur more epochs for convergence if selected clients have irrelevant data. On the one hand, increasing the time between epochs implies a higher overall delay, as federated learning environments are generally synchronous and wait for all clients to respond or time out. On the other hand, selecting clients with higher computational power to reduce the time between epochs may incur more epochs for convergence if the selected clients have statistically insignificant data for the learning task. There is a trade-off between the number of epochs for model training and the total time for each epoch. Thus, the authors simultaneously consider the devices' characteristics and their data distribution relevance to reduce the convergence time of the global model. While Luo et al. propose a non-convex function regarding the expected number of epochs based on clients' previous updates, Lai et al. avoids the linear programming,

creating a selection function that evaluates clients' training losses.

Hierarchical Split Federated Learning (Hier-SFL) [12] is an FL framework that divides the neural network into three different levels, client-edge-cloud, to train the model. Thus, each one of the neural network parts has an associated loss function, which is adjusted separately. Hier-SFL objective is to reduce the communication overhead during the training. However, the distribution of data generated by clients is not taken into account, which can affect the performance of models in scenarios with non-IID distributions.

The selection of clients used by the above-mentioned approaches decreases the convergence time and increases the model's accuracy. The client selection, however, is insufficient for practical purposes in environments with heterogeneous data. Thus, an alternative is model personalization, which fine-tunes clients' models according to their local data consequently improving their performance.

### B. Personalized Models in Federated Learning

Dennis et al. [13] leverage data heterogeneity to produce an unsupervised federated learning algorithm. The clients train a clustering model with their local data and send the vectors that indicate the position of the clusters found in the sample space to an aggregation server. The server then runs a second clustering model based on clients' responses to identify the clusters in the environment. The greater the distance between client distributions, the more effective the detection of different clusters. The proposal is applied to both the selection of clients and the model personalization.

IFCA (Iterative Federated Clustering Algorithm) [14] forms client clusters to personalize models. In the proposal, clients are responsible for choosing their clusters. In addition, IFCA employs multi-task learning, which consists of sharing some neural network weights for clients who have data distributions with intersections but are in different clusters. Delegating the process of identifying clusters to the clients, however, the environment can be susceptible to malicious behavior and requires more computational cost from the clients' devices. Another disadvantage of this proposal is to assume that the number of clusters is previously known, which may be unfeasible and can either overestimate or underestimate the number of existing clusters.

CFL (Clustered Federated Learning) [15] is a proposal that recursively partitions federated learning clients into more homogeneous clusters to mitigate the problems generated by non-IID distributions. Partitioning occurs whenever the loss gradient vector exceeds a pre-established distance threshold. Nonetheless, clients' recursive partition leads to computational overhead on the aggregation server because it runs the procedure at each global training epoch.

Fraboni et al. propose a clustering sampling method for client selection [16]. The authors provide two approaches for clustering clients: client sample size and model similarity. Nevertheless, the first approach highly depends on the clients'
informing their exact sample size to the aggregation server for cluster definition. Therefore, this approach is susceptible to clients' malicious behaviors. On the other hand, the method uses all model weights in the model similarity proposal, which is not efficient.

We propose a clustering system, dividing FL clients into groups with an unsupervised clustering algorithm. Unlike previous client selection proposals that only consider optimizing a generic model across the entire federated learning system, our proposal aims at clustering clients with homogeneous data. Thus, each computed cluster holds IID data, benefiting the system participants individually. Furthermore, clustering clients can improve model convergence in the cluster and increase the performance for specific learning tasks. In addition, the proposal is agnostic to the clustering algorithm and eliminates the need for prior knowledge of the number of existing clusters.

### III. CLIENT CLUSTERING SYSTEM

This section presents the details of our proposal for accurate model creation with the clustering of clients. Firstly, we approach the client's clustering for the creation of cluster models, which is the name given for the models trained inside a cluster. Later, we discuss the composition of more generic models using cluster models.

We assume that messages exchanged between client and server are encrypted, and only the server knows the identification of clients. This prevents malicious clients from getting information about other clients and using it to degrade the model, assuming the server has not been compromised. Furthermore, it is assumed that the server is honest, running the FedAVG algorithm for aggregating the gradients correctly and combining the models on demand. Clients are assumed to have stationary data so that the distance between the individual loss gradient vectors of the selected clients and the average vector of the cluster for a specific epoch remains the same.

#### A. Proposed System

The proposed system operation for training federated models is composed of an initialization followed by two phases. During the system initialization, the server sent to all the clients in the system a testing neural network model. Nevertheless, it was experimentally verified that the final layers of the model retain more information about the data. Thus, one can reduce the communication cost of the system by sending only part of the deep neural network to the server. Clients locally calculate the neural network weights with their data and return the result to the server. After receiving the response from the clients, the server executes the first phase after initialization, which is to uniquely allocate clients into clusters. The server uses a clustering algorithm to determine the number of clusters and the clients of each cluster. As a result, clients with similar neural network weights are allocated to the same homogeneous cluster. Note that cluster creation

occurs without the need to access the client's samples, which is a requirement for keeping privacy. After cluster creation, the second phase starts, consisting of traditional federated learning training with FedAVG among clients in the same cluster. Therefore, in the proposed system, there are at least $k \ge 1$ specific models, where $k$ is the number of clusters generated by the server. Unlike traditional federated learning, the proposal calls the federated learning server a clustering and aggregation server. In addition to aggregating the results, the server is responsible for selecting and storing each client's cluster.

#### B. System initialization

The steps in Fig. 1 are performed to initialize the system. In the first step, the server shares the test model with an initial set of clients. The clients adjust the test model parameters with their local data and return only the vector from the last layer to the clustering and aggregation server. The server runs the clustering algorithm and saves the existing clusters. Once clusters are established, new clients are allocated to existing ones. The initial model of each cluster may differ from the test model, as the server can adjust the model hyperparameters according to the data from each cluster. Adding new clients to a cluster only affects training if the model has not converged. After model convergence, new clients only receive the final model from the cluster. Furthermore, the proposal predicts a fixed number of clusters after initialization, assuming that clients have stationary data. If all clients in a cluster fail simultaneously, the aggregation server stores information about the cluster, consisting of the training state and current model. This is important as the failed clients can be recovered, and new clients can be eventually allocated to the cluster. The system is agnostic to the clustering model used, allowing the administrator to select the clustering algorithm that best suits the requirements. The system is applied in a horizontal federated learning environment, where clients' sample data has the same features. Thus, the proposal assumes that the administrator has relevant information to adjust the clustering algorithms before defining the clusters, e.g., existing classes and information about clients' metadata, such as the number of features and learning tasks. Another hypothesis is that the clients' learning tasks are the same, e.g., image classification.

#### C. Dynamics of Client Participation in the System

After the system's initialization, other clients might arrive to participate in the FL training. Thus, we modify the procedure to integrate new clients into clusters. The new client who wants to participate in FL training requests the server to be included in a cluster. The server sends the test model to the client to obtain statistical information, and the client executes the same steps explained on system initialization. After receiving the response from the client, the server runs the clustering algorithm already trained at system startup to determine which cluster the new client belongs to. The cluster information is sent to the client and their identity is
<figure><img src="image_2.png" /><figcaption>Figure 1. Initialization of the proposed system. (1) The initial set of clients receives the test model. (2) Clients adjust the neural network weights and return the model's last layer vector to the server. (3) The clustering and aggregation server runs the clustering algorithm to get the clusters from the system. (4) The server associates new clients with clusters.</figcaption></figure>

added to the cluster list. Then, the server sends the current $z_t$ cluster's model to the client, allowing it to participate in the federated training. From this stage, federated training is traditionally executed in the $g_j$ cluster, in which the client was allocated. At each $E_q$ cluster's epoch, the server randomly selects a fraction of clients to adjust the parameters of the cluster model. If the client is selected, it calculates the new $w'$ weights and sends them to the server for aggregation. Finally, the server sends the updated model $z_{t+1}$ to the clients of the cluster $g_j$, and a new epoch, $E_{q+1}$, for the cluster, begins. The process is repeated until a stopping condition is reached, such as a model performance convergence criterion, e.g., accuracy within a low variation threshold in consecutive epochs, or a maximum number of global epochs executed $E_{qmax}$.

## IV. PROTOTYPE DEVELOPMENT AND EVALUATION

A prototype of the system was developed using Python v3.9.1 with flower v0.18.0 for the development of the federated learning environment and the scikit-learn library v1.0.1 to create the cluster models. The experiments were carried out on an Intel i7-8700 CPU 3.20 GHz server with 6 processing cores and 32 GB of RAM. The experimental results of the evaluation of the models present the average obtained among all the clients and a confidence interval of 95%. For training the models and evaluating the performance of the proposal, the dataset CIFAR-10 [17] is used. The dataset consists of 60,000 color images with a dimension of 32x32 belonging to 10 different classes. Each class has the same number of samples. In the experiments, the term traditional federated learning is equivalent to FedAVG without the clusters. We use the MobileNet [18] as our deep learning model architecture in the performance evaluation.

### A. Evaluation of Cluster Models

The first step of the system clusters the clients from the update vector sent to the server. Thus, Experiment I evaluates ways of clustering clients based on the information provided. In this experiment, the hyperparameters of three clustering

algorithms are analyzed: K-Means [9], OPTICS [8] and DBSCAN [7].

K-Means is a widely adopted algorithm for clustering tasks. Its main hyperparameter is the number of $k$ clusters existing in a given dataset. Nevertheless, defining this hyperparameter in an unknown dataset is an arduous task, as there is a prior need to know the number of existing clusters. Thus, there is a higher computational cost to use an iterative approach and estimate the best number of clusters. In this way, OPTICS appears as a more practical alternative, as it requires the minimum number of samples to form a cluster as a hyperparameter. Moreover, setting a minimum number of samples per large cluster can still generate heterogeneous clusters. Finally, an alternative to the previous algorithms is DBSCAN. Besides the ability to configure a minimum number of samples to create a cluster, DBSCAN uses the maximum distance between two samples as a hyperparameter. Therefore, it is possible to practically establish a direct relationship between the clients' replies to define if they have IID data.

The clustering results show that the algorithm that best divides clients when varying the scenarios was DBSCAN using a minimum distance of 0.0279 between client vectors. In the three scenarios, DBSCAN optimally clustered the clients, overcoming OPTICS in the IID scenario by identifying only one cluster. Furthermore, the three clustering models generated the same clusters in the non-IID scenarios. The next experiments evaluate the models generated after clustering the clients.

The three algorithms were evaluated with respect to the clustering of clients for different initialization hyperparameters. The K-Means algorithm establishes an optimal threshold for the performance evaluation experiments of the proposal because as the non-IID scenarios are created in a controlled way, the optimal number of clusters is known in advance. For the IID case, K-Means is configured to create clusters with at least two clients in order to guarantee the consistency of federated learning. The purpose of this configuration is to verify the impact on accuracy when the number of clusters is overestimated, dividing clients unnecessarily. So three is the number of clusters that maintain a minimum of two clients per cluster for K-Means.

A relevant evaluation is to determine the distance that allows DBSCAN to identify the clusters whose data are IID in the experimental scenarios. Thus, it is necessary to vary the distance hyperparameter and check how many clusters the model returns. The distance between the neural network weights of the clients strongly depends on the number of local epochs of the clients. Increasing the number of local epochs implies specializing the neural networks on the training data and, thus, producing vectors that are more distant from each other. On the other hand, clients that have datasets with similar distributions, when specializing their weights, produce vectors that remain close.
## B. Performance Evaluation of Specific Models

The second experiment evaluates the behavior of the proposal when all clients' datasets are IID. The goal is to evaluate the performance of the proposal when using different clustering algorithms and verify the impact on accuracy when there are more clusters than needed. The hyperparameter *k* of K-Means was initially set to 5 was set to 5. This hyperparameter is decremented until clusters with the minimum number of participants are established. So, through this approach, Experiment II uses *k* = 3. OPTICS was configured with a minimum number of clients equal to 2, and it also generated clusters unnecessarily. On the other hand, DBSCAN, when analyzing the vectors of clients, correctly identified only one cluster, once the dataset is IID.

<figure><img src="image_3.png" /><figcaption>Figure 2. Accuracy evolution as a function of global epoch for customers with IID datasets. The red line represents the traditional approach of federated learning, while the blue and the green lines represent the proposal of this paper with clustering through DBSCAN and OPTICS, respectively.</figcaption></figure>

The result of Fig. 2 indicates the average accuracy behavior in all 10 clients using the traditional approach and the current proposal. To do this, the 6,000 samples of a class are split among ten clients to create balanced datasets. The proposal, even using K-Means and OPTICS for clustering, has an accuracy value close to the traditional case, despite dividing clients into clusters unnecessarily in this experiment. Thus, it can be concluded that there is no significant loss of performance even when clients have IID datasets and are divided into clusters. Furthermore, the experiment demonstrates the ability of the DBSCAN algorithm to identify that creating clusters is unnecessary in this case.

In the third experiment, the data distributions on the clients are non-IID, with only two classes present in their datasets. The goal of this experiment is to compare the performance of the model generated through traditional federated learning with the one created by the current proposal, in which the data distribution is non-IID. Fig. 3 displays the experimental mean result among all clusters, illustrated in the figure as the blue line, and the traditional result, represented by the red

<figure><img src="image_4.png" /><figcaption>Figure 3. Evolution of the test accuracy as a function of global epoch for clients with datasets that contain only two classes. The red line represents the traditional approach, while the blue line represents the proposal of this paper.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 4. Test accuracy evolution as a function of overall epoch for clients with datasets containing only five classes. The red line represents the traditional approach, while the blue line represents the proposal in this paper.</figcaption></figure>

line. Traditional federated learning is affected by the level of data heterogeneity, while the current proposal allows for high classification performance. We observe that our proposal increases the accuracy by around 16%, showing how effective is the approach in non-IID scenarios.

The fourth experiment again evaluates the performance of the proposal for non-IID datasets. The difference from the third experiment is that we increased the number of classes of each client to five classes. The results shown in Fig. 4 show similar behavior to the third experiment for both cases, but with worse performance. This fact can be explained by the problem difficulty, which was increased to a multiclass classification.
The experiment also analyzes the individual behavior of the clusters' models. The results show that the first cluster has an accuracy of (56 ± 3)%, while the second has an accuracy of (63 ± 3)%. Although there is a cluster with higher performance, even the low performance of the first cluster is higher than that obtained when using the traditional proposal. Thus, it can be concluded that the proposal successfully mitigated the effects of heterogeneity in the scenario with five classes per client.

<figure><img src="image_6.png" /><figcaption>Figure 5. Evolution of the test accuracy as a function of global epoch for clients with datasets that contain only two classes. The clients' datasets are generated with the LDA using α = 0.5. The red line represents the traditional approach, while the blue line represents our current proposal.</figcaption></figure>

The fifth experiment evaluates the clustering approach in a non-IID scenario where the clients' datasets are not limited to samples of a subset of classes, thus creating a more realistic scenario. We build the datasets of this experiment through the Label-based Dirichlet Partition (LDA), which generates the data based on the number of clients and non-IID degree. Figure 5 shows the results of Experiment V. Our approach increases by more than 14% the accuracy compared with traditional federated learning.

V. CONCLUSION

We proposed a system to increase the performance of fed-
erated learning when clients have non-IID data distributions.
The DBSCAN clustering algorithm, which requires only the
minimum distance between samples as a hyperparameter,
provided a high ability to detect the existing clusters even
when clients have IID datasets. Besides, the proposed system
outperformed traditional federated learning when the clients'
datasets are non-IID in all the evaluated scenarios. The
accuracy showed an improvement of approximately 16% in
the best case. The proposal will be extended to non-stationary
scenarios in future work to deal with the concept drift of the
data, where the aggregation server can change clusters on
demand.

VI. ACKNOWLEDGMENT

This study was financed in part by the Coordenação de
Aperfeiçoamento de Pessoal de Nível Superior – Brasil
(CAPES) – Finance Code 001. This paper was also funded
by CNPq, FAPERJ, and Fundação de Amparo à Pesquisa
do Estado de São Paulo (FAPESP), Grants 2015/24494-8,
2018/23292-0, 2015/24485-9, 2014/50937-1.

REFERENCES

[1] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, "Communication-efficient Learning of Deep Networks from Decentralized Data," *Artificial Intelligence and Statistics*, pp. 1273–1282, 2017.

[2] L. A. C. de Souza, G. A. F. Rebello, G. F. Camilo, L. C. Guimarães, and O. C. M. Duarte, "DFedForest: Decentralized Federated Forest," in *International Conference on Blockchain*. IEEE, 2020, pp. 90–97.

[3] H. N. C. Neto, J. Hribar, I. Dusparic, N. C. Fernandes, and D. M. Mattos, "FedSBS: Federated-Learning Participant-Selection Method for Intrusion Detection Systems," *Computer Networks*, vol. 244, p. 110351, 2024.

[4] X. Ma, J. Zhu, Z. Lin, S. Chen, and Y. Qin, "A State-of-the-Art Survey on Solving Non-IID Data in Federated Learning," *Future Generation Computer Systems*, vol. 135, pp. 244–258, 2022.

[5] Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V. Chandra, "Federated Learning with Non-IID Data," *arXiv preprint arXiv:1806.00582*, 2018.

[6] H. Wang, Z. Kaplan, D. Niu, and B. Li, "Optimizing Federated Learning on Non-IID Data with Reinforcement Learning," in *IEEE INFOCOM*, 2020, pp. 1698–1707.

[7] M. Ester, H.-P. Kriegel, J. Sander, X. Xu *et al.*, "A Density-based Algorithm for Discovering Clusters in Large Spatial Databases with Noise," in *KDD*, 1996, pp. 226–231.

[8] M. Ankerst, M. M. Breunig, H.-P. Kriegel, and J. Sander, "OPTICS: Ordering Points to Identify the Clustering Structure," *ACM Sigmod record*, pp. 49–60, 1999.

[9] J. MacQueen, "Some Methods for Classification and Analysis of Multi-variate Observations," in *Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability*, 1967, pp. 281–297.

[10] B. Luo, W. Xiao, S. Wang, J. Huang, and L. Tassiulas, "Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Sampling," in *IEEE INFOCOM*, 2022, pp. 1739–1748.

[11] F. Lai, X. Zhu, H. V. Madhyastha, and M. Chowdhury, "Oort: Efficient Federated Learning via Guided Participant Selection," in *USENIX OSDI*, 2021, pp. 19–35.

[12] T. Qin, G. Cheng, Y. Wei, and Z. Yao, "Hier-SFL: Client-Edge-Cloud Collaborative Traffic Classification Framework based on Hierarchical Federated Split Learning," *Future Generation Computer Systems*, 2023.

[13] D. K. Dennis, T. Li, and V. Smith, "Heterogeneity for the Win: One-Shot Federated Clustering," *arXiv preprint arXiv:2103.00697*, 2021.

[14] A. Ghosh, J. Chung, D. Yin, and K. Ramchandran, "An Efficient Framework for Clustered Federated Learning," *arXiv preprint arXiv:2006.04088*, 2020.

[15] F. Sattler, K.-R. Müller, and W. Samek, "Clustered Federated Learning: Model-Agnostic Distributed Multitask Optimization under Privacy Constraints," *IEEE Transactions on Neural Networks and Learning Systems*, 2020.

[16] Y. Fraboni, R. Vidal, L. Kameni, and M. Lorenzi, "Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated Learning," in *International Conference on Machine Learning*. PMLR, 2021, pp. 3407–3416.

[17] A. Krizhevsky et al., "Learning Multiple Layers of Features from Tiny Images," 2009.

[18] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications," *arXiv preprint arXiv:1704.04861*, 2017.