Title: Heterogeneity-Guided Client Sampling: Towards Fast and Efficient Non-IID Federated Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Federated Learning

Client Selection

Non-IID data
Abstract: Statistical heterogeneity of data present at client devices in a federated learning (FL) system renders the training of a global model in such systems difficult. Particularly challenging are the settings where due to communication resource constraints only a small fraction of clients can participate in any given round of FL. Recent approaches to training a global model in FL systems with non-IID data have focused on developing client selection methods that aim to sample clients with more informative updates of the model. However, existing client selection techniques either introduce significant computation overhead or perform well only in the scenarios where clients have data with similar heterogeneity profiles. In this paper, we propose HiCS-FL (Federated Learning via Hierarchical Clustered Sampling), a novel client selection method in which the server estimates statistical heterogeneity of a client's data using the client’s update of the network’s output layer and relies on this information to cluster and sample the clients. We analyze the ability of the proposed techniques to compare heterogeneity of different datasets, and characterize convergence of the training process that deploys the introduced client selection method. Extensive experimental results demonstrate that in non-IID settings HiCS-FL achieves faster convergence than state-of-the-art FL client selection schemes. Notably, HiCS-FL drastically reduces computation cost compared to existing selection schemes and is adaptable to different heterogeneity scenarios.

Heterogeneity-Guided Client Sampling: Towards Fast and Efficient Non-IID Federated Learning

Abstract

Statistical heterogeneity of data present at client devices in a federated learning (FL) system renders the training of a global model in such systems difficult. Particularly challenging are the settings where due to communication resource constraints only a small fraction of clients can participate in any given round of FL. Recent approaches to training a global model in FL systems with non-IID data have focused on developing client selection methods that aim to sample clients with more informative updates of the model. However, existing client selection techniques either introduce significant computation overhead or perform well only in the scenarios where clients have data with similar heterogeneity profiles. In this paper, we propose HiCS-FL (Federated Learning via Hierarchical Clustered Sampling), a novel client selection method in which the server estimates statistical heterogeneity of a client's data using the client's update of the network's output layer and relies on this information to cluster and sample the clients. We analyze the ability of the proposed techniques to compare heterogeneity of different datasets, and characterize convergence of the training process that deploys the introduced client selection method. Extensive experimental results demonstrate that in non-IID settings HiCS-FL achieves faster convergence than state-of-the-art FL client selection schemes. Notably, HiCS-FL drastically reduces computation cost compared to existing selection schemes and is adaptable to different heterogeneity scenarios.

1 Introduction

The federated learning (FL) framework enables privacy-preserving collaborative training of machine learning (ML) models across a number of devices (clients) by avoiding the need to collect private data stored at those devices. The participating clients typically experience both the system as well as statistical heterogeneity [18]. The former describes settings where client devices have varying degree of computational resources, communication bandwidth and fault tolerance, while the latter refers to the fact that the data owned by the clients may be drawn from different distributions. In this paper, we focus on FL under statistical heterogeneity and leave studies of system heterogeneity to future work.

An early FL method, FedAvg [21], performs well in the settings where the devices train on independent and identically distributed (IID) data. However, compared to the IID scenario, training on non-IID data is detrimental to the convergence speed, variance and accuracy of the learned model. This has motivated numerous studies aiming to reduce the variance and improve convergence of FL on non-IID data [6, 9, 14, 17, 19, 30].

On another note, constraints on communication resources and therefore on the number of clients that may participate in training additionally complicate implementation of FL schemes. It would be particularly unrealistic to require regular contributions to training from all the clients in a largescale cross-device FL system. Instead, only a fraction of clients participate in any given training round; unfortunately, this further aggravates detrimental effects of statistical heterogeneity. Selecting informative clients in non-IID FL settings is an open problem that has received considerable attention

from the research community [8, 11, 12]. Since privacy concerns typically prohibit clients from sharing their local data label distributions, existing studies focus on estimating informativeness of a client's update by analyzing the update itself. This motivated a family of methods that rely on the norms of local updates to assign probabilities of sampling the clients [7, 23]. Aiming to enable efficient use of the available communication and computation resources, another set of methods groups clients with similar data distributions into clusters based on the similarity between clients' model updates [2, 11]. Across the board, the existing methods still struggle to deliver desired performance in an efficient manner and cannot distinguish clients with balanced data from the clients with imbalanced data.

In this paper, we consider training a neural network model for classification tasks via federated learning and propose a novel adaptive clustering-based sampling method for identifying and selecting informative clients. The method, referred to as Federated Learning via Hierarchical Clustered Sampling (HiCS-FL), relies on the updates of the (fully connected) output layer in the network to determine how diverse is the clients' data and, based on that, decide which clients to sample. In particular, HiCS-FL enables heterogeneity-guided client selection by utilizing general properties of the gradients of the output layer to distinguish between clients with balanced from those with imbalanced data. Unlike the Clustered Sampling strategies [11] where the clusters of clients are sampled uniformly, HiCS-FL allocates different probabilities (importance) to the clusters according to their average estimated data heterogeneity. Numerous experiments conducted on vision datasets FMNIST, CIFAR10, Mini-ImageNet and a NLP dataset THUC news demonstrate that HiCS-FL achieves significantly faster training convergence and lower variance than the competing methods. Finally, we conduct convergence analysis of HiCS-FL and discuss implications of the results.

In summary, the contributions of the paper include: (1) Analytical characterization of the correlation between local updates of the output layer and the FL clients' data label distribution, along with an efficient method for estimating data heterogeneity; (2) a novel clustering-based algorithm for heterogeneity-guided client selection; (3) extensive simulation results demonstrating HiCS-FL provides significant improvement in terms of convergence speed and variance over competing approaches; and (4) theoretical analysis of the proposed schemes.

2 Background and Related Work

Assume the cross-device federated learning setting with N clients, where client k owns private local dataset B k with |B k | samples. The plain vanilla FL considers the objective

where θ denotes parameters of the global model, F k ( θ ) is the loss (empirical risk) of model θ on B k , and p k denotes the weight assigned to client k , ∑ N k =1 p k = 1 . In FedAvg, the weights are set to p k = |B k | / ∑ N i =1 |B i | . In training round t , the server collects clients' model updates θ t k formed by training on local data and aggregates them to update global model as θ t +1 = ∑ N k =1 p k θ t k .

When an FL system operates under resource constraints, typically only K ≪ N clients are selected to participate in any given round of training; denote the set of clients selected in round t by S t . In departure from FedAvg, FedProx [19] proposes an alternative strategy for sampling clients based on a multinomial distribution where the probability of selecting a client is proportional to the size of its local dataset; the global model is then formed as the average of the collected local models θ t +1 = 1 K ∑ k ∈S t θ t k . This sampling strategy is unbiased since the the updated global model is on expectation equal to the one obtained by the framework with full client participation as Eq.1.

AFL [12] is the first study to utilize local validation loss as a value function for computing client sampling probabilities; Power-of-Choice [8] takes a step further to propose a greedy approach to sampling clients with the largest local loss. Both of these methods require all clients to compute the local validation loss, which is often unrealistic. To address this problem, FedCor [28] models the local loss by a Gaussian Process (GP), estimates the GP parameters from experiments, and uses the GP model to predict clients' local losses without requiring them to perform validation. In [7], Optimal Client Sampling scheme aiming to minimize the variance of local updates by assigning sampling probabilities proportional to the Euclidean norm of the updates is proposed. The study in [23] models the progression of model's weights by an Ornstein-Uhlenbeck process and proposes a strategy, optimal under that assumption, for selecting clients with significant weight updates.

The clustering-based sampling method proposed in [11] uses cosine similarity [24] to group together clients with similar local updates, and proceeds to sample one client per cluster in attempt to avoid redundant gradient information. DivFL [2] follows the same principle of identifying representative clients but does so by constructing a submodular set and greedily selecting diverse clients. Both of these techniques are computationally expensive due to the high dimension of the gradients that they need to process.

In general, the overviewed methods either: (1) select diverse clients to reduce redundant information; or (2) select clients with a perceived significant contributions to the global model (high loss, large update or low class-imbalance). Efficient and effective client selection in FL remains an open challenge, motivating the heterogeneity-guided adaptive client selection method presented next.

3 HiCS-FL: Federated Learning via Hierarchical Clustered Sampling

Existing client sampling methods including Clustered Sampling [11] and DivFL [2] aim to select clients such that the resulting model update is an unbiased estimate of the true update (i.e., the update in the case of full client participation) while minimizing the variance

clusters based on representative gradients [24], and randomly selects one client from each cluster to contribute to the global model update. Such an approach unfortunately fails to differentiate between model updates formed on data with balanced and those formed on data with imbalanced label distributions - indeed, in either case the updates are treated as being equally important. However, a number of studies in centralized learning has shown that class-imbalanced datasets have significant detrimental effect on the performance of learning classification tasks [3, 4, 26]. This intuition carries over to the FL settings where one expects the updates from clients training on relatively more balanced local data to have a more beneficial impact on the performance of the system. The Federated Learning via Hierarchical Clustered Sampling (HiCS-FL) framework described in this section adapts to the clients' data heterogeneity in the following way: if the levels of heterogeneity (as quantified by the entropy of data label distribution) vary from one cluster to another, HiCS-FL is more likely to sample clusters containing clients with more balanced data; if the clients grouped in different clusters have similar heterogeneity levels, HiCS-FL is more likely to select diverse clients (i.e., sample uniformly across clusters, thus reducing to the conventional clustered sampling strategy).

3.1 Class-imbalance Causes Objective Drift

A number of studies explored detrimental effects of non-IID training data on the performance of a global model learned via FedAvg. An example is SCAFFOLD [14] which demonstrates objective drift in non-IID FL manifested through large differences between local models θ ∗ k trained on substantially different data distributions. The drift is due to FedAvg updating the global model in the direction of the weighted average of local optimal models, which is not necessarily leading towards the optimal global model θ ∗ . The optimal model θ ∗ , in principle obtained by solving optimization in Eq. 1, achieves minimal empirical error on the data with uniform label distribution and is intuitively closer to the local optimal models trained on balanced data. Recent work [36] empirically verified this conjecture through extensive experiments. Let ∇ F ( θ t ) denote the gradient of F ( θ t ) given the global model θ t at round t ; the difference between ∇ F ( θ t ) and the local gradient ∇ F k ( θ t ) computed on client k 's data is typically assumed to be bounded [7, 11, 31]. To proceed, we formalize the assumption about the relationship between gradients and data label distributions.

Assumption 3.1 (Bounded Dissimilarity.) Gradient ∇ F k ( θ t ) of the k -th local model at global round t is such that

where D ( k ) is the data label distribution of client k , D 0 denotes uniform distribution, H ( · ) is Shannon's entropy of a stochastic vector, and β > 0 , κ > ρ > 0 .

The assumption commonly encountered in literature is recovered by setting the right-hand side of (3) to σ 2 m = max k σ 2 k . Intuitively, if the data label distribution of client k is highly imbalanced (i.e., H ( D ( k ) ) is small), the local gradient ∇ F k ( θ t ) may significantly differ from the global gradient ∇ F ( θ t ) (as reflected by the bound above). Analytically, connecting the gradients to the local data label distributions allows one to characterize the effects of client selection on the variance and the rate of convergence. The results of extensive experiments that empirically verify the above assumption are reported in Appendix A.2.

3.2 Estimating Client's Data Heterogeneity

If the server were given access to clients' data label distributions, selecting clients would be relatively straightforward [32]. However, privacy concerns typically discourage clients from sharing such information. Previous studies have explored the use of multi-arm bandits for inferring clients' data heterogeneity from local model parameters, or have utilized a validation dataset at the server to accomplish the same [27, 34, 36]. In this section, we demonstrate how to efficiently and accurately estimate data heterogeneity using local updates of the output layer of a neural network in a classification task. Figure 1 illustrates the last two layers in a typical neural network. The prediction q ∈ R C is computed by forming a weighted average of signals z ∈ R L utilizing the weight matrix W ∈ R C × L and bias b ∈ R C .

3.2.1 Local updates of the output layer

An empirical investigation of the gradients of the output layer's weights while training with FedAvg using mini-batch stochastic gradient descent (SGD) as an optimizer is reported in [5, 29]. There, the focus is on detecting the presence of specific labels in a batch rather than on exploring the effects of class imbalance on the local update. To pursue the latter, we focus on the correlation between local updates of the output layer's bias and the client's data label distribution; we start by analyzing the training via FedAvg that employs SGD and then extend the results to other FL algorithms that utilize optimizers beyond SGD. We assume that the model is trained by minimizing the cross-entropy (CE) loss over one-hot labels - a widely used multi-class classification framework. The gradient is computed by averaging contributions of the samples in mini-batches, i.e., ∇ b L ce = 1 Bl ∑ l j =1 ∑ B n =1 ∇ b L ( j,n ) ce ( x ( j,n ) , y ( j,n ) ) , where B denotes the batch size, l is the number of minibatches, x ( j,n ) is the n -th point in the j -th mini-batch and y ( j,n ) ∈ [ C ] is its label. The contribution of x ( j,n ) to the i -th component of the gradient of the output layer's bias b can be found as (details provided in Appendix A.3)

̸

̸

where I {·} is an indicator, q ( j,n ) = W · z ( j,n ) + b is the output logit for signals z ( j,n ) ∈ R L corresponding to training point ( x ( j,n ) , y ( j,n ) ) (see Fig. 1), and where C denotes the number of classes. We make the following observations: (1) the sign of y ( j,n ) -th component of ∇ b L ( j,n ) ce is opposite of the sign of other components; and (2) the y ( j,n ) -th component of ∇ b L ( j,n ) ce is equal in magnitude to all the other components combined. Note that the above two observations are standard for neural networks using CE loss for supervised multi-class classification tasks.

In each global round t of FedAvg, the selected client k starts from the global model θ t and proceeds to compute local update in R local epochs employing an SGD optimizer with learning rate η . According to Eq. 4, the i -th component of local update ∆ b ( k ) is computed as

where ∇ b i L ( j,n,r ) ce denotes the gradient of bias at local epoch r . Note that the local update of client k , ∆ b ( k ) , is dependent on the label distribution of client k 's data, D ( k ) = [ D ( k ) 1 , . . . , D ( k ) C ] T and the label-specific components of q ( j,n ) which change during training. We proceed by relating expected local updates to the label distributions; for convenience, we first introduce the following definition.

Definition 3.2 Let B -i be the subset of local data B that excludes points with label i . Let s -i ( x ) ∈ [0 , 1] C be the softmax output of a trained neural network for a training point ( x , y ) ∈ B -i . The i -th component of s -i ( x ) , s -i i ( x ) , indicates the level of confidence in (erroneously) classifying x as having label i . For convenience, we define E i = E ( x ,y ) ∼B -i [ s -i i ( x ) ] , ∀ i ∈ [ C ] .

In an untrained/initialized neural network where classifier makes random predictions, E i = 1 /C ; as training proceeds, E i decreases. By taking expectation and simplifying, we obtain (details provided in Appendix A.4)

where D ( k ) i denotes the true fraction of samples with label i in client k 's data, ∑ C i =1 D ( k ) i = 1 .

3.2.2 Estimating local data heterogeneity

We quantify the heterogeneity of clients' data by an entropy-like measure defined below. Let D ( k ) denote the label distribution of client k 's data; its entropy is defined as H ( D ( k ) ) ≜ -∑ C i =1 D ( k ) i ln D ( k ) i ≤ ln C . Recall that more balanced data results in higher entropy, and that H ( D ( k ) ) takes the maximal value when D ( k ) is uniform. The server does not know D ( k ) and therefore cannot compute H ( D ( k ) ) directly. We define

here T is a scaling hyper-parameter (so-called temperature ). Note that even though we can compute ˆ H ( D ( k ) ) to characterize heterogeneity, D ( k ) i and E i remain unknown to the server (details in A.5).

Theorem 3.3 Consider an FL system in which clients collaboratively train a model for a classification task over C classes. Let D ( u ) and D ( k ) denote data label distributions of an arbitrary pair of clients u and k , respectively. Moreover, let U denote the uniform distribution, and let η and R be the learning rate and the number of local epochs, respectively. Then

As an illustration, consider the scenario where client u has a balanced dataset while the dataset of client k is imbalanced; then ∥D ( k ) -U ∥ 2 2 is relatively large compared to ∥D ( u ) -U ∥ ∞ . The bound in (8) also depends on δ , which is reflective of how misleading on average can a class be; small δ suggests that no class is universally misleading. As shown in Appendix A.4, during training δ gradually decreases to 0 as ∑ C i =1 E i decreases to 0 .

3.2.3 Generalizing beyond FedAvg and SGD

The proposed method for estimating clients' data heterogeneity relies on the properties of the gradient for the cross-entropy loss objective discussed in Section 3.2.1. However, for FL algorithms other than FedAvg, such as FedProx [19], FedDyn [1] and Moon [16], which add regularization to combat overfitting, the aforementioned properties may not hold. Moreover, optimization algorithms using second-order momentum such as Adam [15] deploy update rules different from SGD, making the local updates no longer proportional to the gradients. Nevertheless, HiCS-FL remains capable of distinguishing between clients with imbalanced and balanced data, which will be demonstrated in our experiments. Further theoretical discussion of various FL algorithms with optimizers beyond SGD are in appendix A.8 and A.9.

3.3 Heterogeneity-guided Clustering

Clustered Sampling [11] uses cosine similarity [24] between gradients to quantify proximity between clients' data distributions and subsequently group them into clusters. However, cosine similarity

Algorithm 1 HiCS-FL

Input:

Datasets distributed across N clients, the number of clients to sample K , total global rounds T .

1: Initialize updates of bias ∆ b ( k ) ← 0 ∀ k ∈ [ N ] , global model θ t ← θ 1 , S 0 = [ N ] .

2: for t = 1 , . . . , T do

3: if t ≤ ⌈ N/K ⌉ then

4: S t ← randomly sample min( K, | S 0 | ) clients from S 0 , update S 0 ← S 0 -S t ;

5: else

6: estimate ˆ H t ( D ( k ) ) and cluster N clients into M groups based on Eq. 9;

7: S t ←∅ ;

8: while |S t | < K do

9: sample group G t m according to π t ; 10: sample client k in G t m based on ˜ p m ; 11: S t ←S t ∪ k ;

12:

13:

14:

15:

16:

17:

18:

end while end if

for

k

∈ S

θ

t

k

t

do

LocalUpdate end for

θ

∆

b

(

k

19:

end for

Output:

The global model θ T +1

cannot help distinguish between clients with balanced and those with imbalanced datasets. Motivated by this observation, we introduce a new distance measure that incorporates estimates of data heterogeneity ˆ H ( D ( k ) ) . In particular, the proposed measure of distance between clients u and k that we use to form clusters is defined as

where the first term is akin to the cosine similarity used by CS with the major difference that we compute it using only the updates of the bias in the output layer, which is much more efficient than using the weights of the entire network; λ is a pre-defined hyper-parameter (set to 10 in all our experiments). For large λ , the second term dominates when there are clients with different levels of statistical heterogeneity; this allows emergence of clusters that group together clients with balanced datasets. The second term is small when clients have data with similar levels of statistical heterogeneity; in that case, the distance measure reduces to the conventional cosine similarity.

3.4 Hierarchical Clustered Sampling

To select K out of N clients in an FL system, we first organize the clients into M ≥ K groups via the proposed Hierarchical Clustered Sampling (HiCS) technique. In particular, during the first ⌈ N/K ⌉ training rounds the server randomly (without replacement) selects clients and collects from them local updates of ∆ b ( k ) ; the server then estimates ˆ H t ( D ( k ) ) for each selected client k and clusters the clients using the distance measure defined in Eq. 9. Let G t 1 , . . . , G t M denote the resulting M clusters at global round t , and let ¯ H t m = 1 | G m | ∑ k ∈ G m ˆ H t ( D ( k ) ) characterize the average heterogeneity of clients in cluster m , m ∈ [ M ] . Having computed ¯ H t m , HiCS selects a cluster according to the probability vector π t , and then from the selected cluster selects a client according to the probability vector ˜ p t m . The two probability vectors π t and ˜ p t m are defined as

where k 1 , . . . , k | G m | are the indices of clients in cluster G m , γ t = γ 0 (1 -t T ) denotes an annealing hyper-parameter, and T is the number of global rounds. The annealing parameter is scheduled such that at first it promotes sampling clients with balanced data, thus accelerating and stabilizing the convergence of the global model. To avoid overfitting potentially caused by repeatedly selecting a small subset of clients, the annealing parameter is gradually reduced to γ t ≈ 0 , when the server samples the clusters uniformly. The described procedure is formalized as Algorithm 1.

3.5 Convergence Analysis

Adopting the standard assumptions of smoothness, unbiased gradients and bounded variance [7], the following theorem holds for FedAvg with SGD optimizer.

t

+1

(

1

∑

t

K

)

θ

θ

t

)

;

∆

k

∈

(

b

k

S

)

,

t

∀

k

k

∈ S

t

,

;

∆

(

b

k

)

∈

θ

t

k

-

θ

t

Theorem 3.4 Assume F k ( · ) is L -smooth for all k ∈ [ N ] . Let θ t denote parameters of the global model and let F ( · ) be defined as in Eq. 1. Furthermore, assume the stochastic gradient estimator g k ( θ t ) is unbiased and the variance is bounded such that E ∥ g k ( θ t ) -∇ F k ( θ t ) ∥ 2 ≤ σ 2 . Let η and R be the learning rate and the number of local epochs, respectively. If the learning rate is such that η ≤ 1 8 LR , R ≥ 2 , then

where A 1 , A 2 , Φ are positive constants, and ω t k is the probability of sampling client k at round t .

Note that only the second term in the parenthesis on the right-hand side of the bound in Theorem 3.4 is related to the sampling method Π . Under Assumption 3.1,

If the server samples clients with weights proportional to p k , the statistical heterogeneity of the entire FL system may be characterized by H S = ∑ N k =1 p k exp( β ( H ( D ( k ) )) exp( β ( H ( D 0 )) ρ . If all clients have classimbalanced data, H S is small and thus random sampling leads to unsatisfactory convergence rate (as indicated by Theorem 3.4). On the other hand, since the clients sharing a cluster have similar data entropy, the proposed HiCS-FL leads to ω t k = p k exp( γ t ˆ H t ( D ( k ) )) ∑ N j =1 p j exp( γ t ˆ H t ( D ( j ) )) . When training starts, H Π is large because the server tends to sample clients with higher p k exp( γ t H ( D ( k ) )) ; as γ t decreases, H Π eventually approaches H S . Further details and the proof of the theorem are in Appendix A.7.

4 Experiments

Setup. We evaluate the proposed HiCS-FL algorithm on four benchmark datasets (FMNIST, CIFAR10, Mini-ImageNet and THUC news) using different model architectures. We use four baselines: random sampling, pow-d [8], clustered sampling (CS) [11], DivFL [2] and FedCor [28]. To generate non-IID data partitions, we follow the strategy in [35], utilizing Dirichlet distribution with different concentration parameters α which controls the level of heterogeneity (smaller α leads to generating less balanced data). In a departure from previous works we utilize several different α to generate data partitions for a single experiment, leading to a realistic scenario of varied data heterogeneity across different clients. To quantify the performance of the tested methods, we use two metrics: (1) average training loss, and (2) test accuracy of the learned global model. For better visualization, data points in the results are smoothened by a Savitzky-Golay filter with window length 13 and the polynomial order set to 3 . Further details of the experimental setting and a visualization of data partitions are in Appendix A.1 and A.10.

4.1 Comparison on Test Accuracy and Training Loss

FMNIST. We run FedAvg with SGD to train a global model which has CNN architecture in an FL system with 50 clients, where 10% of clients are selected to participate in each round of training. The data partitions are generated using one of 3 sets of the concentration parameter α values: (1) { 0 . 001 , 0 . 002 , 0 . 005 , 0 . 01 , 0 . 5 } ; (2) { 0 . 001 , 0 . 002 , 0 . 005 , 0 . 01 , 0 . 2 } ; (3) { 0 . 001 } . These are used to generate clients' data so as to emulate the following scenarios: (1) 80% of clients have severely imbalanced data while the remaining 20% have balanced data; (2) 80% clients have severely imbalanced data while the remaining 20% have mildly imbalanced data; (3) all clients have severely imbalanced data. Note that H M monotonically decreases as we go through settings (1) to (3). For a fair comparison, pow-d and DivFL are deployed with their ideal settings where the server requires all clients to precompute in each round a metric that is then used for client selection. Figure 2 shows that HiCS-FL outperforms other methods across different settings, exhibiting the fastest convergence rates and the least amount of variance. Particularly significant is the acceleration of convergence in setting (1) where 20% of the participating clients have balanced data. Figure 3 shows that HiCS-FL is helping achieve significant reduction of training variations (as expected, see Section 3.5) as evident by a smooth loss trajectory.

01

(a) FMNIST (1)

(b) FMNIST (2)

(c) FMNIST (3)

(d) CIFAR10 (1)

(e) CIFAR10 (2)

(f) CIFAR10 (3)

test accuracy forthe globalmodel(%)

pe

80

test accuracy for the global model(%)

=

pow-d, d = 50

DivFL (ideal)

Clustered Sampling

82

pow-d, d = 50

DivFL (ideal)

Clustered Sampling

2

3

pow-d, d = 50

DivFL (ideal)

Clustered Sampling

HiCS-FL (ours)

FedCor

HiCS-FL (ours)

FedCor

10

HiCS-FL (ours)

FedCor

0

20

40

60

80100120140160180 200

global rounds

20

40

60

80100120140160180200

global rounds

0

20

40

80100120

global rounds

000810910

60

%peqaee

50

50

40

40

2

pow-d, d = 50

Clustered Sampling

20

pow-d, d = 50

Clustered Sampling

pow-d.d50

Clustered Sampling

DivFL (ideal)

DivFL (ideal)

DivFL (ideal)

10

HiCS-FL (ours)

FedCor

10

HiCS-FL (ours)

FedCor

10

HiCS-FL (ours)

FedCor

0

50

100150

200250300

global rounds

350  400 450 500

50

100150 200 250 300

global rounds

350 400 450 500

0

50

100150200250300

global rounds

350 400 450 500

(a) FMNIST (1)

(b) CIFAR10 (1)

(c) Mini-ImageNet (1)

average training loss for all clients

3.0

Clustered Sampling

pow-d, d = 50

average training loss for all clients

=- Random Sampling

pow-d, d = 50

Clustered Sampling

average training loss for all lients

4.5

pow-d, d = 50

Clustered Sampling

DivFL (ideal)

DivFL (ideal)

DivFL (ideal)

FedCor

HiCS-FL (ours)

3.0

FedCor

HiCS-FL (ours)

4.0

FedCor

HiCS-FL (ours)

2.0

2.5

3.5

1.5

2.0

3.0

1.0

1.5

2.5

0.5

1.0

2.0

0

20

40

60

global rounds

80100120140160180200

0

50

global rounds

10

20

30

global rounds

40

50

60

70

80

90

100

CIFAR10. Here we compare the performance of HiCS-FL to FedProx [19] running CNN model with Adam optimizer on the task of training an FL system with 50 clients, where 20% of clients are selected to participate in each training round. Similar to the experiments on FMNIST, 3 sets of the concentration parameter α are considered: (1) { 0 . 001 , 0 . 01 , 0 . 1 , 0 . 5 , 1 } ; (2) { 0 . 001 , 0 . 002 , 0 . 005 , 0 . 01 , 0 . 5 }; (3) { 0 . 001 , 0 . 002 , 0 . 005 , 0 . 01 , 0 . 1 } . The interpretation of the scenarios emulated by these setting is as same as in the FMNIST experiments. Figure 2 demonstrates improvement of HiCS-FL over all the other methods. HiCS-FL exhibits particularly significant improvements in settings (2) and (3), where 80% of the clients with extremely imbalanced data benefit from 20% of the clients with either balanced or mildly imbalanced data. The advantage of HiCS-FL in setting (1) where all clients have relatively high data heterogeneity is relatively modest (see Fig.2.(d)) because the system's H S is relatively large (see discussion in Section 3.5).

Mini-ImageNet. As in the Mini-ImageNet experiments, we compare HiCS-FL to FedProx running ResNet18 with Adam optimizer but now consider training of an FL system with 100 clients, where 20% of the clients are selected to participate in each round of training. We consider two settings of the concentration parameter α : (1) { 0 . 001 , 0 . 01 , 0 . 1 , 0 . 5 , 1 } and (2) { 0 . 001 , 0 . 005 , 0 . 01 , 0 . 1 , 1 } . Setting (1) emulates the scenario where clients have a range of heterogeneity profiles, from extremely imbalanced, through mildly imbalanced, to balanced, while setting (2) corresponds to the scenario where 80% of the clients have extremely imbalanced data while the remaining 20% have balanced data. The system's H (1) S for setting (1) is larger than H (2) S for setting (2), which is reflected in a more significant improvements achieved by HiCS-FL in the latter setting, as shown in Figure 4.

THUCnews. To evaluate our method on data from a different domain, we conduct experiments involving text classification on the THUC news dataset in Chinese language ( 10 labels). Similar to the aforementioned experiments, we allocate data to 50 clients by emulating heterogeneous data distributions scenarios with parameter α set to: (1) { 0 . 001 , 0 . 01 , 0 . 1 , 0 . 2 , 1 }; (2) { 0 . 001 , 0 . 002 , 0 . 01 , 0 . 1 , 0 . 5 } ; and (3) { 0 . 001 , 0 . 002 , 0 . 005 , 0 . 01 , 0 . 1 }. We trained TextRNNs [20] with BiLSTM architecture as

the classifiers using Adam optimizer. The test accuracy of the global model trained with different schemes for 100 global rounds, reported in Table 1, show that our method outperforms baselines in all the settings, demonstrating efficacy of our proposed algorithm in a simple NLP task.

4.2 Accelerating the Training Convergence

In this section we report the communication costs required to achieve convergence when using HiCS-FL, and compare those results with the competing schemes. For brevity, we select one result from each experiment conducted on the considered four datasets, and display them in Table 2. As can be seen from the table, HiCS-FL significantly reduces the number of communication rounds needed to reach target test accuracy. On FMNIST, HiCS-FL needs 60 rounds to reach test accuracy 0 . 75 , achieving it 2 . 5 times faster than the random sampling scheme. On CIFAR10, HiCS-FL requires only 123 rounds to reach 0 . 6 test accuracy, which is 7 . 3 times faster than random sampling. Significant speedup appears on THUC dataset, in which HiCS-FL only needs 27 rounds to achieve 0 . 8 test accuracy, 3 . 1 times faster than the baseline. Acceleration on Mini-ImageNet is relatively modest but HiCS-FL still outperforms other methods, and does so up to 2 . 2 times faster than random sampling.

Table 2 also shows that HiCS-FL provides the reported improvements without introducing major computational and communication overhead. The only additional computation is due to estimating data heterogeneity and performing clustering utilizing bias updates, which scales with the total number of classes but does not increase with the size of the neural network model | θ t | . Remarkably, HiCS-FL outperforms pow-d, Clustered Sampling, DivFL and FedCor in terms of convergence speed, variance and test accuracy while requiring significantly less computations. More details are provided in Appendix A.11.

4.3 Number of Clustering Groups

As discussed at the end of Section 3.3, the distance function in Equation 9 can be reduced to the conventional cosine similarity when clients exhibit similar levels of statistical heterogeneity, despite potential differences in data distribution. Under these circumstances, our HiCS-FL method can recover the performance of the previously established CS approach [11]. While CS suggests that the number of clusters M should be greater than or equal to the number of selected clients K , our HiCS-FL does not require M > K but adheres to the CS settings to ensure a fair comparison. To elucidate the impact of the number of clusters, we conducted supplementary experiments with

HiCS-FL using varying numbers of clusters M and compared these results to those obtained with M = K as presented in the paper. The results of those experiments can be found in Table. 3. As shown there, HiCS-FL can perform well with smaller M < K as long as M is not too small, such as M = 3 .

4.4 Dynamic Availability of Clients

The purpose of the warm-up phase ( t < ⌈ N/K ⌉ ) shown in Alg. 1 is to collect updates of the output layer from all the available clients in the system in order to facilitate clustering. Although we conduct all the experiments in the setting where clients have fixed availability, our HiCS-FL does not assume all the clients are available in the warm-up phase and can be adapted to more practical scenarios where clients have dynamic availability.

In such a scenario, the warm-up phase can be implemented by the available clients at the beginning of training. The proposed HiCS-FL is then implemented only among the available clients; the available clients with more balanced data are preferred. When new clients join the system at the global round t , the server can obtain the information of availability and selects these new clients at round t +1 to approximate their data heterogeneity. To provide more insights, we conduct additional experiments on CIFAR10 dataset; the results are reported in Table. 4. As can be seen there, HiCS-FL outperforms baselines that consider clients' availability.

5 Conclusion

In this paper, we studied federated learning systems where clients that own non-IID data collaboratively train a global model; the system operates under communication constraints and thus only a fraction of clients participates in any given round of training. We developed HiCS-FL, a hierarchical clustered sampling method which estimates clients' data heterogeneity and uses this information to cluster and select clients to participate in training. We analyzed the performance of the proposed heterogeneity estimation method, and the convergence of training a FL system that deploys HiCS-FL. Extensive benchmarking experiments on four datasets demonstrated significant benefits of the proposed method, including improvement in convergence speed, variance and test accuracy, accomplished with only a minor computational overhead.

References

Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough, and Venkatesh Saligrama. 2021. Federated learning based on dynamic regularization. arXiv preprint arXiv:2111.04263 .

Ravikumar Balakrishnan, Tian Li, Tianyi Zhou, Nageen Himayat, Virginia Smith, and Jeff Bilmes. 2022. Diverse client selection for federated learning via submodular maximization. In International Conference on Learning Representations .

Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. 2018. A systematic study of the class imbalance problem in convolutional neural networks. Neural networks , 106:249-259.

Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research , 16:321-357.

Huancheng Chen and Haris Vikalo. 2024. Recovering labels from local updates in federated learning. arXiv preprint arXiv:2405.00955 .

Huancheng Chen, Chaining Wang, and Haris Vikalo. 2023. The best of both worlds: Accurate global and personalized models through federated learning with data-free hyper-knowledge distillation. In The Eleventh International Conference on Learning Representations .

Wenlin Chen, Samuel Horvath, and Peter Richtarik. 2020. Optimal client sampling for federated learning. arXiv preprint arXiv:2010.13723 .

Yae Jee Cho, Jianyu Wang, and Gauri Joshi. 2020. Client selection in federated learning: Convergence analysis and power-of-choice selection strategies. arXiv preprint arXiv:2010.01243 .

Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. 2021. Exploiting shared representations for personalized federated learning. In International Conference on Machine Learning , pages 2089-2099. PMLR.

Sever S Dragomir, Marcel L Scholz, and Jadranka Sunde. 2000. Some upper bounds for relative entropy and applications. Computers & Mathematics with Applications , 39(9-10):91-100.

Yann Fraboni, Richard Vidal, Laetitia Kameni, and Marco Lorenzi. 2021. Clustered sampling: Lowvariance and improved representativity for clients selection in federated learning. In International Conference on Machine Learning , pages 3407-3416. PMLR.

Jack Goetz, Kshitiz Malik, Duc Bui, Seungwhan Moon, Honglei Liu, and Anuj Kumar. 2019. Active federated learning. arXiv preprint arXiv:1909.12641 .

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. 2020. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning , pages 5132-5143. PMLR.

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 .

Qinbin Li, Bingsheng He, and Dawn Song. 2021. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10713-10722.

Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. 2021. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning , pages 6357-6368. PMLR.

Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. 2020. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine , 37(3):50-60.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2020. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems , 2:429-450.

Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning. arXiv preprint arXiv:1605.05101 .

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32.

Monica Ribero and Haris Vikalo. 2020. Communication-efficient federated learning via optimal client sampling. arXiv preprint arXiv:2007.15197 .

Felix Sattler, Klaus-Robert Müller, and Wojciech Samek. 2020. Clustered federated learning: Modelagnostic distributed multitask optimization under privacy constraints. IEEE transactions on neural networks and learning systems , 32(8):3710-3722.

Ronald W Schafer. 2011. What is a savitzky-golay filter?[lecture notes]. IEEE Signal processing magazine , 28(4):111-117.

Li Shen, Zhouchen Lin, and Qingming Huang. 2016. Relay backpropagation for effective learning of deep convolutional neural networks. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VII 14 , pages 467-482. Springer.

Fang Shi, Weiwei Lin, Lisheng Fan, Xiazhi Lai, and Xiumin Wang. 2023. Efficient client selection based on contextual combinatorial multi-arm bandits. IEEE Transactions on Wireless Communications .

Minxue Tang, Xuefei Ning, Yitu Wang, Jingwei Sun, Yu Wang, Hai Li, and Yiran Chen. 2022. Fedcor: Correlation-based active client selection strategy for heterogeneous federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10102-10111.

Aidmar Wainakh, Fabrizio Ventola, Till Müßig, Jens Keim, Carlos Garcia Cordero, Ephraim Zimmer, Tim Grube, Kristian Kersting, and Max Mühlhäuser. 2021. User label leakage from gradients in federated learning. arXiv preprint arXiv:2105.09369 .

Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. 2020. Federated learning with matched averaging. arXiv preprint arXiv:2002.06440 .

Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. 2020. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems , 33:7611-7623.

Joel Wolfrath, Nikhil Sreekumar, Dhruv Kumar, Yuanli Wang, and Abhishek Chandra. 2022. Haccs: Heterogeneity-aware clustered client selection for accelerated federated learning. In 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS) , pages 985-995. IEEE.

Haibo Yang, Minghong Fang, and Jia Liu. 2021. Achieving linear speedup with partial worker participation in non-iid federated learning. arXiv preprint arXiv:2101.11203 .

Miao Yang, Ximin Wang, Hongbin Zhu, Haifeng Wang, and Hua Qian. 2021. Federated learning with class imbalance reduction. In 2021 29th European Signal Processing Conference (EUSIPCO) , pages 2174-2178. IEEE.

Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. 2019. Bayesian nonparametric federated learning of neural networks. In International conference on machine learning , pages 7252-7261. PMLR.

Jianyi Zhang, Ang Li, Minxue Tang, Jingwei Sun, Xiang Chen, Fan Zhang, Changyou Chen, Yiran Chen, and Hai Li. 2022. Fed-cbs: A heterogeneity-aware client sampling mechanism for federated learning via class-imbalance reduction. arXiv preprint arXiv:2209.15245 .

A Appendix

A.1 Details of the Experiments

A.1.1 General Settings

The experimental results were obtained using Pytorch [22]. In the experiments involving FMNIST, each client used a CNN-based classifier with two 5 × 5 -convolutional layers and two 2 × 2 -maxpooling layers (with a stride of 2 ), followed by a fully-connected layer. In the experiments involving CIFAR10, each client used a CNN-based classifier with three 3 × 3 -convolutional layers and two 2 × 2 -maxpooling layers (with a stride of 2 ), followed by two fully-connected layers; dimension of the hidden layer was 64 . In the experiments involving Mini-ImageNet and THUC news, each client fine-tuned a pretrained ResNet18 [13] and learned a TextRNNs [20], respectively. The optimizers used for model training in the experiments on FMNIST and CIFAR10/Mini-ImageNet/THUC news were the mini-batch stochastic gradient descent (SGD) and Adam [15], respectively. The learning rate was initially set to 0 . 001 and then decreased every 10 iterations, with a decay factor 0 . 5 . The number of global communication rounds was set to 200, 500, 100 and 100 for the experiments on FMNIST, CIFAR10, Mini-ImageNet and THUC news, respectively. In all the experiments, the number of local epochs R was set to 2 and the size of a mini-batch was set to 64 . The sampling rate (fraction of the clients participating in a training round) was set to 0 . 1 for the experiments on FMNIST/THUC news, and to 0 . 2 for the experiments on CIFAR10/Mini-ImageNet. For the sake of visualization, data points in the presented graphs were smoothened by a Savitzky-Golay filter [25] with window length 13 and the polynomial order set to 3 .

A.1.2 Hyper-parameters

In all experiments, the hyper-parameter µ of the regularization term in FedProx [19] was set to 0 . 1 . In the Power-of-Choice (pow-d) [8] selection strategy, d was set to the total number of clients: 50 in the experiments on FMNIST, CIFAR10 and THUC news, 100 in the experiments on Mini-ImageNet. When running DivFL [2], we used the ideal setting where 1-step gradients were requested from all client in each round (regardless of their participation status), similar to the Power-of-Choice settings. For FedCor [28], we followed all settings in the paper and set the annealing coefficient β controlling the sampling strategy to 0.9 as suggested in the paper. For HiCS-FL (our method), the scaling parameter T (temperature) used in data heterogeneity estimation was set to 0 . 0025 in the experiments on FMNIST and to 0 . 0015 in the experiments on CIFAR10/Mini-ImageNet. In all experiments, parameter λ which multiplies the difference between clients' estimated data heterogeneity (used in clustering) was set to 10 . In all experiments, the number of clusters m was for convenience set to be equal to the number of selected clients K . The coefficient γ 0 was set to 4 in the experiments on FMNIST and CIFAR10 while set to 2 in the experiments on Mini-ImageNet. To group clients, both Clustered Sampling [11] and HiCS-FL (our method) utilized an off-the-shelf clustering algorithm performing hierarchical clustering with Ward's Method.

A.2 Empirical Validation of Assumption 3.1

To illustrate and empirically validate Assumption 3.1, we conducted extensive experiments on FMNIST and CIFAR10 with the same model mentioned in Section A.1. In particular, we varied α over 250 values in the interval [0 . 01 , 50] to generate data partitions allocated to 250 clients; entropy of the generated label distributions ranged from 0 to ln 10 (maximum). In these experiments, we allowed all clients to participate in each of 500 training rounds. To facilitate the desired study, in addition to these 250 clients we also simulated a super-client which owns a data set aggregating the data from all the clients (the set of labels in the aggregated dataset is uniformly distributed). In each round, clients start from the initialized global model and compute local gradients on their datasets; the super-client does the same on the aggregated dataset. The server computes and records squared Euclidean norm of the difference between the local gradients and the 'true" gradient (i.e., the super-client's gradient). In each round, the difference between the local gradient and the true gradient changes in a pattern similar to what is stated in Assumption 3.1. As an illustration, we plot all such gradient differences computed during the entire training process of a client. Specifically, the server computes the difference between local gradient and the true gradient in each round of training, obtaining 250 × 500 = 12500 data points that correspond to 250 data partitions. For better visualization, we merged adjacent points.

The results obtained by following these steps in experiments on FMNIST and CIFAR10 are shown in Figure 5. For a more informative visualization, the horizontal coordinate of a point in the scatter plot is H ( D ( k ) ) , while the vertical coordinate is ∥ η t ∇ F k ( θ t ) -η t ∇ F ( θ t ) ∥ 2 . The dashed lines correspond to the curves y = -exp( β [ x -H ( D 0 )]) ρ + κ that envelop the majority of the generated points. In the case of FMNIST, the blue dashed line is parametrized by β = 1 . 0 , ρ = 0 . 13 , and κ = 0 . 14 while the green dashed line is parametrized by β = 1 . 5 , ρ = 0 . 025 , and κ = 0 . 022 ; these two lines envelop 95% of the generated points. In the case of CIFAR10, the blue dashed line is parametrized by β = 2 . 0 , ρ = 0 . 30 , and κ = 0 . 36 while the green dashed line is parametrized by β = 1 . 8 , ρ = 0 . 15 , and κ = 0 . 20 ; as in the other plot, these two lines envelop 95% of the generated points. As the plots indicate, the difference between the local gradient and the true gradient increases as H ( D ( k ) ) decreases, implying that the local gradient computed by a client with more balanced data is closer to the true gradient.

(a) FMNIST

(b) CIFAR10

FittingsimulationpointsonFMNiSTwithnegativeexponentialfunctions

FittingsimulationpointsonCiFAR10withnegativeexponentialfunctions

gradient and true gradient

0.14

diffencebetweenlocal gradient and true gradient

0.35

0.12

0.30

0.10

0.08

0.25

diffence between localg

0.06

0.20

0.04

0.15

0.02

simulation points

0.10

simulation points

0.00

y = -0.13*np.exp(1.0*(x-ln10)) + 0.15

y = -0.30*exp(2.0*(x-ln10) + 0.36

y = -0.025*np.exp(1.5*(x-ln10)) + 0.022

0.05

y = -0.15*exp(1.8*(x-ln10) + 0.20

0.0

0.5

1.0

1.5

2.0

2.5

0.0

0.5

1.0

1.5

2.0

X: H(Dk)

X: H(Dk))

A.3 Gradient of the output (fully connected) layer's bias

Given a batch of samples ( x ( j,n ) , y ( j,n ) ) , the cross-entropy loss is readily computed as

where B is the batchsize; l is the number of mini-batches; C is the number of classes; d is the dimension of the hidden space; z ( j,n ) d denotes the d -th feature in the hidden space given sample x ( j,n ) in the j -th batch; w d,c and b c denote the weight of z ( j,n ) d and the bias for the neuron that outputs the probability of the class c , respectively; and q ( j,n ) c is the corresponding output logit on class c . The gradient of the bias b i given sample x ( j,n ) can be computed by the chain rule as

where

Then

If i = y ( j,n ) ,

̸

By plugging Eq. 18 and 19 in Eq. 15, we obtain

̸

A.4 Expectation of the local update ∆ b ( k )

By combining Eq. 4 and 5 and taking expectation, we obtain

̸

̸

̸

̸

̸

̸

̸

̸

Note that

̸

A comparison to L CE in Eq. 13 reveals that as L CE decreases during training, so does ∑ C i =1 E i . Given an untrained/initialized neural network model, E 0 i = 1 /C for ∀ i ∈ [ C ] , i.e., ∑ C i =1 E 0 i = -1 C -1 + C C -1 = 1 . At global round T , if L ∗ CE = 0 , then ∑ C i =1 E T i = -C C -1 + C C -1 = 0 .

A.5 Privacy of D ( k )

According to Eq. 6, the server is able to obtain C linear equations from each client,

where C denotes the number of classes. Suppose E [∆ b ( k ) i ] are known by the server. Then D ( k ) i , the variables in the aforementioned equations, cannot be determined uniquely since there are C variables and C +1 equations. Therefore, the server is unable to infer clients' true data label distribution and the privacy of D ( k ) is protected.

A.6 Proof of Theorem 3.3

In Section A.3 we derived an expression for the gradient of the bias in the output layer given a single sample ( x ( j,n ) , y ) in the mini-batch. It is worthwhile making the following two observations:

the sign of the y ( j,n ) -th component of ∇ b L ( j,n ) CE ( x ( j,n ) , y ( j,n ) ) is opposite of the sign of the other components; and

the y ( j,n ) -th component of ∇ b L ( j,n ) CE ( x ( j,n ) , y ( j,n ) ) is equal in magnitude to all other components combined.

Proof: Let ∆ b ( k ) = [∆ b ( k ) 1 , . . . , ∆ b ( k ) C ] denote the local update (made by client k ) of the bias in the output layer of the neural network model, and let D ( k ) = [ D ( k ) 1 , . . . , D ( k ) C ] be the (unknown) true data label distribution, ∑ C i =1 D ( k ) i = 1 . Assuming the learning rate η and R local epochs, the expectation of the local update of ∆ b ( k ) is

Data heterogeneity can be captured via entropy, H ( D ( k ) ) = -∑ C c =1 D ( k ) i ln D ( k ) i , where higher H ( D ( k ) ) indicates that client k has more balanced data. However, since we do not have access to

̸

the client's data distribution, we instead define and use as a measure of heterogeneity ˆ H ( D ( k ) ) ≜ H ( softmax (∆ b ( k ) , T )) , where

and where T denotes the temperature of the softmax operator. Suppose there are two clients, u and k , with class-balanced and class-imbalanced data; let D ( u ) and D ( k ) denote their data label distributions, respectively, while ˆ D ( u ) and ˆ D ( k ) are computed by softmax (∆ b ( u ) , T ) and softmax (∆ b ( k ) , T ) . Without a loss of generality, we can re-parameterize ˆ D ( u ) as

where U = [ 1 C , . . . , 1 C ] denotes uniform distribution; i -th component of Z i is 1 while the remaining components are 0 ; ϵ and ϵ i are all non-negative such that ϵ + ∑ C i =1 ϵ i = 1 . We can always set min j ϵ j = 0 ; otherwise, let ϵ ′ = ϵ +min j ϵ j and ϵ ′ i = ϵ i -min j ϵ j , ∀ i ∈ [ C ] ; ϵ quantifies how close is ˆ D ( u ) to U . Due to the concavity of entropy,

We will find the following lemma useful.

Lemma A.1 For two probability vectors p and q with dimension C, the Kullback-Leibler divergence between p and q satisfies

For the proof of the lemma, please see [10]. Applying it, we obtain

Combining Eq. 28 and Eq. 30, we obtain

By taking expectations of both sides,

Since ∥ ∥ ∥ ˆ D ( k ) -U ∥ ∥ ∥ 2 2 is convex (composition of the Euclidean norm and softmax), according to Jensen's inequality

where

Selecting T such that ηR ( D ( k ) i ∑ C c =1 E c -E i ) /T is sufficiently small and applying the first-order Taylor's expansion of e x around 0 , we obtain

where ∑ C j =1 D ( k ) j = 1 . This leads to a simplified ˆ D ( k ) ( E [∆ b ( k ) ]) ,

Substituting Eq. 36 for the second term on the right-hand side of ineq. 33 leads to

Now, consider

Taking expectations of both sides,

The above inequality holds component-wise, so for the j -component ( ϵ j = 0 )

Taking absolute value of both sides yields

By applying the triangle inequality we obtain

Let δ = max i ∣ ∣ ∣ 1 C ∑ C c =1 E c -E i ∣ ∣ ∣ . Since ∑ C c =1 E c ≤ C 1 C = 1 , it holds that

Furthermore, since E [ ϵ ] -1 < 0 ,

Note that

Therefore,

Substituting the above expression in Eq. 33, we obtain

and, therefore,

where C = ηR ( ηR + C 2 T ln C ) C 2 T 2 .

A.7 Convergence Analysis

Here we present the convergence analysis of an FL system deploying FedAvg with SGD wherein only a small fraction of clients participates in any given round of training. Recall that the objective function that comes up when training a neural network model is generally non-convex; we make the standard assumptions of smoothness, unbiased gradient estimate, and bounded variance.

Assumption A.2 (Smoothness) Each local objective function F k ( · ) is L -smooth,

Assumption A.3 (Gradient oracle) The stochastic gradient estimator g k ( θ t,r k ) = ∇ F k ( θ t,r k ) + ζ t,r k for each global round t and local epoch r is such that

■

and

With these three assumptions in place, we provide the proof of Theorem 3.4 stated in the main paper. The proof relies on the technique previously used in [7, 33], where the sampling method is unbiased and thus E [ 1 K ∑ k ∈S t ∑ R r =1 g k ( θ t,r k ) ] = ∑ N k =1 ∑ R r =1 p k ∇ F k ( θ t,r k ) . We provide a generalization that holds for any sampling strategy, resulting in E [ 1 K ∑ k ∈S t ∑ R r =1 g k ( θ t,r k ) ] = ∑ N k =1 ∑ R r =1 ω t k ∇ F k ( θ t,r k ) , where ω t k denotes the probability of sampling client k in round t under sampling strategy Π . Note that ∑ N k =1 ω t k = 1 . We assume that all clients deploy the same number of local epochs R and use learning rate η at round t .

A.7.1 key lemma

Lemma A.4 (Lemma 2 in [33]) Instate Assumptions 3.1, A.2 and A.3. For any step size η such that η ≤ 1 8 LR , for any client k it holds that

Proof of Lemma A.4: For any client k ∈ [ N ] and r ∈ [ R ] ,

Unrolling the recursion yields

A.7.2 Proof of Theorem 3.4

The model update at global round t is formed as

where θ t +1 and θ t denote parameters of the global model at rounds t +1 and t , respectively, and θ t,r k denotes parameters of the local model of client k after r local training epochs. Let

Taking the expectations (conditioned on θ t ) of both sides, we obtain

Inequality (a) in the expression above holds due to the smoothness of F ( · ) (see Assumption A.2). Note that the term A 1 can be bounded as

where equality (a) follows from ⟨ x , y ⟩ = 1 2 ( ∥ x ∥ 2 + ∥ y ∥ 2 - ∥ x -y ∥ 2 ) , inequality (b) is due to ∥ x + y ∥ 2 ≤ 2 ∥ x ∥ 2 + 2 ∥ y ∥ 2 , inequality (c) holds because ∥ ∑ n i =1 z i ∥ 2 ≤ n ∑ n i =1 ∥ z i ∥ 2 , and inequality (d) follows from Assumptions 3.1 and A.2. By selecting η < 1 8 LR and applying Lemma A.4 we obtain

Furthermore,

where equation (a) holds because E [ g k ( θ t,r k ) -∇ F k ( θ t,r k ) ] = 0 , inequality (b) stems from the Jensen's inequality, and inequality (c) is due to Assumption A.3. Substituting inequalities (61) and (62) into inequality (59) yields

If η < 1 8 LR , it must be that 1 2 -30 L 2 R 2 η 2 > 0 and L 2 η 2 -η 2 R < 0 , leading to

By rearranging and summing from t = 0 to t = T 1 we obtain

Let θ ∗ denote the optimal model's parameters, i.e., F ( θ ∗ ) ≤ F ( θ t ) ∀ t ∈ [ T ] . Then

where A 1 = Rη ( 1 2 -30 L 2 R 2 η 2 ) , A 2 = 60 L 2 R 3 η 3 +2 Rη Rη (1 -60 L 2 R 2 η 2 ) and Φ = ( 10 L 2 Rη 2 + Lη ) σ 2 1 -60 L 2 R 2 η 2 .

■

A.8 Regularization Terms in the Objective Function

The proposed method for estimating clients' data heterogeneity relies on the properties of gradient computed for the cross-entropy loss objective. However, the method also applies to the FL algorithms other than FedAvg, in particular those that add a regularization term to combat overfitting, including FedProx [19], FedDyn[1] and Moon [16]. In the following discussion, we demonstrate that HiCS-FL remains capable of distinguishing between clients with imbalanced and balanced data when using these other FL algorithms.

A.8.1 FedProx

The objective function used by FedProx [19] is

where θ t,r k is the vector of client k 's local model parameters in the r -th local epoch at global round t . Therefore, contribution of sample ( x ( j,n ) , y ( j,n ) ) to the gradient of L prox in local epoch r is

where b t,r = [ b t,r 1 , . . . , b t,r C ] denotes parameters of bias in the output layer of the local model, and b t = [ b t 1 , . . . , b t C ] denotes parameters of the global model at round t . We assume the model is trained by SGD as the optimizer, and hence

Therefore, and thus

Taking expectation of both sides yields

̸

where c r = (1 -η t µ ) r -1 > 0 provided η t and µ are sufficiently small. Therefore, the expectation of the local update of bias in the output layer satisfies

where C = ∑ R r =1 c r . Eq. (73) is similar to the expression for the expectation of the local updates of bias when applying FedAvg presented in the main paper; clearly, the analysis of HiCS-FL done in the context of FedAvg extends to FedProx.

A.8.2 FedDyn

For FedDyn [1], the objective function in local epoch r at global round t is

where R denotes the total number of local epochs. The first order condition for local optima implies

̸

̸

and, therefore,

where b t,r = [ b t,r 1 , . . . , b t,r C ] denotes the bias parameters in the output layer of the local model at local epoch r , and where ∆ b τ = [∆ b τ 1 , . . . , ∆ b τ C ] is the local update of the bias at round τ . Since

it holds that and

By induction,

Therefore, the expectation of the local update of bias in the output layer at round t can be computed as

Since the objective function of E [ ∆ b ( k ) , 1 i ] coincides with that of FedAvg,

where η 1 is the learning rate at global round t = 1 . Then,

where a 1 = η 1 R and a 2 = (1 + µη 2 ) R -1 . Furthermore,

and

By induction,

where a t = (1 + µη t ) R -1 and a = a 1 a t ( 1 + ∑ t -3 i =0 ∑ t -1 τ =2 I ( τ + i < t ) ∏ τ + i i = τ a s ) > 0 . After comparing Eq. (89) with its counterpart in the case of FedAvg, we conclude that the previously presented analysis of HiCS-FL extends to FedDyn.

A.8.3 Model-Contrastive Federated Learning (Moon)

Moon [16] relies on the objective function with a contrastive term

where z ( j,n ) denotes the output of the feature extractor of the local model θ t k , z ( j,n ) glob is the output of the feature extractor of the global model θ t , and z ( j,n ) prev is the output of the feature extractor of the local model in the previous round θ t -1 k . Since the contrastive term does not depend on the parameters of bias in the output layer, it holds that

Since the expectation of the local updates of bias in the output layer coincides with the one in case of FedAvg, previously presented analysis of HiCS-FL extends to Moon.

A.9 Optimization Algorithms Beyond SGD

Optimizers beyond SGD utilize different model update rules which in principle may lead to different properties of the local update of the bias in the output layer. However, for several variants of SGD, the properties of the local update of the bias remain such that our presented analysis still applies.

A.9.1 SGD with momentum

In each local epoch r , SGD with momentum updates the model according to

where m t,r k denotes the momentum in the r -th local epoch, µ is the weight for the momentum, and m t, 1 k = ∇L t, 1 CE . Then

where

Therefore, and thus we have

Similar to the discussion in the previous section,

where a = η t ( ∑ R r =2 ( µ r -1 +(1 -µ ) ∑ r τ =2 µ r -τ ) +1 ) > 0 . Similar result is obtained when SGD applies Nesterov acceleration as long as the optimizers are not using second-order momentum.

A.9.2 Adam Optimizer

Recall that the two observations regarding the gradient of L CE still hold when training the model with an adaptive optimizer such as Adam [15]. However, Adam updates the model differently from SGD. In particular, each entry of the gradient has an adaptive learning rate tied to its magnitude. With an SGD optimizer, the magnitude of the i -th entry of the local update of bias ∆ b ( k ) is approximately proportional to the fraction of the samples with label i , D ( k ) i (if E i is small),

However, this observation does not hold when using the Adam optimizer for the local update because each entry has a different learning rate η t,i and thus

Although the magnitude of E [ ∆ b ( k ) i ] is no longer approximately proportional to D ( k ) i , we can utilize the sign of E [ ∆ b ( k ) i ] , i.e.,

Suppose client k has highly imbalanced data, i.e., H ( D ( k ) ) is small. Then the maximal component max i D ( k ) i is much larger than the other components; in fact, it is likely to have only one positive

component in the local update of bias ∆ b ( k ) . On the contrary, suppose client u has balanced data and thus H ( D ( u ) ) is large. The maximal component max i D ( u ) i is then very close to the other components, and it is likely to observe larger number of positive components in the local update of ∆ b ( u ) . While characterizing P ( E [∆ b ( k ) i ] > 0) appears challenging, we can empirically infer that client u with more balanced data has a local update of bias ∆ b ( u ) with more positive components. With

ˆ H ( D ( u ) ) is more likely to be larger than ˆ H ( D ( k ) ) . The examples of estimated entropy when utilizing Adam as the optimizer are provided in Section. A.12.

A.10 Visualization of Data Partitions

To generate non-IID data partitions we follow the strategy in [35], utilizing Dirichlet distribution with different concentration parameters α to control the heterogeneity levels. In particular, the number of samples with label i owned by client k is set to X ( k ) i N i ∑ N j =1 X ( j ) i ,where X (1) i , . . . , X ( N ) i are drawn from

Dir ( α ) and N i denotes the total number of samples with label i in the overall dataset. For the setting with multiple α , we divide the overall training set into | α | equal parts and generate data partitions according to the method above. Figures 6 and 7 illustrate the class distribution of local clients by displaying the number of samples with different labels; colors distinguish between magnitudes - the darker the color, the more samples are in the class.

(a)

(b)

(c)

Heatmap of data partitions

Heatmap of data partitions

Heatmap of data partitions

1000

89

1000

89

N

1000

驰忆乙。

800

800

800

Client ID

600

Client ID

600

Client ID

600

400

400

400

-200

200

200

48

48

2

3

4

5

6

7

8

3

4

5

6

8

6

口

3

4

5

6

8

Class ID

Class ID

Class ID

(a)

(b)

Heatmap of datapartitions

80

100

688449552780

0

Heatmap of data partitions

80

100

Client ID

60

Client ID

60

40

40

-20

20

-0

-0

16204 28 25404 48 25

Class ID

8

22160048

Class ID

8

A.11 Computational and Communication Complexity

We compare the communication and computational costs of HiCS-FL with those of the competing methods, including Power of Choice (pow-d) [8], Clustered Sampling [11] and DivFL [2], and map them against random sampling, as shown in Table. 5. In its ideal setting, pow-d selects K clients with the largest local validation loss among all N clients. To compute the local validation loss at the beginning of a global training round t , the server must send the global model to all clients. Compared to the random sampling strategy where the global model is sent to only K clients, pow-d must transmit additional ( N -K ) | θ t | model parameters. Moreover, pow-d requires all clients to compute validation loss of the global model θ t on local datasets, which incurs additional O ( N | θ t | ) computations. While communication requirements of Clustered Sampling do not exceed those of random sampling, the server must run a clustering algorithm on the local updates of dimension | θ t | (the same as gradients). DivFL relies on maximizing a submodular function to select the most diverse clients based on all clients' gradients, leading to a transmission overhead and additional computation involving | θ t | -dimensional gradients. In our experiments, DivFL has consistently required the longest training time and memory usage due to its dependence on the submodularity maximizer. FedCor [28] cliams that only partial clients participating in the global update after warm-up stage but still needs all clients to perform inference for computing validation loss in the warm-up stage. Our proposed method, HiCS-FL, does not require any additional transmission of model parameters; furthermore, in HiCS-FL the server clusters clients based on their local updates of the bias in the output layer, which is low-dimensional and model-agnostic. Overall, HiCS-FL requires negligible computational overhead to significantly improve the performance of non-iid Federated Learning.

A.12 Examples of Estimated Entropy

To further illustrate the proposed framework, here we show a comparison between the estimated entropy of data label distribution and the true entropy. Specifically, Figures 8 and 9 show that the entropy estimated by the proposed method is close to the true entropy; the experiments were conducted on FMNIST and Mini-ImageNet, using SGD and Adam as optimizers, respectively. As stated in Theorem 3.3, the clients with larger true entropy are likely to have lager estimated entropy. In case where the model is trained with Adam, estimated entropy of data label distribution is not as accurate as in the case of using SGD. Figures 10 and 11 compare the performance of estimating entropy with SGD and Adam optimizers for the same setting of α . Notably, as shown in the figures, the method is capable of distinguishing clients with extremely imbalanced data from those with balanced data.

(a)

(b)

FMNIST Data Set alphas = [0.01, 0.02, 0.05, 0.1, 0.2]

1.75

Estimated Entropy

True Entropy

Entropy of Data Label Distribution

1.50

1.25

1.00

0.75

0.50

0.25

0.00

0

10

20

30

40

50

Client Index

FMNIST Data Set alphas = [0.001, 0.002, 0.005, 0.01, 0.5]

2.00 -

Estimated Entropy

1.75

True Entropy

Entropy of Data Label Distribution

1.50

1.25

1.00

0.75

0.50

0.25

0.00

0

10

20

30

40

50

Client Index

(a)

(b)

CIFAR100 Dataset alphas = [0.001, 0.01, 0.1, 0.5, 1]

Estimated Entropy

4.0

True Entropy

Entropy of Data Label Distribution

3.5

MMMwy

3.0

2.5

2.0

1.5

1.0

0

20

40

60

80

100

Client Index

CIFAR100 alpha = [0.001, 0.005, 0.01, 0.1, 1]

Estimated Entropy

4.0

True Entropy

M

3.5

3.0

Adoqug

2.5

2.0

1.5

1.0

20

40

Client Index

09

80

100

(a)

(b)

CIFAR10 Data Set alphas = [0.001, 0.01, 0.1, 0.5, 1.0]

Estimated Entropy

2.0

True Entropy

WW

Entropy of Data Label Distribution

1.5

1.0

0.5

0°0

0

10

20

30

40

50

Client Index

CIFAR10 Data Set alphas = [0.001, 0.01, 0.1, 0.5, 1.0]

Estimated Entropy

2.0

True Entropy

Entropy of Data Label Distribution

1.5

1.0

0.5

0°0

0

10

20

30

40

50

Client Index

(a)

(b)

CIFAR10 Dataset alphas = [0.001, 0.002, 0.005, 0.01, 0.5]

2.0

Estimated Entropy

True Entropy

Entropy of Data Label Distribution

1.5

1.0

0.5

0.0

0

10

20

30

40

50

Client Index

CIFAR10 Dataset alphas = [0.001, 0.002, 0.005, 0.01, 0.5]

Estimated Entropy

True Entropy

Entropy of Data Label Distribution

2.0

1.5

1.0

0.5

-0°0

0

10

20

30

40

50

Client Index

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of exNoecution) needed to reproduce the experiments?

Answer: [No]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper provides an interesting approach HiCS-FL to investigate the client sampling problem in federated learning, especially for the non-iid setting. This paper estimates the clients' data statistical heterogeneity (label distributions) via the client-updated gradients of the output layer's weights. By using this distance information, the server can distinguish which client owns more balanced data.

STRENGTHS:
- This paper found an interesting relationship between the last layer's bias and the label distributions of a given client data.
- The proposed distance metric can be used to estimate the local data heterogeneity and distinguish which one's data is more balanced.
- The paper is well-written and easy to follow.

WEAKNESSES:
- A potential privacy issue may be raised since it needs to access the individual client update (i.e., gradient) information.  More client's data information may be leaked from the gradient inversion attacks [1]. 


[1] Evaluating Gradient Inversion Attacks and Defenses in Federated Learning, NeurIPS 2021.

QUESTIONS:
- Why the first training rounds set to be $\lceil N / K\rceil$ in line 235?

LIMITATIONS:
See Weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

comments: SUMMARY:
The paper address data heterogeneity by clustering clients via the gradients of the output layer to distinguish between clients with balanced from those with imbalanced data. HiCS-FL assigns different importance to the clusters according to their average estimated data heterogeneity. The paper found that there is a correlation between the gradient of the output layer and the label distribution.

STRENGTHS:
The paper is very well written. The paper did a good job of summarizing what others have done and how HiCS-FL differs. 

Figure 2, and Figure 3 shows considerable gain over the baselines on common FL datasets. 

The method doesn't incur computation or communication overhead and leads to 2x speed up over other methods.

WEAKNESSES:
The approach of clustering to address heterogeneity might be practical as there are simpler methods that can achieve the same goal. 
The experiments were done on a small set of clients, 50. As the number of clients increases, the more pronounced the effect of heterogeneity can have.

QUESTIONS:
Data heterogeneity can be mitigated with pre-training and personalization. Would pre-training make clustering unnecessary?

LIMITATIONS:
The paper doesn't list limitation but I think a limitation there is the experimental setting. Using  Dirichlet to generate clients is unrealistic (I know this is common in FL papers), and the paper focuses only on classification tasks.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenges posed by non-IID data in FL systems, particularly under communication constraints where only a small fraction of clients can participate in each training round. It introduces HiCS-FL, a novel client selection method. HiCS-FL estimates the statistical heterogeneity of a client’s data using updates from the network’s output layer to cluster and sample clients more effectively. Based on the heterogeneity information, clients are clustered, and those with more balanced data are preferentially sampled. Experimental results demonstrate HiCS-FL's superior performance compared to state-of-the-art methods.\n\nSTRENGTHS:\nThe introduction of a hierarchical clustering-based method that considers data heterogeneity is a significant advancement over existing client selection techniques. The solution considers the balanced and imbalanced client data to train the model, which is a very interesting perspective. The extensive experiments on multiple datasets and comparison with several baselines demonstrate the robustness and generalizability of HiCS-FL. This paper is well-written.\n\nWEAKNESSES:\n- The paper primarily focuses on label imbalance when discussing data heterogeneity. However, other types of imbalances, such as feature imbalances within the data, are not considered. Addressing these other forms of data imbalance could provide a more comprehensive solution.\n\n- The method involves collecting updates from the fully connected layer, which may inadvertently reveal the label distribution of a client’s data. This could raise privacy concerns, as it potentially compromises the confidentiality of the client's data distribution.\n\n- The paper does not compare HiCS-FL with other relevant algorithms, such as Oort and Auxo, which also focus on client selection and clustering. Including these comparisons or discussions would provide a clearer picture of HiCS-FL's relative performance and effectiveness.\n\n[1] Oort: Efficient Federated Learning via Guided Participant Selection \n[2] Auxo: Heterogeneity-Mitigating Federated Learning via Scalable Client Clustering\n\n- The assumption that clients can be clustered into M groups, where M is larger than the number of selected clients (K) per round, may not be practical in large-scale systems. Specifically, the paper assumes that all clients are available during the initial clustering phase and that no new clients join the system later, which is unrealistic in dynamic FL environments.\n\nThere might be additional overhead associated with the proposed clustering method. The paper should ensure that the experimental comparisons account for this overhead to maintain fairness in the evaluation of HiCS-FL’s performance.\n\nQUESTIONS:\n- The rationale behind using gradients from different rounds for clustering clients is not clearly explained. Given that gradients can vary significantly across rounds, further clarification is needed to justify this approach and its impact on the effectiveness of the clustering process.\n\nLIMITATIONS:\nSee weakness", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel client selection method to address federated learning scenarios where clients exhibit varying degrees of data imbalance. The authors estimate the label distribution entropy of clients on the server side using the gradient of the output layer's bias. Based on this estimation, they cluster and sample clients, utilizing those with more balanced data to train the global model. Experiments in the paper demonstrate that the proposed method, HiCS-FL, achieves better and faster convergence compared to baselines. The authors provide detailed theoretical proofs to support their approach.\n\nSTRENGTHS:\n1. This paper proposes HiCS-FL, a new federated learning method designed to adaptively handle clients with varying label distributions. It achieves better convergence and greater efficiency compared to baseline methods in experiments.\n\n2. HiCS-FL utilizes the gradients of the output layer's bias to estimate clients' statistical data heterogeneity. This estimation is then used to cluster and sample clients, enabling efficient training of the global model with relatively data-balanced clients.\n\n3. This paper provides a detailed theoretical analysis of HiCS-FL.\n\nWEAKNESSES:\nSee questions.\n\nQUESTIONS:\n1. Is it due to privacy concerns that clients cannot directly compute and upload label distribution entropy to the server? If so, does estimating label distribution entropy through $ \\Delta \\mathbf{b}^{(k)} $ still pose privacy concerns for clients? Additionally, could you consider conducting an experiment comparing the federated learning performance (accuracy, convergence speed, etc.) using the true label distribution entropy versus the gradient-based label distribution entropy estimation proposed in this paper?\n\n2. Why wasn't an experimental analysis conducted on different values of $ \\lambda $ in Eq. 9?\n\n3. Will the final trained global model be used for inference on local clients? If so, wouldn't the client selection method that prefers more data-balanced clients be suboptimal for local clients with severe data imbalance?\n\nLIMITATIONS:\nThe authors mention leaving studies of system heterogeneity to future work in the first paragraph of the Introduction, but there is no further discussion on the limitations of the proposed approach. There are no concerns raised about the societal impacts of the research.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-22"}