Title: DELTA: Diverse Client Sampling for Fasting Federated Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: federated learning

client sampling
Abstract: Partial client participation has been widely adopted in Federated Learning (FL) to reduce the communication burden efficiently. However, an inadequate client sampling scheme can lead to the selection of unrepresentative subsets, resulting in significant variance in model updates and slowed convergence. Existing sampling methods are either biased or can be further optimized for faster convergence.
In this paper, we present DELTA, an unbiased sampling scheme designed to alleviate these issues. DELTA characterizes the effects of client diversity and local variance, and samples representative clients with valuable information for global model updates. In addition, DELTA is a proven optimal unbiased sampling scheme that minimizes variance caused by partial client participation and outperforms other unbiased sampling schemes in terms of convergence.  Furthermore, to address full-client gradient dependence, we provide a practical version of DELTA depending on the available clients' information, and also analyze its convergence. Our results are validated through experiments on both synthetic and real-world datasets.

DELTA: Diverse Client Sampling for Fasting Federated Learning

Abstract

Partial client participation has been widely adopted in Federated Learning (FL) to reduce the communication burden efficiently. However, an inadequate client sampling scheme can lead to the selection of unrepresentative subsets, resulting in significant variance in model updates and slowed convergence. Existing sampling methods are either biased or can be further optimized for faster convergence. In this paper, we present DELTA, an unbiased sampling scheme designed to alleviate these issues. DELTA characterizes the effects of client diversity and local variance, and samples representative clients with valuable information for global model updates. In addition, DELTA is a proven optimal unbiased sampling scheme that minimizes variance caused by partial client participation and outperforms other unbiased sampling schemes in terms of convergence. Furthermore, to address full-client gradient dependence, we provide a practical version of DELTA depending on the available clients' information, and also analyze its convergence. Our results are validated through experiments on both synthetic and real-world datasets.

1 Introduction

Federated Learning (FL) is a distributed learning paradigm that allows a group of clients to collaborate with a central server to train a model. Edge clients can perform local updates without sharing their data, which helps to protect their privacy. However, communication can be a bottleneck in FL, as edge devices often have limited bandwidth and connection availability [58]. To reduce the communication burden, only a subset of clients are typically selected for training. However, an improper client sampling strategy, such as uniform client sampling used in FedAvg [38], can worsen the effects of data heterogeneity in FL. This is because the randomly selected unrepresentative subsets can increase the variance introduced by client sampling and slow down convergence.

Existing sampling strategies can be broadly classified into two categories: biased and unbiased. Unbiased sampling is important because it can preserve the optimization objective. However, only a few unbiased sampling strategies have been proposed in FL, such as multinomial distribution (MD) sampling and cluster sampling. Specifically, cluster sampling can include clustering based on sample size and clustering based on similarity. Unfortunately, these sampling methods often suffer from slow convergence, large variance, and computation overhead issues [2, 13].

To accelerate the convergence of FL with partial client participation, Importance Sampling (IS), an unbiased sampling strategy, has been proposed in recent literature [5, 49]. IS selects clients with a large gradient norm, as shown in Figure 1. Another sampling method shown in Figure 1 is

Postive

Negative

Cluster B

Cluster A

Gradient of Not Selected

Client

Gradient of Selected Client

Gradient of Global Model

DELTA (Ours)

Cluster-based IS

cluster-based IS, which first clusters clients according to the gradient norm and then uses IS to select clients with a large gradient norm within each cluster.

Though IS and cluster-based IS have their advantages, 1) IS could be inefficient because it can result in the transfer of excessive similar updates from the clients to the server . This problem has been pointed out in recent works [52, 63], and efforts are being made to address it. One approach is to use cluster-based IS, which groups similar clients together. This can help, but 2) cluster-based IS has its drawbacks in terms of convergence speed and clustering effect. Figure 2 illustrates that both of these sampling methods can perform poorly at times. Specifically, compared with cluster-based IS, IS cannot fully utilize the diversity of gradients, leading to redundant sampling and a lack of substantial improvement in accuracy [52, 2]. While the inclusion of clients from small gradient groups in cluster-based IS leads to slow convergence as it approaches convergence, as shown by experimental results in Figure 6 and 7 in Appendix B.2. Furthermore, the clustering algorithm's performance tends to vary when applied to different client sets with varying parameter configurations, such as different numbers of clusters, as observed in prior works [52, 51, 56].

To address the limitations of IS and cluster-based IS, namely excessive similar updates and poor convergence performance, we propose a novel sampling method for Federated Learning termed D iv E rse c L ien T s A mpling (DELTA). Compared to IS and cluster-based IS methods, DELTA tends to select clients with diverse gradients, as shown in Figure 1. This allows DELTA to utilize the advantages of a large gradient norm for convergence acceleration while also overcoming the issue of gradient similarity.

Additionally, we propose practical algorithms for DELTA and IS that rely on accessible information from partial clients, addressing the limitations of existing analysis based on full client gradients [35, 5]. We also provide convergence rates for these algorithms. We replace uniform client sampling with DELTA in FedAvg, referred to as FedDELTA , and replace uniform client sampling with IS in FedAvg, referred to as FedIS . Their practical versions are denoted as FedPracDELTA and FedPracIS .

Toy Example and Motivation. We present a toy example to illustrate our motivation, where each client has a regression model. The detailed settings of each model and the calculation of each sampling algorithm's gradient are provided in Appendix B.1. Figure 3 shows that IS deviates from the ideal global model when aggregating gradients from clients with large norms. This motivates us to consider the correlation between local and global gradients in addition to gradient norms when sampling clients. Compared to IS, DELTA selects clients with large gradient diversities, which exploits the clients' information of both gradient norms and directions, resulting in a closer alignment to the ideal global model.

Our contributions. In this paper, we propose an efficient unbiased sampling scheme in the sense that (i) It effectively addresses the issue of excessive similar gradients without the need for additional clustering, while taking advantage of the accelerated convergence of gradient-norm-based IS and (ii) it is provable better than uniform sampling or gradient norm-based sampling. The sampling scheme is versatile and can be easily integrated with other optimization techniques, such as momentum, to improve convergence further.

As our key contributions,

We present DELTA, an unbiased FL sampling scheme based on gradient diversity and local variance. Our refined analysis shows that FedDELTA surpasses the state-of-the-art FedAvg in convergence rate by eliminating the O ( 1 / T 2 / 3 ) term and a σ 2 G -related term of O ( 1 / T 1 / 2 ) .

We present a novel theoretical analysis of nonconvex FedIS, which yields a superior convergence rate compared to existing works while relying on a more lenient assumption. Moreover, our analysis eliminates the O ( 1 / T 2 / 3 ) term of the convergence rate, in contrast to FedAvg.

We present a practical algorithm for DELTA in partial participation settings, utilizing available information to mitigate the reliance on full gradients. We prove that the convergence rates of these practical algorithms can attain the same order as the theoretical optimal sampling probabilities for DELTA and IS.

2 Related Work

Client sampling in federated learning (FL) can be categorized into unbiased and biased methods [14]. Unbiased methods, including multinomial sampling and importance sampling [30, 5, 49], ensure that the expected client aggregation is equivalent to the deterministic global aggregation when all clients participate. Unlike unbiased sampling, which has received comparatively little attention, biased sampling has been extensively examined in the context of federated learning, such as selecting clients with higher loss [7] or larger updates [48]. Recently, cluster-based client selection, which involves grouping clients into clusters and sampling from these clusters, has been proposed to sample diverse clients and reduce variance [41, 12, 52]. Nevertheless,the clustering will require extra communication and computational resources. The proposed DELTA algorithm can be seen as a muted version of a diverse client clustering algorithm without clustering operation.

While recent works [57, 28] have achieved comparable convergence rates to ours using variance reduction techniques, it is worth noting that these techniques are orthogonal to ours and can be easily integrated with our approach. Although [60] achieved the same convergence rate as ours, but their method requires dependent sampling and mixing participation conditions, which can lead to security problems and exceed the communication capacity of the server. In contrast, our method avoids these issues by not relying on such conditions.

A more comprehensive discussion of the related work can be found in Appendix A.

3 Theoretical Analysis and An Improved FL Sampling Strategy

This section presents FL preliminaries and analyzes sampling algorithms, including the convergence rate of nonconvex FedIS in Section 3.2, improved convergence analysis for FL sampling in Section 3.3, and proposal and convergence rate of the DELTA sampling algorithm in Section 3.4.

In FL, the objective of the global model is a sum-structured optimization problem:

where F i ( x ) = E ξ i ∼ D i [ F i ( x, ξ i )] represents the local objective function of client i over data distribution D i , and ξ i means the sampled data of client i . m is the total number of clients and w i represents the weight of client i . With partial client participation, FedAvg randomly selects | S t | = n clients ( n ≤ m ) to communicate and update model. Then the loss function of actual participating users in each round can be expressed as:

M 2 = σ 2 L +4 Kσ 2 G , ˆ M 2 = σ 2 L + K (1 -n / m ) σ 2 G , ˜ M 2 = σ 2 L +6 Kσ 2 G , ` M 2 = σ 2 L +4 Kζ 2 G , ´ M 2 = Kζ 2 G + Kσ 2 L .

Convexity: S and N are abbreviations for strong convex and nonconvex, respectively. ρ assumption: Bound of the similarity among local gradients. Mix participation: the number of participating clients is random, from none to full participation.

For ease of theoretical analysis, we make the following commonly used assumptions:

3.1 Assumptions

Assumption 1 (L-Smooth) . There exists a constant L > 0 , such that ∥∇ F i ( x ) -∇ F i ( y ) ∥ ≤ L ∥ x -y ∥ , ∀ x, y ∈ R d , and i = 1 , 2 , . . . , m .

Assumption 2 (Unbiased Local Gradient Estimator and Local Variance) . Let ξ i t be a random local data sample in the round t at client i : E [ ∇ F i ( x t , ξ i t ) ] = ∇ F i ( x t ) , ∀ i ∈ [ m ] . The function F i ( x t , ξ i t ) has a bounded local variance of σ L,i > 0 , satisfying E [ ∥ ∥ ∇ F i ( x t , ξ i t ) -∇ F i ( x t ) ∥ ∥ 2 ] = σ 2 L,i ≤ σ 2 L . Assumption 3 (Bound Dissimilarity) . There exists constants σ G ≥ 0 and A ≥ 0 such that E ∥∇ F i ( x ) ∥ 2 ≤ ( A 2 + 1) ∥∇ f ( x ) ∥ 2 + σ 2 G . When all local loss functions are identical, A 2 = 0 and σ 2 G = 0 .

The above assumptions are commonly used in both non-convex optimization and FL literature, see e.g. [21, 27, 60].

We notice that Assumption 3 can be further relaxed by Assumption 2 of [24]. We also provide Proposition C.4 in Appendix C to show all our convergence analysis, including Theorem 3.1,3.5 and Corollary 4.1,4.2 can be easily extended to the relaxed assumption while keeping the order of convergence rate unchanged.

3.2 Convergence Analysis of FedIS

As discussed in the introduction, IS faces an excessive gradient similarity problem, necessitating the development of a novel diversity sampling method. Prior to delving into the specifics of our new sampling strategy, we first present the convergence rate of FL under standard IS analysis in this section; this analysis itself is not well explored, particularly in the nonconvex setting. The complete FedIS algorithm is provided in Algorithm 2 of Appendix D, which differs from DELTA only in sampling probability (line 2) by using p i ∝ ∥ ∑ K -1 k =0 g i t,k ∥ .

Theorem 3.1 (Convergence rate of FedIS) . Let constant local and global learning rates η L and η be chosen as such that η L < min (1 / (8 LK ) , C ) , where C is obtained from the condition that 1 2 -10 L 2 K 2 ( A 2 + 1) η 2 L -L 2 ηK ( A 2 +1) 2 n η L > 0 ,and η ≤ 1 / ( η L L ) . In particular, suppose η L = O ( 1 √ TKL ) and η = O ( √ Kn ) , under Assumptions 1-3, the expected gradient norm of FedIS algorithm 2 will be bounded as follows:

where T is the total communication round, K is the total local epoch times, f 0 = f ( x 0 ) , f ∗ = f ( x ∗ ) , M = σ 2 L +4 Kσ 2 G and the expectation is over the local dataset samples among clients.

The FedIS sampling probability p t i is determined by minimizing the variance of convergence with respect to p t i . The variance term Φ is:

where Var ( 1 / ( mp t i ) ˆ g t i ) is called update variance . By optimizing the update variance, we get the sampling probability FedIS:

where ˆ g t i = ∑ K -1 k =0 ∇ F i ( x i k,t , ξ i k,t ) is the sum of the gradient updates of multiple local updates. The proof details of Theorem 3.1 and derivation of sampling probability FedIS are detailed in Appendix D and Appendix F.1.

Remark 3.2 (Explanation for the convergence rate) . It is worth mentioning that although a few works provide the convergence upper bound of FL with gradient-based sampling, several limitations exist in these analyses and results:

1) [49, 35] analyzed FL with IS using a strongly

Algorithm 1 FedDELTA and FedPracDELTA Federated learning with unbiased diverse sampling

:

Require: initial weights x 0 , global learning rate η , local learning rate η l , number of local epoch K , number of training rounds T

convex condition, whereas we extended the analysis to the non-convex problem.

2) Our analysis results, compared to the very recent non-convex analysis of FedIS [5] and FedAvg, remove the term O ( T -2 3 ) , although all these works choose a learning rate of O ( T -1 2 ) . Thus, our result achieves a tighter convergence rate when we use O (1 /T +1 /T 2 / 3 ) (provided by [43]) as our lower bound of convergence (see Table 1).

The comparison results in Table 1 reveal that even when σ G is large and becomes a dependency term for convergence rate, FedIS (ours) is still better than FedAvg and FedIS (others) since our result reduces the coefficient of σ G in the dominant term O ( T -1 2 ) .

Remark 3.3 (Novelty of our FedIS analysis) . Despite the existence of existing convergence analysis of partial participant FL [65, 47], including FedIS that builds on this analysis [35, 4], none of them take full advantage of the nature of unbiased sampling, and thus yield an imprecise upper bound on convergence. To tighten the FedIS upper bound, we first derive a tighter convergence upper bound for unbiased sampling FL. By adopting uniform sampling for unbiased probability, we achieve a tighter FedAvg convergence rate. Leveraging this derived bound, we optimize convergence variance using IS.

Compared with existing unbiased sampling FL works, including FedAvg and FedIS (others), our analysis on FedIS entails: (1) A tighter Local Update Bound Lemma: We establish Lemma C.3 using Assumption 3, diverging from the stronger assumption ∥∇ F i ( x t )) -∇ f ( x t ) ∥ 2 ≤ σ 2 G (used in [65, 47]), and the derived Lemma C.3 achieves a tighter upper bound than other works (Lemma 4 in [47], Lemma 2 in [65]). (2) A tighter upper bound on aggregated model updates E ∥ ∆ t ∥ 2 : By fully utilizing the nature of unbiased sampling, we convert the bound analysis of A 2 = E ∥ ∆ t ∥ 2 equally to a bound analysis of participant variance V ( 1 mp t i ˆ g t i ) and aggregated model update with full user participation. In contrast, instead of exploring the property the unbiased sampling, [47] repeats to use Lemma 4 and [65] uses Lemma 2 for bound A 2 . This inequality transform imposes a loose upper bound for A 2 , resulting in a convergence variance term determined by η 3 L , which reacts to the rate order being O ( T -2 3 ) . (3) Relying on a more lenient assumption: Beyond the aforementioned analytical improvement, our IS analysis obviates the necessity for unusual assumptions in other FedIS analysis such as Mix Participation [35] and ρ -Assumption [4].

Remark 3.4 (Extending FedIS to practical algorithm) . The existing analysis of IS algorithms [35, 5] relies on information from full clients, which is not available in partial participation FL. We propose a practical algorithm for FedIS that only uses information from available clients and provide its convergence rate in Corollary 4.1 in Section 4.

Despite its success in reducing the variance term in the convergence rate, FedIS is far from optimal due to issues with high gradient similarity and the potential for further minimizing the variance term (i.e., the global variance σ G and local variance σ L in Φ ). In the next section, we will discuss how to address this challenging variance term.

3.3 An Improved Convergence Analysis for FedDELTA

FedIS and FedDELTA have different approaches to analyzing objectives, with FedIS analyzing the global objective and FedDELTA analyzing a surrogate objective ˜ f ( x ) (cf. (7)). This leads to different convergence variance and sampling probabilities between the two methods. A flowchart (Figure 8 in Appendix E) has been included to illustrate the differences between FedIS and FedDELTA.

The limitations of FedIS. As shown in Figure 1, IS may have excessive similar gradient selection. The variance Φ in (4) reveals that the standard IS strategy can only control the update variance Var ( 1 / ( mp t i ) ˆ g t i , leaving other terms in Φ , namely σ L and σ G , untouched. Therefore, the standard IS is ineffective at addressing the excessive similar gradient selection problem, motivating the need for a new sampling strategy to address the issue of σ L and σ G .

The decomposition of the global objective. As inspired by the proof of Theorem 3.1 as well as the corresponding Lemma C.1 (stated in Appendix) proposed for unbiased sampling, the gradient of global objective can be decomposed into the gradient of surrogate objective ˜ f ( x t ) and update gap,

Intuitively, the surrogate objective represents the practical objective of the participating clients in each round, while the update gap χ t represents the distance between partial client participation and full client participation. The convergence behavior of the update gap χ 2 t is analogous to the update variance in Φ , and the convergence of the surrogate objective E ∥ ∥ ∥ ∇ ˜ f S t ( x t ) ∥ ∥ ∥ depends on the other variance terms in Φ , namely the local variance and global variance.

2

Minimizing the surrogate objective allows us to further reduce the variance of convergence, and we will focus on analyzing surrogate objective below. We first formulate the surrogate objective with an arbitrary unbiased sampling probability.

Surrogate objective formulation. The expression of the surrogate objective relies on the property of IS. In particular, IS aims to substitute the original sampling distribution p ( z ) with another arbitrary sampling distribution q ( z ) while keeping the expectation unchanged: E q ( z ) [ F i ( z )] = E p ( z ) [ q i ( z ) / p i ( z ) F i ( z )] . According to the Monte Carlo method, when q ( z ) follows the uniform distribution, we can estimate E q ( z ) [ F i ( z )] by 1 / m ∑ m i =1 F i ( z ) and E p ( z ) [ q i ( z ) / p i ( z ) F i ( z )] by 1 / n ∑ i ∈ S t 1 / mp i F i ( z ) , where m and | S t | = n are the sample sizes.

Based on IS property, we formulate the surrogate objective:

where m is the total number of clients, | S t | = n is the number of participating clients in each round, and p i t is the probability that client i is selected at round t . As noted in Lemma C.2 in the appendix, we have: 2 :

Then the convergence rate of the global objective can be formulated as follows:

Theorem 3.5 (Convergence upper bound of FedDELTA) . Under Assumption 1-3 and let local and global learning rates η and η L satisfy η L < 1 / (2 √ 10 KL √ 1 n ∑ l m =1 1 mp t l ) and ηη L ≤ 1 / KL , the minimal gradient norm will be bounded as below:

where f 0 = f ( x 0 ) , f ∗ = f ( x ∗ ) , c is a constant, and the expectation is over the local dataset samples among all workers. The combination of variance ˜ Φ represents combinations of local variance and client gradient diversity.

We derive the convergence rates for both sampling with replacement and sampling without replacement. For sampling without replacement:

For sampling with replacement,

where ζ G,i,t = ∥∇ F i ( x t ) -∇ f ( x t ) ∥ and let ζ G be a upper bound for all i , i.e., ζ G,i,t ≤ ζ G . The proof details of Theorem 3.5 can be found in Appendix E.

Remark 3.6 (The novelty of DELTA analysis) . IS focuses on minimizing V ( 1 mp t i ˆ g t i ) in convergence variance Φ (Eq. (4)), while leaving other terms like σ L and σ G unreduced. Unlike IS roles to reduce the update gap, we propose analyzing the surrogate objective for additional variance reduction.

Compared with FedIS, our analysis of DELTA entails: Focusing on surrogate objective, introducing a novel Lemma and bound: (1) we decompose global objective convergence into surrogate objective and update gap (6) . For surrogate objective analysis, we introduce Lemma E.8 to bound local updates. (2) leveraging the unique surrogate objective expression and Lemma E.8, we link sampling probability with local variance and gradient diversity, deriving novel upper bounds for A 1 and A 2 . (3) by connecting update gap's convergence behavior to surrogate objective through Definition E.1 and Lemma C.2, along with (6) , we establish ˜ Φ as the new global objective convergence variance. Optimizing convergence variance through novel ˜ Φ : FedIS aims to reduce the update variance term V ( 1 ( mp t i ) ˆ g t i ) in Φ , while FedDELTA aims to minimize the entire convergence variance ˜ Φ , which is composed of both gradient diversity and local variance. By minimizing ˜ Φ , we get the sampling method DELTA, which further reduces the variance terms of Φ that cannot be minimized through IS.

3.4 Proposed Sampling Strategy: DELTA

The expression of the convergence upper bound suggests that utilizing sampling to optimize the convergence variance can accelerate the convergence. Hence, we can formulate an optimization problem that minimizes the variance ˜ Φ with respect to the proposed sampling probability p t i :

where ˜ Φ is a linear combination of local variance σ L,i and gradient diversity ζ G,i,t (cf. Theorem 3.5). Corollary 3.7 (Optimal sampling probability of DELTA) . By solving the above optimization problem, the optimal sampling probability is determined as follows:

where α 1 and α 2 are constants defined as α 1 = 20 K 2 Lη L and α 2 = 5 KLη L + η n .

Remark 3.8. We note that a tension exists between the optimal sampling probability (13) and the setting of partial participation for FL. Thus, we also provide a practical implementation version for DELTA and analyze its convergence in Section 4. In particular, we will show that the convergence rate of the practical implementation version keeps the same order with a coefficient difference.

Corollary 3.9 (Convergence rate of FedDELTA) . Let η L = O ( 1 √ TKL ) , η = O ( √ Kn ) and substitute the optimal sampling probability (13) back to ˜ Φ . Then for sufficiently large T, the expected norm of DELTA algorithm 1 satisfies:

Difference between FedDELTA and FedIS. The primary distinction between FedDELTA and FedIS lies in the difference between ˜ Φ and Φ . FedIS aims to decrease the update variance term Var ( 1 / ( mp t i ) ˆ g t i ) in Φ , while FedDELTA aims to reduce the entire quantity ˜ Φ , which is composed of both gradient diversity and local variance. By minimizing ˜ Φ , we can further reduce the terms of Φ that cannot be minimized through FedIS. This leads to different expressions for the optimal sampling probability. The difference between the two resulting update gradients is discussed in Figure 3. Additionally, as seen in Table 1, FedDELTA achieves a superior convergence rate of O ( G 2 / ϵ 2 ) compared to other unbiased sampling algorithms.

Compare DELTA with uniform sampling. According to the Cauchy-Schwarz inequality, DELTA is at least better than uniform sampling by reducing variance: ˜ Φ uniform ˜ Φ DELTA = m ∑ i m =1 ( √ α 1 σ 2 L + α 2 ζ 2 G,i,t ) 2 ( ∑ i m =1 √ α 1 σ 2 L + α 2 ζ 2 G,i,t ) 2 ≥ 1 . ( ∑ i m √ α σ 2 + α ζ 2 ) 2

This implies that DELTA does reduce the variance, especially when =1 1 L 2 G,i,t ∑ i m =1 ( √ α 1 σ 2 L + α 2 ζ 2 G,i,t ) 2 ≪ m .

The significance of DELTA. (1) DELTA is the first unbiased sampling algorithm, to the best of our knowledge, that considers both gradient diversity and local variance in sampling, accelerating convergence. (2) Developing DELTA inspires an improved convergence analysis by focusing on the surrogate objective, leading to a superior convergence rate for FL. (3) Moreover, DELTA can be seen as an unbiased version with the complete theoretical justification for the existing heuristic or biased diversity sampling algorithm of FL, such as [2].

4 FedPracDELTA and FedPracIS: The Practical Algorithms

The gradient-norm-based sampling method necessitates the calculation of the full gradient in every iteration [10, 70]. However, acquiring each client's gradient in advance is generally impractical in FL. To overcome this obstacle, we leverage the gradient from the previous participated round to estimate the gradient of the current round, thus reducing computational resources [49].

For FedPracIS, at round 0, all probabilities are set to 1 / m . Then, during the i th iteration, once participating clients i ∈ S t have sent the server their updated gradients, the sampling probabilities are updated as follows:

where the multiplicative factor ensures that all probabilities sum to 1. The FedPracIS algorithm is shown in Algorithm 2 of Appendix D.

For FedPracDELTA, we use the average of the latest participated clients' gradients to approximate the true gradient of the global model. For local variance, it is obtained by the local gradient's variance over local batches. Specifically, ζ G,i,t = ∥ ˆ g i,t - ∇ ˆ f ( x t ) ∥ , where ∇ ˆ f ( x t ) = 1 n ∑ i ∈ S t ˆ g i,t = 1 n ∑ i ∈ S t ∑ K -1 k =0 ∇ F i ( x i k,t , ξ i k,t ) and σ 2 L,i = 1 | B | ∑ b ∈ B (ˆ g b i,t -1 | B | ∑ b ∈ B ˆ g b i,t ) 2 , where b ∈ B is the local data batch. Then the sampling probabilities are updated as follows:

The FedPracDELTA algorithm is shown in Algorithm 1. Specifically, for α , the default value is 0.5, whereas ζ G and σ L can be implemented by computing the locally obtained gradients.

Assumption 4 (Local gradient norm bound) . The gradients ∇ F i ( x ) are uniformly upper bounded (by a constant G > 0 ) ∥∇ F i ( x ) ∥ 2 ≤ G 2 , ∀ i.

Assumption 4 is a general assumption in IS community to bound the gradient norm [70, 10, 23], and it is also used in the FL community to analyze convergence [2, 68]. This assumption tells us a useful fact that will be used later: ∥∇ F i ( x t,k , ξ t,k ) / ∇ F i ( x s,k , ξ s,k ) ∥ ≤ U . While, for DELTA, the assumption used is a more relaxed version of Assumption 4, namely, E ∥∇ F i ( x ) -∇ f ( x ) ∥ 2 ≤ G 2 (further details are provided in Appendix G).

Corollary 4.1 (Convergence rate of FedPracIS) . Under Assumption 1-4, the expected norm of FedPracIS will be bounded as follows:

(a)

ν

= 20

(b)

ν

= 30

(c)

ν

= 40

FedAvg

FedAvg

0

FedAvg

2

Powerofchoice

0

Powerofchoice

Power of choice

FedIS

Fedis

-2

FedIS

0

FedDELTA

-2

FedDELTA

FedDELTA

S501

-2

S501

S501

6

6o1

6

6

8

8-

8-

10

10

10

12

0

250

500

750100012501500

1750

2000

0

250

500

750

100012501500

1750

2000

0

250

500

750

1000

1250

1500

1750

2000

communication rounds

communication rounds

communication rounds

where M = σ 2 L + 4 Kσ 2 G,s , σ G,s is the gradient dissimilarity bound of round s , and ∥∇ F i ( x t,k , ξ t,k ) / ∇ F i ( x s,k , ξ s,k ) ∥ ≤ U for all i and k .

Corollary 4.2 (Convergence rate of FedPracDELTA) . Under Assumption 1-4, the expected norm of FedPracDELTA satisfies:

where ˜ U is a constant that ∥∇ F i ( x t ) - ∇ f ( x t ) ∥ / ∥∇ F i ( x s ) - ∇ f ( x s ) ∥ ≤ ˜ U 1 ≤ ˜ U and ∥ σ L,t /σ L,s ∥ ≤ ˜ U 2 ≤ ˜ U , and ζ G,s is the gradient diversity bound of round s for all clients.

Remark 4.3. The analysis of the FedPracIS and FedPracDELTA is independent of the unavailable information in the partial participation setting. The convergence rates are of the same order as that of our theoretical algorithm but with an added coefficient constant term that limits the gradient changing rate, as shown in Table 1.

The complete derivation and discussion of the practical algorithm can be found in Appendix G.

5 Experiments

In this section, we evaluate the efficiency of the theoretical algorithm FedDELTA and the practical algorithm FedPracDELTA on various datasets. Our code is available at https://github.com/L3030/DELTA_FL.

Datasets. (1) We evaluate FedDELTA on synthetic data and split-FashionMNIST. The synthetic data follows y = log ( ( A i x -b i ) 2 / 2 ) and "split" means letting 10% of clients own 90% of the data. (2) We evaluate FedPracDELTA on non-iid FashionMNIST, CIFAR-10 and LEAF [3]. Details of data generation and partitioning are provided in Appendix H.2.

Baselines and Models. We compare our algorithm, Fed(Prac)DELTA (Algorithm 1), with Fed(Prac)IS (Algorithm 2 in Appendix D), FedAVG [38], which uses random sampling, and Power-ofchoice [7], which uses loss-based sampling and Cluster-based IS [52]. We utilize the regression model on synthetic date, the CNN model on Fashion-MNIST and Leaf, and the ResNet-18 on CIFAR-10. All algorithms are compared under the same experimental settings, such as lr and batch size. Full details of the sampling process of baselines and the setup of experiments are provided in Appendix H.2.

The maximum values reported in Table 2 are observed during the last 4% of rounds, where these algorithms have already reached convergence. The term 'maximum five accuracies' refers to the mean of the five highest accuracy values obtained within the plateau region of the accuracy curve.

Figure 4 illustrates the theoretical FedDELTA outperforms other biased and unbiased methods in convergence speed on synthetic datasets. The superiority of the theoretical DELTA is also confirmed on split-FashionMNIST, as shown in Appendix H in Figure 14(a). Additional experimental results, which include a range of different choices of regression parameters A i , b i , noise ν , and client numbers, are presented in Figure 11, Figure 12, and Figure 13 in Appendix H.3.

Table 2 shows the FedPracDELTA has better performance in accuracy, communication rounds, and training wall-clock times. Notably, FedPracDELTA significantly accelerates convergence by requiring fewer training rounds and less time to achieve the threshold accuracy in FashionMNIST,

CIFAR-10, FEMNIST, and CelebA. Additionally, on the natural federated dataset LEAF (FEMNIST and CelebA), our results demonstrate that both FedPracDELTA and FedPracIS exhibit substantial improvements over FedAvg. Figure 14(b) in Appendix H.3 illustrates the superior convergence of FedPracDELTA, showcasing the accuracy curves of sampling algorithms on FEMNIST.

Table 3 demonstrates that when compatible with momentum or proximal regularization, our method keeps its superiority in convergence. We combine various optimization methods such as proximal regularization [29], momentum [34], and VARP [18] with sampling algorithms to assess their performance on FEMNIST and FashionMNIST. Additional results for proximal and momentum on CIFAR-10, and for VARP on FashionMNIST, are available in Table 4 and Table 5 in Appendix H.3.

Ablation studies. We also provide ablation studies of heterogeneity α in Table 9 and the impact of the number of sampled clients on accuracy in Figure 15 in Appendix H.3.

6 Conclusions, Limitations, and Future Works

This work studies the unbiased client sampling strategy to accelerate the convergence speed of FL by leveraging diverse clients. To address the prevalent issue of full-client gradient dependence in gradient-based FL [36, 4], we extend the theoretical algorithm DELTA to a practical version that utilizes information from the available clients.

Nevertheless, addressing the backdoor attack defense issue remains crucial in sampling algorithms. Furthermore, there is still significant room for developing an efficient and effective practical algorithm for gradient-based sampling methods. We will prioritize this as a future research direction.

References

Guillaume Alain, Alex Lamb, Chinnadhurai Sankar, Aaron Courville, and Yoshua Bengio. Variance reduction in sgd by distributed importance sampling. arXiv preprint arXiv:1511.06481 , 2015.

Ravikumar Balakrishnan, Tian Li, Tianyi Zhou, Nageen Himayat, Virginia Smith, and Jeff Bilmes. Diverse client selection for federated learning: Submodularity and convergence analysis. In ICML 2021 International Workshop on Federated Learning for User Privacy and Data Confidentiality , Virtual, July 2021.

Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Koneˇ cn` y, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.01097 , 2018.

Wenlin Chen, Samuel Horvath, and Peter Richtarik. Optimal client sampling for federated learning. arXiv preprint arXiv:2010.13723 , 2020.

Wenlin Chen, Samuel Horváth, and Peter Richtárik. Optimal client sampling for federated learning. Transactions on Machine Learning Research , 2022.

Yae Jee Cho, Samarth Gupta, Gauri Joshi, and Osman Ya˘ gan. Bandit-based communicationefficient client selection strategies for federated learning. In 2020 54th Asilomar Conference on Signals, Systems, and Computers , pages 1066-1069. IEEE, 2020.

Yae Jee Cho, Jianyu Wang, and Gauri Joshi. Towards understanding biased client selection in federated learning. In International Conference on Artificial Intelligence and Statistics , pages 10351-10375. PMLR, 2022.

Yongheng Deng, Feng Lyu, Ju Ren, Huaqing Wu, Yuezhi Zhou, Yaoxue Zhang, and Xuemin Shen. Auction: Automated and quality-aware client selection framework for efficient federated learning. IEEE Transactions on Parallel and Distributed Systems , 33(8):1996-2009, 2022.

Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. arXiv preprint arXiv:2010.01264 , 2020.

Víctor Elvira and Luca Martino. Advances in importance sampling. arXiv preprint arXiv:2102.05407 , 2021.

Maximilian Fiedler and Peter Gritzmann. Coresets for weight-constrained anisotropic assignment and clustering. arXiv preprint arXiv:2203.10864 , 2022.

Yann Fraboni, Richard Vidal, Laetitia Kameni, and Marco Lorenzi. Clustered sampling: Lowvariance and improved representativity for clients selection in federated learning, 2021.

Yann Fraboni, Richard Vidal, Laetitia Kameni, and Marco Lorenzi. A general theory for client sampling in federated learning. In IJCAI 2022-31st International joint conférence on artificial intellignce , 2022.

Yann Fraboni, Richard Vidal, Laetitia Kameni, and Marco Lorenzi. A general theory for client sampling in federated learning. In Trustworthy Federated Learning: First International Workshop, FL 2022, Held in Conjunction with IJCAI 2022, Vienna, Austria, July 23, 2022, Revised Selected Papers , pages 46-58. Springer, 2023.

Yongxin Guo, Tao Lin, and Xiaoying Tang. Towards federated learning on time-evolving heterogeneous data. arXiv preprint arXiv:2112.13246 , 2021.

Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335 , 2019.

Tiansheng Huang, Weiwei Lin, Li Shen, Keqin Li, and Albert Y. Zomaya. Stochastic client selection for federated learning with volatile clients. IEEE Internet of Things Journal , 9(20):2005520070, 2022.

Divyansh Jhunjhunwala, PRANAY SHARMA, Aushim Nagarkatti, and Gauri Joshi. Fedvarp: Tackling the variance due to partial client participation in federated learning. In The 38th Conference on Uncertainty in Artificial Intelligence , 2022.

Tyler B Johnson and Carlos Guestrin. Training deep models faster with robust, approximate importance sampling. Advances in Neural Information Processing Systems , 31:7265-7275, 2018.

Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank J Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Mime: Mimicking centralized stochastic algorithms in federated learning. arXiv preprint arXiv:2008.03606 , 2020.

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning , pages 5132-5143. PMLR, 2020.

Angelos Katharopoulos and François Fleuret. Biased importance sampling for deep neural network training. arXiv preprint arXiv:1706.00043 , 2017.

Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning , pages 2525-2534. PMLR, 2018.

Ahmed Khaled and Peter Richtárik. Better theory for sgd in the nonconvex world. arXiv preprint arXiv:2002.03329 , 2020.

Yeongwoo Kim, Ezeddin Al Hakim, Johan Haraldson, Henrik Eriksson, José Mairton B da Silva, and Carlo Fischione. Dynamic clustering in federated learning. In ICC 2021-IEEE International Conference on Communications , pages 1-6. IEEE, 2021.

Haneul Ko, Jaewook Lee, Sangwon Seo, Sangheon Pack, and Victor C. M. Leung. Joint client selection and bandwidth allocation algorithm for federated learning. IEEE Transactions on Mobile Computing , pages 1-1, 2021.

Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning , pages 5381-5393. PMLR, 2020.

Bo Li, Mikkel N Schmidt, Tommy S Alstrøm, and Sebastian U Stich. Partial variance reduction improves non-convex federated learning on heterogeneous data. arXiv preprint arXiv:2212.02191 , 2022.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. arXiv preprint arXiv:1812.06127 , 2018.

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems , 2:429-450, 2020.

Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:1907.02189 , 2019.

Shiyun Lin, Yuze Han, Xiang Li, and Zhihua Zhang. Personalized federated learning towards communication efficiency, robustness and fairness. Advances in Neural Information Processing Systems , 2022.

Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. arXiv preprint arXiv:2006.07242 , 2020.

Wei Liu, Li Chen, Yunfei Chen, and Wenyi Zhang. Accelerating federated learning via momentum gradient descent. IEEE Transactions on Parallel and Distributed Systems , 31(8):1754-1766, 2020.

Bing Luo, Wenli Xiao, Shiqiang Wang, Jianwei Huang, and Leandros Tassiulas. Tackling system and statistical heterogeneity for federated learning with adaptive client sampling. In IEEE INFOCOM 2022-IEEE Conference on Computer Communications , pages 1739-1748. IEEE, 2022.

Bing Luo, Wenli Xiao, Shiqiang Wang, Jianwei Huang, and Leandros Tassiulas. Tackling system and statistical heterogeneity for federated learning with adaptive client sampling. In IEEE INFOCOM 2022 - IEEE Conference on Computer Communications , pages 1739-1748, 2022.

Jiahua Ma, Xinghua Sun, Wenchao Xia, Xijun Wang, Xiang Chen, and Hongbo Zhu. Client selection based on label quantity information for federated learning. In 2021 IEEE 32nd Annual International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC) , pages 1-6, 2021.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Aritra Mitra, Rayana Jaafar, George J Pappas, and Hamed Hassani. Achieving linear convergence in federated learning under objective and systems heterogeneity. arXiv preprint arXiv:2102.07053 , 2021.

Ihab Mohammed, Shadha Tabatabai, Ala Al-Fuqaha, Faissal El Bouanani, Junaid Qadir, Basheer Qolomany, and Mohsen Guizani. Budgeted online selection of candidate iot clients to participate in federated learning. IEEE Internet of Things Journal , 8(7):5938-5952, 2021.

Mahdi Morafah, Saeed Vahidian, Weijia Wang, and Bill Lin. Flis: Clustered federated learning via inference similarity for non-iid data distribution. IEEE Open Journal of the Computer Society , 4:109-120, 2023.

Khalil Muhammad, Qinqin Wang, Diarmuid O'Reilly-Morgan, Elias Tragos, Barry Smyth, Neil Hurley, James Geraci, and Aonghus Lawlor. Fedfast: Going beyond average for faster training of federated recommender systems. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 1234-1242, 2020.

Kumar Kshitij Patel, Lingxiao Wang, Blake E Woodworth, Brian Bullins, and Nati Srebro. Towards optimal communication complexity in distributed non-convex optimization. Advances in Neural Information Processing Systems , 35:13316-13328, 2022.

Jake Perazzone, Shiqiang Wang, Mingyue Ji, and Kevin S. Chan. Communication-efficient device scheduling for federated learning using stochastic optimization. In IEEE INFOCOM 2022 - IEEE Conference on Computer Communications , pages 1449-1458, 2022.

Zhe Qu, Rui Duan, Lixing Chen, Jie Xu, Zhuo Lu, and Yao Liu. Context-aware online client selection for hierarchical federated learning, 2021.

Zhe Qu, Rui Duan, Lixing Chen, Jie Xu, Zhuo Lu, and Yao Liu. Context-aware online client selection for hierarchical federated learning. IEEE Transactions on Parallel and Distributed Systems , 33(12):4353-4367, 2022.

Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Koneˇ cn` y, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00295 , 2020.

Monica Ribero and Haris Vikalo. Communication-efficient federated learning via optimal client sampling. arXiv preprint arXiv:2007.15197 , 2020.

Elsa Rizk, Stefan Vlaski, and Ali H Sayed. Federated learning under importance sampling. arXiv preprint arXiv:2012.07383 , 2020.

Yichen Ruan and Carlee Joe-Wong. Fedsoft: Soft clustered federated learning with proximal local updating. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 8124-8131, 2022.

Gaganpreet Sharma. Pros and cons of different sampling techniques. International journal of applied research , 3(7):749-752, 2017.

Guangyuan Shen, Dehong Gao, DuanXiao Song, Xukai Zhou, Shirui Pan, Wei Lou, Fang Zhou, et al. Fast heterogeneous federated learning with hybrid client selection. arXiv preprint arXiv:2208.05135 , 2022.

Chengshuai Shi and Cong Shen. Federated multi-armed bandits. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 9603-9611, 2021.

Sebastian U Stich, Anant Raj, and Martin Jaggi. Safe adaptive importance sampling. arXiv preprint arXiv:1711.02637 , 2017.

Canh T. Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 21394-21405. Curran Associates, Inc., 2020.

Steven K Thompson. Adaptive cluster sampling. Journal of the American Statistical Association , 85(412):1050-1059, 1990.

Alexander Tyurin and Peter Richtárik. A computation and communication efficient method for distributed nonconvex problems in the partial participation setting. arXiv preprint arXiv:2205.15580 , 2022.

Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan AlShedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. A field guide to federated optimization. arXiv preprint arXiv:2107.06917 , 2021.

Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. arXiv preprint arXiv:2007.07481 , 2020.

Shiqiang Wang and Mingyue Ji. A unified analysis of federated learning with arbitrary client participation. arXiv preprint arXiv:2205.13648 , 2022.

Joel Wolfrath, Nikhil Sreekumar, Dhruv Kumar, Yuanli Wang, and Abhishek Chandra. Haccs: Heterogeneity-aware clustered client selection for accelerated federated learning. In 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS) , pages 985-995, 2022.

Wenchao Xia, Tony QS Quek, Kun Guo, Wanli Wen, Howard H Yang, and Hongbo Zhu. Multiarmed bandit-based client scheduling for federated learning. IEEE Transactions on Wireless Communications , 19(11):7108-7123, 2020.

Han Xie, Jing Ma, Li Xiong, and Carl Yang. Federated graph classification over non-iid graphs. Advances in neural information processing systems , 34:18839-18852, 2021.

Xiaohui Xu, Sijing Duan, Jinrui Zhang, Yunzhen Luo, and Deyu Zhang. Optimizing federated learning on device heterogeneity with a sampling strategy. In 2021 IEEE/ACM 29th International Symposium on Quality of Service (IWQOS) , pages 1-10, 2021.

Haibo Yang, Minghong Fang, and Jia Liu. Achieving linear speedup with partial worker participation in non-iid federated learning. arXiv preprint arXiv:2101.11203 , 2021.

Hao Yu, Sen Yang, and Shenghuo Zhu. Parallel restarted sgd with faster convergence and less communication: Demystifying why model averaging works for deep learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 5693-5700, 2019.

Hangjia Zhang, Zhijun Xie, Roozbeh Zarei, Tao Wu, and Kewei Chen. Adaptive client selection in resource constrained federated learning systems: A deep reinforcement learning approach. IEEE Access , 9:98423-98432, 2021.

Xinwei Zhang, Mingyi Hong, Sairaj Dhople, Wotao Yin, and Yang Liu. Fedpd: A federated learning framework with optimal rates and adaptivity to non-iid data. arXiv preprint arXiv:2005.11418 , 2020.

Boxin Zhao, Ziqi Liu, Chaochao Chen, Mladen Kolar, Zhiqiang Zhang, and Jun Zhou. Adaptive client sampling in federated learning via online learning with bandit feedback. arXiv preprint arXiv:2112.14332 , 2021.

Peilin Zhao and Tong Zhang. Stochastic optimization with importance sampling for regularized loss minimization. In international conference on machine learning , pages 1-9. PMLR, 2015.

Hongbin Zhu, Yong Zhou, Hua Qian, Yuanming Shi, Xu Chen, and Yang Yang. Online client selection for asynchronous federated learning with fairness consideration. IEEE Transactions on Wireless Communications , pages 1-1, 2022.

Contents of Appendix

A An Expanded Version of The Related Work

FedAvg is proposed by [38] as a de facto algorithm of FL, in which multiple local SGD steps are executed on the available clients to alleviate the communication bottleneck. While communication efficient, heterogeneity, such as system heterogeneity [29, 31, 59, 39, 9], and statistical/objective heterogeneity [33, 21, 29, 59, 15], results in inconsistent optimization objectives and drifted clients models, impeding federated optimization considerably.

Objective inconsistency in FL. Several works also encounter difficulties from the objective inconsistency caused by partial client participation [31, 7, 2]. [31, 7] use the local-global gap f ∗ -1 m ∑ m i =1 F ∗ i to measure the distance between the global optimum and the average of all local personal optima, where the local-global gap results from objective inconsistency at the final optimal point. In fact, objective inconsistency occurs in each training round, not only at the final optimal point. [2] also encounter objective inconsistency caused by partial client participation. However, they use | 1 n ∑ n i =1 ∇ F i ( x t ) -∇ f ( x t ) | ≤ ϵ as an assumption to describe such update inconsistency caused by objective inconsistency without any analysis on it. To date, the objective inconsistency caused by partial client participation has not been fully analyzed, even though it is prevalent in FL, even in homogeneous local updates. Our work provides a fundamental convergence analysis on the influence of the objective inconsistency of partial client participation.

Client selection in FL. In general, sampling methods in federated learning (FL) can be classified as biased or unbiased. Unbiased sampling guarantees that the expected value of client aggregation is equal to that of global deterministic aggregation when all clients participate. Conversely, biased sampling may result in suboptimal convergence. A prominent example of unbiased sampling in FL is multinomial sampling (MD), which samples clients based on their data ratio [59, 12]. Additionally, importance sampling (IS), an unbiased sampling method, has been utilized in FL to reduce convergence variance. For instance, [4] use update norm as an indicator of importance to sample clients, [49] sample clients based on data variability, and [40] use test accuracy as an estimation of importance. Meanwhile, various biased sampling strategies have been proposed to speed up training, such as selecting clients with higher loss [7], as many clients as possible under a threshold [45], clients

with larger updates [48], and greedily sampling based on gradient diversity [2]. However, these biased sampling methods can exacerbate the negative effects of objective inconsistency and only converge to a neighboring optimal solution. Another line of research focuses on reinforcement learning for client sampling, treating each client as an agent and aiming to find the optimal action [69, 62, 6, 53, 67]. There are also works that consider online FL, in which the client selection must consider the client's connection ability [44, 17, 26, 71, 46, 8]. Recently, cluster-based client selection has gained some attention in FL [12, 64, 42, 52, 37, 50, 25, 41, 61]. Though clustering adds additional computation and memory overhead, [12, 52] show that it is helpful for sampling diverse clients and reducing variance. Although some studies employ adaptive cluster-based IS to address the issue of slow convergence due to small gradient groups [52, 11], these approaches differ from our method as they still require an additional clustering operation. The proposed DELTA 3 in Algorithm 1 can be viewed as a muted version of the diverse client clustering algorithm, while promising to be unbiased.

Importance sampling. Importance sampling is a statistical method that allows for the estimation of certain quantities by sampling from a distribution that is different from the distribution of interest. It has been applied in a wide range of areas, including Monte Carlo integration [10, 70, 1], Bayesian inference [22, 23], and machine learning [54, 19].

In a recent parallel work, [49] demonstrated mean square convergence of strongly convex federated learning under the assumption of a bounded distance between the global optimal model and the local optimal models.[4] analyzed the convergence of strongly convex and nonconvex federated learning by studying the improvement factor, which is the ratio of the participation variance using importance sampling and the participation variance using uniform sampling. This algorithm dynamically selects clients without any constraints on the number of clients, potentially violating the principle of partial user participation. It is worth noting that both of these sampling methods are based on the gradient norm, ignoring the effect of the direction of the gradient. Other works have focused on the use of importance sampling in the context of online federated learning, where the client selection must consider the client's connection ability. For example, [69] proposed an adaptive client selection method based on reinforcement learning, which takes into account the communication cost and the accuracy of the local model when selecting clients to participate in training. [62] also employed reinforcement learning for adaptive client selection, treating each client as an agent and aiming to find the optimal action that maximizes the accuracy of the global model.[6] introduced a bandit-based federated learning algorithm that uses importance sampling to select the most informative clients in a single communication round. [53] considered the problem of federated learning with imperfect feedback, where the global model is updated based on noisy and biased local gradients, and proposed an importance sampling method to adjust for the bias and reduce the variance of convergence.

B Toy Example and Experiments for Illustrating Our Observation

B.1 Toy example

Figure 5 is a separate illustrated version of each sampling algorithm provided in Figure 3.

We consider a regression problem involving three clients, each with a unique square function: F 1 ( x, y ) = x 2 + y 2 ; F 2 ( x, y ) = 4( x -1 2 ) 2 + 1 2 y 2 ; F 3 ( x, y ) = 3 x 2 + 3 2 ( y -2) 2 . Suppose ( x t , y t ) = (1 , 1) at current round t , the gradients of three clients are ∇ F 1 = (2 , 2) , ∇ F 2 = (4 , 1) , and ∇ F 3 = (6 , -3) . Suppose only two clients are selected to participate in training. The closer the selected user's update is to the global model, the better.

For ideal global model , ∇ F global = 1 3 ∑ 3 i =1 ∇ F i = (4 , 0) , which is the average over all clients.

For FedIS , ∇ F FedIS = 1 2 ( ∇ F 2 + ∇ F 3 ) = (5 , -1) : It tends to select Client 2 and 3 who have large gradient norms, as ∥∇ F 3 ∥ > ∥∇ F 2 ∥ > ∥∇ F 1 ∥ .

For DELTA , ∇ F DELTA = 1 2 ( ∇ F 1 + ∇ F 3 ) = (4 , -1 2 ) : It tends to select Client 1 and 3 who have the largest gradient diversity than that of other clients pair, where the gradient diversity can be formulated by div i = ∥∇ F i ( x t , y t ) -∇ F global ( x t , y t ) ∥ [55, 32].

For FedAvg , ∇ F FedAvg = 1 2 ( ∇ F 1 + ∇ F 2 ) = (3 , 3 2 ) : It assigns each client with equal sampling probability. Compared to FedIS and DELTA, FedAvg is more likely to select Client 1 and 2. To facilitate the comparison, FedAvg is assumed to select Client 1 and 2 here.

From Figure 3, we can observe that the gradient produced by DELTA is closest to that of the ideal global model. Specifically, using L 2 norm as the distance function D , we have D ( ∇ F DELTA , ∇ F global ) < D ( ∇ F FedIS , ∇ F global ) < D ( ∇ F FedAvg , ∇ F global ) . This illustrates the selection of more diverse clients better approaches the ideal global model, thereby making it more efficient.

(a) Overview of different methods.

(b) FedAvg.

(c) FedIS.

(d) DELTA.

Client

1

Client

1

Ct+1,FedAug

Client2

Ct+1,global

t+1,DELTA

Ct+1,FedIS

Ct+1,FedAvg

Client2

Ct+1,global

Ct

ct

Client3

Client3

Client

1

Client

1

Client2

Ct+1,global

Ct+1,FedIS

Client2

Ct+1,global

Ct+1,DELTA

Ct

Ct

Client3

Client3

B.2 Experiments for illustrating our observation.

Experiment setting. For the experiments to illustrate our observation in the introduction, we apply a logistic regression model on the non-iid MNIST dataset. 10 clients are selected from 200 clients to participate in training in each round. We set 2 cluster centers for cluster-based IS. And we set the mini batch-size to 32, the learning rate to 0.01, and the local update time to 5 for all methods. We run 500 communication rounds for each algorithm. We report the average of each round's selected clients' gradient norm and the minimum of each round's selected clients' gradient norm.

Performance of gradient norm. We report the gradient norm performance of cluster-based IS and IS to show that cluster-based IS selects clients with small gradients. As we mentioned in the introduction, the cluster-based IS always selects some clients from the cluster with small gradients, which will slow the convergence in some cases. We provide the average gradient norm comparison between IS and cluster-based IS in Figure 6(a). In addition, we also provide the minimal gradient norm comparison between IS and cluster-based IS in Figure 6(b).

Performance of removing small gradient clusters. We report on a comparison of the accuracy and loss performance between vanilla cluster-based IS and the removal of cluster-based IS with small gradient clusters. Specifically, we consider a setting with two cluster centers. After 250 rounds, we replace the clients in the cluster containing the smaller gradient with the clients in the cluster containing the larger gradient while maintaining the same total number of participating clients. The experimental results are shown in Figure 7. We can observe that vanilla cluster-based IS performs

0

100

200

300

400

500

Rounds

2

3

4

5

6

7

8

average gradient norm

IS

cluster-based IS

(a) Average gradient norm comparison

0

100

200

300

400

500

Rounds

1

2

3

4

5

6

7

minimal gradient norm

IS

cluster-based IS

(b) Minimal gradient norm comparison

worse than cluster-based IS without small gradients, indicating that small gradients are a contributing factor to poor performance.

0

50

100

150

200

250

300

350

400

450

500

Rounds

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

cluster-based IS without small gradients

(a) Accuracy performance comparison

0

100

200

300

400

500

Rounds

0.5

1

1.5

2

2.5

Loss

cluster-based IS without small gradients

(b) Loss performance comparison

C Techniques

Here, we present some technical lemmas that are useful in the theoretical proof. We substitute 1 m for n i N to simplify the writing in all subsequent proofs. n i N is the data ratio of client i . All of our proofs can be easily extended from f ( x t ) = 1 m ∑ m i =1 F i ( x t ) to f ( x t ) = ∑ m i =1 n i N F i ( x t ) .

Lemma C.1. (Unbiased Sampling). Importance sampling is unbiased sampling. E ( 1 n ∑ i ∈ S t 1 mp i ∇ F i ( x t )) = 1 m ∑ m i =1 ∇ F i ( x t ) , no matter whether the sampling is with replacement or without replacement.

Lemma C.1 proves that the importance sampling is an unbiased sampling strategy, either in sampling with replacement or sampling without replacement.

Proof. For with replacement:

For without replacement:

where I m ≜ { 1 if x l ∈ S t , 0 otherwise .

In the expectation, there are three sources of stochasticity. They are client sampling, local SGD, and the filtration of x t . Therefore, the expectation is taken over all of these sources of randomness. Here, S represents the sources of stochasticity other than client sampling. More precisely, S represents the filtration of the stochastic process x j , j = 1 , 2 , 3 . . . at time t and the stochasticity of local SGD.

Lemma C.2 (update gap bound) .

where the first equation follows from E [ x -E ( x )] 2 = E [ x 2 ] -[ E ( x )] 2 and Lemma C.1.

For ease of understanding, we give a detailed derivation of the Lemma C.2.

where E ( x | S ) means the expectation on x over the sampling space. We have E ( ∥∇ ˜ f ( x t ) | S ) = ∇ f ( x t ) and E ( ∥∇ f ( x t ) ∥ 2 | S ) = ∥∇ f ( x t ) ∥ 2 ( ∥∇ f ( x ) ∥ is a constant for stochasticity S and the expectation over a constant is the constant itself.) Therefore, we conclude

We can further take the expectation on both sides of the inequality according to our needs, without changing the relationship.

The following lemma follows from Lemma 4 of [47], but with a looser condition Assumption 3, instead of σ 2 G bound. With some effort, we can derive the following lemma:

Lemma C.3 (Local updates bound.) . For any step-size satisfying η L ≤ 1 8 LK , we can have the following results:

Proof.

Unrolling the recursion, we obtain:

In the following Proposition, we will demonstrate that the convergence rate in this paper with the relaxed version of Assumption 3 remains unchanged.

Proposition C.4 (convergence under relaxed Assumption 3 [24]) . The relaxed version of Assumption 3 in this paper is:

Since we have f ( x ) -f inf ≤ f 0 -f inf ≤ F , where F is a positive constant. This implies that we can substitute σ g with 2 BF + σ G in all analyses without altering the outcomes (one can directly conclude this from using the above bound in Lemma C.3). In the final convergence rate, it is straightforward to see that the convergence rate remains unchanged, yet the constant term σ g becomes 2 BF + σ G .

Thus, we can assert that we have furnished the analysis under the relaxed assumption condition.

D Convergence of FedIS, Proof of Theorem 3.1

The complete version of FedIS algorithm is shown below:

We first restate the convergence theorem (Theorem 3.1) more formally, then prove the result for the nonconvex case.

Theorem D.1. Under Assumptions 1-3 and the sampling strategy FedIS, the expected gradient norm will converge to a stationary point of the global objective. More specifically, if the number of communication rounds T is predetermined and the learning rate η and η L are constant, then the expected gradient norm will be bounded as follows:

where F = f ( x 0 ) -f ( x ∗ ) , M 2 = σ 2 L + 4 Kσ 2 G , and the expectation is over the local datasets samples among workers.

Require: initial weights x 0 , global learning rate η , local learning rate η l , number of training rounds T

Let η L < min (1 / (8 LK ) , C ) , where C is obtained from the condition that 1 2 -10 L 2 K 2 ( A 2 + 1) η 2 L -L 2 ηK ( A 2 +1) 2 n η L > 0 ,and η ≤ 1 / ( η L L ) , it then holds that:

where c is a constant that satisfies 1 2 -10 L 2 K 2 ( A 2 + 1) η 2 L -L 2 ηK ( A 2 +1) 2 n η L > c > 0 , and V ( 1 mp t i ˆ g t i ) = E ∥ 1 mp t i ˆ g t i -1 m ∑ m i =1 ˆ g t i ∥ 2 .

Corollary D.2. Suppose η L and η are such that the conditions mentioned above are satisfied, η L = O ( 1 √ TKL ) and η = O ( √ Kn ) , and let the sampling probability be FedIS (75) . Then for sufficiently large T, the iterates of Theorem 3.1 satisfy:

Proof.

where (a1) follows from the Lipschitz continuous condition. The expectation is conditioned on everything prior to the current step k of round t . Specifically, it is taken over the sampling of clients, the sampling of local data, and the current round's model x t .

Firstly we consider A 1 :

where (a2) follows from Assumption 2 and LemmaC.1. (a3) is due to ⟨ x, y ⟩ = 1 2 [ ∥ x ∥ 2 + ∥ y ∥ 2 -∥ x -y ∥ 2 ] and (a4) comes from Assumption 1.

Then we consider A 2 . Let ˆ g t i = ∑ K -1 k =0 g t i,k = ∑ K -1 k =0 ∇ F i ( x i t,k , ξ i t,k )

The third equality follows from independent sampling.

Specifically, for sampling with replacement, due to every index being independent, we utilize E ∥ x 2 1 + ... + x n ∥ 2 = E [ ∥ x 1 ∥ 2 + ... + ∥ x n ∥ 2 ] .

For sampling without replacement:

From the above, we observe that it is possible to achieve a speedup by sampling from the distribution that minimizes V ( 1 mp t i ˆ g t i ) . Furthermore, as we discussed earlier, the optimal sampling probability is p ∗ i = ∥ ˆ g t i ∥ ∑ i m =1 ∥ ˆ g t i ∥ . For MD sampling [31], which samples according to the data ratio, the optimal sampling probability is p ∗ i,t = q i ∥ ˆ g t i ∥ ∑ i m =1 q i ∥ ˆ g t i ∥ , where q i = n i N .

Now we substitute the expressions of A 1 and A 2 :

where the last inequality follows from ( ηη L 2 K -Lη 2 η 2 L 2 ) ≥ 0 if ηη l ≤ 1 KL , and (a9) holds because there exists a constant c > 0 (for some η L ) satisfying 1 2 -10 L 2 1 m ∑ m i -1 K 2 η 2 L ( A 2 +1) > c > 0 . Rearranging and summing from t = 0 , . . . , T -1 ,we have:

Which implies:

where

D.1 Proof for convergence rate of FedIS (Theorem 3.1) under Assumption 1-3.

In this section, we compare the convergence rate of FedIS with and without Assumption 4. For comparison, we first provide the convergence result under Assumption 4.

First we show Assumption 4 can be used to bound the update variance V ( 1 mp t i ˆ g t i ) , and under the sampling probability FedIS (73):

While for using Assumption 3 instead of additional Assumption 4, we can also bound the update variance:

We replace the variance back to equation (35):

This shows that the requirement for η L is different. It needs that there exists a constant c > 0 (for some η L ) satisfying 1 2 -10 L 2 K 2 η 2 L ( A 2 + 1) -Lηη L K ( A 2 +1) 2 n > c > 0 . One can still guarantee that there exists a constant for η L to satisfy this inequality according to the properties of quadratic functions. Specifically, for the quadratic equation -10 L 2 K 2 ( A 2 +1) η 2 L -LηK ( A 2 +1) 2 n η L + 1 2 , we know that -10 L 2 K 2 ( A 2 +1) < 0 , -LηK ( A 2 +1) 2 n and 1 2 > 0 . Based on the solution of quadratic equations, we can ensure that there exists a η L > 0 solution.

Then we can substitute equation (35) with equation (41) and let η L = O ( 1 √ TKL ) and η = O ( √ Kn ) , yielding the convergence rate of FedIS under Assumptions 1- 3:

E Convergence of DELTA. Proof of Theorem 3.5

E.1 Convergence rate with improved analysis method for getting DELTA

As we see FedIS can only reduce the update variance term in Φ . Since we want to reduce the convergence variance as much as possible, the other term σ L and σ G still needs to be optimized. However, it is not straightforward to derive the optimization problem from Φ . In order to further reduce the variance in Φ (cf. 4), i.e., local variance ( σ L ) and global variance ( σ G ), we divide the convergence of the global objective into a surrogate objective and an update gap and analyze them separately. The analysis framework is shown in Figure 8.

surrogateobjectivef(ct)

[Lemma

Global objective Vf(ct)

B.1

EIVf(x）I1²=E|If（x）I1²+x²

Lemma

Theorem

Objective Gap Xt

B.2

x²≤EIf(ct)2

4.3

/Varisboundedby

the convergence of

Convergencevarianceofglobalobjective

surrogateobjective

Convergencevarianceofsurrogateobjective

eq(12):

eq(7)or eq(8):

Theorem 4.1

Φ = A10² + A20 + As Var(m9)

=B²+B

Corollary 4.2

E.1

FedIS:p=

DELTA:p=

=1√a1VF;(x)Vf(x)²+a20²

+（x)(x)△

Optimizing thevariancetermw.r.ttop

Optimizing the variance term w.r.t to p

As for the update gap, inspired by the expression form of the update variance, we formally define it as follows:

Definition E.1 (Update gap) . In order to measure the update inconsistency, we define the update gap:

Here, the expectation is taken over the distribution of all clients. When all clients participate, we have χ 2 t = 0 . The update inconsistency exists as long as only a partial set of clients participate.

The update gap is a direct manifestation of the objective inconsistency in the update process. The presence of an update gap makes the analysis of the global objective different from the analysis of the surrogate objective. However, by ensuring the convergence of the update gap, we can re-derive the convergence result for the global objective. Formally, the update gap allows us to connect global objective convergence and surrogate objective convergence as follows:

The equation follows from the property of unbiasedness, as shown in Lemma C.1.

To deduce the convergence rate of the global objective, we begin by examining the convergence analysis of the surrogate objective.

Theorem E.2 (Convergence rate of surrogate objective) . Under Assumption 1-3 and let local and global learning rates η and η L satisfy η L < 1 / ( √ 40 KL √ 1 n ∑ l m =1 1 mp t l ) and ηη L ≤ 1 / KL , the minimal gradient norm of surrogate objective will be bounded as below:

where f 0 = f ( x 0 ) , f ∗ = f ( x ∗ ) , the expectation is over the local dataset samples among workers.

˜ Φ is the new combination of variance , representing combinations of local variance and client gradient diversity.

For sampling without replacement:

For sampling with replacement:

where ζ G,i represents client gradient diversity: ζ G,i = ∥∇ F i ( x t ) -∇ f ( x t ) ∥ 4 , and ˜ c is a constant. The proof of Theorem E.2 is provided in Appendix E.2.1 and Appendix E.2.2. Specifically, the proof for sampling with replacement is shown in Appendix E.2.1, while the proof for sampling without replacement is shown in Appendix E.2.2.

Remark E.3. We observe that there is no update variance in ˜ Φ , but the local variance and global variance are still present. Additionally, the new combination of variance ˜ Φ can be minimized by optimizing the sampling probability, as will be shown later.

Derive the convergence from surrogate objective to global objective. As shown in Lemma C.1, unbiased sampling guarantees that the expected partial client updates are equal to the participation of all clients. With sufficient training rounds, unbiased sampling can ensure that the update gap χ 2 will converge to zero. However, we still need to know the convergence speed of χ 2 t to recover the convergence rate of the global objective. Fortunately, we can bound the convergence behavior of χ 2 t by the convergence rate of the surrogate objective according to Definition E.1 and Lemma C.2. This means that the update gap can achieve at least the same convergence rate as the surrogate objective.

Corollary E.4 (New convergence rate of global objective) . Under Assumption 1-3 and based on the above analysis that update variance is bounded, the global objective will converge to a stationary point. Its gradient is bounded as:

Theorem E.5 (Restate of Theorem 3.5) . Under Assumptions 1-3 and the same conditions as in Theorem 3.1, the minimal gradient norm of the surrogate objective will be bounded as follows by setting η L = 1 √ TKL and η √ Kn . Let the local and global learning rates η and η L satisfy η L < 1 √ 40 KL √ 1 n ∑ l m =1 1 mp t l and ηη L ≤ 1 KL . Under Assumptions 1-3 and with partial worker participation, the sequence of outputs x k generated by Algorithm 1 satisfies:

where F = f ( x 0 ) -f ( x ∗ ) , and the expectation is over the local dataset samplings among workers. c is a constant. ζ G,i is defined as client gradient diversity: ζ G,i = ∥∇ F i ( x t ) -∇ f ( x t ) ∥ .

For sample with replacement: ˜ Φ = 5 L 2 Kη 2 L 2 m 2 ∑ m l =1 1 p t l ( σ 2 L,l +4 Kζ 2 G,l ) + Lη L η 2 n ∑ m l =1 1 m 2 p t l σ 2 L,i .

For sampling without replacement: ˜ Φ = 5 L 2 Kη 2 L 2 mn ∑ m l =1 1 p t l ( σ 2 L,l +4 Kζ 2 G,l ) + Lη L η 2 n ∑ m l =1 1 m 2 p t l σ 2 L,l .

Remark E.6 (Condition of η L ) . Here, though the condition expression for η L relies on a dynamic sampling probability p t l , we can still guarantee that there a constant η L satisfies this condition.

Specifically, one can substitute the optimal sampling probability 1 p t i = ∑ j m =1 √ α 1 ζ 2 G,j + α 2 σ 2 L,j √ α 1 ζ 2 G,i + α 2 σ 2 L,i back to the above inequality condition. As long as the gradient ∇ F i ( x t ) is bounded, we can ensure 1 m 2 ∑ m i =1 ∑ j m =1 √ α 1 ζ 2 G,j + α 2 σ 2 L,j √ α 1 ζ 2 G,i + α 2 σ 2 L,i ≤ max j √ α 1 ζ 2 G,j + α 2 σ 2 L,j min i √ α 1 ζ 2 G,i + α 1 σ 2 L,i ≤ ˜ G , therefore 1 2 √ 10( A 2 +1) KL √ √ √ √ 1 m 2 ∑ i m =1 ∑ j m =1 √ α 1 ζ 2 G,j + α 2 σ 2 L,j √ α 1 ζ 2 G,i + α 2 σ 2 L,i ≥ 1 2 √ 10( A 2 +1) KL √ ˜ G ≥ C , where ˜ G and C are positive constants. Thus, we can always find a constant η L to satisfy this inequality under dynamic sampling probability p t i .

Corollary E.7 (Convergence rate of DELTA) . Suppose η L and η are such that the conditions mentioned above are satisfied, η L = O ( 1 √ TKL ) and η = O ( √ Kn ) . Then for sufficiently large T, the iterates of Theorem 3.5 satisfy:

Lemma E.8. For any step-size satisfying η L ≤ 1 8 LK , we can have the following results:

where ζ G,i = ∥∇ F ( x t ) - ∇ f ( x t ) ∥ , and the expectation is over local SGD and filtration of x t , without the stochasticity of client sampling.

Proof.

Unrolling the recursion, we get:

E.2 Proof for Theorem E.2.

In Section E.2.1 and Section E.2.2, we provide the proof for Theorem E.2. Specifically, the proof for sampling with replacement is shown in Appendix E.2.1, while the proof for sampling without replacement is shown in Appendix E.2.2.

E.2.1 Sample with replacement

where ˜ Φ = 5 L 2 Kη 2 L 2 m 2 ∑ m l =1 1 p t l ( σ 2 L +4 Kζ 2 G,i ) + Lη L η 2 n ∑ m l =1 1 m 2 p t l σ 2 L .

Proof.

Where (a1) follows from the Lipschitz continuity condition. Here, the expectation is over the local data SGD and the filtration of x t . However, in the next analysis, the expectation is over all randomness, including client sampling .This is achieved by taking expectation on both sides of the above equation over client sampling.

To begin, let us consider A 1 :

where (a2) follows from Assumption 2, and (a3) is due to ⟨ x, y ⟩ = 1 2 [ ∥ x ∥ 2 + ∥ y ∥ 2 -∥ x -y ∥ 2 ] for x = √ Kη L ∇ ˜ f ( x t ) and y = √ η L K [ -1 n ∑ i ∈ S t 1 mp t i ∑ K -1 k =0 ∇ F i ( x i t,k ) + K ∇ ˜ f ( x t )] .

To bound A 1 , we need to bound the following part:

where (a4) follows from the fact that E | x 1 + · · · + x n | 2 ≤ n E ( | x 1 | 2 + · · · + | x n | 2 ) , (a5) is a consequence of Assumption 1, and (a6) is a result of Lemma E.8. Combining the above expressions, we obtain:

Next, we consider bounding A 2 :

where S represents the whole sample space and (a7) is due to Assumption 2.

Now we substitute the expressions for A 1 and A 2 and take the expectation over the client sampling distribution on both sides. It should be noted that the derivation of A 1 and A 2 above is based on considering the expectation over the sampling distribution:

where (a8) comes from Lemma C.2, (a9) follows from ( ηη L 2 K -Lη 2 η 2 L 2 ) ≥ 0 if ηη l ≤ 1 KL , and (a10) holds because there exists a constant c > 0 satisfying ( 1 2 -20 K 2 η 2 L L 2 ( A 2 +1) m 2 ∑ m l =1 1 p t l ) > c > 0 if η L < 1 2 √ 10( A 2 +1) KL √ 1 m ∑ l m =1 1 mp t l .

Rearranging and summing from t = 0 , . . . , T -1 , we have:

Which implies:

where ˜ Φ = 5 L 2 Kη 2 L 2 m 2 ∑ m l =1 1 p t l ( σ 2 L +4 Kζ 2 G,i ) + Lη L η 2 n ∑ m l =1 1 m 2 p t l σ 2 L .

E.2.2 Sample without replacement

where ˜ Φ = 5 L 2 Kη 2 L 2 mn ∑ m l =1 1 p t l ( σ 2 L +4 Kζ 2 G,i ) + Lη L η 2 n ∑ m l =1 1 m 2 p t l σ 2 L .

Proof.

Where the first inequality follows from Lipschitz continuous condition. The expectation here is taken over both the local SGD and the filtration of x t . However, in the subsequent analysis, the expectation is taken over all sources of randomness, including client sampling.

Similarly, we consider A 1 first:

Since x i are sampled from S t without replacement, this causes pairs x i 1 and x i 2 to no longer be independent. We introduce the activation function as follows:

Then we obtain the following bound:

̸

̸

where (b1) follows from ∥ ∑ m i =1 t i ∥ 2 = ∑ i ∈ [ m ] ∥ t i ∥ 2 + ∑ i = j ⟨ t i , t j ⟩ c 1 = ∑ i ∈ [ m ] m ∥ t i ∥ 2 -1 2 ∑ i = j ∥ t i -t j ∥ 2 ((c1) here is due to ⟨ x, y ⟩ = 1 2 [ ∥ x ∥ 2 + ∥ y ∥ 2 -∥ x -y ∥ 2 ] ), (b2) is due to E ∥ x 1 + · · · + x n ∥ 2 ≤ n E ( ∥ x 1 ∥ 2 + · · · + ∥ x n ∥ 2 ) , and (b3) comes from Lemma E.8.

̸

Therefore, we have the bound of A 1 :

The expression for A 2 is as follows:

Now we substitute the expressions for A 1 and A 2 and take the expectation over the client sampling distribution on both sides. It should be noted that the derivation of A 1 and A 2 above is based on considering the expectation over the sampling distribution:

Also, for (b4), step sizes need to satisfy ( ηη L 2 K -Lη 2 η 2 L 2 ) ≥ 0 if ηη l ≤ 1 KL , and there exists a constant c > 0 satisfying ( 1 2 -20 K 2 η 2 L L 2 ( A 2 +1) mn ∑ m l =1 1 p t l ) > c > 0 if η L < 1 2 √ 10( A 2 +1) KL √ 1 n ∑ l m =1 1 mp t l .

Rearranging and summing from t = 0 , . . . , T -1 ,we have:

Which implies:

Remark E.9. ζ G used in DELTA can be easily transformed into a σ G related term, thus it is fair to compare DELTA with FedIS. In particular, by taking the expectation on ζ G , it equates to E ∥∇ F i ( x t ) - ∇ f ( x t ) ∥ 2 . As demonstrated in [1], one can derive E ∥∇ F i ( x t ) - ∇ f ( x t ) ∥ 2 = E ∥∇ F i ( x t ) ∥ 2 - ∥∇ f ( x t ) ∥ 2 ≤ A ∥∇ f ( x t ) ∥ 2 + σ 2 G . Shifting A ∥∇ f ( x t ) ∥ 2 to the left side of the convergence result, ζ G can be directly transformed into σ G .

F Proof of the Optimal Sampling Probability

F.1 Sampling probability FedIS

Corollary F.1 (Optimal sampling probability for FedIS) .

Solving the above optimization problem, we obtain the expression for the optimal sampling probability:

where ˆ g t i = ∑ K -1 k =0 g i k is the sum of the gradient updates across multiple updates.

Recall Theorem 3.1; only the last variance term in the convergence term Φ is affected by sampling. In other words, we need to minimize the variance term with respect to probability. We formalize this as follows:

This optimization problem can be solved in closed form using the KKT conditions. It is straightforward to verify that the solution to the optimization problem is:

Under the optimal sampling probability, the variance will be:

Therefore, the variance term is bounded by:

Remark: If the uniform distribution is adopted with p t i = 1 m , it is easy to observe that the variance of the stochastic gradient is bounded by ∑ i m =1 | g i | 2 m .

this implies that importance sampling does improve convergence rate, especially when ( ∑ i m =1 ∥ g i ∥ ) 2 ∑ i m =1 ∥ g i ∥ 2 << m .

F.2 Sampling probability of DELTA

Our result is of the following form:

It is easy to see that the sampling strategy only affects ˜ Φ . To enhance the convergence rate, we need to minimize ˜ Φ with respect to p t l . As shown, the expression for ˜ Φ with and without replacement is similar, and only differs in the values of n and m . Here, we will consider the case with replacement. Specifically, we need to solve the following optimization problem:

Solving this optimization problem, we find that the optimal sampling probability is:

For simplicity, we rewrite the optimal sampling probability as:

where α 1 = 20 K 2 Lη L , α 2 = 5 KLη L + η n .

Remark: Now, we will compare this result with the uniform sampling strategy:

this implies that our sampling method does improve the convergence rate (our sampling approach might be n times faster in convergence than uniform sampling), especially when ( ∑ l m =1 √ α 1 ζ 2 G,l + α 2 σ 2 L,l ) 2 ∑ l m =1 ( √ α 1 ζ 2 G,l + α 2 σ 2 L,l ) 2 << m .

G Convergence Analysis of The Practical Algorithm

In order to provide the convergence rate of the practical algorithm, we need an additional Assumption 4 ( ∥∇ F i ( x ) ∥ 2 ≤ G 2 , ∀ i ). This assumption tells us a useful fact that will be used later:

It can be shown that ∥∇ F i ( x t,k , ξ t,k ) / ∇ F i ( x s,k , ξ s,k ) ∥ ≤ U for all i and k , where the subscript s refers to the last round in which client i participated, and U is a constant upper bound. This tells us

For uniform p l = 1 m :

Norm of gradient

Norm of gradient

8

7

6

6

^

4

4

r2

2

1

0

m

409

7

m

wuou jeuxew

minimal norm

maximal norm

minimal norm

(a) FedIS's gradient norm on MNIST

(b) FedIS' gradient norm on FashionMNIST

Norm of gradient diversity

Norm of gradient diversity

7

6

5

4

4

3

2

1

0

0

604

7

1.

minimal norm

maximal norm

minimal norm

(a) Gradient diversity norm of DELTA on MNIST (b) Gradient diversity norm of DELTA on FashionMNIST

that the change in the norm of the client's gradient is bounded. U comes from the following inequality constraint procedure:

We establish the upper bound U based on two factors: (1) Assumption 4, and (2) the definition of importance sampling E q ( z ) ( F i ( z )) = E p ( z ) ( q i ( z ) /p i ( z ) F i ( z )) , where there exists a positive constant γ such that p i ( z ) ≥ γ > 0 . Thus, for p s i = ˆ g s i ∑ j ˆ g s j ≥ γ , we can easily ensure ∥ g t i ∥ ∥ g s i ∥ ≤ U since ˆ g s i > 0 is consistently bounded.

In general, the gradient norm tends to become smaller as training progresses, which leads to ∥∇ F i ( x t,k , ξ t,k ) / ∇ F i ( x s,k , ξ s,k ) ∥ going to zero. Even if there are some oscillations in the gradient norm, the gradient will vary within a limited range and will not diverge to infinity. Figures 9 and Figure 10 depict the norms of gradients and gradient diversity across all clients in each round. Notably, these figures demonstrate that in the case of practical IS and practical DELTA, the change ratio of both gradient and gradient diversity remains limited, with the maximum norm being under 8 and the minimum norm exceeding 0.5.

Based on Assumption 4 and Assumption 3, we can re-derive the convergence analysis for both convergence variance Φ (4) and ˜ Φ (46). In particular, for Assumption 3 ( E ∥∇ F i ( x ) ∥ 2 ≤ ( A 2 + 1) ∥∇ f ( x ) ∥ 2 + σ 2 G ), we use σ G,s and σ G,t instead of a unified σ G for the sake of comparison.

Specifically, Φ = 1 c [ 5 η 2 L L 2 K 2 m ∑ m i =1 ( σ 2 L + 4 Kσ 2 G ) + ηη L L 2 m σ 2 L + Lηη L 2 nK V ( 1 mp t i ˆ g t i )] , where ˆ g t i = ∑ K k =1 ∇ F i ( x k,s , ξ k,s ) . With the practical sampling probability p s i of FedIS:

According to Assumption 4, we know ∥ ˆ g t i ˆ g s i ∥ 2 = ∥ ∑ K k =1 ∇ F i ( x i t,k ,ξ i t,k ) ∑ K k =1 ∇ F i ( x i s,k ,ξ i s,k ) ∥ ≤ U 2 . Then we get

Similar to the previous proof, based on Assumption 3. we can get the new convergence rate:

where M = σ 2 L +4 Kσ 2 G,s .

Remark G.1 (Discussion on U and convergence rate.) . It is worth noting that ∥∇ F i ( x t,k , ξ t,k ) / ∇ F i ( x s,k , ξ s,k ) ∥ is typically relatively small because the gradient tends to go to zero as the training process progresses. This means that U can be relatively small, more specifically, U < 1 in the upper bound term O ( KU 2 σ 2 G,s √ nKT ) . However, this does not necessarily mean that the practical algorithm is better than the theoretical algorithm because the values of σ G are different, as we stated at the beginning. Typically, the value of σ G,s for the practical algorithm is larger than the value of σ G,t , which also comes from the fact that the gradient tends to go to zero as the training process progresses. Additionally, due to the presence of the summation over both i and k , the gap between σ G,s and σ G,t is multiplied, and σ G,s /σ G,t ∼ m 2 K 2 1 U 2 . Thus, the practical algorithm leads to a slower convergence than the theoretical algorithm.

Similarly, as long as the gradient is consistently bounded, we can assume that ∥∇ F i ( x t ) -∇ f ( x t ) ∥ / ∥∇ F i ( x s ) - ∇ f ( x s ) ∥ ≤ ˜ U 1 ≤ ˜ U and ∥ σ L,t /σ L,s ∥ ≤ ˜ U 2 ≤ ˜ U for all i , where σ 2 L,s = E [∥ ∥ ∇ F i ( x s , ξ i s ) -∇ F i ( x s ) ∥ ∥ ] . Then, we can obtain a similar conclusion by following the same analysis on ˜ Φ .

Specifically, we have ˜ Φ = Lη L 2 m 2 c ∑ m i =1 1 p s i ( α 1 ζ 2 G,i + α 2 σ 2 L,i ) , where α 1 and α 2 are constants defined in (13). For the sake of comparison of different participation rounds s and t , we rewrite the symbols as ζ i G,s and σ i L,s . Then, using the practical sampling probability p s i of DELTA, and letting R s i = √ α 1 ζ i G,s 2 + α 2 σ i L,s 2 , we have:

Therefore, compared to the theoretical algorithm of DELTA, the practical algorithm of DELTA has the following convergence rate:

This discussion of the effect of ˜ U on the convergence rate is similar to the discussion of U in Remark G.1.

H Additional Experiment Results and Experiment Details.

H.1 Experimental Environment

For all experiments, we use NVIDIA GeForce RTX 3090 GPUs. Each simulation trail with 500 communication rounds and three random seeds.

H.2 Experiment setup

Setup for the synthetic dataset. To demonstrate the validity of our theoretical results, we first conduct experiments using logistic regression on synthetic datasets. Specifically, we randomly generate ( x, y ) pairs using the equation y = log ( ( Ax -b ) 2 2 ) with given values for A i and b i as training data for clients. Each client's local dataset contains 1000 samples. In each round, we select 10 out of 20 clients to participate in training (we also provide the results of 10 out of 200 clients in Figure 13).

To simulate gradient noise, we calculate the gradient for each client i using the equation g i = ∇ f i ( A i , b i , D i ) + ν i , where A i and b i are the model parameters, D i is the local dataset for client i , and ν i is a zero-mean random variable that controls the heterogeneity of client i . The larger the value of E ∥ ν i ∥ 2 , the greater the heterogeneity of client i .

We demonstrate the experiment on different functions with different values of A and b . Each function is set with noise levels of 20, 30, and 40 to illustrate our theoretical results. To construct different functions, we set A = 8 , 10 and b = 2 , 1 , respectively, to observe the convergence behavior of different functions.

All the algorithms run in the same environment with a fixed learning rate of 0 . 001 . We train each experiment for 2000 rounds to ensure that the global loss has a stable convergence performance.

Setup for FashionMNIST and CIFAR-10. To evaluate the performance of DELTA and FedIS, we train a two-layer CNN on the non-iid FashionMNIST dataset and a ResNet-18 on the non-iid CIFAR-10 dataset, respectively. CIFAR-10 is composed of 32 × 32 images with three RGB channels, belonging to 10 different classes with 60000 samples.

The "non-iid" follows the idea introduced in [66, 16], where we leverage Latent Dirichlet Allocation (LDA) to control the distribution drift with the Dirichlet parameter α . Larger α indicates smaller drifts. Unless otherwise stated, we set the Dirichlet parameter α = 0 . 5 .

Unless specifically mentioned otherwise, our studies use the following protocol: all datasets are split with a parameter of α = 0 . 5 , the server chooses n = 20 clients according to our proposed probability from the total of m = 300 clients, and each is trained for T = 500 communication rounds with K = 5 local epochs. The default local dataset batch size is 32. The learning rates are set the same for all algorithms, specifically lr global = 1 and lr local = 0 . 01 .

All algorithms use FedAvg as the backbone. We compare DELTA, FedIS and Cluster-based IS with FedAvg on different datasets with different settings.

Setup for Split-FashionMNIST. In this section, we evaluate our algorithms on the splitFashionMNIST dataset. In particular, we let 10% clients own 90% of the data, and the detailed split data process is shown below:

Divide the dataset by labels. For example, divide FashionMNIST into 10 groups, and assign each client one label

Random select one client

Reshuffle the data in the selected client

Equally divided into 100 clients

Setup for LEAF. To test our algorithm's efficiency on diverse real datasets, we use the non-IID FEMNIST dataset and non-IID CelebA dataset in LEAF, as given in [3]. All baselines use a 4-layer CNN for both datasets with a learning rate of lr local = 0 . 1 , batch size of 32, sample ratio of 20% and

(a)

A

= 8

, b

= 2

, ν

= 20

(b)

A

= 8

, b

= 2

, ν

= 30

(c)

A

= 8

, b

= 2

, ν

= 40

(d)

A

= 10

, b

= 1

, ν

= 20

(e)

A

= 10

, b

= 1

, ν

= 30

(f)

A

= 10

, b

= 1

, ν

= 40

FedAvg

FedAvg

FedAvg

0

FedIS

Power of choice

Power of Choice

FedIS

0

Powerofchoice

FedIS

-2

DELTA

DELTA

2

DELTA

SSO

-4

SSO|

6

SSO

6

6

8-

8-

10

10

8

12

10

0

250

500

750

1000

1250

1500

1750

2000

0

250

500

750

1000

1250

1500

1750

2000

0

250

500

750

1000

1250

1500

1750

2000

communication rounds

communication rounds

communication rounds

FedAvg

FedAvg

0

FedAvg

2

Powerofchoice

FedIS

0

Powerofchoice

FedIS

FedIS

Power of choice

0

DELTA

2

DELTA

DELTA

SSO1

2

SS01

SSO1

-6

-4

6

-6

-8

-10

8

8-

10

-10

12

0

250

500

750

1000

1250

1500

1750

2000

0

250

500

750

1000

1250

1500

1750

2000

0

250

500

750

1000

1250

1500

1750

2000

communication rounds

communication rounds

communication rounds

communication round of T = 500 . The reported results are averaged over three runs with different random seeds.

The implementation detail of different sampling algorithms. The power-of-choice sampling method is proposed by [7]. The sampling strategy is to first sample 20 clients randomly from all clients, and then choose 10 of the 20 clients with the largest loss as the selected clients. FedAvg samples clients according to their data ratio. Thus, FedAvg promises to be unbiased, which is given in [12, 31] to be an unbiased sampling method. As for FedIS, the sampling strategy follows Equation (5). For cluster-based IS, it first clusters clients following the gradient norm and then uses the importance sampling strategy similar to FedIS in each cluster. And for DELTA, the sampling probability follows Equation (13). For the practical implementation of FedIS and DELTA, the sampling probability follows the strategy described in Section 4.

H.3 Additional Experimental Results

Performance of algorithms on the synthetic dataset. We display the log of the global loss of different sampling methods on synthetic dataset in Figure 11, where the Power-of-Choice is a biased sampling strategy that selects clients with higher loss [7].

We also show the convergence behavior of different sampling algorithms under small noise, as shown in Figure 12.

To simulate a large number of clients, we increased the client number from 20 to 200, with only 10 clients participating in each round. The results in Figure 13 demonstrate the effectiveness of DELTA.

Convergence performance of theoretical DELTA on split-FashionMNIST and practical DELTA on FEMNIST. Figure 14(a) illustrates the theoretical DELTA outperforms other methods in convergence speed. Figure 14(b) indicates that cluster-based IS and practical DELTA exhibit rapid initial accuracy improvement, while practical DELTA and practical IS achieve higher accuracy in the end.

Ablation study for DELTA with different sampled numbers. Figure 15 shows the accuracy performance of practical DELTA algorithms on FEMNIST with different sampled numbers of clients. In particular, the larger number of sampled clients, the faster the convergence speed is. This is consistent with our theoretical result (Corollary 4.2).

(a)

ν

= 10

(b)

ν

= 5

(c)

ν

= 1

(d)

ν

= 0

.

5

(e)

ν

= 0

.

1

0

fedavg

fedavg

fedavg

power of choice

0

power of choice

0

power of choice

-2

FedIS

FedIS

FedIS

-4

DELTA

2

DELTA

2

DELTA

6

SS01

4

SS01

-4

6

8

6

8

10

8-

10

12

10

12

0

250

500

750

1000

1250

1500

1750

2000

0

250

500

750

1000

1250

1500

1750

2000

0

250

500

750

1000

1250

1500

1750

2000

communication rounds

communication rounds

communication rounds

fedavg

fedavg

0

power of choice

0

power of choice

FedIS

FedIS

2

DELTA

2

DELTA

SSO1

-4

SSO1

-4

6

6

8

8

10

-10

0

250

500

750

1000

1250

1500

1750

2000

0

250

500

750

1000

1250

1500

1750

2000

communication rounds

communication rounds

(a)

ν

= 30

(b)

ν

= 20

(c)

ν

= 10

(d)

ν

= 5

(e)

ν

= 1

fedavg

fedavg

fedavg

0

power of choice

FedIS

0

power of choice

FedIS

0

power of choice

FedIS

2

DELTA

5

DELTA

5

DELTA

SS0

-4

SSO1

10

SSO1

10

6

15

-15

8

20

20

-10

25

25

0

250

500

750

1000

1250

1500

1750

2000

0

250

500

750

1000

1250

1500

1750

2000

0

250

500

750

1000

1250

1500

1750

2000

communication rounds

communication rounds

communication rounds

fedavg

fedavg

0

power of choice

0

power of choice

FedIS

FedIS

5

DELTA

-2

DELTA

SS01

10

SSOI

-4

15

6

20

8

25

0

250

500

750

1000

1250

1500

1750

2000

10

0

250

500

750

1000

1250

1500

1750

2000

communication rounds

communication rounds

Performance on FashionMNIST and CIFAR-10. For CIFAR-10, we report the mean of the best 10 test accuracies on test data. In Table 2, we compare the performance of DELTA, FedIS, and FedAvg on non-IID FashionMNIST and CIFAR-10 datasets. Specifically, we use α = 0 . 1 for FashionMNIST and α = 0 . 5 for CIFAR-10 to split the datasets. As for Multinomial Distribution (MD) sampling [29], it samples based on the clients' data ratio and average aggregates. It is symmetric in sampling and aggregation with FedAvg, with similar performance. It can be seen that DELTA has better accuracy than FedIS, while both DELTA and FedIS outperform FedAvg with the same number of communication rounds.

Assessing the Compatibility of FedIS with Other Optimization Methods. In Table 4, we demonstrate that DELTA and FedIS are compatible with other FL optimization algorithms, such as Fedprox [29] and FedMIME [20]. Furthermore, DELTA maintains its superiority in this setting.

0

100

200

300

400

500

Round

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

FedAvg

Cluster-based IS

FedPracIS

FedPracDELTA

0

50

100

150

200

250

300

350

400

Round

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

FedAvg

Cluster-based IS

FedPracIS

FedPracDELTA

(a) Performance of algorithms on splitFashionMNIST

(b) Performance of algorithms on FEMNIST

0

100

200

300

400

500

Round

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

sampled number=5

sampled number=10

sampled number=20

sampled number=40

sampled number=80

In Table 5, we demonstrate that DELTA and FedIS are compatible with other variance reduction algorithms, like FedVARP [18].

It is worth noting that FedV ARP utilizes the historic update to approximate the unparticipated clients' updates. However, in this setting, the improvement of the sampling strategy on the results is somewhat reduced. This is because the sampling strategy is slightly redundant when all users are involved. Thus, when VARP and DELTA/FedIS are combined, instead of reassigning weights in the aggregation step, we use (75) or (13) to select the current round update clients and then average aggregate the updates of all clients. One can see that the combination of DELTA/FedIS and VARP can still show the advantages of sampling.

In addition to the above optimization methods like VARP, we also conduct experiments on LEAF (FEMNIST and CelebA) with other non-vanilla SGD algorithms, namely Adagrad and Adam, to show that the proposed client selection framework be applied to federated learning algorithms other than vanilla SGD. The results are shown in Table 6 and Table 7.

Table 8 provide results under some common thresholds, including 50% for CIFAR-10 and 80% for CelebA, to replace 54% for CIFAR-10 and 85% for CelebA in Table 2.

Albation study for α . In Table 9, we experiment with different choices of heterogeneity α in the CIFAR-10 dataset. The parameter of heterogeneity α changes from 0 . 1 to 0 . 5 to 1 . We observe a consistent improvement of DELTA compared to the other algorithms. This shows that DELTA is robust to changes in the level of heterogeneity in the data distribution.

Peer reviews:
comments: SUMMARY:
The paper proposed an unbiased client sampling method in Federated Learning. The proposed method is motivated by considering both variance and gradient diversity when doing client sampling. The authors theoretically proved that the proposed sampling method can achieve better convergence rate for non-convex objective functions. Besides, the authors also demonstrated its practical advantages by numerical experiments.

STRENGTHS:
Considering gradient diversity is a novel idea in FL client sampling research. The paper demonstrates its proposed method by both theory and experiments. The motivation is clear, and the paper is in general good writing. Given that client sampling is an important problem of federated learning and the claimed advantage of the proposed method, the paper can have significant impact on the area.

WEAKNESSES:
Assumption 4 is a very strong assumption. It can be violated in many problems. For example, the simple least square regression problem can violate this assumption.

Besides, the paper seems to have too much content for a nine-page paper. As a result, the key theoretical result is lack of intuition.

QUESTIONS:
1.	May the authors discuss what will happen if Assumption 4 does not hold?
2.	How large is U used in the theory of practical IS and practical DELTA?
3.	May the authors discuss what is the key intuition of theoretical advantage of the new IS analysis? In my understanding, the importance sampling can only improve the constants rather than the convergence order. Thus, I find the theoretical result quite surprising. Unfortunately, I do not have enough time to check the correction of the theory.

LIMITATIONS:
The authors mentioned that their proposed method does not address the backdoor attack concern.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce DELTA (Diverse Client Sampling) which is an unbiased method for client selection in Federated Learning (FL). DELTA is  heavily inspired by Importance Sampling (IS) and cluster-based IS, resulting in sampling diverse clients with significant gradients however without the clustering. They provide convergence rate analysis of FL under IS scheme (FedIS) and then of FL under their DELTA scheme (FedDELTA) and closed form expressions for the sampling probabilities are given and discussed. Practical versions of those algorithms FedPracIS and FedPracDELTA are also analyzed. In the experiments loss profiles, accuracies and timings of proposed algorithms against baselines (including FedAvg, Power-of-Choice and cluster-based IS) and over a collection of real and synthetic datasets are presented and discussed, exhibiting the empirical advantages of the sampling scheme.\n\nSTRENGTHS:\n- Comprehensive and nicely structured, easy to follow presentation, with a good balance of theoretical, practical and intuitive/explanatory information.\n\n- Rich empirical results also communicating the benefits of proposed, analyzed sampling approaches.\n\n- Simplicity of resulting FL algorithm: Basically the well-known, seminal FedAvg idea/template with a novel client sampling scheme (DELTA, PracticalDELTA) on top (Algorithm 1).\n\nWEAKNESSES:\n- Minor: \"developing an efficient and effective practical algorithm for gradient-based sampling methods\" in future work section (Section 6) could confuse the reader (since DELTA is exactly such an algorithm). Perhaps  a different wording, emphasizing room of improving over this new/existing development?\n\nQUESTIONS:\n- In the experiments, there seems to be a variety of thresholds (70%, 54%, 85%, 80%) for different experiments and metrics in Tables 2 and 3. I would expect a more uniform way of presentation, some common threshold.\n\n- On a related topic: Since for Table 2, 500 communication rounds were completed for all (dataset, algorithm) pairs why mean of maximum five accuracies? Are all these maximum values observed within the last x% of rounds? How do these accuracy profiles evolve? Have they saturated after 500 rounds or there is still room for improvements? I mean some short comment summarizing the full profile plots (similar to Figures 12a and 12b in the appendix) would be more than sufficient.\n\n- How do you implement a sampling strategy like that of Equation (16) for FedPracDELTA in the experiments. There are parameters like alpha's, sigma's and zeta's in those: it would be great to include or refer to brief hints on how you choose values for those for conducting the empirical evaluation.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes a client sampling scheme in federated learning for faster convergence. The authors argue that the previous client sampling method based on importance sampling ignores gradient diversity. Convergence analysis is provided, as well as experiments on four image datasets.

STRENGTHS:
The problem of client selection in federated learning is interesting and worth investigating.

WEAKNESSES:
[1] The paper overall is not well-written. Both organization and writing could be improved. There are multiple grammar errors even in the Introduction. 

[2] Both theoretical and main method seems to rely on gradient information. However, gradient information is usually unavailable in a federated learning setting. It is unclear to me what's the contribution of this paper either theoretically or practically.

QUESTIONS:
See weakness.

LIMITATIONS:
The authors didn't really discuss limitation, even though it has a limitation session.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel unbiased client sampling scheme called DELTA for Federated Learning (FL). The authors address the issue of unrepresentative client subsets in FL, which can lead to  significant variance in model updates and slow convergence. They show that existing unbiased sampling methods have limitations in terms of convergence speed and redundancy. In contrast, DELTA selects diverse clients  with significant gradients without clustering operations. The authors also propose practical algorithms for DELTA and Importance Sampling (IS) that rely on accessible client information. The results are validated through experiments  on both synthetic and real-world datasets.\n\nSTRENGTHS:\n1. The paper presents a novel unbiased sampling scheme for FL that addresses the limitations of existing methods. The approach of selecting diverse clients based on gradient diversity without  clustering operations appears to be new and promising.\n2. The paper provides theoretical analyses and experimental results to support the proposed sampling schemes. The convergence rates of the practical algorithms are shown to attain the same  order as the theoretical optimal sampling probabilities.\n3. The paper is well-organized and clearly presents the motivation, methodology, and results of the study. The figures effectively illustrate the concepts and comparisons between different sampling methods.\n\nWEAKNESSES:\n1. In Corollary 4.2, the convergence rate of the practical algorithm depends on a term $\\tilde{U}$, which can be very large if $\\left\\|\\nabla F_i\\left(x_s\\right)-\\nabla f\\left(x_s\\right)\\right\\|$ is small. \n\n2. Can the proposed client selection framework be applied to federated learning algorithms other than vanilla SGD. I believe a unified theoretical analysis would show the effectiveness of the proposed method.\n\nQUESTIONS:\nSee the weakness part above.\n\nLIMITATIONS:\nThe paper does not explicitly address the limitations and potential negative societal impacts of the proposed work, but I do not think that this is a major issue.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the sampling schemes in federated learning. In particular, the authors first develop a new analysis method for important sampling (IS) strategy, which achieves a better convergence rate and then they propose a new sampling approach called DELTA, which can outperform the IS scheme. In addition, the authors also proposed a practically implementable version of DELTA and the performance is evaluated using experiments.\n\nSTRENGTHS:\n1. The authors develop a new convergence analysis for IS for nonconvex objective functions. \n\n2. The authors propose a new sampling approach called DELTA and propose a practically implementable version of DELTA.\n\n3. The proposed algorithm is evaluated using experiments.\n\nWEAKNESSES:\n1. The novelty of the analysis is unclear. It seems a standard \"by round\" analysis. Since the authors claim the analysis is one of the major contributions of this paper. I suggest the authors to elaborate this point in detail. \n\n2. The Assumption 4 seems strong and can significantly simplify the analysis in general. In addition, it  can be violated easily in practice. \n\n3. The wring of the paper is not careful enough. Many parameters are presented without explanation although it is not very hard to figure them out. For example, in Theorem 3.1, many parameters such as K and T are not explained. In addition, instead of just mentioning some lemmas and assumptions in the appendix or other papers, it will be much better to state them in the main paper to make the reading easier.\n\nQUESTIONS:\n1. The improvement of the proposed analytical approach is unclear. It seems all the approach achieve a $O(1/\\epsilon^2)$ of the communication complexity. \n\n2. When IS is analyzed, Assumption 3 is used, whereas DELTA is analyzed, it seems that Assumption 3 is not used while the gradient diversion $\\zeta_{G,i,t}$ is used, since $\\sigma_G$ is not part of Theorem 3.4. From Corollary 3.7 and compare it to Theorem 3.1, it seems the advantage of DELTA is to eliminate the term of $\\sigma_G$. If it is correct, it seems that it is a little bit unfair to use different assumptions to analyze the two different approaches. \n\n3. The novelty of the analysis has to be more elaborated. Please clearly state the main differences between the proposed analysis and other analysis in the literature such as IS and many works related to FedAvg. To me, it is a pretty standard \"by round\" analysis for nonconvex objective functions.\n\nLIMITATIONS:\nThe authors discussed the limitations of the work in the end.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}