SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Intuitively, the surrogate objective represents the practical objective of the participating clients in each round, while the update gap  $\chi_t$  represents the distance between partial client participation and full client participation. The convergence behavior of the update gap  $\chi_t^2$  is analogous to the update variance in  $\Phi$ , and the convergence of the surrogate objective  $\mathbb{E}\left\| \nabla \tilde{f}_{S_t}(x_t)\right\|^2$  depends on the other variance terms in  $\Phi$ , namely the local variance and global variance.
Minimizing the surrogate objective allows us to further reduce the variance of convergence, and we will focus on analyzing surrogate objective below. We first formulate the surrogate objective with an arbitrary unbiased sampling probability.
Surrogate objective formulation. The expression of the surrogate objective relies on the property of IS. In particular, IS aims to substitute the original sampling distribution  $p(z)$  with another arbitrary sampling distribution  $q(z)$  while keeping the expectation unchanged:  $\mathbb{E}_{q(z)}[F_i(z)] = \mathbb{E}_{p(z)}[q_i(z) / p_i(z)F_i(z)]$ . According to the Monte Carlo method, when  $q(z)$  follows the uniform distribution, we can estimate  $\mathbb{E}_{q(z)}[F_i(z)]$  by  $1/m\sum_{i=1}^{m} F_i(z)$  and  $\mathbb{E}_{p(z)}[q_i(z) / p_i(z)F_i(z)]$  by  $1/n\sum_{i\in S_t}1/mp_iF_i(z)$ , where  $m$  and  $|S_t| = n$  are the sample sizes.
Based on IS property, we formulate the surrogate objective:
$$
\tilde {f} _ {S _ {t}} \left(x _ {t}\right) = \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} F _ {i} \left(x _ {t}\right), \tag {7}
$$
where  $m$  is the total number of clients,  $|S_{t}| = n$  is the number of participating clients in each round, and  $p_t^i$  is the probability that client  $i$  is selected at round  $t$ .
As noted in Lemma C.2 in the appendix, we have:
$$
\min  _ {t \in [ T ]} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} = \min  _ {t \in [ T ]} \mathbb {E} \| \nabla \tilde {f} (x _ {t}) \| ^ {2} + \mathbb {E} \| \chi_ {t} ^ {2} \| \leq \min  _ {t \in [ T ]} 2 \mathbb {E} \| \nabla \tilde {f} (x _ {t}) \| ^ {2}. \tag {8}
$$
Then the convergence rate of the global objective can be formulated as follows:
Theorem 3.5 (Convergence upper bound of FedDELTA). Under Assumption 1-3 and let local and global learning rates  $\eta$  and  $\eta_L$  satisfy  $\eta_L < \frac{1}{2\sqrt{10K}L}\sqrt{\frac{1}{n}\sum_{l=1}^{m}\frac{1}{mp_l^t}}$  and  $\eta \eta_L \leq \frac{1}{KL}$ , the minimal gradient norm will be bounded as below:
$$
\min  _ {t \in [ T ]} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {f ^ {0} - f ^ {*}}{c \eta \eta_ {L} K T} + \frac {\tilde {\Phi}}{c}, \tag {9}
$$
2With slight abuse of notation, we use the  $\tilde{f} (x_{t})$  for  $\tilde{f}_{S_t}(x_t)$  in this paper.
where  $f^0 = f(x_0)$ ,  $f^* = f(x_*)$ ,  $c$  is a constant, and the expectation is over the local dataset samples among all workers. The combination of variance  $\tilde{\Phi}$  represents combinations of local variance and client gradient diversity.
We derive the convergence rates for both sampling with replacement and sampling without replacement. For sampling without replacement:
$$
\tilde {\Phi} = \frac {5 L ^ {2} K \eta_ {L} ^ {2}}{2 m n} \sum_ {i = 1} ^ {m} \frac {1}{p _ {i} ^ {t}} \left(\sigma_ {L, i} ^ {2} + 4 K \zeta_ {G, i, t} ^ {2}\right) + \frac {L \eta_ {L} \eta}{2 n} \sum_ {i = 1} ^ {m} \frac {1}{m ^ {2} p _ {i} ^ {t}} \sigma_ {L, i} ^ {2}. \tag {10}
$$
For sampling with replacement,
$$
\tilde {\Phi} = \frac {5 L ^ {2} K \eta_ {L} ^ {2}}{2 m ^ {2}} \sum_ {i = 1} ^ {m} \frac {1}{p _ {i} ^ {t}} \left(\sigma_ {L, i} ^ {2} + 4 K \zeta_ {G, i, t} ^ {2}\right) + \frac {L \eta_ {L} \eta}{2 n} \sum_ {i = 1} ^ {m} \frac {1}{m ^ {2} p _ {i} ^ {t}} \sigma_ {L, i} ^ {2}, \tag {11}
$$
where  $\zeta_{G,i,t} = \| \nabla F_i(x_t) - \nabla f(x_t)\|$  and let  $\zeta_G$  be a upper bound for all  $i$ , i.e.,  $\zeta_{G,i,t}\leq \zeta_G$ . The proof details of Theorem 3.5 can be found in Appendix E.
Remark 3.6 (The novelty of DELTA analysis). IS focuses on minimizing  $V\left(\frac{1}{mp_i^t}\hat{g}_i^t\right)$  in convergence variance  $\Phi$  (Eq. (4)), while leaving other terms like  $\sigma_L$  and  $\sigma_G$  unreduced. Unlike IS roles to reduce the update gap, we propose analyzing the surrogate objective for additional variance reduction.
Compared with FedIS, our analysis of DELTA entails: Focusing on surrogate objective, introducing a novel Lemma and bound: (1) we decompose global objective convergence into surrogate objective and update gap (6). For surrogate objective analysis, we introduce Lemma E.8 to bound local updates. (2) leveraging the unique surrogate objective expression and Lemma E.8, we link sampling probability with local variance and gradient diversity, deriving novel upper bounds for  $A_{1}$  and  $A_{2}$ . (3) by connecting update gap's convergence behavior to surrogate objective through Definition E.1 and Lemma C.2, along with (6), we establish  $\tilde{\Phi}$  as the new global objective convergence variance.
Optimizing convergence variance through novel  $\tilde{\Phi}$ : FedIS aims to reduce the update variance term  $V\left(\frac{1}{(mp_i^t)}\hat{g}_i^t\right)$  in  $\Phi$ , while FedDELTA aims to minimize the entire convergence variance  $\tilde{\Phi}$ , which is composed of both gradient diversity and local variance. By minimizing  $\tilde{\Phi}$ , we get the sampling method DELTA, which further reduces the variance terms of  $\Phi$  that cannot be minimized through IS.
3.4 Proposed Sampling Strategy: DELTA
The expression of the convergence upper bound suggests that utilizing sampling to optimize the convergence variance can accelerate the convergence. Hence, we can formulate an optimization problem that minimizes the variance  $\tilde{\Phi}$  with respect to the proposed sampling probability  $p_i^t$ :
$$
\min  _ {p _ {i} ^ {t}} \tilde {\Phi} \quad \text {s . t .} \quad \sum_ {i = 1} ^ {m} p _ {i} ^ {t} = 1, \tag {12}
$$
where  $\tilde{\Phi}$  is a linear combination of local variance  $\sigma_{L,i}$  and gradient diversity  $\zeta_{G,i,t}$  (cf. Theorem 3.5). Corollary 3.7 (Optimal sampling probability of DELTA). By solving the above optimization problem, the optimal sampling probability is determined as follows:
$$
p _ {i} ^ {t} = \frac {\sqrt {\alpha_ {1} \zeta_ {G , i , t} ^ {2} + \alpha_ {2} \sigma_ {L , i} ^ {2}}}{\sum_ {j = 1} ^ {m} \sqrt {\alpha_ {1} \zeta_ {G , j , t} ^ {2} + \alpha_ {2} \sigma_ {L , j} ^ {2}}}, \tag {13}
$$
where  $\alpha_{1}$  and  $\alpha_{2}$  are constants defined as  $\alpha_{1} = 20K^{2}L\eta_{L}$  and  $\alpha_{2} = 5KL\eta_{L} + \frac{\eta}{n}$ .
Remark 3.8. We note that a tension exists between the optimal sampling probability (13) and the setting of partial participation for FL. Thus, we also provide a practical implementation version for DELTA and analyze its convergence in Section 4. In particular, we will show that the convergence rate of the practical implementation version keeps the same order with a coefficient difference.
Corollary 3.9 (Convergence rate of FedDELTA). Let  $\eta_L = \mathcal{O}\left(\frac{1}{\sqrt{TKL}}\right)$ ,  $\eta = \mathcal{O}\left(\sqrt{Kn}\right)$  and substitute the optimal sampling probability (13) back to  $\tilde{\Phi}$ . Then for sufficiently large  $T$ , the expected norm of DELTA algorithm 1 satisfies:
$$
\min  _ {t \in [ T ]} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} \leq \mathcal {O} \left(\frac {f ^ {0} - f ^ {*}}{\sqrt {n K T}}\right) + \underbrace {\mathcal {O} \left(\frac {\sigma_ {L} ^ {2}}{\sqrt {n K T}}\right) + \mathcal {O} \left(\frac {\sigma_ {L} ^ {2} + 4 K \zeta_ {G} ^ {2}}{K T}\right)} _ {\text {o r d e r o f} \bar {\Phi}}. \tag {14}
$$
Difference between FedDELTA and FedIS. The primary distinction between FedDELTA and FedIS lies in the difference between  $\tilde{\Phi}$  and  $\Phi$ . FedIS aims to decrease the update variance term  $\mathrm{Var}\left(\frac{1}{(mp_i^t)\hat{g}_i^t}\right)$  in  $\Phi$ , while FedDELTA aims to reduce the entire quantity  $\tilde{\Phi}$ , which is composed of both gradient diversity and local variance. By minimizing  $\tilde{\Phi}$ , we can further reduce the terms of  $\Phi$  that cannot be minimized through FedIS. This leads to different expressions for the optimal sampling probability. The difference between the two resulting update gradients is discussed in Figure 3. Additionally, as seen in Table 1, FedDELTA achieves a superior convergence rate of  $\mathcal{O}\left(\frac{G^2}{\epsilon^2}\right)$  compared to other unbiased sampling algorithms.
Compare DELTA with uniform sampling. According to the Cauchy-Schwarz inequality, DELTA is at least better than uniform sampling by reducing variance:  $\frac{\tilde{\Phi}_{\text{uniform}}}{\tilde{\Phi}_{\text{DELTA}}} = \frac{m \sum_{i=1}^{m} \left( \sqrt{\alpha_1 \sigma_L^2 + \alpha_2 \zeta_G^2}_{G,i,t} \right)^2}{\left( \sum_{i=1}^{m} \sqrt{\alpha_1 \sigma_L^2 + \alpha_2 \zeta_G^2}_{G,i,t} \right)^2} \geq 1$ .
This implies that DELTA does reduce the variance, especially when  $\frac{\left(\sum_{i=1}^{m}\sqrt{\alpha_1\sigma_L^2 + \alpha_2\zeta_{G,i,t}^2}\right)^2}{\sum_{i=1}^{m}\left(\sqrt{\alpha_1\sigma_L^2 + \alpha_2\zeta_{G,i,t}^2}\right)^2} \ll m$ .
The significance of DELTA. (1) DELTA is the first unbiased sampling algorithm, to the best of our knowledge, that considers both gradient diversity and local variance in sampling, accelerating convergence. (2) Developing DELTA inspires an improved convergence analysis by focusing on the surrogate objective, leading to a superior convergence rate for FL. (3) Moreover, DELTA can be seen as an unbiased version with the complete theoretical justification for the existing heuristic or biased diversity sampling algorithm of FL, such as [2].
4 FedPracDELTA and FedPracIS: The Practical Algorithms
The gradient-norm-based sampling method necessitates the calculation of the full gradient in every iteration [10, 70]. However, acquiring each client's gradient in advance is generally impractical in FL. To overcome this obstacle, we leverage the gradient from the previous participated round to estimate the gradient of the current round, thus reducing computational resources [49].
For FedPracIS, at round 0, all probabilities are set to  $1 / m$ . Then, during the  $i_{th}$  iteration, once participating clients  $i \in S_t$  have sent the server their updated gradients, the sampling probabilities are updated as follows:
$$
p _ {i, t + 1} ^ {*} = \frac {\left\| \hat {g} _ {i , t} \right\|}{\sum_ {i \in S _ {t}} \left\| \hat {g} _ {i , t} \right\|} \left(1 - \sum_ {i \in S _ {t} ^ {c}} p _ {i, t} ^ {*}\right), \tag {15}
$$
where the multiplicative factor ensures that all probabilities sum to 1. The FedPracIS algorithm is shown in Algorithm 2 of Appendix D.
For FedProcDELTA, we use the average of the latest participated clients' gradients to approximate the true gradient of the global model. For local variance, it is obtained by the local gradient's variance over local batches. Specifically,  $\zeta_{G,i,t} = \| \hat{g}_{i,t} - \nabla \hat{f}(x_t)\|$ , where  $\nabla \hat{f}(x_t) = \frac{1}{n}\sum_{i\in S_t}\hat{g}_{i,t} = \frac{1}{n}\sum_{i\in S_t}\sum_{k=0}^{K-1}\nabla F_i(x_{k,t}^i,\xi_{k,t}^i)$  and  $\sigma_{L,i}^2 = \frac{1}{|B|}\sum_{b\in B}(\hat{g}_{i,t}^b - \frac{1}{|B|}\sum_{b\in B}\hat{g}_{i,t}^b)^2$ , where  $b\in B$  is the local data batch. Then the sampling probabilities are updated as follows:
$$
p _ {i, t + 1} ^ {*} = \frac {\sqrt {\alpha_ {1} \zeta_ {G , i , t} ^ {2} + \alpha_ {2} \sigma_ {L , i} ^ {2}}}{\sum_ {i \in S _ {t}} \sqrt {\alpha_ {1} \zeta_ {G , i , t} ^ {2} + \alpha_ {2} \sigma_ {L , i} ^ {2}}} \left(1 - \sum_ {j \in S _ {t} ^ {c}} p _ {i, t} ^ {*}\right). \tag {16}
$$
The FedProcDELTA algorithm is shown in Algorithm 1. Specifically, for  $\alpha$ , the default value is 0.5, whereas  $\zeta_G$  and  $\sigma_L$  can be implemented by computing the locally obtained gradients.
Assumption 4 (Local gradient norm bound). The gradients  $\nabla F_{i}(x)$  are uniformly upper bounded (by a constant  $G > 0$ )  $\| \nabla F_i(x)\|^2\leq G^2,\forall i$
Assumption 4 is a general assumption in IS community to bound the gradient norm [70, 10, 23], and it is also used in the FL community to analyze convergence [2, 68]. This assumption tells us a useful fact that will be used later:  $\| \nabla F_i(x_{t,k},\xi_{t,k}) / \nabla F_i(x_{s,k},\xi_{s,k})\| \leq U$ . While, for DELTA, the assumption used is a more relaxed version of Assumption 4, namely,  $\mathbb{E}\| \nabla F_i(x) - \nabla f(x)\| ^2\leq G^2$  (further details are provided in Appendix G).
Corollary 4.1 (Convergence rate of FedPracIS). Under Assumption 1-4, the expected norm of FedPracIS will be bounded as follows:
$$
\min  _ {t \in [ T ]} E \| \nabla f \left(x _ {t}\right) \| ^ {2} \leq \mathcal {O} \left(\frac {f ^ {0} - f ^ {*}}{\sqrt {n K T}}\right) + \mathcal {O} \left(\frac {\sigma_ {L} ^ {2}}{\sqrt {n K T}}\right) + \mathcal {O} \left(\frac {M ^ {2}}{T}\right) + \mathcal {O} \left(\frac {K U ^ {2} \sigma_ {G , s} ^ {2}}{\sqrt {n K T}}\right), \tag {17}
$$
(a)  $\nu = 20$
(b)  $\nu = 30$
Figure 4: Performance of different algorithms on the regression model. The loss is calculated by  $f(x, y) = \left\| y - \log \left( (A_i x - b_i)^2 / 2 \right) \right\|^2$ ,  $A_i = 10$ ,  $b_i = 1$ . The logarithm of global loss is reported for various degrees of gradient noise,  $\nu$ , and all methods are well-tuned to yield the best results for each algorithm under each setting.
(c)  $\nu = 40$
where  $M = \sigma_L^2 + 4K\sigma_{G,s}^2$ ,  $\sigma_{G,s}$  is the gradient dissimilarity bound of round  $s$ , and  $\| \nabla F_i(x_{t,k},\xi_{t,k}) / \nabla F_i(x_{s,k},\xi_{s,k}) \| \leq U$  for all  $i$  and  $k$ .
Corollary 4.2 (Convergence rate of FedPracDELTA). Under Assumption 1-4, the expected norm of FedPracDELTA satisfies:
$$
\min  _ {t \in [ T ]} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} \leq \mathcal {O} \left(\frac {f ^ {0} - f ^ {*}}{\sqrt {n K T}}\right) + \mathcal {O} \left(\frac {\bar {U} ^ {2} \sigma_ {L , s} ^ {2}}{\sqrt {n K T}}\right) + \mathcal {O} \left(\frac {\bar {U} ^ {2} \sigma_ {L , s} ^ {2} + 4 K \bar {U} ^ {2} \zeta_ {G , s} ^ {2}}{K T}\right), \tag {18}
$$
where  $\tilde{U}$  is a constant that  $\| \nabla F_i(x_t) - \nabla f(x_t)\| /\| \nabla F_i(x_s) - \nabla f(x_s)\| \leq \tilde{U}_1\leq \tilde{U}$  and  $\| \sigma_{L,t} / \sigma_{L,s}\| \leq \tilde{U}_2\leq \tilde{U}$ , and  $\zeta_{G,s}$  is the gradient diversity bound of round  $s$  for all clients.
Remark 4.3. The analysis of the FedPracIS and FedPracDELTA is independent of the unavailable information in the partial participation setting. The convergence rates are of the same order as that of our theoretical algorithm but with an added coefficient constant term that limits the gradient changing rate, as shown in Table 1.
The complete derivation and discussion of the practical algorithm can be found in Appendix G.
In this section, we evaluate the efficiency of the theoretical algorithm FedDELTA and the practical algorithm FedPracDELTA on various datasets. Our code is available at https://github.com/L3030/DELTA_FL.
Datasets. (1) We evaluate FedDELTA on synthetic data and split-FashionMNIST. The synthetic data follows  $y = \log \left( {{\left( {A}_{i}x - {b}_{i}\right) }^{2}/2}\right)$  and "split" means letting 10% of clients own 90% of the data. (2) We evaluate FedPracDELTA on non-iid FashionMNIST, CIFAR-10 and LEAF [3]. Details of data generation and partitioning are provided in Appendix H.2.
Baselines and Models. We compare our algorithm, Fed(Prac)DELTA (Algorithm 1), with Fed(Prac)IS (Algorithm 2 in Appendix D), FedAVG [38], which uses random sampling, and Power-of-choice [7], which uses loss-based sampling and Cluster-based IS [52]. We utilize the regression model on synthetic data, the CNN model on Fashion-MNIST and Leaf, and the ResNet-18 on CIFAR-10. All algorithms are compared under the same experimental settings, such as lr and batch size. Full details of the sampling process of baselines and the setup of experiments are provided in Appendix H.2.
The maximum values reported in Table 2 are observed during the last  $4\%$  of rounds, where these algorithms have already reached convergence. The term 'maximum five accuracies' refers to the mean of the five highest accuracy values obtained within the plateau region of the accuracy curve.
Figure 4 illustrates the theoretical FedDELTA outperforms other biased and unbiased methods in convergence speed on synthetic datasets. The superiority of the theoretical DELTA is also confirmed on split-FashionMNIST, as shown in Appendix H in Figure 14(a). Additional experimental results, which include a range of different choices of regression parameters  $A_{i}$ ,  $b_{i}$ , noise  $\nu$ , and client numbers, are presented in Figure 11, Figure 12, and Figure 13 in Appendix H.3.
Table 2 shows the FedPracDELTA has better performance in accuracy, communication rounds, and training wall-clock times. Notably, FedPracDELTA significantly accelerates convergence by requiring fewer training rounds and less time to achieve the threshold accuracy in FashionMNIST,
Table 2: Performance of algorithms over various datasets. We run 500 communication rounds on FashionMNIST, CIFAR-10, FEMNIST, and CelebA for each algorithm. We report the mean of maximum 5 accuracies for test datasets and the average number of communication rounds and time to reach the threshold accuracy.
Algorithm	FashionMNIST	CIFAR-10
Acc (%)	Rounds for 70%	Time (s) for 70%	Acc (%)	Rounds for 54%	Time (s) for 54%
FedAvg	70.35±0.51	426 (1.0×)	1795.12 (1.0×)	54.28±0.29	338 (1.0×)	3283.14 (1.0×)
Cluster-based IS	71.21 ±0.24	362 (1.17×)	1547.41 (1.16×)	54.83±0.02	323 (1.05×)	3188.54 (1.03×)
FedPracIS	71.69±0.43	404 (1.05×)	1719.26 (1.04×)	55.05±0.27	313 (1.08×)	3085.05 (1.06×)
FedPracDELTA	72.10±0.49	322 (1.32×)	1372.33 (1.31×)	55.20 ±0.26	303 (1.12×)	2989.98 (1.1×)
Algorithm	FEMNIST	CelebA
Acc (%)	Rounds for 70%	Time (s) for 70%	Acc (%)	Rounds for 85%	Time (s) for 85%
FedAvg	71.82±0.93	164 (1.0×)	330.02 (1.0×)	85.92±0.89	420 (1.0×)	3439.81 (1.0×)
Cluster-based IS	70.42±0.66	215 (0.76×)	453.56 (0.73×)	86.77±0.11	395 (1.06×)	3474.50 (1.01×)
FedPracIS	80.11±0.29	110 (1.51×)	223.27 (1.48×)	88.12±0.71	327 (1.28×)	2746.82 (1.25×)
FedPracDELTA	81.44±0.28	98 (1.67×)	198.95 (1.66×)	89.67 ±0.56	306 (1.37×)	2607.12 (1.32×)
Table 3: Performance of sampling algorithms integration with other optimization methods on FEMNIST. PracIS and PracDELTA are the sampling methods of Algorithm FedPracIS and FedPracDELTA, respectively, using the sampling probabilities defined in equations (15) and (16). For proximal and momentum methods, we use the default hyperparameter setting  $\mu = 0.01$  and  $\gamma = 0.9$ .
Backbone with Sampling	Uniform Sampling	Cluster-based IS	PracIS	PracDELTA
Acc (%)	Rounds for 80%	Acc (%)	Rounds for 80%	Acc (%)	Rounds for 80%	Acc (%)	Rounds for 80%
FedAvg	71.82±0.93	164 (for 70%)	70.42±0.66	215 (for 70%)	80.11±0.29	110 (for 70%)	81.44±0.28	98 (for 70%)
FedAvg + momentum	80.86±0.49	268	80.86±0.49	281	81.80±0.05	246	82.58±0.44	200
FedAvg + proximal	81.41±0.34	313	80.88±0.38	326	81.28±0.25	289	82.54±0.57	245
CIFAR-10, FEMNIST, and CelebA. Additionally, on the natural federated dataset LEAF (FEMNIST and CelebA), our results demonstrate that both FedPracDELTA and FedPracIS exhibit substantial improvements over FedAvg. Figure 14(b) in Appendix H.3 illustrates the superior convergence of FedPracDELTA, showcasing the accuracy curves of sampling algorithms on FEMNIST.
Table 3 demonstrates that when compatible with momentum or proximal regularization, our method keeps its superiority in convergence. We combine various optimization methods such as proximal regularization [29], momentum [34], and VARP [18] with sampling algorithms to assess their performance on FEMNIST and FashionMNIST. Additional results for proximal and momentum on CIFAR-10, and for VARP on FashionMNIST, are available in Table 4 and Table 5 in Appendix H.3.
Ablation studies. We also provide ablation studies of heterogeneity  $\alpha$  in Table 9 and the impact of the number of sampled clients on accuracy in Figure 15 in Appendix H.3.
6 Conclusions, Limitations, and Future Works
This work studies the unbiased client sampling strategy to accelerate the convergence speed of FL by leveraging diverse clients. To address the prevalent issue of full-client gradient dependence in gradient-based FL [36, 4], we extend the theoretical algorithm DELTA to a practical version that utilizes information from the available clients.
Nevertheless, addressing the backdoor attack defense issue remains crucial in sampling algorithms. Furthermore, there is still significant room for developing an efficient and effective practical algorithm for gradient-based sampling methods. We will prioritize this as a future research direction.
This work is supported in part by the National Natural Science Foundation of China under Grant No. 62001412, in part by the funding from Shenzhen Institute of Artificial Intelligence and Robotics for Society, in part by the Shenzhen Key Lab of Crowd Intelligence Empowered Low-Carbon Energy Network (Grant No. ZDSYS20220606100601002), and in part by the Guangdong Provincial Key Laboratory of Future Networks of Intelligence (Grant No. 2022B1212010001). This work is also supported in part by the Research Center for Industries of the Future (RCIF) at Westlake University, and Westlake Education Foundation.
[69] Boxin Zhao, Ziqi Liu, Chaochao Chen, Mladen Kolar, Zhiqiang Zhang, and Jun Zhou. Adaptive client sampling in federated learning via online learning with bandit feedback. arXiv preprint arXiv:2112.14332, 2021.
[70] Peilin Zhao and Tong Zhang. Stochastic optimization with importance sampling for regularized loss minimization. In international conference on machine learning, pages 1-9. PMLR, 2015.
[71] Hongbin Zhu, Yong Zhou, Hua Qian, Yuanming Shi, Xu Chen, and Yang Yang. Online client selection for asynchronous federated learning with fairness consideration. IEEE Transactions on Wireless Communications, pages 1-1, 2022.
A An Expanded Version of The Related Work 16
B Toy Example and Experiments for Illustrating Our Observation 17
B.1 Toy example 17
B.2 Experiments for illustrating our observation. 18
C Techniques 19
D Convergence of FedIS, Proof of Theorem 3.1 21
D.1 Proof for convergence rate of FedIS (Theorem 3.1) under Assumption 1-3. 25
E Convergence of DELTA. Proof of Theorem 3.5 26
E.1 Convergence rate with improved analysis method for getting DELTA 26
E.2 Proof for Theorem E.2. 28
E.2.1 Sample with replacement 28
E.2.2 Sample without replacement 32
F Proof of the Optimal Sampling Probability 35
F.1 Sampling probability FedIS 35
F.2 Sampling probability of DELTA 36
G Convergence Analysis of The Practical Algorithm 36
H Additional Experiment Results and Experiment Details. 39
H.1 Experimental Environment 39
H.2 Experiment setup 39
H.3 Additional Experimental Results 40
A An Expanded Version of The Related Work
FedAvg is proposed by [38] as a de facto algorithm of FL, in which multiple local SGD steps are executed on the available clients to alleviate the communication bottleneck. While communication efficient, heterogeneity, such as system heterogeneity [29, 31, 59, 39, 9], and statistical/objective heterogeneity [33, 21, 29, 59, 15], results in inconsistent optimization objectives and drifted client models, impeding federated optimization considerably.