SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} f (x _ {t + 1}) \leq f (x _ {t}) - \eta \eta_ {L} K \mathbb {E} _ {t} \left\| \nabla \tilde {f} (x _ {t}) \right\| ^ {2} + \eta \mathbb {E} _ {t} \left\langle \nabla \tilde {f} (x _ {t}), \Delta_ {t} + \eta_ {L} K \nabla \tilde {f} (x _ {t}) \right\rangle + \frac {L}{2} \eta^ {2} \mathbb {E} _ {t} \| \Delta_ {t} \| ^ {2} \\ \stackrel {(b 4)} {\leq} f (x _ {t}) - \eta \eta_ {L} K \left(\frac {1}{2} - \frac {2 0 L ^ {2} K ^ {2} (A ^ {2} + 1) \eta_ {L} ^ {2}}{n m} \sum_ {l = 1} ^ {m} \frac {1}{p _ {l} ^ {t}}\right) \mathbb {E} _ {t} \| \nabla \tilde {f} (x _ {t}) \| ^ {2} + \frac {2 K ^ {2} \eta \eta_ {L} ^ {3} L ^ {2}}{2 n m} \sum_ {l = 1} ^ {m} \frac {1}{p _ {l} ^ {t}} \left(\sigma_ {L} ^ {2} \right. \\ \left. + 4 K \zeta_ {G, i} ^ {2}\right) + \frac {L \eta^ {2} \eta_ {L} ^ {2} K}{2 n} \sum_ {l = 1} ^ {m} \frac {1}{p _ {l} ^ {t}} \sigma_ {L} ^ {2} - \left(\frac {\eta \eta_ {L}}{2 K} - \frac {L \eta^ {2} \eta_ {L} ^ {2}}{2}\right) \mathbb {E} _ {t} \left\| - \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \right\| ^ {2} \\ \leq f \left(x _ {t}\right) - c \eta \eta_ {L} K \mathbb {E} _ {t} \| \nabla \tilde {f} \left(x _ {t}\right) \| ^ {2} + \frac {2 K ^ {2} \eta \eta_ {L} ^ {3} L ^ {2}}{2 n m} \sum_ {l = 1} ^ {m} \frac {1}{p _ {l} ^ {t}} \left(\sigma_ {L} ^ {2} + 4 K \zeta_ {G, i} ^ {2}\right) + \frac {L \eta^ {2} \eta_ {L} ^ {2} K}{2 n} \sum_ {l = 1} ^ {m} \frac {1}{p _ {l} ^ {t}} \sigma_ {L} ^ {2}. \tag {70} \\ \end{array}
$$
Also, for (b4), step sizes need to satisfy  $\left(\frac{\eta\eta_L}{2K} -\frac{L\eta^2\eta_L^2}{2}\right)\geq 0$  if  $\eta \eta_{l}\leq \frac{1}{KL}$  , and there exists a constant  $c > 0$  satisfying  $(\frac{1}{2} -\frac{20K^2\eta_L^2L^2(A^2 + 1)}{mn}\sum_{l = 1}^{m}\frac{1}{p_l^t}) > c > 0$  if  $\eta_L <   \frac{1}{2\sqrt{10(A^2 + 1)}KL\sqrt{\frac{1}{n}\sum_{l = 1}^{m}\frac{1}{mp_l^t}}}$
Rearranging and summing from  $t = 0, \dots, T - 1$ , we have:
$$
\sum_ {t = 1} ^ {T - 1} c \eta \eta_ {L} K \mathbb {E} \| \nabla \tilde {f} (x _ {t}) \| ^ {2} \leq f (x _ {0}) - f (x _ {T}) + T (\eta \eta_ {L} K) \tilde {\Phi}. \tag {71}
$$
$$
\min  _ {t \in [ T ]} \mathbb {E} \| \nabla \tilde {f} (x _ {t}) \| ^ {2} \leq \frac {f _ {0} - f _ {*}}{c \eta \eta_ {L} K T} + \frac {1}{c} \tilde {\Phi}, \tag {72}
$$
Remark E.9.  $\zeta_G$  used in DELTA can be easily transformed into a  $\sigma_G$  related term, thus it is fair to compare DELTA with FedIS. In particular, by taking the expectation on  $\zeta_G$ , it equates to  $E\|\nabla F_i(x_t) - \nabla f(x_t)\|^2$ . As demonstrated in [1], one can derive  $E\|\nabla F_i(x_t) - \nabla f(x_t)\|^2 = E\|\nabla F_i(x_t)\|^2 - \|\nabla f(x_t)\|^2 \leq A\|\nabla f(x_t)\|^2 + \sigma_G^2$ . Shifting  $A\|\nabla f(x_t)\|^2$  to the left side of the convergence result,  $\zeta_G$  can be directly transformed into  $\sigma_G$ .
F Proof of the Optimal Sampling Probability
F.1 Sampling probability FedIS
Corollary F.1 (Optimal sampling probability for FedIS).
$$
\min  _ {p _ {l} ^ {t}} \Phi \qquad s. t. \sum_ {l = 1} ^ {m} p _ {l} ^ {t} = 1  .
$$
Solving the above optimization problem, we obtain the expression for the optimal sampling probability:
$$
p _ {i} ^ {t} = \frac {\| g _ {i} ^ {t} \|}{\sum_ {j = 1} ^ {m} \| g _ {j} ^ {t} \|}, \tag {73}
$$
where  $\hat{g}_i^t = \sum_{k=0}^{K-1} g_k^i$  is the sum of the gradient updates across multiple updates.
Recall Theorem 3.1; only the last variance term in the convergence term  $\Phi$  is affected by sampling. In other words, we need to minimize the variance term with respect to probability. We formalize this as follows:
$$
\min  _ {p _ {i} ^ {t} \in [ 0, 1 ], \sum_ {i = 1} ^ {m} p _ {i} ^ {t} = 1} V \left(\frac {1}{m p _ {i} ^ {t}} \hat {g} _ {i} ^ {t}\right) \Leftrightarrow \min  _ {p _ {i} ^ {t} \in [ 0, 1 ], \sum_ {i = 1} ^ {m} p _ {i} ^ {t} = 1} \frac {1}{m ^ {2}} \sum_ {i = 1} ^ {m} \frac {1}{p _ {i} ^ {t}} \| \hat {g} _ {i} ^ {t} \| ^ {2}. \tag {74}
$$
This optimization problem can be solved in closed form using the KKT conditions. It is straightforward to verify that the solution to the optimization problem is:
$$
p _ {i, t} ^ {*} = \frac {\left\| \sum_ {k = 0} ^ {K - 1} g _ {t , k} ^ {i} \right\|}{\sum_ {i = 1} ^ {m} \left\| \sum_ {k = 0} ^ {K - 1} g _ {t , k} ^ {i} \right\|}, \forall i \in 1, 2, \dots , m. \tag {75}
$$
Under the optimal sampling probability, the variance will be:
$$
V \left(\frac {1}{m p _ {i} ^ {t}} \hat {g} _ {i} ^ {t}\right) \leq \mathbb {E} \left\| \frac {\sum_ {i = 1} ^ {m} \hat {g} _ {i} ^ {t}}{m} \right\| ^ {2} = \frac {1}{m ^ {2}} \mathbb {E} \| \sum_ {i = 1} ^ {m} \sum_ {k = 1} ^ {K} \nabla F _ {i} \left(x _ {t, k}, \xi_ {k, t}\right) \| ^ {2} \tag {76}
$$
Therefore, the variance term is bounded by:
$$
V \left(\frac {1}{m p _ {i} ^ {t}} \hat {g} _ {i} ^ {t}\right) \leq \frac {1}{m} \sum_ {i = 1} ^ {m} K \sum_ {k = 1} ^ {K} \mathbb {E} \| \nabla F _ {i} \left(x _ {t, k}, \xi_ {k, t}\right) \| ^ {2} \leq K ^ {2} G ^ {2} \tag {77}
$$
Remark: If the uniform distribution is adopted with  $p_i^t = \frac{1}{m}$ , it is easy to observe that the variance of the stochastic gradient is bounded by  $\frac{\sum_{i=1}^{m} |g_i|^2}{m}$ .
According to Cauchy-Schwarz inequality,
$$
\left. \frac {\sum_ {i = 1} ^ {m} \| \hat {g} _ {i} ^ {t} \| ^ {2}}{m} \right/ \left(\frac {\sum_ {i = 1} ^ {m} \| \hat {g} _ {i} \|}{m}\right) ^ {2} = \frac {m \sum_ {i = 1} ^ {m} \| \hat {g} _ {i} \| ^ {2}}{\left(\sum_ {i = 1} ^ {m} \| \hat {g} _ {i} \|\right) ^ {2}} \geq 1, \tag {78}
$$
this implies that importance sampling does improve convergence rate, especially when  $\frac{\left(\sum_{i=1}^{m}\|g_i\|\right)^2}{\sum_{i=1}^{m}\|g_i\|^2} < m$ .
F.2 Sampling probability of DELTA
Our result is of the following form:
$$
\underset {t \in [ T ]} {\min } \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {f _ {0} - f _ {*}}{c \eta \eta_ {L} K T} + \tilde {\Phi}, \tag {79}
$$
It is easy to see that the sampling strategy only affects  $\tilde{\Phi}$ . To enhance the convergence rate, we need to minimize  $\tilde{\Phi}$  with respect to  $p_l^t$ . As shown, the expression for  $\tilde{\Phi}$  with and without replacement is similar, and only differs in the values of  $n$  and  $m$ . Here, we will consider the case with replacement. Specifically, we need to solve the following optimization problem:
$$
\min _ {p _ {l} ^ {t}} \tilde {\Phi} = \frac {1}{c} (\frac {5 L ^ {2} K \eta_ {L} ^ {2}}{2 m ^ {2}} \sum_ {l = 1} ^ {m} \frac {1}{p _ {l} ^ {t}} (\sigma_ {L, l} ^ {2} + 4 K \zeta_ {G, i} ^ {2}) + \frac {L \eta_ {L} \eta}{2 n} \sum_ {l = 1} ^ {m} \frac {1}{m ^ {2} p _ {l} ^ {t}} \sigma_ {L, i} ^ {2}) \qquad s. t. \sum_ {l = 1} ^ {m} p _ {l} ^ {t} = 1.
$$
Solving this optimization problem, we find that the optimal sampling probability is:
$$
p _ {i, t} ^ {*} = \frac {\sqrt {5 K L \eta_ {L} \left(\sigma_ {L , i} ^ {2} + 4 K \zeta_ {G , i} ^ {2}\right) + \frac {\eta}{n} \sigma_ {L , l} ^ {2}}}{\sum_ {l = 1} ^ {m} \sqrt {5 K L \eta_ {L} \left(\sigma_ {L , l} ^ {2} + 4 K \zeta_ {G , l} ^ {2}\right) + \frac {\eta}{n} \sigma_ {L , l} ^ {2}}}. \tag {80}
$$
For simplicity, we rewrite the optimal sampling probability as:
$$
p _ {i, t} ^ {*} = \frac {\sqrt {\alpha_ {1} \zeta_ {G , i} ^ {2} + \alpha_ {2} \sigma_ {L , i} ^ {2}}}{\sum_ {l = 1} ^ {m} \sqrt {\alpha_ {1} \zeta_ {G , l} ^ {2} + \alpha_ {2} \sigma_ {L , l} ^ {2}}}, \tag {81}
$$
where  $\alpha_{1} = 20K^{2}L\eta_{L}$ $\alpha_{2} = 5KL\eta_{L} + \frac{\eta}{n}$
Remark: Now, we will compare this result with the uniform sampling strategy:
$$
\Phi_ {D E L T A} = \frac {L \eta_ {L}}{2 c} \left(\frac {\sum_ {l = 1} ^ {m} \sqrt {\alpha_ {1} \zeta_ {G , l} ^ {2} + \alpha_ {2} \sigma_ {L , l} ^ {2}}}{m}\right) ^ {2}. \tag {82}
$$
For uniform  $p_l = \frac{1}{m}$ :
$$
\Phi_ {u n i f o r m} = \frac {L \eta_ {L}}{2 c} \frac {\sum_ {l = 1} ^ {m} \left(\sqrt {\alpha_ {1} \zeta_ {G , l} ^ {2} + \alpha_ {2} \sigma_ {L , l} ^ {2}}\right) ^ {2}}{m}. \tag {83}
$$
According to Cauchy-Schwarz inequality:
$$
\frac {\sum_ {l = 1} ^ {m} \left(\sqrt {\alpha_ {1} \zeta_ {G , l} ^ {2} + \alpha_ {2} \sigma_ {L , l} ^ {2}}\right) ^ {2}}{m} / \left(\frac {\sum_ {l = 1} ^ {m} \sqrt {\alpha_ {1} \zeta_ {G , l} ^ {2} + \alpha_ {2} \sigma_ {L , l} ^ {2}}}{m}\right) ^ {2} = \frac {m \sum_ {l = 1} ^ {m} \left(\sqrt {\alpha_ {1} \zeta_ {G , l} ^ {2} + \alpha_ {2} \sigma_ {L , l} ^ {2}}\right) ^ {2}}{\left(\sum_ {l = 1} ^ {m} \sqrt {\alpha_ {1} \zeta_ {G , l} ^ {2} + \alpha_ {2} \sigma_ {L , l} ^ {2}}\right) ^ {2}} \geq 1, \tag {84}
$$
this implies that our sampling method does improve the convergence rate (our sampling approach might be  $n$  times faster in convergence than uniform sampling), especially when
$$
\begin{array}{r} \frac {\left(\sum_ {l = 1} ^ {m} \sqrt {\alpha_ {1} \zeta_ {G , l} ^ {2} + \alpha_ {2} \sigma_ {L , l} ^ {2}}\right) ^ {2}}{\sum_ {l = 1} ^ {m} \left(\sqrt {\alpha_ {1} \zeta_ {G , l} ^ {2} + \alpha_ {2} \sigma_ {L , l} ^ {2}}\right) ^ {2}} <   <   m. \end{array}
$$
G Convergence Analysis of The Practical Algorithm
In order to provide the convergence rate of the practical algorithm, we need an additional Assumption 4  $(\| \nabla F_i(x)\|^2 \leq G^2, \forall i)$ . This assumption tells us a useful fact that will be used later:
It can be shown that  $\| \nabla F_i(x_{t,k},\xi_{t,k}) / \nabla F_i(x_{s,k},\xi_{s,k})\| \leq U$  for all  $i$  and  $k$ , where the subscript  $s$  refers to the last round in which client  $i$  participated, and  $U$  is a constant upper bound. This tells us
(a) FedIS's gradient norm on MNIST
(b) FedIS' gradient norm on FashionMNIST
Figure 9: Performance of gradient norm of FedIS. We evaluate the performance of FedIS on MNIST and FashionMNIST datasets. In each round, we report the maximal and minimal gradient norm among all clients.
Figure 10: Performance of gradient diversity norm of DELTA. We evaluate the performance of DELTA on MNIST and FashionMNIST datasets. In each round, we report the maximal and minimal gradient diversity norm among all clients.
(a) Gradient diversity norm of DELTA on MNIST
(b) Gradient diversity norm of DELTA on FashionMNIST
that the change in the norm of the client's gradient is bounded.  $U$  comes from the following inequality constraint procedure:
$$
\begin{array}{l} V \left(\frac {1}{m p _ {i} ^ {s}} \hat {g} _ {i} ^ {t}\right) = E | | \frac {1}{m p _ {i} ^ {s}} \hat {g} _ {i} ^ {t} - \frac {1}{m} \sum_ {i = 1} ^ {m} \hat {g} _ {i} ^ {t} | | ^ {2} \leq E | | \frac {1}{m p _ {i} ^ {t}} \hat {g} _ {i} ^ {t} | | ^ {2} = E \left\| \frac {1}{m} \frac {\hat {g} _ {i} ^ {t}}{| | \hat {g} _ {i} ^ {s} | |} \sum_ {j = 1} ^ {m} | | \hat {g} _ {j} ^ {s} | | \right\| ^ {2} \\ \leq E \left(\left| \left| \frac {1}{m} \right| \right| ^ {2} \frac {\left| \hat {g _ {i} ^ {t}} \right| ^ {2}}{\left| \left| \hat {g _ {i} ^ {s}} \right| \right| ^ {2}} \left| \sum_ {j = 1} ^ {m} \left| \left| \hat {g _ {j} ^ {s}} \right| \right| \right| ^ {2}\right) \leq \frac {1}{m ^ {2}} U ^ {2} m \sum_ {j = 1} ^ {m} K \sum_ {k = 1} ^ {K} E \| \nabla F _ {j} \left(x _ {k, s}, \xi_ {k, s}\right) \| ^ {2}. \tag {85} \\ \end{array}
$$
We establish the upper bound  $U$  based on two factors: (1) Assumption 4, and (2) the definition of importance sampling  $E_{q(z)}(F_i(z)) = E_{p(z)}(q_i(z) / p_i(z)F_i(z))$ , where there exists a positive constant  $\gamma$  such that  $p_i(z)\geq \gamma >0$ . Thus, for  $p_i^s = \frac{\hat{g}_i^s}{\sum_j\hat{g}_j^s}\geq \gamma$ , we can easily ensure  $\frac{\|g_i^t\|}{\|g_i^s\|}\leq U$  since  $\hat{g}_i^s >0$  is consistently bounded.
In general, the gradient norm tends to become smaller as training progresses, which leads to  $\| \nabla F_i(x_{t,k},\xi_{t,k}) / \nabla F_i(x_{s,k},\xi_{s,k})\|$  going to zero. Even if there are some oscillations in the gradient norm, the gradient will vary within a limited range and will not diverge to infinity. Figures 9 and Figure 10 depict the norms of gradients and gradient diversity across all clients in each round. Notably, these figures demonstrate that in the case of practical IS and practical DELTA, the change ratio of both gradient and gradient diversity remains limited, with the maximum norm being under 8 and the minimum norm exceeding 0.5.
Based on Assumption 4 and Assumption 3, we can re-derive the convergence analysis for both convergence variance  $\Phi$  (4) and  $\tilde{\Phi}$  (46). In particular, for Assumption 3  $(\mathbb{E}\| \nabla F_i(x)\|^2\leq (A^2 +1)\| \nabla f(x)\|^2 +\sigma_G^2)$ , we use  $\sigma_{G,s}$  and  $\sigma_{G,t}$  instead of a unified  $\sigma_{G}$  for the sake of comparison.
Specifically,  $\Phi = \frac{1}{c} \left[ \frac{5\eta_L^2 L^2 K}{2m} \sum_{i=1}^{m} (\sigma_L^2 + 4K\sigma_G^2) + \frac{\eta\eta_L L}{2m} \sigma_L^2 + \frac{L\eta\eta_L}{2nK} V \left( \frac{1}{mp_i^t} \hat{g}_i^t \right) \right]$ , where  $\hat{g}_i^t = \sum_{k=1}^{K} \nabla F_i(x_{k,s}, \xi_{k,s})$ . With the practical sampling probability  $p_i^s$  of FedIS:
$$
V \left(\frac {1}{m p _ {i} ^ {s}} \hat {g} _ {i} ^ {t}\right) = E \| \frac {1}{m p _ {i} ^ {s}} \hat {g} _ {i} ^ {t} - \frac {1}{m} \sum_ {i = 1} ^ {m} \hat {g} _ {i} ^ {t} \| ^ {2} \leq E \| \frac {1}{m p _ {i} ^ {t}} \hat {g} _ {i} ^ {t} \| ^ {2} = E \| \frac {1}{m} \frac {\hat {g} _ {i} ^ {t}}{\hat {g} _ {i} ^ {s}} \sum_ {j = 1} ^ {m} \hat {g} _ {j} ^ {s} \| ^ {2}. \tag {86}
$$
According to Assumption 4, we know  $\| \frac{\hat{g}_i^t}{\hat{g}_i^s}\|^2 = \| \frac{\sum_{k=1}^{K} \nabla F_i(x_{t,k}^i, \xi_{t,k}^i)}{\sum_{k=1}^{K} \nabla F_i(x_{s,k}^i, \xi_{s,k}^i)}\|^2 \leq U^2$ . Then we get
$$
\begin{array}{l} V \left(\frac {1}{m p _ {i} ^ {s}} \hat {g} _ {i} ^ {t}\right) \leq E \left(\| \frac {1}{m} \| ^ {2} \| \| \frac {\hat {g} _ {i} ^ {t}}{\hat {g} _ {i} ^ {s}} \| ^ {2} \| \sum_ {j = 1} ^ {m} \hat {g} _ {j} ^ {s} \| ^ {2}\right) \leq \frac {1}{m ^ {2}} U ^ {2} E \| \sum_ {i = 1} ^ {m} \sum_ {k = 1} ^ {K} \nabla F _ {i} (x _ {k, s}, \xi_ {k, s}) \| ^ {2} \\ \leq \frac {1}{m ^ {2}} U ^ {2} m \sum_ {i = 1} ^ {m} K \sum_ {k = 1} ^ {K} E \| \nabla F _ {i} \left(x _ {k, s}, \xi_ {k, s}\right) \| ^ {2} \tag {87} \\ \end{array}
$$
Similar to the previous proof, based on Assumption 3. we can get the new convergence rate:
$$
\min  _ {t \in [ T ]} E \| \nabla f (x _ {t}) \| ^ {2} \leq \mathcal {O} \left(\frac {f ^ {0} - f ^ {*}}{\sqrt {n K T}}\right) + \underbrace {\mathcal {O} \left(\frac {\sigma_ {L} ^ {2}}{\sqrt {n K T}}\right) + \mathcal {O} \left(\frac {M ^ {2}}{T}\right) + \mathcal {O} \left(\frac {K U ^ {2} \sigma_ {G , s} ^ {2}}{\sqrt {n K T}}\right)} _ {\text {o r d e r o f} \Phi}. \tag {88}
$$
where  $M = \sigma_L^2 +4K\sigma_{G,s}^2$
Remark G.1 (Discussion on  $U$  and convergence rate.). It is worth noting that  $\| \nabla F_i(x_{t,k},\xi_{t,k}) / \nabla F_i(x_{s,k},\xi_{s,k})\|$  is typically relatively small because the gradient tends to go to zero as the training process progresses. This means that  $U$  can be relatively small, more specifically,  $U < 1$  in the upper bound term  $\mathcal{O}\left(\frac{K U^2\sigma_{G,s}^2}{\sqrt{n K T}}\right)$ . However, this does not necessarily mean that the practical algorithm is better than the theoretical algorithm because the values of  $\sigma_G$  are different, as we stated at the beginning. Typically, the value of  $\sigma_{G,s}$  for the practical algorithm is larger than the value of  $\sigma_{G,t}$ , which also comes from the fact that the gradient tends to go to zero as the training process progresses. Additionally, due to the presence of the summation over both  $i$  and  $k$ , the gap between  $\sigma_{G,s}$  and  $\sigma_{G,t}$  is multiplied, and  $\sigma_{G,s} / \sigma_{G,t} \sim m^2 K^2 \frac{1}{U^2}$ . Thus, the practical algorithm leads to a slower convergence than the theoretical algorithm.
Similarly, as long as the gradient is consistently bounded, we can assume that  $\| \nabla F_i(x_t) - \nabla f(x_t)\| /\| \nabla F_i(x_s) - \nabla f(x_s)\| \leq \tilde{U}_1\leq \tilde{U}$  and  $\| \sigma_{L,t} / \sigma_{L,s}\| \leq \tilde{U}_2\leq \tilde{U}$  for all  $i$ , where  $\sigma_{L,s}^2 = \mathbb{E}\left[\left\| \nabla F_i(x_s,\xi_s^i) - \nabla F_i(x_s)\right\|\right]$ . Then, we can obtain a similar conclusion by following the same analysis on  $\tilde{\Phi}$ .
Specifically, we have  $\tilde{\Phi} = \frac{L\eta_L}{2m^2c}\sum_{i = 1}^{m}\frac{1}{p_i^s}\left(\alpha_1\zeta_{G,i}^2 +\alpha_2\sigma_{L,i}^2\right)$ , where  $\alpha_{1}$  and  $\alpha_{2}$  are constants defined in (13). For the sake of comparison of different participation rounds  $s$  and  $t$ , we rewrite the symbols as  $\zeta_{G,s}^{i}$  and  $\sigma_{L,s}^{i}$ . Then, using the practical sampling probability  $p_i^s$  of DELTA, and letting  $R_{i}^{s} = \sqrt{\alpha_{1}\zeta_{G,s}^{i}{}^{2} + \alpha_{2}\sigma_{L,s}^{i}{}^{2}}$ , we have:
$$
\begin{array}{l} \tilde {\Phi} = \frac {L \eta_ {L}}{2 m ^ {2} c} \sum_ {i = 1} ^ {m} \frac {1}{p _ {i} ^ {s}} (R _ {i} ^ {t}) ^ {2} = \frac {L \eta_ {L}}{2 m ^ {2} c} \sum_ {i = 1} ^ {m} \frac {(R _ {i} ^ {t}) ^ {2}}{R _ {i} ^ {s}} \sum_ {j = 1} ^ {m} (R _ {j} ^ {s}) ^ {2} = \frac {L \eta_ {L}}{2 m ^ {2} c} \sum_ {i = 1} ^ {m} \left(\frac {R _ {i} ^ {t}}{R _ {i} ^ {s}}\right) ^ {2} R _ {i} ^ {s} \sum_ {j = 1} ^ {m} R _ {j} ^ {s} \\ \leq \frac {L \eta_ {L}}{2 m ^ {2} c} \tilde {U} ^ {2} \sum_ {i = 1} ^ {m} R _ {i} ^ {s} \sum_ {j = 1} ^ {m} R _ {j} ^ {s} = \frac {L \eta_ {L}}{2 m ^ {2} c} \tilde {U} ^ {2} \left(\sum_ {i = 1} ^ {m} R _ {i} ^ {s}\right) ^ {2} \leq \frac {L \eta_ {L}}{2 m ^ {2} c} \tilde {U} ^ {2} m \sum_ {i = 1} ^ {m} (R _ {i} ^ {s}) ^ {2} \\ \leq \frac {L \eta_ {L}}{2 c} \tilde {U} ^ {2} \left(5 K L \eta_ {L} \left(\sigma_ {L, s} ^ {2} + 4 K \zeta_ {G, s} ^ {2}\right) + \frac {\eta}{n} \sigma_ {L} ^ {2}\right) \tag {89} \\ \end{array}
$$
Therefore, compared to the theoretical algorithm of DELTA, the practical algorithm of DELTA has the following convergence rate:
$$
\min  _ {t \in [ T ]} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} \leq \mathcal {O} \left(\frac {f ^ {0} - f ^ {*}}{\sqrt {n K T}}\right) + \underbrace {\mathcal {O} \left(\frac {\tilde {U} ^ {2} \sigma_ {L , s} ^ {2}}{\sqrt {n K T}}\right) + \mathcal {O} \left(\frac {\tilde {U} ^ {2} \sigma_ {L , s} ^ {2} + 4 K \tilde {U} ^ {2} \zeta_ {G , s} ^ {2}}{K T}\right)} _ {\text {o r d e r o f} \tilde {\Phi}}. \tag {90}
$$
This discussion of the effect of  $\tilde{U}$  on the convergence rate is similar to the discussion of  $U$  in Remark G.1.
H Additional Experiment Results and Experiment Details.
H.1 Experimental Environment
For all experiments, we use NVIDIA GeForce RTX 3090 GPUs. Each simulation trail with 500 communication rounds and three random seeds.
H.2 Experiment setup
Setup for the synthetic dataset. To demonstrate the validity of our theoretical results, we first conduct experiments using logistic regression on synthetic datasets. Specifically, we randomly generate  $(x,y)$  pairs using the equation  $y = \log \left(\frac{(Ax - b)^2}{2}\right)$  with given values for  $A_{i}$  and  $b_{i}$  as training data for clients. Each client's local dataset contains 1000 samples. In each round, we select 10 out of 20 clients to participate in training (we also provide the results of 10 out of 200 clients in Figure 13).
To simulate gradient noise, we calculate the gradient for each client  $i$  using the equation  $g_{i} = \nabla f_{i}(A_{i},b_{i},D_{i}) + \nu_{i}$ , where  $A_{i}$  and  $b_{i}$  are the model parameters,  $D_{i}$  is the local dataset for client  $i$ , and  $\nu_{i}$  is a zero-mean random variable that controls the heterogeneity of client  $i$ . The larger the value of  $\mathbb{E}\|\nu_i\|^2$ , the greater the heterogeneity of client  $i$ .
We demonstrate the experiment on different functions with different values of  $A$  and  $b$ . Each function is set with noise levels of 20, 30, and 40 to illustrate our theoretical results. To construct different functions, we set  $A = 8, 10$  and  $b = 2, 1$ , respectively, to observe the convergence behavior of different functions.
All the algorithms run in the same environment with a fixed learning rate of 0.001. We train each experiment for 2000 rounds to ensure that the global loss has a stable convergence performance.
Setup for FashionMNIST and CIFAR-10. To evaluate the performance of DELTA and FedIS, we train a two-layer CNN on the non-iid FashionMNIST dataset and a ResNet-18 on the non-iid CIFAR-10 dataset, respectively. CIFAR-10 is composed of  $32 \times 32$  images with three RGB channels, belonging to 10 different classes with 60000 samples.
The "non-iid" follows the idea introduced in [66, 16], where we leverage Latent Dirichlet Allocation (LDA) to control the distribution drift with the Dirichlet parameter  $\alpha$ . Larger  $\alpha$  indicates smaller drifts. Unless otherwise stated, we set the Dirichlet parameter  $\alpha = 0.5$ .
Unless specifically mentioned otherwise, our studies use the following protocol: all datasets are split with a parameter of  $\alpha = 0.5$ , the server chooses  $n = 20$  clients according to our proposed probability from the total of  $m = 300$  clients, and each is trained for  $T = 500$  communication rounds with  $K = 5$  local epochs. The default local dataset batch size is 32. The learning rates are set the same for all algorithms, specifically  $lr_{global} = 1$  and  $lr_{local} = 0.01$ .
All algorithms use FedAvg as the backbone. We compare DELTA, FedIS and Cluster-based IS with FedAvg on different datasets with different settings.
Setup for Split-FashionMNIST. In this section, we evaluate our algorithms on the split-FashionMNIST dataset. In particular, we let  $10\%$  clients own  $90\%$  of the data, and the detailed split data process is shown below:
- Divide the dataset by labels. For example, divide FashionMNIST into 10 groups, and assign each client one label
- Random select one client
- Reshuffle the data in the selected client
- Equally divided into 100 clients
Setup for LEAF. To test our algorithm's efficiency on diverse real datasets, we use the non-IID FEMNIST dataset and non-IID CelebA dataset in LEAF, as given in [3]. All baselines use a 4-layer CNN for both datasets with a learning rate of  $lr_{local} = 0.1$ , batch size of 32, sample ratio of  $20\%$  and
(a)  $A = 8,b = 2,\nu = 20$
(b)  $A = 8,b = 2,\nu = 30$
(c)  $A = 8,b = 2,\nu = 40$
(d)  $A = 10,b = 1,\nu = 20$
(e)  $A = 10,b = 1,\nu = 30$
(f)  $A = 10,b = 1,\nu = 40$
Figure 11: Performance of different algorithms on the regression model. The loss is calculated by  $f(x, y) = \left\| y - \log \left(\frac{(A_i x - b_i)^2}{2}\right) \right\|^2$ , we report the logarithm of the global loss with different degrees of gradient noise  $\nu$ . All methods are well-tuned, and we report the best result of each algorithm under each setting.
communication round of  $T = 500$ . The reported results are averaged over three runs with different random seeds.
The implementation detail of different sampling algorithms. The power-of-choice sampling method is proposed by [7]. The sampling strategy is to first sample 20 clients randomly from all clients, and then choose 10 of the 20 clients with the largest loss as the selected clients. FedAvg samples clients according to their data ratio. Thus, FedAvg promises to be unbiased, which is given in [12, 31] to be an unbiased sampling method. As for FedIS, the sampling strategy follows Equation (5). For cluster-based IS, it first clusters clients following the gradient norm and then uses the importance sampling strategy similar to FedIS in each cluster. And for DELTA, the sampling probability follows Equation (13). For the practical implementation of FedIS and DELTA, the sampling probability follows the strategy described in Section 4.
H.3 Additional Experimental Results
Performance of algorithms on the synthetic dataset. We display the log of the global loss of different sampling methods on synthetic dataset in Figure 11, where the Power-of-Choice is a biased sampling strategy that selects clients with higher loss [7].
We also show the convergence behavior of different sampling algorithms under small noise, as shown in Figure 12.
To simulate a large number of clients, we increased the client number from 20 to 200, with only 10 clients participating in each round. The results in Figure 13 demonstrate the effectiveness of DELTA.