SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} A _ {1} = \left\langle \nabla f (x _ {t}), \mathbb {E} _ {t} \left[ \Delta_ {t} + \eta_ {L} K \nabla f (x _ {t}) \right] \right\rangle \\ = \left\langle \nabla f (x _ {t}), \mathbb {E} _ {t} [ - \frac {1}{| S _ {t} |} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \eta_ {L} g _ {t, k} ^ {i} + \eta_ {L} K \nabla f (x _ {t}) ] \right\rangle \\ \stackrel {(a 2)} {=} \left\langle \nabla f (x _ {t}), \mathbb {E} _ {t} [ - \frac {1}{m} \sum_ {i = 1} ^ {m} \sum_ {k = 0} ^ {K - 1} \eta_ {L} \nabla F _ {i} (x _ {t, k} ^ {i}) + \eta_ {L} K \nabla f (x _ {t}) ] \right\rangle \\ = \left\langle \sqrt {\eta_ {L} K} \nabla f (x _ {t}), - \frac {\sqrt {\eta_ {L}}}{\sqrt {K}} \mathbb {E} _ {t} [ \frac {1}{m} \sum_ {i = 1} ^ {m} \sum_ {k = 0} ^ {K - 1} (\nabla F _ {i} (x _ {t, k} ^ {i}) - \nabla F _ {i} (x _ {t})) ] \right\rangle \\ \stackrel {(a 3)} {=} \frac {\eta_ {L} K}{2} \| \nabla f (x _ {t}) \| ^ {2} + \frac {\eta_ {L}}{2 K} \mathbb {E} _ {t} \left\| \frac {1}{m} \sum_ {i = 1} ^ {m} \sum_ {k = 0} ^ {K - 1} (\nabla F _ {i} (x _ {t, k} ^ {i}) - \nabla F _ {i} (x _ {t})) \right\| ^ {2} \\ - \frac {\eta_ {L}}{2 K} \mathbb {E} _ {t} \| \frac {1}{m} \sum_ {i = 1} ^ {m} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \| ^ {2} \\ \stackrel {(a 4)} {\leq} \frac {\eta_ {L} K}{2} \| \nabla f (x _ {t}) \| ^ {2} + \frac {\eta_ {L} L ^ {2}}{2 m} \sum_ {i = 1} ^ {m} \sum_ {k = 0} ^ {K - 1} \mathbb {E} _ {t} \left\| x _ {t, k} ^ {i} - x _ {t} \right\| ^ {2} - \frac {\eta_ {L}}{2 K} \mathbb {E} _ {t} \| \frac {1}{m} \sum_ {i = 1} ^ {m} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} (x _ {t, k} ^ {i}) \| ^ {2} \\ \leq \left(\frac {\eta_ {L} K}{2} + 1 0 K ^ {3} L ^ {2} \eta_ {L} ^ {3} (A ^ {2} + 1)\right) \| \nabla f (x _ {t}) \| ^ {2} + \frac {5 L ^ {2} \eta_ {L} ^ {3}}{2} K ^ {2} \sigma_ {L} ^ {2} + 1 0 \eta_ {L} ^ {3} L ^ {2} K ^ {3} \sigma_ {G} ^ {2} \\ - \frac {\eta_ {L}}{2 K} \mathbb {E} _ {t} \| \frac {1}{m} \sum_ {i = 1} ^ {m} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \| ^ {2}, \tag {32} \\ \end{array}
$$
where (a2) follows from Assumption 2 and LemmaC.1. (a3) is due to  $\langle x,y\rangle = \frac{1}{2}\left[\| x\| ^2 +\| y\| ^2 -\| x - y\| ^2\right]$  and (a4) comes from Assumption 1.
Then we consider  $A_{2}$ . Let  $\hat{g}_i^t = \sum_{k=0}^{K-1} g_{i,k}^t = \sum_{k=0}^{K-1} \nabla F_i(x_{t,k}^i, \xi_{t,k}^i)$
$$
\begin{array}{l} A _ {2} = \mathbb {E} _ {t} \| \Delta_ {t} \| ^ {2} \\ = \mathbb {E} _ {t} \left\| \eta_ {L} \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} g _ {t, k} ^ {i} \right\| ^ {2} \\ = \eta_ {L} ^ {2} \frac {1}{n} \mathbb {E} _ {t} \left\| \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} g _ {t, k} ^ {i} - \frac {1}{m} \sum_ {i = 1} ^ {m} \sum_ {k = 0} ^ {K - 1} g _ {t, k} ^ {i} \right\| ^ {2} \\ + \eta_ {L} ^ {2} \mathbb {E} _ {t} \left\| \frac {1}{m} \sum_ {i = 1} ^ {m} \sum_ {k = 0} ^ {K - 1} g _ {i} \left(x _ {t, k} ^ {i}\right) \right\| ^ {2} \\ = \frac {\eta_ {L} ^ {2}}{n} V (\frac {1}{m p _ {i} ^ {t}} \hat {g} _ {i} ^ {t}) \\ + \eta_ {L} ^ {2} \mathbb {E} \| \frac {1}{m} \sum_ {i = 1} ^ {m} \sum_ {k = 0} ^ {K - 1} \left[ g _ {i} \left(x _ {t, k} ^ {i}\right) - \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) + \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \right] \| ^ {2} \\ \leq \frac {\eta_ {L} ^ {2}}{n} V \left(\frac {1}{m p _ {i}} \hat {g _ {i} ^ {t}}\right) \\ + \eta_ {L} ^ {2} \frac {1}{m ^ {2}} \sum_ {i = 1} ^ {m} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \| g _ {i} \left(x _ {t, k} ^ {i}\right) - \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \| ^ {2} + \eta_ {L} ^ {2} \mathbb {E} \| \frac {1}{m} \sum_ {i = 1} ^ {m} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \| ^ {2} \\ \leq \frac {\eta_ {L} ^ {2}}{n} V \left(\frac {1}{m p _ {i} ^ {t}} \hat {g} _ {i} ^ {t}\right) + \eta_ {L} ^ {2} \frac {K}{m} \sigma_ {L} ^ {2} + \eta_ {L} ^ {2} \mathbb {E} \| \frac {1}{m} \sum_ {i = 1} ^ {m} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \| ^ {2}. \tag {33} \\ \end{array}
$$
The third equality follows from independent sampling.
Specifically, for sampling with replacement, due to every index being independent, we utilize  $\mathbb{E}\| x_1^2 +\ldots +x_n\|^2 = \mathbb{E}[\| x_1\|^2 +\ldots +\| x_n\|^2 ]$
For sampling without replacement:
$$
\begin{array}{l} \mathbb {E} \| \frac {1}{n} \sum_ {i \in S _ {t}} \left(\frac {1}{m p _ {i} ^ {t}} \hat {g} _ {i} ^ {t} - \frac {1}{m} \sum_ {i = 1} ^ {m} \hat {g} _ {i} ^ {t}\right) \| ^ {2} \\ = \frac {1}{n ^ {2}} \mathbb {E} \| \sum_ {i = 1} ^ {m} \mathbb {I} _ {i} \left(\frac {1}{m p _ {i} ^ {t}} \hat {g} _ {i} ^ {t} - \frac {1}{m} \sum_ {i = 1} ^ {m} \hat {g} _ {i} ^ {t}\right) \| ^ {2} \\ = \frac {1}{n ^ {2}} \mathbb {E} \left(\| \sum_ {i = 1} ^ {m} \mathbb {I} _ {i} \left(\frac {1}{m p _ {i} ^ {t}} \hat {g} _ {i} ^ {t} - \frac {1}{m} \sum_ {i = 1} ^ {m} \hat {g} _ {i} ^ {t}\right) \| ^ {2} \mid \mathbb {I} _ {i} = 1\right) \times \mathbb {P} (\mathbb {I} _ {i} = 1) \\ + \frac {1}{n ^ {2}} \mathbb {E} \left(\| \sum_ {i = 1} ^ {m} \mathbb {I} _ {i} \left(\frac {1}{m p _ {i} ^ {t}} \hat {g} _ {i} ^ {t} - \frac {1}{m} \sum_ {i = 1} ^ {m} \hat {g} _ {i} ^ {t}\right) \| ^ {2} \mid \mathbb {I} _ {i} = 0\right) \times \mathbb {P} (\mathbb {I} _ {i} = 0) \\ = \frac {1}{n} \sum_ {i = 1} ^ {m} p _ {i} ^ {t} \| \frac {1}{m p _ {i} ^ {t}} \hat {g} _ {i} ^ {t} - \frac {1}{m} \sum_ {i = 1} ^ {m} \hat {g} _ {i} ^ {t} \| ^ {2} \\ = \frac {1}{n} E \| \frac {1}{m p _ {i} ^ {t}} \hat {g} _ {i} ^ {t} - \frac {1}{m} \sum_ {i = 1} ^ {m} \hat {g} _ {i} ^ {t} \| ^ {2}. \tag {34} \\ \end{array}
$$
From the above, we observe that it is possible to achieve a speedup by sampling from the distribution that minimizes  $V\left(\frac{1}{mp_i^{\hat{t}}}\hat{g}_i^t\right)$ . Furthermore, as we discussed earlier, the optimal sampling probability is  $p_i^* = \frac{\|\hat{g}_i^t\|}{\sum_{i=1}^{m}\|\hat{g}_i^t\|}$ . For MD sampling [31], which samples according to the data ratio, the optimal sampling probability is  $p_{i,t}^* = \frac{q_i\|\hat{g}_i^t\|}{\sum_{i=1}^{m}q_i\|\hat{g}_i^t\|}$ , where  $q_i = \frac{n_i}{N}$ .
Now we substitute the expressions of  $A_{1}$  and  $A_{2}$ :
$$
\begin{array}{l} \mathbb {E} _ {t} [ f (x _ {t + 1}) ] \leq f (x _ {t}) - \eta \eta_ {L} K \| \nabla f (x _ {t}) \| ^ {2} + \eta \langle \nabla f (x _ {t}), \mathbb {E} _ {t} [ \Delta_ {t} + \eta_ {L} K \nabla f (x _ {t}) ] \rangle + \frac {L}{2} \eta^ {2} \mathbb {E} _ {t} \| \Delta_ {t} \| ^ {2} \\ \leq f (x _ {t}) - \eta \eta_ {L} K \left(\frac {1}{2} - 1 0 L ^ {2} K ^ {2} \eta_ {L} ^ {2} \left(A ^ {2} + 1\right)\right) \| \nabla f (x _ {t}) \| ^ {2} + \frac {5 \eta \eta_ {L} ^ {3} L ^ {2} K ^ {2}}{2} \left(\sigma_ {L} ^ {2} + 4 K \sigma_ {G} ^ {2}\right) \\ + \frac {\eta^ {2} \eta_ {L} ^ {2} K L}{2 m} \sigma_ {L} ^ {2} + \frac {L \eta^ {2} \eta_ {L} ^ {2}}{2 n} V (\frac {1}{m p _ {i} ^ {t}} \hat {g _ {i} ^ {t}}) - \left(\frac {\eta \eta_ {L}}{2 K} - \frac {L \eta^ {2} \eta_ {L} ^ {2}}{2}\right) \mathbb {E} _ {t} \left\| \frac {1}{m} \sum_ {i = 1} ^ {m} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} (x _ {t, k} ^ {i}) \right\| ^ {2} \\ \leq f \left(x _ {t}\right) - c \eta \eta_ {L} K \| \nabla f \left(x _ {t}\right) \| ^ {2} + \frac {5 \eta \eta_ {L} ^ {3} L ^ {2} K ^ {2}}{2} \left(\sigma_ {L} ^ {2} + 4 K \sigma_ {G} ^ {2}\right) + \frac {\eta^ {2} \eta_ {L} ^ {2} K L}{2 m} \sigma_ {L} ^ {2} + \frac {L \eta^ {2} \eta_ {L} ^ {2}}{2 n} V \left(\frac {1}{m p _ {i} ^ {t}} \hat {g} _ {i} ^ {t}\right), \tag {35} \\ \end{array}
$$
where the last inequality follows from  $\left(\frac{\eta\eta_L}{2K} -\frac{L\eta^2\eta_L^2}{2}\right)\geq 0$  if  $\eta \eta_{l}\leq \frac{1}{KL}$ , and (a9) holds because there exists a constant  $c > 0$  (for some  $\eta_L$ ) satisfying  $\frac{1}{2} -10L^{2}\frac{1}{m}\sum_{i = 1}^{m}K^{2}\eta_{L}^{2}(A^{2} + 1) > c > 0$ . Rearranging and summing from  $t = 0,\ldots ,T - 1$ , we have:
$$
\sum_ {t = 1} ^ {T - 1} c \eta \eta_ {L} K \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} \leq f (x _ {0}) - f (x _ {T}) + T (\eta \eta_ {L} K) \Phi . \tag {36}
$$
Which implies:
$$
\underset {t \in [ T ]} {\min } \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {f _ {0} - f _ {*}}{c \eta \eta_ {L} K T} + \Phi , \tag {37}
$$
$$
\Phi = \frac {1}{c} [ \frac {5 \eta_ {L} ^ {2} K L ^ {2}}{2} (\sigma_ {L} ^ {2} + 4 K \sigma_ {G} ^ {2}) + \frac {\eta \eta_ {L} L}{2 m} \sigma_ {L} ^ {2} + \frac {L \eta \eta_ {L}}{2 n K} V (\frac {1}{m p _ {i} ^ {t}} \hat {g _ {i} ^ {t}}) ]. (3 8)
$$
D.1 Proof for convergence rate of FedIS (Theorem 3.1) under Assumption 1-3.
In this section, we compare the convergence rate of FedIS with and without Assumption 4. For comparison, we first provide the convergence result under Assumption 4.
First we show Assumption 4 can be used to bound the update variance  $V\left(\frac{1}{mp_i^t}\hat{g}_i^t\right)$ , and under the sampling probability FedIS (73):
$$
V \left(\frac {1}{m p _ {i} ^ {t}} \hat {g _ {i} ^ {t}}\right) \leq \frac {1}{m ^ {2}} \mathbb {E} \| \sum_ {i = 1} ^ {m} \sum_ {k = 1} ^ {K} \nabla F _ {i} \left(x _ {t, k}, \xi_ {k, t}\right) \| ^ {2} \leq \frac {1}{m} \sum_ {i = 1} ^ {m} K \sum_ {k = 1} ^ {K} \mathbb {E} \| \nabla F _ {i} \left(x _ {t, k}, \xi_ {k, t}\right) \| ^ {2} \leq K ^ {2} G ^ {2} \tag {39}
$$
While for using Assumption 3 instead of additional Assumption 4, we can also bound the update variance:
$$
\begin{array}{l} V \left(\frac {1}{m p _ {i} ^ {t}} \hat {g} _ {i} ^ {t}\right) \leq \frac {1}{m ^ {2}} \mathbb {E} \| \sum_ {i = 1} ^ {m} \sum_ {k = 1} ^ {K} \nabla F _ {i} (x _ {t, k}, \xi_ {k, t}) \| ^ {2} \leq \frac {1}{m} \sum_ {i = 1} ^ {m} K \sum_ {k = 1} ^ {K} \mathbb {E} \| \nabla F _ {i} (x _ {t, k}, \xi_ {k, t}) \| ^ {2} \\ \leq K ^ {2} \sigma_ {G} ^ {2} + K ^ {2} \left(A ^ {2} + 1\right) \| \nabla f (x _ {t}) \| ^ {2} \tag {40} \\ \end{array}
$$
We replace the variance back to equation (35):
$$
\begin{array}{l} \mathbb {E} _ {t} [ f (x _ {t + 1}) ] \leq f (x _ {t}) - \eta \eta_ {L} K \| \nabla f (x _ {t}) \| ^ {2} + \eta \left\langle \nabla f (x _ {t}), \mathbb {E} _ {t} [ \Delta_ {t} + \eta_ {L} K \nabla f (x _ {t}) ] \right\rangle + \frac {L}{2} \eta^ {2} \mathbb {E} _ {t} \| \Delta_ {t} \| ^ {2} \\ \leq f (x _ {t}) - \eta \eta_ {L} K \left(\frac {1}{2} - 1 0 L ^ {2} K ^ {2} \eta_ {L} ^ {2} (A ^ {2} + 1)\right) \| \nabla f (x _ {t}) \| ^ {2} + \frac {5 \eta \eta_ {L} ^ {3} L ^ {2} K ^ {2}}{2} (\sigma_ {L} ^ {2} + 4 K \sigma_ {G} ^ {2}) \\ + \frac {\eta^ {2} \eta_ {L} ^ {2} K L}{2 m} \sigma_ {L} ^ {2} + \frac {L \eta^ {2} \eta_ {L} ^ {2}}{2 n} V \left(\frac {1}{m p _ {i} ^ {t}} \hat {g _ {i} ^ {t}}\right) - \left(\frac {\eta \eta_ {L}}{2 K} - \frac {L \eta^ {2} \eta_ {L} ^ {2}}{2}\right) \mathbb {E} _ {t} \left\| \frac {1}{m} \sum_ {i = 1} ^ {m} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} (x _ {t, k} ^ {i}) \right\| ^ {2} \\ \leq f (x _ {t}) - \eta \eta_ {L} K \left(\frac {1}{2} - 1 0 L ^ {2} K ^ {2} \eta_ {L} ^ {2} (A ^ {2} + 1) - \frac {L \eta \eta_ {L} K (A ^ {2} + 1)}{2 n}\right) \| \nabla f (x _ {t}) \| ^ {2} \\ + \frac {5 \eta \eta_ {L} ^ {3} L ^ {2} K ^ {2}}{2} (\sigma_ {L} ^ {2} + 4 K \sigma_ {G} ^ {2}) + \frac {\eta^ {2} \eta_ {L} ^ {2} K L}{2 m} \sigma_ {L} ^ {2} + \frac {L \eta^ {2} \eta_ {L} ^ {2}}{2 n} K ^ {2} \sigma_ {G} ^ {2} \\ - \left(\frac {\eta \eta_ {L}}{2 K} - \frac {L \eta^ {2} \eta_ {L} ^ {2}}{2}\right) \mathbb {E} _ {t} \left\| \frac {1}{m} \sum_ {i = 1} ^ {m} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \right\| ^ {2}. \tag {41} \\ \end{array}
$$
This shows that the requirement for  $\eta_L$  is different. It needs that there exists a constant  $c > 0$  (for some  $\eta_L$ ) satisfying  $\frac{1}{2} - 10L^2 K^2 \eta_L^2 (A^2 + 1) - \frac{L \eta \eta_L K (A^2 + 1)}{2n} > c > 0$ . One can still guarantee that there exists a constant for  $\eta_L$  to satisfy this inequality according to the properties of quadratic functions. Specifically, for the quadratic equation  $-10L^2 K^2 (A^2 + 1) \eta_L^2 - \frac{L \eta K (A^2 + 1)}{2n} \eta_L + \frac{1}{2}$ , we know that  $-10L^2 K^2 (A^2 + 1) < 0, -\frac{L \eta K (A^2 + 1)}{2n}$  and  $\frac{1}{2} > 0$ . Based on the solution of quadratic equations, we can ensure that there exists a  $\eta_L > 0$  solution.
Then we can substitute equation (35) with equation (41) and let  $\eta_{L} = \mathcal{O}\left(\frac{1}{\sqrt{T}KL}\right)$  and  $\eta = \mathcal{O}\left(\sqrt{Kn}\right)$ , yielding the convergence rate of FedIS under Assumptions 1-3:
$$
\min  _ {t \in [ T ]} E \| \nabla f (x _ {t}) \| ^ {2} \leq \mathcal {O} \left(\frac {f ^ {0} - f ^ {*}}{\sqrt {n K T}}\right) + \underbrace {\mathcal {O} \left(\frac {\sigma_ {L} ^ {2}}{\sqrt {n K T}}\right) + \mathcal {O} \left(\frac {M ^ {2}}{T}\right) + \mathcal {O} \left(\frac {K \sigma_ {G} ^ {2}}{\sqrt {n K T}}\right)} _ {\text {o r d e r o f} \Phi}. \tag {42}
$$
E Convergence of DELTA. Proof of Theorem 3.5
E.1 Convergence rate with improved analysis method for getting DELTA
As we see FedIS can only reduce the update variance term in  $\Phi$ . Since we want to reduce the convergence variance as much as possible, the other term  $\sigma_{L}$  and  $\sigma_{G}$  still needs to be optimized. However, it is not straightforward to derive the optimization problem from  $\Phi$ . In order to further reduce the variance in  $\Phi$  (cf. 4), i.e., local variance  $(\sigma_{L})$  and global variance  $(\sigma_{G})$ , we divide the convergence of the global objective into a surrogate objective and an update gap and analyze them separately. The analysis framework is shown in Figure 8.
Figure 8: Theoretical analysis flow. The figure shows the theoretical analysis flow of FedIS (left) and DELTA (right), highlighting the differences in sampling probability due to variance.
As for the update gap, inspired by the expression form of the update variance, we formally define it as follows:
Definition E.1 (Update gap). In order to measure the update inconsistency, we define the update gap:
$$
\chi_ {t} = \mathbb {E} \left[ \left\| \nabla \tilde {f} \left(x _ {t}\right) - \nabla f \left(x _ {t}\right) \right\| \right]. \tag {43}
$$
Here, the expectation is taken over the distribution of all clients. When all clients participate, we have  $\chi_t^2 = 0$ . The update inconsistency exists as long as only a partial set of clients participate.
The update gap is a direct manifestation of the objective inconsistency in the update process. The presence of an update gap makes the analysis of the global objective different from the analysis of the surrogate objective. However, by ensuring the convergence of the update gap, we can re-derive the convergence result for the global objective. Formally, the update gap allows us to connect global objective convergence and surrogate objective convergence as follows:
$$
\mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} = \mathbb {E} \| \nabla \tilde {f} (x _ {t}) \| ^ {2} + \chi_ {t} ^ {2}. \tag {44}
$$
The equation follows from the property of unbiasedness, as shown in Lemma C.1.
To deduce the convergence rate of the global objective, we begin by examining the convergence analysis of the surrogate objective.
Theorem E.2 (Convergence rate of surrogate objective). Under Assumption 1-3 and let local and global learning rates  $\eta$  and  $\eta_L$  satisfy  $\eta_L < \frac{1}{\sqrt{40KL}\sqrt{\frac{1}{n}\sum_{l=1}^{m}\frac{1}{mp_l^f}}}$  and  $\eta \eta_L \leq \frac{1}{KL}$ , the minimal gradient norm of surrogate objective will be bounded as below:
$$
\min  _ {t \in [ T ]} \mathbb {E} \left\| \nabla \tilde {f} (x _ {t}) \right\| ^ {2} \leq \frac {f ^ {0} - f ^ {*}}{\tilde {c} \eta \eta_ {L} K T} + \frac {\tilde {\Phi}}{\tilde {c}}, \tag {45}
$$
where  $f^0 = f(x_0)$ ,  $f^{*} = f(x_{*})$ , the expectation is over the local dataset samples among workers.
$\tilde{\Phi}$  is the new combination of variance, representing combinations of local variance and client gradient diversity.
$$
\tilde {\Phi} = \frac {5 L ^ {2} K \eta_ {L} ^ {2}}{2 m n} \sum_ {i = 1} ^ {m} \frac {1}{p _ {i} ^ {t}} \left(\sigma_ {L, i} ^ {2} + 4 K \zeta_ {G, i} ^ {2}\right) + \frac {L \eta_ {L} \eta}{2 n} \sum_ {i = 1} ^ {m} \frac {1}{m ^ {2} p _ {i} ^ {t}} \sigma_ {L, i} ^ {2}, \tag {46}
$$
For sampling with replacement:
$$
\tilde {\Phi} = \frac {5 L ^ {2} K \eta_ {L} ^ {2}}{2 m ^ {2}} \sum_ {i = 1} ^ {m} \frac {1}{p _ {i} ^ {t}} \left(\sigma_ {L, i} ^ {2} + 4 K \zeta_ {G, i} ^ {2}\right) + \frac {L \eta_ {L} \eta}{2 n} \sum_ {i = 1} ^ {m} \frac {1}{m ^ {2} p _ {i} ^ {t}} \sigma_ {L, i} ^ {2} \tag {47}
$$
where  $\zeta_{G,i}$  represents client gradient diversity:  $\zeta_{G,i} = \|\nabla F_i(x_t) - \nabla f(x_t)\|^4$ , and  $\tilde{c}$  is a constant. The proof of Theorem E.2 is provided in Appendix E.2.1 and Appendix E.2.2. Specifically, the proof for sampling with replacement is shown in Appendix E.2.1, while the proof for sampling without replacement is shown in Appendix E.2.2.
Remark E.3. We observe that there is no update variance in  $\tilde{\Phi}$ , but the local variance and global variance are still present. Additionally, the new combination of variance  $\tilde{\Phi}$  can be minimized by optimizing the sampling probability, as will be shown later.
Derive the convergence from surrogate objective to global objective. As shown in Lemma C.1, unbiased sampling guarantees that the expected partial client updates are equal to the participation of all clients. With sufficient training rounds, unbiased sampling can ensure that the update gap  $\chi^2$  will converge to zero. However, we still need to know the convergence speed of  $\chi_t^2$  to recover the convergence rate of the global objective. Fortunately, we can bound the convergence behavior of  $\chi_t^2$  by the convergence rate of the surrogate objective according to Definition E.1 and Lemma C.2. This means that the update gap can achieve at least the same convergence rate as the surrogate objective.
Corollary E.4 (New convergence rate of global objective). Under Assumption 1-3 and based on the above analysis that update variance is bounded, the global objective will converge to a stationary point. Its gradient is bounded as:
$$
\min  _ {t \in [ T ]} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} = \min  _ {t \in [ T ]} \mathbb {E} \| \nabla \tilde {f} (x _ {t}) \| ^ {2} + \mathbb {E} \| \chi_ {t} ^ {2} \| \leq \min  _ {t \in [ T ]} 2 \mathbb {E} \| \nabla \tilde {f} (x _ {t}) \| ^ {2} \leq \frac {f ^ {0} - f ^ {*}}{c \eta_ {L} K T} + \frac {\tilde {\Phi}}{c}. \tag {48}
$$
Theorem E.5 (Restate of Theorem 3.5). Under Assumptions 1-3 and the same conditions as in Theorem 3.1, the minimal gradient norm of the surrogate objective will be bounded as follows by setting  $\eta_L = \frac{1}{\sqrt{TKL}}$  and  $\eta \sqrt{Kn}$ . Let the local and global learning rates  $\eta$  and  $\eta_L$  satisfy  $\eta_L < \frac{1}{\sqrt{40KL}\sqrt{\frac{1}{n}\sum_{l=1}^{m}\frac{1}{mp_l^t}}}$  and  $\eta \eta_L \leq \frac{1}{KL}$ . Under Assumptions 1-3 and with partial worker participation, the sequence of outputs  $x_k$  generated by Algorithm 1 satisfies:
$$
\underset {t \in [ T ]} {\min } \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} \leq \frac {F}{c \eta \eta_ {L} K T} + \frac {1}{c} \tilde {\Phi}, \tag {49}
$$
where  $F = f(x_0) - f(x_*)$ , and the expectation is over the local dataset samplings among workers.  $c$  is a constant.  $\zeta_{G,i}$  is defined as client gradient diversity:  $\zeta_{G,i} = \| \nabla F_i(x_t) - \nabla f(x_t)\|$ .
For sample with replacement:  $\tilde{\Phi} = \frac{5L^2K\eta_L^2}{2m^2}\sum_{l = 1}^{m}\frac{1}{p_l^t} (\sigma_{L,l}^2 +4K\zeta_{G,l}^2) + \frac{L\eta_L\eta}{2n}\sum_{l = 1}^{m}\frac{1}{m^2p_l^t}\sigma_{L,i}^2.$
For sampling without replacement:  $\tilde{\Phi} = \frac{5L^2K\eta_L^2}{2mn}\sum_{l = 1}^{m}\frac{1}{p_l^t} (\sigma_{L,l}^2 +4K\zeta_{G,l}^2) + \frac{L\eta_L\eta}{2n}\sum_{l = 1}^{m}\frac{1}{m^2p_l^t}\sigma_{L,l}^2.$
Remark E.6 (Condition of  $\eta_L$ ). Here, though the condition expression for  $\eta_L$  relies on a dynamic sampling probability  $p_l^t$ , we can still guarantee that there is a constant  $\eta_L$  satisfying this condition.
Specifically, one can substitute the optimal sampling probability  $\frac{1}{p_i^t} = \frac{\sum_{j=1}^{m} \sqrt{\alpha_1 \zeta_{G,j}^2 + \alpha_2 \sigma_{L,j}^2}}{\sqrt{\alpha_1 \zeta_{G,i}^2 + \alpha_2 \sigma_{L,i}^2}}$  back to the above inequality condition. As long as the gradient  $\nabla F_i(x_t)$  is bounded, we can ensure  $\frac{1}{m^2} \sum_{i=1}^{m} \frac{\sum_{j=1}^{m} \sqrt{\alpha_1 \zeta_{G,j}^2 + \alpha_2 \sigma_{L,j}^2}}{\sqrt{\alpha_1 \zeta_{G,i}^2 + \alpha_2 \sigma_{L,i}^2}} \leq \frac{\max_j \sqrt{\alpha_1 \zeta_{G,j}^2 + \alpha_2 \sigma_{L,j}^2}}{\min_i \sqrt{\alpha_1 \zeta_{G,i}^2 + \alpha_1 \sigma_{L,i}^2}} \leq \tilde{G}$ , therefore  $\frac{1}{2\sqrt{10(A^2+1)}KL\sqrt{\frac{1}{m^2} \sum_{i=1}^{m} \frac{\sum_{j=1}^{m} \sqrt{\alpha_1 \zeta_{G,j}^2 + \alpha_2 \sigma_{L,j}^2}}{\sqrt{\alpha_1 \zeta_{G,i}^2 + \alpha_2 \sigma_{L,i}^2}}} \geq \frac{1}{2\sqrt{10(A^2+1)}KL\sqrt{\tilde{G}}} \geq C$ , where  $\tilde{G}$  and  $C$  are positive constants. Thus, we can always find a constant  $\eta_L$  to satisfy this inequality under dynamic sampling probability  $p_i^t$ .
In the Appendix, we abbreviate  $\zeta_{G,i,t}$  to  $\zeta_{G,i}$  for the sake of simplicity in notation, without any loss of generality.
Corollary E.7 (Convergence rate of DELTA). Suppose  $\eta_L$  and  $\eta$  are such that the conditions mentioned above are satisfied,  $\eta_L = \mathcal{O}\left(\frac{1}{\sqrt{T}KL}\right)$  and  $\eta = \mathcal{O}\left(\sqrt{Kn}\right)$ . Then for sufficiently large  $T$ , the iterates of Theorem 3.5 satisfy:
$$
\min  _ {t \in [ T ]} \mathbb {E} \| \nabla f (x _ {t}) \| ^ {2} \leq \mathcal {O} \left(\frac {F}{\sqrt {n K T}}\right) + \mathcal {O} \left(\frac {\sigma_ {L} ^ {2}}{\sqrt {n K T}}\right) + \mathcal {O} \left(\frac {\sigma_ {L} ^ {2} + 4 K \zeta_ {G} ^ {2}}{K T}\right). \tag {50}
$$
Lemma E.8. For any step-size satisfying  $\eta_L \leq \frac{1}{8LK}$ , we can have the following results:
$$
\mathbb {E} \| x _ {i, k} ^ {t} - x _ {t} \| ^ {2} \leq 5 K \left(\eta_ {L} ^ {2} \sigma_ {L} ^ {2} + 4 K \eta_ {L} ^ {2} \zeta_ {G, i} ^ {2}\right) + 2 0 K ^ {2} \left(A ^ {2} + 1\right) \eta_ {L} ^ {2} \| \nabla f (x _ {t}) \| ^ {2}. \tag {51}
$$
where  $\zeta_{G,i} = \| \nabla F(x_t) - \nabla f(x_t)\|$ , and the expectation is over local SGD and filtration of  $x_{t}$  without the stochasticity of client sampling.
$$
\begin{array}{l} \mathbb {E} _ {t} \| x _ {t, k} ^ {i} - x _ {t} \| ^ {2} \\ = \mathbb {E} _ {t} \| x _ {t, k - 1} ^ {i} - x _ {t} - \eta_ {L} g _ {t, k - 1} ^ {t} \| ^ {2} \\ = \mathbb {E} _ {t} \| x _ {t, k - 1} ^ {i} - x _ {t} - \eta_ {L} (g _ {t, k - 1} ^ {t} - \nabla F _ {i} (x _ {t, k - 1} ^ {i}) + \nabla F _ {i} (x _ {t, k - 1} ^ {i}) - \nabla F _ {i} (x _ {t}) + \nabla F _ {i} (x _ {t})) \| ^ {2} \\ \leq (1 + \frac {1}{2 K - 1}) \mathbb {E} _ {t} \| x _ {t, k - 1} ^ {i} - x _ {t} \| ^ {2} + \mathbb {E} _ {t} \| \eta_ {L} (g _ {t, k - 1} ^ {t} - \nabla F _ {i} (x _ {t, k} ^ {i})) \| ^ {2} \\ + 4 K \mathbb {E} _ {t} [ \| \eta_ {L} (\nabla F _ {i} (x _ {t, K - 1} ^ {i}) - \nabla F _ {i} (x _ {t})) \| ^ {2} ] + 4 K \eta_ {L} ^ {2} \mathbb {E} _ {t} \| \nabla F _ {i} (x _ {t}) \| ^ {2} \\ \leq (1 + \frac {1}{2 K - 1}) \mathbb {E} _ {t} \| x _ {t, k - 1} ^ {i} - x _ {t} \| ^ {2} + \eta_ {L} ^ {2} \sigma_ {L} ^ {2} + 4 K \eta_ {L} ^ {2} L ^ {2} \mathbb {E} _ {t} \| x _ {t, k - 1} ^ {i} - x _ {t} \| ^ {2} \\ + 4 K \eta_ {L} ^ {2} \zeta_ {G, i} ^ {2} + 4 K \eta_ {L} ^ {2} (A ^ {2} + 1) \| \nabla f (x _ {t}) \| ^ {2} \\ \leq \left(1 + \frac {1}{K - 1}\right) \mathbb {E} \| x _ {t, k - 1} ^ {i} - x _ {t} \| ^ {2} + \eta_ {L} ^ {2} \sigma_ {L} ^ {2} + 4 K \eta_ {L} ^ {2} \zeta_ {G, i} ^ {2} + 4 K \left(A ^ {2} + 1\right) \| \eta_ {L} \nabla f (x _ {t}) \| ^ {2}. \tag {52} \\ \end{array}
$$
Unrolling the recursion, we get: