SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} _ {t} \| x _ {t, k} ^ {i} - x _ {t} \| ^ {2} \leq \sum_ {p = 0} ^ {k - 1} (1 + \frac {1}{K - 1}) ^ {p} \left[ \eta_ {L} ^ {2} \sigma_ {L} ^ {2} + 4 K \eta_ {L} ^ {2} \zeta_ {G, i} ^ {2} + 4 K (A ^ {2} + 1) \| \eta_ {L} \nabla f (x _ {t}) \| ^ {2} \right] \\ \leq (K - 1) \left[ \left(1 + \frac {1}{K - 1}\right) ^ {K} - 1 \right] \left[ \eta_ {L} ^ {2} \sigma_ {L} ^ {2} + 4 K \eta_ {L} ^ {2} \zeta_ {G, i} ^ {2} + 4 K (A ^ {2} + 1) \| \eta_ {L} \nabla f (x _ {t}) \| ^ {2} \right] \\ \leq 5 K \left(\eta_ {L} ^ {2} \sigma_ {L} ^ {2} + 4 K \eta_ {L} ^ {2} \zeta_ {G, i} ^ {2}\right) + 2 0 K ^ {2} \left(A ^ {2} + 1\right) \eta_ {L} ^ {2} \| \nabla f (x _ {t}) \| ^ {2}. \tag {53} \\ \end{array}
$$
E.2 Proof for Theorem E.2.
In Section E.2.1 and Section E.2.2, we provide the proof for Theorem E.2. Specifically, the proof for sampling with replacement is shown in Appendix E.2.1, while the proof for sampling without replacement is shown in Appendix E.2.2.
E.2.1 Sample with replacement
$$
\min  _ {t \in [ T ]} \mathbb {E} \| \nabla \tilde {f} (x _ {t}) \| ^ {2} \leq \frac {f _ {0} - f _ {*}}{c \eta \eta_ {L} K T} + \frac {1}{c} \tilde {\Phi}, \tag {54}
$$
where  $\tilde{\Phi} = \frac{5L^2K\eta_L^2}{2m^2}\sum_{l = 1}^{m}\frac{1}{p_l^t} (\sigma_L^2 +4K\zeta_{G,i}^2) + \frac{L\eta_L\eta}{2n}\sum_{l = 1}^{m}\frac{1}{m^2p_l^t}\sigma_L^2.$
$$
\begin{array}{l} \mathbb {E} _ {t} [ \tilde {f} (x _ {t + 1}) ] \stackrel {(a 1)} {\leq} \tilde {f} (x _ {t}) + \left\langle \nabla \tilde {f} (x _ {t}), \mathbb {E} _ {t} [ x _ {t + 1} - x _ {t} ] \right\rangle + \frac {L}{2} \mathbb {E} _ {t} [ \| x _ {t + 1} - x _ {t} \| ^ {2} ] \\ = \tilde {f} (x _ {t}) + \left\langle \nabla \tilde {f} (x _ {t}), \mathbb {E} _ {t} [ \eta \Delta_ {t} + \eta \eta_ {L} K \nabla \tilde {f} (x _ {t}) - \eta \eta_ {L} K \nabla \tilde {f} (x _ {t}) ] \right\rangle + \frac {L}{2} \eta^ {2} \mathbb {E} _ {t} [ \| \Delta_ {t} \| ^ {2} ] \\ = \tilde {f} (x _ {t}) - \eta \eta_ {L} K \left\| \nabla \tilde {f} (x _ {t}) \right\| ^ {2} + \eta \underbrace {\left\langle \nabla \tilde {f} (x _ {t}) , \mathbb {E} _ {t} [ \Delta_ {t} + \eta_ {L} K \nabla \tilde {f} (x _ {t}) ] \right\rangle} _ {A _ {1}} + \frac {L}{2} \eta^ {2} \underbrace {\mathbb {E} _ {t} \| \Delta_ {t} \| ^ {2}} _ {A _ {2}} \tag {55} \\ \end{array}
$$
Where (a1) follows from the Lipschitz continuity condition. Here, the expectation is over the local data SGD and the filtration of  $x_{t}$ . However, in the next analysis, the expectation is over all randomness, including client sampling. This is achieved by taking expectation on both sides of the above equation over client sampling.
To begin, let us consider  $A_{1}$ :
$$
\begin{array}{l} A _ {1} = \left\langle \nabla \tilde {f} (x _ {t}), \mathbb {E} _ {t} \left[ \Delta_ {t} + \eta_ {L} K \nabla \tilde {f} (x _ {t}) \right] \right\rangle \\ = \left\langle \nabla \tilde {f} (x _ {t}), \mathbb {E} _ {t} \left[ - \frac {1}{| S _ {t} |} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \eta_ {L} g _ {t, k} ^ {i} + \eta_ {L} K \nabla \tilde {f} (x _ {t}) \right] \right\rangle \\ \stackrel {(a 2)} {=} \left\langle \nabla \tilde {f} (x _ {t}), \mathbb {E} _ {t} [ - \frac {1}{| S _ {t} |} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \eta_ {L} \nabla F _ {i} (x _ {t, k} ^ {i}) + \eta_ {L} K \nabla \tilde {f} (x _ {t}) ] \right\rangle \\ = \left\langle \sqrt {K \eta_ {L}} \nabla \tilde {f} (x _ {t}), \frac {\sqrt {\eta_ {L}}}{\sqrt {K}} \mathbb {E} _ {t} \left[ - \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) + K \nabla \tilde {f} (x _ {t}) \right] \right\rangle \\ \stackrel {(a 3)} {=} \frac {K \eta_ {L}}{2} \| \nabla \tilde {f} (x _ {t}) \| ^ {2} + \frac {\eta_ {L}}{2 K} \mathbb {E} _ {t} \left(\| - \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} (x _ {t, k} ^ {i}) + K \nabla \tilde {f} (x _ {t}) \| ^ {2}\right) \\ - \frac {\eta_ {L}}{2 K} \mathbb {E} _ {t} \| - \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \| ^ {2}, \tag {56} \\ \end{array}
$$
where (a2) follows from Assumption 2, and (a3) is due to  $\langle x,y\rangle = \frac{1}{2}\left[\| x\| ^2 +\| y\| ^2 -\| x - y\| ^2\right]$  for  $x = \sqrt{K\eta_L}\nabla \tilde{f} (x_t)$  and  $y = \frac{\sqrt{\eta_L}}{K} [-\frac{1}{n}\sum_{i\in S_t}\frac{1}{mp_i^t}\sum_{k = 0}^{K - 1}\nabla F_i(x_{t,k}^i) + K\nabla \tilde{f} (x_t)]$
To bound  $A_{1}$ , we need to bound the following part:
$$
\begin{array}{l} \mathbb {E} _ {t} \| \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} (x _ {t, k} ^ {i}) - K \nabla \tilde {f} (x _ {t}) \| ^ {2} \\ = \mathbb {E} _ {t} \| \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) - \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} \left(x _ {t}\right) \| ^ {2} \\ \stackrel {(a 4)} {\leq} \frac {K}{n} \sum_ {i \in S _ {t}} \sum_ {k = 0} ^ {K - 1} \mathbb {E} _ {t} \| \frac {1}{m p _ {i} ^ {t}} (\nabla F _ {i} (x _ {t, k} ^ {i}) - \nabla F _ {i} (x _ {t})) \| ^ {2} \\ = \frac {K}{n} \sum_ {i \in S _ {t}} \sum_ {k = 0} ^ {K - 1} \mathbb {E} _ {t} \left\{\mathbb {E} _ {t} \left(\| \frac {1}{m p _ {i} ^ {t}} \left(\nabla F _ {i} \left(x _ {t, k} ^ {i}\right) - \nabla F _ {i} \left(x _ {t}\right)\right) \| ^ {2} \mid S\right) \right\} \\ = \frac {K}{n} \sum_ {i \in S _ {t}} \sum_ {k = 0} ^ {K - 1} \mathbb {E} _ {t} \left(\sum_ {l = 1} ^ {m} \frac {1}{m ^ {2} p _ {l} ^ {t}} \| \nabla F _ {l} \left(x _ {t, k} ^ {l}\right) - \nabla F _ {l} \left(x _ {t}\right) \| ^ {2}\right) \\ = K \sum_ {k = 0} ^ {K - 1} \sum_ {l = 1} ^ {m} \frac {1}{m ^ {2} p _ {l} ^ {t}} \mathbb {E} _ {t} \| \nabla F _ {l} \left(x _ {t, k} ^ {l}\right) - \nabla F _ {l} \left(x _ {t}\right) \| ^ {2} \\ \stackrel {(a 5)} {\leq} \frac {K ^ {2}}{m ^ {2}} \sum_ {l = 1} ^ {m} \frac {L ^ {2}}{p _ {l} ^ {t}} \mathbb {E} \| x _ {t, k} ^ {l} - x _ {t} \| ^ {2} \\ \stackrel {(a 6)} {\leq} \frac {L ^ {2} K ^ {2}}{m ^ {2}} \sum_ {l = 1} ^ {m} \frac {1}{p _ {l} ^ {t}} \left(5 K \left(\eta_ {L} ^ {2} \sigma_ {L} ^ {2} + 4 K \eta_ {L} ^ {2} \zeta_ {G, i} ^ {2}\right) + 2 0 K ^ {2} \left(A ^ {2} + 1\right) \eta_ {L} ^ {2} \| \nabla f (x _ {t}) \| ^ {2}\right) \\ = \frac {5 L ^ {2} K ^ {3} \eta_ {L} ^ {2}}{m ^ {2}} \sum_ {l = 1} ^ {m} \frac {1}{p _ {l} ^ {t}} \left(\sigma_ {L} ^ {2} + 4 K \sigma_ {G} ^ {2}\right) + \frac {2 0 L ^ {2} K ^ {4} \eta_ {L} ^ {2} \left(A ^ {2} + 1\right)}{m ^ {2}} \sum_ {l = 1} ^ {m} \frac {1}{p _ {l} ^ {t}} \| \nabla f (x _ {t}) \| ^ {2}, \tag {57} \\ \end{array}
$$
where (a4) follows from the fact that  $\mathbb{E}|x_1 + \dots + x_n|^2 \leq n\mathbb{E}\left(|x_1|^2 + \dots + |x_n|^2\right)$ , (a5) is a consequence of Assumption 1, and (a6) is a result of Lemma E.8.
Combining the above expressions, we obtain:
$$
\begin{array}{l} A _ {1} \leq \frac {K \eta_ {L}}{2} \| \nabla \tilde {f} (x _ {t}) \| ^ {2} + \frac {\eta_ {L}}{2 K} \left[ \frac {5 L ^ {2} K ^ {3} \eta_ {L} ^ {2}}{m ^ {2}} \sum_ {l = 1} ^ {m} \frac {1}{p _ {l} ^ {t}} (\sigma_ {L} + 4 K \zeta_ {G, i} ^ {2}) \right. \\ \left. + \frac {2 0 L ^ {2} K ^ {4} \eta_ {L} ^ {2} \left(A ^ {2} + 1\right)}{m ^ {2}} \sum_ {l = 1} ^ {m} \frac {1}{p _ {l} ^ {t}} \| \nabla f (x _ {t}) \| ^ {2} \right] - \frac {\eta_ {L}}{2 K} \mathbb {E} _ {t} \| - \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \| ^ {2}. \tag {58} \\ \end{array}
$$
Next, we consider bounding  $A_{2}$ :
$$
\begin{array}{l} A _ {2} = \mathbb {E} _ {t} \| \Delta_ {t} \| ^ {2} \\ = \mathbb {E} _ {t} \left\| - \eta_ {L} \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} g _ {t, k} ^ {i} \right\| ^ {2} \\ = \eta_ {L} ^ {2} \mathbb {E} _ {t} \left\| \frac {1}{n} \sum_ {i \in S _ {t}} \sum_ {k = 0} ^ {K - 1} \left(\frac {1}{m p _ {i} ^ {t}} g _ {t, k} ^ {i} - \frac {1}{m p _ {i} ^ {t}} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right)\right) \right\| ^ {2} + \eta_ {L} ^ {2} \mathbb {E} _ {t} \left\| - \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \right\| ^ {2} \\ = \eta_ {L} ^ {2} \frac {1}{n ^ {2}} \sum_ {i \in S _ {t}} \sum_ {k = 0} ^ {K - 1} \mathbb {E} _ {t} \left\| \frac {1}{m p _ {i} ^ {t}} g _ {t, k} ^ {i} - \frac {1}{m p _ {i} ^ {t}} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \right\| ^ {2} + \eta_ {L} ^ {2} \mathbb {E} _ {t} \left\| - \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \right\| ^ {2} \\ = \eta_ {L} ^ {2} \frac {1}{n ^ {2}} \sum_ {k = 0} ^ {K - 1} \mathbb {E} _ {t} \left(\mathbb {E} \left\| \frac {1}{m p _ {i} ^ {t}} \left(g _ {t, k} ^ {i} - \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \right. \right\| ^ {2} \mid S\right) + \eta_ {L} ^ {2} \mathbb {E} _ {t} \left\| - \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \right\| ^ {2} \\ = \eta_ {L} ^ {2} \frac {1}{n ^ {2}} \sum_ {k = 0} ^ {K - 1} \mathbb {E} _ {t} \left(\sum_ {l = 1} ^ {m} \frac {1}{m ^ {2} p _ {l} ^ {t}} \left\| g _ {t, k} ^ {i} - \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \right\| ^ {2}\right) + \eta_ {L} ^ {2} \mathbb {E} _ {t} \left\| - \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \right\| ^ {2} \\ \stackrel {(a 7)} {\leq} \eta_ {L} ^ {2} \frac {K}{n} \sum_ {l = 1} ^ {m} \frac {1}{m ^ {2} p _ {l} ^ {t}} \sigma_ {L} ^ {2} + \eta_ {L} ^ {2} \mathbb {E} _ {t} \left\| - \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \right\| ^ {2}, \tag {59} \\ \end{array}
$$
where  $S$  represents the whole sample space and (a7) is due to Assumption 2.
Now we substitute the expressions for  $A_{1}$  and  $A_{2}$  and take the expectation over the client sampling distribution on both sides. It should be noted that the derivation of  $A_{1}$  and  $A_{2}$  above is based on considering the expectation over the sampling distribution:
$$
\begin{array}{l} f \left(x _ {t + 1}\right) \leq f \left(x _ {t}\right) - \eta \eta_ {L} K \mathbb {E} _ {t} \left\| \nabla \tilde {f} \left(x _ {t}\right) \right\| ^ {2} + \eta \mathbb {E} _ {t} \left\langle \nabla \tilde {f} \left(x _ {t}\right), \Delta_ {t} + \eta_ {L} K \nabla \tilde {f} \left(x _ {t}\right) \right\rangle + \frac {L}{2} \eta^ {2} \mathbb {E} _ {t} \| \Delta_ {t} \| ^ {2} \\ \stackrel {(a 8)} {\leq} f (x _ {t}) - K \eta \eta_ {L} \left(\frac {1}{2} - \frac {2 0 K ^ {2} \eta_ {L} ^ {2} L ^ {2} (A ^ {2} + 1)}{m ^ {2}} \sum_ {l = 1} ^ {m} \frac {1}{p _ {l} ^ {t}}\right) \mathbb {E} _ {t} \left\| \nabla \tilde {f} (x _ {t}) \right\| ^ {2} \\ + \frac {5 L ^ {2} K ^ {2} \eta_ {L} ^ {3} \eta}{2 m ^ {2}} \sum_ {l = 1} ^ {m} \frac {1}{p _ {l} ^ {t}} \left(\sigma_ {L} + 4 K \zeta_ {G, i} ^ {2}\right) \\ + \frac {L \eta_ {L} ^ {2} \eta^ {2} K}{2 n} \sum_ {l = 1} ^ {m} \frac {1}{m ^ {2} p _ {l} ^ {t}} \sigma_ {L} ^ {2} - \left(\frac {\eta \eta_ {L}}{2 K} - \frac {L \eta^ {2} \eta_ {L} ^ {2}}{2}\right) \mathbb {E} _ {t} \left\| - \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla f _ {i} \left(x _ {t, k} ^ {i}\right) \right\| ^ {2} \\ \stackrel {(a 9)} {\leq} f (x _ {t}) - K \eta \eta_ {L} \left(\frac {1}{2} - \frac {2 0 K ^ {2} \eta_ {L} ^ {2} L ^ {2} (A ^ {2} + 1)}{m ^ {2}} \sum_ {l = 1} ^ {m} \frac {1}{p _ {l} ^ {t}}\right) \mathbb {E} _ {t} \| \nabla \tilde {f} (x _ {t}) \| ^ {2} \\ + \frac {5 L ^ {2} K ^ {2} \eta_ {L} ^ {3} \eta}{2 m ^ {2}} \sum_ {l = 1} ^ {m} \frac {1}{p _ {l} ^ {t}} \left(\sigma_ {L} + 4 K \zeta_ {G, i} ^ {2}\right) + \frac {L \eta_ {L} ^ {2} \eta^ {2} K}{2 n} \sum_ {l = 1} ^ {m} \frac {1}{m ^ {2} p _ {l} ^ {t}} \sigma_ {L} ^ {2} \\ \stackrel {(a 1 0)} {\leq} f \left(x _ {t}\right) - c K \eta \eta_ {L} \mathbb {E} _ {t} \| \nabla \tilde {f} \left(x _ {t}\right) \| ^ {2} + \frac {5 L ^ {2} K ^ {2} \eta_ {L} ^ {3} \eta}{2 m ^ {2}} \sum_ {l = 1} ^ {m} \frac {1}{p _ {l} ^ {t}} \left(\sigma_ {L} ^ {2} + 4 K \zeta_ {G, i} ^ {2}\right) + \frac {L \eta_ {L} ^ {2} \eta^ {2} K}{2 n} \sum_ {l = 1} ^ {m} \frac {1}{m ^ {2} p _ {l} ^ {t}} \sigma_ {L} ^ {2}, \tag {60} \\ \end{array}
$$
where (a8) comes from Lemma C.2, (a9) follows from  $\left(\frac{\eta\eta_L}{2K} -\frac{L\eta^2\eta_L^2}{2}\right)\geq 0$  if  $\eta \eta_{l}\leq \frac{1}{KL}$ , and (a10) holds because there exists a constant  $c > 0$  satisfying  $(\frac{1}{2} -\frac{20K^2\eta_L^2L^2(A^2 + 1)}{m^2}\sum_{l = 1}^{m}\frac{1}{p_l^t}) > c > 0$  if  $\eta_L < \frac{1}{2\sqrt{10(A^2 + 1)}KL\sqrt{\frac{1}{m}\sum_{l = 1}^{m}\frac{1}{mp_l^t}}}$ .
Rearranging and summing from  $t = 0,\dots ,T - 1$  , we have:
$$
\begin{array}{l} \sum_ {t = 1} ^ {T - 1} c \eta_ {L} K \mathbb {E} \| \nabla \tilde {f} (x _ {t}) \| ^ {2} \leq f (x _ {0}) - f (x _ {T}) \\ + T \left(\eta_ {L} K\right) \left(\frac {5 L ^ {2} K \eta_ {L} ^ {2}}{2 m ^ {2}} \sum_ {l = 1} ^ {m} \frac {1}{p _ {l} ^ {t}} \left(\sigma_ {L} ^ {2} + 4 K \zeta_ {G, i} ^ {2}\right) + \frac {L \eta_ {L} \eta}{2 n} \sum_ {l = 1} ^ {m} \frac {1}{m ^ {2} p _ {l} ^ {t}} \sigma_ {L} ^ {2}\right). \tag {61} \\ \end{array}
$$
$$
\min  _ {t \in [ T ]} \mathbb {E} \| \nabla \tilde {f} (x _ {t}) \| ^ {2} \leq \frac {f _ {0} - f _ {*}}{c \eta \eta_ {L} K T} + \frac {1}{c} \tilde {\Phi}, \tag {62}
$$
E.2.2 Sample without replacement
$$
\min  _ {t \in [ T ]} \mathbb {E} \| \nabla \tilde {f} (x _ {t}) \| ^ {2} \leq \frac {f _ {0} - f _ {*}}{c \eta \eta_ {L} K T} + \frac {1}{c} \tilde {\Phi}, \tag {63}
$$
where  $\tilde{\Phi} = \frac{5L^2K\eta_L^2}{2mn}\sum_{l = 1}^{m}\frac{1}{p_l^t} (\sigma_L^2 +4K\zeta_{G,i}^2) + \frac{L\eta_L\eta}{2n}\sum_{l = 1}^{m}\frac{1}{m^2p_l^t}\sigma_L^2.$
$$
\begin{array}{l} \mathbb {E} [ \tilde {f} (x _ {t + 1}) ] \leq \tilde {f} (x _ {t}) + \left\langle \nabla \tilde {f} (x _ {t}), \mathbb {E} [ x _ {t + 1} - x _ {t} ] \right\rangle + \frac {L}{2} \mathbb {E} _ {t} [ \| x _ {t + 1} - x _ {t} \| ] \\ = \tilde {f} (x _ {t}) + \left\langle \nabla \tilde {f} (x _ {t}), \mathbb {E} _ {t} [ \eta \Delta_ {t} + \eta \eta_ {L} K \nabla \tilde {f} (x _ {t}) - \eta \eta_ {L} K \nabla \tilde {f} (x _ {t}) ] \right\rangle + \frac {L}{2} \eta^ {2} \mathbb {E} _ {t} [ \| \Delta_ {t} \| ^ {2} ] \\ = \tilde {f} (x _ {t}) - \eta \eta_ {L} K \left\| \nabla \tilde {f} (x _ {t}) \right\| ^ {2} + \eta \underbrace {\left\langle \nabla \tilde {f} (x _ {t}) , \mathbb {E} _ {t} [ \Delta_ {t} + \eta_ {L} K \nabla \tilde {f} (x _ {t}) ] \right\rangle} _ {A _ {1}} + \frac {L}{2} \eta^ {2} \underbrace {\mathbb {E} _ {t} \| \Delta_ {t} \| ^ {2}} _ {A _ {2}}. \tag {64} \\ \end{array}
$$
Where the first inequality follows from Lipschitz continuous condition. The expectation here is taken over both the local SGD and the filtration of  $x_{t}$ . However, in the subsequent analysis, the expectation is taken over all sources of randomness, including client sampling.
Similarly, we consider  $A_{1}$  first:
$$
\begin{array}{l} A _ {1} = \left\langle \nabla \tilde {f} (x _ {t}), \mathbb {E} _ {t} [ \Delta_ {t} + \eta_ {L} K \nabla \tilde {f} (x _ {t}) ] \right\rangle \\ = \left\langle \nabla \tilde {f} (x _ {t}), \mathbb {E} _ {t} \left[ - \frac {1}{| S _ {t} |} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \eta_ {L} g _ {t, k} ^ {i} + \eta_ {L} K \nabla \tilde {f} (x _ {t}) \right] \right\rangle \\ = \left\langle \nabla \tilde {f} (x _ {t}), \mathbb {E} _ {t} \left[ - \frac {1}{| S _ {t} |} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \eta_ {L} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) + \eta_ {L} K \nabla \tilde {f} (x _ {t}) \right] \right\rangle \\ = \left\langle \sqrt {K \eta_ {L}} \nabla \tilde {f} (x _ {t}), \frac {\sqrt {\eta_ {L}}}{\sqrt {K}} \mathbb {E} _ {t} \left[ - \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} (x _ {t, k} ^ {i}) + K \nabla \tilde {f} (x _ {t}) \right] \right\rangle \\ = \frac {K \eta_ {L}}{2} \left\| \nabla \tilde {f} (x _ {t}) \right\| ^ {2} + \frac {\eta_ {L}}{2 K} \mathbb {E} _ {t} \left\| - \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} (x _ {t, k} ^ {i}) + K \nabla \tilde {f} (x _ {t}) \right\| ^ {2} \\ - \frac {\eta_ {L}}{2 K} \mathbb {E} _ {t} \left\| - \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \right\| ^ {2}. \tag {65} \\ \end{array}
$$
Since  $x_{i}$  are sampled from  $S_{t}$  without replacement, this causes pairs  $x_{i1}$  and  $x_{i2}$  to no longer be independent. We introduce the activation function as follows:
$$
\mathbb {I} _ {m} \triangleq \left\{ \begin{array}{l l} 1 & i f x \in S _ {t}, \\ 0 & \text {o t h e r w i s e}. \end{array} \right. \tag {66}
$$
Then we obtain the following bound:
$$
\begin{array}{l} \mathbb {E} _ {t} \left\| \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} (x _ {t, k} ^ {i}) - K \nabla \tilde {f} (x _ {t}) \right\| ^ {2} \\ = \mathbb {E} _ {t} \left\| \frac {1}{n} \sum_ {l = 1} ^ {m} \mathbb {I} _ {m} \frac {1}{m p _ {l} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {l} \left(x _ {t, k} ^ {l}\right) - \frac {1}{n} \sum_ {l = 1} ^ {m} \mathbb {I} _ {m} \frac {1}{m p _ {l} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {l} \left(x _ {t}\right) \right\| ^ {2} \\ \stackrel {(b 1)} {\leq} \frac {m}{n ^ {2}} \sum_ {l = 1} ^ {m} \mathbb {E} _ {t} \left\| \mathbb {I} _ {m} \frac {1}{m p _ {l} ^ {t}} \sum_ {k = 0} ^ {K - 1} \left(\nabla F _ {l} \left(x _ {t, k} ^ {l}\right) - \nabla F _ {l} \left(x _ {t}\right)\right) \right\| ^ {2} \\ - \frac {1}{n ^ {2}} \sum_ {l _ {1} \neq l _ {2}} \mathbb {E} _ {t} \left\| \left\{\mathbb {I} _ {m} \frac {1}{m p _ {l _ {1}}} \sum_ {k = 0} ^ {K - 1} \left(\nabla F _ {l _ {1}} (x _ {t, k} ^ {l _ {1}}) - \nabla F _ {l _ {1}} (x _ {t})\right) \right\} \right. \\ \left. - \left\{\mathbb {I} _ {m} \frac {1}{m p _ {l _ {2}}} \sum_ {k = 0} ^ {K - 1} \left(\nabla F _ {l _ {2}} \left(x _ {t, k} ^ {l _ {2}}\right) - \nabla F _ {l _ {2}} \left(x _ {t}\right)\right) \right\} \right\| ^ {2} \\ \leq \frac {m}{n ^ {2}} \sum_ {l = 1} ^ {m} \mathbb {E} _ {t} \left\| \mathbb {I} _ {m} \frac {1}{m p _ {l} ^ {t}} \sum_ {k = 0} ^ {K - 1} \left(\nabla F _ {l} \left(x _ {t, k} ^ {l}\right) - \frac {1}{m p _ {l} ^ {t}} \nabla F _ {l} \left(x _ {t}\right)\right) \right\| ^ {2} \\ = \frac {m}{n ^ {2}} \sum_ {l = 1} ^ {m} \mathbb {E} _ {t} \left\{\left\| \mathbb {I} _ {m} \frac {1}{m p _ {l} ^ {t}} \sum_ {k = 0} ^ {K - 1} \left(\nabla F _ {l} (x _ {t, k} ^ {l}) - \frac {1}{m p _ {l} ^ {t}} \nabla F _ {l} (x _ {t})\right) \right\| ^ {2} | \mathbb {I} _ {m} = 1 \right\} \times P (\mathbb {I} _ {m} = 1) \\ + \mathbb {E} _ {t} \left\{ \right.\left\| \right. \mathbb {I} _ {m} \left( \right.\frac {1}{m p _ {l} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {l} \left(x _ {t, k} ^ {l}\right) - \frac {1}{m p _ {l} ^ {t}} \nabla F _ {l} \left(x _ {t}\right)\left. \right\| ^ {2} \mid \mathbb {I} _ {m} = 0 \left. \right\} \times P (\mathbb {I} _ {m} = 0)) \\ = \frac {m}{n ^ {2}} \sum_ {l = 1} ^ {m} n p _ {l} ^ {t} \mathbb {E} \left\| \frac {1}{m p _ {l} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {l} \left(x _ {t, k} ^ {l}\right) - \frac {1}{m p _ {l} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {l} \left(x _ {t}\right) \right\| ^ {2} \\ \stackrel {(b 2)} {\leq} \frac {L ^ {2} K}{m n} \sum_ {k = 0} ^ {K - 1} \sum_ {l = 1} ^ {m} \frac {1}{p _ {l} ^ {t}} \mathbb {E} \| x _ {t, k} ^ {l} - x _ {t} \| ^ {2} \\ \stackrel {(b 3)} {\leq} \frac {L ^ {2} K ^ {2}}{n} \left(5 K \frac {\eta_ {L} ^ {2}}{m} \sum_ {l = 1} ^ {m} \frac {1}{p _ {l} ^ {t}} \left(\sigma_ {L} ^ {2} + 4 K \zeta_ {G, i} ^ {2}\right) + 2 0 K ^ {2} \left(A ^ {2} + 1\right) \eta_ {L} ^ {2} \| \nabla f (x _ {t}) \| ^ {2} \frac {1}{m} \sum_ {l = 1} ^ {m} \frac {1}{p _ {l} ^ {t}}\right), \tag {67} \\ \end{array}
$$
where (b1) follows from  $\| \sum_{i = 1}^{m}t_{i}\|^{2} = \sum_{i\in [m]}\| t_{i}\|^{2} + \sum_{i\neq j}\langle t_{i},t_{j}\rangle \stackrel {c1}{=}\sum_{i\in [m]}m\| t_{i}\|^{2} - \frac{1}{2}\sum_{i\neq j}\| t_{i} - t_{j}\|^{2}$  ((c1) here is due to  $\langle x,y\rangle = \frac{1}{2}\left[\| x\| ^2 +\| y\| ^2 -\| x - y\| ^2\right]$ , (b2) is due to  $\mathbb{E}\| x_1 + \dots +x_n\| ^2\leq n\mathbb{E}\left(\| x_1\| ^2 +\dots +\| x_n\| ^2\right)$ , and (b3) comes from Lemma E.8.
Therefore, we have the bound of  $A_{1}$ :
$$
\begin{array}{l} A _ {1} \leq \frac {K \eta_ {L}}{2} \| \nabla \tilde {f} (x _ {t}) \| ^ {2} + \frac {\eta_ {L} L ^ {2} K}{2 n} \left(5 K \frac {\eta_ {L} ^ {2}}{m} \sum_ {l = 1} ^ {m} \frac {1}{p _ {l} ^ {t}} \left(\sigma_ {L} ^ {2} + 4 K \zeta_ {G, i} ^ {2}\right) \right. \\ \left. + 2 0 K ^ {2} \left(A ^ {2} + 1\right) \eta_ {L} ^ {2} \| \nabla f (x _ {t}) \| ^ {2} \frac {1}{m} \sum_ {l = 1} ^ {m} \frac {1}{p _ {l} ^ {t}}\right) - \frac {\eta_ {L}}{2 K} \mathbb {E} _ {t} \left\| - \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \right\| ^ {2}. \tag {68} \\ \end{array}
$$
The expression for  $A_{2}$  is as follows:
$$
\begin{array}{l} A _ {2} = \mathbb {E} _ {t} \| \Delta_ {t} \| ^ {2} \\ = \mathbb {E} _ {t} \left\| - \eta_ {L} \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} g _ {t, k} ^ {i} \right\| ^ {2} \\ = \eta_ {L} ^ {2} \mathbb {E} _ {t} \left\| \frac {1}{n} \sum_ {i \in S _ {t}} \sum_ {k = 0} ^ {K - 1} \left(\frac {1}{m p _ {i} ^ {t}} g _ {t, k} ^ {i} - \frac {1}{m p _ {i} ^ {t}} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right)\right) \right\| ^ {2} + \eta_ {L} ^ {2} \mathbb {E} _ {t} \left\| - \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \right\| ^ {2} \\ = \eta_ {L} ^ {2} \frac {1}{n ^ {2}} \mathbb {E} _ {t} \left\| \sum_ {l = 1} ^ {m} \mathbb {I} _ {m} \sum_ {k = 0} ^ {K - 1} \frac {1}{m p _ {l} ^ {t}} (g _ {t, k} ^ {l} - \nabla F _ {i} (x _ {t, k} ^ {i})) \right\| ^ {2} + \eta_ {L} ^ {2} \mathbb {E} _ {t} \left\| - \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} (x _ {t, k} ^ {i}) \right\| ^ {2} \\ = \eta_ {L} ^ {2} \frac {1}{n ^ {2}} \sum_ {l = 1} ^ {m} \mathbb {E} _ {t} \left\| \sum_ {l = 1} ^ {m} \mathbb {I} _ {m} \sum_ {k = 0} ^ {K - 1} \frac {1}{m p _ {l} ^ {t}} \left(g _ {t, k} ^ {l} - \nabla F _ {i} \left(x _ {t, k} ^ {i}\right)\right) \right\| ^ {2} + \eta_ {L} ^ {2} \mathbb {E} _ {t} \left\| - \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \right\| ^ {2} \\ = \eta_ {L} ^ {2} \frac {1}{n ^ {2}} \sum_ {l = 1} ^ {m} n p _ {l} ^ {t} \mathbb {E} _ {t} \left\| \sum_ {k = 0} ^ {K - 1} \frac {1}{m p _ {l} ^ {t}} \left(g _ {t, k} ^ {l} - \nabla F _ {i} \left(x _ {t, k} ^ {i}\right)\right) \right\| ^ {2} + \eta_ {L} ^ {2} \mathbb {E} _ {t} \left\| - \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \right\| ^ {2} \\ \leq \eta_ {L} ^ {2} \frac {K}{n} \sum_ {l = 1} ^ {m} \frac {1}{m ^ {2} p _ {l} ^ {t}} \sigma_ {L} ^ {2} + \eta_ {L} ^ {2} \mathbb {E} _ {t} \left\| - \frac {1}{n} \sum_ {i \in S _ {t}} \frac {1}{m p _ {i} ^ {t}} \sum_ {k = 0} ^ {K - 1} \nabla F _ {i} \left(x _ {t, k} ^ {i}\right) \right\| ^ {2}. \tag {69} \\ \end{array}
$$