SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| \nabla f (\boldsymbol {x}) - \nabla f (\boldsymbol {y}) \right\| \leq \left(A L _ {0} + B L _ {1} \| \nabla f (\boldsymbol {x}) \|\right) \| \boldsymbol {x} - \boldsymbol {y} \|,
$$
where  $A = 1 + e^{C} - \frac{e^{C} - 1}{C}$  and  $B = \frac{e^C - 1}{C}$ .
Lemma 3 (Lemma A.3 in [51]). Let  $f: \mathbb{R}^d \to \mathbb{R}$  be  $(L_0, L_1)$ -smooth and  $C > 0$ . For any  $\pmb{x}, \pmb{y} \in \mathbb{R}^d$  with  $\| \pmb{x} - \pmb{y} \| \leq \frac{C}{L_1}$ ,
$$
f (\boldsymbol {y}) \leq f (\boldsymbol {x}) + \langle \nabla f (\boldsymbol {x}), \boldsymbol {y} - \boldsymbol {x} \rangle + \frac {A L _ {0} + B L _ {1} \| \nabla f (\boldsymbol {x}) \|}{2} \| \boldsymbol {x} - \boldsymbol {y} \| ^ {2},
$$
Lemma 4 (Lemma B.1 in [51]). For any  $\mu \geq 0$  and  $\pmb{x},\pmb{y}\in \mathbb{R}^d$
$$
- \left\langle \boldsymbol {x}, \frac {\boldsymbol {y}}{\| \boldsymbol {y} \|} \right\rangle \leq - \mu \| \boldsymbol {x} \| - (1 - \mu) \| \boldsymbol {y} \| + (1 + \mu) \| \boldsymbol {y} - \boldsymbol {x} \|.
$$
Lemma 5 (Lemma 1 in [30]). Assume that  $Z_{1},\ldots ,Z_{T}$  is a martingale difference sequence with respect to the filtration  $\mathcal{F}_t$  and  $\mathbb{E}_t[\exp (Z_t^2 /\sigma_t^2)]\leq \exp (1)$  for all  $t$ , where  $\sigma_1,\dots,\sigma_T$  is a sequence of random variables such that  $\sigma_t\in \mathcal{F}_t$ . Then for any fixed  $\lambda >0$  and  $\delta \in (0,1)$ , with probability at least  $1 - \delta$ ,
$$
\sum_ {t = 1} ^ {T} Z _ {t} \leq \frac {3}{4} \lambda \sum_ {t = 1} ^ {T} \sigma_ {t} ^ {2} + \frac {1}{\lambda} \log \frac {1}{\delta}.
$$
Lemma 6 (Improved Serfling's inequality, Proposition 2.3 in [3]). Let  $\mathcal{X} = (x_1,\dots ,x_N)$  be a finite population of  $N$  real numbers and  $X_{1},\ldots ,X_{n}$  denote a random sample without replacement from  $\mathcal{X}$ . Let  $\mu_{\mathcal{X}} = \frac{1}{N}\sum_{i = 1}^{N}x_{i}$ ,  $\sigma_{\mathcal{X}}^2 = \frac{1}{N}\sum_{i = 1}^{N}(x_i - \mu)^2$ , and  $\hat{\mu}_n = \frac{1}{n}\sum_{i = 1}^{n}X_{i}$ . If  $\max_{i\in [N]}|x_i|\leq b$ , then for any  $\lambda >0$
$$
\mathbb {E} [ \exp (\lambda (\hat {\mu} _ {n} - \mu_ {\mathcal {X}})) ] \leq \exp \left(\frac {b ^ {2} \lambda^ {2}}{2} \frac {n + 1}{n ^ {2}} \left(1 - \frac {n}{N}\right)\right).
$$
Lemma 7 (Lemma 12 in [10]). Suppose  $X_{1},\ldots ,X_{T}$  is a martingale difference sequence in a Hilbert space such that  $\| X_t\| \leq b$  almost surely for some constant  $b$ . Further, assume  $\mathbb{E}_{t - 1}[\| X_t\|^2 ]\leq \sigma_t^2$  with probability 1 for some constants  $\sigma_{t}$ . Then with probability at least  $1 - 3\delta$ , for all  $k$ :
$$
\left\| \sum_ {t = 1} ^ {k} X _ {t} \right\| \leq 3 b \max  \left(1, \log \frac {1}{\delta}\right) + 3 \sqrt {\sum_ {t = 1} ^ {k} \sigma_ {t} ^ {2} \max  \left(1 , \log \frac {1}{\delta}\right)}
$$
A.3 Proof of Lemma 1
$$
\Xi_ {r} := \frac {1}{N I} \sum_ {i = 1} ^ {N} \sum_ {k = 0} ^ {I - 1} \| \bar {\boldsymbol {x}} _ {r} - \boldsymbol {y} _ {r, k} ^ {i} \|.
$$
$\Xi_r$  is the average "lag" (over clients) of the correction  $G_r^i$  behind  $\nabla f(\bar{\boldsymbol{x}}_r)$  due to client sampling, since each  $G_r^i$  is set according to stochastic gradients at  $\boldsymbol{y}_{r,k}^i$ .
For any  $\delta \in (0,1)$ , define
$$
K = \left\{\begin{array}{l l}\left\lceil \frac {\log \left(\frac {R N}{\delta}\right)}{\log \left(\frac {N}{N - \mathcal {S}}\right)} \right\rceil&S <   N\\1&S = N\end{array}\right.
$$
$$
\mathcal {E} = \left\{\max  _ {0 \leq r \leq R - 1, 1 \leq i \leq N} r - q _ {r} ^ {i} \leq K \right\}. \tag {10}
$$
The event  $\mathcal{E}$  occurs when every round in which a client is sampled occurs after no more than  $K$  rounds following the previous round in which that client was sampled. This means that under  $\mathcal{E}$ , the correction  $\pmb{G}_r^i$  was computed within  $K$  rounds of round  $r$ . The next lemma shows that  $\mathcal{E}$  occurs with probability  $1 - \delta$ .
Lemma 8. For any  $\delta$  with  $0 < \delta < 1$ ,  $\mathbb{P}(\mathcal{E}) \geq 1 - \delta$ .
Proof. If  $S = N$ , then  $r - q_r^i = 1$ , and we are done. Otherwise, since each round's participating clients are sampled uniformly without replacement and independently at each round,
$$
\begin{array}{l} \mathbb {P} (r - q _ {r} ^ {i} \leq k) = \sum_ {i = 1} ^ {k} \mathbb {P} (r - q _ {r} ^ {i} = i) \\ = \left\{ \begin{array}{l l} \sum_ {i = 1} ^ {k} \frac {S}{N} \left(1 - \frac {S}{N}\right) ^ {i - 1} & k <   r \\ 1 & \text {o t h e r w i s e} \end{array} \right. \\ \geq \frac {S}{N} \sum_ {i = 1} ^ {k} \left(1 - \frac {S}{N}\right) ^ {i - 1} \\ = \frac {S}{N} \frac {1 - \left(1 - \frac {S}{N}\right) ^ {k}}{1 - \left(1 - \frac {S}{N}\right)} \\ = 1 - \left(1 - \frac {S}{N}\right) ^ {k}. \\ \end{array}
$$
Also by the choice of  $K$ :  $K \geq \frac{\log\left(\frac{RN}{\delta}\right)}{\log\left(\frac{N}{N - S}\right)}$ , we can show that  $\left(1 - \frac{S}{N}\right)^{K} \leq \frac{\delta}{RN}$ . So  $\mathbb{P}(r - q_r^i \leq K) \geq 1 - \frac{\delta}{RN}$ . Therefore
$$
\mathbb {P} (\mathcal {E}) = \mathbb {P} \left(r - q _ {r} ^ {i} \leq K, \forall 0 \leq r \leq R - 1, 1 \leq i \leq N\right) \geq 1 - \sum_ {r = 0} ^ {R - 1} \sum_ {i = 1} ^ {N} \mathbb {P} \left(r - q _ {r} ^ {i} > K\right) \leq 1 - \delta .
$$
Lemma 9. Let  $r \geq 0$ . If  $\|\bar{\pmb{x}}_r - \pmb{y}_{r,k}^i\| \leq \frac{C}{L_1}$  for all  $i \in [N]$  and  $k \in \{0, \dots, I-1\}$ , and  $BL_1\rho \Xi_r < 1$ , then
$$
\| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| \leq \frac {\sigma + \| \boldsymbol {G} _ {r} \| + (A L _ {0} + B L _ {1} \kappa) \Xi_ {r}}{1 - B L _ {1} \rho \Xi_ {r}}.
$$
Proof. Due to the assumption  $\| \bar{\pmb{x}}_r - \pmb{y}_{r,k}^i\| \leq \frac{C}{L_1}$ , we have
$$
\begin{array}{l} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| \leq \| \nabla f (\bar {\boldsymbol {x}} _ {r}) - \boldsymbol {G} _ {r} \| + \| \boldsymbol {G} _ {r} \| \\ \leq \frac {1}{N} \sum_ {i = 1} ^ {N} \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \boldsymbol {G} _ {r} ^ {i} \| + \| \boldsymbol {G} _ {r} \| \\ \leq \frac {1}{N I} \sum_ {i = 1} ^ {N} \sum_ {k = 0} ^ {I - 1} \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \nabla F _ {i} \left(\boldsymbol {y} _ {r, k} ^ {i}; \xi_ {q _ {r, k} ^ {i}} ^ {i}\right) \| + \| \boldsymbol {G} _ {r} \| \\ \leq \frac {1}{N I} \sum_ {i = 1} ^ {N} \sum_ {k = 0} ^ {I - 1} \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \nabla f _ {i} (\boldsymbol {y} _ {r, k} ^ {i}) \| + \frac {1}{N I} \sum_ {i = 1} ^ {N} \sum_ {k = 0} ^ {I - 1} \| \nabla f _ {i} (\boldsymbol {y} _ {r, k} ^ {i}) - \nabla F _ {i} (\boldsymbol {y} _ {r, k} ^ {i}; \xi_ {q _ {r, k} ^ {i}} ^ {i}) \| + \| \boldsymbol {G} _ {r} \| \\ \stackrel {(i)} {\leq} \frac {1}{N I} \sum_ {i = 1} ^ {N} \sum_ {k = 0} ^ {I - 1} \left(A L _ {0} + B L _ {1} \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \|\right) \| \bar {\boldsymbol {x}} _ {r} - \boldsymbol {y} _ {r, k} ^ {i} \| + \sigma + \| \boldsymbol {G} _ {r} \| \\ \stackrel {(i i)} {\leq} \left(A L _ {0} + B L _ {1} (\kappa + \rho \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \|\right)) \frac {1}{N I} \sum_ {i = 1} ^ {N} \sum_ {k = 0} ^ {I - 1} \| \bar {\boldsymbol {x}} _ {r} - \boldsymbol {y} _ {r, k} ^ {i} \| + \sigma + \| \boldsymbol {G} _ {r} \| \\ \leq \left(A L _ {0} + B L _ {1} (\kappa + \rho \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \|\right) \Xi_ {r} + \sigma + \| \boldsymbol {G} _ {r} \|, \\ \end{array}
$$
where we used Lemma 2 in (i) and Assumption 1(iv) in (ii). Rearranging to isolate  $\| \nabla f(\bar{\pmb{x}}_r)\|$  gives the result.
Lemma 10. Suppose that Equation 9 holds, then for any  $i\in [N]$  and  $k\in [I]$
$$
\left\| \boldsymbol {x} _ {0, k} ^ {i} - \bar {\boldsymbol {x}} _ {0} \right\| \leq 2 I (2 \sigma \eta + \gamma). \tag {11}
$$
Proof. Notice that, due to the clipping operation,
$$
\mathbb {1} _ {\bar {A} _ {0}} \left\| \boldsymbol {x} _ {0, k} ^ {i} - \bar {\boldsymbol {x}} _ {0} \right\| = k \gamma \leq 2 k (2 \sigma \eta + \gamma). \tag {12}
$$
holds for any  $k \leq I$ . Next we verify Equation 11 under the event  $\mathcal{A}_0$ . To see this, we first consider the case  $k = 1$
$$
\begin{array}{l} \mathbb {1} _ {\mathcal {A} _ {0}} \left\| \boldsymbol {x} _ {0, 1} ^ {i} - \bar {\boldsymbol {x}} _ {0} \right\| = \mathbb {1} _ {\mathcal {A} _ {0}} \cdot \eta \left\| \nabla F _ {i} (\bar {\boldsymbol {x}} _ {0}; \xi_ {0, 0} ^ {i}) - \boldsymbol {G} _ {0} ^ {i} + \boldsymbol {G} _ {0} \right\| \\ \leq \mathbb {1} _ {\mathcal {A} _ {0}} \left(\eta \| \nabla F _ {i} (\bar {\boldsymbol {x}} _ {0}; \xi_ {0, 0} ^ {i}) - \nabla f _ {i} (\bar {\boldsymbol {x}} _ {0}) \| + \eta \| \boldsymbol {G} _ {0} ^ {i} - \nabla f _ {i} (\bar {\boldsymbol {x}} _ {0}) \| + \eta \| \boldsymbol {G} _ {0} \|\right) \\ \leq 2 \sigma \eta + \gamma , \\ \end{array}
$$
where we used the initialization  $G_0^i = \nabla F_i(\bar{\boldsymbol{x}}_0; \tilde{\xi}^i)$  and  $\| G_0 \| \leq \frac{\gamma}{\eta}$  under the event  $\mathcal{A}_0$ . This indicates that Equation 11 holds with  $k = 1$ .
Now suppose that Equation 11 holds for some  $k$  with  $1 \leq k < I$ . Then we have
$$
\mathbb {1} _ {\mathcal {A} _ {0}} \| \pmb {x} _ {0, k + 1} ^ {i} - \bar {\pmb {x}} _ {0} \| = \mathbb {1} _ {\mathcal {A} _ {0}} \| \pmb {x} _ {0, k} ^ {i} - \eta (\nabla F _ {i} (\pmb {x} _ {0, k} ^ {i}; \xi_ {0, k} ^ {i}) - \pmb {G} _ {0} ^ {i} + \pmb {G} _ {0}) - \bar {\pmb {x}} _ {0} \|
$$
$$
\leq \mathbb {1} _ {\mathcal {A} _ {0}} \left(\| \boldsymbol {x} _ {0, k} ^ {i} - \bar {\boldsymbol {x}} _ {0} \| + \eta \| \nabla F _ {i} \left(\boldsymbol {x} _ {0, k} ^ {i}; \xi_ {0, k} ^ {i}\right) - \boldsymbol {G} _ {0} ^ {i} + \boldsymbol {G} _ {0} \|\right)
$$
$$
\begin{array}{l} \stackrel {(i)} {\leq} \mathbb {1} _ {\mathcal {A} _ {0}} \Big (\| \boldsymbol {x} _ {0, k} ^ {i} - \bar {\boldsymbol {x}} _ {0} \| + \eta \| \nabla F _ {i} (\boldsymbol {x} _ {0, k} ^ {i}; \xi_ {0, k} ^ {i}) - \nabla f _ {i} (\boldsymbol {x} _ {0, k} ^ {i}) \| + \eta \| \boldsymbol {G} _ {0} ^ {i} - \nabla f _ {i} (\bar {\boldsymbol {x}} _ {0}) \| \\ \left. + \eta \| \nabla f _ {i} \left(\boldsymbol {x} _ {0, k} ^ {i}\right) - \nabla f _ {i} \left(\bar {\boldsymbol {x}} _ {0}\right) \| + \eta \| \boldsymbol {G} _ {0} \|\right) \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(i i)} {\leq} \mathbb {1} _ {\mathcal {A} _ {0}} \left(\| \boldsymbol {x} _ {0, k} ^ {i} - \bar {\boldsymbol {x}} _ {0} \| + 2 \sigma \eta + \gamma + \eta \| \nabla f _ {i} (\boldsymbol {x} _ {0, k} ^ {i}) - \nabla f _ {i} (\bar {\boldsymbol {x}} _ {0}) \|\right) \\ \stackrel {(i i i)} {\leq} \mathbb {1} _ {\mathcal {A} _ {0}} \left(\| \boldsymbol {x} _ {0, k} ^ {i} - \bar {\boldsymbol {x}} _ {0} \| + 2 \sigma \eta + \gamma + \eta \left(A L _ {0} + B L _ {1} \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {0}) \|\right) \| \boldsymbol {x} _ {0, k} ^ {i} - \bar {\boldsymbol {x}} _ {0} \|\right) \\ \leq \mathbb {1} _ {\mathcal {A} _ {0}} \left(\| \boldsymbol {x} _ {0, k} ^ {i} - \bar {\boldsymbol {x}} _ {0} \| + 2 \sigma \eta + \gamma + \eta (A L _ {0} + B L _ {1} (\kappa + \rho \| \nabla f (\bar {\boldsymbol {x}} _ {0}) \|)) \| \boldsymbol {x} _ {0, k} ^ {i} - \bar {\boldsymbol {x}} _ {0} \|\right). \\ \end{array}
$$
where  $(i)$  holds due to the initialization  $G_0 = \nabla F_i(\bar{\boldsymbol{x}}_0;\tilde{\xi}^i)$ ;  $(ii)$  follows from  $\| G_0\| \leq \frac{\gamma}{\eta}$ ; and  $(iii)$  follows from Lemma 2. Note that the conditions of Lemma 2 are satisfied here by the inductive hypothesis together with Equation 9.
Notice that  $\Xi_0 = 0$ , so by Lemma 9, we have
$$
\mathbb {1} _ {\mathcal {A} _ {0}} \| \nabla f (\bar {\boldsymbol {x}} _ {0}) \| \leq \| \boldsymbol {G} _ {0} - \nabla f (\bar {\boldsymbol {x}} _ {0}) \| + \mathbb {1} _ {\mathcal {A} _ {0}} \| \boldsymbol {G} _ {0} \| \leq \frac {1}{N} \sum_ {i = 1} ^ {N} \left\| \nabla F _ {i} (\bar {\boldsymbol {x}} _ {0}; \tilde {\xi} ^ {i}) - \nabla f _ {i} (\bar {\boldsymbol {x}} _ {0}) \right\| + \frac {\gamma}{\eta} \leq \sigma + \frac {\gamma}{\eta}.
$$
$$
\begin{array}{l} \mathbb {1} _ {\mathcal {A} _ {0}} \left\| \boldsymbol {x} _ {0, k + 1} ^ {i} - \bar {\boldsymbol {x}} _ {0} \right\| \leq \left\| \boldsymbol {x} _ {0, k} ^ {i} - \bar {\boldsymbol {x}} _ {0} \right\| + 2 \sigma \eta + \gamma + \eta \left(A L _ {0} + B L _ {1} \kappa + B L _ {1} \rho \left(\sigma + \frac {\gamma}{\eta}\right)\right) \left\| \boldsymbol {x} _ {0, k} ^ {i} - \bar {\boldsymbol {x}} _ {0} \right\| \\ \stackrel {(i)} {\leq} \left\| \boldsymbol {x} _ {0, k} ^ {i} - \bar {\boldsymbol {x}} _ {0} \right\| + 2 \sigma \eta + \gamma + \frac {1}{4 I} \left\| \boldsymbol {x} _ {0, k} ^ {i} - \bar {\boldsymbol {x}} _ {0} \right\| \\ \stackrel {(i i)} {\leq} 2 \sigma \eta + \gamma + \left(1 + \frac {1}{4 I}\right) 2 k (2 \sigma \eta + \gamma) \\ = 2 \left(k + \frac {k}{4 I} + \frac {1}{2}\right) (2 \sigma \eta + \gamma) \\ \leq 2 (k + 1) (2 \sigma \eta + \gamma), \\ \end{array}
$$
where  $(i)$  holds due to Equation 9 and  $(ii)$  holds because of the inductive hypothesis that Equation 11 holds for  $k$ . This completes the induction over  $k$  under the event  $\mathcal{A}_0$ . This fact together with Equation 12 proves that Equation 11 holds for all  $k = 1, \ldots, I$ .
Lemma 11. Suppose that Equation 9 holds. If  $\max_{k\in [I]}\| \pmb{x}_{q,k}^i -\bar{\pmb{x}}_q\| \leq 2I(2\sigma \eta +\gamma)$  holds for any  $q\leq r - 1$ , then we have
$$
\mathbb {1} _ {\mathcal {E}} \max  _ {k \in [ I ]} \| \boldsymbol {x} _ {r, k} ^ {i} - \bar {\boldsymbol {x}} _ {r} \| \leq 2 I (2 \sigma \eta + \gamma), \quad f o r a n y i \in \mathcal {S} _ {r}, \tag {13}
$$
$$
\mathbb {1} _ {\varepsilon} \max  _ {k \in [ I ]} \| \boldsymbol {y} _ {r, k} ^ {i} - \bar {\boldsymbol {x}} _ {r} \| \leq 2 (K + 1) I (2 \sigma \eta + \gamma), \quad f o r a n y i \in [ N ],. \tag {14}
$$
Proof. Under  $\bar{\mathcal{A}}_r$ , Equation 13 trivially holds due to clipping. Next we verify Equation 13 under  $\mathcal{A}_r$ . First consider the case  $k = 1$ .
$$
\begin{array}{l} \mathbb {1} _ {\mathcal {A} _ {r}} \left\| \boldsymbol {x} _ {r, 1} ^ {i} - \bar {\boldsymbol {x}} _ {r} \right\| \\ = \mathbb {1} _ {\mathcal {A} _ {r}} \cdot \eta \| \nabla F _ {i} (\bar {\boldsymbol {x}} _ {r}; \xi_ {r, 0} ^ {i}) - \boldsymbol {G} _ {r} ^ {i} + \boldsymbol {G} _ {r} \| \\ \leq \eta \left\| \nabla F _ {i} \left(\bar {\boldsymbol {x}} _ {r}; \xi_ {r, 0} ^ {i}\right) - \frac {1}{I} \sum_ {k = 0} ^ {I - 1} \nabla F _ {i} \left(\boldsymbol {y} _ {r, k} ^ {i}; \xi_ {q _ {r} ^ {i}, k} ^ {i}\right) \right\| + \eta \mathbb {1} _ {\mathcal {A} _ {r}} \| \boldsymbol {G} _ {r} \| \\ \leq \eta \| \nabla F _ {i} (\bar {\boldsymbol {x}} _ {r}; \xi_ {r, 0} ^ {i}) - \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \| + \frac {\eta}{I} \sum_ {k = 0} ^ {I - 1} \| \nabla F _ {i} (\boldsymbol {y} _ {r, k} ^ {i}) - \nabla F _ {i} (\boldsymbol {y} _ {r, k} ^ {i}; \xi_ {q _ {r} ^ {i}, k} ^ {i}) \| + \\ \frac {\eta}{I} \sum_ {k = 0} ^ {I - 1} \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \nabla f _ {i} (\boldsymbol {y} _ {r, k} ^ {i}) \| + \eta \mathbb {1} _ {\mathcal {A} _ {r}} \| \boldsymbol {G} _ {r} \| \\ \leq 2 \sigma \eta + \gamma + \frac {\eta}{I} \sum_ {k = 0} ^ {I - 1} \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \nabla f _ {i} (\boldsymbol {y} _ {r, k} ^ {i}) \| \tag {15} \\ \end{array}
$$
Recall that  $\pmb{y}_{r,k}^{i} = \pmb{x}_{q_{r}^{i},k}^{i}$  from the definitions Equation 7 and Equation 8, so
$$
\begin{array}{l} \left\| \bar {\boldsymbol {x}} _ {r} - \boldsymbol {y} _ {r, k} ^ {i} \right\| \leq \left\| \bar {\boldsymbol {x}} _ {r} - \bar {\boldsymbol {x}} _ {q _ {r} ^ {i}} \right\| + \left\| \bar {\boldsymbol {x}} _ {q _ {r} ^ {i}} - \boldsymbol {x} _ {q _ {r} ^ {i}, k} \right\| \leq \sum_ {q = q _ {r} ^ {i}} ^ {r - 1} \left\| \bar {\boldsymbol {x}} _ {q + 1} - \bar {\boldsymbol {x}} _ {q} \right\| + \left\| \bar {\boldsymbol {x}} _ {q _ {r} ^ {i}} - \boldsymbol {x} _ {q _ {r} ^ {i}, k} ^ {i} \right\| \\ \stackrel {(i)} {\leq} \left(r - q _ {r} ^ {i}\right) 2 I (2 \sigma \eta + \gamma) + 2 k (2 \sigma \eta + \gamma) \\ \stackrel {(i i)} {\leq} 2 (K + 1) I (2 \sigma \eta + \gamma) \\ \stackrel {(i i i)} {\leq} \frac {C}{L _ {1}}, \tag {16} \\ \end{array}
$$
where  $(i)$  holds due to inductive hypothesis for rounds  $q\in \{0,\ldots ,r - 1\}$ ;  $(ii)$  holds due to the event  $\mathcal{E}$  such that  $r - q_r^i\leq K$ ; and  $(iii)$  follows from Equation 9. The above bound on  $\| \bar{\pmb{x}}_r - \pmb{y}_{r,k}^i\|$  verifies Equation 14 and shows that the condition of Lemma 2 is satisfied. Then applying it to Equation 15 yields
$$
\begin{array}{l} \| \boldsymbol {x} _ {r, 1} ^ {i} - \bar {\boldsymbol {x}} _ {r} \| _ {\mathbb {1} _ {\mathcal {A} _ {r}}} \leq 2 \sigma \eta + \gamma + \eta \mathbb {1} _ {\mathcal {A} _ {r}} \left(A L _ {0} + B L _ {1} \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \|\right) \frac {1}{I} \sum_ {k = 0} ^ {I - 1} \| \bar {\boldsymbol {x}} _ {r} - \boldsymbol {y} _ {r, k} ^ {i} \| \\ \leq 2 \sigma \eta + \gamma + 2 (K + 1) \eta \mathbb {1} _ {\mathcal {A} _ {r}} I \left(A L _ {0} + B L _ {1} \kappa + B L _ {1} \rho \| \nabla f (\bar {x} _ {r}) \|\right) (2 \sigma \eta + \gamma) \\ \leq (1 + 2 (K + 1) \eta I \left(A L _ {0} + B L _ {1} \kappa + B L _ {1} \rho \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {x} _ {r}) \|\right)) (2 \sigma \eta + \gamma), \tag {17} \\ \end{array}
$$
where we used Equation 16 and Assumption 1 (iii) and (iv). Invoking Equation 9 and Equation 16 gives
$$
B L _ {1} \rho \Xi_ {r} = B L _ {1} \rho \frac {1}{N I} \sum_ {i = 1} ^ {N} \sum_ {k = 0} ^ {I - 1} \| \bar {\boldsymbol {x}} _ {r} - \boldsymbol {y} _ {r, k} ^ {i} \| \leq 2 (K + 1) B L _ {1} \rho I (2 \sigma \eta + \gamma) \leq \frac {1}{2}.
$$
Therefore the conditions of Lemma 9 are satisfied, and we can bound  $\| \nabla f(\bar{\boldsymbol{x}}_r) \|$  as follows:
$$
\begin{array}{l} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| _ {\mathcal {A} _ {r}} \leq \frac {\sigma + \frac {\gamma}{\eta} + (A L _ {0} + B L _ {1} \kappa) \Xi_ {r}}{1 - B L _ {1} \rho \Xi_ {r}} \\ \leq 2 \left(\sigma + \frac {\gamma}{\eta} + \left(A L _ {0} + B L _ {1} \kappa\right) \Xi_ {r}\right) \\ \leq 2 \left(\sigma + \frac {\gamma}{\eta} + 2 (K + 1) I \left(A L _ {0} + B L _ {1} \kappa\right) (2 \sigma \eta + \gamma)\right) \\ \leq 2 \left(2 \sigma + \frac {\gamma}{\eta}\right) (1 + 2 (K + 1) \eta I \left(A L _ {0} + B L _ {1} \kappa\right)) \\ \leq 4 \left(2 \sigma + \frac {\gamma}{\eta}\right), \tag {18} \\ \end{array}
$$
where we used Equation 9. Finally, plugging back into Equation 17 yields
$$
\begin{array}{l} \left\| \boldsymbol {x} _ {r, 1} ^ {i} - \bar {\boldsymbol {x}} _ {r} \right\| \mathbb {1} _ {\mathcal {A} _ {r}} \leq \left(1 + 2 (K + 1) \eta I \left(A L _ {0} + B L _ {1} \kappa + 4 B L _ {1} \rho \left(2 \sigma + \frac {\gamma}{\eta}\right)\right)\right) (2 \sigma \eta + \gamma) \\ \leq 2 (2 \sigma \eta + \gamma), \\ \end{array}
$$
where we used Equation 9 again. This proves Equation 13 for the base case  $k = 1$ .
Now suppose that Equation 13 holds for some  $k$  with  $1 \leq k < I$ . Then
$$
\begin{array}{l} \| \pmb {x} _ {r, k + 1} ^ {i} - \bar {\pmb {x}} _ {r} \| _ {\mathcal {A} _ {r}} = \mathbb {1} _ {\mathcal {A} _ {r}} \| \pmb {x} _ {r, k} ^ {i} - \eta (\nabla F _ {i} (\pmb {x} _ {r, k} ^ {i}; \xi_ {r, k} ^ {i}) - \pmb {G} _ {r} ^ {i} + \pmb {G} _ {r}) - \bar {\pmb {x}} _ {r} \| \\ \leq \mathbf {1} _ {\mathcal {A} _ {r}} (\| \boldsymbol {x} _ {r, k} ^ {i} - \bar {\boldsymbol {x}} _ {r} \| + \eta \| \nabla F _ {i} (\boldsymbol {x} _ {r, k} ^ {i}; \xi_ {r, k} ^ {i}) - \boldsymbol {G} _ {r} ^ {i} + \boldsymbol {G} _ {r} \|) \\ \leq \mathbb {1} _ {\mathcal {A} _ {r}} \left(\| \boldsymbol {x} _ {r, k} ^ {i} - \bar {\boldsymbol {x}} _ {r} \| + \eta \| \nabla F _ {i} \left(\boldsymbol {x} _ {r, k} ^ {i}; \xi_ {r, k} ^ {i}\right) - \nabla f _ {i} \left(\boldsymbol {x} _ {r, k} ^ {i}\right)\right) \\ + \mathbb {1} _ {\mathcal {A} _ {r}} \frac {\eta}{I} \sum_ {j = 0} ^ {I - 1} \| \nabla F _ {j} \left(\boldsymbol {y} _ {r, j} ^ {i}; \xi_ {q _ {r} ^ {i}; j} ^ {i}\right) - \nabla f _ {i} \left(\boldsymbol {y} _ {r, j} ^ {i}\right) \| \\ + \mathbb {1} _ {\mathcal {A} _ {r}} \frac {\eta}{I} \sum_ {j = 0} ^ {I - 1} \| \nabla f _ {i} (\boldsymbol {x} _ {r, k} ^ {i}) - \nabla f _ {i} (\boldsymbol {y} _ {r, j} ^ {i}) \| + \eta \mathbb {1} _ {\mathcal {A} _ {r}} \| \boldsymbol {G} _ {r} \| \\ \leq \mathbb {1} _ {\mathcal {A} _ {r}} \| \boldsymbol {x} _ {r, k} ^ {i} - \bar {\boldsymbol {x}} _ {r} \| + 2 \sigma \eta + \gamma + \eta \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f _ {i} (\boldsymbol {x} _ {r, k} ^ {i}) - \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \| \\ + \mathbb {1} _ {\mathcal {A} _ {r}} \frac {\eta}{I} \sum_ {j = 0} ^ {I - 1} \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \nabla f _ {i} \left(\boldsymbol {y} _ {r, j} ^ {i}\right) \|, \tag {19} \\ \end{array}
$$
where we used  $\| G_r\| \leq \frac{\gamma}{\eta}$  under the event  $\mathcal{A}_r$  and Assumption 1 (iii). We can individually bound each of the two terms  $\mathbb{1}_{\mathcal{A}_r}\| \nabla f_i(\pmb{x}_{r,k}^i) - \nabla f_i(\bar{\pmb{x}}_r)\|$  and  $\mathbb{1}_{\mathcal{A}_r}\| \nabla f_i(\bar{\pmb{x}}_r) - \nabla f_i(\pmb{y}_{r,j}^i)\|$ . By the inductive hypothesis over  $k$ , together with Equation 9:  $\| \pmb{x}_{r,k}^i -\bar{\pmb{x}}_r\| \leq 2I(2\sigma \eta +\gamma)\leq \frac{C}{L_1}$ . So we can apply Lemma 2 and obtain:
$$
\begin{array}{l} \| \nabla f _ {i} \left(\boldsymbol {x} _ {r, k} ^ {i}\right) - \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \| \mathbb {1} _ {\mathcal {A} _ {r}} \leq \mathbb {1} _ {\mathcal {A} _ {r}} \left(A L _ {0} + B L _ {1} \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \|\right) \| \boldsymbol {x} _ {r, k} ^ {i} - \bar {\boldsymbol {x}} _ {r} \| \\ \leq \mathbb {1} _ {\mathcal {A} _ {r}} \left(A L _ {0} + B L _ {1} \kappa + B L _ {1} \rho \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \|\right) \| \boldsymbol {x} _ {r, k} ^ {i} - \bar {\boldsymbol {x}} _ {r} \| \\ \leq \mathbb {1} _ {\mathcal {A} _ {r}} \left(A L _ {0} + B L _ {1} \kappa + 4 B L _ {1} \rho \left(2 \sigma + \frac {\gamma}{\eta}\right)\right) \| \boldsymbol {x} _ {r, k} ^ {i} - \bar {\boldsymbol {x}} _ {r} \| \\ \leq \frac {1}{4 \eta I} \left\| \boldsymbol {x} _ {r, k} ^ {i} - \bar {\boldsymbol {x}} _ {r} \right\|, \tag {20} \\ \end{array}
$$
where we used Assumption 1 (iii) and (iv), Equation 18, and Equation 9. Similarly, by the inductive hypothesis over  $r$ ,
$$
\left\| \bar {\boldsymbol {x}} _ {r} - \boldsymbol {y} _ {r, j} ^ {i} \right\| \leq \left\| \bar {\boldsymbol {x}} _ {r} - \bar {\boldsymbol {x}} _ {q _ {r} ^ {i}} \right\| + \left\| \bar {\boldsymbol {x}} _ {q _ {r} ^ {i}} - \boldsymbol {x} _ {q _ {r} ^ {i}, j} ^ {i} \right\| \leq 2 (K + 1) I (2 \sigma \eta + \gamma) \leq \frac {C}{L _ {1}}.
$$
So we can apply Lemma 2:
$$
\begin{array}{l} \left\| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \nabla f _ {i} \left(\boldsymbol {y} _ {r, j} ^ {i}\right) \right\| \leq \left(A L _ {0} + B L _ {1} \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \|\right) \| \bar {\boldsymbol {x}} _ {r} - \boldsymbol {y} _ {r, j} ^ {i} \| \\ \leq \left(A L _ {0} + B L _ {1} \kappa + 4 B L _ {1} \rho \left(2 \sigma + \frac {\gamma}{\eta}\right)\right) \| \bar {\boldsymbol {x}} _ {r} - \boldsymbol {y} _ {r, j} ^ {i} \| \\ \leq \frac {1}{4 (K + 1) \eta I} \left\| \bar {\boldsymbol {x}} _ {r} - \boldsymbol {y} _ {r, j} ^ {i} \right\| \tag {21} \\ \end{array}
$$
where the last inequality follows from the assumption Equation 9.
Finally, plugging Equation 20 and Equation 21 into Equation 19 gives