SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where the last line uses Markov's inequality. Repeating this argument for  $-\mathbb{1}_{\mathcal{A}_r}\langle \nabla f(\bar{\boldsymbol{x}}_r),\boldsymbol{\epsilon}_r^{(1)}\rangle$ , we can obtain
$$
\mathbb {P} \left(\left| \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \boldsymbol {\epsilon} _ {r} ^ {(1)} \rangle \right| \geq \lambda_ {2} \sum_ {r = 0} ^ {R - 1} M _ {r} + \frac {1}{\lambda_ {2}} \log \frac {1}{\delta}\right) \geq 1 - 2 \delta ,
$$
Here  $\mathbb{E}_r[\cdot ]$  takes expectation over the randomness of subsampling, that is  $\mathcal{S}_r$ .
$$
B _ {2} \leq \eta I \lambda_ {2} \left(\kappa + 5 \rho \left(2 \sigma + \frac {\gamma}{\eta}\right)\right) ^ {2} \left(\frac {1}{S} - \frac {1}{N}\right) \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| ^ {2} + \frac {\eta I}{\lambda_ {2}} \log \frac {1}{\delta}, \tag {35}
$$
holds with probability  $1 - 2\delta$
The same argument can be applied to  $B_{3}$ . Notice that
$$
\mathbb {1} _ {\mathcal {A} _ {r}} \langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \boldsymbol {\epsilon} _ {r} ^ {(2)} \rangle = \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \mathbb {1} _ {\mathcal {A} _ {r}} \langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \rangle - \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {1} _ {\mathcal {A} _ {r}} \langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \rangle ,
$$
so we must upper bound  $\mathbb{1}_{\mathcal{A}_r}|\langle \nabla f(\bar{\boldsymbol{x}}_r),\nabla f_i(\bar{\boldsymbol{x}}_r)\rangle |$  to apply Lemma 6. Using Equation 25 in Corollary 2, we have
$$
\mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \| \leq \kappa + \rho \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| \leq \kappa + 4 \rho \left(2 \sigma + \frac {\gamma}{\eta}\right) \leq \kappa + 5 \rho \left(2 \sigma + \frac {\gamma}{\eta}\right).
$$
This matches the corresponding upper bound in Equation 33 from the bound of  $B_{2}$ . We may therefore apply an identical argument as in the case of  $B_{2}$  and obtain
$$
B _ {3} \leq \eta I \lambda_ {2} \left(\kappa + 5 \rho \left(2 \sigma + \frac {\gamma}{\eta}\right)\right) ^ {2} \left(\frac {1}{S} - \frac {1}{N}\right) \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| ^ {2} + \frac {\eta I}{\lambda_ {2}} \log \frac {2}{\delta}, \tag {36}
$$
with probability  $1 - 2\delta$
Combining Equation 31, Equation 35, and Equation 36 into Equation 30, yields that, with probability  $1 - 5\delta$ ,
$$
\begin{array}{l} A _ {1} \leq - \eta I \left(\frac {7}{8} - \frac {3}{4} \sigma^ {2} \lambda_ {1} - 2 \lambda_ {2} \left(\kappa + 5 \rho \left(2 \sigma + \frac {\gamma}{\eta}\right)\right) ^ {2} \left(\frac {1}{S} - \frac {1}{N}\right)\right) \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| ^ {2} + \\ 2 \eta^ {2} I ^ {2} (A L _ {0} + B L _ {1} \kappa) \left(2 \sigma + \frac {\gamma}{\eta}\right) \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| + \eta \left(\frac {1}{\lambda_ {1}} + \frac {2 I}{\lambda_ {2}}\right) \log \frac {1}{\delta} \\ \leq - \frac {3}{4} \eta I \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| ^ {2} + 2 \eta^ {2} I ^ {2} (A L _ {0} + B L _ {1} \kappa) \left(2 \sigma + \frac {\gamma}{\eta}\right) \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| + \\ \eta \left(1 2 \sigma^ {2} + 6 4 I \left(\kappa + 5 \rho \left(2 \sigma + \frac {\gamma}{\eta}\right)\right) ^ {2} \left(\frac {1}{S} - \frac {1}{N}\right)\right) \log \frac {1}{\delta}, \tag {37} \\ \end{array}
$$
where we chose
$$
\lambda_ {1} = \frac {1}{1 2 \sigma^ {2}},
$$
$$
\lambda_ {2} = \frac {1}{3 2 \left(\kappa + 5 \rho \left(2 \sigma + \frac {\gamma}{\eta}\right)\right) ^ {2} \left(\frac {1}{S} - \frac {1}{N}\right)}.
$$
Bounding  $\mathbf{A}_2$ . From Lemma 4,
$$
- \left\langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \frac {\boldsymbol {g} _ {r , k} ^ {i}}{\| \boldsymbol {g} _ {r , k} ^ {i} \|} \right\rangle \leq - \mu \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| - (1 - \mu) \| \boldsymbol {g} _ {r, k} ^ {i} \| + (1 + \mu) \| \boldsymbol {g} _ {r, k} ^ {i} - \nabla f (\bar {\boldsymbol {x}} _ {r}) \|
$$
for any  $\mu \geq 0$ . Also,
$$
\begin{array}{l} \mathbb {1} _ {\tilde {\mathcal {A}} _ {r}} \left\| \boldsymbol {g} _ {r, k} ^ {i} \right\| = \mathbb {1} _ {\tilde {\mathcal {A}} _ {r}} \left\| \nabla F _ {i} \left(\boldsymbol {x} _ {r, k} ^ {i}; \xi_ {r, k} ^ {i}\right) - \boldsymbol {G} _ {r} ^ {i} + \boldsymbol {G} _ {r} \right\| \\ \geq \mathbb {1} _ {\bar {A} _ {r}} \left(\| \boldsymbol {G} _ {r} \| - \left\| \nabla F _ {i} \left(\boldsymbol {x} _ {r, k} ^ {i}; \xi_ {r, k} ^ {i}\right) - \boldsymbol {G} _ {r} ^ {i} \right\|\right) \\ \geq \mathbb {1} _ {\bar {\mathcal {A}} _ {r}} \left(\frac {\gamma}{\eta} - \left\| \nabla F _ {i} (\boldsymbol {x} _ {r, k} ^ {i}; \boldsymbol {\xi} _ {r, k} ^ {i}) - \nabla f _ {i} (\boldsymbol {x} _ {r, k} ^ {i}) \right\| - \left\| \nabla f _ {i} (\boldsymbol {x} _ {r, k} ^ {i}) - \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \right\| - \left\| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \boldsymbol {G} _ {r} ^ {i} \right\|\right) \\ \geq \mathbb {1} _ {\bar {A} _ {r}} \left(\frac {\gamma}{\eta} - \sigma - \left(A L _ {0} + B L _ {1} \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \|\right) \left\| \boldsymbol {x} _ {r, k} ^ {i} - \bar {\boldsymbol {x}} _ {r} \right\| - \left\| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \boldsymbol {G} _ {r} ^ {i} \right\|\right) \\ \geq \mathbb {1} _ {\bar {A} _ {r}} \left(\frac {\gamma}{\eta} - \sigma - \gamma I \big (A L _ {0} + B L _ {1} \kappa + B L _ {1} \rho \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| \big) - \left\| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \boldsymbol {G} _ {r} ^ {i} \right\|\right), \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {1} _ {\bar {A} _ {r}} \left\| \boldsymbol {g} _ {r, k} ^ {i} - \nabla f (\bar {\boldsymbol {x}} _ {r}) \right\| = \mathbb {1} _ {\bar {A} _ {r}} \left\| \nabla F _ {i} \left(\boldsymbol {x} _ {r, k} ^ {i}; \xi_ {r, k} ^ {i}\right) - \boldsymbol {G} _ {r} ^ {i} + \boldsymbol {G} _ {r} - \nabla f (\bar {\boldsymbol {x}} _ {r}) \right\| \\ \leq \mathbb {1} _ {\bar {A} _ {r}} \left(\left\| \nabla F _ {i} \left(\boldsymbol {x} _ {r, k} ^ {i}; \xi_ {r, k} ^ {i}\right) - \boldsymbol {G} _ {r} ^ {i} \right\| + \| \boldsymbol {G} _ {r} - \nabla f (\bar {\boldsymbol {x}} _ {r}) \|\right) \\ \leq \mathbb {1} _ {\bar {\mathcal {A}} _ {r}} \left(\left\| \nabla F _ {i} \left(\boldsymbol {x} _ {r, k} ^ {i}; \xi_ {r, k} ^ {i}\right) - \nabla f _ {i} \left(\boldsymbol {x} _ {r, k} ^ {i}\right) \right\| + \left\| \nabla f _ {i} \left(\boldsymbol {x} _ {r, k} ^ {i}\right) - \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \right\|\right) \\ \left. + \left\| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \boldsymbol {G} _ {r} ^ {i} \right\| + \frac {1}{N} \sum_ {j = 1} ^ {N} \left\| \boldsymbol {G} _ {r} ^ {j} - \nabla F _ {j} (\bar {\boldsymbol {x}} _ {r}) \right\|\right) \\ \leq \mathbb {1} _ {\bar {\mathcal {A}} _ {r}} \left(\sigma + \left(A L _ {0} + B L _ {1} \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \|\right) \| \boldsymbol {x} _ {r, k} ^ {i} - \bar {\boldsymbol {x}} _ {r} \| \right. \\ \left. + \left\| \nabla f _ {i} \left(\bar {\boldsymbol {x}} _ {r}\right) - \boldsymbol {G} _ {r} ^ {i} \right\| + \frac {1}{N} \sum_ {j = 1} ^ {N} \left\| \boldsymbol {G} _ {r} ^ {j} - \nabla F _ {j} \left(\bar {\boldsymbol {x}} _ {r}\right) \right\|\right) \\ \leq \mathbb {1} _ {\bar {A} _ {r}} \left(\sigma + \gamma I \left(A L _ {0} + B L _ {1} \kappa + B L _ {1} \rho \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \|\right) \right. \\ \left. + \left\| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \boldsymbol {G} _ {r} ^ {i} \right\| + \frac {1}{N} \sum_ {j = 1} ^ {N} \left\| \boldsymbol {G} _ {r} ^ {j} - \nabla F _ {j} (\bar {\boldsymbol {x}} _ {r}) \right\|\right). \\ \end{array}
$$
$$
\begin{array}{l} - \mathbb {1} _ {\bar {\mathcal {A}} _ {r}} \left\langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \frac {\boldsymbol {g} _ {r , k} ^ {i}}{\| \boldsymbol {g} _ {r , k} ^ {i} \|} \right\rangle \\ \leq \mathbb {1} _ {\bar {\mathcal {A}} _ {r}} \left(- \mu \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| - (1 - \mu) \frac {\gamma}{\eta} + 2 \sigma + 2 \gamma I \left(A L _ {0} + B L _ {1} \kappa + B L _ {1} \rho \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \|\right) \right. \\ \left. + 2 \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \boldsymbol {G} _ {r} ^ {i} \| + \frac {1 + \mu}{N} \sum_ {j = 1} ^ {N} \| \boldsymbol {G} _ {r} ^ {j} - \nabla F _ {j} (\bar {\boldsymbol {x}} _ {r}) \|\right), \\ \end{array}
$$
$$
\begin{array}{l} - \mathbb {1} _ {\bar {\mathcal {A}} _ {r}} \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \left\langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \frac {\boldsymbol {g} _ {r , k} ^ {i}}{\| \boldsymbol {g} _ {r , k} ^ {i} \|} \right\rangle \\ \leq \mathbb {1} _ {\bar {\mathcal {A}} _ {r}} \left(- \mu \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| - (1 - \mu) \frac {\gamma}{\eta} + 2 \sigma + 2 \gamma I \left(A L _ {0} + B L _ {1} \kappa + B L _ {1} \rho \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \|\right) + (3 + \mu) \left(2 \sigma + \frac {\gamma}{3 0 \eta}\right)\right) \\ \leq \mathbb {1} _ {\bar {\mathcal {A}} _ {r}} \left(- (\mu - 2 B L _ {1} \rho \gamma I) \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| + \left(- \frac {9}{1 0} + \frac {3 1}{3 0} \mu\right) \frac {\gamma}{\eta} + (5 + \mu) \sigma + 2 \gamma I (A L _ {0} + B L _ {1} \kappa)\right), \\ \end{array}
$$
where we used Equation 26. Finally
$$
\begin{array}{l} A _ {2} = - \gamma \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\bar {A} _ {r}} \left\langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \frac {1}{S} \sum_ {i \in S _ {r}} \sum_ {k = 0} ^ {I - 1} \frac {\boldsymbol {g} _ {r , k} ^ {i}}{\| \boldsymbol {g} _ {r , k} ^ {i} \|} \right\rangle \\ \leq \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\bar {\mathcal {A}} _ {r}} \left[ - \gamma I (\mu - 2 B L _ {1} \rho \gamma I) \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| + \left(- \frac {9}{1 0} + \frac {3 1}{3 0} \mu\right) \frac {\gamma^ {2} I}{\eta} + (5 + \mu) \sigma \gamma I + 2 \gamma^ {2} I ^ {2} (A L _ {0} + B L _ {1} \kappa) \right] \\ \leq \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\bar {\mathcal {A}} _ {r}} \left[ - \gamma I \left(\frac {3}{4} - 2 B L _ {1} \rho \gamma I\right) \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| - \gamma I \left(\frac {1}{8} \frac {\gamma}{\eta} - 6 \sigma - 2 \gamma I (A L _ {0} + B L _ {1} \kappa)\right) \right], \\ \end{array}
$$
where we used the choice  $\mu = \frac{3}{4}$ .
Bounding  $\mathbf{A_3}$ . To bound  $A_{3}$ , we begin by bounding  $\mathbb{1}_{\mathcal{A}_r}\| \bar{\boldsymbol{x}}_{r + 1} - \bar{\boldsymbol{x}}_r\|^2$
where we denoted  $\tilde{\epsilon}_{r,k}^{i} = \epsilon_{q_{r,k}^{i},k}^{i} = \nabla F_{i}(\pmb{y}_{r,k}^{i};\xi_{q_{r,k}^{i}}^{i}) - \nabla f_{i}(\pmb{y}_{r,k}^{i})$ . The first three terms on the RHS of Equation 38 can be bounded with similar arguments as those used previously:
$$
\begin{array}{l} \mathbb {1} _ {\mathcal {A} _ {r}} \left\| \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \sum_ {k = 0} ^ {I - 1} \nabla f _ {i} \left(\boldsymbol {x} _ {r, k} ^ {i}\right) - \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \right\| ^ {2} \\ \leq I \mathbb {1} _ {\mathcal {A} _ {r}} \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \sum_ {k = 0} ^ {I - 1} \left| \left| \nabla f _ {i} \left(\boldsymbol {x} _ {r, k} ^ {i}\right) - \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \right| \right| ^ {2} \\ \leq I \mathbb {1} _ {\mathcal {A} _ {r}} \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \left(A L _ {0} + B L _ {1} \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \|\right) ^ {2} \sum_ {k = 0} ^ {I - 1} \left\| \boldsymbol {x} _ {r, k} ^ {i} - \bar {\boldsymbol {x}} _ {r} \right\| ^ {2} \\ \stackrel {(i)} {\leq} 4 \mathbb {1} _ {\mathcal {A} _ {r}} \eta^ {2} I ^ {4} (A L _ {0} + B L _ {1} \kappa + B L _ {1} \rho \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \|) ^ {2} \left(2 \sigma + \frac {\gamma}{\eta}\right) ^ {2} \\ \leq 8 \eta^ {2} I ^ {4} B ^ {2} L _ {1} ^ {2} \rho^ {2} \left(2 \sigma + \frac {\gamma}{\eta}\right) ^ {2} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| ^ {2} + 8 \mathbb {1} _ {\mathcal {A} _ {r}} \eta^ {2} I ^ {4} (A L _ {0} + B L _ {1} \kappa) ^ {2} \left(2 \sigma + \frac {\gamma}{\eta}\right) ^ {2} \\ \stackrel {(i i)} {\leq} \frac {1}{3 2} I ^ {2} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| ^ {2} + \mathbb {1} _ {\mathcal {A} _ {r}} 8 \Gamma_ {1} ^ {2} \eta^ {2} I ^ {4} \left(2 \sigma + \frac {\gamma}{\eta}\right) ^ {2}, \\ \end{array}
$$
where  $(i)$  holds due to Equation 13 and Assumption 1(iv); and  $(ii)$  follows from the condition on  $\eta$  in Equation 9. Due to Lemma 1, we can use the same argument to get
$$
\mathbb {1} _ {\mathcal {A} _ {r}} \left\| \frac {1}{S I} \sum_ {i \in \mathcal {S} _ {r}} \sum_ {k = 0} ^ {I - 1} \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \nabla f _ {i} (\boldsymbol {y} _ {r, k} ^ {i}) \right\| ^ {2} \leq \frac {1}{3 2} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| ^ {2} + \mathbb {1} _ {\mathcal {A} _ {r}} 8 \Gamma_ {1} ^ {2} \eta^ {2} I ^ {2} \left(2 \sigma + \frac {\gamma}{\eta}\right) ^ {2},
$$
$$
\mathbb {1} _ {\mathcal {A} _ {r}} \left\| \frac {1}{N I} \sum_ {i = 1} ^ {N} \sum_ {k = 0} ^ {I - 1} \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \nabla f _ {i} (\boldsymbol {y} _ {r, k} ^ {i}) \right\| ^ {2} \leq \frac {1}{3 2} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| ^ {2} + \mathbb {1} _ {\mathcal {A} _ {r}} 8 \Gamma_ {1} ^ {2} \eta^ {2} I ^ {2} \left(2 \sigma + \frac {\gamma}{\eta}\right) ^ {2}.
$$
Plugging back into Equation 38,
$$
\begin{array}{l} \mathbb {1} _ {\mathcal {A} _ {r}} \left\| \bar {\boldsymbol {x}} _ {r + 1} - \bar {\boldsymbol {x}} _ {r} \right\| ^ {2} \leq 5 \eta^ {2} I ^ {2} \mathbb {1} _ {\mathcal {A} _ {r}} \left\| \nabla f (\bar {\boldsymbol {x}} _ {r}) \right\| ^ {2} + \mathbb {1} _ {\mathcal {A} _ {r}} 1 9 2 \Gamma_ {1} ^ {2} \eta^ {4} I ^ {4} \left(2 \sigma + \frac {\gamma}{\eta}\right) ^ {2} \\ + 8 \eta^ {2} \mathbb {1} _ {\mathcal {A} _ {r}} \left\| \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \sum_ {k = 0} ^ {I - 1} \boldsymbol {\epsilon} _ {r, k} ^ {i} \right\| ^ {2} + 8 \eta^ {2} \mathbb {1} _ {\mathcal {A} _ {r}} \left\| \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \sum_ {k = 0} ^ {I - 1} \tilde {\boldsymbol {\epsilon}} _ {r, k} ^ {i} \right\| ^ {2} \\ + 8 \eta^ {2} \mathbb {1} _ {\mathcal {A} _ {r}} \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {k = 0} ^ {I - 1} \tilde {\epsilon} _ {r, k} ^ {i} \right\| ^ {2}. \tag {39} \\ \end{array}
$$
We can bound the remaining terms using the concentration inequality from Lemma 7. We write the index of  $S_{r}$  as  $\{i_1,\dots,i_S\}$  For  $k = 0,\dots,I - 1$  and for  $j\in [S]$ , let  $X_{kS + j} = \varepsilon_{r,k}^{i_j}$  and  $\mathcal{H}_{kS + j} = \sigma \left(\mathcal{F}_r,\mathcal{S}_r,\{\xi_{r,\ell}^i\}_{0\leq \ell \leq k - 1,i\in \mathcal{S}_r},\{\xi_{r,k}^{i_m}\}_{m\leq j}\right)$  for  $k\geq 1$ , where  $\{\xi_{r,\ell}^i\}_{0\leq \ell \leq -1,i\in \mathcal{S}_r} = \emptyset$ . For any  $t = kS + j$ , it holds that
$$
\mathbb {E} \left[ X _ {t} \mid \mathcal {H} _ {t - 1} \right] = \mathbb {E} \left[ \boldsymbol {\varepsilon} _ {r, k} ^ {i _ {j}} \mid \mathcal {H} _ {k S + j - 1} \right] = \mathbb {E} \left[ \boldsymbol {\varepsilon} _ {r, k} ^ {i _ {j}} \mid \mathcal {F} _ {r}, \{\xi_ {r, \ell} ^ {i _ {j}} \} _ {\ell \leq k - 1} \right] = 0,
$$
where the second inequality holds due to the independence between different clients given  $\mathcal{F}_r$ . Then  $\{X_t\}_{t \leq IS}$  is a martingale difference sequence with respect to  $\{\mathcal{H}_t\}_{t \leq IS}$ . Also,  $\|X_t\| \leq \sigma$  almost surely for all  $t \in [IS]$  and  $\mathbb{E}_{t-1}[\|X_t\|] \leq \sigma^2$ . Therefore, by Lemma 7, with probability  $5$  at least  $1 - \delta$ ,
$$
\begin{array}{l} \left\| \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \sum_ {k = 0} ^ {I - 1} \boldsymbol {\epsilon} _ {r, k} ^ {i} \right\| ^ {2} \leq \frac {1}{S ^ {2}} \left(3 \sigma \log \frac {1}{\delta} + 3 \sqrt {S I \sigma^ {2} \log \frac {1}{\delta}}\right) ^ {2} \\ \leq \frac {9 \sigma^ {2} \log^ {2} \frac {1}{\delta}}{S ^ {2}} + \frac {9 I \sigma^ {2} \log \frac {1}{\delta}}{S} \\ \leq \frac {9 \sigma^ {2} I \log \frac {1}{\delta}}{S} \left(\frac {\log \frac {1}{\delta}}{S I} + 1\right) \\ \leq \frac {1 8 \sigma^ {2} I \log^ {2} \frac {1}{\delta}}{S}. \\ \end{array}
$$
Without loss of generality, we assume  $\log (1 / \delta)\geq 1$ . We can apply the exact same argument to bound the remaining noise terms:
$$
\left\| \frac {1}{S} \sum_ {i \in S _ {r}} \sum_ {k = 0} ^ {I - 1} \tilde {\boldsymbol {\epsilon}} _ {r, k} ^ {i} \right\| ^ {2} \leq \frac {1 8 \sigma^ {2} I \log \frac {1}{\delta}}{S},
$$
$$
\left\| \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {k = 0} ^ {I - 1} \tilde {\boldsymbol {\epsilon}} _ {r, k} ^ {i} \right\| ^ {2} \leq \frac {1 8 \sigma^ {2} I \log \frac {1}{\delta}}{N} \leq \frac {1 8 \sigma^ {2} I \log \frac {1}{\delta}}{S},
$$
In fact, we first verify that the bound holds with probability at least  $1 - \delta$  given  $\mathcal{F}_r$  and  $S_{r}$ . Since the upper bound does not depend on  $\mathcal{F}_r$  and  $S_{r}$ , we can conclude the bound holds with the same unconditional probability.
each with probability  $1 - 3\delta$ . Plugging back into Equation 39,
$$
\mathbb {1} _ {\mathcal {A} _ {r}} \| \bar {\boldsymbol {x}} _ {r + 1} - \bar {\boldsymbol {x}} _ {r} \| ^ {2} \leq \mathbb {1} _ {\mathcal {A} _ {r}} \left(5 \eta^ {2} I ^ {2} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| ^ {2} + 1 9 2 \Gamma_ {1} ^ {2} \eta^ {4} I ^ {4} \left(2 \sigma + \frac {\gamma}{\eta}\right) ^ {2} + \frac {4 3 2 \sigma^ {2} \eta^ {2} I \log \frac {1}{\delta}}{S}\right),
$$
with probability  $1 - 9\delta$ . Also, using Equation 25,
$$
\mathbb {1} _ {\mathcal {A} _ {r}} \left(A L _ {0} + B L _ {1} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \|\right) \leq \mathbb {1} _ {\mathcal {A} _ {r}} \left(A L _ {0} + 4 B L _ {1} \left(2 \sigma + \frac {\gamma}{\eta}\right)\right) \leq \mathbb {1} _ {\mathcal {A} _ {r}} \Gamma_ {1}.
$$
$$
\begin{array}{l} A _ {3} = \frac {1}{2} \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} (A L _ {0} + B L _ {1} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \|) \| \bar {\boldsymbol {x}} _ {r + 1} - \bar {\boldsymbol {x}} _ {r} \| ^ {2} \\ \leq \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \left(\frac {5}{2} \Gamma_ {1} \eta^ {2} I ^ {2} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| ^ {2} + 9 6 \Gamma_ {1} ^ {3} \eta^ {4} I ^ {4} \left(2 \sigma + \frac {\gamma}{\eta}\right) ^ {2} + 2 1 6 (A L _ {0} + B L _ {1} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \|) \frac {\sigma^ {2} \eta^ {2} I \log \frac {1}{\delta}}{S}\right) \\ \leq \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \left[ \frac {5}{8} \eta I \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| ^ {2} + \frac {2 1 6 B L _ {1} \sigma^ {2} \eta^ {2} I \log \frac {1}{\delta}}{S} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| \right. \\ \left. + 9 6 \Gamma_ {1} ^ {3} \eta^ {4} I ^ {4} \left(2 \sigma + \frac {\gamma}{\eta}\right) ^ {2} + \frac {2 1 6 A L _ {0} \sigma^ {2} \eta^ {2} I \log \frac {1}{\delta}}{S} \right], \\ \end{array}
$$
with probability  $1 - 9\delta$
Bounding  $\mathbf{A}_4$ : Due to normalization of the update  $g_{r,k}^i$  under  $\bar{\mathcal{A}}_r$ , we have
$$
\begin{array}{l} A _ {4} = \gamma^ {2} \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\bar {\mathcal {A}} _ {r}} \frac {A L _ {0} + B L _ {1} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \|}{2} \left\| \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \sum_ {k = 0} ^ {I - 1} \frac {\boldsymbol {g} _ {r , k} ^ {i}}{\| \boldsymbol {g} _ {r , k} ^ {i} \|} \right\| ^ {2} \\ \leq \gamma^ {2} I ^ {2} \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\bar {\mathcal {A}} _ {r}} \frac {A L _ {0} + B L _ {1} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \|}{2}. \\ \end{array}
$$
Combining the respective bounds for  $A_{1}, A_{2}, A_{3}, A_{4}$  into Equation 28,
$$
\begin{array}{l} f (\bar {\boldsymbol {x}} _ {R}) - f (\bar {\boldsymbol {x}} _ {0}) \\ \leq \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \left[ - \frac {1}{8} \eta I \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| ^ {2} + \left(2 \Gamma_ {1} \eta^ {2} I ^ {2} \left(2 \sigma + \frac {\gamma}{\eta}\right) + \frac {2 1 6 B L _ {1} \sigma^ {2} \eta^ {2} I \log \frac {1}{\delta}}{S}\right) \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| \right. \\ \left. + 9 6 \Gamma_ {1} ^ {3} \eta^ {4} I ^ {4} \left(2 \sigma + \frac {\gamma}{\eta}\right) ^ {2} + \frac {2 1 6 A L _ {0} \sigma^ {2} \eta^ {2} I \log \frac {1}{\delta}}{S} \right] \\ + \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\bar {A} _ {r}} \left[ - \gamma I \left(\frac {3}{4} - 2 B L _ {1} \rho \gamma I - \frac {1}{2} B L _ {1} \gamma I\right) \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| - \gamma I \left(\frac {1}{8} \frac {\gamma}{\eta} - 6 \sigma - 2 A L _ {0} \gamma I - \frac {5}{2} B L _ {1} \kappa \gamma I\right) \right] \\ + \eta \left(1 2 \sigma^ {2} + 6 4 I \left(\kappa + 5 \rho \left(2 \sigma + \frac {\gamma}{\eta}\right)\right) ^ {2} \left(\frac {1}{S} - \frac {1}{N}\right)\right) \log \frac {1}{\delta}, \\ \end{array}
$$
which is the desired result. Note that the bound on  $A_{1}$  holds with probability  $1 - 5\delta$  and the bound on  $A_{2}$  holds with probability  $1 - 9\delta$ , and we initially supposed the event  $\mathcal{E}$ , which holds with probability  $1 - \delta$ . So the overall result holds with probability at least  $1 - 15\delta$ .
Theorem 1 restated. Let  $\epsilon \leq \frac{AL_0}{16BL_1\rho}$  and  $\delta \in (0,1)$ . Denote  $K = \left\lceil \frac{\log(RN / \delta)}{\log(N / (N - S))} \right\rceil$ ,  $\Gamma_1 := AL_0 + BL_1\kappa + 4BL_1\rho \left(2\sigma + \frac{\gamma}{\eta}\right)$  and  $\Gamma_2 := 64\left(\kappa + 5\rho \left(2\sigma + \frac{\gamma}{\eta}\right)\right)^2\left(\frac{1}{S} - \frac{1}{N}\right)$ . If
$$
\eta \leq \min  \left\{\frac {1}{9 0 (K + 1) \Gamma_ {1} I}, \frac {\epsilon}{3 2 \Gamma_ {1} I \left(7 4 \sigma + \frac {A L _ {0}}{B L _ {1} \rho}\right)}, \frac {S \epsilon^ {2}}{2 1 6 A L _ {0} \sigma^ {2} \log \frac {1}{\delta}}, \frac {\Delta}{\log \frac {1}{\delta}} \min  \left\{\frac {1}{1 2 \sigma^ {2}}, \frac {1}{\Gamma_ {2} I} \right\} \right\},
$$
and  $\gamma = \left(72\sigma +\frac{AL_0}{BL_1\rho}\right)\eta$ , then Algorithm 1 satisfies  $\frac{1}{R}\sum_{r = 0}^{R - 1}\| \nabla f(\bar{\boldsymbol{x}}_r)\| \leq 35\epsilon$  with probability at least  $1 - 15\delta$ , as long as  $R\geq \frac{8\Delta}{\epsilon^2\eta I}$ .
Proof. First, under our choice of  $\eta$  and  $\gamma$ ,
$$
(K + 1) \Gamma_ {1} \eta I \leq \frac {1}{6 0},
$$
$$
2 (K + 1) \eta I \left(2 \sigma + \frac {\gamma}{\eta}\right) \leq \frac {\left(2 \sigma + \frac {\gamma}{\eta}\right)}{3 0 \Gamma_ {1}} = \frac {\left(2 \sigma + \frac {\gamma}{\eta}\right)}{3 0 \left(A L _ {0} + B L _ {1} \kappa + 4 B L _ {1} \rho \left(2 \sigma + \frac {\gamma}{\eta}\right) \right.} \leq \frac {1}{1 2 0 B L _ {1} \rho} \leq \frac {1}{1 2 0 L _ {1}},
$$
where the last line follows from  $B, \rho \geq 1$ . Therefore Equation 9 holds under our choice of  $\eta$  and  $\gamma$ , so the condition of Lemma 12 is satisfied. Denoting
$$
U (\pmb {x}) = - \gamma I \left(\frac {3}{4} - 2 B L _ {1} \rho \gamma I - \frac {1}{2} B L _ {1} \gamma I\right) \| \nabla f (\pmb {x}) \| - \gamma I \left(\frac {1}{8} \frac {\gamma}{\eta} - 6 \sigma - 2 A L _ {0} \gamma I - \frac {5}{2} B L _ {1} \kappa \gamma I\right),
$$