SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {1} _ {\mathcal {A} _ {r}} \left\| \boldsymbol {x} _ {r, k + 1} ^ {i} - \bar {\boldsymbol {x}} _ {r} \right\| \leq 2 \sigma \eta + \gamma + \left(1 + \frac {1}{4 I}\right) \left\| \boldsymbol {x} _ {r, k} ^ {i} - \bar {\boldsymbol {x}} _ {r} \right\| + \frac {1}{4 (K + 1) I ^ {2}} \sum_ {j = 0} ^ {I - 1} \| \bar {\boldsymbol {x}} _ {r} - \boldsymbol {y} _ {r, j} ^ {i} \| \\ \leq 2 \sigma \eta + \gamma + 2 k \left(1 + \frac {1}{4 I}\right) (2 \sigma \eta + \gamma) + \frac {1}{4 (K + 1) I ^ {2}} \sum_ {j = 0} ^ {I - 1} 2 (K + 1) I (2 \sigma \eta + \gamma) \\ \leq 2 \left(\frac {1}{2} + k + \frac {k}{4 I} + \frac {1}{4}\right) (2 \sigma \eta + \gamma) \\ \leq 2 (k + 1) (2 \sigma \eta + \gamma), \\ \end{array}
$$
where we used the inductive hypothesis over  $k$  and the inductive hypothesis over  $r$ . This fact together with  $\mathbb{1}_{\bar{A}_r} \| \pmb{x}_{r,k+1}^i - \bar{\pmb{x}}_r \| \leq 2(k+1)(2\sigma \eta + \gamma)$  (which trivially holds due to the clipping operator), we prove that Equation 13 holds for  $k+1$ , completing the induction over  $k$ . Therefore Equation 13 holds for all  $k \in [I]$ .
Lemma 1 restated. Suppose that Equation 9 holds. Then for any  $r \geq 0$  we have
$$
\mathbb {1} _ {\varepsilon} \max  _ {k \in [ I ]} \| \boldsymbol {x} _ {r, k} ^ {i} - \bar {\boldsymbol {x}} _ {r} \| \leq 2 I (2 \sigma \eta + \gamma), \quad \text {f o r a n y} i \in \mathcal {S} _ {r}, \tag {22}
$$
$$
\mathbb {1} _ {\mathcal {E}} \max  _ {k \in [ I ]} \| \boldsymbol {y} _ {r, k} ^ {i} - \bar {\boldsymbol {x}} _ {r} \| \leq 2 (K + 1) I (2 \sigma \eta + \gamma), \quad \text {f o r a n y} i \in [ N ]. \tag {23}
$$
Proof. We will use induction to prove Equation 22 by an induction over  $r$ . And Equation 23 holds as a consequence of the inductive assumption due to Lemma 11. Invoking Lemma 10, we can verify the base case of the induction over  $r$ . Now we assume  $\max_{k\in [I],i\in [N]}\| \pmb{x}_{q,k}^i -\bar{\pmb{x}}_q\| \leq 2(K + 1)I(2\sigma \eta +\gamma)$  holds for any  $q\leq r - 1$ . Then according to Lemma 11, we can prove Equation 22 and Equation 23 immediately. It also completes the induction of Equation 22 over  $r$ .
Corollary 2. Suppose that Equation 9 holds. Then for any  $r \geq 0$  and  $i \in [N]$ ,
$$
\mathbb {1} _ {\varepsilon} \Xi_ {r} \leq 2 (K + 1) \eta I \left(\sigma + \frac {\gamma}{\eta}\right) \tag {24}
$$
$$
\mathbb {1} _ {\mathcal {E} \cap \mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| \leq 4 \left(2 \sigma + \frac {\gamma}{\eta}\right) \tag {25}
$$
$$
\mathbb {1} _ {\mathcal {E} \cap \mathcal {A} _ {r}} \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \boldsymbol {G} _ {r} ^ {i} \| \leq 2 \sigma + \frac {\gamma}{3 0 \eta} \tag {26}
$$
$$
\mathbb {1} _ {\mathcal {E} \cap \mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) - \boldsymbol {G} _ {r} \| \leq 2 \sigma + \frac {\gamma}{3 0 \eta}. \tag {27}
$$
Proof. All proofs of this corollary are under the event  $\mathcal{E}$ . Equation 24 follows directly from Equation 14 and the definition  $\Xi_r = \frac{1}{NI} \sum_{i=1}^N \sum_{k=1}^I \|\pmb{y}_{r,k}^i - \bar{\pmb{x}}_r\|$ . Equation 25 is exactly Equation 18.
Due to Lemma 1, we can apply Lemma 2 such that
$$
\begin{array}{l} \mathbb {1} _ {\mathcal {E}} \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \boldsymbol {G} _ {r} ^ {i} \| \leq \mathbb {1} _ {\mathcal {E}} \left(\frac {1}{I} \sum_ {k = 0} ^ {I - 1} \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \nabla f _ {i} (\boldsymbol {y} _ {r, k} ^ {i}) \| + \frac {1}{I} \sum_ {k = 0} ^ {I - 1} \| \nabla f _ {i} (\boldsymbol {y} _ {r, k} ^ {i}) - \nabla F _ {i} (\boldsymbol {y} _ {r, k} ^ {i}; \xi_ {q _ {r, k} ^ {i}} ^ {i}) \|\right) \\ \leq \mathbb {1} _ {\mathcal {E}} \left(\frac {1}{I} \sum_ {k = 0} ^ {I - 1} \left(A L _ {0} + B L _ {1} \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \|\right) \| \bar {\boldsymbol {x}} _ {r} - \boldsymbol {y} _ {r, k} ^ {i} \| + \sigma\right) \\ \leq \left(A L _ {0} + B L _ {1} \kappa + B L _ {1} \rho \mathbb {1} _ {\mathcal {E}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \|\right) \Xi_ {r} + \sigma . \\ \end{array}
$$
So, using Equation 24 and Equation 25,
$$
\begin{array}{l} \mathbb {1} _ {\mathcal {E} \cap \mathcal {A} _ {r}} \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \boldsymbol {G} _ {r} ^ {i} \| \leq 2 (K + 1) \eta I \left(2 \sigma + \frac {\gamma}{\eta}\right) \left(A L _ {0} + B L _ {1} \kappa + 4 B L _ {1} \rho \left(2 \sigma + \frac {\gamma}{\eta}\right)\right) + \sigma \\ \leq \frac {1}{3 0} \left(2 \sigma + \frac {\gamma}{\eta}\right) + \sigma \\ \leq 2 \sigma + \frac {\gamma}{3 0 \eta}, \\ \end{array}
$$
where we used Equation 9. Equation 27 follows immediately from Equation 26 by
$$
\mathbb {1} _ {\mathcal {E} \cap \mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) - \boldsymbol {G} _ {r} \| \leq \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {1} _ {\mathcal {E} \cap \mathcal {A} _ {r}} \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \boldsymbol {G} _ {r} ^ {i} \| \leq 2 \sigma + \frac {\gamma}{3 0 \eta}.
$$
A.4 Proof of descent inequality
Lemma 12. Suppose Equation 9 holds. Then with probability  $1 - 15\delta$ , Algorithm 1 satisfies
$$
\begin{array}{l} f (\bar {x} _ {R}) - f (\bar {x} _ {0}) \\ \leq \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \left[ - \frac {1}{8} \eta I \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| ^ {2} + \left(2 \Gamma_ {1} \eta^ {2} I ^ {2} \left(2 \sigma + \frac {\gamma}{\eta}\right) + \frac {2 1 6 B L _ {1} \sigma^ {2} \eta^ {2} I \log \frac {1}{\delta}}{S}\right) \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| \right. \\ \left. + 9 6 \Gamma_ {1} ^ {3} \eta^ {4} I ^ {4} \left(2 \sigma + \frac {\gamma}{\eta}\right) ^ {2} + \frac {1 0 8 A L _ {0} \sigma^ {2} \eta^ {2} I \log \frac {1}{\delta}}{S} \right] \\ + \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\bar {A} _ {r}} \left[ - \gamma I \left(\frac {3}{4} - 2 B L _ {1} \rho \gamma I - \frac {1}{2} B L _ {1} \gamma I\right) \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| \right. \\ \left. - \gamma I \left(\frac {1}{8} \frac {\gamma}{\eta} - 6 \sigma - 2 A L _ {0} \gamma I - \frac {5}{2} B L _ {1} \kappa \gamma I\right) \right] \\ + \eta \left(1 2 \sigma^ {2} + 6 4 I \left(\kappa + 5 \rho \left(2 \sigma + \frac {\gamma}{\eta}\right)\right) ^ {2} \left(\frac {1}{S} - \frac {1}{N}\right)\right) \log \frac {1}{\delta}, \\ \end{array}
$$
$$
w h e r e \Gamma_ {1} = A L _ {0} + B L _ {1} \kappa + 4 B L _ {1} \rho \left(2 \sigma + \frac {\gamma}{\eta}\right).
$$
Proof. By Lemma 8,  $\mathbb{P}(\mathcal{E})\geq 1 - \delta$ . The remainder of the proof will suppose the event  $\mathcal{E}$  happens, and the results will hold with probability  $1 - \delta$ . By Lemma 1 and Equation 9,  $\| \bar{\boldsymbol{x}}_{r + 1} - \bar{\boldsymbol{x}}_r\| \leq \frac{C}{L_1}$ . Denote  $\mathcal{A}_r = \{\| G_r\| \leq \gamma /\eta \}$ . Therefore we may apply Lemma 3 to obtain
$$
\begin{array}{l} f \left(\bar {\boldsymbol {x}} _ {r + 1}\right) - f \left(\bar {\boldsymbol {x}} _ {r}\right) \\ \leq \left\langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \bar {\boldsymbol {x}} _ {r + 1} - \bar {\boldsymbol {x}} _ {r} \right\rangle + \frac {A L _ {0} + B L _ {1} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \|}{2} \| \bar {\boldsymbol {x}} _ {r + 1} - \bar {\boldsymbol {x}} _ {r} \| ^ {2} \\ \leq - \mathbb {1} _ {\mathcal {A} _ {r}} \eta \left\langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \sum_ {k = 0} ^ {I - 1} \boldsymbol {g} _ {r, k} ^ {i} \right\rangle - \mathbb {1} _ {\bar {\mathcal {A}} _ {r}} \gamma \left\langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \sum_ {k = 0} ^ {I - 1} \frac {\boldsymbol {g} _ {r , k} ^ {i}}{\| \boldsymbol {g} _ {r , k} ^ {i} \|} \right\rangle + \\ \mathbb {1} _ {\mathcal {A} _ {r}} \frac {A L _ {0} + B L _ {1} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \|}{2} \| \bar {\boldsymbol {x}} _ {r + 1} - \bar {\boldsymbol {x}} _ {r} \| ^ {2} + \mathbb {1} _ {\bar {\mathcal {A}} _ {r}} \frac {A L _ {0} + B L _ {1} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \|}{2} \| \bar {\boldsymbol {x}} _ {r + 1} - \bar {\boldsymbol {x}} _ {r} \| ^ {2}. \\ \end{array}
$$
Summing over  $r = 0,\ldots ,R - 1$  yields
$$
\begin{array}{l} f(\bar{\boldsymbol{x}}_{R}) - f(\bar{\boldsymbol{x}}_{0})\leq \underbrace{-\eta\sum_{r = 0}^{R - 1}\mathbb{1}_{\mathcal{A}_{r}}\left\langle\nabla f(\bar{\boldsymbol{x}}_{r}),\frac{1}{S}\sum_{i\in\mathcal{S}_{r}}\sum_{k = 0}^{I - 1}\boldsymbol{g}_{r,k}^{i}\right\rangle}_{A_{1}} \\ \underbrace {- \gamma \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\bar {\mathcal {A}} _ {r}} \left\langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \sum_ {k = 0} ^ {I - 1} \frac {\boldsymbol {g} _ {r , k} ^ {i}}{\| \boldsymbol {g} _ {r , k} ^ {i} \|} \right\rangle} _ {A _ {2}} \\ + \underbrace {\sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \frac {A L _ {0} + B L _ {1} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \|}{2} \| \bar {\boldsymbol {x}} _ {r + 1} - \bar {\boldsymbol {x}} _ {r} \| ^ {2}} _ {A _ {3}} \\ + \underbrace {\sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\bar {\mathcal {A}} _ {r}} \frac {A L _ {0} + B L _ {1} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \|}{2} \| \bar {\boldsymbol {x}} _ {r + 1} - \bar {\boldsymbol {x}} _ {r} \| ^ {2}} _ {A _ {4}}. \tag {28} \\ \end{array}
$$
We proceed by bounding each of the four terms in Equation 28.
Bounding  $\mathbf{A_1}$  .Denote
$$
\boldsymbol {\epsilon} _ {r, k} ^ {i} = \nabla F _ {i} (\boldsymbol {x} _ {r, k} ^ {i}; \xi_ {r, k} ^ {i}) - \nabla f _ {i} (\boldsymbol {x} _ {r, k} ^ {i}),
$$
$$
\boldsymbol {\epsilon} _ {r} ^ {(1)} = \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \boldsymbol {G} _ {r} ^ {i} - \boldsymbol {G} _ {r},
$$
$$
\boldsymbol {\epsilon} _ {r} ^ {(2)} = \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \nabla f (\bar {\boldsymbol {x}} _ {r}).
$$
Then we can write
$$
\begin{array}{l} \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \boldsymbol {g} _ {r, k} ^ {i} = \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \nabla F _ {i} (\boldsymbol {x} _ {r, k} ^ {i}; \boldsymbol {\xi} _ {r, k} ^ {i}) - \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \boldsymbol {G} _ {r} ^ {i} + \boldsymbol {G} _ {r} \\ = \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \nabla f _ {i} (\boldsymbol {x} _ {r, k} ^ {i}) + \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \boldsymbol {\epsilon} _ {r, k} ^ {i} - \boldsymbol {\epsilon} _ {r} ^ {(1)} \\ = \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) + \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \left(\nabla f _ {i} (\boldsymbol {x} _ {r, k} ^ {i}) - \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r})\right) + \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \boldsymbol {\epsilon} _ {r, k} ^ {i} - \boldsymbol {\epsilon} _ {r} ^ {(1)} \\ = \nabla f (\bar {\boldsymbol {x}} _ {r}) - \boldsymbol {\epsilon} _ {r} ^ {(1)} + \boldsymbol {\epsilon} _ {r} ^ {(2)} + \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \left(\nabla f _ {i} (\boldsymbol {x} _ {r, k} ^ {i}) - \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r})\right) + \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \boldsymbol {\epsilon} _ {r, k} ^ {i}. \\ \end{array}
$$
$$
\begin{array}{l} A _ {1} = - \eta \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \sum_ {k = 0} ^ {I - 1} \left\langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \boldsymbol {g} _ {r, k} ^ {i} \right\rangle \\ = - \eta I \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| ^ {2} - \eta \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \left\langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \sum_ {k = 0} ^ {I - 1} \nabla f _ {i} \left(\boldsymbol {x} _ {r, k} ^ {i}\right) - \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \right\rangle \\ - \eta \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \sum_ {k = 0} ^ {I - 1} \left\langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \boldsymbol {\epsilon} _ {r, k} ^ {i} \right\rangle + \eta I \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \left\langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \boldsymbol {\epsilon} _ {r} ^ {(1)} - \boldsymbol {\epsilon} _ {r} ^ {(2)} \right\rangle \\ \leq - \eta I \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| ^ {2} - \eta \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \left\langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \sum_ {k = 0} ^ {I - 1} \nabla f _ {i} \left(\boldsymbol {x} _ {r, k} ^ {i}\right) - \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \right\rangle \\ + \eta \left| \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \sum_ {k = 0} ^ {I - 1} \left\langle \nabla f (\bar {x} _ {r}), \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \boldsymbol {\epsilon} _ {r, k} ^ {i} \right\rangle \right| + \eta I \left| \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \left\langle \nabla f (\bar {x} _ {r}), \boldsymbol {\epsilon} _ {r} ^ {(1)} \right\rangle \right| \\ + \eta I \left| \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \left\langle \nabla f \left(\bar {\boldsymbol {x}} _ {r}\right), \boldsymbol {\epsilon} _ {r} ^ {(2)} \right\rangle \right|. \tag {29} \\ \end{array}
$$
Since  $\| \pmb{x}_{r,k}^i -\bar{\pmb{x}}_r\| \leq C / L_1$  due to Equation 13, we can apply Lemma 2 such that
$$
- \eta \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \left\langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \sum_ {k = 0} ^ {I - 1} \nabla f _ {i} \left(\boldsymbol {x} _ {r, k} ^ {i}\right) - \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \right\rangle
$$
$$
\leq \eta \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| \left\| \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \sum_ {k = 0} ^ {I - 1} \nabla f _ {i} \left(\boldsymbol {x} _ {r, k} ^ {i}\right) - \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \right\|
$$
$$
\leq \eta \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| \cdot \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \sum_ {k = 0} ^ {I - 1} \left\| \nabla f _ {i} \left(\boldsymbol {x} _ {r, k} ^ {i}\right) - \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \right\|
$$
$$
\leq \eta \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| \cdot \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \left(A L _ {0} + B L _ {1} \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \|\right) \sum_ {k = 0} ^ {I - 1} \left\| \boldsymbol {x} _ {r, k} ^ {i} - \bar {\boldsymbol {x}} _ {r} \right\|
$$
$$
\stackrel {(i)} {\leq} 2 \eta^ {2} I ^ {2} \left(2 \sigma + \frac {\gamma}{\eta}\right) \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| (A L _ {0} + B L _ {1} \kappa + B L _ {1} \rho \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \|)
$$
$$
\begin{array}{l} \stackrel {(i i)} {\leq} 2 \eta^ {2} I ^ {2} \left(A L _ {0} + B L _ {1} \kappa\right) \left(2 \sigma + \frac {\gamma}{\eta}\right) \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| \\ + 2 \eta^ {2} I ^ {2} B L _ {1} \rho \left(2 \sigma + \frac {\gamma}{\eta}\right) \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| ^ {2} \\ \end{array}
$$
$$
\stackrel {(i i i)} {\leq} 2 \eta^ {2} I ^ {2} (A L _ {0} + B L _ {1} \kappa) \left(2 \sigma + \frac {\gamma}{\eta}\right) \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| + \frac {1}{8} \eta I \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| ^ {2},
$$
where  $(i)$  holds due to Equation 13;  $(ii)$  follows from Assumption 1(iv); and  $(iii)$  follows from Equation 9. Plugging into Equation 29,
$$
\begin{array}{l} A _ {1} \leq - \frac {7}{8} \eta I \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| ^ {2} + 2 \eta^ {2} I ^ {2} (A L _ {0} + B L _ {1} \kappa) \left(2 \sigma + \frac {\gamma}{\eta}\right) \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| \\ + \underbrace {\eta \left| \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \sum_ {k = 0} ^ {I - 1} \left\langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \boldsymbol {\epsilon} _ {r , k} ^ {i} \right\rangle \right|} _ {B _ {1}} \\ + \underbrace {\eta I \left| \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \left\langle \nabla f (\bar {\boldsymbol {x}} _ {r}) , \boldsymbol {\epsilon} _ {r} ^ {(1)} \right\rangle \right|} _ {B _ {2}} \\ + \underbrace {\eta I \left| \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \left\langle \nabla f \left(\bar {\boldsymbol {x}} _ {r}\right) , \boldsymbol {\epsilon} _ {r} ^ {(2)} \right\rangle \right|} _ {B _ {3}}. \tag {30} \\ \end{array}
$$
Denote each of the three last terms (those involving  $\epsilon_{r,k}^{i}$ ,  $\epsilon_r^{(1)}$ , and  $\epsilon_r^{(2)}$ , respectively) as  $B_{1}, B_{2}, B_{3}$ .  $B_{1}$  can be bounded using the martingale concentration bound in Lemma 5.
Denoting  $\mathcal{G}_{r,k} = \sigma (\mathcal{F}_r,\mathcal{S}_r,\{\xi_{r,\ell}^i:i\in \mathcal{S}_r,\ell \leq k\})$  , we have
$$
\mathbb {E} \left[ \mathbb {1} _ {\mathcal {A} _ {r}} \left\langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \boldsymbol {\epsilon} _ {r, k} ^ {i} \right\rangle \mid \mathcal {G} _ {r, k - 1} \right] = 0
$$
so that  $\left\{\mathbb{1}_{\mathcal{A}_r}\left\langle \nabla f(\bar{\boldsymbol{x}}_r),\frac{1}{S}\sum_{i\in \mathcal{S}_r}\epsilon_{r,k}^i\right\rangle \right\}_{r,k}$  is a martingale difference sequence with respect to  $\mathcal{G}_{r,k}$ . Since  $\mathbb{1}_{\mathcal{A}_r}\in \mathcal{F}_r\in \mathcal{G}_{r,k}$  for any  $0\leq k\leq I$ , we have
$$
\begin{array}{l} \mathbb {E} \left[ \exp \left(\frac {\mathbb {1} _ {\mathcal {A} _ {r}} \left\langle \nabla f (\bar {\boldsymbol {x}} _ {r}) , \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \boldsymbol {\epsilon} _ {r , k} ^ {i} \right\rangle^ {2}}{\mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| ^ {2} \sigma^ {2}}\right) | \mathcal {G} _ {r, k - 1} \right] \\ \leq \mathbb {E} \left[ \exp \left(\frac {\frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \| \boldsymbol {\epsilon} _ {r , k} ^ {i} \| ^ {2}}{\sigma^ {2}}\right) \mid \mathcal {G} _ {r, k - 1} \right] \\ \leq \exp (1), \\ \end{array}
$$
where we used Cauchy-Schwarz, Jensen's inequality, and  $\| \pmb{\epsilon}_{r,k}^{i}\| \leq \sigma$ . Therefore we can apply Lemma 5 to obtain
$$
B _ {1} \leq \frac {3}{4} \eta \sigma^ {2} I \lambda_ {1} \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| ^ {2} + \frac {\eta}{\lambda_ {1}} \log \frac {1}{\delta}, \tag {31}
$$
with probability  $1 - \delta$  for any  $\lambda_1 > 0$ .
The concentration bounds for  $B_{2}$  and  $B_{3}$  will rely on the concentration bound for sampling without replacement in Lemma 6. We start with  $B_{2}$ . Notice that
$$
\mathbb {1} _ {\mathcal {A} _ {r}} \left\langle \nabla f \left(\bar {\boldsymbol {x}} _ {r}\right), \boldsymbol {\epsilon} _ {r} ^ {(1)} \right\rangle = \frac {1}{S} \sum_ {i \in \mathcal {S} _ {r}} \mathbb {1} _ {\mathcal {A} _ {r}} \left\langle \nabla f \left(\bar {\boldsymbol {x}} _ {r}\right), \boldsymbol {G} _ {r} ^ {i} \right\rangle - \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {1} _ {\mathcal {A} _ {r}} \left\langle \nabla f \left(\bar {\boldsymbol {x}} _ {r}\right), \boldsymbol {G} _ {r} ^ {i} \right\rangle , \tag {32}
$$
so we must upper bound  $\mathbb{1}_{\mathcal{A}_r}|\langle \nabla f(\bar{\pmb{x}}_r),\pmb{G}_r^i\rangle |$  to apply Lemma 6. Under  $\mathcal{A}_r$
$$
\begin{array}{l} \left\| \boldsymbol {G} _ {r} ^ {i} \right\| \leq \left\| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) \right\| + \left\| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \boldsymbol {G} _ {r} ^ {i} \right\| \\ \leq \kappa + \rho \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| + \| \nabla f _ {i} (\bar {\boldsymbol {x}} _ {r}) - \boldsymbol {G} _ {r} ^ {i} \| \\ \stackrel {(i)} {\leq} \kappa + 4 \rho \left(2 \sigma + \frac {\gamma}{\eta}\right) + \left(2 \sigma + \frac {\gamma}{\eta}\right) \\ \leq \kappa + (4 \rho + 1) \left(2 \sigma + \frac {\gamma}{\eta}\right) \\ \stackrel {(i i)} {\leq} \kappa + 5 \rho \left(2 \sigma + \frac {\gamma}{\eta}\right). \tag {33} \\ \end{array}
$$
where  $(i)$  holds due to Equation 25 and Equation 26; and  $(ii)$  holds since  $\rho \geq 1$ . Therefore
$$
\mathbb {1} _ {\mathcal {A} _ {r}} | \langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \boldsymbol {G} _ {r} ^ {i} \rangle | \leq \left(\kappa + 5 \rho \left(2 \sigma + \frac {\gamma}{\eta}\right)\right) \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \|,
$$
and the condition of Lemma 6 is satisfied. Notice that  $\mathbb{1}_{\mathcal{A}_r} \in \mathcal{F}_r$  because  $\pmb{G}_r$  is constructed by the history gradients before round  $r$ . Therefore, applying Lemma 6 to Equation 32 gives<sup>4</sup>
$$
\begin{array}{l} \mathbb {E} _ {r} \left[ \exp \left(\lambda_ {2} \mathbb {1} _ {\mathcal {A} _ {r}} \left\langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \epsilon_ {r} ^ {(1)} \right\rangle\right) \right] \\ \leq \exp \left\{\frac {1}{2} \lambda_ {2} ^ {2} \left(\kappa + 5 \rho \left(2 \sigma + \frac {\gamma}{\eta}\right)\right) ^ {2} \frac {S + 1}{S ^ {2}} \left(1 - \frac {S}{N}\right) \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| ^ {2} \right\} \\ \leq \exp \left\{\lambda_ {2} ^ {2} \left(\kappa + 5 \rho \left(2 \sigma + \frac {\gamma}{\eta}\right)\right) ^ {2} \left(\frac {1}{S} - \frac {1}{N}\right) \mathbb {1} _ {\mathcal {A} _ {r}} \| \nabla f (\bar {\boldsymbol {x}} _ {r}) \| ^ {2} \right\} \\ =: \exp \left\{\lambda_ {2} ^ {2} M _ {r} \right\} \tag {34} \\ \end{array}
$$
$$
\begin{array}{l} Y _ {r} = \exp \left\{\sum_ {q = 0} ^ {r} \lambda_ {2} \left(\mathbb {1} _ {\mathcal {A} _ {q}} \langle \nabla f (\bar {\boldsymbol {x}} _ {q}), \boldsymbol {\epsilon} _ {q} ^ {(1)} \rangle - \lambda_ {2} M _ {r}\right) \right\} \\ = Y _ {r - 1} \exp \left\{\lambda_ {2} \mathbb {1} _ {\mathcal {A} _ {r}} \langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \boldsymbol {\epsilon} _ {r} ^ {(1)} \rangle - \lambda_ {2} ^ {2} M _ {r} \right\}. \\ \end{array}
$$
Then we know  $\mathbb{E}_r[Y_r] \leq Y_{r-1}$  by Equation 34, which implies
$$
\mathbb {E} \left[ Y _ {R - 1} \right] = \mathbb {E} \left[ \mathbb {E} _ {R - 1} \left[ Y _ {R - 1} \right] \right] \leq \mathbb {E} \left[ Y _ {R - 2} \right] \leq \dots \leq \mathbb {E} \left[ Y _ {0} \right] \leq 1.
$$
Therefore for any  $\lambda_2 > 0$ , we have
$$
\begin{array}{l} \mathbb {P} \left(\sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \boldsymbol {\epsilon} _ {r} ^ {(1)} \rangle \geq \lambda_ {2} \sum_ {r = 0} ^ {R - 1} M _ {r} + \frac {1}{\lambda_ {2}} \log \frac {1}{\delta}\right) \\ = \mathbb {P} \left(\lambda_ {2} \sum_ {r = 0} ^ {R - 1} \mathbb {1} _ {\mathcal {A} _ {r}} \langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \boldsymbol {\epsilon} _ {r} ^ {(1)} \rangle \geq \lambda_ {2} ^ {2} \sum_ {r = 0} ^ {R - 1} M _ {r} + \log \frac {1}{\delta}\right) \\ = \mathbb {P} \left(\exp \left\{\sum_ {r = 0} ^ {R - 1} \lambda_ {2} \left(\mathbb {1} _ {\mathcal {A} _ {r}} \langle \nabla f (\bar {\boldsymbol {x}} _ {r}), \boldsymbol {\epsilon} _ {r} ^ {(1)} \rangle - \lambda_ {2} M _ {r}\right) \right\} \geq \frac {1}{\delta}\right) \\ = \mathbb {P} \left(Y _ {R - 1} \geq \frac {1}{\delta}\right) \\ \leq \delta , \\ \end{array}
$$