SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} r _ {K} \stackrel {(2 5)} {\leq} r _ {0} \exp (- a \gamma (K + 1)) + c _ {1} \gamma + c _ {2} \gamma^ {2} \\ = \quad \widetilde {\mathcal {O}} \left(r _ {0} \exp \left(- \frac {\ln \left(\max  \left\{2 , \min  \left\{a r _ {0} K / c _ {1} , a ^ {2} r _ {0} K ^ {2} / c _ {2} \right\} \right\}\right)}{a (K + 1)} a (K + 1)\right)\right) \\ + \widetilde {\mathcal {O}} \left(\frac {c _ {1}}{a K} + \frac {c _ {2}}{a ^ {2} K ^ {2}}\right) \\ = \left. \right. \widetilde {\mathcal {O}} \left(r _ {0} \exp \left(- \ln \left(\max  \left\{2, \min  \left\{\frac {a r _ {0} K}{c _ {1}}, \frac {a ^ {2} r _ {0} K ^ {2}}{c _ {2}} \right\}\right\}\right)\right)\right) \\ + \widetilde {\mathcal {O}} \left(\frac {c _ {1}}{a K} + \frac {c _ {2}}{a ^ {2} K ^ {2}}\right) \\ { = } { \widetilde { \mathcal { O } } \left( \frac { c _ { 1 } } { a K } + \frac { c _ { 2 } } { a ^ { 2 } K ^ { 2 } } \right) . } \\ \end{array}
$$
2. If  $\gamma_0 \leq \frac{\ln\left(\max\{2, \min\{aK / c_1, a^2r_0K^2 / c_2\}\}\right)}{a(K + 1)}$  then we choose  $\gamma = \gamma_0$  which implies that
$$
\begin{array}{l} r _ {K} \stackrel {(2 5)} {\leq} r _ {0} \exp \left(- a \gamma_ {0} (K + 1)\right) + c _ {1} \gamma_ {0} + c _ {2} \gamma_ {0} ^ {2} \\ = \widetilde {\mathcal {O}} \left(r _ {0} \exp (- a \gamma_ {0} (K + 1)) + \frac {c _ {1}}{a K} + \frac {c _ {2}}{a ^ {2} K ^ {2}}\right). \\ \end{array}
$$
Combining the obtained bounds we get the result.
D Methods that use robust aggregators
First of we provide the result of Karimireddy et al. [2022] that describes error of RAGG, where  $\overline{\pmb{m}}^t = \alpha \overline{\pmb{g}}^t + (1 - \alpha) \overline{\pmb{m}}^{t-1}$ .
Lemma D.1 (Aggregation error Karimireddy et al. [2022]). Given that RAGG satisfies 1.1 holds, the error between the ideal average momentum  $\overline{\boldsymbol{m}}^t$  and the output of the robust aggregation rule  $\boldsymbol{m}^t$  for any  $t\geq 1$  can be bounded as
$$
\mathbb {E} \left\| \boldsymbol {m} ^ {t} - \overline {{\boldsymbol {m}}} ^ {t} \right\| ^ {2} \leq c \delta (\rho^ {t}) ^ {2},
$$
where we define for  $t \geq 1$
$$
\left(\rho^ {t}\right) ^ {2} := 4 \left(6 \alpha \sigma^ {2} + 3 \zeta^ {2}\right) + 4 \left(6 \sigma^ {2} - 3 \zeta^ {2}\right) (1 - \alpha) ^ {t + 1}.
$$
For  $t = 0$  we can simplify the bound as  $\left(\rho^0\right)^2 \coloneqq 24\sigma^2 + 12\zeta^2$ .
Moreover, one can state a uniform bound for  $(\rho^t)^2$
$$
\left(\rho^ {t}\right) ^ {2} \leq \rho^ {2} = 2 4 \sigma^ {2} + 1 2 \zeta^ {2}. \tag {26}
$$
D.1 Proofs for SGDA-RA
Algorithm 1 SGDA-RA
Input: RAGG,  $\gamma$   
1: for  $t = 0, \ldots$  do  
2: for worker  $i \in [n]$  in parallel  
3:  $\pmb{g}_i^t \gets \pmb{g}_i(\pmb{x}^t, \pmb{\xi}_i)$   
4: send  $\pmb{g}_i^t$  if  $i \in \mathcal{G}$ , else send * if Byzantine  
5:  $\widehat{\pmb{g}}^t = \mathrm{RAGG}(\pmb{g}_1^t, \dots, \pmb{g}_n^t)$  and  $\pmb{x}^{t+1} \gets \pmb{x}^t - \gamma \widehat{\pmb{g}}^t$ . // update params using robust aggregate
D.1.1 Quasi-Strongly Monotone Case
Theorem (Theorem 1 duplicate). Let Assumptions 1, 2, 4 and 6 hold. Then after  $T$  iterations SGDA-RA (Algorithm 1) with  $(\delta, c)$ -RAGG and  $\gamma \leq \frac{1}{2\ell}$  outputs  $\boldsymbol{x}^T$  such that
$$
\mathbb {E} \left\| \boldsymbol {x} ^ {T} - \boldsymbol {x} ^ {*} \right\| ^ {2} \leq \left(1 - \frac {\gamma \mu}{2}\right) ^ {T} \left\| \boldsymbol {x} ^ {0} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \frac {2 \gamma \sigma^ {2}}{\mu G} + \frac {2 \gamma c \delta \rho^ {2}}{\mu} + \frac {c \delta \rho^ {2}}{\mu^ {2}},
$$
where  $\rho^2 = 24\sigma^2 +12\zeta^2$  by Lemma D.1 with  $\alpha = 1$
Proof of Theorem 1. We start the proof with
$$
\left\| \boldsymbol {x} ^ {t + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} = \left\| \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {*} - \gamma \widehat {\boldsymbol {g}} ^ {t} \right\| ^ {2} = \left\| \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {*} \right\| ^ {2} - 2 \gamma \langle \widehat {\boldsymbol {g}} ^ {t}, \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {*} \rangle + \gamma^ {2} \left\| \widehat {\boldsymbol {g}} ^ {t} \right\| ^ {2}.
$$
Since  $\widehat{\pmb{g}}^t = \widehat{\pmb{g}}^t - F^t + F^t$  one has
$$
\left\| \boldsymbol {x} ^ {t + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} = \left\| \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {*} \right\| ^ {2} - 2 \gamma \langle \widehat {\boldsymbol {g}} ^ {t} - \overline {{\boldsymbol {g}}} ^ {t}, \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {*} \rangle - 2 \gamma \langle \overline {{\boldsymbol {g}}} ^ {t}, \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {*} \rangle + \gamma^ {2} \| \widehat {\boldsymbol {g}} ^ {t} \| ^ {2}.
$$
Applying (11) for  $\langle \widehat{\pmb{g}}^t -\overline{\pmb{g}}^t,\pmb{x}^t -\pmb{x}^*\rangle$  with  $\lambda = \frac{\gamma\mu}{2}$  and (12) for  $\| \widehat{\pmb{g}}^t\|^2 = \left\| \widehat{\pmb{g}}^t -\overline{\pmb{g}}^t +\overline{\pmb{g}}^t\right\|^2$  we derive
$$
\begin{array}{l} \left\| \boldsymbol {x} ^ {t + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \leq \left(1 + \frac {\gamma \mu}{2}\right) \left\| \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {*} \right\| ^ {2} - 2 \gamma \langle \bar {\boldsymbol {g}} ^ {t}, \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {*} \rangle \\ + \frac {2 \gamma}{\mu} \left\| \widehat {\boldsymbol {g}} ^ {t} - \overline {{\boldsymbol {g}}} ^ {t} \right\| ^ {2} + 2 \gamma^ {2} \left\| \widehat {\boldsymbol {g}} ^ {t} - \overline {{\boldsymbol {g}}} ^ {t} \right\| ^ {2} + 2 \gamma^ {2} \left\| \overline {{\boldsymbol {g}}} ^ {t} \right\| ^ {2}. \\ \end{array}
$$
Next by taking an expectation  $\mathbb{E}_{\xi}$  of both sides of the above inequality and rearranging terms obtain
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {\xi}} \left\| \boldsymbol {x} ^ {t + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \leq \left(1 + \frac {\gamma \mu}{2}\right) \left\| \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {*} \right\| ^ {2} - 2 \gamma \langle F (x ^ {t}), \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {*} \rangle \\ + \frac {2 \gamma}{\mu} \mathbb {E} _ {\pmb {\xi}} \left\| \widehat {\pmb {g}} ^ {t} - \overline {{\pmb {g}}} ^ {t} \right\| ^ {2} + 2 \gamma^ {2} \mathbb {E} _ {\pmb {\xi}} \left\| \widehat {\pmb {g}} ^ {t} - \overline {{\pmb {g}}} ^ {t} \right\| ^ {2} + 2 \gamma^ {2} \mathbb {E} _ {\pmb {\xi}} \left\| \overline {{\pmb {g}}} ^ {t} \right\| ^ {2}. \\ \end{array}
$$
Next we use Lemmas C.1 and D.1 to derive
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {\xi}} \left\| \boldsymbol {x} ^ {t + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \leq \left(1 + \frac {\gamma \mu}{2}\right) \left\| \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \left(2 \gamma^ {2} \ell - 2 \gamma\right) \langle F (\boldsymbol {x} ^ {t}), \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {*} \rangle \\ + \frac {2 \gamma^ {2} \sigma^ {2}}{G} + 2 c \delta \rho^ {2} \left(\frac {\gamma}{\mu} + \gamma^ {2}\right), \\ \end{array}
$$
that together with the choice of  $\gamma \leq \frac{1}{2\ell}$  and Assumption (QSM) allows to obtain
$$
\mathbb {E} _ {\boldsymbol {\xi}} \left\| \boldsymbol {x} ^ {t + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \leq \left(1 - \frac {\gamma \mu}{2}\right) \left\| \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \frac {2 \gamma^ {2} \sigma^ {2}}{G} + 2 c \delta \rho^ {2} \left(\frac {\gamma}{\mu} + \gamma^ {2}\right).
$$
Next we take full expectation of both sides and obtain
$$
\mathbb {E} \left\| \boldsymbol {x} ^ {t + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \leq \left(1 - \frac {\gamma \mu}{2}\right) \mathbb {E} \left\| \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \frac {2 \gamma^ {2} \sigma^ {2}}{G} + 2 c \delta \rho^ {2} \left(\frac {\gamma}{\mu} + \gamma^ {2}\right).
$$
The latter implies
$$
\mathbb {E} \left\| \boldsymbol {x} ^ {T} - \boldsymbol {x} ^ {*} \right\| ^ {2} \leq \left(1 - \frac {\gamma \mu}{2}\right) ^ {T} \left\| \boldsymbol {x} ^ {0} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \frac {4 \gamma \sigma^ {2}}{\mu G} + \frac {4 \gamma c \delta \rho^ {2}}{\mu} + \frac {4 c \delta \rho^ {2}}{\mu^ {2}},
$$
where  $\rho$  is bounded by Lemma D.1 with  $\alpha = 1$ .
Corollary 1. Let assumptions of Theorem 1 hold. Then  $\mathbb{E}\left\| \boldsymbol{x}^T - \boldsymbol{x}^*\right\|^2 \leq \varepsilon$  holds after
$$
T \geq \left(4 + \frac {4 \ell}{\mu} + \frac {1}{3 c \delta G}\right) \ln \frac {3 R ^ {2}}{\varepsilon}
$$
iterations of SGDA-RA with  $\gamma = \min \left(\frac{1}{2\ell},\frac{1}{2\mu + \frac{\mu}{6c\delta G}}\right)$  and  $b\geq \frac{72c\delta\sigma^2}{\mu^2\varepsilon}$ .
Proof. If  $\zeta = 0$ ,  $\rho^2 = 24\sigma^2$  the result of Theorem 1 can be simplified as
$$
\mathbb {E} \left\| \boldsymbol {x} ^ {T} - \boldsymbol {x} ^ {*} \right\| ^ {2} \leq \left(1 - \frac {\gamma \mu}{2}\right) ^ {T} \left\| \boldsymbol {x} ^ {0} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \frac {2 \gamma \sigma^ {2}}{\mu G} + \frac {4 8 \gamma c \delta \sigma^ {2}}{\mu} + \frac {2 4 c \delta \sigma^ {2}}{\mu^ {2}}.
$$
Applying Lemma C.3 to the last bound we get the result of the corollary.
D.2 Proofs for SEG-RA
Algorithm 2 SEG-RA
Input: RAGG,  $\gamma$   
1: for  $t = 1, \ldots$  do  
2: for worker  $i \in [n]$  in parallel  
3:  $g_{\xi_i}^t \gets g_i(x^t, \xi_i)$   
4: send  $g_{\xi_i}^t$  if  $i \in \mathcal{G}$ , else send * if Byzantine  
5:  $\widehat{g}_{\xi^t}(x^t) = \mathrm{RAGG}(g_{\xi_1}^t, \ldots, g_{\xi_n}^t)$   
6:  $\widetilde{x}^t \gets x^t - \gamma_1 \widehat{g}_{\xi^t}(x^t)$ . // update params using robust aggregate  
7: for worker  $i \in [n]$  in parallel  
8:  $g_{\eta_i}^t \gets g_i(\widetilde{x}^t, \eta_i)$   
9: send  $g_{\eta_i}^t$  if  $i \in \mathcal{G}$ , else send * if Byzantine  
10:  $\widehat{g}_{\eta^t}(\widetilde{x}^t) = \mathrm{RAGG}(g_{\eta_1}^t, \ldots, g_{\eta_n}^t)$   
11:  $x^{t+1} \gets x^t - \gamma_2 \widehat{g}_{\eta^t}(\widetilde{x}^t)$ . // update params using robust aggregate
To analyze the convergence of SEG introduce the following notation
$$
\bar {g} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) = g _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) = \frac {1}{G} \sum_ {i \in \mathcal {G}} g _ {i} \left(\boldsymbol {x} ^ {k}, \boldsymbol {\xi} _ {i} ^ {k}\right)
$$
$$
\widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) = \operatorname {R A G G} \left(\boldsymbol {g} _ {1} \left(\boldsymbol {x} ^ {k}, \boldsymbol {\xi} _ {1} ^ {k}\right), \dots , \boldsymbol {g} _ {n} \left(\boldsymbol {x} ^ {k}, \boldsymbol {\xi} _ {n} ^ {k}\right)\right),
$$
$$
\widetilde {\boldsymbol {x}} ^ {k} = \boldsymbol {x} ^ {k} - \gamma_ {1} \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}),
$$
$$
\bar {g} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) = \bar {g} _ {\boldsymbol {\eta} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) = \frac {1}{G} \sum_ {i \in \mathcal {G}} g _ {i} \left(\boldsymbol {x} ^ {k}, \boldsymbol {\eta} _ {i} ^ {k}\right)
$$
where  $\pmb{\xi}_i^k$ ,  $i \in \mathcal{G}$  and  $\pmb{\eta}_j^k$ ,  $j \in \mathcal{G}$  are i.i.d. samples satisfying Assumption 1, i.e., due to the independence we have
Corollary 2. Suppose that the operator  $F$  is given in the form (1) and Assumption 1 holds. Then
$$
\mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \| \overline {{\boldsymbol {g}}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) - F (\boldsymbol {x} ^ {k}) \| ^ {2} \right] \leq \frac {\sigma^ {2}}{G}, \tag {27}
$$
$$
\mathbb {E} _ {\boldsymbol {\eta} ^ {k}} \left[ \| \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\tilde {\boldsymbol {x}} ^ {k}\right) - F \left(\tilde {\boldsymbol {x}} ^ {k}\right) \| ^ {2} \right] \leq \frac {\sigma^ {2}}{G}. \tag {28}
$$
D.2.1 Auxiliary results
Lemma D.2. Let Assumptions 2, 3, 4 and Corollary 2 hold. If
$$
\gamma_ {1} \leq \frac {1}{2 L} \tag {29}
$$
for SEG-RA (Algorithm 2), then  $\overline{g}_{\pmb{\eta}^k}(\widetilde{\pmb{x}}^k) = \overline{g}_{\pmb{\eta}^k}\left(\pmb{x}^k - \gamma_1\widehat{g}_{\pmb{\xi}^k}(\pmb{x}^k)\right)$  satisfies the following inequality
$$
\gamma_ {1} ^ {2} \mathbb {E} \left[ \left\| \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2} \mid \boldsymbol {x} ^ {k} \right] \leq 2 \widehat {P} _ {k} + \frac {8 \gamma_ {1} ^ {2} \sigma^ {2}}{G} + 4 \gamma_ {1} ^ {2} c \delta \rho^ {2}, \tag {30}
$$
where  $\widehat{P}_k = \gamma_1\mathbb{E}_{\pmb{\xi}^k,\pmb{\eta}^k}\left[\langle \overline{\pmb{g}}_{\pmb{\eta}^k}(\widetilde{\pmb{x}}^k),\pmb{x}^k -\pmb{x}^*\rangle \right]$  and  $\rho^2 = 24\sigma^2 +12\zeta^2$  by Lemma D.1 with  $\alpha = 1$
Proof. Using the auxiliary iterate  $\widehat{\pmb{x}}^{k + 1} = \pmb{x}^k -\gamma_1\overline{\pmb{g}}_{\pmb{\eta}^k}(\widetilde{\pmb{x}}^k)$ , we get
$$
\begin{array}{l} \left\| \widehat {\boldsymbol {x}} ^ {k + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} = \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - 2 \gamma_ {1} \left\langle \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*}, \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) \right\rangle + \gamma_ {1} ^ {2} \left\| \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) \right\| ^ {2} (31) \\ = \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - 2 \gamma_ {1} \left\langle \boldsymbol {x} ^ {k} - \gamma \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) - \boldsymbol {x} ^ {*}, \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\rangle (32) \\ \left. - 2 \gamma_ {1} ^ {2} \left\langle \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right), \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\rangle + \gamma_ {1} ^ {2} \left\| \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2}. \right. (33) \\ \end{array}
$$
Taking the expectation  $\mathbb{E}_{\pmb{\xi}^k,\pmb{\eta}^k}\left[\cdot \right] = \mathbb{E}\left[\cdot \mid \pmb{x}^k\right]$  conditioned on  $\pmb{x}^k$  from the above identity, using tower property  $\mathbb{E}_{\pmb{\xi}^k,\pmb{\eta}^k}\left[\cdot \right] = \mathbb{E}_{\pmb{\xi}^k}\left[\mathbb{E}_{\pmb{\eta}^k}\left[\cdot \right]\right]$ , and  $\mu$ -quasi strong monotonicity of  $F(x)$ , we derive
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\| \widehat {\boldsymbol {x}} ^ {k + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \right] \\ = \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\| \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) \right\| ^ {2} \right] \\ - 2 \gamma_ {1} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\langle \boldsymbol {x} ^ {k} - \gamma_ {1} \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) - \boldsymbol {x} ^ {*}, \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\rangle \right] \\ - 2 \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\langle \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right), \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\rangle \right] \\ = \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} \\ - 2 \gamma_ {1} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\langle \boldsymbol {x} ^ {k} - \gamma_ {1} \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) - \boldsymbol {x} ^ {*}, F \left(\boldsymbol {x} ^ {k} - \gamma_ {1} \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right)\right) \right\rangle \right] \\ - 2 \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\langle \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right), \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\rangle \right] + \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\| \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {\text {(Q S M), (1 4)}} {\leq} \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \right\| ^ {2} \right] \\ + \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) - \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) \right\| ^ {2} \right]. \\ \end{array}
$$
To upper bound the last term we use simple inequality (16), and apply  $L$ -Lipschitzness of  $F(x)$ :
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\| \widehat {\boldsymbol {x}} ^ {k + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \right] \quad \stackrel {(1 6)} {\leq} \quad \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \right\| ^ {2} \right] \\ + 4 \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\| \bar {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) - \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) \right\| ^ {2} \right] \\ + 4 \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\| F (\boldsymbol {x} ^ {k}) - F \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2} \right] \\ + 4 \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\| \overline {{g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) - F (\boldsymbol {x} ^ {k}) \right\| ^ {2} \right] \\ + 4 \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\| \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\tilde {\boldsymbol {x}} ^ {k}\right) - F \left(\tilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} \leq \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - \gamma_ {1} ^ {2} \left(1 - 4 L ^ {2} \gamma_ {1} ^ {2}\right) \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) \right\| ^ {2} \right] \tag {Lip,(27),(28)} \\ + 4 \gamma_ {1} ^ {2} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\| \bar {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) - \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) \right\| ^ {2} \right] \\ + \frac {4 \gamma_ {1} ^ {2} \sigma^ {2}}{G} + \frac {4 \gamma_ {1} ^ {2} \sigma^ {2}}{G} \\ \end{array}
$$
$$
\begin{array}{l} \leq \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - \gamma_ {1} ^ {2} \left(1 - 4 \gamma_ {1} ^ {2} L ^ {2}\right) \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) \right\| ^ {2} \right] \tag {16} \\ + \frac {8 \gamma_ {1} ^ {2} \sigma^ {2}}{G} + 4 \gamma_ {1} ^ {2} c \delta \rho^ {2} \\ \stackrel {(2 9)} {\leq} \quad \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \frac {8 \gamma_ {1} ^ {2} \sigma^ {2}}{G} + 4 \gamma_ {1} ^ {2} c \delta \rho^ {2}. \\ \end{array}
$$
Finally, we use the above inequality together with (31):
$$
\left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - 2 \widehat {P} _ {k} + \gamma_ {1} ^ {2} \mathbb {E} \left[ \left\| \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) \right\| ^ {2} \mid \boldsymbol {x} ^ {k} \right] \leq \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \frac {8 \gamma_ {1} ^ {2} \sigma^ {2}}{G} + 4 \gamma_ {1} ^ {2} c \delta \rho^ {2},
$$
where  $\widehat{P}_k = \gamma_1\mathbb{E}_{\pmb{\xi}^k,\pmb{\eta}^k}\left[\langle \overline{\pmb{g}}_{\pmb{\eta}^k}(\widetilde{\pmb{x}}^k),\pmb{x}^k -\pmb{x}^*\rangle \right]$ . Rearranging the terms, we obtain (30).
Lemma D.3. Consider SEG-RA (Algorithm 2). Let Assumptions 2, 3, 4 and Corollary 2 hold. If
$$
\gamma_ {1} \leq \frac {1}{2 \mu + 2 L}, \tag {34}
$$
then  $\overline{\mathbf{g}}_{\pmb{\eta}^k}(\widetilde{\pmb{x}}^k) = \overline{\mathbf{g}}_{\pmb{\eta}^k}\left(\pmb{x}^k - \gamma_1 \widehat{\mathbf{g}}_{\pmb{\xi}^k}(\pmb{x}^k)\right)$  satisfies the following inequality
$$
\widehat {P} _ {k} \geq \frac {\mu \gamma_ {1}}{2} \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \frac {\gamma_ {1} ^ {2}}{4} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\| \overline {{g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) \right\| ^ {2} \right] - \frac {8 \gamma_ {1} ^ {2} \sigma^ {2}}{G} - \frac {9 \gamma_ {1} ^ {2} c \delta \rho^ {2}}{2}, \tag {35}
$$
$$
{- \widehat {P} _ {k}} \leq {- \frac {\mu \gamma_ {1}}{2} \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \frac {4 \gamma_ {1} ^ {2} \sigma^ {2}}{G} + 4 \gamma_ {1} ^ {2} c \delta \rho^ {2}}
$$
Proof. Since  $\mathbb{E}_{\pmb{\xi}^k,\pmb{\eta}^k}[\cdot ] = \mathbb{E}[\cdot |\pmb{x}^k ]$  and  $\overline{g}_{\pmb{\eta}^k}(\widetilde{\pmb{x}}^k) = \overline{g}_{\pmb{\eta}^k}\left(\pmb{x}^k -\gamma_1\widehat{g}_{\pmb{\xi}^k}(\pmb{x}^k)\right)$ , we have
$$
- \widehat {P} _ {k}
$$
$$
\begin{array}{l} = - \gamma_ {1} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}, \boldsymbol {\eta} ^ {k}} \left[ \left\langle \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right), \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\rangle \right] \\ = - \gamma_ {1} \mathbb {E} _ {\boldsymbol {\xi} ^ {k}} \left[ \left\langle \mathbb {E} _ {\boldsymbol {\eta} ^ {k}} \left[ \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right], \boldsymbol {x} ^ {k} - \gamma_ {1} \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) - \boldsymbol {x} ^ {*} \right\rangle \right] \\ - \gamma_ {1} ^ {2} \mathbb {E} \left[ \left\langle \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right), \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) \right\rangle \right] \\ \end{array}
$$