SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1:  $\widehat{\pmb{x}}^0 = \pmb{x}^0$
2: for  $t = 1,2,\ldots ,r$  do
3: Run SGDA-CC (Alg. 5) for  $K_{t}$  iterations with stepsize  $\gamma_{t}$ , starting point  $\widehat{\pmb{x}}^{t - 1}$ .
4: Define  $\widehat{\pmb{x}}^t$  as  $\widehat{\pmb{x}}^t = \frac{1}{K_t}\sum_{k = 0}^{K_t}\pmb{x}^{k,t}$ , where  $\pmb{x}^{0,t},\pmb{x}^{1,t},\dots,\pmb{x}^{K_t,t}$  are the iterates produced by SGDA-CC (Alg. 5).
Output:  $\widehat{\pmb{x}}^r$
Theorem (Theorem 5 duplicate). Let Assumptions 1, 4 and 6 hold. Then, after  $r = \left\lceil \log_2 \frac{R^2}{\varepsilon} \right\rceil - 1$  restarts R-SGDA-CC (Algorithm 6) with  $\gamma_t = \min \left\{ \frac{1}{2\ell}, \sqrt{\frac{(n - 2B - m)R^2}{6\sigma^2 2^t K_t}}, \sqrt{\frac{m^2 R^2}{72q\sigma^2 2^t B^2 n^2}} \right\}$  and  $K_t = \left\lceil \max \left\{ \frac{8\ell}{\mu}, \frac{96\sigma^2 2^t}{(n - 2B - m)\mu^2 R^2}, \frac{34n\sigma B \sqrt{q 2^t}}{m\mu R} \right\} \right\rceil$ , where  $R \geq \| x^0 - x^* \|$ , outputs  $\hat{x}^r$  such that  $\mathbb{E}\| \hat{x}^r - x^* \|^2 \leq \varepsilon$ . Moreover, the total number of executed iterations of SGDA-CC is
$$
\sum_ {t = 1} ^ {r} K _ {t} = \mathcal {O} \left(\frac {\ell}{\mu} \log \frac {\mu R _ {0} ^ {2}}{\varepsilon} + \frac {\sigma^ {2}}{(n - 2 B - m) \mu \varepsilon} + \frac {n B \sigma}{m \sqrt {\mu \varepsilon}}\right). \tag {55}
$$
With  $q = 2C^2 +12 + \frac{12}{n - 2B - m}$  and  $C = \mathcal{Q}(1)$  by Lemma E.1.
Proof of Theorem 5.  $\overline{\pmb{x}}^K = \frac{1}{K}\sum_{k = 0}^{K - 1}\pmb {x}^k$
Theorem 9 implies that SGDA-CC with
$$
\gamma = \min \left\{\frac {1}{2 \ell}, \sqrt {\frac {(n - 2 B - m) R _ {0} ^ {2}}{6 \sigma^ {2} K}}, \sqrt {\frac {m ^ {2} R _ {0} ^ {2}}{7 2 \rho^ {2} B ^ {2} n ^ {2}}} \right\}
$$
$$
\mu \mathbb {E} \left[ \left\| \overline {{\boldsymbol {x}}} ^ {K} - \boldsymbol {x} ^ {*} \right\| ^ {2} \right] \leq \frac {2}{R _ {0} ^ {2}} \gamma K
$$
after  $K$  iterations.
After the first restart we have
$$
\mathbb {E} \left[ \left\| \widehat {\pmb {x}} ^ {1} - \pmb {x} ^ {*} \right\| ^ {2} \right] \leq \frac {2 R _ {0} ^ {2}}{\mu \gamma_ {1} K _ {1}} \leq \frac {R _ {0} ^ {2}}{2}.
$$
Next, assume that we have  $\mathbb{E}[\| \widehat{\pmb{x}}^t -\pmb{x}^*\| ^2 ]\leq \frac{R_0^2}{2^t}$  for some  $t\leq r - 1$ . Then, Theorem 9 implies that
$$
\mathbb {E} \Big [ \big \| \widehat {\pmb {x}} ^ {t + 1} - \pmb {x} ^ {*} \big \| ^ {2} \mid \widehat {\pmb {x}} ^ {t} \Big ] \leq \frac {2 \| \widehat {\pmb {x}} ^ {t} - \pmb {x} ^ {*} \| ^ {2}}{\mu \gamma_ {t} K _ {t}}.
$$
Taking the full expectation from the both sides of previous inequality we get
$$
\mathbb {E} \left[ \left\| \widehat {\boldsymbol {x}} ^ {t + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \right] \leq \frac {2 \mathbb {E} [ \| \widehat {\boldsymbol {x}} ^ {t} - \boldsymbol {x} ^ {*} \| ^ {2} ]}{\mu \gamma_ {t} K _ {t}} \leq \frac {2 R _ {0} ^ {2}}{2 ^ {t} \mu \gamma_ {t} K _ {t}} \leq \frac {R _ {0} ^ {2}}{2 ^ {t + 1}}.
$$
Therefore, by mathematical induction we have that for all  $t = 1,\dots ,r$
$$
\mathbb {E} \left[ \| \widehat {\boldsymbol {x}} ^ {t} - \boldsymbol {x} ^ {*} \| ^ {2} \right] \leq \frac {R _ {0} ^ {2}}{2 ^ {t}}.
$$
Then, after  $r = \left\lceil \log_2\frac{R_0^2}{\varepsilon}\right\rceil -1$  restarts of SGDA-CC we have  $\mathbb{E}\left[\| \widehat{\pmb{x}}^r -\pmb{x}^*\| ^2\right]\leq \varepsilon$ . The total number of iterations executed by SGDA-CC is
$$
\begin{array}{l} \sum_ {t = 1} ^ {r} K _ {t} = \mathcal {O} \left(\sum_ {t = 1} ^ {r} \max \left\{\frac {\ell}{\mu}, \frac {\sigma^ {2} 2 ^ {t}}{(n - 2 B - m) \mu^ {2} R _ {0} ^ {2}}, \frac {n B \rho 2 ^ {\frac {t}{2}}}{m \mu R _ {0}} \right\}\right) \\ = \mathcal {O} \left(\frac {\ell}{\mu} r + \frac {\sigma^ {2} 2 ^ {r}}{(n - 2 B - m) \mu^ {2} R _ {0} ^ {2}} + \frac {n B \rho 2 ^ {\frac {r}{2}}}{m \mu R _ {0}}\right) \\ { = } { \mathcal { O } \left( \frac { \ell } { \mu } \log \frac { \mu R _ { 0 } ^ { 2 } } { \varepsilon } + \frac { \sigma ^ { 2 } } { ( n - 2 B - m ) \mu ^ { 2 } R _ { 0 } ^ { 2 } } \cdot \frac { \mu R _ { 0 } ^ { 2 } } { \varepsilon } + \frac { n B \rho } { m \mu R _ { 0 } } \cdot \sqrt { \frac { \mu R _ { 0 } ^ { 2 } } { \varepsilon } } \right) } \\ { = } { \mathcal { O } \left( \frac { \ell } { \mu } \log \frac { \mu R _ { 0 } ^ { 2 } } { \varepsilon } + \frac { \sigma ^ { 2 } } { ( n - 2 B - m ) \mu \varepsilon } + \frac { n B \rho } { m \sqrt { \mu \varepsilon } } \right) . } \\ \end{array}
$$
Corollary 8. Let assumptions of 5 hold. Then  $\mathbb{E}\left[\| \widehat{\pmb{x}}^r -\pmb {x}^*\| ^2\right]\leq \varepsilon$  holds after
$$
\sum_ {t = 1} ^ {r} K _ {t} = \mathcal {O} \left(\frac {\ell}{\mu} \log \frac {\mu R ^ {2}}{\varepsilon} + \frac {\sigma^ {2}}{n \mu \varepsilon} + \frac {n ^ {2} \sigma}{m \sqrt {\mu \varepsilon}}\right) \tag {56}
$$
Proof. Using the definition of  $\rho$  from Lemma E.1 and if  $B \leq \frac{n}{4}$ ,  $m << n$  the bound for  $\sum_{t=1}^{r} K_t$  can be easily derived.
Algorithm 7 SEG-CC
Input: RAGG,  $\gamma$   
1: for  $t = 1, \ldots$  do  
2: for worker  $i \in [n]$  in parallel  
3:  $g_{\xi_i}^t \gets g_i(\boldsymbol{x}^t, \boldsymbol{\xi}_i)$   
4: send  $g_{\xi_i}^t$  if  $i \in \mathcal{G}_t$ , else send * if Byzantine  
5:  $\widehat{g}_{\xi^t}(\boldsymbol{x}^t) = \frac{1}{W_t - \frac{1}{2}} \sum_{i \in W_t - \frac{1}{2}} g_{\xi_i}^t$ ,  $W_{t - \frac{1}{2}} = (\mathcal{G}_t \cup \mathcal{B}_t) \setminus \mathcal{C}_t$   
6: if  $\left|\left\{i \in \mathcal{W}_{t - \frac{1}{2}} \mid \left\| \widehat{g}_{\xi^t}(\boldsymbol{x}^t) - g_{\xi_i^t}^t \right\| \leq C\sigma\right\}\right| \geq \left|W_{t - \frac{1}{2}}\right| / 2$  then  
7:  $\widetilde{\boldsymbol{x}}^t \gets \boldsymbol{x}^t - \gamma_1 \widehat{g}_{\xi^t}(\boldsymbol{x}^t)$   
8: else  
9: recompute  
10: end if  
11:  $\mathcal{C}_{t + \frac{1}{2}}, \mathcal{G}_{t + \frac{1}{2}} \cup \mathcal{B}_{t + \frac{1}{2}} = \text{CheckComputations}(\mathcal{C}_t, \mathcal{G}_t \cup \mathcal{B}_t)$   
12: for worker  $i \in [n]$  in parallel  
13:  $g_{\eta_i}^t \gets g_i(\widetilde{\boldsymbol{x}}^t, \eta_i)$   
14: send  $g_{\eta_i}^t$  if  $i \in \mathcal{G}$ , else send * if Byzantine  
15:  $\widehat{g}_{\eta^t}(\widetilde{\boldsymbol{x}}^t) = \frac{1}{W_t} \sum_{i \in W_t} g_{\eta_i}^t$ ,  $W_t = (\mathcal{G}_{t + \frac{1}{2}} \cup \mathcal{B}_{t + \frac{1}{2}}) \setminus \mathcal{C}_t + \frac{1}{2}$   
16: if  $\left|\left\{i \in \mathcal{W}_t \mid \| \widehat{g}_{\eta^t}(\widetilde{\boldsymbol{x}}^t) - g_{\eta_i}^t \| \leq C\sigma\right\}\right| \geq |\mathcal{W}_t| / 2$  then  
17:  $x^{t + 1} \gets x^t - \gamma_2 \widehat{g}_{\eta^t}(\widetilde{x}^t)$   
18: else  
19: recompute  
20: end if  
21:  $C_{t + 1}, C_{t + 1} \cup B_{t + 1} = \text{CheckComputations}(\mathcal{C}_{t + \frac{1}{2}}, \mathcal{G}_{t + \frac{1}{2}} \cup B_{t + \frac{1}{2}})$
E.3.1 Auxiliary results
Similarly to Section E.1 we state the following. If a Byzantine peer deviates from the protocol at iteration  $k$ , it will be detected with some probability  $p_k$  during the next iteration. One can lower bound this probability as
$$
\sum_ {l = 0} ^ {\infty} \mathbb {E} \left[ \mathbb {1} _ {l} \right] + \sum_ {l = 0} ^ {\infty} \mathbb {E} \left[ \mathbb {1} _ {l - \frac {1}{2}} \right] \leq \frac {n B}{m}. \tag {57}
$$
Lemma E.2. Let Assumption 3 holds. Let Algorithm 7 is run with  $\gamma_1 \leq 1/2L$  and  $\beta = \gamma_2 / \gamma_1 \leq 1/2$ .
Then its iterations satisfy
$$
\begin{array}{l} 2 \gamma_ {2} \left\langle \bar {\mathbf {g}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}), \widetilde {\boldsymbol {x}} ^ {k} - \boldsymbol {u} \right\rangle \leq \left\| \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\| ^ {2} - \left\| \boldsymbol {x} ^ {k + 1} - \boldsymbol {u} \right\| ^ {2} - 2 \gamma_ {2} \left\langle \widehat {\mathbf {g}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) - \bar {\mathbf {g}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}), \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\rangle \\ + 2 \gamma_ {2} ^ {2} \left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) - \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2} + 4 \gamma_ {1} \gamma_ {2} \left\| \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) - F \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2} \\ + 4 \gamma_ {1} \gamma_ {2} \| F (\boldsymbol {x} ^ {k}) - \overline {{\boldsymbol {g}}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \| ^ {2} + 4 \gamma_ {1} \gamma_ {2} \| \overline {{\boldsymbol {g}}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) - \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \| ^ {2}. \\ \end{array}
$$
Proof. Since  $\pmb{x}^{k + 1} = \pmb{x}^k -\gamma_2\widehat{\pmb{g}}_{\pmb{\eta}^k}(\widetilde{\pmb{x}}^k)$ , we have
$$
\begin{array}{l} \left\| \boldsymbol {x} ^ {k + 1} - \boldsymbol {u} \right\| ^ {2} = \left\| \boldsymbol {x} ^ {k} - \gamma_ {2} \widehat {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) - \boldsymbol {u} \right\| ^ {2} \\ = \left\| \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\| ^ {2} - 2 \gamma_ {2} \langle \widehat {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}), \boldsymbol {x} ^ {k} - \boldsymbol {u} \rangle + \gamma_ {2} ^ {2} \left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) \right\| ^ {2}. \\ \end{array}
$$
E.3 Proofs for SEG-CC
Rearranging the terms gives that
$$
\begin{array}{l} 2 \gamma_ {2} \left\langle \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right), \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\rangle = \left\| \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\| ^ {2} - \left\| \boldsymbol {x} ^ {k + 1} - \boldsymbol {u} \right\| ^ {2} \\ - 2 \gamma_ {2} \left\langle \widehat {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) - \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right), \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\rangle + \gamma_ {2} ^ {2} \left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2}. \\ \end{array}
$$
Next we use (14)
$$
\begin{array}{l} 2 \left\langle \bar {g} _ {\eta^ {k}} (\tilde {\boldsymbol {x}} ^ {k}), \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\rangle = 2 \left\langle \bar {g} _ {\eta^ {k}} (\tilde {\boldsymbol {x}} ^ {k}), \boldsymbol {x} ^ {k} - \tilde {\boldsymbol {x}} ^ {k} \right\rangle + 2 \left\langle \bar {g} _ {\eta^ {k}} (\tilde {\boldsymbol {x}} ^ {k}), \tilde {\boldsymbol {x}} ^ {k} - \boldsymbol {u} \right\rangle \\ = 2 \gamma_ {1} \left\langle \overline {{g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right), \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) \right\rangle + 2 \left\langle \overline {{g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right), \widetilde {\boldsymbol {x}} ^ {k} - \boldsymbol {u} \right\rangle \\ \stackrel {(1 4)} {=} - \gamma_ {1} \left(\left\| \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) - \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \right\| ^ {2} - \left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \right\| ^ {2} - \left\| \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) \right\| ^ {2}\right) \\ + 2 \left\langle \overline {{g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right), \widetilde {\boldsymbol {x}} ^ {k} - \boldsymbol {u} \right\rangle \\ \end{array}
$$
and obtain the following
$$
\begin{array}{l} 2 \gamma_ {2} \left\langle \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}), \widetilde {\boldsymbol {x}} ^ {k} - \boldsymbol {u} \right\rangle = \left\| \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\| ^ {2} - \left\| \boldsymbol {x} ^ {k + 1} - \boldsymbol {u} \right\| ^ {2} - 2 \gamma_ {2} \left\langle \widehat {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) - \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}), \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\rangle \\ + \gamma_ {2} ^ {2} \left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2} \\ + \gamma_ {1} \gamma_ {2} \left(\left\| \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\tilde {\boldsymbol {x}} ^ {k}\right) - \hat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) \right\| ^ {2} - \left\| \hat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) \right\| ^ {2} - \left\| \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\tilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2}\right) \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(1 6)} {\leq} \left\| \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\| ^ {2} - \left\| \boldsymbol {x} ^ {k + 1} - \boldsymbol {u} \right\| ^ {2} - 2 \gamma_ {2} \left\langle \widehat {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) - \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}), \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\rangle \\ + 2 \gamma_ {2} ^ {2} \left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) - \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2} + 2 \gamma_ {2} ^ {2} \left\| \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2} \\ + \gamma_ {1} \gamma_ {2} \left(\left\| \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) - \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) \right\| ^ {2} - \left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) \right\| ^ {2} - \left\| \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2}\right). \\ \end{array}
$$
$$
\text {I f} \beta = \gamma_ {2} / \gamma_ {1} \leq 1 / 2
$$
$$
\begin{array}{l} 2 \gamma_ {2} \left\langle \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}), \widetilde {\boldsymbol {x}} ^ {k} - \boldsymbol {u} \right\rangle \leq \left\| \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\| ^ {2} - \left\| \boldsymbol {x} ^ {k + 1} - \boldsymbol {u} \right\| ^ {2} - 2 \gamma_ {2} \left\langle \widehat {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) - \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}), \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\rangle \\ + 2 \gamma_ {2} ^ {2} \left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) - \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2} \\ + \gamma_ {1} \gamma_ {2} \left(\left\| \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) - \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) \right\| ^ {2} - \left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) \right\| ^ {2}\right). \\ \end{array}
$$
Combining the latter with the result of the following chain
$$
\begin{array}{l} \left\| \overline {{g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) - \widehat {g} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) \right\| ^ {2} \stackrel {(1 6)} {\leq} 4 \left\| \overline {{g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) - F \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2} + 4 \left\| F \left(\widetilde {\boldsymbol {x}} ^ {k}\right) - F \left(\boldsymbol {x} ^ {k}\right) \right\| ^ {2} \\ + 4 \left\| F (\boldsymbol {x} ^ {k}) - \overline {{\boldsymbol {g}}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \right\| ^ {2} + 4 \left\| \overline {{\boldsymbol {g}}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) - \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \right\| ^ {2} \\ \stackrel {(3)} {\leq} 4 \left\| \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) - F \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2} + 4 L ^ {2} \left\| \widetilde {\boldsymbol {x}} ^ {k} - \boldsymbol {x} ^ {k} \right\| ^ {2} \\ + 4 \left\| F (\boldsymbol {x} ^ {k}) - \overline {{\boldsymbol {g}}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \right\| ^ {2} + 4 \left\| \overline {{\boldsymbol {g}}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) - \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \right\| ^ {2} \\ = 4 \left\| \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) - F \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2} + 4 \left\| F \left(\boldsymbol {x} ^ {k}\right) - \overline {{\boldsymbol {g}}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) \right\| ^ {2} \\ + 4 \left\| \overline {{g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) - \widehat {g} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \right\| ^ {2} + 4 \gamma_ {1} ^ {2} L ^ {2} \left\| \widehat {g} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \right\| ^ {2} \\ \end{array}
$$
we obtain if  $\gamma_1\leq 1 / 2L$
$$
\begin{array}{l} 2 \gamma_ {2} \left\langle \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}), \widetilde {\boldsymbol {x}} ^ {k} - \boldsymbol {u} \right\rangle \leq \left\| \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\| ^ {2} - \left\| \boldsymbol {x} ^ {k + 1} - \boldsymbol {u} \right\| ^ {2} - 2 \gamma_ {2} \left\langle \widehat {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) - \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}), \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\rangle \\ + 2 \gamma_ {2} ^ {2} \left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) - \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) \right\| ^ {2} + 4 \gamma_ {1} \gamma_ {2} \left\| \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) - F (\widetilde {\boldsymbol {x}} ^ {k}) \right\| ^ {2} \\ + 4 \gamma_ {1} \gamma_ {2} \| F (\boldsymbol {x} ^ {k}) - \bar {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \| ^ {2} + 4 \gamma_ {1} \gamma_ {2} \| \bar {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) - \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \| ^ {2}. \tag {58} \\ \end{array}
$$
E.3.2 Lipschitz Case
Theorem 11. Let Assumptions 1, 3, 5 hold. And let
$$
\gamma_ {1} = \min  \left\{\frac {1}{2 L}, \sqrt {\frac {(n - 2 B - m) R ^ {2}}{1 6 \sigma^ {2} K}}, \sqrt {\frac {m R ^ {2}}{8 \rho^ {2} B n}} \right\}, \tag {59}
$$
$$
\gamma_ {2} = \min  \left\{\frac {1}{4 L}, \sqrt {\frac {m ^ {2} R ^ {2}}{6 4 \rho^ {2} B ^ {2} n ^ {2}}}, \sqrt {\frac {(n - 2 B - m) R ^ {2}}{6 4 \sigma^ {2} K}} \right\}, \tag {60}
$$
where  $\rho^2 = q\sigma^2$  with  $q = 2C^2 +12 + \frac{12}{n - 2B - m}$  and  $C = \mathcal{O}(1)$  by Lemma E.1. Then iterations of SEG-CC (Algorithm 7) satisfy for  $k\geq 1$
$$
\mathbb {E} \left[ R _ {k} ^ {2} \right] \leq 2 R, \tag {61}
$$
$$
\sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\langle F \left(\widetilde {\boldsymbol {x}} ^ {k}\right), \widetilde {\boldsymbol {x}} ^ {k} - \boldsymbol {x} ^ {*} \right\rangle \right] \leq \frac {R ^ {2}}{\gamma_ {2}}.
$$
where  $R_{k} = \| \pmb{x}^{k} - \pmb{x}^{*}\|$  and  $R\geq \| \pmb{x}^0 -\pmb{x}^*\|$
Proof. Substituting  $\mathbf{u} = \mathbf{x}^*$  into the result of Lemma E.2 and taking expectation over  $\eta^k$  one obtains
$$
\begin{array}{l} 2 \gamma_ {2} \mathbb {E} _ {\boldsymbol {\eta} ^ {k}} \left[ \left\langle \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right), \widetilde {\boldsymbol {x}} ^ {k} - \boldsymbol {x} ^ {*} \right\rangle \right] \\ \stackrel {(1 6)} {\leq} \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - \mathbb {E} _ {\boldsymbol {\eta} ^ {k}} \left[ \left\| \boldsymbol {x} ^ {k + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \right] - 2 \gamma_ {2} \mathbb {E} _ {\boldsymbol {\eta} ^ {k}} \left[ \left\langle \widehat {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) - \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}), \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\rangle \right] \\ + \gamma_ {1} \gamma_ {2} 4 \mathbb {E} _ {\boldsymbol {\eta} ^ {k}} \left[ \left\| \bar {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\tilde {\boldsymbol {x}} ^ {k}\right) - F \left(\tilde {\boldsymbol {x}} ^ {k}\right) \right\| ^ {2} \right] + \gamma_ {1} \gamma_ {2} 4 \left\| F \left(\boldsymbol {x} ^ {k}\right) - \bar {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {k}} \left(\boldsymbol {x} ^ {k}\right) \right\| ^ {2} \\ + \gamma_ {1} \gamma_ {2} 4 \left\| \left| \overline {{g}} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) - \widehat {g} _ {\boldsymbol {\xi} ^ {k}} (\boldsymbol {x} ^ {k}) \right| \right\| ^ {2} + 2 \gamma_ {2} ^ {2} \mathbb {E} _ {\boldsymbol {\eta} ^ {k}} \left[ \left\| \widehat {g} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) - \overline {{g}} _ {\boldsymbol {\eta} ^ {k}} (\widetilde {\boldsymbol {x}} ^ {k}) \right\| ^ {2} \right] \\ \end{array}
$$
To estimate the inner product in the right-hand side we apply Cauchy-Schwarz inequality:
$$
\begin{array}{l} - 2 \gamma_ {2} \mathbb {E} _ {\boldsymbol {\eta} ^ {k}} \left[ \left\langle \widehat {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) - \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right), \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\rangle \right] \\ \leq 2 \gamma_ {2} \mathbb {E} _ {\boldsymbol {\eta} ^ {k}} \left[ \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| \left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) - \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta} ^ {k}} \left(\widetilde {\boldsymbol {x}} ^ {k}\right) \right\| \right] \\ \end{array}
$$
$$
\begin{array}{r l} \overset {L e m m a E. 1} {\leq} & 2 \gamma_ {2} \rho \big \| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \big \| \mathbb {1} _ {k}. \end{array}
$$
Then Assumption 6 implies