SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \alpha \gamma \sum_ {j = 0} ^ {t} (1 - \alpha) ^ {t - j} \langle F ^ {j}, \boldsymbol {x} ^ {j} - \boldsymbol {x} ^ {*} \rangle \\ \leq \left(1 - \frac {\mu \gamma \alpha}{2}\right) \left\| \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {*} \right\| ^ {2} - \mathbb {E} _ {\boldsymbol {\xi}} \left\| \boldsymbol {x} ^ {t + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \\ + \frac {2 \gamma}{\mu \alpha} \mathbb {E} _ {\boldsymbol {\xi}} \left\| \widehat {\boldsymbol {m}} ^ {t} - \overline {{\boldsymbol {m}}} ^ {t} \right\| ^ {2} + 4 \gamma^ {2} \sum_ {j = 1} ^ {t} (1 - \alpha) ^ {t - j} \mathbb {E} _ {\boldsymbol {\xi}} \left\| \widehat {\boldsymbol {m}} ^ {j} - \overline {{\boldsymbol {m}}} ^ {j} \right\| ^ {2} \\ + 3 \gamma^ {2} \alpha \ell \sum_ {j = 1} ^ {t} (1 - \alpha) ^ {t - j} \sum_ {i = 0} ^ {j} (1 - \alpha) ^ {j - i} \langle F (\boldsymbol {x} ^ {i}), \boldsymbol {x} ^ {i} - \boldsymbol {x} ^ {*} \rangle \\ + \frac {3 \gamma^ {2} \alpha \sigma^ {2}}{G} \sum_ {j = 1} ^ {t} (1 - \alpha) ^ {t - j} \sum_ {i = 0} ^ {j} (1 - \alpha) ^ {j - i}. \\ \end{array}
$$
$$
\begin{array}{l} \alpha \gamma \sum_ {j = 0} ^ {t} (1 - \alpha) ^ {t - j} \mathbb {E} \langle F ^ {j}, \boldsymbol {x} ^ {j} - \boldsymbol {x} ^ {*} \rangle \\ \leq \left(1 - \frac {\mu \gamma \alpha}{2}\right) \mathbb {E} \left\| \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {*} \right\| ^ {2} - \mathbb {E} \left\| \boldsymbol {x} ^ {t + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \\ + \frac {2 \gamma}{\mu \alpha} \mathbb {E} \left\| \widehat {\boldsymbol {m}} ^ {t} - \overline {{\boldsymbol {m}}} ^ {t} \right\| ^ {2} + 4 \gamma^ {2} \sum_ {j = 1} ^ {t} (1 - \alpha) ^ {t - j} \mathbb {E} \left\| \widehat {\boldsymbol {m}} ^ {j} - \overline {{\boldsymbol {m}}} ^ {j} \right\| ^ {2} \\ + 3 \gamma^ {2} \alpha \ell \sum_ {j = 1} ^ {t} (1 - \alpha) ^ {t - j} \sum_ {i = 0} ^ {j} (1 - \alpha) ^ {j - i} \mathbb {E} \langle F (\boldsymbol {x} ^ {i}), \boldsymbol {x} ^ {i} - \boldsymbol {x} ^ {*} \rangle \\ + \frac {3 \gamma^ {2} \sigma^ {2}}{\alpha G}. \\ \end{array}
$$
Introducing the following notation
$$
\boldsymbol {Z} ^ {t} = \sum_ {j = 0} ^ {t} (1 - \alpha) ^ {t - j} \mathbb {E} \langle F ^ {j}, \boldsymbol {x} ^ {j} - \boldsymbol {x} ^ {*} \rangle
$$
and using that  $\mathbb{E}\left\| \widehat{\boldsymbol{m}}^t - \overline{\boldsymbol{m}}^t \right\|^2 \leq c\delta \rho^2$ , where  $\rho$  is given by (26) we have
$$
\begin{array}{l} \alpha \gamma \boldsymbol {Z} ^ {t} \leq \left(1 - \frac {\mu \gamma \alpha}{2}\right) \mathbb {E} \left\| \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {*} \right\| ^ {2} - \mathbb {E} \left\| \boldsymbol {x} ^ {t + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \frac {2 \gamma c \delta \rho^ {2}}{\mu \alpha} \\ + \frac {4 \gamma^ {2} c \delta \rho^ {2}}{\alpha} + 3 \gamma^ {2} \alpha \ell \sum_ {j = 1} ^ {t} (1 - \alpha) ^ {t - j} \boldsymbol {Z} ^ {j} + \frac {3 \gamma^ {2} \sigma^ {2}}{\alpha G}. \\ \end{array}
$$
Next we sum the above inequality  $T$  times with weights  $w_{t} = \left(1 - \frac{\mu\gamma\alpha}{2}\right)^{-t - 1}$  where  $W_{T} = \sum_{t = 0}^{T}w_{t}$
$$
\begin{array}{l} \alpha \gamma \sum_ {t = 0} ^ {T} w _ {t} \boldsymbol {Z} ^ {t} \leq \left(1 - \frac {\mu \gamma \alpha}{2}\right) \sum_ {t = 0} ^ {T} w _ {t} \mathbb {E} \| \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {*} \| ^ {2} - \sum_ {t = 0} ^ {T} w _ {t} \mathbb {E} \| \boldsymbol {x} ^ {t + 1} - \boldsymbol {x} ^ {*} \| ^ {2} \\ + 3 \gamma^ {2} \alpha \ell \sum_ {t = 0} ^ {T} w _ {t} \sum_ {j = 0} ^ {t} (1 - \alpha) ^ {t - j} Z ^ {j} \\ + W _ {T} \left(\frac {2 \gamma c \delta \rho^ {2}}{\mu \alpha} + \frac {4 \gamma^ {2} c \delta \rho^ {2}}{\alpha} + \frac {3 \gamma^ {2} \sigma^ {2}}{\alpha G}\right). \\ \end{array}
$$
Since  $\left(1 - \frac{\mu\gamma\alpha}{2}\right)w_{t} = w_{t - 1}$
$$
\begin{array}{l} \alpha \gamma \sum_ {t = 0} ^ {T} w _ {t} \boldsymbol {Z} ^ {t} \leq \left\| \boldsymbol {x} ^ {0} - \boldsymbol {x} ^ {*} \right\| ^ {2} + 3 \gamma^ {2} \alpha \ell \sum_ {t = 0} ^ {T} w _ {t} \sum_ {j = 0} ^ {t} (1 - \alpha) ^ {t - j} \boldsymbol {Z} ^ {j} \\ + W _ {T} \left(\frac {2 \gamma c \delta \rho^ {2}}{\mu \alpha} + \frac {4 \gamma^ {2} c \delta \rho^ {2}}{\alpha} + \frac {3 \gamma^ {2} \sigma^ {2}}{\alpha G}\right). \\ \end{array}
$$
If  $\gamma \leq \frac{1}{\mu}$  we have
$$
w _ {t} = \left(1 - \frac {\mu \gamma \alpha}{2}\right) ^ {- t + i} w _ {i} \leq \left(1 + \frac {\mu \gamma \alpha}{2}\right) ^ {t - i} w _ {i} \leq \left(1 + \frac {\alpha}{2}\right) ^ {t - i} w _ {i}.
$$
$$
\begin{array}{l} \alpha \gamma \sum_ {t = 0} ^ {T} w _ {t} \boldsymbol {Z} ^ {t} \leq \left\| \boldsymbol {x} ^ {0} - \boldsymbol {x} ^ {*} \right\| ^ {2} \\ + 3 \gamma^ {2} \alpha \ell \sum_ {t = 0} ^ {T} \sum_ {j = 0} ^ {t} \left(1 + \frac {\alpha}{2}\right) ^ {t - j} (1 - \alpha) ^ {t - j} w _ {j} \boldsymbol {Z} ^ {j} \\ + W _ {T} \left(\frac {2 \gamma c \delta \rho^ {2}}{\mu \alpha} + \frac {4 \gamma^ {2} c \delta \rho^ {2}}{\alpha} + \frac {3 \gamma^ {2} \sigma^ {2}}{\alpha G}\right) \\ \leq \left\| \boldsymbol {x} ^ {0} - \boldsymbol {x} ^ {*} \right\| ^ {2} + 3 \gamma^ {2} \alpha \ell \sum_ {t = 0} ^ {T} \sum_ {j = 0} ^ {t} \left(1 - \frac {\alpha}{2}\right) ^ {t - j} w _ {j} \boldsymbol {Z} ^ {j} \\ + W _ {T} \left(\frac {2 \gamma c \delta \rho^ {2}}{\mu \alpha} + \frac {4 \gamma^ {2} c \delta \rho^ {2}}{\alpha} + \frac {3 \gamma^ {2} \sigma^ {2}}{\alpha G}\right) \\ \leq \left\| \boldsymbol {x} ^ {0} - \boldsymbol {x} ^ {*} \right\| ^ {2} \\ + 3 \gamma^ {2} \alpha \ell \left(\sum_ {t = 0} ^ {T} w _ {t} \boldsymbol {Z} ^ {t}\right) \left(\sum_ {t = 0} ^ {\infty} \left(1 - \frac {\alpha}{2}\right) ^ {t}\right) \\ + W _ {T} \left(\frac {2 \gamma c \delta \rho^ {2}}{\mu \alpha} + \frac {4 \gamma^ {2} c \delta \rho^ {2}}{\alpha} + \frac {3 \gamma^ {2} \sigma^ {2}}{\alpha G}\right) \\ \leq \left. \left\| \boldsymbol {x} ^ {0} - \boldsymbol {x} ^ {*} \right\| ^ {2} + 6 \gamma^ {2} \ell \left(\sum_ {t = 0} ^ {T} w _ {t} \boldsymbol {Z} ^ {t}\right) \right. \\ + W _ {T} \left(\frac {2 \gamma c \delta \rho^ {2}}{\mu \alpha} + \frac {4 \gamma^ {2} c \delta \rho^ {2}}{\alpha} + \frac {3 \gamma^ {2} \sigma^ {2}}{\alpha G}\right). \\ \end{array}
$$
If  $\gamma \leq \frac{\alpha}{12\ell}$  then the following is true
$$
\frac {\alpha \gamma}{2} \sum_ {t = 0} ^ {T} w _ {t} \mathbf {Z} ^ {t} \leq \left\| \mathbf {x} ^ {0} - \mathbf {x} ^ {*} \right\| ^ {2} + W _ {T} \left(\frac {2 \gamma c \delta \rho^ {2}}{\mu \alpha} + \frac {4 \gamma^ {2} c \delta \rho^ {2}}{\alpha} + \frac {3 \gamma^ {2} \sigma^ {2}}{\alpha G}\right). \tag {40}
$$
Using the notations for  $Z^t$  and (QSM) we have
$$
\boldsymbol {Z} ^ {t} = \sum_ {j = 0} ^ {t} (1 - \alpha) ^ {t - j} \mathbb {E} \langle F ^ {j}, \boldsymbol {x} ^ {j} - \boldsymbol {x} ^ {*} \rangle \geq \mu \sum_ {j = 0} ^ {t} (1 - \alpha) ^ {t - j} \mathbb {E} \| \boldsymbol {x} ^ {j} - \boldsymbol {x} ^ {*} \|.
$$
and consequently by Jensen's inequality
$$
\boldsymbol {Z} ^ {t} \geq \mu \sum_ {j = 0} ^ {t} (1 - \alpha) ^ {t - j} \mathbb {E} \| \boldsymbol {x} ^ {j} - \boldsymbol {x} ^ {*} \| \geq \mu \frac {1 - (1 - \alpha) ^ {t + 1}}{\alpha} \mathbb {E} \left\| \boldsymbol {x} ^ {*} - \sum_ {j = 0} ^ {t} \frac {\alpha (1 - \alpha) ^ {t - j}}{1 - (1 - \alpha) ^ {t + 1}} \boldsymbol {x} ^ {j} \right\|.
$$
With the definition  $\widehat{\pmb{x}}^t = \frac{\alpha}{1 - (1 - \alpha)^{t + 1}}\sum_{j = 0}^{t}(1 - \alpha)^{t - j}\pmb{x}^j$  then the above implies that
$$
\boldsymbol {Z} ^ {t} \geq \mu \mathbb {E} \| \widehat {\boldsymbol {x}} ^ {t} - \boldsymbol {x} ^ {*} \|,
$$
that together with (40) gives
$$
\frac {\alpha \gamma \mu}{2} \sum_ {t = 0} ^ {T} w _ {t} \mathbb {E} \| \widehat {\boldsymbol {x}} ^ {t} - \boldsymbol {x} ^ {*} \| \leq \| \boldsymbol {x} ^ {0} - \boldsymbol {x} ^ {*} \| ^ {2} + W _ {T} \left(\frac {2 \gamma c \delta \rho^ {2}}{\mu \alpha} + \frac {4 \gamma^ {2} c \delta \rho^ {2}}{\alpha} + \frac {3 \gamma^ {2} \sigma^ {2}}{\alpha G}\right). \tag {41}
$$
Applying the Jensen inequality again with  $\overline{\pmb{x}}^T = \frac{1}{W_T}\sum_{t = 0}^{T}w_t\widehat{\pmb{x}}^t$  we derive the final result
$$
\mathbb {E} \left[ \left\| \overline {{\boldsymbol {x}}} ^ {T} - \boldsymbol {x} ^ {*} \right\| ^ {2} \right] \leq \frac {2 \left\| \boldsymbol {x} ^ {0} - \boldsymbol {x} ^ {*} \right\| ^ {2}}{\mu \gamma \alpha W _ {T}} + \frac {4 c \delta \rho^ {2}}{\mu^ {2} \alpha^ {2}} + \frac {8 \gamma c \delta \rho^ {2}}{\mu \alpha^ {2}} + \frac {6 \gamma \sigma^ {2}}{\mu \alpha^ {2} G}, \tag {42}
$$
together with the bound on  $\rho$  given by Lemma D.1.
Corollary 4. Let assumptions of Theorem 3 hold. Then  $\mathbb{E}\left\| \overline{\boldsymbol{x}}^T - \boldsymbol{x}^*\right\|^2 \leq \varepsilon$  holds after
$$
T \geq \frac {1}{\alpha} \left(4 + \frac {2 4 \ell}{\mu \alpha} + \frac {1}{8 c \delta G}\right) \ln \frac {3 R ^ {2}}{\varepsilon}
$$
iterations of M-SGDA-RA with  $\gamma = \min \left(\frac{\alpha}{12\ell},\frac{1}{2\mu + \frac{\mu}{16c\delta G}}\right)$ .
Proof. If  $\zeta = 0$ ,  $\rho^2 = 24\sigma^2$  the result of Theorem 3 can be simplified as
$$
\mathbb {E} \left[ \left\| \overline {{\boldsymbol {x}}} ^ {T} - \boldsymbol {x} ^ {*} \right\| ^ {2} \right] \leq \frac {2 \left\| \boldsymbol {x} ^ {0} - \boldsymbol {x} ^ {*} \right\| ^ {2}}{\mu \gamma \alpha W _ {T}} + \frac {4 \cdot 2 4 c \delta \sigma^ {2}}{\mu^ {2} \alpha^ {2}} + \frac {8 \cdot 2 4 \gamma c \delta \sigma^ {2}}{\mu \alpha^ {2}} + \frac {6 \gamma \sigma^ {2}}{\mu \alpha^ {2} G}.
$$
Since  $\frac{2}{\mu\gamma\alpha W_T} \leq \left(1 - \frac{\mu\gamma\alpha}{2}\right)^{T + 1}$  we can apply Lemma C.3 and get the result of the corollary.
E Methods with random check of computations
We replace  $(\delta_{\mathrm{max}}, c)$ -RAGG with the simple mean, but introduce additional verification that have to be passed to accept the mean. The advantage of such aggregation that it coincides with "good" mean if there are no peers violating the protocol But if there is at least one peer violating the protocol at iteration  $t$  we can bound the variance similar to Lemma D.1.
Algorithm 4 CheckComputations
Input:  $t, \mathcal{G}_t \cup \mathcal{B}_t, \mathcal{C}_t$ , Banned  $t = \emptyset$
1:  $\mathcal{C}_{t + 1} = \{c_1^{t + 1},\ldots ,c_m^{t + 1}\}$ ,  $\mathcal{C}_{t + 1}\subset (\mathcal{C}_t\cup \mathcal{B}_t)\setminus \mathcal{C}_t$  and  $\mathcal{U}_{t + 1} = \{u_1^{t + 1},\dots ,u_m^{t + 1}\}$ ,  $\mathcal{U}_{t + 1}\subset (\mathcal{C}_t\cup \mathcal{B}_t)\setminus \mathcal{C}_t$ , where  $2m$  workers  $c_{1}^{t + 1},\ldots ,c_{m}^{t + 1},u_{1}^{t + 1},\ldots ,u_{m}^{t + 1}$  are chosen uniformly at random without replacement.
2: for  $i = 1, \dots, m$  in parallel  $c_i^{t+1}$  checks computations of  $u_i^{t+1}$  during the next iteration
3:  $c_{i}^{t + 1}$  receives a query to recalculate  $\pmb {g}(\pmb{x}^t,\pmb{\xi}_{u_i^{t + 1}}^t)$
4:  $c_{i}^{t + 1}$  sends the recalculated  $\pmb {g}(\pmb{x}^t,\pmb{\xi}_{u_i^{t + 1}}^t)$
5: for  $i = 1,\dots,m$  do
6: if  $g(\pmb{x}^t, \pmb{\xi}_{u_i^t}) \neq g_{u_i^t}^t$  then
7: Banned $_t =$  Banned $_t \cup \{u_i^t, c_i^t\}$ .
8: end if
Output:  $\mathcal{C}_{t + 1},\mathcal{G}_t\cup \mathcal{B}_t\setminus \mathsf{Banned}_t$
Lemma E.1. Let Assumption 2 is satisfied with  $\zeta = 0$ . Then the error between the ideal average  $\overline{\pmb{g}}^t$  and the average with the recomputation rule  $\pmb{g}^t$  can be bounded as
$$
\mathbb {E} _ {\boldsymbol {\xi}} \left\| \widehat {\boldsymbol {g}} ^ {t} - \overline {{\boldsymbol {g}}} ^ {t} \right\| ^ {2} \leq \rho^ {2} \mathbb {1} _ {t},
$$
where  $\rho^2 = q\sigma^2$  with  $q = 2C^2 +12 + \frac{12}{n - 2B - m}$  and  $C = \mathcal{O}(1)$ .
Proof of Lemma E.1. Denote the set  $\widetilde{\mathcal{G}}$  in the following way  $\widetilde{\mathcal{G}} = \{i\in \mathcal{G}_t\setminus \mathcal{C}_t:\| \widehat{\pmb{g}}^t -\pmb {g}_i^t\| \leq C\sigma \}$
$$
\widehat {\boldsymbol {g}} ^ {t} = \left\{ \begin{array}{l l} \frac {1}{n} \sum_ {i = 1} ^ {n} \boldsymbol {g} _ {i} ^ {t}, & \text {i f n u m b e r o f w o r k e r s > \frac {n}{2}}, \\ \mathbf {r e c o m p u t e}, & \text {o t h e r w i s e}. \end{array} \right.
$$
So that we have
$$
\begin{array}{l} \left\| \widehat {\boldsymbol {g}} ^ {t} - \overline {{\boldsymbol {g}}} ^ {t} \right\| ^ {2} = \left\| \widehat {\boldsymbol {g}} ^ {t} - \frac {1}{| \widetilde {\mathcal {G}} |} \sum_ {i \in \widetilde {\mathcal {G}}} \boldsymbol {g} _ {i} ^ {t} + \frac {1}{| \widetilde {\mathcal {G}} |} \sum_ {i \in \widetilde {\mathcal {G}}} \boldsymbol {g} _ {i} ^ {t} - \overline {{\boldsymbol {g}}} ^ {t} \right\| ^ {2} \\ \leq 2 \left\| \widehat {\boldsymbol {g}} ^ {t} - \frac {1}{| \widetilde {\mathcal {Q}} |} \sum_ {i \in \widetilde {\mathcal {G}}} \boldsymbol {g} _ {i} ^ {t} \right\| ^ {2} + 2 \left\| \frac {1}{| \widetilde {\mathcal {Q}} |} \sum_ {i \in \widetilde {\mathcal {G}}} \boldsymbol {g} _ {i} ^ {t} - \overline {{\boldsymbol {g}}} ^ {t} \right\| ^ {2} \\ \leq 2 C ^ {2} \sigma^ {2} + 2 \left\| \frac {1}{| \widetilde {\mathcal {G}} |} \sum_ {i \in \widetilde {\mathcal {G}}} \boldsymbol {g} _ {i} ^ {t} - \overline {{\boldsymbol {g}}} ^ {t} \right\| ^ {2} \\ \end{array}
$$
If  $\delta \leq 1/4$  then an acceptance of  $\widehat{\pmb{g}}^t$  implies that  $|\widetilde{\mathcal{G}}| > n/4$  and  $|\widetilde{\mathcal{G}}| > |\mathcal{G}_t \backslash \mathcal{C}_t| / 3$ .
$$
\begin{array}{l} \left\| \frac {1}{| \widetilde {\mathcal {G}} |} \sum_ {i \in \widetilde {\mathcal {G}}} \boldsymbol {g} _ {i} ^ {t} - \overline {{\boldsymbol {g}}} ^ {t} \right\| ^ {2} \leq \frac {1}{| \widetilde {\mathcal {G}} |} \sum_ {i \in \widetilde {\mathcal {G}}} \| \boldsymbol {g} _ {i} ^ {t} - \overline {{\boldsymbol {g}}} ^ {t} \| ^ {2} \leq \frac {1}{| \widetilde {\mathcal {G}} |} \sum_ {i \in \mathcal {G} _ {t} \backslash \mathcal {C} _ {t}} \| \boldsymbol {g} _ {i} ^ {t} - \overline {{\boldsymbol {g}}} ^ {t} \| ^ {2} \\ \leq \frac {3}{| \mathcal {G} _ {t} \setminus \mathcal {C} _ {t} |} \sum_ {i \in \mathcal {G} _ {t} \setminus \mathcal {C} _ {t}} \| \boldsymbol {g} _ {i} ^ {t} - \overline {{\boldsymbol {g}}} ^ {t} \| ^ {2} \\ \end{array}
$$
Bringing the above results together gives that
$$
\mathbb {E} \big \| \widehat {\pmb {g}} ^ {t} - \overline {{\pmb {g}}} ^ {t} \big \| ^ {2} \leq 2 C ^ {2} \sigma^ {2} + \frac {6}{| \mathcal {G} _ {t} \setminus \mathcal {C} _ {t} |} \sum_ {i \in \mathcal {G}} \mathbb {E} \big \| \pmb {g} _ {i} ^ {t} - \overline {{\pmb {g}}} ^ {t} \big \| ^ {2}
$$
Since checks of computations are only possible in homogeneous case  $(\zeta = 0)$  then  $\mathbb{E}\| \pmb {g}_i^t -F^t\| ^2 = \sigma^2$  and
$$
\left. \mathbb {E} _ {\boldsymbol {\xi}} \left\| \boldsymbol {g} _ {i} ^ {t} - \overline {{\boldsymbol {g}}} ^ {t} \right\| ^ {2} \leq 2 \mathbb {E} _ {\boldsymbol {\xi}} \left\| \boldsymbol {g} _ {i} ^ {t} - F ^ {t} \right\| ^ {2} + 2 \mathbb {E} _ {\boldsymbol {\xi}} \left\| F ^ {t} - \overline {{\boldsymbol {g}}} ^ {t} \right\| ^ {2} \leq 2 \sigma^ {2} + \frac {2 \sigma^ {2}}{| \mathcal {G} |}. \right. \tag {43}
$$
Since  $|\mathcal{G}_t\setminus \mathcal{C}_t| > n - 2B - m$
$$
\mathbb {E} _ {\pmb {\xi}} \left\| \widehat {\pmb {g}} ^ {t} - \overline {{\pmb {g}}} ^ {t} \right\| ^ {2} \leq 2 C ^ {2} \sigma^ {2} + 1 2 \sigma^ {2} + \frac {1 2 \sigma^ {2}}{| \mathcal {G} _ {t} \setminus \mathcal {C} _ {t} |} \leq 2 C ^ {2} \sigma^ {2} + 1 2 \sigma^ {2} + \frac {1 2 \sigma^ {2}}{n - 2 B - m}.
$$
E.1 Proofs for SGDA-CC
Algorithm 5 SGDA-CC
Input:  $\gamma$   
1:  $\mathcal{C}_0 = \emptyset$   
2: for  $t = 1, \ldots$  do  
3: for worker  $i \in (\mathcal{G}_t \cup \mathcal{B}_t) \setminus \mathcal{C}_t$  in parallel  
4: send  $\boldsymbol{g}_i^t = \begin{cases} \boldsymbol{g}_i(\boldsymbol{x}^t, \boldsymbol{\xi}_i), & \text{if } i \in \mathcal{G}_t \setminus \mathcal{C}_t, \\ *, & \text{if } i \in \mathcal{B}_t \setminus \mathcal{C}_t, \end{cases}$   
5:  $\widehat{\boldsymbol{g}}^t = \frac{1}{\mathcal{W}_t} \sum_{i \in \mathcal{W}_t} \boldsymbol{g}_i^t$ ,  $\mathcal{W}_t = (\mathcal{G}_t \cup \mathcal{B}_t) \setminus \mathcal{C}_t$   
6: if  $|\{i \in \mathcal{W}_t | \| \widehat{\boldsymbol{g}}^t - \boldsymbol{g}_i^t \| \leq C\sigma\}| \geq |\mathcal{W}_t|/2$  then  
7:  $\boldsymbol{x}^{t+1} \gets \boldsymbol{x}^t - \gamma \widehat{\boldsymbol{g}}^t$ .  
9: else  
10: recompute  
11: end if  
12:  $\mathcal{C}_{t+1}, \mathcal{G}_{t+1} \cup \mathcal{B}_{t+1} = \text{CheckComputations}(\mathcal{C}_t, \mathcal{G}_t \cup \mathcal{B}_t)$
E.1.1 Star Co-coercive Case
Next we provide convergence guarantees for SGDA-CC (Algorithm 5) under Assumption 6.
Theorem 9. Let Assumptions 1 and 6 hold. Next, assume that
$$
\gamma = \min  \left\{\frac {1}{2 \ell}, \sqrt {\frac {(n - 2 B - m) R ^ {2}}{6 \sigma^ {2} K}}, \sqrt {\frac {m ^ {2} R ^ {2}}{7 2 \rho^ {2} B ^ {2} n ^ {2}}} \right\} \tag {44}
$$
where  $\rho^2 = q\sigma^2$  with  $q = 2C^2 +12 + \frac{12}{n - 2B - m}$  and  $C = \mathcal{O}(1)$  by Lemma E.1 and  $R\geq \| \pmb{x}^0 -\pmb{x}^*\|$ . Then after  $K$  iterations of SGDA-CC (Algorithm 5) it outputs  $\pmb{x}^T$  such that
$$
\sum_ {k = 0} ^ {K - 1} \mathbb {E} \left\| F (\boldsymbol {x} ^ {k}) \right\| \leq \ell \sum_ {k = 0} ^ {K - 1} \mathbb {E} [ \langle F (\boldsymbol {x} ^ {k}), \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \rangle ] \leq \frac {2 \ell R ^ {2}}{\gamma}.
$$
Proof. Since  $|\mathcal{G}_t \setminus \mathcal{C}_t| \geq n - 2B - m$  one can derive using the results of Lemmas C.1 and E.1 that
$$
\begin{array}{l} \mathbb {E} \left[ \| \boldsymbol {x} ^ {k + 1} - \boldsymbol {x} ^ {*} \| ^ {2} \mid \boldsymbol {x} ^ {k} \right] = \mathbb {E} \left[ \| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} - \gamma \widehat {\boldsymbol {g}} ^ {k} \| ^ {2} \mid \boldsymbol {x} ^ {k} \right] \\ = \| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \| ^ {2} - 2 \gamma \mathbb {E} \left[ \langle \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*}, \widehat {\boldsymbol {g}} ^ {k} \rangle | \boldsymbol {x} ^ {k} \right] + \gamma^ {2} \mathbb {E} \left[ \| \widehat {\boldsymbol {g}} ^ {k} \| ^ {2} | \boldsymbol {x} ^ {k} \right] \\ \leq \| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \| ^ {2} - 2 \gamma \langle \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*}, F (\boldsymbol {x} ^ {k}) \rangle + 2 \ell \gamma^ {2} \langle \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*}, F (\boldsymbol {x} ^ {k}) \rangle \\ - 2 \gamma \mathbb {E} \left[ \langle \pmb {x} ^ {k} - \pmb {x} ^ {*}, \widehat {\pmb {g}} ^ {k} - \overline {{\pmb {g}}} ^ {k} \rangle \mid \pmb {x} ^ {k} \right] + 2 \gamma^ {2} \rho^ {2} \mathbb {1} _ {k} + \frac {2 \gamma^ {2} \sigma^ {2}}{n - 2 B - m}, \\ \end{array}
$$
where  $\mathbb{1}_k$  is an indicator function of the event that at least 1 Byzantine peer violates the protocol at iteration  $k$ .
To estimate the inner product in the right-hand side we apply Cauchy-Schwarz inequality and then Lemma E.1
$$
\begin{array}{l} - 2 \gamma \mathbb {E} \left[ \langle \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*}, \widehat {\boldsymbol {g}} ^ {k} - \overline {{\boldsymbol {g}}} ^ {k} \rangle \mid \boldsymbol {x} ^ {k} \right] \leq 2 \gamma \| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \| \mathbb {E} \left[ \| \widehat {\boldsymbol {g}} ^ {k} - \overline {{\boldsymbol {g}}} ^ {k} \| \mid \boldsymbol {x} ^ {k} \right] \\ \leq 2 \gamma \| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \| \sqrt {\mathbb {E} \left[ \| \hat {\boldsymbol {g}} ^ {k} - \overline {{\boldsymbol {g}}} ^ {k} \| ^ {2} \mid \boldsymbol {x} ^ {k} \right]} \\ \leq 2 \gamma \rho \| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \| _ {\mathbb {1} _ {k}}. \\ \end{array}
$$
Since  $\gamma \leq \frac{1}{2\ell}$  the above results imply
$$
\begin{array}{l} \gamma \langle \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*}, F (\boldsymbol {x} ^ {k}) \rangle \leq \| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \| ^ {2} - \mathbb {E} \left[ \| \boldsymbol {x} ^ {k + 1} - \boldsymbol {x} ^ {*} \| ^ {2} \mid \boldsymbol {x} ^ {k} \right] \\ - 2 \gamma \mathbb {E} \left[ \langle \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*}, \widehat {\boldsymbol {g}} ^ {k} - \overline {{\boldsymbol {g}}} ^ {k} \rangle \mid \boldsymbol {x} ^ {k} \right] + 2 \gamma^ {2} \rho^ {2} \mathbb {1} _ {k} + \frac {2 \gamma^ {2} \sigma^ {2}}{n - 2 B - m}. \\ \leq \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} - \mathbb {E} \left[ \left\| \boldsymbol {x} ^ {k + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \mid \boldsymbol {x} ^ {k} \right] \\ + 2 \gamma^ {2} \rho^ {2} \mathbb {1} _ {k} + \frac {2 \gamma^ {2} \sigma^ {2}}{n - 2 B - m} + 2 \gamma \rho \| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \| \mathbb {1} _ {k}. \\ \end{array}
$$
Taking the full expectation from the both sides of the above inequality and summing up the results for  $k = 0,1,\ldots ,K - 1$  we derive
$$
\begin{array}{l} \frac {\gamma}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} [ \langle F (\boldsymbol {x} ^ {k}), \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \rangle ] \\ \leq \frac {1}{K} \sum_ {k = 0} ^ {K - 1} \left(\mathbb {E} \left[ \| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \| ^ {2} \right] - \mathbb {E} \left[ \| \boldsymbol {x} ^ {k + 1} - \boldsymbol {x} ^ {*} \| ^ {2} \right]\right) + \frac {2 \gamma^ {2} \sigma^ {2}}{n - 2 B - m} \\ + \frac {2 \gamma \rho}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ | | \pmb {x} ^ {k} - \pmb {x} ^ {*} | | \mathbb {1} _ {k} \right] + \frac {2 \gamma^ {2} \rho^ {2}}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} [ \mathbb {1} _ {k} ] \\ \leq \frac {\| \boldsymbol {x} ^ {0} - \boldsymbol {x} ^ {*} \| ^ {2} - \mathbb {E} [ \| \boldsymbol {x} ^ {K} - \boldsymbol {x} ^ {*} \| ^ {2} ]}{K} + \frac {2 \gamma^ {2} \sigma^ {2}}{n - 2 B - m} \\ + \frac {2 \gamma \rho}{K} \sum_ {k = 0} ^ {K - 1} \sqrt {\mathbb {E} \left[ \| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \| ^ {2} \right] \mathbb {E} [ \mathbb {1} _ {k} ]} + \frac {2 \gamma^ {2} \rho^ {2}}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} [ \mathbb {1} _ {k} ]. \\ \end{array}
$$
Since  $F$  satisfies Assumption 6,  $\sum_{k=0}^{K-1} \mathbb{E}[\langle F(\pmb{x}^k), \pmb{x}^k - \pmb{x}^* \rangle] \geq 0$ . Using this and new notation  $R_k = \| \pmb{x}^k - \pmb{x}^* \|, k > 0$ ,  $R_0 \geq \| \pmb{x}^0 - \pmb{x}^* \|$  we get
$$
\begin{array}{l} { 0 } { \leq } { \frac { R _ { 0 } ^ { 2 } - \mathbb { E } [ R _ { K } ^ { 2 } ] } { K } + \frac { 2 \gamma ^ { 2 } \sigma ^ { 2 } } { n - 2 B - m } } \\ + \frac {2 \gamma \rho}{K} \sum_ {k = 0} ^ {K - 1} \sqrt {\mathbb {E} \left[ R _ {k} ^ {2} \right] \mathbb {E} \left[ \mathbb {1} _ {k} \right]} + \frac {2 \gamma^ {2} \rho^ {2}}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \mathbb {1} _ {k} \right] \tag {45} \\ \end{array}
$$
implying (after changing the indices) that
$$
\mathbb {E} \left[ R _ {k} ^ {2} \right] \leq R _ {0} ^ {2} + \frac {2 \gamma^ {2} \sigma^ {2} k}{n - 2 B - m} + 2 \gamma \rho \sum_ {l = 0} ^ {k - 1} \sqrt {\mathbb {E} \left[ R _ {l} ^ {2} \right] \mathbb {E} [ \mathbb {1} _ {l} ]} + 2 \gamma^ {2} \rho^ {2} \sum_ {l = 0} ^ {k - 1} \mathbb {E} [ \mathbb {1} _ {l} ] \tag {46}
$$
holds for all  $k \geq 0$ . In the remaining part of the proof we derive by induction that
$$
R _ {0} ^ {2} + \frac {2 \gamma^ {2} \sigma^ {2} k}{n - 2 B - m} + 2 \gamma \rho \sum_ {l = 0} ^ {k - 1} \sqrt {\mathbb {E} \left[ R _ {l} ^ {2} \right] \mathbb {E} \left[ \mathbb {1} _ {l} \right]} + 2 \gamma^ {2} \rho^ {2} \sum_ {l = 0} ^ {k - 1} \mathbb {E} \left[ \mathbb {1} _ {k} \right] \leq 2 R _ {0} ^ {2} \tag {47}
$$
for all  $k = 0, \dots, K$ . For  $k = 0$  this inequality trivially holds. Next, assume that it holds for all  $k = 0, 1, \dots, T - 1, T \leq K - 1$ . Let us show that it holds for  $k = T$  as well. From (46) and (47) we have that  $\mathbb{E}[R_k^2] \leq 2R_0^2$  for all  $k = 0, 1, \dots, T - 1$ . Therefore,