SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} [ R _ {T} ^ {2} ] \leq R _ {0} ^ {2} + \frac {2 \gamma^ {2} \sigma^ {2} T}{n - 2 B - m} + 2 \gamma \rho \sum_ {l = 0} ^ {T - 1} \sqrt {\mathbb {E} [ R _ {l} ^ {2} ] \mathbb {E} [ \mathbb {1} _ {l} ]} + 2 \gamma^ {2} \rho^ {2} \sum_ {l = 0} ^ {T - 1} \mathbb {E} [ \mathbb {1} _ {l} ] \\ \leq R _ {0} ^ {2} + \frac {2 \gamma^ {2} \sigma^ {2} T}{n - 2 B - m} + 2 \sqrt {2} \gamma \rho R _ {0} \sum_ {l = 0} ^ {T - 1} \sqrt {\mathbb {E} [ \mathbb {1} _ {l} ]} + 2 \gamma^ {2} \rho^ {2} \sum_ {l = 0} ^ {T - 1} \mathbb {E} [ \mathbb {1} _ {l} ]. \\ \end{array}
$$
If a Byzantine peer deviates from the protocol at iteration  $k$ , it will be detected with some probability  $p_k$  during the next iteration. One can lower bound this probability as
$$
p _ {k} \geq m \cdot \frac {G _ {k}}{n _ {k}} \cdot \frac {1}{n _ {k}} = \frac {m (1 - \delta_ {k})}{n _ {k}} \geq \frac {m}{n}.
$$
Therefore, each individual Byzantine worker can violate the protocol no more than  $1 / p$  times on average implying that
$$
\mathbb {E} [ R _ {T} ^ {2} ] \leq R _ {0} ^ {2} + \frac {2 \gamma^ {2} \sigma^ {2} T}{n - 2 B - m} + \frac {2 \sqrt {2} \gamma \rho R _ {0} n B}{m} + \frac {2 \gamma^ {2} \rho^ {2} n B}{m}
$$
Taking
$$
\gamma = \min  \left\{\frac {1}{2 \ell}, \sqrt {\frac {(n - 2 B - m) R _ {0} ^ {2}}{6 \sigma^ {2} K}}, \sqrt {\frac {m ^ {2} R _ {0} ^ {2}}{7 2 \rho^ {2} B ^ {2} n ^ {2}}} \right\}
$$
we ensure that
$$
\frac {2 \gamma^ {2} \sigma^ {2} T}{n - 2 B - m} + \frac {2 \sqrt {2} \gamma \rho R _ {0} n B}{m} + \frac {2 \gamma^ {2} \rho^ {2} n B}{m} \leq \frac {R _ {0} ^ {2}}{3} + \frac {R _ {0} ^ {2}}{3} + \frac {R _ {0} ^ {2}}{3} = R _ {0} ^ {2},
$$
and, as a result, we get
$$
\mathbb {E} \left[ R _ {T} ^ {2} \right] \leq 2 R _ {0} ^ {2} \equiv 2 R \tag {48}
$$
. Therefore, (47) holds for all  $k = 0,1,\ldots ,K$  . Together with (45) it implies
$$
\sum_ {k = 0} ^ {K - 1} \mathbb {E} [ \langle F (\boldsymbol {x} ^ {k}), \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \rangle ] \leq \frac {2 R _ {0} ^ {2}}{\gamma}.
$$
The last inequality together with Assumption 6 implies
$$
\sum_ {k = 0} ^ {K - 1} \mathbb {E} \left\| F (\boldsymbol {x} ^ {k}) \right\| \leq \frac {2 \ell R _ {0} ^ {2}}{\gamma}.
$$
Corollary 5. Let assumptions of Theorem 9 hold. Then  $\frac{1}{K}\sum_{k = 0}^{K - 1}\mathbb{E}\big\| F(\pmb{x}^k)\big\| \leq \varepsilon$  holds after
$$
K = \mathcal {O} \left(\frac {\ell^ {2} R ^ {2}}{\varepsilon} + \frac {\sigma^ {2} \ell^ {2} R ^ {2}}{n \varepsilon^ {2}} + \frac {\sigma n ^ {2} \ell R}{m \varepsilon}\right)
$$
iterations of SGDA-CC.
$$
\begin{array}{l} \frac {1}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \| F (\boldsymbol {x} ^ {k}) \| \leq \frac {2 \ell R ^ {2}}{\gamma K} \leq \frac {2 \ell R ^ {2}}{K} \left(2 \ell + \sqrt {\frac {6 \sigma^ {2} K}{(n - 2 B - m) R ^ {2}}} + \sqrt {\frac {7 2 \rho^ {2} B ^ {2} n ^ {2}}{m ^ {2} R ^ {2}}}\right) \\ \leq \frac {4 \ell^ {2} R ^ {2}}{K} + \sqrt {\frac {2 4 \sigma^ {2} \ell^ {2} R ^ {2}}{(n - 2 B - m) K}} + \frac {1 7 \rho B n \ell R}{m K} \\ \end{array}
$$
Let us chose  $K$  such that each of the last three terms less or equal  $\varepsilon /3$ , then
$$
K = \max  \left(\frac {6 \ell^ {2} R ^ {2}}{\varepsilon}, \frac {2 1 6 \sigma^ {2} \ell^ {2} R ^ {2}}{(n - 2 B - m) \varepsilon^ {2}}, \frac {5 1 \rho B n \ell R}{m \varepsilon}\right)
$$
where  $\rho^2 = q\sigma^2$  with  $q = 2C^2 +12 + \frac{12}{n - 2B - m}$  and  $C = \mathcal{O}(1)$  by Lemma E.1. The latter implies that
$$
\frac {1}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \| F (\boldsymbol {x} ^ {k}) \| \leq \varepsilon .
$$
Using the definition of  $\rho$  from Lemma E.1 and if  $B \leq \frac{n}{4}$ ,  $m << n$  the bound for  $K$  can be easily derived.
E.1.2 Quasi-Strongly Monotone Case
Theorem (Theorem 4 duplicate). Let Assumptions 1, 4 and 6 hold. Then after  $T$  iterations SGDA-CC (Algorithm 5) with  $\gamma \leq \frac{1}{2\ell}$  outputs  $\mathbf{x}^T$  such that
$$
\mathbb {E} \left\| \boldsymbol {x} ^ {T + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \leq \left(1 - \frac {\gamma \mu}{2}\right) ^ {T + 1} \left\| \boldsymbol {x} ^ {0} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \frac {4 \gamma \sigma^ {2}}{\mu (n - 2 B - m)} + \frac {2 \rho^ {2} n B}{m} \left(\frac {\gamma}{\mu} + \gamma^ {2}\right).
$$
where  $\rho^2 = q\sigma^2$  with  $q = 2C^2 +12 + \frac{12}{n - 2B - m}$  and  $C = \mathcal{O}(1)$  by Lemma E.1.
Proof of Theorem 4. The proof is similar to the proof of Theorem 1
$$
\begin{array}{l} \mu \mathbb {E} \left[ \left\| \overline {{\boldsymbol {x}}} ^ {K} - \boldsymbol {x} ^ {*} \right\| ^ {2} \right] = \mu \mathbb {E} \left[ \left\| \frac {1}{K} \sum_ {k = 0} ^ {K - 1} \left(\boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*}\right) \right\| ^ {2} \right] \leq \mu \mathbb {E} \left[ \frac {1}{K} \sum_ {k = 0} ^ {K - 1} \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} \right] \\ = \frac {\mu}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| ^ {2} \right] \stackrel {(Q S M)} {\leq} \frac {1}{K} \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \langle F (\boldsymbol {x} ^ {k}), \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \rangle \right] \\ \end{array}
$$
Applying (11) for  $\langle \overline{\pmb{g}}^t, \pmb{x}^t - \pmb{x}^* \rangle$  with  $\lambda = \frac{\gamma \mu}{2}$  and (12) for  $\| \widehat{\pmb{g}}^t \|^2 = \left\| \widehat{\pmb{g}}^t - \overline{\pmb{g}}^t + \overline{\pmb{g}}^t \right\|^2$  we derive
The difference with the proof of Theorem 1 is that we suppose that the number of peer violating the protocol at an iteration  $t$  is known to any "good" peer. So the result of Lemma E.1 writes as follows
$$
\left. \mathbb {E} _ {\boldsymbol {\xi}} \left\| \widehat {\boldsymbol {g}} ^ {t} - \overline {{\boldsymbol {g}}} ^ {t} \right\| ^ {2} \leq \rho^ {2} \mathbb {1} _ {t}, \right.
$$
where  $\mathbb{1}_t$  is an indicator function of the event that at least 1 Byzantine peer violates the protocol at iteration  $t$ .
Together with Lemma C.1 we can proceed as follows m
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {\xi}} \left\| \boldsymbol {x} ^ {t + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \leq \left(1 + \frac {\gamma \mu}{2}\right) \left\| \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \left(2 \gamma^ {2} \ell - 2 \gamma\right) \langle F (\boldsymbol {x} ^ {t}), \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {*} \rangle \\ + \frac {2 \gamma^ {2} \sigma^ {2}}{G} + 2 \mathbb {1} _ {t} \rho^ {2} \left(\frac {\gamma}{\mu} + \gamma^ {2}\right), \\ \end{array}
$$
Since  $\gamma \leq \frac{1}{2\ell}$  and Assumption (QSM) holds we derive
$$
\mathbb {E} _ {\boldsymbol {\xi}} \left\| \boldsymbol {x} ^ {t + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \leq \left(1 - \frac {\gamma \mu}{2}\right) \left\| \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \frac {2 \gamma^ {2} \sigma^ {2}}{G} + 2 \mathbb {1} _ {t} \rho^ {2} \left(\frac {\gamma}{\mu} + \gamma^ {2}\right).
$$
$$
\mathbb {E} \left\| \boldsymbol {x} ^ {t + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \leq \left(1 - \frac {\gamma \mu}{2}\right) \mathbb {E} \left\| \boldsymbol {x} ^ {t} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \frac {2 \gamma^ {2} \sigma^ {2}}{n - 2 B - m} + 2 \rho^ {2} \left(\frac {\gamma}{\mu} + \gamma^ {2}\right) \mathbb {E} \mathbb {1} _ {t}.
$$
$$
\begin{array}{l} \mathbb {E} \left\| \boldsymbol {x} ^ {T + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \leq \left(1 - \frac {\gamma \mu}{2}\right) ^ {T + 1} \left\| \boldsymbol {x} ^ {0} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \frac {4 \gamma \sigma^ {2}}{\mu (n - 2 B - m)} \\ + 2 \rho^ {2} \left(\frac {\gamma}{\mu} + \gamma^ {2}\right) \sum_ {i} ^ {T} \mathbb {E} \mathbb {1} _ {i} \left(1 - \frac {\gamma \mu}{2}\right) ^ {T - i}. \\ \end{array}
$$
If a Byzantine peer deviates from the protocol at iteration  $t$ , it will be detected with some probability  $p_t$  during the next iteration. One can lower bound this probability as
$$
p _ {t} \geq m \cdot \frac {G _ {t}}{n _ {t}} \cdot \frac {1}{n _ {t}} = \frac {m (1 - \delta_ {t})}{n _ {t}} \geq \frac {m}{n}.
$$
Therefore, each individual Byzantine worker can violate the protocol no more than  $\frac{1}{p}$  times on average implying that
$$
\mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \mathbb {1} _ {t} \right] \leq \frac {n B}{m}
$$
$$
\mathbb {E} \left[ \sum_ {i} ^ {T} \mathbb {1} _ {i} \left(1 - \frac {\gamma \mu}{2}\right) ^ {T - i} \right] \leq \mathbb {E} \left[ \sum_ {i} ^ {T} \mathbb {1} _ {i} \right] \leq \frac {n B}{m}. \tag {49}
$$
The latter together with the above bound on  $\mathbb{E}\left\| \boldsymbol{x}^{T + 1} - \boldsymbol{x}^*\right\|^2$  implies the result of the theorem.
Corollary 6. Let assumptions of Theorem 4 hold. Then  $\mathbb{E}\left\| \pmb{x}^T - \pmb{x}^*\right\|^2 \leq \varepsilon$  holds after
$$
T = \widetilde {\mathcal {O}} \bigg (\frac {\ell}{\mu} + \frac {\sigma^ {2}}{\mu^ {2} (n - 2 B - m) \varepsilon} + \frac {q \sigma^ {2} B n}{\mu^ {2} m \varepsilon} + \frac {q \sigma^ {2} B n}{\mu^ {2} m \sqrt {\varepsilon}} \bigg)
$$
iterations of SGDA-CC (Alg. 5) with
$$
\gamma = \min  \left\{\frac {1}{2 \ell}, \frac {2 \ln \left(\max  \left\{2 , \min  \left\{\frac {m (n - 2 B - m) \mu^ {2} R ^ {2} K}{8 m \sigma^ {2} + 4 q \sigma^ {2} n B (n - 2 B - m)} , \frac {m \mu^ {2} R ^ {2} K ^ {2}}{8 q n B \sigma^ {2}} \right\} \right\}\right)}{\mu (K + 1)} \right\}.
$$
Proof. Using the definition of  $\rho (\rho^2 = q\sigma^2 = \mathcal{O}(\sigma^2))$  from Lemma E.1 and if  $B\leq \frac{n}{4}$ ,  $m < < n$  the result of Theorem 4 can be simplified as
$$
\mathbb {E} \left\| \boldsymbol {x} ^ {T + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \leq \left(1 - \frac {\gamma \mu}{2}\right) ^ {T + 1} \left\| \boldsymbol {x} ^ {0} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \frac {4 \gamma \sigma^ {2}}{\mu (n - 2 B - m)} + \frac {2 q \sigma^ {2} n B}{m} \left(\frac {\gamma}{\mu} + \gamma^ {2}\right).
$$
Applying Lemma C.4 to the last bound we get the result of the corollary.
E.1.3 Monotone Case
Theorem 10. Suppose the assumptions of Theorem 9 and Assumption 5 hold. Next, assume that
$$
\gamma = \min  \left\{\frac {1}{2 \ell}, \sqrt {\frac {(n - 2 B - m) R ^ {2}}{6 \sigma^ {2} K}}, \sqrt {\frac {m ^ {2} R ^ {2}}{7 2 \rho^ {2} B ^ {2} n ^ {2}}} \right\} \tag {50}
$$
where  $\rho^2 = q\sigma^2$  with  $q = 2C^2 +12 + \frac{12}{n - 2B - m}$  and  $C = \mathcal{O}(1)$  by Lemma E.1 and  $R\geq \| \pmb{x}^0 -\pmb{x}^*\|$ . Then after  $K$  iterations of SGDA-CC (Algorithm 5)
$$
\mathbb {E} \left[ \operatorname {G a p} _ {B _ {R} \left(x ^ {*}\right)} \left(\overline {{\boldsymbol {x}}} ^ {K}\right) \right] \leq \frac {3 R ^ {2}}{\gamma K}, \tag {51}
$$
where  $\mathbf{G}a\mathbf{p}_{B_R(x^*)}\big(\overline{\mathbf{x}}^K\big) = \max_{u\in B_R(x^*)}\bigl \langle F(\boldsymbol {u}),\overline{\boldsymbol{x}}^K -\boldsymbol {u}\bigr \rangle ,\overline{\boldsymbol{x}}^K = \frac{1}{K}\sum_{k = 0}^{K - 1}\boldsymbol{x}^k$  and  $R\geq \| \pmb{x}^0 -\pmb{x}^*\|$ .
Proof. Combining (16), (14) one can derive
$$
\begin{array}{l} 2 \gamma \left\langle F (\boldsymbol {x} ^ {k}), \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\rangle \leq \left\| \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\| ^ {2} - \left\| \boldsymbol {x} ^ {k + 1} - \boldsymbol {u} \right\| ^ {2} \\ - 2 \gamma \left\langle \widehat {\boldsymbol {g}} ^ {k} - \overline {{\boldsymbol {g}}} ^ {k}, \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\rangle - 2 \gamma \left\langle \overline {{\boldsymbol {g}}} ^ {k} - F ^ {k}, \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\rangle \\ + 2 \gamma^ {2} \left\| \widehat {\boldsymbol {g}} ^ {k} - \overline {{\boldsymbol {g}}} ^ {k} \right\| ^ {2} + 2 \gamma^ {2} \left\| \overline {{\boldsymbol {g}}} ^ {k} \right\| ^ {2}. \\ \end{array}
$$
Assumption 5 implies that
$$
\left\langle F (\boldsymbol {u}), \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\rangle \leq \left\langle F \left(\boldsymbol {x} ^ {k}\right), \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\rangle \tag {52}
$$
and consequently by Jensen inequality
$$
\begin{array}{l} 2 \gamma K \left\langle F (\boldsymbol {u}), \overline {{\boldsymbol {x}}} ^ {K} - \boldsymbol {u} \right\rangle \leq \left\| \boldsymbol {x} ^ {0} - \boldsymbol {u} \right\| ^ {2} - 2 \gamma \sum_ {k = 0} ^ {K - 1} \left\langle \widehat {\boldsymbol {g}} ^ {k} - \overline {{\boldsymbol {g}}} ^ {k}, \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\rangle \\ - 2 \gamma \sum_ {k = 0} ^ {K - 1} \left\langle \bar {\boldsymbol {g}} ^ {k} - F ^ {k}, \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\rangle + 2 \gamma^ {2} \sum_ {k = 0} ^ {K - 1} \left(\left\| \hat {\boldsymbol {g}} ^ {k} - \bar {\boldsymbol {g}} ^ {k} \right\| ^ {2} + \left\| \bar {\boldsymbol {g}} ^ {k} \right\| ^ {2}\right). \\ \end{array}
$$
Then maximization in  $\pmb{u}$  gives
$$
\begin{array}{l} 2 \gamma K \operatorname {G a p} _ {B _ {R} (x ^ {*})} (\overline {{\boldsymbol {x}}} ^ {K}) \leq \max  _ {u \in B _ {R} (x ^ {*})} \| \boldsymbol {x} ^ {0} - \boldsymbol {u} \| ^ {2} + 2 \gamma^ {2} \sum_ {k = 0} ^ {K - 1} \left(\left\| \widehat {\boldsymbol {g}} ^ {k} - \overline {{\boldsymbol {g}}} ^ {k} \right\| ^ {2} + \left\| \overline {{\boldsymbol {g}}} ^ {k} \right\| ^ {2}\right) \\ + 2 \gamma \max  _ {u \in B _ {R} \left(x ^ {*}\right)} \left(\sum_ {k = 0} ^ {K - 1} \left\langle \bar {\boldsymbol {g}} ^ {k} - \widehat {\boldsymbol {g}} ^ {k}, \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\rangle\right) \\ + 2 \gamma \max  _ {u \in B _ {R} (x ^ {*})} \left(\sum_ {k = 0} ^ {K - 1} \left\langle F ^ {k} - \overline {{\boldsymbol {g}}} ^ {k}, \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\rangle\right). \\ \end{array}
$$
Taking the full expectation from the both sides of the previous inequality gives
$$
\begin{array}{l} 2 \gamma K \mathbb {E} \left[ \operatorname {G a p} _ {B _ {R} \left(x ^ {*}\right)} \left(\overline {{\boldsymbol {x}}} ^ {K}\right) \right] \leq \max  _ {u \in B _ {R} \left(x ^ {*}\right)} \left\| \boldsymbol {x} ^ {0} - \boldsymbol {u} \right\| ^ {2} \\ + 2 \gamma \mathbb {E} \left[ \max  _ {u \in B _ {R} \left(x ^ {*}\right)} \left(\sum_ {k = 0} ^ {K - 1} \left\langle \overline {{\boldsymbol {g}}} ^ {k} - \widehat {\boldsymbol {g}} ^ {k}, \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\rangle\right) \right] \\ + 2 \gamma \mathbb {E} \left[ \max  _ {u \in B _ {R} (x ^ {*})} \left(\sum_ {k = 0} ^ {K - 1} \left\langle F ^ {k} - \overline {{\boldsymbol {g}}} ^ {k}, \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\rangle\right) \right]. \\ + 2 \gamma^ {2} \mathbb {E} \left[ \sum_ {k = 0} ^ {K - 1} \left(\left\| \widehat {\boldsymbol {g}} ^ {k} - \overline {{\boldsymbol {g}}} ^ {k} \right\| ^ {2} + \left\| \overline {{\boldsymbol {g}}} ^ {k} \right\| ^ {2}\right) \right] \\ \end{array}
$$
Firstly obtain the bound for the terms that do not depend on  $\mathbf{u}$  using Assumption 1, Lemma E.1 and Theorem 9
$$
\begin{array}{l} 2 \gamma^ {2} \mathbb {E} \left[ \sum_ {k = 0} ^ {K - 1} \left(\left\| \widehat {\boldsymbol {g}} ^ {k} - \overline {{\boldsymbol {g}}} ^ {k} \right\| ^ {2} + \left\| \overline {{\boldsymbol {g}}} ^ {k} \right\| ^ {2}\right) \right] \\ \leq 2 \gamma^ {2} \mathbb {E} \left[ \sum_ {k = 0} ^ {K - 1} \left\| \widehat {\boldsymbol {g}} ^ {k} - \overline {{\boldsymbol {g}}} ^ {k} \right\| ^ {2} \right] + 2 \gamma^ {2} \mathbb {E} \left[ \sum_ {k = 0} ^ {K - 1} \left\| \overline {{\boldsymbol {g}}} ^ {k} \right\| ^ {2} \right] \\ \leq 2 \gamma^ {2} \rho^ {2} \mathbb {E} \left[ \sum_ {k = 0} ^ {K - 1} \mathbb {1} _ {k} \right] + \frac {2 \gamma^ {2} K \sigma^ {2}}{| \mathcal {G} _ {t} \backslash \mathcal {C} _ {t} |} + 2 \gamma^ {2} \ell \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left\langle F ^ {k}, \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\rangle \\ \leq \frac {2 \gamma^ {2} n B c \rho^ {2}}{m} + \frac {2 \gamma^ {2} K \sigma^ {2}}{| \mathcal {G} _ {t} \setminus \mathcal {C} _ {t} |} + 4 \ell \gamma R ^ {2}. \\ \end{array}
$$
Since  $\mathbb{E}\big[\| \pmb{x}^k - \pmb{u}\|\big] \leq \mathbb{E}\big[\| \pmb{x}^k - \pmb{x}^*\| \big] + \| \pmb{x}^* - \pmb{u}\| \leq \mathbb{E}\big[\| \pmb{x}^k - \pmb{x}^*\| \big] + \max_{u \in B_R(x^*)} \| \pmb{x}^* - \pmb{u}\|^{\frac{(48)}{2}}$  one can derive that
$$
\begin{array}{l} 2 \gamma \mathbb {E} \left[ \max  _ {u \in B _ {R} (x ^ {*})} \left(\sum_ {k = 0} ^ {K - 1} \left\langle \bar {\boldsymbol {g}} ^ {k} - \hat {\boldsymbol {g}} ^ {k}, \boldsymbol {x} ^ {k} - \boldsymbol {u} \right\rangle\right) \right] \\ \leq 2 \gamma \mathbb {E} \left[ \max  _ {u \in B _ {R} \left(x ^ {*}\right)} \left(\sum_ {k = 0} ^ {K - 1} \left\langle \overline {{\boldsymbol {g}}} ^ {k} - \widehat {\boldsymbol {g}} ^ {k}, \boldsymbol {x} ^ {*} - \boldsymbol {u} \right\rangle\right) + \sum_ {k = 0} ^ {K - 1} \left\langle \overline {{\boldsymbol {g}}} ^ {k} - \widehat {\boldsymbol {g}} ^ {k}, \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\rangle \right] \\ \leq 2 \gamma \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \max  _ {u \in B _ {R} \left(x ^ {*}\right)} \left\langle \overline {{g}} ^ {k} - \widehat {\boldsymbol {g}} ^ {k}, \boldsymbol {x} ^ {*} - \boldsymbol {u} \right\rangle \right] + 2 \gamma \mathbb {E} \left[ \sum_ {k = 0} ^ {K - 1} \left\langle \overline {{g}} ^ {k} - \widehat {\boldsymbol {g}} ^ {k}, \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\rangle \right] \\ \leq 2 \gamma \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ \max  _ {u \in B _ {R} \left(x ^ {*}\right)} \left\| \overline {{\boldsymbol {g}}} ^ {k} - \widehat {\boldsymbol {g}} ^ {k} \right\| \left\| \boldsymbol {x} ^ {*} - \boldsymbol {u} \right\| \right] + 2 \gamma \mathbb {E} \left[ \mathbb {E} \left[ \sum_ {k = 0} ^ {K - 1} \left\| \overline {{\boldsymbol {g}}} ^ {k} - \widehat {\boldsymbol {g}} ^ {k} \right\| \left\| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \right\| | x ^ {k} \right] \right] \\ \leq 2 \gamma \sum_ {k = 0} ^ {K - 1} \mathbb {E} \left[ R \| \overline {{\boldsymbol {g}}} ^ {k} - \widehat {\boldsymbol {g}} ^ {k} \| \right] + 2 \gamma \mathbb {E} \left[ \sum_ {k = 0} ^ {K - 1} \rho \mathbb {1} _ {k} \| \boldsymbol {x} ^ {k} - \boldsymbol {x} ^ {*} \| \right] \\ \leq 2 \gamma \rho R \mathbb {E} \left[ \sum_ {k = 0} ^ {K - 1} \mathbb {1} _ {k} \right] + 4 \gamma \rho R \mathbb {E} \left[ \sum_ {k = 0} ^ {K - 1} \mathbb {1} _ {k} \right] \leq 6 \gamma R \rho \mathbb {E} \left[ \sum_ {k = 0} ^ {K - 1} \mathbb {1} _ {k} \right] \leq \frac {6 n B \gamma R \rho}{m} \\ \end{array}
$$
Following Beznosikov et al. [2023] one can derive he bound for the next term:
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {k = 0} ^ {K - 1} \langle F ^ {k} - \overline {{\boldsymbol {g}}} ^ {k}, \boldsymbol {x} ^ {k} \rangle \right] = \mathbb {E} \left[ \sum_ {k = 0} ^ {K - 1} \langle \mathbb {E} [ F ^ {k} - \overline {{\boldsymbol {g}}} ^ {k} \mid x ^ {k} ], \boldsymbol {x} ^ {k} \rangle \right] = 0, \\ \mathbb {E} \left[ \sum_ {k = 0} ^ {K - 1} \langle F ^ {k} - \overline {{\boldsymbol {g}}} ^ {k}, \boldsymbol {x} ^ {0} \rangle \right] = \sum_ {k = 0} ^ {K - 1} \left\langle \mathbb {E} [ F ^ {k} - \overline {{\boldsymbol {g}}} ^ {k} ], \boldsymbol {x} ^ {0} \right\rangle = 0, \\ \end{array}
$$
$$
\begin{array}{l} 2 \gamma \mathbb {E} \left[ \max  _ {\boldsymbol {u} \in B _ {R} (x ^ {*})} \sum_ {k = 0} ^ {K - 1} \langle F ^ {k} - \overline {{\boldsymbol {g}}} ^ {k}, \boldsymbol {x} ^ {k} - \boldsymbol {u} \rangle \right] \\ = 2 \gamma \mathbb {E} \left[ \sum_ {k = 0} ^ {K - 1} \langle F ^ {k} - \bar {\boldsymbol {g}} ^ {k}, \boldsymbol {x} ^ {k} \rangle \right] + 2 \gamma \mathbb {E} \left[ \max  _ {\boldsymbol {u} \in B _ {R} (x ^ {*})} \sum_ {k = 0} ^ {K - 1} \langle F ^ {k} - \bar {\boldsymbol {g}} ^ {k}, - \boldsymbol {u} \rangle \right] \\ = 2 \gamma \mathbb {E} \left[ \max  _ {\boldsymbol {u} \in B _ {R} \left(x ^ {*}\right)} \sum_ {k = 0} ^ {K - 1} \left\langle F ^ {k} - \overline {{\boldsymbol {g}}} ^ {k}, - \boldsymbol {u} \right\rangle \right] \\ = 2 \gamma \mathbb {E} \left[ \sum_ {k = 0} ^ {K - 1} \langle F ^ {k} - \overline {{\boldsymbol {g}}} ^ {k}, \boldsymbol {x} ^ {0} \rangle \right] \\ + 2 \gamma \mathbb {E} \left[ \max  _ {\boldsymbol {u} \in B _ {R} (x ^ {*})} \sum_ {k = 0} ^ {K - 1} \langle F ^ {k} - \bar {\boldsymbol {g}} ^ {k}, - \boldsymbol {u} \rangle \right] \\ = 2 \gamma K \mathbb {E} \left[ \max  _ {\boldsymbol {u} \in B _ {R} \left(x ^ {*}\right)} \left\langle \frac {1}{K} \sum_ {k = 0} ^ {K - 1} \left(F ^ {k} - \overline {{\boldsymbol {g}}} ^ {k}\right), \boldsymbol {x} ^ {0} - \boldsymbol {u} \right\rangle \right] \\ \stackrel {(1 1)} {\leq} \quad 2 \gamma K \mathbb {E} \left[ \max  _ {\boldsymbol {u} \in B _ {R} (x ^ {*})} \left\{\frac {\gamma}{2} \left\| \frac {1}{K} \sum_ {k = 0} ^ {K - 1} \left(F ^ {k} - \overline {{\boldsymbol {g}}} ^ {k}\right) \right\| ^ {2} + \frac {1}{2 \gamma} \| \boldsymbol {x} ^ {0} - \boldsymbol {u} \| ^ {2} \right\} \right] \\ = \gamma^ {2} \mathbb {E} \left[ \left\| \sum_ {k = 0} ^ {K - 1} \left(F ^ {k} - \overline {{\boldsymbol {g}}} ^ {k}\right) \right\| ^ {2} \right] + \max  _ {\boldsymbol {u} \in B _ {R} \left(x ^ {*}\right)} \| \boldsymbol {x} ^ {0} - \boldsymbol {u} \| ^ {2}. \\ \end{array}
$$
We notice that  $\mathbb{E}[F^k -\overline{\pmb{g}}^k\mid F^0 -\overline{\pmb{g}}^0,\dots ,F^{k - 1} - \overline{\pmb{g}}^{k - 1}] = 0$  for all  $k\geq 1$ , i.e., conditions of Lemma C.2 are satisfied. Therefore, applying Lemma C.2, we get
$$
\begin{array}{l} 2 \gamma \mathbb {E} \left[ \max  _ {\boldsymbol {u} \in B _ {R} (x ^ {*})} \sum_ {k = 0} ^ {K - 1} \langle F ^ {k} - \overline {{\boldsymbol {g}}} ^ {k}, \boldsymbol {x} ^ {k} - \boldsymbol {u} \rangle \right] \leq \gamma^ {2} \sum_ {k = 0} ^ {K - 1} \mathbb {E} [ \| F ^ {k} - \overline {{\boldsymbol {g}}} ^ {k} \| ^ {2} ] \\ + \max  _ {\boldsymbol {u} \in B _ {R} \left(x ^ {*}\right)} \| \boldsymbol {x} ^ {0} - \boldsymbol {u} \| ^ {2} (53) \\ \leq \frac {\gamma^ {2} K \sigma^ {2}}{| \mathcal {C} _ {t} \backslash \mathcal {C} _ {t} |} + \max  _ {\boldsymbol {u} \in B _ {R} \left(x ^ {*}\right)} \| \boldsymbol {x} ^ {0} - \boldsymbol {u} \| ^ {2}. (54) \\ \end{array}
$$
Assembling the above results together gives
$$
\begin{array}{l} 2 \gamma K \mathbb {E} \left[ \operatorname {G a p} _ {B _ {R} \left(x ^ {*}\right)} \left(\overline {{\boldsymbol {x}}} ^ {K}\right) \right] \\ \leq 2 \max  _ {u \in B _ {R} \left(x ^ {*}\right)} \left\| \boldsymbol {x} ^ {0} - \boldsymbol {u} \right\| ^ {2} + \frac {2 \gamma^ {2} n B \rho^ {2}}{m} + \frac {3 \gamma^ {2} K \sigma^ {2}}{\left| \mathcal {G} _ {t} \backslash \mathcal {C} _ {t} \right|} + 4 \ell \gamma R ^ {2} + \frac {6 n B \gamma R \rho}{m} \\ \leq 2 \max  _ {u \in B _ {R} (x ^ {*})} \| \boldsymbol {x} ^ {0} - \boldsymbol {u} \| ^ {2} + \frac {2 \gamma^ {2} n B \rho^ {2}}{m} + \frac {3 \gamma^ {2} K \sigma^ {2}}{n - 2 B - m} \\ + 4 \ell \gamma R ^ {2} + \frac {6 n B \gamma R \rho}{m} \stackrel {(4 4)} {\leq} 6 R ^ {2}. \\ \end{array}
$$
Corollary 7. Let assumptions of Theorem 10 hold. Then  $\mathbb{E}\left[\mathsf{G}\mathsf{a}\mathsf{p}_{B_R(x^*)}(\overline{\mathbf{x}}^K)\right] \leq \varepsilon$  holds after
$$
K = \mathcal {O} \left(\frac {\ell R ^ {2}}{\varepsilon} + \frac {\sigma^ {2} R ^ {2}}{n \varepsilon^ {2}} + \frac {\sigma n ^ {2} R}{m \varepsilon}\right)
$$
$$
\begin{array}{l} \mathbb {E} \left[ \operatorname {G a p} _ {B _ {R} \left(x ^ {*}\right)} \left(\overline {{x}} ^ {K}\right) \right] \leq \frac {3 R ^ {2}}{\gamma K} \leq \frac {3 R ^ {2}}{K} \left(2 \ell + \sqrt {\frac {6 \sigma^ {2} K}{(n - 2 B - m) R ^ {2}}} + \sqrt {\frac {7 2 \rho^ {2} B ^ {2} n ^ {2}}{m ^ {2} R ^ {2}}}\right) \\ \leq \frac {6 \ell R ^ {2}}{K} + \sqrt {\frac {5 4 \sigma^ {2} R ^ {2}}{(n - 2 B - m) K}} + \frac {2 6 \rho B n R}{m K} \\ \end{array}
$$
$$
K = \max \bigg (\frac {1 8 \ell R ^ {2}}{\varepsilon}, \frac {9 \cdot 5 4 \sigma^ {2} R ^ {2}}{(n - 2 B - m) \varepsilon^ {2}}, \frac {7 8 \rho B n R}{m \varepsilon} \bigg)
$$
guarantees that
$$
\mathbb {E} \left[ \operatorname {G a p} _ {B _ {R} \left(x ^ {*}\right)} \left(\overline {{\boldsymbol {x}}} ^ {K}\right) \right] \leq \varepsilon .
$$
E.2 Proofs for R-SGDA-CC
E.2.1 Quasi-Strongly Monotone Case
Algorithm 6 R-SGDA-CC
Input:  $x^0$  - starting point,  $r$  - number of restarts,  $\{\gamma_t\}_{t=1}^r$  - stepsizes for SGDA-CC (Alg. 5),  $\{K_t\}_{t=1}^r$  - number of iterations for SGDA-CC (Alg. 5)