SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A. Adibi, A. Mitra, G. J. Pappas, and H. Hassani. Distributed statistical min-max learning in the presence of byzantine agents. In 2022 IEEE 61st Conference on Decision and Control (CDC), pages 4179-4184. IEEE, 2022.
D. Alistarh, Z. Allen-Zhu, and J. Li. Byzantine stochastic gradient descent. Advances in Neural Information Processing Systems, 31, 2018.
Z. Allen-Zhu, F. Ebrahimianghazani, J. Li, and D. Alistarh. Byzantine-resilient non-convex stochastic gradient descent. In International Conference on Learning Representations, 2021.
Y. Allouah, S. Farhadkhani, R. Guerraoui, N. Gupta, R. Pinot, and J. Stephan. Fixing by mixing: A recipe for optimal byzantine ml under heterogeneity. In International Conference on Artificial Intelligence and Statistics, pages 1232-1300. PMLR, 2023.
M. Baruch, G. Baruch, and Y. Goldberg. A little is enough: Circumventing defenses for distributed learning, 2019.
A. Beznosikov, E. Gorbunov, H. Berard, and N. Loizou. Stochastic gradient descent-ascent: Unified theory and new efficient methods. In International Conference on Artificial Intelligence and Statistics, pages 172-235. PMLR, 2023.
P. Blanchard, E. M. El Mhamdi, R. Guerraoui, and J. Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. Advances in Neural Information Processing Systems, 30, 2017.
J. Bose, G. Gidel, H. Berard, A. Cianflone, P. Vincent, S. Lacoste-Julien, and W. Hamilton. Adversarial example games. Advances in neural information processing systems, 33:8921-8934, 2020.
F. E. Browder. Existence and approximation of solutions of nonlinear variational inequalities. Proceedings of the National Academy of Sciences, 56(4):1080-1086, 1966.
Y. Chen, L. Su, and J. Xu. Distributed statistical machine learning in adversarial settings: Byzantine gradient descent. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 1 (2):1-25, 2017.
G. Damaskinos, E.-M. El-Mhamdi, R. Guerraoui, A. Guirguis, and S. Rouault. Aggregathor: Byzantine machine learning via robust gradient aggregation. Proceedings of Machine Learning and Systems, 1:81-106, 2019.
V. F. Dem'yanov and A. B. Pevnyi. Numerical methods for finding saddle points. USSR Computational Mathematics and Mathematical Physics, 12(5):11-52, 1972.
M. Diskin, A. Bukhtiyarov, M. Ryabinin, L. Saulnier, A. Sinitsin, D. Popov, D. V. Pyrkin, M. Kashirin, A. Borzunov, A. Villanova del Moral, et al. Distributed deep learning in open collaborations. Advances in Neural Information Processing Systems, 34:7879-7897, 2021.
F. Facchinei and J.-S. Pang. Finite-dimensional variational inequalities and complementarity problems. Springer, 2003.
G. Gidel, H. Berard, G. Vignoud, P. Vincent, and S. Lacoste-Julien. A variational inequality perspective on generative adversarial networks. In International Conference on Learning Representations, 2018. URL https://openreview.net/pdf?id=r1laEnA5Ym.
A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D. Silver, and K. Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning, pages 3540-3549. PMLR, 2017.
G. Wayne and L. Abbott. Hierarchical control using networks trained with higher-level forward models. Neural computation, 26(10):2163-2193, 2014.
Z. Wu, Q. Ling, T. Chen, and G. B. Giannakis. Federated variance-reduced stochastic gradient descent with robustness to byzantine attacks. IEEE Transactions on Signal Processing, 68:4583-4596, 2020.
C. Xie, S. Koyejo, and I. Gupta. Fall of empires: Breaking Byzantine-tolerant sgd by inner product manipulation, 2019.
Y.-R. Yang and W.-J. Li. Basgd: Buffered asynchronous sgd for byzantine learning. In International Conference on Machine Learning, pages 11751-11761. PMLR, 2021.
D. Yin, Y. Chen, R. Kannan, and P. Bartlett. Byzantine-robust distributed learning: Towards optimal statistical rates. In International Conference on Machine Learning, pages 5650-5659. PMLR, 2018.
G. Zhang, Y. Wang, L. Lessard, and R. B. Grosse. Near-optimal local convergence of alternating gradient descent-ascent for minimax optimization. In International Conference on Artificial Intelligence and Statistics, pages 7659-7679. PMLR, 2022.
H. Zhu and Q. Ling. Broadcast: Reducing both stochastic and compression noise to robustify communication-efficient federated learning. arXiv preprint arXiv:2104.06685, 2021.
M. Zinkevich, M. Weimer, L. Li, and A. Smola. Parallelized stochastic gradient descent. Advances in neural information processing systems, 23, 2010.
1.1 Setting 2
1.2 Our Contributions 4
1.3 Related Work 5
2 Main Results 5
2.1 Methods with Robust Aggregation 5
2.2 Client Momentum 6
2.3 Random Checks of Computations 7
3 Numerical Experiments 9
4 Conclusion 9
A Examples of  $(\delta, c)$ -Robust Aggregation Rules 17
A.1 Aggregators 17
A.2 Bucketing algorithm 17
A.3 Robust Aggregation examples 17
B Further Details on RDEG 19
C Auxiliary results 21
C.1 Basic Inequalities 21
C.2 Usefull Lemmas 21
D Methods that use robust aggregators 24
D.1 Proofs for SGDA-RA 24
D.1.1 Quasi-Strongly Monotone Case 24
D.2 Proofs for SEG-RA 25
D.2.1 Auxiliary results 26
D.2.2 Quasi-Strongly Monotone Case 28
D.3 Proofs for M-SGDA-RA 30
D.3.1 Quasi-Strongly Monotone Case 30
E Methods with random check of computations 35
E.1 Proofs for SGDA-CC 36
E.1.1 Star Co-coercive Case 36
E.1.2 Quasi-Strongly Monotone Case 39
E.1.3 Monotone Case 41
E.2 Proofs for R-SGDA-CC 44
E.2.1 Quasi-Strongly Monotone Case 44
E.3 Proofs for SEG-CC 46
E.3.1 Auxiliary results 46
E.3.2 Lipschitz Case 48
E.3.3 Quasi-Strongly Monotone Case 51
E.3.4 Lipschitz Monotone Case 56
E.4 Proofs for R-SEG-CC 59
E.4.1 Quasi Strongly Monotone Case 59
FExtra Experiments and Experimental details 61
F.1 Quadratic games 61
F.2 Generative Adversarial Networks 68
A Examples of  $(\delta, c)$ -Robust Aggregation Rules
This section is about how to construct an aggregator satisfying 1.1.
A.1 Aggregators
This subsection examines various aggregators that lack robustness. It means that new attacks can be easily designed to exploit the aggregation scheme, causing its failure. We analyze three commonly employed defenses that are representative.
Krum. For  $i \neq j$ , let  $i \rightarrow j$  denote that  $\pmb{x}_j$  belongs to the  $n - q - 2$  closest vectors to  $\pmb{x}_i$ . Then,
$$
\operatorname {K R U M} \left(\boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {n}\right) := \underset {i} {\operatorname {a r g m i n}} \sum_ {i \rightarrow j} \left\| \boldsymbol {x} _ {i} - \boldsymbol {x} _ {j} \right\| ^ {2}.
$$
Krum is computationally expensive, requiring  $\mathcal{O}(n^2)$  work by the server Blanchard et al. [2017].
CM. Coordinate-wise median computes for the  $k$ -th coordinate:
$$
\left[ \mathrm {C M} \left(\boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {n}\right) \right] _ {k} := \operatorname {m e d i a n} \left(\left[ \boldsymbol {x} _ {1} \right] _ {k}, \dots , \left[ \boldsymbol {x} _ {n} \right] _ {k}\right) = \underset {i} {\arg \min } \sum_ {j = 1} ^ {n} \left| \left[ \boldsymbol {x} _ {i} \right] _ {k} - \left[ \boldsymbol {x} _ {j} \right] _ {k} \right|.
$$
Coordinate-wise median is fast to implement requiring only  $\mathcal{O}(n)$  time Chen et al. [2017].
RFA. Robust federated averaging (RFA) computes the geometric median
$$
\operatorname {R F A} \left(\boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {n}\right) := \underset {\boldsymbol {v}} {\operatorname {a r g m i n}} \sum_ {i = 1} ^ {n} \| \boldsymbol {v} - \boldsymbol {x} _ {i} \| _ {2}.
$$
Although there is no closed form solution for the geometric median, an approximation technique presented by Pillutla et al. [2022] involves performing several iterations of the smoothed Weiszfeld algorithm, with each iteration requiring a computation of complexity  $\mathcal{O}(n)$ .
A.2 Bucketing algorithm
We use the process of  $s$ -bucketing, propose by [Yang and Li, 2021, Karimireddy et al., 2022] to randomly divide  $n$  inputs,  $\pmb{x}_1$  to  $\pmb{x}_n$ , into  $\lceil n / s\rceil$  buckets, each containing no more than  $s$  elements. After averaging the contents of each bucket to create  $\pmb{y}_1,\dots ,\pmb{y}_{\lceil n / s\rceil}$ , we input them into the aggregator AGGR. The Bucketing Algorithm outlines the procedure. Our approach's main feature is that the resulting set of averaged  $\pmb{y}_1,\dots ,\pmb{y}_{\lceil n / s\rceil}$  are more homogeneous (with lower variance) than the original inputs.
Algorithm Bucketing Algorithm
1: input  $\{\pmb{x}_1,\dots ,\pmb{x}_n\} ,s\in \mathbb{N}$  aggregation rule AGGR
2: pick random permutation  $\pi$  of  $[n]$
3: compute  $\pmb{y}_i\gets \frac{1}{s}\sum_{k = (i - 1)\cdot s + 1}^{\min (n,i\cdot s)}\pmb{x}_{\pi (k)}$  for  $i = \{1,\dots ,\lceil n / s\rceil \}$
4: output  $\widehat{\pmb{x}}\gets \mathrm{AGGR}(\pmb {y}_1,\dots ,\pmb {y}_{\lceil n / s\rceil})$
// aggregate after bucketing
A.3 Robust Aggregation examples
Next we recall the result from [Karimireddy et al., 2022], that shows that aggregators which we saw, can be made to satisfy 1.1 by combining with bucketing.
Theorem 8. Suppose we are given  $n$  inputs  $\{\pmb{x}_1, \dots, \pmb{x}_n\}$  such that  $\mathbb{E}\| \pmb{x}_i - \pmb{x}_j\|^2 \leq \rho^2$  for any fixed pair  $i, j \in \mathcal{G}$  and some  $\rho \geq 0$  for some  $\delta \leq \delta_{\max}$ , with  $\delta_{\max}$  to be defined. Then, running Bucketing Algorithm with  $s = \left\lfloor \frac{\delta_{\max}}{\delta} \right\rfloor$  yields the following:
- Krum:  $\mathbb{E}\| \mathrm{K R U M} \circ \mathrm{B U C K E T I N G}(\pmb{x}_1, \dots, \pmb{x}_n) - \bar{\pmb{x}}\|^2 \leq \mathcal{O}(\delta\rho^2)$  with  $\delta_{\max} < \frac{1}{4}$ .
- Geometric median:  $\mathbb{E}\| \mathrm{RFA} \circ \mathrm{BUCKETING}(\pmb{x}_1, \dots, \pmb{x}_n) - \bar{\pmb{x}}\|^2 \leq \mathcal{O}(\delta\rho^2)$  with  $\delta_{\max} < \frac{1}{2}$ .
- Coordinate-wise median:  $\mathbb{E}\| \mathbf{CM} \circ \mathbf{BUCKETING}(\pmb{x}_1, \dots, \pmb{x}_n) - \bar{\pmb{x}}\|^2 \leq \mathcal{O}(d\delta\rho^2)$  with  $\delta_{\max} < \frac{1}{2}$ .
Note that all these methods satisfy the notion of an agnostic Byzantine robust aggregator (Definition 1.1).
B Further Details on RDEG
Originally RDEG was proposed for min-max problems and represents a variation of SEG with Univariate Trimmed-Mean Estimator aggregation rule. For convenience we give here RDEG pseudocode we used in experiments.
In this section we use the notation  $\pi \in (0,1)$  for a confidence level.
Robust Distributed Extra-Gradient (RDEG)
Univariate Trimmed-Mean Estimator Algorithm Lugosi and Mendelson [2021]
Input:  $\mathrm{TRIM}_{\epsilon, \alpha, \delta}, \gamma$   
1: for  $t = 1, \ldots$  do  
2: for worker  $i \in [n]$  in parallel  
3:  $g_{\xi_i}^t \gets g_i(\boldsymbol{x}^t, \boldsymbol{\xi}_i)$   
4: send  $g_{\xi_i}^t$  if  $i \in \mathcal{G}$ , else send * if Byzantine  
5:  $\widehat{g}_{\xi^t}(\boldsymbol{x}^t) = \mathrm{TRIM}_{\epsilon, \alpha, \delta}(\boldsymbol{g}_{\xi_1}^t, \ldots, \boldsymbol{g}_{\xi_n}^t)$   
6:  $\widetilde{\boldsymbol{x}}^t \gets \boldsymbol{x}^t - \gamma_1 \widehat{g}_{\xi^t}(\boldsymbol{x}^t)$ .  
7: for worker  $i \in [n]$  in parallel  
8:  $g_{\eta_i}^t \gets g_i(\widetilde{\boldsymbol{x}}^t, \eta_i)$   
9: send  $g_{\eta_i}^t$  if  $i \in \mathcal{G}$ , else send * if Byzantine  
10:  $\widehat{g}_{\eta^t}(\widetilde{\boldsymbol{x}}^t) = \mathrm{TRIM}_{\epsilon, \alpha, \delta}(\boldsymbol{g}_{\eta_1}^t, \ldots, \boldsymbol{g}_{\eta_n}^t)$   
11:  $x^{t+1} \gets x^t - \gamma_2 \widehat{g}_{\eta^t}(\widetilde{x}^t)$ .
Performance of Univariate Trimmed-Mean Estimator. The TRIM operator takes as input  $n$  vectors, and applies coordinatewisely the univariate trimmed mean estimator from Lugosi and Mendelson [2021], described below here as Univariate Trimmed-Mean Estimator Algorithm.
Input: Corrupted data set  $Z_{1}, \ldots, Z_{n/2}$ ,  $\widetilde{Z}_{1}, \ldots, \widetilde{Z}_{n/2}$ , corruption fraction  $\delta$ , and confidence level  $\pi$ . 1: Set  $\epsilon = 8\delta + 24\frac{\log(4/\pi)}{n}$ .
2: Let  $Z_1^* \leq Z_2^* \leq \dots \leq Z_{n/2}^*$  represent a non-decreasing arrangement of  $\{Z_i\}_{i \in [n/2]}$ . Compute quantiles:  $\gamma = Z_{\epsilon n/2}^*$  and  $\beta = Z_{(1 - \epsilon)n/2}^*$ .
3: Compute robust mean estimate  $\widehat{\mu}_Z$  as follows:
$$
\widehat {\mu} _ {Z} = \frac {2}{n} \sum_ {i = 1} ^ {n / 2} \phi_ {\gamma , \beta} (\widetilde {Z} _ {i}); \phi_ {\gamma , \beta} (x) = \left\{ \begin{array}{l l} \beta & x > \beta \\ x & x \in [ \gamma , \beta ] \\ \gamma & x <   \gamma \end{array} \right.
$$
The following result on the performance of Univariate Trimmed-Mean Estimator plays a key role in the analysis of RDEG.
Theorem. [Adibi et al., 2022, Theorem 1] Consider the trimmed mean estimator. Suppose  $\delta \in [0,1/16)$ , and let  $\pi \in (0,1)$  be such that  $\pi \geq 4e^{-n/2}$ . Then, there exists an universal constant  $c$  such that with probability at least  $1 - \pi$ ,
$$
| \widehat {\mu} _ {Z} - \mu_ {Z} | \leq c \sigma_ {Z} \left(\sqrt {\delta} + \sqrt {\frac {\log (1 / \pi)}{n}}\right).
$$
Using the latter componentwise result the authors states that
$$
\left\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {t}} \left(\boldsymbol {x} ^ {t}\right) - F \left(\boldsymbol {x} ^ {t}\right) \right\| \leq c \sigma \left(\sqrt {\delta} + \sqrt {\frac {\log (1 / \pi)}{n}}\right).
$$
In fact this result is very similar to the Definition 1.1. The main difference is that for Univariate Trimmed-Mean Estimator we have a bound with some probability. The other difference that using
the following representation of the result with  $\rho^2 = c^2\sigma^2$
$$
\| \widehat {\boldsymbol {g}} _ {\boldsymbol {\xi} ^ {t}} (\boldsymbol {x} ^ {t}) - F (\boldsymbol {x} ^ {t}) \| ^ {2} \leq \delta \rho^ {2} + \frac {\rho^ {2} \log (1 / \pi)}{n}, \quad \mathrm {w . p .} 1 - \pi
$$
Univariate Trimmed-Mean Estimator has the additional term inversely depending on the number of workers.
Moreover, the result requires  $\delta \in [0,1 / 16)$  in contrast to the aggregators we used, that work for wider range of corruption level  $\delta \in [0,1 / 5]$ .
Performance guarantees for RDEG. The authors of [Adibi et al., 2022] consider only homogeneous case.
Theorem. [Adibi et al., 2022, Theorem 3] Suppose Assumptions 3 and 4 hold in conjunction with the assumptions on  $\delta$  and  $n$ : the fraction  $\delta$  of corrupted devices satisfies  $\delta \in [0,1/16)$ , and the number of agents  $n$  is sufficiently large:  $n \geq 48 \log(16dT^2)$ . Then, with  $\pi = 1/(4dT^2)$  and step-size  $\eta \leq 1/(4L)$ , RDEG guarantees the following with probability at least  $1 - 1/T$ :
$$
\left\| \boldsymbol {x} ^ {*} - \boldsymbol {x} ^ {T + 1} \right\| ^ {2} \leq 2 e ^ {- \frac {T}{4 \kappa}} R ^ {2} + \frac {8 c \sigma R \kappa}{L} \left(\sqrt {\delta} + \sqrt {\frac {\log (4 d T ^ {2})}{n}}\right), \tag {10}
$$
where  $\kappa = \mu /L$
The result implies that RDEG benefits of collaboration only when the corruption level is small. In fact, the term  $\frac{\log(4dT^2)}{n} \leq \frac{\log(4dT^2)}{48\log(16dT^2)} \leq 1/48$ , so the corruption level should be less than  $1/48$  to make RDEG significantly benefit of collaboration in contrast to our SEG-CC that requires corruption level only less than  $1/5$ . Moreover, in case of larger corruption level, RDEG converges to a ball centered at the solution with radius  $\widetilde{\mathcal{O}}\left(\sqrt{\frac{\sqrt{\delta}\sigma R\kappa}{L}}\right)$  in contrast to our methods SGDA-RA, SEG-RA and M-SGDA-RA converge to a ball centered at the solution with radius  $\widetilde{\mathcal{O}}\left(\sqrt{\frac{c\delta\sigma^2}{\mu^2}}\right)$ , that has a better dependence on  $\sigma$ . It is crucial with increasing batchsize ( $b =$  batchsize), since  $\sigma^2$  depends on a batchsize as  $\frac{1}{b}$ .
C Auxiliary results
C.1 Basic Inequalities
For all  $a, b \in \mathbb{R}^n$  and  $\lambda > 0$ ,  $q \in (0,1]$
$$
| \langle a, b \rangle | \leq \frac {\| a \| _ {2} ^ {2}}{2 \lambda} + \frac {\lambda \| b \| _ {2} ^ {2}}{2}, \tag {11}
$$
$$
\left\| a + b \right\| _ {2} ^ {2} \leq 2 \left\| a \right\| _ {2} ^ {2} + 2 \| b \| _ {2} ^ {2}, \tag {12}
$$
$$
\left\| a + b \right\| ^ {2} \leq (1 + \lambda) \left\| a \right\| ^ {2} + \left(1 + \frac {1}{\lambda}\right) \left\| b \right\| ^ {2}, \tag {13}
$$
$$
\langle a, b \rangle = \frac {1}{2} \left(\| a + b \| _ {2} ^ {2} - \| a \| _ {2} ^ {2} - \| b \| _ {2} ^ {2}\right), \tag {14}
$$
$$
\langle a, b \rangle = \frac {1}{2} \left(- \| a - b \| _ {2} ^ {2} + \| a \| _ {2} ^ {2} + \| b \| _ {2} ^ {2}\right), \tag {15}
$$
$$
\left\| \sum_ {i = 1} ^ {n} a _ {i} \right\| ^ {2} \leq n \sum_ {i = 1} ^ {n} \| a _ {i} \| ^ {2}, \tag {16}
$$
$$
\left\| a + b \right\| ^ {2} \geq \frac {1}{2} \left\| a \right\| ^ {2} - \left\| b \right\| ^ {2}, \tag {17}
$$
$$
\left(1 - \frac {q}{2}\right) ^ {- 1} \leq 1 + q, \tag {18}
$$
$$
\left(1 + \frac {q}{2}\right) (1 - q) \leq 1 - \frac {q}{2}. \tag {19}
$$
C.2 Usefull Lemmas
We write  $\pmb{g}_i^t$  or simply  $\pmb{g}_i$  instead of  $\pmb{g}_i(\pmb{x}^t,\pmb{\xi}_i^t)$  when there is no ambiguity.
Lemma C.1. Suppose that the operator  $F$  is given in the form (1) and Assumptions 1 and 6 hold. Then
$$
\mathbb {E} _ {\boldsymbol {\xi}} \left\| \overline {{\boldsymbol {g}}} (\boldsymbol {x}, \boldsymbol {\xi}) \right\| ^ {2} \leq \ell \left\langle F (\boldsymbol {x}), \boldsymbol {x} - \boldsymbol {x} ^ {*} \right\rangle + \frac {\sigma^ {2}}{G},
$$
where  $\mathbb{E}_{\pmb{\xi}} := \Pi_i \mathbb{E}_{\pmb{\xi}_i}$  and  $\overline{\pmb{g}}(\pmb{x}, \pmb{\xi}) = \frac{1}{G} \sum_{i \in G} \pmb{g}_i(\pmb{x}; \pmb{\xi}_i)$ .
Proof of Lemma C.1. First of one can decomposed a squared norm of a difference and obtain
$$
\mathbb {E} _ {\boldsymbol {\xi}} \left\| \bar {\boldsymbol {g}} (\boldsymbol {x}, \boldsymbol {\xi}) - F (\boldsymbol {x}) \right\| ^ {2} = \mathbb {E} _ {\boldsymbol {\xi}} \left\| \bar {\boldsymbol {g}} (\boldsymbol {x}, \boldsymbol {\xi}) \right\| ^ {2} - 2 \langle \mathbb {E} _ {\boldsymbol {\xi}} \bar {\boldsymbol {g}} (\boldsymbol {x}, \boldsymbol {\xi}), F (\boldsymbol {x}) \rangle + \| F (\boldsymbol {x}) \| ^ {2}.
$$
Since  $\overline{\pmb{g}} (\pmb {x},\pmb {\xi}) = \frac{1}{G}\sum_{i\in \mathcal{G}}\pmb{g}_i(\pmb {x};\pmb{\xi}_i)$ , by the definition (1) of  $F$  and by Assumption 1 one has
$$
\mathbb {E} _ {\boldsymbol {\xi}} \bar {\boldsymbol {g}} (\boldsymbol {x}, \boldsymbol {\xi}) = \frac {1}{G} \sum_ {i \in \mathcal {G}} \mathbb {E} _ {\boldsymbol {\xi} _ {i}} \boldsymbol {g} _ {i} (\boldsymbol {x}, \boldsymbol {\xi} _ {i}) = \frac {1}{G} \sum_ {i \in \mathcal {G}} F _ {i} (\boldsymbol {x}) = F (x),
$$
$$
\left. \mathbb {E} _ {\boldsymbol {\xi}} \| \bar {\boldsymbol {g}} (\boldsymbol {x}, \boldsymbol {\xi}) \| ^ {2} = \mathbb {E} _ {\boldsymbol {\xi}} \| \bar {\boldsymbol {g}} (\boldsymbol {x}, \boldsymbol {\xi}) + F (\boldsymbol {x}) \| ^ {2} - \| F (\boldsymbol {x}) \| ^ {2}. \right. \tag {20}
$$
One can bound  $\mathbb{E}_{\pmb{\xi}}\| \overline{\pmb{g}} (\pmb {x},\pmb {\xi}) - F(\pmb {x})\| ^2$  as
$$
\mathbb {E} _ {\boldsymbol {\xi}} \| \overline {{\boldsymbol {g}}} (\boldsymbol {x}, \boldsymbol {\xi}) - F (\boldsymbol {x}) \| ^ {2} \quad = \quad \mathbb {E} _ {\boldsymbol {\xi}} \left\| \frac {1}{G} \sum_ {i \in \mathcal {G}} \left(\boldsymbol {g} _ {i} (\boldsymbol {x}; \boldsymbol {\xi} _ {i}) - F _ {i} (\boldsymbol {x})\right) \right\| ^ {2}
$$
$$
\stackrel {\text {i n d e p e n d e n c e o f} \boldsymbol {\xi} _ {i}} {=} \frac {1}{G ^ {2}} \sum_ {i \in \mathcal {G}} \mathbb {E} _ {\boldsymbol {\xi} _ {i}} \| \boldsymbol {g} _ {i} (\boldsymbol {x}; \boldsymbol {\xi} _ {i}) - F _ {i} (\boldsymbol {x}) \| ^ {2} \leq \frac {\sigma^ {2}}{G},
$$
where the last inequality of the above chain follows from (SC). The above chain together with (20) and (SC) implies the statement of the theorem.
Lemma C.2. Let  $K > 0$  be a positive integer and  $\eta_1, \eta_2, \ldots, \eta_K$  be random vectors such that  $\mathbb{E}_k[\eta_k] \stackrel{\text{def}}{=} \mathbb{E}[\eta_k \mid \eta_1, \ldots, \eta_{k-1}] = 0$  for  $k = 2, \ldots, K$ . Then
$$
\mathbb {E} \left[ \left\| \sum_ {k = 1} ^ {K} \eta_ {k} \right\| ^ {2} \right] = \sum_ {k = 1} ^ {K} \mathbb {E} [ \| \eta_ {k} \| ^ {2} ]. \tag {21}
$$
Proof. We start with the following derivation:
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \sum_ {k = 1} ^ {K} \eta_ {k} \right\| ^ {2} \right] = \mathbb {E} [ \| \eta_ {K} \| ^ {2} ] + 2 \mathbb {E} \left[ \left\langle \eta_ {K}, \sum_ {k = 1} ^ {K - 1} \eta_ {k} \right\rangle \right] + \mathbb {E} \left[ \left\| \sum_ {k = 1} ^ {K - 1} \eta_ {k} \right\| ^ {2} \right] \\ { = } { \mathbb { E } [ \| \eta _ { K } \| ^ { 2 } ] + 2 \mathbb { E } \left[ \mathbb { E } _ { K } \left[ \left\langle \eta _ { K } , \sum _ { k = 1 } ^ { K - 1 } \eta _ { k } \right\rangle \right] \right] + \mathbb { E } \left[ \left\| \sum _ { k = 1 } ^ { K - 1 } \eta _ { k } \right\| ^ { 2 } \right] } \\ = \mathbb {E} [ \| \eta_ {K} \| ^ {2} ] + 2 \mathbb {E} \left[ \left\langle \mathbb {E} _ {K} [ \eta_ {K} ], \sum_ {k = 1} ^ {K - 1} \eta_ {k} \right\rangle \right] + \mathbb {E} \left[ \left\| \sum_ {k = 1} ^ {K - 1} \eta_ {k} \right\| ^ {2} \right] \\ = \mathbb {E} [ \| \eta_ {K} \| ^ {2} ] + \mathbb {E} \left[ \left\| \sum_ {k = 1} ^ {K - 1} \eta_ {k} \right\| ^ {2} \right]. \\ \end{array}
$$
Applying similar steps to  $\mathbb{E}\left[\left\| \sum_{k = 1}^{K - 1}\eta_k\right\| ^2\right],\mathbb{E}\left[\left\| \sum_{k = 1}^{K - 2}\eta_k\right\| ^2\right],\ldots ,\mathbb{E}\left[\left\| \sum_{k = 1}^2\eta_k\right\| ^2\right]$ , we get the result.
Lemma C.3. Suppose
$$
r _ {K} \leq r _ {0} (1 - a \gamma) ^ {K} + \frac {c _ {1} \gamma}{b} + \frac {c _ {0}}{b} \tag {22}
$$
holds for  $\gamma \leq \gamma_0$ . Then the choice of
$$
b \geq \frac {3 c _ {0}}{\varepsilon}
$$
$$
\gamma \leq \min  \left(\gamma_ {0}, \frac {c _ {0}}{c _ {1}}\right)
$$
implies that  $r_K \leq \varepsilon$  for
$$
K \geq \frac {1}{a} \max  \left(\frac {c _ {1}}{c _ {0}}, \frac {1}{\gamma_ {0}}\right) \ln \frac {3 r _ {0}}{\varepsilon}
$$
Proof. Since  $b \geq \frac{3c_0}{\varepsilon}$  then  $\frac{c_0}{b} \leq \frac{\varepsilon}{3}$  and  $\frac{c_1\gamma}{b} \leq \frac{c_1\gamma\varepsilon}{3c_0}$ . The choice of  $\gamma \leq \min \left(\gamma_0, \frac{c_0}{c_1}\right)$  implies that  $\frac{c_1\gamma}{b} \leq \frac{\varepsilon}{3}$ .
The choice of  $K \geq \frac{1}{a} \max \left(\frac{c_1}{c_0}, \frac{1}{\gamma_0}\right) \ln \frac{3r_0}{\varepsilon}$  implies that  $r_0(1 - a\gamma)^K \leq \frac{\varepsilon}{3}$  and finishes the proof.
Lemma C.4 (see also Lemma 2 from Stich [2019] and Lemma D.2 from Gorbunov et al. [2020]). Let  $\{r_k\}_{k\geq 0}$  satisfy
$$
r _ {K} \leq r _ {0} (1 - a \gamma) ^ {K + 1} + c _ {1} \gamma + c _ {2} \gamma^ {2} \tag {23}
$$
for all  $K \geq 0$  with some constants  $a, c_2 > 0$ ,  $c_1 \geq 0$ ,  $\gamma \leq \gamma_0$ .
Then for
$$
\gamma = \min  \left\{\gamma_ {0}, \frac {\ln \left(\max  \left\{2 , \min  \left\{a r _ {0} K / c _ {1} , a ^ {2} r _ {0} K ^ {2} / c _ {2} \right\} \right\}\right)}{a (K + 1)} \right\} \tag {24}
$$
we have that
$$
r _ {K} = \widetilde {\mathcal {O}} \left(r _ {0} \exp \left(- a \gamma_ {0} (K + 1)\right) + \frac {c _ {1}}{a K} + \frac {c _ {2}}{a ^ {2} K ^ {2}}\right).
$$
Moreover  $r_K \leq \varepsilon$  after
$$
K = \widetilde {\mathcal {O}} \left(\frac {1}{a \gamma_ {0}} + \frac {c _ {1}}{a \varepsilon} + \frac {c _ {2}}{a ^ {2} \sqrt {\varepsilon}}\right)
$$
$$
r _ {K} \leq r _ {0} (1 - a \gamma) ^ {K + 1} + c _ {1} \gamma + c _ {2} \gamma^ {2} \leq r _ {0} \exp (- a \gamma (K + 1)) + c _ {1} \gamma + c _ {2} \gamma^ {2}. \tag {25}
$$
Next we consider two possible situations.
1. If  $\gamma_0 \geq \frac{\ln\left(\max\{2, \min\{a r_0 K / c_1, a^2 r_0 K^2 / c_2\}\}\right)}{a(K + 1)}$  then we choose  $\gamma = \frac{\ln\left(\max\{2, \min\{a r_0 K / c_1, a^2 r_0 K^2 / c_2\}\}\right)}{a(K + 1)}$  and get that