SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \widetilde {\boldsymbol {x}} ^ {t} = \boldsymbol {x} ^ {t} - \gamma_ {1} \text {R A G G} (\boldsymbol {g} _ {\boldsymbol {\xi} _ {1}} ^ {t}, \ldots , \boldsymbol {g} _ {\boldsymbol {\xi} _ {n}} ^ {t}), \text {w h e r e} \boldsymbol {g} _ {\boldsymbol {\xi} _ {i}} ^ {t} = \boldsymbol {g} _ {i} (\boldsymbol {x} ^ {t}, \boldsymbol {\xi} _ {i} ^ {t}), \forall i \in \mathcal {G} \text {a n d} \boldsymbol {g} _ {\boldsymbol {\xi} _ {i}} ^ {t} = *   \forall i \in \mathcal {B}, \\ \boldsymbol {x} ^ {t + 1} = \boldsymbol {x} ^ {t} - \gamma_ {2} \text {R A G G} (\boldsymbol {g} _ {\boldsymbol {\eta} _ {1}} ^ {t}, \ldots , \boldsymbol {g} _ {\boldsymbol {\eta} _ {n}} ^ {t}), \text {w h e r e} \boldsymbol {g} _ {\boldsymbol {\eta} _ {i}} ^ {t} = \boldsymbol {g} _ {i} (\widetilde {\boldsymbol {x}} ^ {t}, \boldsymbol {\eta} _ {i} ^ {t}), \forall i \in \mathcal {G} \text {a n d} \boldsymbol {g} _ {\boldsymbol {\eta} _ {i}} ^ {t} = *   \forall i \in \mathcal {B}, \end{array}
$$
where  $\{\pmb{g}_{\eta_i}^t\}_{i\in \mathcal{G}}$  and  $\{\pmb{g}_{\eta_i}^t\}_{i\in \mathcal{G}}$  are sampled independently. Our main convergence result for SEG-RA is presented in the following theorem; see Appendix D.2 for the proof.
Theorem 2. Let Assumptions  $1, 2, 3$  and  $4$  hold. Then after  $T$  iterations SEG-RA (Algorithm 2) with  $(\delta, c)$ -RAGG,  $\gamma_1 \leq \frac{1}{2\mu + 2L}$  and  $\beta = \gamma_2 / \gamma_1 \leq 1/4$  outputs  $\boldsymbol{x}^T$  such that
$$
\mathbb {E} \left\| \boldsymbol {x} ^ {T} - \boldsymbol {x} ^ {*} \right\| ^ {2} \leq \left(1 - \frac {\mu \beta \gamma_ {1}}{4}\right) ^ {T} \left\| \boldsymbol {x} ^ {0} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \frac {8 \gamma_ {1} \sigma^ {2}}{\mu \beta G} + 8 c \delta (2 4 \sigma^ {2} + 1 2 \zeta^ {2}) \left(\frac {\gamma_ {1}}{\beta \mu} + \frac {2}{\mu^ {2}}\right).
$$
Similar to the case of SGDA-RA, the bound for SEG-RA has the term that cannot be reduced without increasing batchsize even in the homogeneous data case. RDEG, which is also a modification of SEG, has the same linearly convergent term, but SEG-RA has a better dependence on the batchsize, needed to obtain the convergence to any predefined accuracy, that is  $\mathcal{O}(\varepsilon^{-1})$  versus  $\mathcal{O}(\varepsilon^{-2})$  for RDEG; see Cor. 3.
In heterogeneous case when  $\sigma = 0$ , SEG-RA also converges linearly to the ball centered at the solution with radius  $\mathcal{O}(\sqrt{c\delta}\zeta /\mu)$  that matches the lower bound.
2.2 Client Momentum
Next, we focus on the version of SGDA-RA that utilizes worker momentum  $m_{i}^{t}$ , i.e.,
$$
\boldsymbol {x} ^ {t + 1} = \boldsymbol {x} ^ {t} - \gamma \operatorname {R A G G} \left(\boldsymbol {m} _ {1} ^ {t}, \dots , \boldsymbol {m} _ {n} ^ {t}\right), \text {w i t h} \boldsymbol {m} _ {i} ^ {t} = (1 - \alpha) \boldsymbol {m} _ {i} ^ {t - 1} + \alpha \boldsymbol {g} _ {i} ^ {t},
$$
where  $\pmb{g}_i^t = \pmb{g}_i(\pmb{x}^t,\pmb{\xi}_i^t)$ ,  $\forall i\in \mathcal{G}$  and  $\pmb{g}_i^t = *\forall i\in \mathcal{B}$  and  $\{\pmb{g}_{\pmb{\xi}_i}^t\}_{i\in \mathcal{G}}$  are sampled independently. Our main convergence result for this version called M-SGDA-RA is summarized in the following theorem.
Theorem 3. Let Assumptions 1, 2, 4, and 6 hold. Then after  $T$  iterations M-SGDA-RA (Algorithm 3) with  $(\delta, c)$ -RAGG outputs  $\overline{\boldsymbol{x}}^T$  such that
$$
\mathbb {E} \left[ \left\| \overline {{\boldsymbol {x}}} ^ {T} - \boldsymbol {x} ^ {*} \right\| ^ {2} \right] \leq \frac {2 \left\| \boldsymbol {x} ^ {0} - \boldsymbol {x} ^ {*} \right\| ^ {2}}{\mu \gamma \alpha W _ {T}} + \frac {8 \gamma c \delta (2 4 \sigma^ {2} + 1 2 \zeta^ {2})}{\mu \alpha^ {2}} + \frac {6 \gamma \sigma^ {2}}{\mu \alpha^ {2} G} + \frac {4 c \delta (2 4 \sigma^ {2} + 1 2 \zeta^ {2})}{\mu^ {2} \alpha^ {2}}.
$$
where  $\overline{\pmb{x}}^T = \frac{1}{W_T}\sum_{t = 0}^T w_t\widehat{\pmb{x}}^t,$ $\widehat{\pmb{x}}^t = \frac{\alpha}{1 - (1 - \alpha)^{t + 1}}\sum_{j = 0}^{t}(1 - \alpha)^{t - j}\pmb {x}^j,$ $w_{t} = \left(1 - \frac{\mu\gamma\alpha}{2}\right)^{-t - 1},$  and  $W_{T} = \sum_{t = 0}^{T}w_{t}$
${}^{5}$  SGDA-based and SEG-based methods are typically analyzed under different assumptions. Although (SC) follows from (Lip) and (QSM) with  $\ell  = {L}^{2}/\mu$  ,some operators may satisfy (SC) with significantly smaller  $\ell$  . Next,when  $\mu  = 0$  ,SGDA is not guaranteed to converge [Gidel et al., 2018], while SEG does
Despite the fact that M-SGDA-RA is the first algorithm (for VIs) non-invariant to permutations, it also requires large batches to achieve convergence to any accuracy. Even in the context of minimization, which is much easier than VI, the known SOTA analysis of Momentum-SGD relies in the convex case on the unbiasedness of the estimator that is not available due to a robust aggregation. Nevertheless, we prove<sup>6</sup> the convergence to the ball centered at the solution with radius  $\mathcal{O}(\sqrt{c\delta(\zeta + \sigma)} / \alpha \mu)$ ; see Appendix D.3. Moreover, we show that M-SGDA-RA outperforms in the experiments other methods that require large batches.
2.3 Random Checks of Computations
We start with the Stochastic Gradient Descent-Accent with Checks of Computations (SGDA-CC). At each iteration of SGDA-CC, the server selects  $m$  workers (uniformly at random) and requests them to check the computations of other  $m$  workers from the previous iteration. Let  $V_{t}$  be the set of workers that verify/check computations,  $A_{t}$  are active workers at iteration  $t$ , and  $V_{t} \cap A_{t} = \emptyset$ . Then, the update of SGDA-CC can be written as
$$
\boldsymbol {x} ^ {t + 1} = \boldsymbol {x} ^ {t} - \gamma \overline {{\boldsymbol {g}}} ^ {t}, \text {i f} \overline {{\boldsymbol {g}}} ^ {t} = \frac {1}{| A _ {t} |} \sum_ {i \in A _ {t}} \boldsymbol {g} _ {i} (\boldsymbol {x} ^ {t}, \boldsymbol {\xi} _ {i} ^ {t}) \quad \text {i s a c c e p t e d},
$$
where  $\{\pmb{g}_i(\pmb{x}^t,\pmb{\xi}_i^t)\}_{i\in \mathcal{G}}$  are sampled independently.
The acceptance (of the update) event occurs when the condition  $\left\| \overline{\boldsymbol{g}}^t - \boldsymbol{g}_i(\boldsymbol{x}^t, \boldsymbol{\xi}_i^t)\right\| \leq C\sigma$  holds for the majority of workers. If  $\overline{\boldsymbol{g}}^t$  is rejected, then all workers re-sample  $\boldsymbol{g}_i(\boldsymbol{x}^t, \boldsymbol{\xi}_i^t)$  until acceptance is achieved. The rejection probability is bounded, as per [Gorbunov et al., 2022b], and can be adjusted by choosing a constant  $C = \mathcal{O}(1)$ . We assume that the server knows the seeds for generating randomness on workers, and thus, verification of computations is possible. Following each aggregation of  $\boldsymbol{g}_i(\boldsymbol{x}^t, \boldsymbol{\xi}_i^t)_{i \in \mathcal{G}}$ , the server selects uniformly at random  $2m$  workers:  $m$  workers check the computations at the previous step of the other  $m$  workers. For instance, at the  $(t + 1)$ -th iteration, the server asks a checking peer  $i$  to compute  $\boldsymbol{g}_j(\boldsymbol{x}^t, \boldsymbol{\xi}_j^t)$ , where  $j$  is a peer being checked. This is possible if all seeds are broadcasted at the start of the training. Workers assigned to checking do not participate in the training while they check and do not contribute to  $\overline{\boldsymbol{g}}^t$ . Therefore, each Byzantine peer is checked at each iteration with a probability of  $\sim m/n$  by some good worker (see the proof of Theorem 4). If the results are mismatched, then both the checking and checked peers are removed from training.
This design ensures that every such mismatch, whether it is caused by honest or Byzantine peers, eliminates at least one Byzantine peer and at most one honest peer (see details in Appendix E.1). It's worth noting that we assume any information is accessible to Byzantines except when each of them will be checked. As such, Byzantine peers can only reduce their relative numbers, which leads us to the main result for SGDA-CC, which is presented below.
Theorem 4. Let Assumptions 1, 4 and 6 hold. Then after  $T$  iterations SGDA-CC (Algorithm 5) with  $\gamma \leq \frac{1}{2\ell}$  outputs  $\pmb{x}^T$  such that
$$
\mathbb {E} \left\| \boldsymbol {x} ^ {T + 1} - \boldsymbol {x} ^ {*} \right\| ^ {2} \leq \left(1 - \frac {\gamma \mu}{2}\right) ^ {T + 1} \left\| \boldsymbol {x} ^ {0} - \boldsymbol {x} ^ {*} \right\| ^ {2} + \frac {4 \gamma \sigma^ {2}}{\mu (n - 2 B - m)} + \frac {2 q \sigma^ {2} n B}{m} \left(\frac {\gamma}{\mu} + \gamma^ {2}\right),
$$
where  $q = 2C^2 +12 + \frac{12}{n - 2B - m}$  .  $q = \mathcal{O}(1)$  since  $C = \mathcal{O}(1)$
The above theorem (see Appendix E.1 for the proof) provides the first result that does not require large batchesizes to converge to any predefined accuracy. The first and the second terms in the convergence bound correspond to the SOTA results for SGDA [Loizou et al., 2021]. Similarly to the vanilla SGDA, the convergence can be obtained by decreasing stepsize, however, such an approach does not benefit from collaboration, since the dominating term  $\frac{\gamma\sigma^2nB}{\mu m}$  (coming from the presence of Byzantine workers) is not inversely dependent on  $n$ . Moreover, the result is even worse than for single node SGDA in terms of dependence on  $n$ .
In contrast to Theorems 1-2, the result from Theorem 3 is given for the averaged iterate. We consider the averaged iterate to make the analysis simpler. We believe that one can extend the analysis to the last iterate as well, but we do not do it since we expect that the same problem (the need for large batches) will remain in the last-iterate analysis.
To overcome this issue we consider the restart technique for SGDA-CC and propose the next algorithm called R-SGDA-CC. This method consists of  $r$  stages. On the  $t$ -th stage R-SGDA-CC runs SGDA-CC with  $\gamma_{t}$  for  $K_{t}$  iterations from the starting point  $\widehat{\boldsymbol{x}}^t$ , which is the output from the previous stage, and defines the obtained point as  $\widehat{\boldsymbol{x}}^{t + 1}$  (see details in Appendix E.2). The main result for R-SGDA-CC is given below.
Theorem 5. Let Assumptions 1, 4 and 6 hold. Then, after  $r = \left\lceil \log_2 \frac{R^2}{\varepsilon} \right\rceil - 1$  restarts R-SGDA-CC (Algorithm 6) with  $\gamma_t = \min \left\{ \frac{1}{2\ell}, \sqrt{\frac{(n - 2B - m)R^2}{6\sigma^2 2^t K_t}}, \sqrt{\frac{m^2 R^2}{72q\sigma^2 2^t B^2 n^2}} \right\}$  and  $K_t = \left\lceil \max \left\{ \frac{8\ell}{\mu}, \frac{96\sigma^2 2^t}{(n - 2B - m)\mu^2 R^2}, \frac{34n\sigma B \sqrt{q 2^t}}{m\mu R} \right\} \right\rceil$ , where  $R \geq \| x^0 - x^* \|$ , outputs  $\widehat{x}^r$  such that  $\mathbb{E}\| \widehat{x}^r - x^* \|^2 \leq \varepsilon$ . Moreover, the total number of executed iterations of SGDA-CC is
$$
\sum_ {t = 1} ^ {r} K _ {t} = \mathcal {O} \left(\frac {\ell}{\mu} \log \frac {\mu R _ {0} ^ {2}}{\varepsilon} + \frac {\sigma^ {2}}{(n - 2 B - m) \mu \varepsilon} + \frac {n B \sigma}{m \sqrt {\mu \varepsilon}}\right). \tag {5}
$$
The above result implies that R-SGDA-CC also converges to any accuracy without large batch-sizes (see Appendix E.2 for details). However, as the accuracy tends to zero, the dominant term  $\frac{\sigma^2}{(n - 2B - m)\mu\varepsilon}$  inversely depends on the number of workers. This makes R-SGDA-CC benefit from collaboration, as the algorithm becomes more efficient with an increasing number of workers. Moreover, when  $B$  and  $m$  are small the derived complexity result for R-SGDA-CC matches the one for parallel SGDA [Loizou et al., 2021], which is obtained for the case of no Byzantine workers.
Next, we present a modification of Stochastic Extragradient with Checks of Computations (SEG-CC):
$$
\widetilde {\boldsymbol {x}} ^ {t} = \boldsymbol {x} ^ {t} - \gamma_ {1} \overline {{\boldsymbol {g}}} _ {\boldsymbol {\xi}} ^ {t}, \quad \text {i f} \overline {{\boldsymbol {g}}} _ {\boldsymbol {\xi}} ^ {t} = \frac {1}{| A _ {t} |} \sum_ {i \in A _ {t}} \boldsymbol {g} _ {i} (\boldsymbol {x} ^ {t}, \boldsymbol {\xi} _ {i} ^ {t}) \text {i s a c c e p t e d},
$$
$$
\boldsymbol {x} ^ {t + 1} = \boldsymbol {x} ^ {t} - \gamma_ {2} \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta}} ^ {t}, \quad \text {i f} \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta}} ^ {t} = \frac {1}{| A _ {t} |} \sum_ {i \in A _ {t}} \boldsymbol {g} _ {i} (\widetilde {\boldsymbol {x}} ^ {t}, \boldsymbol {\eta} _ {i} ^ {t}) \quad \text {i s a c c e p t e d},
$$
where  $\{\pmb{g}_i(\pmb{x}^t,\pmb{\xi}_i^t)\}_{i\in \mathcal{G}}$  and  $\{\pmb {g}_i(\widetilde{\pmb{x}}^t,\pmb {\eta}_i^t)\}_{i\in \mathcal{G}}$  are sampled independently. The events of acceptance  $\overline{\pmb{g}}_{\pmb{\eta}}^{t}$  (or  $\overline{\pmb{g}}_{\pmb{\xi}}^{t}$ ) happens if
$$
\left\| \overline {{\boldsymbol {g}}} ^ {t} - \boldsymbol {g} _ {i} (\boldsymbol {x} ^ {t}, \boldsymbol {\xi} _ {i} ^ {t}) \right\| \leq C \sigma \left(\text {o r} \| \overline {{\boldsymbol {g}}} _ {\boldsymbol {\eta}} ^ {t} - \boldsymbol {g} _ {i} (\widetilde {\boldsymbol {x}} ^ {t}, \boldsymbol {\eta} _ {i} ^ {t}) \| \leq C \sigma\right)
$$
holds for the majority of workers. An iteration of SEG-CC actually represents two subsequent iteration of SGDA-CC, so we refer to the beginning of the section for more details. Our main convergence results for SEG-CC are summarized in the following theorem; see Appendix E.3 for the proof.
Theorem 6. Let Assumptions 1, 3 and 4 hold. Then after  $T$  iterations SEG-CC (Algorithm 7) with  $\gamma_1 \leq \frac{1}{2\mu + 2L}$  and  $\beta = \gamma_2 / \gamma_1 \leq 1/4$  outputs  $\mathbf{x}^T$  such that
$$
\mathbb {E} \left\| \boldsymbol {x} ^ {T} - \boldsymbol {x} ^ {*} \right\| ^ {2} \leq \left(1 - \frac {\mu \beta \gamma_ {1}}{4}\right) ^ {T} \left\| \boldsymbol {x} ^ {0} - \boldsymbol {x} ^ {*} \right\| ^ {2} + 2 \sigma^ {2} \left(\frac {4 \gamma_ {1}}{\beta \mu^ {2} (n - 2 B - m)} + \frac {\gamma_ {1} q n B}{m}\right),
$$
Similarly to SGDA-CC, SEG-CC does not require large batchsizes to converge to any predefined accuracy and does not benefit of collaboration, though the first two terms correspond to the SOTA convergence results for SEG under bounded variance assumption [Juditsky et al., 2011]. The last term appears due to the presence of the Byzantine workers. The restart technique can also be applied; see Appendix E.4 for the proof.
Theorem 7. Let Assumptions 1, 3, 4 hold. Then, after  $r = \left\lceil \log_2 \frac{R^2}{\varepsilon} \right\rceil - 1$  restarts R-SEG-CC (Algothm 8) with  $\gamma_{1_t} = \min \left\{\frac{1}{2L}, \sqrt{\frac{(G-B-m)R^2}{16\sigma^22^tK_t}}, \sqrt{\frac{mR^2}{8q\sigma^22^tBn}}\right\}$ ,  $\gamma_{2_t} = \min \left\{\frac{1}{4L}, \sqrt{\frac{m^2R^2}{64q\sigma^22^tB^2n^2}}, \sqrt{\frac{(G-B-m)R^2}{64\sigma^2K_t}}\right\}$  and  $K_t = \left\lceil \max \left\{\frac{8L}{\mu}, \frac{16n\sigma B\sqrt{q^2t}}{m\mu R}, \frac{256\sigma^22^t}{(G-B-m)\mu^2R^2}\right\} \right\rceil$ , where  $R \geq \| x^0 - x^*\|$  outputs  $\widehat{x}^r$  such that  $\mathbb{E}\|\widehat{x}^r - x^*\|^2 \leq \varepsilon$ . Moreover, the total number of executed iterations of SEG-CC is
$$
\sum_ {t = 1} ^ {r} K _ {t} = \mathcal {O} \left(\frac {\ell}{\mu} \log \frac {\mu R _ {0} ^ {2}}{\varepsilon} + \frac {\sigma^ {2}}{(n - 2 B - m) \mu \varepsilon} + \frac {n B \sigma}{m \sqrt {\mu \varepsilon}}\right). \tag {6}
$$
The above result states that R-SEG-CC also converges to any accuracy without large batchsizes; see Appendix E.4. But with accuracy tending to zero ( $\varepsilon \to 0$ ) the dominating term  $\frac{\sigma^2}{(n - 2B - m)\mu\varepsilon}$  inversely depends on the number of workers, hence R-SEG-CC benefits from collaboration. Moreover, when  $B$  and  $m$  are small the derived complexity result for R-SEG-CC matches the one for parallel/minibatched SEG [Juditsky et al., 2011], which is obtained for the case of no Byzantine workers.
3 Numerical Experiments
Quadratic game. To illustrate our theoretical results, we conduct numerical experiments on a quadratic game
$$
\min _ {y} \max _ {z} \frac {1}{s} \sum_ {i = 1} ^ {s} \frac {1}{2} y ^ {\top} \mathbf {A} _ {1, i} y + y ^ {\top} \mathbf {A} _ {2, i} z - \frac {1}{2} z ^ {\top} \mathbf {A} _ {3, i} z + b _ {1, i} ^ {\top} y - b _ {2, i} ^ {\top} z.
$$
The above problem can be re-formulated as a special case of (1) with  $F$  defined as follows:
$$
F (\boldsymbol {x}) = \frac {1}{s} \sum_ {i = 1} ^ {s} \mathbf {A} _ {i} \boldsymbol {x} + b _ {i}, \text {w h e r e} \boldsymbol {x} = (y ^ {\top}, z ^ {\top}) ^ {\top}, b _ {i} = \left(b _ {1, i} ^ {\top}, b _ {2, i} ^ {\top}\right) ^ {\top}, \tag {7}
$$
with symmetric matrices  $\mathbf{A}_{j,i}$  s.t.  $\mu \mathbf{I} \preccurlyeq \mathbf{A}_{j,i} \preccurlyeq \ell \mathbf{I}$ ,  $\mathbf{A}_i \in \mathbb{R}^{d \times d}$  and  $b_i \in \mathbb{R}^d$ ; see Appendix F for the detailed description.
We set  $\ell = 100$ ,  $\mu = 0.1$ ,  $s = 1000$  and  $d = 50$ . Only one peer checked the computations on each iteration ( $m = 1$ ). We used RFA (geometric median) with bucketing as an aggregator since it showed the best performance. For approximating the median we used Weiszfeld's method with 10 iterations and parameter  $\nu = 0.1$  [Pillutla et al., 2022]. RDEG [Adibi et al., 2022] provably works only if  $n \geq 100$ , so here we provide experiments with  $n = 150$ ,  $B = 20$ ,  $\gamma = 2e - 5$ . We set the parameter  $\alpha = 0.1$  for M-SGDA-RA, and the following parameters for RDEG:  $\alpha_{\mathrm{RDEG}} = 0.06$ ,  $\delta_{\mathrm{RDEG}} = 0.9$  and theoretical value of  $\epsilon$ ; see Appendix F for more experiments. We tested the algorithms under the following attacks: bit flipping (BF), random noise (RN), inner product manipulation (IPM) Xie et al. [2019] and "a little is enough" (ALIE) Baruch et al. [2019].
Robust Neural Networks training. Let  $f(u; x, y)$  be the loss function of a neural network with parameters  $u \in \mathbb{R}^d$  given input  $x \in \mathbb{R}^m$  and label  $y$ . For example, in our experiments, we let  $f$  be the cross entropy loss, and  $\{(x_i, y_i)\}_1^N$  is the MNIST dataset. Now consider the following objective:
$$
\min  _ {u \in \mathbb {R} ^ {d}} \max  _ {v \in \mathbb {R} ^ {m}} \frac {1}{N} \sum_ {i = 1} ^ {N} f (u; x _ {i} + v, y _ {i}) + \frac {\lambda_ {1}}{2} \| u \| _ {2} ^ {2} - \frac {\lambda_ {2}}{2} \| v \| _ {2} ^ {2}. \tag {8}
$$
This min-max objective adds an extra adversarial noise variable to the input data such that it maximizes the loss, so the neural network should become robust to such noise as it minimizes the loss. We can reformulate this objective as a variational inequality with
$$
\boldsymbol {x} = \binom {u} {v}, \quad F _ {i} (\boldsymbol {x}) = \binom {\nabla_ {u} f (u ; x _ {i} + v, y _ {i}) + \lambda_ {1} u} {- \nabla_ {v} f (u ; x _ {i} + v, y _ {i}) + \lambda_ {2} v}, \quad F (\boldsymbol {x}) = \frac {1}{N} \sum_ {i = 1} ^ {N} F _ {i} (\boldsymbol {x}). \tag {9}
$$
We let  $n = 20$ ,  $B = 4$ ,  $\lambda_1 = 0$ , and  $\lambda_2 = 100$ . We fix the learning rate to 0.01 and use a batch size of 32. We run the algorithm for 50 epochs and average our results across 3 runs. We test the algorithms under the following attacks: i) bit flipping (BF), ii) label flipping (LF), iii) inner product manipulation (IPM) Xie et al. [2019], and iv) a little is enough (ALIE) Baruch et al. [2019]. We compare our algorithm SGDA-CC against the following algorithms: i) SGDA-RA, ii) M-SGDA-RA, and iii) RDEG Adibi et al. [2022]. We use RFA with bucket size 2 as the robust aggregator. The results are shown in Figure 2. Specifically, we show the validation error on MNIST after each epoch. We can see that SGDA-CC performs the best, followed closely by M-SGDA-RA.
This paper proposes several new algorithms for Byzantine-robust distributed variational inequalities and provides rigorous theoretical convergence analysis for the developed methods. In particular, we
Figure 1: Error plots for quadratic games experiments under different Byzantine attacks. The first row shows the outperformance of M-SGDA-RA over methods without checks of computations. The second row illustrates advantages of SGDA-CC and SEG-CC.
Figure 2: Error plots for the robust neural network experiment on MNIST under different byzantine attacks (BF, LF, IPM, and ALIE). Each algorithm is shown with a consistent choice of color and style across plots, as indicated in the legends.
propose the first methods in this setting that provably converge to any predefined accuracy in the case of homogeneous data. We believe this is an important step towards building a strong theory of Byzantine robustness in the case of distributed VIs.
However, our work has several limitations. First of all, one can consider different/more general assumptions about operators [Beznosikov et al., 2023, Gorbunov et al., 2022a, 2023b] in the analysis of the proposed methods. Next, as we mention in the discussion after Theorem 3, our result for M-SGDA-RA requires large batchesizes, and it remains unclear to us whether this requirement can be removed. Finally, the only results that do not require large batchesizes are derived using the checks of computations that create (although small) computation overhead. Obtaining similar results without checks of computations remains an open problem. Addressing these limitations is a prominent direction for future research.
This work of N. Tupitsa was supported by a grant for research centers in the field of artificial intelligence, provided by the Analytical Center for the Government of the Russian Federation in accordance with the subsidy agreement (agreement identifier 000000D730321P5Q0002) and the agreement with the Moscow Institute of Physics and Technology dated November 1, 2021 No. 70-2021-00138.