SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\langle \nabla \mathcal {L} _ {\mathcal {H}} (\theta_ {t}), R _ {t} \rangle = \frac {1}{2} \left(\| \nabla \mathcal {L} _ {\mathcal {H}} (\theta_ {t}) \| ^ {2} + \| R _ {t} \| ^ {2} - \| \nabla \mathcal {L} _ {\mathcal {H}} (\theta_ {t}) - R _ {t} \| ^ {2}\right).
$$
Substituting the above in (49) we obtain that
$$
\begin{array}{l} \mathcal {L} _ {\mathcal {H}} (\theta_ {t + 1}) - \mathcal {L} _ {\mathcal {H}} (\theta_ {t}) \leq - \frac {\gamma}{2} \left(\| \nabla \mathcal {L} _ {\mathcal {H}} (\theta_ {t}) \| ^ {2} + \| R _ {t} \| ^ {2} - \| \nabla \mathcal {L} _ {\mathcal {H}} (\theta_ {t}) - R _ {t} \| ^ {2}\right) + \frac {1}{2} \gamma^ {2} L \| R _ {t} \| ^ {2} \\ = - \frac {\gamma}{2} \left\| \nabla \mathcal {L} _ {\mathcal {H}} (\theta_ {t}) \right\| ^ {2} - \frac {\gamma}{2} (1 - \gamma L) \left\| R _ {t} \right\| ^ {2} + \frac {\gamma}{2} \left\| \nabla \mathcal {L} _ {\mathcal {H}} (\theta_ {t}) - R _ {t} \right\| ^ {2}. \\ \end{array}
$$
Substituting  $\gamma = \frac{1}{L}$  in the above we obtain that
$$
\mathcal {L} _ {\mathcal {H}} \left(\theta_ {t + 1}\right) - \mathcal {L} _ {\mathcal {H}} \left(\theta_ {t}\right) \leq - \frac {1}{2 L} \left\| \nabla \mathcal {L} _ {\mathcal {H}} \left(\theta_ {t}\right) \right\| ^ {2} + \frac {1}{2 L} \left\| R _ {t} - \nabla \mathcal {L} _ {\mathcal {H}} \left(\theta_ {t}\right) \right\| ^ {2}. \tag {50}
$$
As we assume the aggregation  $F$  to satisfy  $(f,\kappa)$ -robustness, by Definition 4, we also have that
$$
\begin{array}{l} \| R _ {t} - \nabla \mathcal {L} _ {\mathcal {H}} (\theta_ {t}) \| ^ {2} = \left\| F \Big (g _ {t} ^ {(1)}, \ldots , g _ {t} ^ {(n)} \Big) - \frac {1}{| \mathcal {H} |} \sum_ {i \in \mathcal {H}} g _ {t} ^ {(i)} \right\| ^ {2} \leq \frac {\kappa}{| \mathcal {H} |} \sum_ {i \in \mathcal {H}} \left\| g _ {t} ^ {(i)} - \frac {1}{| \mathcal {H} |} \sum_ {i \in \mathcal {H}} g _ {t} ^ {(i)} \right\| ^ {2} \\ = \frac {\kappa}{| \mathcal {H} |} \sum_ {i \in \mathcal {H}} \| \nabla \mathcal {L} _ {i} \left(\theta_ {t}\right) - \nabla \mathcal {L} _ {\mathcal {H}} \left(\theta_ {t}\right) \| ^ {2}. \tag {51} \\ \end{array}
$$
Besides, Assumption 1 implies that for all  $\theta \in \mathbb{R}^d$  we have
Using the above in (51) yields
$$
\left\| R _ {t} - \nabla \mathcal {L} _ {\mathcal {H}} (\theta_ {t}) \right\| ^ {2} \leq \kappa G ^ {2} + \kappa B ^ {2} \left\| \nabla \mathcal {L} _ {\mathcal {H}} (\theta_ {t}) \right\| ^ {2}.
$$
Substituting the above in (50) yields
$$
\mathcal {L} _ {\mathcal {H}} \left(\theta_ {t + 1}\right) - \mathcal {L} _ {\mathcal {H}} \left(\theta_ {t}\right) \leq - \frac {1}{2 L} \left\| \nabla \mathcal {L} _ {\mathcal {H}} \left(\theta_ {t}\right) \right\| ^ {2} + \frac {1}{2 L} \left(\kappa G ^ {2} + \kappa B ^ {2} \left\| \nabla \mathcal {L} _ {\mathcal {H}} \left(\theta_ {t}\right) \right\| ^ {2}\right). \tag {52}
$$
Multiplying both sides in (52) by  $2L$  and rearranging the terms, we get
$$
\left(1 - \kappa B ^ {2}\right) \left\| \nabla \mathcal {L} _ {\mathcal {H}} \left(\theta_ {t}\right) \right\| ^ {2} \leq \kappa G ^ {2} + 2 L \left(\mathcal {L} _ {\mathcal {H}} \left(\theta_ {t}\right) - \mathcal {L} _ {\mathcal {H}} \left(\theta_ {t + 1}\right)\right). \tag {53}
$$
Recall that  $t$  in the above was arbitrary in  $\{0, \dots, T - 1\}$ . Averaging over all  $t \in \{0, \dots, T - 1\}$  yields
$$
\begin{array}{l} \left(1 - \kappa B ^ {2}\right) \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla \mathcal {L} _ {\mathcal {H}} (\theta_ {t}) \| ^ {2} \leq \kappa G ^ {2} + \frac {2 L}{T} \sum_ {t = 0} ^ {T - 1} (\mathcal {L} _ {\mathcal {H}} (\theta_ {t}) - \mathcal {L} _ {\mathcal {H}} (\theta_ {t + 1})) = \kappa G ^ {2} + \frac {2 L}{T} (\mathcal {L} _ {\mathcal {H}} (\theta_ {0}) - \mathcal {L} _ {\mathcal {H}} (\theta_ {T})) \\ \leq \kappa G ^ {2} + \frac {2 L}{T} \left(\mathcal {L} _ {\mathcal {H}} (\theta_ {0}) - \mathcal {L} _ {* , \mathcal {H}}\right). \\ \end{array}
$$
Finally, since we assume that  $1 - \kappa B^{2} > 0$ , dividing both sides in the above by  $1 - \kappa B^{2}$  yields
The above concludes the proof for the non-convex case.
C.1.2 Strongly Convex Case
Proof. Assume now that  $\mathcal{L}_{\mathcal{H}}$  is  $\mu$ -PL. Following the proof of the non-convex case up until (53) yields
$$
\begin{array}{l} \left(1 - \kappa B ^ {2}\right) \| \nabla \mathcal {L} _ {\mathcal {H}} \left(\theta_ {t}\right) \| ^ {2} \leq \kappa G ^ {2} + 2 L \left(\mathcal {L} _ {\mathcal {H}} \left(\theta_ {t}\right) - \mathcal {L} _ {\mathcal {H}} \left(\theta_ {t + 1}\right)\right) \\ = \kappa G ^ {2} + 2 L \left(\mathcal {L} _ {\mathcal {H}} \left(\theta_ {t}\right) - \mathcal {L} _ {*}, \mathcal {H} + \mathcal {L} _ {*}, \mathcal {H} - \mathcal {L} _ {\mathcal {H}} \left(\theta_ {t + 1}\right)\right). \\ \end{array}
$$
$$
2 L \left(\mathcal {L} _ {\mathcal {H}} \left(\theta_ {t + 1}\right) - \mathcal {L} _ {*, \mathcal {H}}\right) \leq \kappa G ^ {2} - \left(1 - \kappa B ^ {2}\right) \left\| \nabla \mathcal {L} _ {\mathcal {H}} \left(\theta_ {t}\right) \right\| ^ {2} + 2 L \left(\mathcal {L} _ {\mathcal {H}} \left(\theta_ {t}\right) - \mathcal {L} _ {*, \mathcal {H}}\right).
$$
Since  $\mathcal{L}_{\mathcal{H}}$  is  $\mu$ -PL, as per Definition 3, we obtain
$$
\begin{array}{l} 2 L \left(\mathcal {L} _ {\mathcal {H}} \left(\theta_ {t + 1}\right) - \mathcal {L} _ {*}, \mathcal {H}\right) \leq \kappa G ^ {2} - 2 \mu \left(1 - \kappa B ^ {2}\right) \left(\mathcal {L} _ {\mathcal {H}} \left(\theta_ {t}\right) - \mathcal {L} _ {*}, \mathcal {H}\right) + 2 L \left(\mathcal {L} _ {\mathcal {H}} \left(\theta_ {t}\right) - \mathcal {L} _ {*}, \mathcal {H}\right) \\ = \kappa G ^ {2} + \left(2 L - 2 \mu \left(1 - \kappa B ^ {2}\right)\right) \left(\mathcal {L} _ {\mathcal {H}} \left(\theta_ {t}\right) - \mathcal {L} _ {*}, \mathcal {H}\right). \\ \end{array}
$$
Dividing both sides by  $2L$ , we get
$$
\mathcal {L} _ {\mathcal {H}} \left(\theta_ {t + 1}\right) - \mathcal {L} _ {*, \mathcal {H}} \leq \frac {\kappa G ^ {2}}{2 L} + \left(1 - \frac {\mu}{L} \left(1 - \kappa B ^ {2}\right)\right) \left(\mathcal {L} _ {\mathcal {H}} \left(\theta_ {t}\right) - \mathcal {L} _ {*, \mathcal {H}}\right). \tag {54}
$$
Recall that  $t$  is arbitrary in  $\{0, \dots, T - 1\}$ . Then, applying (54) recursively (on the right hand side) yields
$$
\begin{array}{l} \mathcal {L} _ {\mathcal {H}} (\theta_ {t + 1}) - \mathcal {L} _ {*, \mathcal {H}} \leq \frac {\kappa G ^ {2}}{2 L} \sum_ {k = 0} ^ {t} \left(1 - \frac {\mu}{L} \left(1 - \kappa B ^ {2}\right)\right) ^ {k} \\ + \left(1 - \frac {\mu}{L} \left(1 - \kappa B ^ {2}\right)\right) ^ {t + 1} (\mathcal {L} _ {\mathcal {H}} (\theta_ {0}) - \mathcal {L} _ {*, \mathcal {H}}) \\ \leq \frac {\kappa G ^ {2}}{2 L} \frac {1}{1 - \left(1 - \frac {\mu}{L} (1 - \kappa B ^ {2})\right)} + \left(1 - \frac {\mu}{L} (1 - \kappa B ^ {2})\right) ^ {t + 1} (\mathcal {L} _ {\mathcal {H}} (\theta_ {0}) - \mathcal {L} _ {\ast , \mathcal {H}}) \\ = \frac {\kappa G ^ {2}}{2 \mu (1 - \kappa B ^ {2})} + \left(1 - \frac {\mu}{L} (1 - \kappa B ^ {2})\right) ^ {t + 1} (\mathcal {L} _ {\mathcal {H}} (\theta_ {0}) - \mathcal {L} _ {*, \mathcal {H}}). \\ \end{array}
$$
Using the fact that  $(1 + x)^n \leq e^{nx}$  for all  $x \in \mathbb{R}$  and substituting  $t = T - 1$  yields
Corollary 1. Assume that the global loss  $\mathcal{L}_{\mathcal{H}}$  is  $\mu$ -PL and  $L$ -smooth, and that for each  $i \in \mathcal{H}$  local loss  $\mathcal{L}_i$  is convex and  $L_i$ -smooth. Denote  $L_{\max} := \max_{i \in \mathcal{H}} L_i$  and assume that  $\kappa\left(\frac{3L_{\max}}{\mu} - 1\right) \leq 1$ . Consider Algorithm 1 with learning rate  $\gamma = \frac{1}{L}$ . If  $F$  is  $(f, \kappa)$ -robust, then for all  $T \geq 1$ , we have
$$
\mathcal {L} _ {\mathcal {H}} (\theta_ {T}) - \mathcal {L} _ {*, \mathcal {H}} \leq \frac {3 \kappa}{\mu} \frac {1}{| \mathcal {H} |} \sum_ {i \in \mathcal {H}} \| \nabla \mathcal {L} _ {i} (\theta_ {*}) \| ^ {2} + e ^ {- \frac {\mu}{3 L} T} (\mathcal {L} _ {\mathcal {H}} (\theta_ {0}) - \mathcal {L} _ {*, \mathcal {H}}).
$$
Proof. Invoking Proposition 1, we know that the loss functions satisfy  $(G,B)$ -gradient dissimilarity with
$$
G ^ {2} = \frac {2}{| \mathcal {H} |} \sum_ {i \in \mathcal {H}} \| \nabla \mathcal {L} _ {i} (\theta_ {*}) \| ^ {2}, \qquad B ^ {2} = 2 \frac {L _ {\max }}{\mu} - 1.
$$
Moreover, since we assume that  $\kappa(3\frac{L_{\max}}{\mu} - 1) \leq 1$ , we have
$$
1 - \kappa B ^ {2} = 1 - \kappa (2 \frac {L _ {\mathrm {m a x}}}{\mu} - 1) = 1 - \frac {2}{3} \kappa (3 \frac {L _ {\mathrm {m a x}}}{\mu} - 1) + \frac {\kappa}{3} \geq 1 - \frac {2}{3} + \frac {\kappa}{3} \geq \frac {1}{3}.
$$
Therefore, since the global loss  $\mathcal{L}_{\mathcal{H}}$  is  $\mu$ -PL and  $L$ -smooth, we can apply Theorem 2 to obtain
$$
\begin{array}{l} \mathcal {L} _ {\mathcal {H}} \left(\theta_ {T}\right) - \mathcal {L} _ {*}, \mathcal {H} \leq \frac {\kappa G ^ {2}}{2 \mu \left(1 - \kappa B ^ {2}\right)} + e ^ {- \frac {\mu}{L} \left(1 - \kappa B ^ {2}\right) T} \left(\mathcal {L} _ {\mathcal {H}} \left(\theta_ {0}\right) - \mathcal {L} _ {*}, \mathcal {H}\right) \\ \leq \frac {3}{2 \mu} \kappa G ^ {2} + e ^ {- \frac {\mu}{3 L} T} \left(\mathcal {L} _ {\mathcal {H}} \left(\theta_ {0}\right) - \mathcal {L} _ {*}\right) \\ = \frac {3 \kappa}{\mu} \frac {1}{| \mathcal {H} |} \sum_ {i \in \mathcal {H}} \| \nabla \mathcal {L} _ {i} (\theta_ {*}) \| ^ {2} + e ^ {- \frac {\mu}{3 L} T} (\mathcal {L} _ {\mathcal {H}} (\theta_ {0}) - \mathcal {L} _ {*}, \mathcal {H}). \\ \end{array}
$$
In this section, we present the full experimental setups of the experiments in Figures 1,2,3, and 4. All our experiments were conducted on the following hardware: MacBook Pro, Apple M1 chip, 8-core CPU and 2 NVIDIA A10-24GB GPUs. Our code is available online through this link.
D.1 Figure 1: brittleness of  $G$ -Gradient Dissimilarity
This first experiment aims to show the gap between existing theory and practice. While in theory  $G$ -gradient dissimilarity does not cover the least square regression problem, we show that it is indeed possible to converge using the Robust D-GD Algorithm with the presence of 3 Byzantine workers out of 10 total workers. All the hyperparameters of this experiments are listed in Table 1.
Number of Byzantine workers	f = 3
Number of honest workers	n - f = 7
Dataset	n - f datapoints in mg LIBSVM [10]
selected uniformly without replacement
Data heterogeneity	Each honest worker holds one distinct point
Model	Linear regression
Algorithm	Robust D-GD
Number of steps	T = 40000
Learning rate	γ = 0.001
Loss function	Regularized Mean Least Square error
l2-regularization term	λ = 1/√m(n-f) = 1/√7
Aggregation rule	NNM [3] coupled with CW Trimmed Mean [34]
Byzantine attacks	sign flipping [2], fall of empires [33],
a little is enough [5] and mimic [20]
Table 1: Setup of Figure 1's experiment
D.2 Figure 2: empirical breakdown point
The second experiment we conduct tends to highlight the empirical breakdown point observed in practice when the fraction of Byzantines becomes too high. Indeed, while existing theory suggests that this breakdown point occurs for a fraction of  $1/2$  Byzantine workers, we show empirically that it occurs even before having  $1/4$  Byzantines. We present all the hyperparameters used for this experiment in Table 2.
Number of Byzantine workers	from f = 1 to f = 9
Number of honest workers	n - f = 10
Dataset	10% of MNIST selected uniformly without replacement
Data heterogeneity	Each worker dataset holds data from a distinct class
Model	Logistic regression
Algorithm	Robust D-GD
Number of steps	T = 500
Learning rate	γ = {0.05 if 0 ≤ T < 350
0.01 if 350 ≤ T < 420
0.002 if 420 ≤ T < 480
0.0004 if 480 ≤ T < 500
Loss function	Negative Log Likelihood (NLL)
l2-regularization term	10-4
Aggregation rule	NNM [3] & CW Trimmed Mean [34], Krum [7],
CW Median [34] or geometric median [30]
Byzantine attacks	sign flipping [2]
Table 2: Setup of Figure 2's experiment
D.3 Figure 3: comparing theoretical upper bounds
In the third experiment, we compare two errors bounds, i.e.,  $f / n - (2 + B^2)f \cdot G^2$  and  $f / n - 2f \cdot \widehat{G}^2$ , guaranteed for robust D-GD under  $(G, B)$ -gradient dissimilarity and  $\widehat{G}$ -gradient dissimilarity, respectively, on the MNIST Dataset. We use a logistic regression model and the Negative Log Likelihood (NLL) loss function as the local loss function for each honest worker.
Before explaining how we compute  $\widehat{G}$  and  $(G,B)$ , let us recall that the local loss functions of honest workers are said to satisfy  $\widehat{G}$ -gradient dissimilarity if for all  $\theta \in \mathbb{R}^d$ , we have
$$
\frac {1}{| \mathcal {H} |} \sum_ {i \in \mathcal {H}} \| \nabla \mathcal {L} _ {i} (\theta) - \nabla \mathcal {L} _ {\mathcal {H}} (\theta) \| ^ {2} \leq \widehat {G} ^ {2}.
$$
Similarly, the local loss functions of honest workers are said to satisfy  $(G,B)$ -gradient dissimilarity if, for all  $\theta \in \mathbb{R}^d$ , we have
Evidently,  $\widehat{G}$  and  $(G,B)$  are difficult to compute since one has to explore the entire space to get tight values. In this paper, we present a first heuristic to compute approximate values of  $\widehat{G}$  and  $(G,B)$ : We first compute  $\theta_{\star}$  by running D-GD without Byzantine workers, then we choose a point  $\theta_0$ , that is arbitrarily far from  $\theta_{\star}$ . Here we choose  $\theta_0 = (1e4,\dots ,1e4)\in \mathbb{R}^d$  with  $d = 784$  for the MNIST dataset. Given  $\theta_{\star}$  and  $\theta_0$  we construct 101 points  $\theta_0,\ldots ,\theta_{100}$  between  $\theta_{\star}$  and  $\theta_0$  such that
$$
\theta_ {t} = \frac {t}{1 0 0} \times \theta_ {\star} + \left(1 - \frac {t}{1 0 0}\right) \times \theta_ {0}, \forall t \in \{0, 1, 2, \dots , 1 0 0 \},
$$
and then compute  $\frac{1}{|\mathcal{H}|}\sum_{i\in \mathcal{H}}\| \nabla \mathcal{L}_i(\theta_t) - \nabla \mathcal{L}_{\mathcal{H}}(\theta_t)\|^2$  and  $\| \nabla \mathcal{L}_{\mathcal{H}}(\theta_t)\|^2$  for all  $t\in \{0,1,2,\ldots ,100\}$ . We set
$$
\widehat {G} ^ {2} = \max _ {t \in \{0, 1, 2, \dots , 1 0 0 \}} \frac {1}{| \mathcal {H} |} \sum_ {i \in \mathcal {H}} \| \nabla \mathcal {L} _ {i} (\theta_ {t}) - \nabla \mathcal {L} _ {\mathcal {H}} (\theta_ {t}) \| ^ {2}.
$$
Also, to compute the couple  $(G,B)$ , we first consider 100 values of  $B^{\prime 2}\in \left[0,\frac{n - 2f}{f}\right]$ , then compute
$$
G _ {B ^ {\prime}} ^ {2} = \max _ {t \in \{0, 1, 2, \dots , 1 0 0 \}} \frac {1}{| \mathcal {H} |} \sum_ {i \in \mathcal {H}} \left\| \nabla \mathcal {L} _ {i} (\theta_ {t}) - \nabla \mathcal {L} _ {\mathcal {H}} (\theta_ {t}) \right\| ^ {2} - B ^ {\prime 2} \left\| \nabla \mathcal {L} _ {\mathcal {H}} (\theta_ {t}) \right\| ^ {2}.
$$
and finally set
$$
(G, B) = \underset {(G _ {B ^ {\prime}}, B ^ {\prime})} {\arg \min} \frac {f}{n - (2 + B ^ {\prime 2}) f} \cdot G _ {B ^ {\prime}} ^ {2}.
$$
In Figure 3, we show the different values of  $f / n - (2 + B^2)f \cdot G^2$  and  $f / n - 2f \cdot \widehat{G}^2$  for  $f = 1$  to  $f = 9$  where there is always  $n - f = 10$  honest workers.
D.4 Figure 4: matching empirical performances
In the last experiments of the paper, we compare the empirical error gap (left-hand side of Corollary 1) with the upper bound (right-hand side of Corollary 1).
Let us first recall Corollary 1:
We first give all the hyperparameters used for the learning in Table 3 and then explain how we computed  $\mu, L, \kappa$  and  $\theta_{\star}$ .
Number of Byzantine workers	from f = 1 to f = 19
Number of honest workers	n - f = 20
Dataset	n - f datapoints in mg LIBSVM [10]
selected uniformly without replacement
Data heterogeneity	Each honest worker holds m = 1 distinct point
Model	Linear regression
Algorithm	Robust D-GD
Number of steps	T = 40000
Learning rate	γ = 0.0001
Loss function	Regularized Mean Least Square error
l2-regularization term	λ = 1/√m(n-f) = 1/√7
Aggregation rule	NNM [3] coupled with CW Trimmed Mean [34]
Byzantine attacks	sign flipping [2], fall of empires [33],
a little is enough [5] and mimic [20]
Table 3: Setup of Figure 4's experiment
Computation of  $\mu$  and  $L$ . Let  $\mathbf{X}$  be the matrix that contains all the  $n - f$  vector data points and  $\lambda$  the  $\ell_2$ -regularization term, we have
$$
L = \operatorname {e i g} _ {\max } \left(\frac {1}{n - f} \mathbf {X} ^ {\top} \mathbf {X} + \lambda \mathbf {I}\right), \quad \mu = \operatorname {e i g} _ {\min } \left(\frac {1}{n - f} \mathbf {X} ^ {\top} \mathbf {X} + \lambda \mathbf {I}\right) \tag {55}
$$
where for any matrix  $\mathbf{A} \in \mathbb{R}^{d \times d}$ ,  $\mathrm{eig}_{\max}(\mathbf{A})$  and  $\mathrm{eig}_{\max}(\mathbf{A})$  refer to the maximum and minimum eigenvalue of  $\mathbf{A}$  respectively.
Computation of  $\kappa$ . We recall that by definition, an aggregation rule  $F\colon \mathbb{R}^{d\times n}\to \mathbb{R}^d$  is said to be  $(f,\kappa)$ -robust if for any vectors  $x_{1},\ldots ,x_{n}\in \mathbb{R}^{d}$ , and any set  $S\subseteq [n]$  of size  $n - f$ , the output  $\hat{x}\coloneqq F(x_1,\dots,x_n)$  satisfies the following:
where  $\overline{x}_S\coloneqq \frac{1}{|S|}\sum_{i\in S}x_i$
Hence, as done in [3], we estimate  $\kappa$  empirically as follows: We first compute for every step  $t\in \{0,\ldots ,T - 1\}$  and every attack  $a\in \{\mathrm{SF},\mathrm{FOE},\mathrm{ALIE},\mathrm{Mimic}\}$  the value of  $\kappa_{t,a}$  such that
$$
\kappa_ {t, a} = \frac {\left\| R _ {t , a} - \overline {{g}} _ {t , a} \right\| ^ {2}}{\frac {1}{| S |} \sum_ {i \in S} \left\| g _ {t , a} ^ {(i)} - \overline {{g}} _ {t , a} \right\| ^ {2}}
$$
where  $\overline{g}_{t,a} \coloneqq \frac{1}{|\mathcal{H}|}\sum_{i\in \mathcal{H}}g_{t,a}^{(i)}$  and  $R_{t,a}$  and  $g_{t,a}^{(i)}$  refer respectively to  $R_{t}$  and  $g_{t}^{(i)}$  when the attack  $a$  is used by the Byzantine workers. Then, we compute the empirical  $\kappa$  following:
$$
\kappa = \max_{\substack{t\in \{0,\ldots ,T - 1\} \\ a\in \{\mathrm{SF},\mathrm{FOE,ALIE,Mimic}\}}} \kappa_{t,a}.
$$
**Computation of  $\theta_{\star}$ .** We compute  $\theta_{\star}$  using the closed form of the solution of a Mean Least Square regression problem:
$$
\theta_ {\star} = \left(\mathbf {X} ^ {\top} \mathbf {X} + (n - f) \lambda \mathbf {I}\right) ^ {- 1} \mathbf {X} ^ {\top} \mathbf {y},
$$
where  $\mathbf{X} \in \mathbb{R}^{(n - f) \times d}$  is the matrix that contains the data points and  $\mathbf{y} \in \mathbb{R}^{n - f}$  the associated vector that contains the labels.