SlowGuess/NeurIPS_2023_OCR huggingface dataset:
One of the crucial components to the above deductions was finding the suitable triplets  $(\alpha, K, z)$  while preserving the  $(G, B)$ -gradient dissimilarity assumption (along with the smoothness and strong convexity assumptions) simultaneously in both the two scenarios, thereby establishing their validity. While the exact calculations are tedious, intuitively,  $B$  constrains the relative difference between the scale parameters  $\alpha$  and  $\frac{1}{K}$ , and  $G$  constrains the separation between the minima, i.e.  $\|z\|^2$ .
Extension to non-convex problems. The lower bound from Theorem 1 assumes that the given distributed algorithm satisfies  $(f, \varepsilon)$ -resilience, which means finding an  $\varepsilon$ -approximate minimizer of the global honest loss  $\mathcal{L}_{\mathcal{H}}$ . The latter may not be possible for the general case of smooth and non-convex functions. In that case we cannot seek an  $\varepsilon$ -approximate minimizer, but rather an  $\varepsilon$ -approximate stationary point [20, 3], i.e.,  $\hat{\theta}$  such that  $\| \nabla \mathcal{L}_{\mathcal{H}}(\hat{\theta}) \|^{2} \leq \varepsilon$ . Then the lower bound in Theorem 1, in conjunction with the  $\mu$ -PL inequality, yields the following lower bound
$$
\varepsilon \geq \frac {1}{4} \cdot \frac {f}{n - (2 + B ^ {2}) f} \cdot G ^ {2}. \tag {7}
$$
Comparison with prior work. The result of Theorem 1 generalizes the existing robustness limits derived under  $G$ -gradient dissimilarity [20]. In particular, setting  $B = 0$  in Theorem 1, we recover the breakdown point  $\frac{1}{2}$  and the optimization lower bound  $\Omega(f_n \cdot G^2)$ . Perhaps, the most striking contrast to prior work [12, 20, 13, 3] is our breakdown point  $\frac{1}{2 + B^2}$ , instead of simply  $\frac{1}{2}$ . We remark that a similar dependence on heterogeneity has been repeatedly assumed in the past, but without any formal justification. For instance, under  $(0, B)$ -gradient dissimilarity, [20, Theorem IV] assumes  $f_n = \mathcal{O}(1 / B^2)$  to obtain a formal robustness guarantee. In the context of robust distributed convex optimization (and robust least-squares regression), the upper bound assumed on the fraction  $f_n$  usually depends upon the condition number of the distributed optimization problem, e.g., see [6, Theorem 3] and [16, Theorem 2]. Our analysis in Theorem 1 justifies these assumptions on the breakdown point in prior work under heterogeneity.
Empirical breakdown point. Interestingly, our breakdown point  $\frac{1}{2 + B^2}$  allows to better understand some empirical observations indicating that the breakdown point of robust distributed learning algorithms is smaller than  $1/2$ . We illustrate this in Figure 2 with a logistic regression model on the MNIST dataset under extreme heterogeneity, i.e., each worker dataset contains data points from a
single class. We consider the state-of-the-art robust variant of distributed gradient descent (detailed later in Section 5.1) that uses the NNM robustness scheme composed with robust aggregation rules, namely, coordinate-wise trimmed-mean (CW Trimmed Mean) [34], Krum [7], coordinate-wise median (CW Median) [34] and geometric median [31, 30, 1]. We observe that all these methods consistently fail to converge as soon as the fraction of Byzantine workers exceeds  $\frac{1}{4}$ , which is well short of the previously known theoretical breakdown point  $\frac{1}{2}$ . Theorem 1, to the best of our knowledge, provides the first formal justification to this empirical observation.
Figure 2: Best training loss (left) and accuracy (right) using robust D-GD (see Section 5.1) with NNM to train a logistic regression model on the MNIST dataset, in the presence of 10 honest workers and 1 to 9 Byzantine workers. The Byzantine workers use the sign flipping attack. More details on the experimental setup can be found in Appendix D.
5 Tight upper bounds under  $(G,B)$ -Gradient Dissimilarity
We demonstrate in this section that the bounds presented in Theorem 1 are tight. Specifically, we show that a robust variant of distributed gradient descent, referred to as robust  $D$ -GD, yields an asymptotic error that matches the lower bound under  $(G,B)$ -gradient dissimilarity, while also proving the tightness of the breakdown point. Lastly, we present empirical evaluations showcasing a significant improvement over existing robustness analyses that relied upon  $G$ -gradient dissimilarity.
5.1 Convergence analysis of robust D-GD
In robust D-GD, the server initially possesses a model  $\theta_0$ . Then, at each step  $t \in \{0, \dots, T - 1\}$ , the server broadcasts model  $\theta_t$  to all workers. Each honest worker  $w_i$  sends the gradient  $g_t^{(i)} = \nabla \mathcal{L}_i(\theta_t)$  of its local loss function at  $\theta_t$ . However, a Byzantine worker  $w_j$  might send an arbitrary value for its gradient. Upon receiving the gradients from all the workers, the server aggregates the local gradients using a robust aggregation rule  $F \colon \mathbb{R}^{d \times n} \to \mathbb{R}^d$ . Specifically, the server computes  $R_t \coloneqq F(g_t^{(1)}, \ldots, g_t^{(n)})$ . Ultimately, the server updates the current model  $\theta_t$  to  $\theta_{t+1} = \theta_t - \gamma R_t$ , where  $\gamma > 0$  is the learning rate. The full procedure is summarized in Algorithm 1.
Algorithm 1: Robust D-GD
Input: Initial model  $\theta_0$ , robust aggregation  $F$ , learning rate  $\gamma$ , and number of steps  $T$ . for  $t = 0\dots T - 1$  do
Server broadcasts  $\theta_{t}$  to all the workers;
for each honest worker  $\mathbf{w}_{\mathrm{i}}$  in parallel do
Compute and send gradient  $g_{t}^{(i)} = \nabla \mathcal{L}_{i}(\theta_{t})$
// A Byzantine worker  $w_{j}$  may send an arbitrary value for  $g_{t}^{(j)}$
Server computes the aggregate gradient:  $R_{t} = F(g_{t}^{(1)},\dots ,g_{t}^{(n)})$
Server updates the model:  $\theta_{t + 1} = \theta_t - \gamma R_t$
To analyze robust D-GD under  $(G,B)$ -gradient dissimilarity, we first recall the notion of  $(f,\kappa)$ -robustness in Definition 4 below. First introduced in [3],  $(f,\kappa)$ -robustness is a general property of robust aggregation that covers several existing aggregation rules.
Definition 4  $((f,\kappa)$ -robustness). Let  $n\geq 1$ ,  $0\leq f < n / 2$  and  $\kappa \geq 0$ . An aggregation rule  $F\colon \mathbb{R}^{d\times n}\to \mathbb{R}^d$  is said to be  $(f,\kappa)$ -robust if for any vectors  $x_{1},\ldots ,x_{n}\in \mathbb{R}^{d}$ , and any set  $S\subseteq [n]$  of size  $n - f$ , the output  $\hat{x}\coloneqq F(x_1,\dots,x_n)$  satisfies the following:
$$
\left\| \hat {x} - \overline {{x}} _ {S} \right\| ^ {2} \leq \kappa \cdot \frac {1}{| S |} \sum_ {i \in S} \left\| x _ {i} - \overline {{x}} _ {S} \right\| ^ {2},
$$
where  $\overline{x}_S\coloneqq \frac{1}{|S|}\sum_{i\in S}x_i$ . We refer to  $\kappa$  as the robustness coefficient of  $F$ .
Closed-form robustness coefficients for multiple aggregation rules can be found in [3]. For example, assuming  $n \geq (2 + \eta)f$ , for some  $\eta > 0$ ,  $\kappa = \Theta\left(\frac{f}{n}\right)$  for coordinate-wise trimmed mean,  $\kappa = \Theta(1)$  for coordinate-wise median, and  $\kappa_{F \circ \mathrm{NNM}} = \Theta\left(\frac{f}{n}(\kappa + 1)\right)$  when  $F$  is composed with NNM [3].
Assuming  $F$  to be an  $(f,\kappa)$ -robust aggregation rule, we show in Theorem 2 below the convergence of robust D-GD in the presence of up to  $f$  Byzantine workers, under  $(G,B)$ -gradient dissimilarity.
Theorem 2. Let  $0 \leq f < n/2$ . Assume that the global loss  $\mathcal{L}_{\mathcal{H}}$  is  $L$ -smooth and that the honest local losses satisfy  $(G, B)$ -gradient dissimilarity (Assumption 1). Consider Algorithm 1 with learning rate  $\gamma = \frac{1}{L}$ . If the aggregation  $F$  is  $(f, \kappa)$ -robust with  $\kappa B^2 < 1$ , then the following holds for all  $T \geq 1$ .
1. In the general case where  $\mathcal{L}_{\mathcal{H}}$  may be non-convex, we have
$$
\frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla \mathcal {L} _ {\mathcal {H}} (\theta_ {t}) \| ^ {2} \leq \frac {\kappa G ^ {2}}{1 - \kappa B ^ {2}} + \frac {2 L (\mathcal {L} _ {\mathcal {H}} (\theta_ {0}) - \mathcal {L} _ {* , \mathcal {H}})}{(1 - \kappa B ^ {2}) T}.
$$
2. In the case where  $\mathcal{L}_{\mathcal{H}}$  is  $\mu$ -PL, we have
$$
\mathcal {L} _ {\mathcal {H}} (\theta_ {T}) - \mathcal {L} _ {*, \mathcal {H}} \leq \frac {\kappa G ^ {2}}{2 \mu (1 - \kappa B ^ {2})} + e ^ {- \frac {\mu}{L} (1 - \kappa B ^ {2}) T} (\mathcal {L} _ {\mathcal {H}} (\theta_ {0}) - \mathcal {L} _ {*, \mathcal {H}}).
$$
Tightness of the result. We recall that the best possible robustness coefficient for an aggregation  $F$  is  $\kappa = f / n - 2f$  (see [3]). For such an aggregation rule, the sufficient condition  $\kappa B^2 < 1$  reduces to  $f / n - 2f\cdot B^2 < 1$  or equivalently  $f / n < 1 / 2 + B^2$ . Besides, robust D-GD guarantees  $(f,\varepsilon)$ -resilience, for  $\mu$ -PL losses, where we have  $\varepsilon = \mathcal{O}\big(f / n - (2 + B^2)f\cdot G^2\big)$  asymptotically in  $T$ . Both these conditions on  $f / n$  and  $\varepsilon$  indeed match the limits shown earlier in Theorem 1. Yet, we are unaware of an aggregation rule with an order-optimal robustness coefficient, i.e., usually  $\kappa >f / n - 2f$ . However, as shown in [3], the composition of nearest neighbor mixing (NNM) with several aggregation rules, such as CW Trimmed Mean, Krum and Geometric Median, yields a robustness coefficient  $\kappa = \Theta (f / n - 2f)$ . Therefore, robust D-GD can indeed achieve  $(f,\varepsilon)$ -resilience with an optimal error  $\varepsilon = \mathcal{O}\big(f / n - (2 + B^2)f\cdot G^2\big)$ , but for a suboptimal breakdown point. The same observation holds for the non-convex case, where the lower bound is given by (7). Lastly, note that when  $B = 0$ , our result recovers the bounds derived in prior work under  $G$ -gradient dissimilarity [20, 3].
We remark that while the convergence rate of robust D-GD shown in Theorem 2 is linear (which is typical to convergence of gradient descent in strongly convex case), it features a slowdown factor of value  $1 - \kappa B^2$ . Hence, suggesting that Byzantine workers might decelerate the training under heterogeneity. This slowdown is also empirically observed (e.g., see Figure 1). Whether this slowdown is fundamental to robust distributed learning is an interesting open question. Investigating such a slowdown in the stochastic case is also of interest, as existing convergence bounds are under  $G$ -gradient dissimilarity only, for strongly convex [4] and non-convex [3] cases.
Comparison with prior work. Few previous works have studied Byzantine robustness under  $(G, B)$ -gradient dissimilarity [20, 14]. While these works do not provide lower bounds, the upper bound they derive (see Appendix E in [20] and Appendix E.4 in [14]) are similar to Theorem 2, with some notable differences. First, unlike the notion of  $(f, \kappa)$ -robustness that we use, the so-called  $(c, \delta)$ -agnostic robustness, used in [20, 14], is a stochastic notion. Under the latter notion, good parameters  $(c, \delta)$  of robust aggregators were only shown when using a randomized method called Bucketing [20]. Consequently, instead of obtaining a deterministic error bound as in Theorem 2,
simply replacing  $c\delta$  with  $\kappa$  in [20, 14] gives an expected bound, which is strictly weaker than the result of Theorem 2. Moreover, the corresponding non-vanishing upper bound term and breakdown point for robust D-GD obtained from the analysis in [20] for several robust aggregation rules (e.g., coordinate-wise median) are worse than what we obtain using  $(f,\kappa)$ -robustness.
5.2 Reducing the gap between theory and practice
In this section, we first argue that, even if we were to assume that the  $G$ -gradient dissimilarity condition (2) holds true, the robustness bounds derived in Theorem 2 under  $(G,B)$ -gradient dissimilarity improve upon the existing bounds [3] that rely on  $G$ -gradient dissimilarity. Next, we compare the empirical observations for robust D-GD with our theoretical upper bounds.
Comparing upper bounds. We consider a logistic regression model on MNIST dataset under extreme heterogeneity. While it is difficult to find tight values for parameters  $G$  and  $B$  satisfying  $(G, B)$ -gradient dissimilarity, we can approximate these parameters through a heuristic method. A similar approach can be used to approximate  $\widehat{G}$  for which the loss functions satisfy the condition of  $\widehat{G}$ -gradient dissimilarity. We defer the details on these approximations to Appendix D. In Figure 3, we compare the error bounds, i.e.,  $f / n - (2 + B^2)f \cdot G^2$  and  $f / n - 2f \cdot \widehat{G}^2$ , guaranteed for robust D-GD under  $(G, B)$ -gradient dissimilarity and  $\widehat{G}$ -gradient dissimilarity, respectively. We observe that the latter bound is extremely large compared to the former, which confirms that the tightest bounds under  $G$ -gradient dissimilarity are vacuous for practical purposes.
Figure 3: Comparison of our upper bound in Theorem 2 with that of  $G$ -gradient dissimilarity on MNIST with a logistic regression model. The number of honest workers is 10, and the number of Byzantine workers varies from 1 to 9.
Figure 4: Comparison between our upper bound in Corollary 1 and the training loss for the least-squares regression on the mg LIBSVM dataset [10]. The number of honest workers is 20 and the number of Byzantine workers varies from 1 to 19.
We further specialize the result of Theorem 2 to the convex case for which the  $(G,B)$ -gradient dissimilarity condition was characterized in Proposition 1. We have the following corollary.
Corollary 1. Assume that the global loss  $\mathcal{L}_{\mathcal{H}}$  is  $\mu$ -PL and  $L$ -smooth, and that for each  $i \in \mathcal{H}$  local loss  $\mathcal{L}_i$  is convex and  $L_i$ -smooth. Denote  $L_{\max} \coloneqq \max_{i \in \mathcal{H}} L_i$  and assume that  $\kappa\left(\frac{3L_{\max}}{\mu} - 1\right) \leq 1$ . Consider Algorithm 1 with learning rate  $\gamma = \frac{1}{L}$ . If  $F$  is  $(f, \kappa)$ -robust, then for all  $T \geq 1$ , we have
$$
\mathcal {L} _ {\mathcal {H}} (\theta_ {T}) - \mathcal {L} _ {*, \mathcal {H}} \leq \frac {3 \kappa}{\mu} \frac {1}{| \mathcal {H} |} \sum_ {i \in \mathcal {H}} \| \nabla \mathcal {L} _ {i} (\theta_ {*}) \| ^ {2} + e ^ {- \frac {\mu}{3 L} T} \left(\mathcal {L} _ {\mathcal {H}} (\theta_ {0}) - \mathcal {L} _ {*, \mathcal {H}}\right).
$$
The non-vanishing term in the upper bound shown in Corollary 1 corresponds to the heterogeneity at the minimum  $\frac{1}{|\mathcal{H}|}\sum_{i\in \mathcal{H}}\| \nabla \mathcal{L}_i(\theta_*)\|^2$ . This quantity is considered to be a natural measure of gradient dissimilarity in classical (non-Byzantine) distributed convex optimization [22, 23]. As such, we believe that this bound cannot be improved upon in general.
Matching empirical performances. Since the upper bound in Corollary 1 requires computing the constants  $\mu, L$ , we choose to conduct this experiment on least-squares regression, where the exact
computation of  $\mu, L$  is possible. We compare the empirical error gap (left-hand side of Corollary 1) with the upper bound (right-hand side of Corollary 1). Our findings, shown in Figure 4, indicate that our theoretical analysis reliably predicts the empirical performances of robust D-GD, especially when the fraction of Byzantine workers is small. Note, however, that our upper bound is non-informative when more than  $\frac{1}{4}$  of the workers are Byzantine, as the predicted error exceeds the initial loss value. We believe this to be an artifact of the proof, i.e. the upper bound is meaningful only up to a multiplicative constant. Indeed, when visualizing the results in logarithmic scale in Figure 4, the shape of empirical measurements and our upper bounds are quite similar.
This paper revisits the theory of robust distributed learning by considering a realistic data heterogeneity model, namely  $(G,B)$ -gradient dissimilarity. Using this model, we show that the breakdown point depends upon heterogeneity (specifically,  $1/2 + B^2$ ) and is smaller than the usual fraction  $1/2$ . We prove a new lower bound on the learning error of any distributed learning algorithm, which is matched using robust D-GD. Moreover, we show that our theoretical guarantees align closely with empirical observations, contrary to prior works which rely upon the stringent model of  $G$ -gradient dissimilarity.
An interesting future research direction is to investigate whether the  $1 - \kappa B^{2}$  slowdown factor in the convergence rate of robust D-GD (Theorem 2) is unavoidable. Another interesting research problem is to derive lower (and upper) bounds independent of the heterogeneity model, thereby elucidating the tightness of the convergence guarantee of robust D-GD in the strongly convex case (Corollary 1).
This work was supported in part by SNSF grant 200021_200477 and an EPFL-INRIA postdoctoral grant. The authors are thankful to the anonymous reviewers for their constructive comments.
[1] Anish Acharya, Abolfazl Hashemi, Prateek Jain, Sujay Sanghavi, Inderjit S. Dhillon, and Ufuk Topcu. Robust training in high dimensions via block coordinate geometric median descent. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 11145-11168. PMLR, 28-30 Mar 2022. URL https://proceedings.mlr.press/v151/acharya22a.html.
[2] Zeyuan Allen-Zhu, Faeze Ebrahimianghazani, Jerry Li, and Dan Alistarh. Byzantine-resilient non-convex stochastic gradient descent. In International Conference on Learning Representations, 2020.
[3] Youssef Allouah, Sadegh Farhadkhani, Rachid Guerraoui, Nirupam Gupta, Rafaël Pinot, and John Stephan. Fixing by mixing: A recipe for optimal Byzantine ML under heterogeneity. In International Conference on Artificial Intelligence and Statistics, pages 1232-1300. PMLR, 2023.
[4] Youssef Allouah, Rachid Guerraoui, Nirupam Gupta, Rafaël Pinot, and John Stephan. On the privacy-robustness-utility trilemma in distributed learning. In International Conference on Machine Learning, number 202, 2023.
[5] Moran Baruch, Gilad Baruch, and Yoav Goldberg. A little is enough: Circumventing defenses for distributed learning. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, 8-14 December 2019, Long Beach, CA, USA, 2019.
[6] Kush Bhatia, Prateek Jain, and Purushottam Kar. Robust regression via hard thresholding. Advances in neural information processing systems, 28, 2015.
[22] Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik. Tighter theory for local SGD on identical and heterogeneous data. In International Conference on Artificial Intelligence and Statistics, pages 4519-4529. PMLR, 2020.
[23] Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized SGD with changing topology and local updates. In International Conference on Machine Learning, pages 5381-5393. PMLR, 2020.
[24] Leslie Lamport, Robert Shostak, and Marshall Pease. The Byzantine generals problem. ACM Trans. Program. Lang. Syst., 4(3):382-401, jul 1982. ISSN 0164-0925. doi: 10.1145/357172.357176. URL https://doi.org/10.1145/357172.357176.
[25] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429-450, 2020.
[26] Shuo Liu, Nirupam Gupta, and Nitin H. Vaidya. Approximate Byzantine fault-tolerance in distributed optimization. In Proceedings of the 2021 ACM Symposium on Principles of Distributed Computing, PODC'21, page 379-389, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450385480. doi: 10.1145/3465084.3467902.
[27] Aritra Mitra, Rayana Jaafar, George J Pappas, and Hamed Hassani. Linear convergence in federated learning: Tackling client heterogeneity and sparse gradients. Advances in Neural Information Processing Systems, 34:14606-14619, 2021.
[28] Yuri Nesterov et al. Lectures on convex optimization, volume 137. Springer, 2018.
[29] Maxence Noble, Aurélien Bellet, and Aymeric Dieuleveut. Differentially private federated learning on heterogeneous data. In International Conference on Artificial Intelligence and Statistics, pages 10110-10145. PMLR, 2022.
[30] Krishna Pillutla, Sham M. Kakade, and Zaid Harchaoui. Robust aggregation for federated learning. IEEE Transactions on Signal Processing, 70:1142-1154, 2022. doi: 10.1109/TSP.2022.3153135.
[31] Christopher G Small. A survey of multidimensional medians. International Statistical Review/Revue Internationale de Statistique, pages 263-277, 1990.
[32] Sharan Vaswani, Francis Bach, and Mark Schmidt. Fast and faster convergence of SGD for over-parameterized models and an accelerated perceptron. In The 22nd international conference on artificial intelligence and statistics, pages 1195-1204. PMLR, 2019.
[33] Cong Xie, Oluwasanmi Koyejo, and Indranil Gupta. Fall of empires: Breaking Byzantine-tolerant SGD by inner product manipulation. In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, page 83, 2019.
[34] Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantine-robust distributed learning: Towards optimal statistical rates. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 5650–5659. PMLR, 10–15 Jul 2018. URL https://proceedings.mlr.press/v80/yin18a.html.
Appendix A contains the proof of Proposition 1. Appendix B contains the impossibility result shown in Theorem 1. Appendix C contains the convergence proofs concerning Robust D-GD (Theorem 2, Corollary 1, and Proposition 1). Appendix D presents our experimental setups.
Proof. Let  $\theta \in \mathbb{R}^d$ . By bias-variance decomposition, we obtain that
$$
\frac {1}{| \mathcal {H} |} \sum_ {i \in \mathcal {H}} \| \nabla \mathcal {L} _ {i} (\theta) - \nabla \mathcal {L} _ {\mathcal {H}} (\theta) \| ^ {2} = \frac {1}{| \mathcal {H} |} \sum_ {i \in \mathcal {H}} \| \nabla \mathcal {L} _ {i} (\theta) \| ^ {2} - \| \nabla \mathcal {L} _ {\mathcal {H}} (\theta) \| ^ {2}. \tag {8}
$$
Using triangle inequality, we have
$$
\left\| \nabla \mathcal {L} _ {i} (\theta) \right\| ^ {2} = \left\| \nabla \mathcal {L} _ {i} (\theta) - \nabla \mathcal {L} _ {i} \left(\theta_ {*}\right) + \nabla \mathcal {L} _ {i} \left(\theta_ {*}\right) \right\| ^ {2} \leq \left(\left\| \nabla \mathcal {L} _ {i} (\theta) - \nabla \mathcal {L} _ {i} \left(\theta_ {*}\right) \right\| + \left\| \nabla \mathcal {L} _ {i} \left(\theta_ {*}\right) \right\|\right) ^ {2}
$$
For any pair of real values  $(a, b)$ , we have  $(a + b)^2 \leq 2a^2 + 2b^2$ . Using this inequality with  $a = \| \nabla \mathcal{L}_i(\theta) - \nabla \mathcal{L}_i(\theta_*)\|$  and  $b = \| \nabla \mathcal{L}_i(\theta_*)\|$  in the above, we obtain that
$$
\left\| \nabla \mathcal {L} _ {i} (\theta) \right\| ^ {2} \leq 2 \left\| \nabla \mathcal {L} _ {i} (\theta) - \nabla \mathcal {L} _ {i} \left(\theta_ {*}\right) \right\| ^ {2} + 2 \left\| \nabla \mathcal {L} _ {i} \left(\theta_ {*}\right) \right\| ^ {2}.
$$
Using the above in (8), we obtain that
$$
\begin{array}{l} \frac {1}{| \mathcal {H} |} \sum_ {i \in \mathcal {H}} \| \nabla \mathcal {L} _ {i} (\theta) - \nabla \mathcal {L} _ {\mathcal {H}} (\theta) \| ^ {2} \leq \frac {1}{| \mathcal {H} |} \sum_ {i \in \mathcal {H}} \left(2 \| \nabla \mathcal {L} _ {i} (\theta) - \nabla \mathcal {L} _ {i} (\theta_ {*}) \| ^ {2} + 2 \| \nabla \mathcal {L} _ {i} (\theta_ {*}) \| ^ {2}\right) \\ - \left\| \nabla \mathcal {L} _ {\mathcal {H}} (\theta) \right\| ^ {2}. \tag {9} \\ \end{array}
$$
For all  $i\in \mathcal{H}$ , since  $\mathcal{L}_i$  is assumed convex and  $L_{i}$ -smooth, we also have (see [28, Theorem 2.1.5]) for all  $\theta^{\prime}\in \mathbb{R}^{d}$  that