SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Given  $\nabla H_{1} = 4$ ,  $\nabla H_{2} = 5$ ,  $\nabla H_{3} = 6$ , and  $\nabla H = 5$ . Now suppose that one of the three users is Byzantine, say user 2, who can send the server an arbitrary value  $g_{2}$  instead of the honest gradient  $\nabla H_{2} = 5$ . To implement NBS, we impose that  $\beta = \alpha = \frac{1}{3}$ . We can verify that (1) always holds, with the RHS equal to 6 and the LHS never reaching 6 no matter what value  $g_{2}$  takes. However, the LHS can infinitely approach 6 if  $g_{2}$  is selected judiciously. For example, for the set of gradients  $g_{1} = 4$ ,  $g_{2} = -5.98$ ,  $g_{3} = 6$ , the LHS is equal to 5.99. In this case, the Byzantine gradient  $g_{2}$  is carefully chosen such that it is able to bypass the screening, and meanwhile exerts maximum adversarial perturbation on the output. Readers can verify that (1) also holds when user 1 or user 3 is Byzantine. This example shows that (1) precisely measures the worst-case deviation by Byzantine users and that it cannot be further tightened.
The advantages of NBS over its counterparts. Aside from NBS, most of the other robust aggregation measures are robust-mean-based [7, 8, 9, 10], with the goal of computing a robust mean
gradient for every single iteration. Given the neat convergence guarantees resulted from NBS, a natural question arises: "Can any of those robust-mean-based methods lead to similar convergence results?" It turns out that the answer to this question might be negative, due to three major theoretical advantages of NBS over its counterparts, which make the latter not only require over-stringent assumptions but also produce less informative bounds.
First, unlike robust-mean-based methods, NBS does not require the assumption that local data are iid. As explained in Section 4.1, bounding the noise term  $\| G - \nabla H\|$  is crucial for convergence, where  $G$  represents the output of robust aggregation, whether it is NBS or one of its counterparts. For the latter, the iid assumption is indispensable since it ensures that the honest gradients are iid samples of the true global gradient  $\nabla H$ . Without this assumption,  $\| G - \nabla H\|$  cannot be upper-bounded theoretically.
Second, unlike its counterparts, NBS does not require local gradients to follow a certain distribution, which often lacks proper justification in practice. For robust-mean-based methods, this distributional assumption determines the level to which honest local gradients are concentrated. Without this assumption,  $\| G - \nabla H \|$  also cannot be upper-bounded theoretically. In contrast, in order to bound  $\| G - \nabla H \|$ , NBS only requires Assumption 1, which always holds as explained previously and does not have the justification issue.
Third (and most importantly), NBS's bound on  $\| G - \nabla H\|$  is tighter, therefore more useful, than that of its counterparts. To reach convergence, the noise term  $\| G - \nabla H\|$  has to be bounded for every iteration, suggesting that the distributional assumption needs to be made repeatedly for robust-mean-based methods. Although the distribution of honest gradients is supposed to become more and more concentrated as the model makes steady progress towards the optimum, such dynamic distributional change is unknown to us and there is no way to accurately incorporate this piece of information into the distributional assumption at each iteration. Therefore, one has little choice but to compromise by imposing the same distributional assumption on local gradients for all iterations, which inevitably leads to a loose bound on  $\| G - \nabla H\|$ . For example, the authors of [9] assume that the partial derivatives of the objective follow the same sub-exponential distribution for all iterations. As a result, they can only bound  $\| G - \nabla H\|$  as a constant  $\Delta$  irrespective of the iteration index (see Theorem 8 and Theorem 11 therein). Clearly, the inequality  $\| G - \nabla H\| \leq \Delta$  is not tight since it completely ignores the ever-concentrating effect of honest gradients as training proceeds. In contrast, NBS leads to (1), i.e.,  $\| G - \nabla H\| \leq C_{\alpha}\| \nabla H\| +\sigma$ , which accurately captures the supposed shrinkage of  $\| G - \nabla H\|$  as  $\| \nabla H\|$  decreases.
To summarize, NBS enjoys the above three distinctive advantages over its robust-mean-based counterparts in theoretically upper-bounding  $\| G - \nabla H\|$ , making it suitable for us to derive meaningful convergence guarantees.
The effects of non-iid data. Although our convergence analysis applies to both iid and non-iid settings, the heterogeneity of local data does have an impact on the convergence results. This is reflected in (2) where the distance between any local gradient and the true global gradient is upper-bounded by  $\sigma$  ( $\max_{1\leq i\leq m}\|\nabla H_i - \nabla H\| \leq \sigma$ ), which is determined by data heterogeneity. In the most homogeneous case where each user has the same data,  $\sigma$  is equal to 0. As data heterogeneity increases,  $\sigma$  should also increase accordingly. Through the adoption of such a universal upper bound  $\sigma$ , we are able to derive meaningful convergence guarantees without the normally required iid assumption. This analytical approach, i.e., making assumptions in the spirit of (2) and imposing a universal upper bound to eliminate local gradients' differences, might serve as a pathway for future works to bypass the non-iid issue theoretically in the convergence analysis.
Dataset and allocation. In this section, we empirically evaluate the performance of our algorithm for a classification task using the logistic regression model on the Spambase dataset [21]. We assign  $\frac{2}{3}$  of the 4601 total samples for training and the other  $\frac{1}{3}$  for testing. We evenly split the training data among  $m = 20$  nodes, with 8 of them exclusively holding spams (labelled as 1) and the remaining 12 exclusively holding non-spams (labelled as 0). Each node has the same number of data points  $n = 153$ . Although in theory there are numerous ways to create a non-iid setting, splitting data
according to their labels seems to be the common practice. In the Byzantine setting, we pick  $\alpha m = 4$  nodes as Byzantine (2 from each group). Upon the completion of training, we use the percentage of correct classifications on the test set as the performance metric of model accuracy.
Objective and fairness metric. To achieve fairness, we adopt  $q$ -FFL [16] where the local objective takes the form of  $H_{i}(\theta) = \frac{1}{q + 1}[F_{i}(\theta)]^{q + 1}$  with  $F_{i}(\theta) = \frac{1}{n}\sum_{j=(i-1)n+1}^{(i-1)n+n}f(\theta; x_{j})$  and  $f(\theta; x_{j})$  being the cross-entropy loss that is common for classification tasks. As explained in Section 3.2,  $q$ -FFL promotes egalitarian fairness. We follow [16] to use the variance of the local classification accuracy as the metric of fairness (only the accuracies of honest users are counted for  $\alpha > 0$ ). Note that our framework is not limited to the  $q$ -FFL objective and it readily suits for other fairness-oriented objectives as long as they can be decomposed in the form of  $H = \frac{1}{m}\left[H_{1} + \cdots + H_{m}\right]$ .
Byzantine models and robust benchmarks. To generate Byzantine gradients, we experiment with four different strategies, i.e., sign-flipping attack [22], label-flipping attack [9], Inner Product Manipulation (IPM) attack (for two regimes of  $\epsilon = 0.5$  and  $\epsilon = 50$  respectively) [23], and "A Little Is Enough" (ALIE) attack [24]. These four types of Byzantine models are commonly considered in the literature [25], and they represent a wide range of attacks varying in aspects such as the gradient norms and adversary's knowledge. To test the effectiveness of NBS, we compare it with several widely-acknowledged robust aggregation measures Krum [7], coordinate-wise median (CM) [9] and coordinate-wise trimmed mean (CTM) [9], as benchmarks, along with the plain averaging (ERM). For Krum, CTM, and NBS, the screening/trimming parameter is chosen based on the Byzantine percentage  $\alpha$ . This allows these robust schemes to effectively aggregate gradient information and achieve better performance compared to schemes that do not consider  $\alpha$ . In practice,  $\alpha$  can be set as the estimated upper bound of the Byzantine percentage, as the exact value of  $\alpha$  may not always be available or reliable during the training process.
5.2 Evaluation
First, we seek to inspect the robustness of NBS compared with other benchmarks, without incorporating  $q$ -FFL for the moment. This is because reasonably high accuracy should always be the first priority of model training, without which other parallel goals such as fairness would be meaningless. In Figure 1, we plot the performance curves of model accuracy for the five considered aggregation measures (ERM, NBS, CTM, CM, Krum) under various attack regimes, using learning rate  $\eta = 1$  and number of iterations  $T = 300$ . From Figure 1, we can see that NBS enjoys the best overall performance, due to its ability to utilize redundant information as well as its resilience to large-norm attacks (such as sign-flipping attack and IPM attack with  $\epsilon = 50$  where ERM performs poorly).
(a) No attack
(b) Sign-flipping attack
(c) Label-flipping attack
(d) IPM attack  $(\epsilon = 0.5)$
(e) IPM attack  $(\epsilon = 50)$
(f) ALIE attack
Figure 1: Model accuracy of different robust aggregation schemes under various attacks.
This result verifies that the theoretical advantages of NBS over other robust benchmarks in handling Byzantine attacks can extend to practical applications. In contrast, CM and Krum fail to converge to a functioning model even in the absence of Byzantine attacks. The only other robust scheme with comparable robustness performance to NBS is CTM, which achieves a worst-case test accuracy of  $79.7\%$  compared to  $85.4\%$  for NBS.
Next, we incorporate the  $q$ -FFL objective (Step 5 in Algorithm 2) into NBS and investigate how NBS, while ensuring Byzantine robustness, affects the fairness performance. In Table 1, we document both the model accuracy and the variance of local accuracies (the metric of model fairness) using different values of  $q$  and screening percentage  $\beta$ . The learning rate  $\eta$  and number of iterations  $T$  for each  $q$  value are carefully selected to ensure fast and stable convergence. Table 1 shows that vanilla  $q$ -FFL (under  $\beta = 0$ ) achieves the intended tradeoff between utility (high accuracy) and fairness (low variance), with larger  $q$  focusing more on fairness. In particular,  $q = 1$  leads to a desirable tradeoff by significantly reducing the variance (from 480 to 269) while only marginally decreasing the model accuracy (from  $92.5\%$  to  $91.3\%$ ). By comparison, the study in [16] reports that  $q$ -FFL reduces the variance of accuracies across devices by an average of  $45\%$ , with a similar decrease in overall accuracy. Therefore, we choose  $q = 1$  for the  $q$ -FFL scheme in the remaining experiments.
Table 1: The impact of screening on the performance of H-nobs (with no attack).
β = 0	β = 10%	β = 20%	β = 30%	(η, T)
q = 0	92.5% (480)	92.7% (389)	92.5% (396)	92.2% (392)	(1, 300)
q = 0.5	92.0% (405)	92.4% (364)	92.3% (368)	92.2% (381)	(0.5, 1000)
q = 1	91.3% (269)	92.0% (327)	91.9% (331)	91.1% (349)	(0.5, 1000)
q = 2	86.9% (156)	89.5% (253)	89.0% (243)	88.9% (260)	(0.2, 1500)
From Table 1, we observe that increasing  $\beta$  does not have an obvious effect on the model accuracy, since screening up to  $30\%$  of gradients still leaves enough redundant information in the absence of Byzantine attacks. On the other hand, the effects of screening on fairness are quite different for small  $q$  and large  $q$  regimes. Specifically, when  $q = 0$ , the variance is at its highest, which suggests a relatively wide gap on the local accuracies. In that case, screening up to  $30\%$  of gradients coincidentally suppresses some of the divisive nodes and makes the results fairer. However, when  $q \geq 1$ , the variance is small, in which case the screening operation begins to compromise fairness by increasing the variance, since it inevitably filters out certain fairness-enhancing gradients. This phenomenon corroborates our hypothesis in Introduction about the internal conflicts between fairness (up-weighted objective) and robustness (filtering of outliers). Despite the inevitable impaired performance on fairness compared to vanilla  $q$ -FFL, H-nobs (e.g., with  $q = 1$  and  $\beta = 20\%$ ) strikes a good balance between robustness and fairness, being able to defend against up to  $20\%$  of Byzantine nodes while securing some fairness gains. In addition, we observe a similar interplay between robustness and fairness in the Law School dataset [26] and the Credit Card Client dataset [27], both of which are commonly considered in the context of fairness-aware machine learning [28]. The results for these two datasets are presented in Appendix E.
Table 2: The influence of various attacks on both robustness (accuracy) and fairness (variance).
q-FFL	q-FFL + NBS	q-FFL + CTM	q-FFL + CM	q-FFL + Krum
No attack	91.3% (269)	91.9% (331)	91.5% (316)	62.0% (N/A)	62.8% (N/A)
Sign-flipping	61.1% (N/A)	91.0% (265)	87.4% (143)	47.3% (N/A)	82.1% (291)
Label-flipping	80.1% (150)	81.5% (220)	87.9% (61)	45.8% (N/A)	57.9% (N/A)
IPM (ε = 0.5)	89.8% (76)	87.1% (437)	90.8% (357)	54.4% (N/A)	57.4% (N/A)
IPM (ε = 50)	61.1% (N/A)	90.3% (94)	71.1% (186)	54.6% (N/A)	88.5% (245)
ALIE	76.0% (211)	90.0% (72)	88.1% (62)	48.4% (N/A)	88.5% (245)
Finally, we compare H-nobs (i.e.,  $q$ -FFL + NBS) with the combinations of  $q$ -FFL and other aggregation schemes in the Byzantine setting with 4 Byzantine nodes. The robustness and fairness performance of candidate algorithms under various attacks is summarized in Table 2, using learning rate  $\eta = 0.5$  and number of iterations  $T = 1000$ . We exclude the variance if the model accuracy falls below  $65\%$ . In Table 2, we can see that in the Byzantine-free case, both NBS and CTM effectively maintain the fairness improvements achieved by  $q$ -FFL, as they have a much lower variance than ERM (480), which uses the traditional training loss  $F$  instead of the fair objective  $H$ . When Byzantine attacks are introduced, NBS demonstrates the highest level of overall robustness, achieving a worst-case test accuracy of  $81.5\%$ . In comparison, the second-best performer, CTM, reaches  $71.1\%$ . This suggesting an even wider performance gap than when considering these two schemes without integrating with  $q$ -FFL, which had worst-case accuracies of  $85.4\%$  and  $79.7\%$  respectively. These findings further support our choice of NBS as the aggregation method, which showcases not only theoretical but also empirical advantages over its robust-mean-based counterparts. In terms of fairness, both NBS and CTM exhibit varying performance depending on the specific attack regimes, due to the unpredictability of Byzantine gradients. As a result, no definitive conclusion can be drawn regarding the tradeoff between robustness and fairness in the presence of Byzantine attacks.
5.3 Summary and discussion
In this section, we first empirically illustrate the effectiveness of NBS against various Byzantine attacks and its advantage over other robust benchmarks. Then, we investigate the influence of screening on the fairness performance for H-nobs, verifying the internal conflicts between robustness and fairness. Finally, we compare H-nobs with other benchmarks under different attack regimes, demonstrating the robust performance of H-nobs and indicating that the fairness gains can vary significantly depending on the attack regimes.
We want to point out two limitations of our empirical exploration. First, our simulations are based on a fixed Byzantine percentage  $\alpha = 20\%$  and a fixed level of data heterogeneity (i.e.,  $\sigma$  in (2)), both of which can have an impact on the convergence as demonstrated in Theorems 1-3. However, as  $\alpha$  or  $\sigma$  increases, overcoming Byzantine failure might become an intractable task due to the insufficient redundancy among the honest users. We leave the exploration on the robustness/fairness tradeoff under different  $\alpha/\sigma$  regimes for future work. Second, our evaluation of fairness is conducted based on the specific  $q$ -FFL objective under the egalitarian fairness notion. Whether or not our empirical results apply to other fairness notions or objectives is an open question and deserves further investigation in the future.
In this paper, we recognize three important but underexplored questions related to fair and robust distributed learning. We address these questions by proposing a new framework H-nobs that achieves joint fairness and robustness. The convergence of the proposed algorithm, even under the challenging non-id settings, is theoretically guaranteed for different types of learning models. We empirically verify the effectiveness of our algorithm under various attack regimes, and we also take the initiative to investigate the influence of robustness-oriented screening on the performance of fairness.
This work was partly supported by the NSF grants No.1939553, No.2003211, and No.2231209.
[18] Leslie Lamport, Robert Shostak, and Marshall Pease. The byzantine generals problem. In Concurrence: The Works of Leslie Lamport, pages 203-226. 2019.
[19] Lingjiao Chen, Hongyi Wang, Zachary Charles, and Dimitris Papailiopoulos. Draco: Byzantine-resilient distributed training via redundant gradients. In International Conference on Machine Learning, pages 903–912. PMLR, 2018.
[20] Avishek Ghosh, Raj Kumar Maity, and Arya Mazumdar. Distributed newton can communicate less and resist byzantine workers. Advances in Neural Information Processing Systems, 33:18028-18038, 2020.
[21] Mark Hopkins, Erik Reeber, George Forman, and Jaap Suermondt. Spambase, 1999.
[22] Liping Li, Wei Xu, Tianyi Chen, Georgios B Giannakis, and Qing Ling. Rsa: Byzantine-robust stochastic aggregation methods for distributed learning from heterogeneous datasets. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1544–1551, 2019.
[23] Cong Xie, Oluwasanmi Koyejo, and Indranil Gupta. Fall of empires: Breaking byzantine-tolerant sgd by inner product manipulation. In Uncertainty in Artificial Intelligence, pages 261-270. PMLR, 2020.
[24] Gilad Baruch, Moran Baruch, and Yoav Goldberg. A little is enough: Circumventing defenses for distributed learning. Advances in Neural Information Processing Systems, 32, 2019.
[25] Shenghui Li, Edith C-H Ngai, and Thiemo Voigt. An experimental study of byzantine-robust aggregation schemes in federated learning. IEEE Transactions on Big Data, 2023.
[26] Linda Wightman. Law school admissions bar passage, 1998.
[27] I-Cheng Yeh. Default of credit card clients, 2016.
[28] Tai Le Quy, Arjun Roy, Vasileios Iosifidis, Wenbin Zhang, and Eirini Ntoutsi. A survey on datasets for fairness-aware machine learning. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 12(3):e1452, 2022.
For  $G = \frac{1}{|\mathcal{U}|} \sum_{i \in \mathcal{U}} g_i$  with  $\mathcal{U} = \{(1), \ldots, ((1 - \beta)m)\}$  and any specific vector  $S$ , we have
$$
\begin{array}{l} \| G - S \| = \left\| \frac {1}{| \mathcal {U} |} \sum_ {i \in \mathcal {U}} g _ {i} - S \right\| \\ = \left\| \frac {1}{| \mathcal {U} |} \sum_ {i \in \mathcal {U}} (g _ {i} - S) \right\| \\ = \frac {1}{| \mathcal {U} |} \left\| \sum_ {i \in \mathcal {U} \cap \mathcal {M}} (g _ {i} - S) + \sum_ {i \in \mathcal {U} \cap \mathcal {B}} (g _ {i} - S) \right\| \\ \leq \frac {1}{| \mathcal {U} |} \left(\sum_ {i \in \mathcal {U} \cap \mathcal {M}} \| g _ {i} - S \| + \sum_ {i \in \mathcal {U} \cap \mathcal {B}} \| g _ {i} - S \|\right). \\ \end{array}
$$
For  $i\in \mathcal{U}\cap \mathcal{M}$ $\| g_i - S\| \leq \Delta$  (we define  $\Delta = \max_{i\in \mathcal{M}}\| g_i - S\|$ ).
For  $i\in \mathcal{U}\cap \mathcal{B}$  , we bound  $\| g_i - S\|$  as follows:
$$
\begin{array}{l} \left\| g _ {i} - S \right\| \leq \left\| g _ {i} \right\| + \left\| S \right\| \\ \leq \left\| g _ {\left((1 - \beta) m\right)} \right\| + \| S \| \\ \leq \left\| g _ {\left((1 - \alpha) m\right)} \right\| + \| S \| \\ \leq \max  _ {i \in \mathcal {M}} \| g _ {i} \| + \| S \| \\ = \max  _ {i \in \mathcal {M}} \| g _ {i} - S + S \| + \| S \| \\ \leq \max  _ {i \in \mathcal {M}} \| g _ {i} - S \| + 2 \| S \| \\ = \Delta + 2 \| S \|. \\ \end{array}
$$
Combining the above results, we have
$$
\begin{array}{l} \| G - S \| \leq \frac {1}{| \mathcal {U} |} \left(| \mathcal {U} \cap \mathcal {M} | \cdot \Delta + | \mathcal {U} \cap \mathcal {B} | \cdot (\Delta + 2 \| S \|)\right) \\ = \frac {1}{| \mathcal {U} |} \left(| \mathcal {U} | \cdot \Delta + 2 | \mathcal {U} \cap \mathcal {B} | \cdot \| S \|\right) \tag {6} \\ = \Delta + \frac {2 | \mathcal {U} \cap \mathcal {B} |}{| \mathcal {U} |} \| S \| \\ \leq \Delta + \frac {2 \alpha}{1 - \beta} \| S \| \\ \end{array}
$$
which is exactly the conclusion in Lemma 1 after we replace  $S$  with  $\nabla H$ . Note that in (6), the last inequality only holds on condition that  $|\mathcal{B}| \leq |\mathcal{U}|$ , i.e.,  $\alpha \leq 1 - \beta$ , which, combined with  $\beta \geq \alpha$ , suggests that  $\alpha \leq \frac{1}{2}$ .
According to Assumption 2,  $H(\theta)$  is  $L_{H}$ -smooth. According to the property of smoothness, we have
$$
\begin{array}{l} H \left(\theta_ {t + 1}\right) \leq H \left(\theta_ {t}\right) + \left\langle \nabla H \left(\theta_ {t}\right), \theta_ {t + 1} - \theta_ {t} \right\rangle + \frac {L _ {H}}{2} \left\| \theta_ {t + 1} - \theta_ {t} \right\| ^ {2} \\ = H (\theta_ {t}) - \eta \langle \nabla H (\theta_ {t}), G (\theta_ {t}) \rangle + \frac {L _ {H}}{2} \eta^ {2} \| G (\theta_ {t}) \| ^ {2} \\ = H \left(\theta_ {t}\right) - \frac {1}{L _ {H}} \langle \nabla H \left(\theta_ {t}\right), G \left(\theta_ {t}\right) - \nabla H \left(\theta_ {t}\right) + \nabla H \left(\theta_ {t}\right) \rangle + \frac {1}{2 L _ {H}} \| G \left(\theta_ {t}\right) - \nabla H \left(\theta_ {t}\right) + \nabla H \left(\theta_ {t}\right) \| ^ {2} \\ = H (\theta_ {t}) - \frac {1}{L _ {H}} \| \nabla H (\theta_ {t}) \| ^ {2} + \frac {1}{2 L _ {H}} \left(\| \nabla H (\theta_ {t}) \| ^ {2} + \| G (\theta_ {t}) - \nabla H (\theta_ {t}) \| ^ {2}\right) \\ = H \left(\theta_ {t}\right) - \frac {1}{2 L _ {H}} \| \nabla H \left(\theta_ {t}\right) \| ^ {2} + \frac {1}{2 L _ {H}} \| G \left(\theta_ {t}\right) - \nabla H \left(\theta_ {t}\right) \| ^ {2} \tag {7} \\ \end{array}
$$
where the first equality follows from  $\theta_{t + 1} = \theta_t - \eta \cdot G(\theta_t)$  and the second equality follows from  $\eta = \frac{1}{L_H}$ . Note that the derivation of (7) is a common trick in analyzing the convergence of smooth functions which shifts the burden of proving convergence into the relatively easy task of quantifying  $\| G(\theta_t) - \nabla H(\theta_t)\|$ .
Combining Lemma 1 and Assumption 1, we have  $\| G(\theta_t) - \nabla H(\theta_t)\| \leq C_\alpha \| \nabla H(\theta_t)\| +\sigma$  , which leads to
$$
\begin{array}{l} \left\| G \left(\theta_ {t}\right) - \nabla H \left(\theta_ {t}\right) \right\| ^ {2} \leq C _ {\alpha} ^ {2} \left\| \nabla H \left(\theta_ {t}\right) \right\| ^ {2} + 2 C _ {\alpha} \left\| \nabla H \left(\theta_ {t}\right) \right\| \sigma + \sigma^ {2} \tag {8} \\ \leq (1 + r) C _ {\alpha} ^ {2} \| \nabla H (\theta_ {t}) \| ^ {2} + (1 + 1 / r) \sigma^ {2} \\ \end{array}
$$
for any  $r > 0$
Combining (7) and (8), we have
$$
H \left(\theta_ {t + 1}\right) \leq H \left(\theta_ {t}\right) - \frac {1 - (1 + r) C _ {\alpha} ^ {2}}{2 L _ {H}} \| \nabla H \left(\theta_ {t}\right) \| ^ {2} + \frac {1 + 1 / r}{2 L _ {H}} \sigma^ {2} \tag {9}
$$
$$
\left\| \nabla H \left(\theta_ {t}\right) \right\| ^ {2} \leq \frac {2 L _ {H}}{1 - (1 + r) C _ {\alpha} ^ {2}} \left[ H \left(\theta_ {t}\right) - H \left(\theta_ {t + 1}\right) \right] + \frac {1 + 1 / r}{1 - (1 + r) C _ {\alpha} ^ {2}} \sigma^ {2}. \tag {10}
$$
Summing up (10) for  $t = 0,1,\ldots ,T - 1$  before divided by  $T$  gives