SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 0} ^ {T - 1} \| \nabla H \left(\theta_ {t}\right) \| ^ {2} \leq \frac {2 L _ {H}}{\left(1 - (1 + r) C _ {\alpha} ^ {2}\right) T} \left[ H \left(\theta_ {0}\right) - H \left(\theta_ {T}\right) \right] + \frac {1 + 1 / r}{1 - (1 + r) C _ {\alpha} ^ {2}} \sigma^ {2} \tag {11} \\ \leq \frac {2 L _ {H}}{\left(1 - (1 + r) C _ {\alpha} ^ {2}\right) T} \left[ H \left(\theta_ {0}\right) - H \left(\theta^ {*}\right) \right] + \frac {1 + 1 / r}{1 - (1 + r) C _ {\alpha} ^ {2}} \sigma^ {2} \\ \end{array}
$$
which is exactly the conclusion in Theorem 1.
Note that the transition from (9) to (10) only stands under the condition that  $1 - (1 + r)C_{\alpha}^{2} > 0$ , which constrains  $r$  to the less than  $\frac{1}{C_{\alpha}^{2}} - 1$ . On the other hand,  $r > 0$ , which requires  $C_{\alpha} = \frac{2\alpha}{1 - \beta} < 1$ , i.e.,  $2\alpha + \beta < 1$ . Since  $\beta \geq \alpha$ , we can conclude that (11) holds if and only if  $\alpha < \frac{1}{3}$  and  $0 < r < \left(\frac{1 - \beta}{2\alpha}\right)^{2} - 1$ .
According to Assumption 3, the convexity of  $H(\theta)$  suggests that  $H(\theta^{*})\geq H(\theta_{t}) + \langle \nabla H(\theta_{t}),\theta^{*} - \theta_{t}\rangle$ , which leads to
$$
\begin{array}{l} H \left(\theta_ {t}\right) - H \left(\theta^ {*}\right) \leq \langle \nabla H \left(\theta_ {t}\right), \theta_ {t} - \theta^ {*} \rangle \\ \leq \left\| \nabla H \left(\theta_ {t}\right) \right\| \cdot \left\| \theta_ {t} - \theta^ {*} \right\| \tag {12} \\ \leq D \| \nabla H (\theta_ {t}) \| \\ \end{array}
$$
in which  $D = k\| \theta_0 - \theta^*\|$ . The third inequality of (12) follows from Assumption 4. As a result, we obtain (13) as a key property for the subsequent analysis.
$$
\left\| \nabla H \left(\theta_ {t}\right) \right\| \geq \frac {1}{D} \left[ H \left(\theta_ {t}\right) - H \left(\theta^ {*}\right) \right] \tag {13}
$$
Since Theorem 2 keeps all the assumptions made in Theorem 1, all the intermediate steps in the proof of Theorem 1 also apply here. In this regard, we borrow (9), i.e.,
$$
H \left(\theta_ {t + 1}\right) - H \left(\theta_ {t}\right) \leq - A \| \nabla H \left(\theta_ {t}\right) \| ^ {2} + B \tag {14}
$$
in which we define  $A = \frac{1 - (1 + r)C_{\alpha}^{2}}{2L_{H}}$  and  $B = \frac{(1 + 1 / r)\sigma^2}{2L_H}$  for convenience.
Next, we consider two cases in regard to the relationship between  $A\| \nabla H(\theta_t)\| ^2$  and  $B$ .
Case 1. Suppose for all  $0 \leq t \leq T - 1$ , it holds that  $B \leq \frac{A}{2} \| \nabla H(\theta_t) \|^2$ . In this case, we have
$$
H \left(\theta_ {t + 1}\right) - H \left(\theta_ {t}\right) \leq - \frac {A}{2} \| \nabla H \left(\theta_ {t}\right) \| ^ {2}. \tag {15}
$$
Combining (13) and (15) gives
$$
\left[ H \left(\theta_ {t}\right) - H \left(\theta^ {*}\right) \right] ^ {2} \leq \frac {2 D ^ {2}}{A} \left(\left[ H \left(\theta_ {t}\right) - H \left(\theta^ {*}\right) \right] - \left[ H \left(\theta_ {t + 1}\right) - H \left(\theta^ {*}\right) \right]\right) \tag {16}
$$
which, after divided by  $\left[H(\theta_t) - H(\theta^*)\right]\left[H(\theta_{t + 1}) - H(\theta^*)\right]$  on both sides, leads to
$$
\frac {H \left(\theta_ {t}\right) - H \left(\theta^ {*}\right)}{H \left(\theta_ {t + 1}\right) - H \left(\theta^ {*}\right)} \leq \frac {2 D ^ {2}}{A} \left(\frac {1}{H \left(\theta_ {t + 1}\right) - H \left(\theta^ {*}\right)} - \frac {1}{H \left(\theta_ {t}\right) - H \left(\theta^ {*}\right)}\right). \tag {17}
$$
According to (15),  $H(\theta_{t + 1}) \leq H(\theta_t)$ . Therefore,  $\frac{H(\theta_t) - H(\theta^*)}{H(\theta_{t + 1}) - H(\theta^*)} \geq 1$ , and (17) can be simplified as
$$
\frac {1}{H \left(\theta_ {t + 1}\right) - H \left(\theta^ {*}\right)} - \frac {1}{H \left(\theta_ {t}\right) - H \left(\theta^ {*}\right)} \geq \frac {A}{2 D ^ {2}}. \tag {18}
$$
Summing up (18) for  $t = 0,1,\ldots ,T - 1$  gives
$$
\begin{array}{l} \frac {1}{H \left(\theta_ {T}\right) - H \left(\theta^ {*}\right)} \geq \frac {A T}{2 D ^ {2}} + \frac {1}{H \left(\theta_ {0}\right) - H \left(\theta^ {*}\right)} \tag {19} \\ \geq \frac {A T}{2 D ^ {2}} \\ \end{array}
$$
$$
H \left(\theta_ {T}\right) - H \left(\theta^ {*}\right) \leq \frac {2 D ^ {2}}{A T}. \tag {20}
$$
Case 2. Suppose  $\exists t_0 \in \{0,1,\dots,T - 1\}$ , such that  $B > \frac{A}{2}\| \nabla H(\theta_{t_0})\| ^2$ . In this case, we have
$$
\left\| \nabla H \left(\theta_ {t _ {0}}\right) \right\| <   \sqrt {\frac {2 B}{A}}. \tag {21}
$$
Combining (13) and (21) gives
$$
H \left(\theta_ {t _ {0}}\right) - H \left(\theta^ {*}\right) <   D \sqrt {\frac {2 B}{A}}. \tag {22}
$$
Next we show by contradiction that for all  $t \geq t_0$ , it holds that
$$
H \left(\theta_ {t}\right) - H \left(\theta^ {*}\right) \leq D \sqrt {\frac {2 B}{A}} + B. \tag {23}
$$
Suppose that there exists  $t_1 \geq t_0$  such that
$$
H \left(\theta_ {t _ {1}}\right) - H \left(\theta^ {*}\right) > D \sqrt {\frac {2 B}{A}} + B. \tag {24}
$$
According to (14), we have
$$
\begin{array}{r l} H \left(\theta_ {t _ {1}}\right) - H \left(\theta_ {t _ {1} - 1}\right) & \leq - A \| \nabla H \left(\theta_ {t _ {1} - 1}\right) \| ^ {2} + B \\ & \leq B. \end{array} \tag {25}
$$
Combining (24) and (25) gives
$$
H \left(\theta_ {t _ {1} - 1}\right) - H \left(\theta^ {*}\right) > D \sqrt {\frac {2 B}{A}}. \tag {26}
$$
Combining (26) and (13) gives
$$
\left\| \nabla H \left(\theta_ {t _ {1} - 1}\right) \right\| > \sqrt {\frac {2 B}{A}}. \tag {27}
$$
Plugging (27) into (14), we obtain  $H(\theta_{t_1 - 1}) \geq H(\theta_{t_1}) + B$ , which suggests that (24) also holds with  $t_1$  replaced by  $t_1 - 1$ . By the same token, we can conclude that (24) should hold with  $t_1$  replaced by all  $t \leq t_1$ . This is in clear contradiction with the incident of  $t = t_0$  as shown in (22). Therefore, (23) is valid for all  $t \geq t_0$  as stated.
Finally, combining the results of Case 1 (20) and Case 2 (23), we achieve that
$$
\begin{array}{l} H \left(\theta_ {T}\right) - H \left(\theta^ {*}\right) \leq \max  \left\{\frac {2 D ^ {2}}{A T}, D \sqrt {\frac {2 B}{A}} + B \right\} \\ = \max \left\{\frac {4 L _ {H} D ^ {2}}{(1 - (1 + r) C _ {\alpha} ^ {2}) T}, \sqrt {\frac {2 (1 + 1 / r)}{1 - (1 + r) C _ {\alpha} ^ {2}}} D \sigma + \frac {(1 + 1 / r) \sigma^ {2}}{2 L _ {H}} \right\} \\ \end{array}
$$
which completes the proof of Theorem 2.
According to Assumption 2,  $H(\theta)$  is  $L_{H}$ -smooth, and according to Assumption 5,  $H(\theta)$  is  $\lambda_{H}$ -strongly convex. In convex optimization theory, it is well known that smooth and strongly convex functions enjoy linear convergence rate with gradient descent. Here we will first establish and then use such a property with a specific convergence factor. We start with the following equality
$$
\left\| \theta_ {t} - \eta \nabla H \left(\theta_ {t}\right) - \theta^ {*} \right\| ^ {2} = \left\| \theta_ {t} - \theta^ {*} \right\| ^ {2} - 2 \eta \langle \nabla H \left(\theta_ {t}\right), \theta_ {t} - \theta^ {*} \rangle + \eta^ {2} \| \nabla H \left(\theta_ {t}\right) \| ^ {2}. \tag {29}
$$
According to the co-coercivity of smooth and strongly convex function, we have
$$
\left\langle \nabla H \left(\theta_ {t}\right) - \nabla H \left(\theta^ {*}\right), \theta_ {t} - \theta^ {*} \right\rangle \geq \frac {1}{L _ {H} + \lambda_ {H}} \| \nabla H \left(\theta_ {t}\right) - \nabla H \left(\theta^ {*}\right) \| ^ {2} + \frac {L _ {H} \lambda_ {H}}{L _ {H} + \lambda_ {H}} \| \theta_ {t} - \theta^ {*} \| ^ {2}. \tag {30}
$$
Since  $\theta^{*}$  is the global minimizer of  $H(\theta)$  and therefore  $\nabla H(\theta^{*}) = 0$ , (30) reduces to
$$
\left\langle \nabla H \left(\theta_ {t}\right), \theta_ {t} - \theta^ {*} \right\rangle \geq \frac {1}{L _ {H} + \lambda_ {H}} \| \nabla H \left(\theta_ {t}\right) \| ^ {2} + \frac {L _ {H} \lambda_ {H}}{L _ {H} + \lambda_ {H}} \| \theta_ {t} - \theta^ {*} \| ^ {2}. \tag {31}
$$
Plugging (31) into (29), we have
$$
\left\| \theta_ {t} - \eta \nabla H \left(\theta_ {t}\right) - \theta^ {*} \right\| ^ {2} \leq \left(1 - 2 \eta \frac {L _ {H} \lambda_ {H}}{L _ {H} + \lambda_ {H}}\right) \left\| \theta_ {t} - \theta^ {*} \right\| ^ {2} + \left(\eta^ {2} - \frac {2 \eta}{L _ {H} + \lambda_ {H}}\right) \| \nabla H (\theta_ {t}) \| ^ {2}. \tag {32}
$$
In order to eliminate the last term in (32), we take  $\eta = \frac{2}{L_H + \lambda_H}$  and simplify (32) as
$$
\left\| \theta_ {t} - \eta \nabla H \left(\theta_ {t}\right) - \theta^ {*} \right\| ^ {2} \leq \left(1 - \frac {4 L _ {H} \lambda_ {H}}{\left(L _ {H} + \lambda_ {H}\right) ^ {2}}\right) \left\| \theta_ {t} - \theta^ {*} \right\| ^ {2} \tag {33}
$$
which is the same as
$$
\left\| \theta_ {t} - \eta \nabla H \left(\theta_ {t}\right) - \theta^ {*} \right\| \leq \frac {L _ {H} - \lambda_ {H}}{L _ {H} + \lambda_ {H}} \left\| \theta_ {t} - \theta^ {*} \right\| \tag {34}
$$
which verifies linear convergence with a factor of  $\frac{L_H - \lambda_H}{L_H + \lambda_H}$ . Note that (34) holds on condition that  $\eta = \frac{2}{L_H + \lambda_H}$ .
Next, we try to evaluate the single-step progress made by our algorithm as follows:
$$
\begin{array}{l} \left\| \theta_ {t + 1} - \theta^ {*} \right\| = \left\| \theta_ {t} - \eta G (\theta_ {t}) - \theta^ {*} \right\| \\ = \| \theta_ {t} - \eta \nabla H (\theta_ {t}) - \theta^ {*} + \eta [ \nabla H (\theta_ {t}) - G (\theta_ {t}) ] \| \\ \leq \left\| \theta_ {t} - \eta \nabla H \left(\theta_ {t}\right) - \theta^ {*} \right\| + \eta \left\| \nabla H \left(\theta_ {t}\right) - G \left(\theta_ {t}\right) \right\| \\ \leq \frac {L _ {H} - \lambda_ {H}}{L _ {H} + \lambda_ {H}} \| \theta_ {t} - \theta^ {*} \| + \frac {2}{L _ {H} + \lambda_ {H}} \| G (\theta_ {t}) - \nabla H (\theta_ {t}) \| \tag {35} \\ \leq \frac {L _ {H} - \lambda_ {H}}{L _ {H} + \lambda_ {H}} \| \theta_ {t} - \theta^ {*} \| + \frac {2 C _ {\alpha}}{L _ {H} + \lambda_ {H}} \| \nabla H (\theta_ {t}) \| + \frac {2 \sigma}{L _ {H} + \lambda_ {H}} \\ \end{array}
$$
in which the second inequality follows from (34) by taking  $\eta = \frac{2}{L_H + \lambda_H}$ , and the third inequality follows from Lemma 1 and Assumption 1.
According to the properties of  $H(\theta)$  being  $L_{H}$ -smooth, we have
$$
\frac {1}{2 L _ {H}} \| \nabla H (\theta_ {t}) \| ^ {2} \leq H (\theta_ {t}) - H (\theta^ {*}) \leq \frac {L _ {H}}{2} \| \theta_ {t} - \theta^ {*} \| ^ {2}
$$
$$
\left\| \nabla H \left(\theta_ {t}\right) \right\| \leq L _ {H} \left\| \theta_ {t} - \theta^ {*} \right\|. \tag {36}
$$
Plugging (36) into (35), we have
$$
\left\| \theta_ {t + 1} - \theta^ {*} \right\| \leq \frac {2 L _ {H} C _ {\alpha} + L _ {H} - \lambda_ {H}}{L _ {H} + \lambda_ {H}} \left\| \theta_ {t} - \theta^ {*} \right\| + \frac {2 \sigma}{L _ {H} + \lambda_ {H}}. \tag {37}
$$
By iterating (37) we obtain
$$
\left\| \theta_ {T} - \theta^ {*} \right\| \leq \left(\frac {2 L _ {H} C _ {\alpha} + L _ {H} - \lambda_ {H}}{L _ {H} + \lambda_ {H}}\right) ^ {T} \left\| \theta_ {0} - \theta^ {*} \right\| + \frac {\sigma}{\lambda_ {H} - L _ {H} C _ {\alpha}} \tag {38}
$$
which is exactly the conclusion in Theorem 3.
Note that the transition from (37) to (38) only stands under the condition that  $\frac{2L_H C_\alpha + L_H - \lambda_H}{L_H + \lambda_H} < 1$ , which requires  $C_\alpha = \frac{2\alpha}{1 - \beta} < \frac{\lambda_H}{L_H}$ , i.e.,  $2\alpha \frac{L_H}{\lambda_H} + \beta < 1$ . Since  $\beta \geq \alpha$ , we can conclude that (38) holds if and only if  $\alpha < \frac{1}{1 + 2L_H / \lambda_H}$ .
E The interplay between robustness and fairness in two additional datasets
Table 3: The performance of H-nobs on the Law School dataset (with no attack).
β = 0	β = 10%	β = 20%	β = 30%	(η, T)
q = 0	90.7% (559)	90.6% (523)	90.6% (512)	90.6% (507)	(0.1, 500)
q = 0.5	89.8% (483)	90.3% (478)	90.3% (470)	90.4% (465)	(0.05, 1000)
q = 1	87.3% (353)	89.9% (449)	90.2% (451)	90.2% (448)	(0.02, 2000)
q = 2	66.2% (228)	73.3% (428)	74.0% (447)	73.9% (445)	(0.005, 5000)
Table 4: The performance of H-nobs on the Credit Card Client dataset (with no attack).
β = 0	β = 10%	β = 20%	β = 30%	(η, T)
q = 0	80.7% (1195)	80.5% (1143)	80.6% (1181)	80.6% (1148)	(0.2, 500)
q = 0.5	81.0% (1123)	80.3% (1081)	80.3% (1140)	80.4% (1105)	(0.1, 2000)
q = 1	81.0% (987)	80.1% (1006)	79.8% (1058)	80.0% (1035)	(0.1, 2000)
q = 2	78.9% (869)	79.1% (887)	79.0% (930)	79.3% (966)	(0.05, 5000)
For both the Law School dataset and the Credit Card Client dataset, we use  $\frac{2}{3}$  as the training/testing split ratio and divide the training data into two groups based on the male/female feature (instead of the label), which is a commonly considered attribute in fairness research. Similar to the previous setup, we distribute the grouped training data across 20 nodes in a balanced manner. The performance of H-nobs on these two datasets is detailed in Table 3 and Table 4, respectively.
Tables 3 and 4 exhibit a pattern similar to that in Table 1: when the variance is high (under small  $q$  regimes), increasing  $\beta$  does not impair fairness; however, when  $q$ -FFL achieves significant fairness gains (under large  $q$  regimes), fairness is compromised by the screening operation. Note that here we only consider the effects of screening, not in combination with attacks. When Byzantine attacks are at play, Table 2 demonstrates satisfying robustness performance of H-nobs, although the fairness gains achieved by  $q$ -FFL can vary considerably depending on the attack regimes.