SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbf {v} = \boldsymbol {H} \left(\boldsymbol {\theta} ^ {*}; p (\mathbf {z})\right) \boldsymbol {\delta} + o \left(\| \boldsymbol {\delta} \|\right). \tag {18}
$$
Conduct the Second-order Taylor expansion on function  $f$  near 1 in  $D_{f}\big(q(\mathbf{z})||p(\mathbf{z})\big)$ , we have
$$
f \left(\frac {q (\mathbf {z})}{p (\mathbf {z})}\right) = f ^ {\prime} (1) r (\mathbf {z}) + \frac {f ^ {\prime \prime} (1)}{2} r ^ {2} (\mathbf {z}) + o \left(r ^ {2} (\mathbf {z})\right). \tag {19}
$$
$$
\begin{array}{l} D _ {f} (q | | p) = \int_ {\mathbf {z}} p (\mathbf {z}) f \left(\frac {q (\mathbf {z})}{p (\mathbf {z})}\right) d \mathbf {z} = \mathbb {E} _ {p} \left[ f \left(\frac {q (\mathbf {z})}{p (\mathbf {z})}\right) \right] = \mathbb {E} _ {p} \left[ f ^ {\prime} (1) r + \frac {f ^ {\prime \prime} (1)}{2} r ^ {2} + o \left(r ^ {2}\right) \right] (20) \\ = \frac {f ^ {\prime \prime} (1)}{2} \mathbb {E} _ {p} \left[ r ^ {2} (\mathbf {z}) \right] + o \left(r ^ {2} (\mathbf {z})\right) = \left(1 + o (1)\right) \frac {f ^ {\prime \prime} (1)}{2} \mathbb {E} _ {p} \left[ r ^ {2} (\mathbf {z}) \right]. (21) \\ \end{array}
$$
Define  $\mathcal{I}[r(\mathbf{z})] = \mathbb{E}_p\left[r^2 (\mathbf{z})\right],\left(D_f(q||p)\right)_{\min} = \big(1 + o(1)\big)f''(1)\left(\mathcal{I}[r(\mathbf{z})]\right)_{\min} / 2.$  The infimum is:
$$
\operatorname {I n f} _ {r (\mathbf {z})} \quad \mathcal {I} [ r (\mathbf {z}) ] = \mathbb {E} _ {p} \left[ r ^ {2} (\mathbf {z}) \right], \tag {22}
$$
$$
\text {s u b j e c t} \mathbb {E} _ {p} [ r (\mathbf {z}) ] = 0, \quad \mathbb {E} _ {p} [ r (\mathbf {z}) \nabla_ {\theta} \mathcal {L} \left(\boldsymbol {\theta} ^ {*}; \mathbf {z}\right) ] = \mathbf {v}. \tag {23}
$$
Define the Lagrange multiplier as  $\mathcal{L}[r(\mathbf{z})] = \frac{1}{2}\mathcal{I}[r(\mathbf{z})] - \alpha \mathbb{E}_p[r(\mathbf{z})] - \beta^{\mathrm{T}}\big(\mathbb{E}_p[r(\mathbf{z})\nabla_\theta \mathcal{L}(\boldsymbol{\theta}^*;\mathbf{z})] - \mathbf{v}\big)$ , when  $\mathcal{I}[r(\mathbf{z})]$  is optimal,  $\delta \mathcal{L}[r(\mathbf{z})] / \delta r(\mathbf{z}) = 0$ , namely:
$$
r (\mathbf {z}) - \alpha - \boldsymbol {\beta} ^ {\mathrm {T}} \nabla_ {\boldsymbol {\theta}} \mathcal {L} \left(\boldsymbol {\theta} ^ {*}; \mathbf {z}\right) = 0. \tag {24}
$$
Therefore,  $r(\mathbf{z}) = \alpha +\beta^{\mathrm{T}}\nabla_{\pmb{\theta}}\mathcal{L}(\pmb{\theta}^{*};\mathbf{z})$  . We have:
$$
0 = \mathbb {E} _ {p} [ r (\mathbf {z}) ] = \alpha + \boldsymbol {\beta} ^ {\mathrm {T}} \mathbb {E} _ {p} \left[ \nabla_ {\boldsymbol {\theta}} \mathcal {L} \left(\boldsymbol {\theta} ^ {*}; \mathbf {z}\right) \right] = \alpha + \boldsymbol {\beta} ^ {\mathrm {T}} \nabla_ {\boldsymbol {\theta}} \mathcal {L} \left(\boldsymbol {\theta} ^ {*}; p (\mathbf {z})\right) = \alpha , \tag {25}
$$
$$
\mathbf {v} = \mathbb {E} _ {p} \left[ r (\mathbf {z}) \nabla_ {\boldsymbol {\theta}} \mathcal {L} \left(\boldsymbol {\theta} ^ {*}; \mathbf {z}\right) \right] = \mathbb {E} _ {p} \left[ \nabla_ {\boldsymbol {\theta}} \mathcal {L} \left(\boldsymbol {\theta} ^ {*}; \mathbf {z}\right) \nabla_ {\boldsymbol {\theta}} \mathcal {L} \left(\boldsymbol {\theta} ^ {*}; \mathbf {z}\right) ^ {\mathrm {T}} \boldsymbol {\beta} \right] = \boldsymbol {H} \left(\boldsymbol {\theta} ^ {*}; p (\mathbf {z})\right) \boldsymbol {\beta}. \tag {26}
$$
To conclude:
$$
\alpha = 0, \quad \boldsymbol {H} \left(\boldsymbol {\theta} ^ {*}; p (\mathbf {z})\right) \boldsymbol {\beta} = \mathbf {v}. \tag {27}
$$
Therefore, when  $\mathcal{I}[r(\mathbf{z})]$  is optimal:
$$
r (\mathbf {z}) = \left(\boldsymbol {H} ^ {- 1} \left(\boldsymbol {\theta} ^ {*}; p (\mathbf {z})\right) \mathbf {v}\right) ^ {\mathrm {T}} \nabla_ {\boldsymbol {\theta}} \mathcal {L} \left(\boldsymbol {\theta} ^ {*}; \mathbf {z}\right) = \mathbf {v} ^ {\mathrm {T}} \boldsymbol {H} ^ {- 1} \left(\boldsymbol {\theta} ^ {*}; p (\mathbf {z})\right) \nabla_ {\boldsymbol {\theta}} \mathcal {L} \left(\boldsymbol {\theta} ^ {*}; \mathbf {z}\right), \tag {28}
$$
thus the optimal  $\mathcal{I}[r(\mathbf{z})]$  is:
$$
\begin{array}{l} \left(\mathcal {I} [ r (\mathbf {z}) ]\right) _ {\min } = \mathbb {E} _ {p} \left[ r (\mathbf {z}) ^ {2} \right] (29) \\ = \mathbb {E} _ {p} \left[ \mathbf {v} ^ {\mathrm {T}} \boldsymbol {H} ^ {- 1} \left(\boldsymbol {\theta} ^ {*}; p (\mathbf {z})\right) \nabla_ {\boldsymbol {\theta}} \mathcal {L} \left(\boldsymbol {\theta} ^ {*}; \mathbf {z}\right) \left(\mathbf {v} ^ {\mathrm {T}} \boldsymbol {H} ^ {- 1} \left(\boldsymbol {\theta} ^ {*}; p (\mathbf {z})\right) \nabla_ {\boldsymbol {\theta}} \mathcal {L} \left(\boldsymbol {\theta} ^ {*}; \mathbf {z}\right)\right) ^ {\mathrm {T}} \right] (30) \\ = \mathbf {v} ^ {\mathrm {T}} \boldsymbol {H} ^ {- 1} \left(\boldsymbol {\theta} ^ {*}; p (\mathbf {z})\right) \mathbb {E} _ {p} \left[ \nabla_ {\boldsymbol {\theta}} \mathcal {L} \left(\boldsymbol {\theta} ^ {*}; \mathbf {z}\right) \nabla_ {\boldsymbol {\theta}} \mathcal {L} \left(\boldsymbol {\theta} ^ {*}; \mathbf {z}\right) ^ {\mathrm {T}} \right] \boldsymbol {H} ^ {- 1} \left(\boldsymbol {\theta} ^ {*}; p (\mathbf {z})\right) \mathbf {v} (31) \\ = \mathbf {v} ^ {\mathrm {T}} \boldsymbol {H} ^ {- 1} \left(\boldsymbol {\theta} ^ {*}; p (\mathbf {z})\right) \mathbf {v}, (32) \\ \end{array}
$$
since  $\mathbf{v} = H\big(\pmb{\theta}^{*}; p(\mathbf{z})\big)\pmb{\delta} + o(\|\pmb{\delta}\|) = \big(1 + o(1)\big)\pmb{H}\big(\pmb{\theta}^{*}; p(\mathbf{z})\big)\pmb{\delta}$ :
$$
\left(\mathcal {I} [ r (\mathbf {z}) ]\right) _ {\min } = \left(1 + o (1)\right) \boldsymbol {\delta} ^ {\mathrm {T}} \boldsymbol {H} \left(\boldsymbol {\theta} ^ {*}; p (\mathbf {z})\right) \boldsymbol {\delta} = \left(1 + o (1)\right) \sum_ {k = 1} ^ {d} H _ {k} ^ {*} \delta_ {k} ^ {2}, \tag {33}
$$
where  $H_{k}^{*} = H_{k}\big(\pmb{\theta}^{*};p(\mathbf{z})\big)$
Therefore, the solution is
$$
\left(D _ {f} (q | | p)\right) _ {\min } = \left(1 + o (1)\right) \frac {f ^ {\prime \prime} (1)}{2} \left(\mathcal {I} [ r (\mathbf {z}) ]\right) _ {\min } = \left(1 + o (1)\right) \frac {f ^ {\prime \prime} (1)}{2} \sum_ {k = 1} ^ {d} H _ {k} ^ {*} \delta_ {k} ^ {2}. \tag {34}
$$
$$
\left. D _ {f} \left(q (\mathbf {z}) | | p (\mathbf {z})\right) \geq \left(D _ {f} (q | | p)\right) _ {\min } = \left(1 + o (1)\right) \frac {f ^ {\prime \prime} (1)}{2} \mathcal {I} _ {\mathrm {F - D i v}}, \quad \mathcal {I} _ {\mathrm {F - D i v}} = \sum_ {k = 1} ^ {d} H _ {k} ^ {*} \delta_ {k} ^ {2}. \right. \tag {35}
$$
A.4 Proofs of Theorem 2
Theorem 2 (Hessian Estimations by Diffusion Theory, Detailed Version). When  $\sqrt{H_k^*} = \sqrt{H_k(\pmb{\theta}^*;p(\mathbf{z}))}$  are small, there exists a constant  $C > 0$  for any  $\epsilon > 0$  that the following inequality holds with a probability higher than  $1 - \epsilon$  for large  $n$ :
$$
\left| \frac {\frac {1}{n} \sum_ {i = 1} ^ {n} \left| u _ {k} ^ {(i)} \right|}{C \sqrt {H _ {k} ^ {*}}} - 1 \right| <   \sqrt {\frac {\pi - 2}{2 n \epsilon}}, \tag {36}
$$
where  $u_{k}^{(i)}$  is the  $k$ -th dimension of  $\mathbf{u}_t^{(i)}$ .
Proof. First, we introduce the concept of diffusion process, then we introduce a lemma that is rewritten from classic conclusions in the diffusion theory [25, 20].
The training dynamics on one client can be modeled as a diffusion process [38, 25, 20] with Stochastic Gradient Noise (SGN):
$$
d \boldsymbol {\theta} = - \nabla_ {\boldsymbol {\theta}} \mathcal {L} (\boldsymbol {\theta}; p (\mathbf {z})) d t + \sqrt {2 D} d \mathbf {W} _ {t}, \tag {37}
$$
where  $dt$  is the unit time or the step size,  $D$  is the diffusion coefficient, and  $d\mathbf{W}_t \sim N(0, Idt)$ . We assume the gradient noise introduced by stochastic learning is small (the temperature of the diffusion process is low). Here we also assume that the data distributions of all clients approximately obey the merged data distribution  $p(\mathbf{z})$ .
The diffusion coefficient matrix  $D$  is a diagonal matrix, and its value in the Stochastic Gradient Descent (SGD) dynamics is:
$$
D _ {k} = \frac {\eta}{2 B} H _ {k} ^ {*}, \tag {38}
$$
where  $\eta = dt$  is the unit time or the step size,  $B$  is the batch size, and  $H_{k}^{*} = H_{k}\big(\pmb{\theta}^{*};p(\mathbf{z})\big)$ .
Its value in the dynamics involving adaptive learning rate mechanisms, take the Adam [17] optimizer for example, can also be seen as  $D_{k} \approx \frac{\eta}{2B} H_{k}^{*}$  when  $\sqrt{H_k^*}$  are small. Since the parameter update is:
$$
\Delta \boldsymbol {\theta} = - \hat {\eta} \odot \mathbf {m}, \tag {39}
$$
where  $\mathbf{m}$  can be seen as an SGD update with the momentum mechanism and  $\mathbb{E}[\mathbf{m}] = \nabla_{\theta}\mathcal{L}(\pmb {\theta};p(\mathbf{z}))$  in a stationary distribution. In Adam,  $\hat{\eta} = \eta (\sqrt{\mathbf{v}} +\epsilon)^{-1}$  and  $\mathbb{E}[\mathbf{v}] = \mathbb{E}_{\mathbf{z}\sim p(\mathbf{z})}[\nabla_{\theta}\mathcal{L}(\pmb {\theta};\mathbf{z})\odot \nabla_{\theta}\mathcal{L}(\pmb {\theta};\mathbf{z})]\approx$ $\mathbf{H}(\pmb {\theta}^{*};p(\mathbf{z}))$  in a stationary distribution. Therefore, when  $\sqrt{\mathbf{v}}$  are small, the weight update can be approximated with:
$$
\Delta \boldsymbol {\theta} \approx - \eta \epsilon^ {- 1} \mathbf {m}, \tag {40}
$$
which can be seen as an SGD update with the learning rate  $\eta \epsilon^{-1}$  and the gradient  $\mathbf{m}$  on a small batch.
To conclude, there exists a constant  $C_1 > 0$  that the diffusion coefficients  $D_{k}$  of (1) all dimensions in the Stochastic Gradient Descent (SGD) dynamics; or (2) dimensions with small  $\sqrt{H_k^*}$  in dynamics involving adaptive learning rate mechanisms, such as Adam [17]; are:
$$
D _ {k} \approx C _ {1} H _ {k} ^ {*}, \tag {41}
$$
We introduce Lemma A.2 that is rewritten from classic conclusions in the diffusion theory [25, 20]:
Lemma A.2. When the training dynamics on the  $i$ -th client can be modeled as a diffusion process with Stochastic Gradient Noise (SGN), there exists a constant  $C_2 > 0$  that the updates obey the Gaussian distributions on all dimensions:
$$
u _ {k} ^ {(i)} \sim N \left(0, C _ {2} D _ {k}\right). \tag {42}
$$
According to Lemma A.2,
$$
\mathbb {E} \left[ \left| u _ {k} ^ {(i)} \right| \right] = \int_ {- \infty} ^ {+ \infty} \frac {\left| x \right| \exp \left(- \frac {x ^ {2}}{2 C _ {2} D _ {k}}\right)}{\sqrt {2 \pi C _ {2} D _ {k}}} d x = \sqrt {\frac {2 C _ {2} D _ {k}}{\pi}}, \tag {43}
$$
$$
\mathbb {E} \left[ \left| u _ {k} ^ {(i)} \right| ^ {2} \right] = \int_ {- \infty} ^ {+ \infty} \frac {\left| x \right| ^ {2} \exp \left(- \frac {x ^ {2}}{2 C _ {2} D _ {k}}\right)}{\sqrt {2 \pi C _ {2} D _ {k}}} d x = C _ {2} D _ {k}, \tag {44}
$$
$$
\mathbb {D} \left[ \left| u _ {k} ^ {(i)} \right| \right] = \mathbb {E} \left[ \left| u _ {k} ^ {(i)} \right| ^ {2} \right] - \mathbb {E} \left[ \left| u _ {k} ^ {(i)} \right| \right] ^ {2} = (1 - \frac {2}{\pi}) C _ {2} D _ {k}. \tag {45}
$$
The Chebyshev Inequality demonstrates that:
$$
P \left(\left| X - \mathbb {E} [ X ] \right| <   c\right) > 1 - \frac {\mathbb {D} [ X ]}{c ^ {2}}, \tag {46}
$$
we choose  $X = \frac{1}{n}\sum_{i = 1}^{n}|u_k^{(i)}|$ , then:
$$
\mathbb {E} [ X ] = \sqrt {\frac {2 C _ {2} D _ {k}}{\pi}}, \quad \mathbb {D} [ X ] = \frac {(1 - \frac {2}{\pi}) C _ {2} D _ {k}}{n}. \tag {47}
$$
We choose  $c = \sqrt{\frac{\mathbb{D}[X]}{\epsilon}}$ , then:
$$
\left| \frac {\sum_ {i = 1} ^ {n} \left| u _ {k} ^ {(i)} \right|}{n} - \sqrt {\frac {2 C _ {2} D _ {k}}{\pi}} \right| <   \sqrt {\frac {\mathbb {D} [ X ]}{\epsilon}}, \tag {48}
$$
holds with a probability higher than  $1 - \epsilon$ .
For dimensions with small Hessians, there exists a constant  $C_1 > 0$  that  $D_k = C_1 H_k^*$ . Therefore, there exists a constant  $C = \sqrt{\frac{2C_2C_1}{\pi}}$  for any  $\epsilon > 0$  that the following inequality holds with a probability higher than  $1 - \epsilon$ :
$$
\left| \frac {\frac {1}{n} \sum_ {i = 1} ^ {n} \left| u _ {k} ^ {(i)} \right|}{C \sqrt {H _ {k} ^ {*}}} - 1 \right| <   \sqrt {\frac {\pi - 2}{2 n \epsilon}}. \tag {49}
$$
A.5 Proofs of Theorem 3
We verify the Byzantine resilience of Fed-FA in Theorem 3.
Preliminary. Suppose  $\mathbf{g} = \mathbb{E}[\mathbf{u}],\mathbf{g}^{*} = \mathbb{E}[\mathbf{u}^{*}]$  are the expected updates for clean updates  $\mathbf{u}$  and malicious updates  $\mathbf{u}^*$ . Suppose  $g_{k},g_{k}^{*},u_{k}$  denote the  $k$ -th dimension of  $\mathbf{g},\mathbf{g}^{*},\mathbf{u}$ .
We also require that the gradient noises are bounded linearly by  $g_{k}^{2}$ , namely, there exists  $\eta >0$  such that  $\mathbb{D}[u_k]\leq \eta |\mathbb{E}[u_k]|^2 = \eta g_k^2$  for any dimension  $k$  for clean updates  $\mathbf{u}$ . We define  $\eta$  as the gradient noise scale. [1] assume that  $\mathbb{D}[u_k] = \sigma^2$  and thus  $\mathbb{D}[u_k]$  is a fixed value  $\sigma^2$ . However, in this work, we allow  $\mathbb{D}[u_k]$  to be arbitrarily large when  $g_{k}^{2}$  grows very large and  $\mathbb{D}[u_k]$  is only required to be bounded with a linear bound of  $g_{k}^{2}$  instead, namely  $\eta g_{k}^{2}$ .
According to Fisher information matrix assumption [30],  $H_{k}^{*} = \mathbb{D}[u_{k}]$
Theorem 3 (Byzantine Resilience of Fed-FA. Detailed Version.). Assume the loss function on the merged dataset  $p(\mathbf{z})$  is locally  $\mu$ -strongly convex and locally  $L$ -smooth near the optimal parameter. For  $m$  malicious clients,  $1 \leq m \leq \lfloor \frac{n - 1}{2} \rfloor$ , when the estimations of indicators are accurate enough and the gradient noise scale  $\eta$  is small enough, there exists  $0 \leq \alpha < \frac{\pi}{2}$  such that Fed-FA aggregation algorithm is  $(\alpha, m)$ -Byzantine resilience:
$$
\left\| \mathbb {E} \left[ \mathcal {A} \left(\left\{\mathbf {u} ^ {(i)} \right\} _ {i = 1} ^ {n}\right) \right] - \mathbf {g} \right\| _ {2} \leq \sin \alpha \| \mathbf {g} \| _ {2}, \quad \mathbf {g} ^ {T} \mathbb {E} \left[ \mathcal {A} \left(\left\{\mathbf {u} ^ {(i)} \right\} _ {i = 1} ^ {n}\right) \right] \geq (1 - \sin \alpha) \| \mathbf {g} \| _ {2} ^ {2}, \tag {50}
$$
where  $\mathbf{g} = \mathbb{E}[\mathbf{u}],\mathbf{g}^{*} = \mathbb{E}[\mathbf{u}^{*}]$  are the expectations for clean updates  $\mathbf{u}$  and malicious updates  $\mathbf{u}^*$ .
Proof. We assume that the estimations of Hessians are accurate, namely the estimations of  $\mathcal{I}_{\mathrm{F - Div}}^{(k)}$  are accurate. Suppose  $\mathcal{I}_{\mathrm{F - Div}}^{(1)}\leq \mathcal{I}_{\mathrm{F - Div}}^{(2)}\leq \mathcal{I}_{\mathrm{F - Div}}^{(3)}\leq \dots \mathcal{I}_{\mathrm{F - Div}}^{(n)}$ .
Then we have  $S = \{1,2,\dots ,\lceil \frac{n + 1}{2}\rceil \}$ ,  $|S| = \lceil \frac{n + 1}{2}\rceil \geq n - m > m$ . Suppose  $M$  is the set of malicious client indexes, and  $|M| = m$ . Define  $M_1 = \{i:i\leq |S|,i\in M\}$ ,  $|M_{1}|\leq |M|\leq m\leq \lfloor \frac{n - 1}{2}\rfloor$ , then,
$$
\mathcal {A} \left(\left\{\mathbf {u} ^ {(i)} \right\} _ {i = 1} ^ {n}\right) = \frac {1}{| S |} \sum_ {i = 1} ^ {| S |} \mathbf {u} ^ {(i)} = \frac {1}{| S |} \left(\sum_ {i \in S \backslash M _ {1}} \mathbf {u} ^ {(i)} + \sum_ {i \in M _ {1}} \mathbf {u} ^ {(i)}\right). \tag {51}
$$
If  $M_1 = \emptyset$ , namely  $S \setminus M_1 = S$ , we have:
$$
\mathbb {E} [ \mathcal {A} (\{\mathbf {u} ^ {(i)} \} _ {i = 1} ^ {n}) ] = \frac {1}{| S |} \left(\sum_ {i \in S \backslash M _ {1}} \mathbb {E} [ \mathbf {u} ^ {(i)} ]\right) = \frac {1}{| S |} \left(\sum_ {i \in S} \mathbb {E} [ \mathbf {u} ^ {(i)} ]\right) = \mathbf {g}, \tag {52}
$$
namely there exists  $\alpha = 0$  such that:
$$
\left\| \mathbb {E} \left[ \mathcal {A} \left(\left\{\mathbf {u} ^ {(i)} \right\} _ {i = 1} ^ {n}\right) \right] - \mathbf {g} \right\| \leq \sin \alpha \| \mathbf {g} \| _ {2}, \quad \mathbf {g} ^ {\mathrm {T}} \mathbb {E} \left[ \mathcal {A} \left(\left\{\mathbf {u} ^ {(i)} \right\} _ {i = 1} ^ {n}\right) \right] = \| \mathbf {g} \| _ {2} ^ {2} \geq (1 - \sin \alpha) \| \mathbf {g} \| _ {2} ^ {2}. \tag {53}
$$
Otherwise, we have  $|M_1| > 0$ ,  $|S| - |M_1| \geq (n - m) - m > 0$ :
$$
\mathbb {E} \left[ \mathcal {A} \left(\left\{\mathbf {u} ^ {(i)} \right\} _ {i = 1} ^ {n}\right) \right] = \frac {1}{| S |} \left(\sum_ {i \in S \backslash M _ {1}} \mathbb {E} \left[ \mathbf {u} ^ {(i)} \right] + \sum_ {i \in M _ {1}} \mathbb {E} \left[ \mathbf {u} ^ {(i)} \right]\right) = \frac {\left(| S | - | M _ {1} |\right) \mathbf {g} + | M _ {1} | \mathbf {g} ^ {*}}{| S |}. \tag {54}
$$
For any  $i \in M_1$ , there exists a clean client  $j \notin S \setminus M_1$  because we have  $n - m$  clean clients but clean client number in  $S$  is  $|S| - |M_1| < |S| = \lceil \frac{n + 1}{2} \rceil \leq n - m$ . Therefore  $\mathcal{I}_{\mathrm{F - Div}}^{(j)} \geq \mathcal{I}_{\mathrm{F - Div}}^{(i)}$ , namely:
$$
\sum_ {k = 1} ^ {d} H _ {k} ^ {*} \left(u _ {k} ^ {(j)} - g _ {k}\right) ^ {2} \geq \sum_ {k = 1} ^ {d} H _ {k} ^ {*} \left(u _ {k} ^ {(i)} - g _ {k}\right) ^ {2} \tag {55}
$$
where estimations of Hessians and  $\delta$  are accurate, namely we can estimate the optimal parameter updates of clean clients  $\theta^{*}$  accurately, thus we replace the  $k$ -th dimension  $\theta^{*}$  or  $\theta^{\mathrm{Avg}}$  with  $g_{k}$ .
Since the loss function is locally  $\mu$ -strongly convex and locally  $L$ -smooth, we have  $\mu \leq H_k^* = \mathbb{D}[u_k] \leq L$ , since the Hessian matrix is diagonal according to the assumption and  $H_k^*$  is the eigenvalue of the Hessian matrix that is in  $[\mu, L]$ , we have,
$$
L \sum_ {k = 1} ^ {d} \left(u _ {k} ^ {(j)} - g _ {k}\right) ^ {2} \geq \sum_ {k = 1} ^ {d} \mathbb {D} \left[ u _ {k} \right] \left(u _ {k} ^ {(j)} - g _ {k}\right) ^ {2} \geq \sum_ {k = 1} ^ {d} \mathbb {D} \left[ u _ {k} \right] \left(u _ {k} ^ {(i)} - g _ {k}\right) ^ {2} \geq \mu \sum_ {k = 1} ^ {d} \left(u _ {k} ^ {(i)} - g _ {k}\right) ^ {2}. \tag {56}
$$
Note that client  $i$  is malicious, the expectation of the right-hand side is,
$$
\begin{array}{l} \mu \sum_ {k = 1} ^ {d} \mathbb {E} \left[ \left(u _ {k} ^ {(i)} - g _ {k}\right) ^ {2} \right] = \mu \sum_ {k = 1} ^ {d} \mathbb {E} \left[ \left(\left(u _ {k} ^ {(i)} - g _ {k} ^ {*}\right) + \left(g _ {k} ^ {*} - g _ {k}\right)\right) ^ {2} \right] (57) \\ = \mu \sum_ {k = 1} ^ {d} \mathbb {E} \left[ \left(u _ {k} ^ {(i)} - g _ {k} ^ {*}\right) ^ {2} + 2 \left(u _ {k} ^ {(i)} - g _ {k} ^ {*}\right) \left(g _ {k} ^ {*} - g _ {k}\right) + \left(g _ {k} ^ {*} - g _ {k}\right) ^ {2} \right] (58) \\ = \mu \sum_ {k = 1} ^ {d} \mathbb {E} \left[ \left(u _ {k} ^ {(i)} - g _ {k} ^ {*}\right) ^ {2} \right] + \mu \sum_ {k = 1} ^ {d} \left(g _ {k} ^ {*} - g _ {k}\right) ^ {2} (59) \\ \geq \mu \sum_ {k = 1} ^ {d} \left(g _ {k} ^ {*} - g _ {k}\right) ^ {2} = \mu \| \mathbf {g} ^ {*} - \mathbf {g} \| _ {2} ^ {2}, (60) \\ \end{array}
$$
and note that client  $j$  is clean, the expectation of the left-hand side is,
$$
L \sum_ {k = 1} ^ {d} \mathbb {E} \left[ \left(u _ {k} ^ {(j)} - g _ {k}\right) ^ {2} \right] = L \sum_ {k = 1} ^ {d} \mathbb {D} [ u _ {k} ] \leq \eta L \sum_ {k = 1} ^ {d} g _ {k} ^ {2} = \eta L \| \mathbf {g} \| _ {2} ^ {2}, \tag {61}
$$
combining them, we have,
$$
\eta L \| \mathbf {g} \| _ {2} ^ {2} \geq \mu \| \mathbf {g} ^ {*} - \mathbf {g} \| _ {2} ^ {2}, \tag {62}
$$
$$
\begin{array}{l} \left\| \mathbb {E} \left[ \mathcal {A} \left(\left\{\mathbf {u} ^ {(i)} \right\} _ {i = 1} ^ {n}\right) \right] - \mathbf {g} \right\| _ {2} = \left\| \frac {\left(| S | - | M _ {1} |\right) \mathbf {g} + | M _ {1} | \mathbf {g} ^ {*}}{| S |} - \mathbf {g} \right\| _ {2} = \frac {| M _ {1} |}{| S |} \| \mathbf {g} ^ {*} - \mathbf {g} \| _ {2} (63) \\ \leq \frac {m}{\left\lceil \frac {n + 1}{2} \right\rceil} \| \mathbf {g} ^ {*} - \mathbf {g} \| _ {2} \leq \frac {m}{\left\lceil \frac {n + 1}{2} \right\rceil} \sqrt {\frac {\eta L}{\mu}} \| \mathbf {g} \| _ {2} (64) \\ \end{array}
$$
When  $\eta$  is small enough and  $\eta < \frac{\lceil\frac{n + 1}{2}\rceil^2\mu}{m^2L}$ , we have,
$$
0 <   \frac {m}{\lceil \frac {n + 1}{2} \rceil} \sqrt {\frac {\eta L}{\mu}} <   1, \tag {65}
$$
therefore there exists:
$$
\alpha = \arcsin \left(\frac {m}{\lceil \frac {n + 1}{2} \rceil} \sqrt {\frac {\eta L}{\mu}}\right) \in (0, \frac {\pi}{2}), \tag {66}
$$
such that Fed-FA aggregation algorithm is  $(\alpha, m)$ -Byzantine resilience:
$$
\left\| \mathbb {E} \left[ \mathcal {A} \left(\left\{\mathbf {u} ^ {(i)} \right\} _ {i = 1} ^ {n}\right) \right] - \mathbf {g} \| _ {2} \leq \sin \alpha \| \mathbf {g} \| _ {2}, \right. \tag {67}
$$
and we can also derive:
$$
\begin{array}{l} \mathbf {g} ^ {\mathrm {T}} \mathbb {E} [ \mathcal {A} (\{\mathbf {u} ^ {(i)} \} _ {i = 1} ^ {n}) ] = \mathbf {g} ^ {\mathrm {T}} \left[ (\mathbb {E} [ \mathcal {A} (\{\mathbf {u} ^ {(i)} \} _ {i = 1} ^ {n}) ] - \mathbf {g}) + \mathbf {g} \right] (68) \\ = \| \mathbf {g} \| ^ {2} + \mathbf {g} ^ {\mathrm {T}} (\mathbb {E} [ \mathcal {A} (\{\mathbf {u} ^ {(i)} \} _ {i = 1} ^ {n}) ] - \mathbf {g}) (69) \\ \geq \left\| \mathbf {g} \right\| ^ {2} - \left\| \mathbf {g} \right\| \left\| \mathbb {E} \left[ \mathcal {A} \left(\left\{\mathbf {u} ^ {(i)} \right\} _ {i = 1} ^ {n}\right) \right] - \mathbf {g} \right\| \geq (1 - \sin \alpha) \| \mathbf {g} \| _ {2} ^ {2}. (70) \\ \end{array}
$$
[1] also proves that the higher-order moments of aggregations are bounded by a linear combination of terms of clean update moments. The proof utilizes the same technique as above.
The bound of  $\sin \alpha$  provided in the proof is:  $\sin \alpha = \left(\frac{m}{\lceil\frac{n + 1}{2}\rceil}\sqrt{\frac{\eta L}{\mu}}\right)$ , a lower  $\sin \alpha$  indicates better Byzantine resilience. It can be concluded from the bound and the assumptions that: (1) higher  $m$  ( $1 \leq m \leq \lfloor \frac{n - 1}{2} \rfloor$ ) leads to worse Byzantine resilience, and when  $m > \lfloor \frac{n - 1}{2} \rfloor$ , the assumption of the theorem does not hold and the Byzantine resilience of Fed-FA is not guaranteed; (2) a poorly conditioned loss function or Hessian matrix (namely the condition number  $L / \mu$  of the Hessian matrix is high) leads to poor Byzantine resilience; (3) a higher gradient noise scale  $\eta$  leads to poor Byzantine resilience; (4) the Byzantine resilience of Fed-FA relies on that the estimations of indicators are accurate; we propose dataset synthesis and embedding Hessian reassignment mechanisms for more accurate Hessian estimations; therefore, a future improvement direction of Fed-FA may be avoiding the interference of attackers on our indicator estimation.
B Detailed experiment setups
In this section, we introduce the detailed experimental setup. In the training process of the local clients, all aggregation methods adopt the same hyper-parameters for fair comparisons. Experiments are conducted on NVIDIA TITAN RTX GPUs. One experiment costs about 30 minutes for one run (including 10 rounds).
B.1 Tasks and datasets
We adopt four text classification tasks, i.e., SST-2 [39], IMDB [24], Amazon [2], and AgNews [52]. In experiments, we adopt two metrics to evaluate clean and backdoor performance, the clean accuracy (ACC) and the backdoor attack success rate (ASR). ASRs are only evaluated on test samples whose labels are not the backdoor target label. ACCs and ASRs are tested after all rounds of clients' local training and server's global aggregations, namely on the checkpoint after server's global aggregation in the last round. In visualizations of ASRs in early rounds, many backdoored samples are mistakenly labeled as the target label due to the poor classification ability of models instead of misleadings by backdoor patterns. Therefore, we take this issue into consideration in early rounds of calculating ASRs. Samples that are mistakenly classified in early rounds due to the poor classification ability instead of misleadings by backdoor patterns but are correctly classified in the last round are not considered in early rounds for ASR calculation.
SST-2 denotes the Stanford Sentiment Treebank dataset [39]. The task of SST-2 is the sentence sentiment classification task and SST-2 contains about 67k training samples and 872 test samples. IMDB denotes the