SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| \mathbb {E} \left[ \mathcal {A} \left(\left\{\mathbf {u} ^ {(i)} \right\} _ {i = 1} ^ {n}\right) \right] - \mathbf {g} \right\| _ {2} \leq \sin \alpha \| \mathbf {g} \| _ {2}, \quad \mathbf {g} ^ {T} \mathbb {E} \left[ \mathcal {A} \left(\left\{\mathbf {u} ^ {(i)} \right\} _ {i = 1} ^ {n}\right) \right] \geq (1 - \sin \alpha) \| \mathbf {g} \| _ {2} ^ {2}, \tag {6}
$$
where  $\mathbf{g} = \mathbb{E}[\mathbf{u}]$  is the expected update for clean clients  $\mathbf{u}$ .
Theorem 3 states the Byzantine resilience of Fed-FA, namely the variations of aggregated updates and ideal clean updates are bounded  $(\| \mathbb{E}[\mathcal{A}(\{\mathbf{u}^{(i)}\}_{i = 1}^n)] - \mathbf{g}\| _2\leq \sin \alpha \| \mathbf{g}\| _2)$ , which indicates that the attacker cannot divert aggregated updates too far from ideal updates. Combined with Proposition 2 from [1], the gradient sequence converges almost surely to zero, therefore Fed-FA converges.
In this section, we introduce experiment setups and main results. Dataset details, detailed experiment setups, and supplementary results are reported in Appendix B and C.
4.1 Experiment setups
Datasets. We adopt four typical text classification tasks, i.e., SST-2 (Stanford Sentiment Treebank) [39], IMDB (IMDB movie reviews) [24], Amazon (Amazon reviews) [2], and AgNews [52]. Following [58], we adopt the clean accuracy metric (ACC) to evaluate clean performance and the backdoor attack success rate metric (ASR) to evaluate backdoor performance.
Models and training. We adopt three typical neural network architectures in NLP tasks, i.e., GRU, LSTM, and CNN. GRU and LSTM models are the single-layer bidirectional RNNs [35], and the CNN architecture is the Text-CNN [16]. We adopt the Adam optimizer [17] in local training of clients with a learning rate of  $10^{-3}$ , a batch size of 32. We train models for 10 rounds. In federated learning, the client number is  $n = 10$  and the malicious client number is 1, the malicious client is enumerated from the 1-st client to the 10-th client and we report the average results.
Backdoor attacks. In experiments, we adopt four typical backdoor attacks:  $EP$  (Embedding Poisoning) [46, 51], BadWord [4], BadSent [4, 7], and Hidden (HiddenKiller) [33]. EP and BadWord choose five low-frequency candidate trigger words, i.e., "cf", "mn", "bb", "tq" and "mb". BadSent adopts "I watched this 3d movie" as the trigger sentence. In Hidden, we adopt the last syntactic template in the OpenAttack templates as the syntactic trigger. The target label is label 0.
Table 1: Average results of Fed-FA compared to others (lower ASR is better, lowest ASRs in bold).
Model (ACC)	Metric FedAvg	Non-discarding Aggregations	Discarding Aggregations (includes Fed-FA)
Median FoolsGold	RFA	CRFL	Residual	Krum M-Krum	Bulyan Dim-Krum	Fed-FA
GRU (86.85)	ACC	86.05	86.04	85.92	85.96	71.25	86.05	76.32	85.09	86.05	84.53	86.36
ASR	86.02	59.56	85.99	86.26	75.96	65.54	74.22	54.24	48.90	33.16	13.66
LSTM (84.42)	ACC	83.49	83.60	73.74	83.75	70.26	83.69	75.68	83.29	83.60	82.91	84.39
ASR	90.51	67.09	90.16	90.68	84.82	70.12	75.52	60.09	61.29	33.08	22.11
CNN (87.11)	ACC	86.32	85.98	86.29	86.33	77.37	86.28	78.14	85.60	86.22	85.38	86.36
ASR	83.47	57.19	83.53	83.58	37.92	62.77	75.02	65.80	50.74	28.46	22.77
Average (86.13)	ACC	85.28	85.61	85.32	85.35	72.96	85.34	76.71	84.66	85.29	84.27	85.70
ASR	86.67	61.28	86.56	86.84	65.98	66.14	74.92	60.04	53.63	31.56	19.51
(a) Average ASRs on GRU.
Figure 2: ASRs in 10 rounds. Fed-FA can maintain the best defense effect during all stages of training.
(b) Average ASRs on LSTM.
(c) Average ASRs on CNN.
(d) Legend.
Aggregation baselines. Robust federated aggregations can be divided into two categories, i.e., non-discarding aggregations and discarding aggregations. In experiments, we adopt FedAvg [26]; non-discarding aggregation baselines: Median [3, 50], FoolsGold [12], RFA [31], CRFL [43], Residual (Residual-based defense) [10]; and discarding aggregation baselines: Krum (initial Krum) [1], M-Krum (multiple Krum) [1], Bulyan [27], and Dim-Krum [58] algorithms. In Dim-Krum, we choose the ratio as  $\rho = 10^{-3}$  and the adaptive noise scale  $\lambda = 2$ .
Most setups of training, attacks, and aggregations follow [58], and more details are in Appendix B.
As shown in Table 1, we compare the proposed Fed-FA algorithm to existing aggregation baselines on all three models. Results are averaged on different attacks and datasets. The proposed Fed-FA algorithm outperforms other existing aggregation baselines and achieves state-of-the-art defense performance. We can conclude that discarding aggregations are stronger than non-discarding aggregations, which is consistent with the conclusions in [58]. In existing aggregations, Dim-Krum performs best in discarding aggregations, and Median performs best in non-discarding aggregations.
Fig. 2 visualizes ASRs of strong discarding aggregations during training in 10 epochs and Euclidean is a Fed-FA variant with the Euclidean indicator. Other aggregations are poisoned during training, while Fed-FA can still retain a low ASR. Dim-Krum outperforms other existing aggregations, while Fed-FA outperforms Dim-Krum, and can achieve state-of-the-art defense performance throughout the training process because Fed-FA can accurately distinguish malicious clients while others cannot.
Results of different datasets and attacks. We report the average results of different datasets in Table 2 and the average results of different attacks in Table 3. Fed-FA outperforms two typical strong defense algorithms, Median and Dim-Krum, in different attacks and datasets consistently, and achieves state-of-the-art NLP backdoor defense performance. Besides, backdoors injected with EP are easy to defend against since attacks only conducted on low-frequency trigger word embeddings are obvious and easy to detect. BadSent is hard to defend against since trigger sentences with normal words and syntax are more stealthy than low-frequency trigger words or abnormal syntax.
Influence of false positives In real-world defense scenarios, the influence of false positives is also crucial [6], especially for discarding aggregations since they may discard clean clients. We validate that the false positives in detection have weak impacts on the clean performance of Fed-FA and the
Table 2: Average results of different datasets. Fed-FA outperforms others consistently.
Dataset	Metric	FedAvg	Median	Dim-Krum	Fed-FA
SST-2	ACC	79.68	79.54	79.96	81.60
ASR	91.28	68.66	43.66	31.94
IMDB	ACC	79.72	79.79	77.62	79.82
ASR	88.45	63.85	54.53	27.93
Amazon	ACC	90.38	90.29	89.14	90.27
ASR	85.85	59.74	24.18	14.75
AgNews	ACC	91.36	91.21	90.44	91.12
ASR	81.09	52.86	4.00	3.43
Table 3: Average results of different attacks. EP is easy to defend against and BadSent is hard.
Attack	Metric	FedAvg	Median	Dim-Krum	Fed-FA
EP	ACC	86.18	85.89	84.47	85.94
ASR	97.64	12.70	14.09	12.13
BadWord	ACC	86.12	85.77	84.56	85.95
ASR	91.40	81.08	34.03	15.40
BadSent	ACC	86.21	85.85	84.51	85.97
ASR	99.17	98.32	48.42	28.30
Hidden	ACC	82.62	83.27	83.54	84.95
ASR	58.45	53.01	29.71	22.21
Table 4: Results of ablation study on Fed-FA variants. Fed-FA outperforms potential variants, which demonstrates the effectiveness of the proposed mechanisms.
Method	ACC	ASR
FedAvg	85.28	86.67
Median	85.61	61.28
Residual	85.34	66.14
Krum	76.71	74.92
M-Krum	84.66	60.04
Bulyan	85.29	53.63
Dim-Krum	84.27	31.56
Fed-FA	85.70	19.51
Method	ACC	ASR
FedAvg	85.28	86.67
Fed-FA with Euclidean indicator	84.78	58.50
Fed-FA with labeled dataset	85.77	20.06
Fed-FA without Hessian reassignment	85.79	39.78
Fed-FA with inverse reassignment	84.44	82.27
Fed-FA with layer-wise reassignment	85.41	52.75
Fed-FA with reassignment within entire model	85.34	42.03
Fed-FA	85.70	19.51
false positive rates of Fed-FA variant designed for malicious client detection are lower than variants of other discarding aggregations. Due to space limit, further analyses are deferred to Appendix D.
In this section, we first report the ablation study results. Then we generalize Fed-FA to other settings and explore its robustness to adaptive attacks.
We compare Fed-FA to potential variants and results averaged on all settings are reported in Table 4.
F-divergence indicator can estimate data divergence more accurately. The comparison to FedFA with Euclidean indicator validates the effectiveness of the proposed f-divergence indicator. We also conduct analytic trials to evaluate the correlations of  $\sqrt{\text{Indicator}}$  and  $\sqrt{D_f(p||q)}$ , here data divergences are controlled with the dataset mixing ratio following [55] that  $\sqrt{\text{Indicator}} \propto \sqrt{D_f(p||q)}$  should hold. Fig. 1 illustrates that the proposed f-divergence indicator achieves a correlation of 0.9847, higher than 0.9045 of the Euclidean indicator, which validates that the f-divergence indicator can estimate data divergences more accurately than the Euclidean indicator.
Dataset synthesis can roughly estimate relatively accurate Hessian scales. Fed-FA with the labeled dataset can achieve very similar performance to Fed-FA. Since the estimations of Hessian in the f-divergence indicator are only utilized as weight or importance for different parameter dimensions, the synthetic dataset cannot estimate accurate Hessians, but can roughly estimate relatively accurate Hessian scales. Therefore, the dataset synthesis mechanism does not require labeled corpus, nor does it cause performance loss, which demonstrates its effectiveness. Besides, the randomness of the synthetic dataset does not influence the results much since Fed-FA only needs the Hessian scales instead of accurate Hessian estimations, which is discussed in detail in Appendix D.
Effectiveness of embedding Hessian reassignment mechanism. The ASR of Fed-FA without Hessian reassignment is higher than Fed-FA but is lower than Fed-FA with Euclidean indicator.
Table 5: Results of defenses on BERT on SST-2. Fed-FA still outperforms others.
Attack	Method	ACC	ASR
BadWord	FedAvg	89.41	96.73
Median	89.39	79.32
M-Krum	88.95	12.93
Dim-Krum	89.56	48.36
Fed-FA	89.41	9.11
BadSent	FedAvg	89.45	95.95
Median	89.45	92.29
M-Krum	89.26	37.38
Dim-Krum	89.45	46.03
Fed-FA	89.03	27.10
Table 6: Results of MNIST backdoor defense task. FedFA can still work in CV. ASRs (< 11) are in bold.
Method	Metric	2 Attackers	3 Attackers	4 Attackers
FedAvg	ACC	96.33	96.23	96.01
ASR	39.20	94.50	98.44
CRFL	ACC	96.64	96.22	96.23
ASR	10.78	12.41	87.37
Median	ACC	96.61	96.44	96.31
ASR	10.59	10.73	10.76
Krum	ACC	95.97	95.76	95.64
ASR	10.02	10.35	10.34
Fed-FA	ACC	96.08	95.82	95.93
ASR	10.20	10.20	10.31
It means that embedding Hessian reassignment can help estimate Hessians more accurately. We also explore Fed-FA with inverse reassignment, which replaces the reassignment principle  $\sqrt{\hat{H}_k^*} \propto \sum_{i=1}^{n} |u_k^{(i)}| / n$  with  $\sqrt{\hat{H}_k^*} \propto \left\{\sum_{i=1}^{n} |u_k^{(i)}| / n\right\}^{-1}$ , and causes very poor performance. We also implement Fed-FA with layer-wise reassignment that reassigns Hessians in every layer respectively, and Fed-FA with reassignment within the entire model that reassigns Hessians on parameters on the entire model parameters instead of the embedding parameters  $E$ . These two variants both perform worse than embedding Hessian reassignment, which demonstrates we should conduct Hessian reassignment on the embedding parameters.
Why does conducting Hessian reassignment on embeddings work best? The premise of Theorem 2 requires that Hessians are small. The Hessian scales on embeddings are usually smaller than other layers: analytic trials show that average Hessian scales are about  $10^{-6}$  on embeddings,  $10^{-4}$  on other layers; thus correlations of  $\sum_{i=1}^{n} |u_k^{(i)}| / n$  and  $\sqrt{H_k^*}$  on embeddings are 0.47, which is much higher than correlations on other layers, 0.02. Thus reassignment should only be conducted on embeddings.
5.2 Results on other settings
Results on pre-trained language models. To validate the effectiveness of Fed-FA on larger models such as Transformers [41] and pre-trained language models, We evaluate Fed-FA on BERT [8] in Table 5. Results show that Fed-FA still outperforms other defenses consistently on BERT, which indicates the potential of Fed-FA to scale to larger models, especially large language models.
Results on federated vision backdoor defense. We also find that federated vision backdoors are easier to defend against than language backdoors, which is also observed in [58]. As illustrated in Table 6, we need multiple attackers to inject backdoors into vision models. Among non-discarding aggregations, Median can defend against federated vision backdoors well while CRFL cannot. Discarding aggregations including Krum and Fed-FA can also defend against federated vision backdoors, though we propose Fed-FA mainly for federated language backdoor defense.
Results on non-IID and multiple attacker cases. We generalize Fed-FA to non-IID and multiple attacker settings in Fig. 3. Here we choose the Dirichlet distribution with the concentration parameter  $\alpha = 0.9$  as the non-IID distribution. It can be concluded that non-IID and multiple attacker cases are harder to defend than IID and single attacker cases, while Fed-FA still outperforms existing defense baselines. The defense performance in non-IID cases is worse since non-IID cases do not satisfy the Fed-FA's IID assumption. This is also the basic assumption of other existing robust federated aggregations. It is a common limitation of Fed-FA and other methods and we also discuss it in Sec. 6.
5.3 Robustness to adaptive attacks
Since the proposed f-divergence indicator is calculated according to parameter update variances, potential adaptive attacks can be conducted with adversarial parameter corruptions [40, 54] or perturbations [14]. We adopt an  $L_{2}$ -penalty regularizer on parameters to make parameters close to
(a) Non-IID and multi-atker cases.
(b) Weight decay attack cases.
Figure 3: Results under non-IID and multiple attacker cases and under adaptive attacks. Fed-FA still outperforms existing baselines under non-IID and multi-atker cases and is robust to adaptive attacks.
(c) Adaptive decay attack cases.
$\pmb{\theta}_{t-1}^{\text{server}}$  in the  $t$ -th round:  $R = \lambda \cdot \sum_{i=1}^{d} (\theta_i^k - \theta_i^{\text{server}})^2$ , where  $\lambda$  denotes the decay coefficient and  $\theta_i^{\text{server}}$  denotes the  $i$ -th dimension of  $\pmb{\theta}_{t-1}^{\text{server}}$ . We also design another adaptive decay regularizer to target to attack Fed-FA:  $R = \lambda \cdot \sum_{i=1}^{d} H_i' (\theta_i^k - \theta_i^{\text{server}})^2$ , where  $H_i'$  is Hessians estimated by attackers.
As shown in Fig. 3, when the decay in regularizer is weak, adaptive attacks can inject backdoors to FedAvg whereas it can neither fool Fed-FA nor Euclidean since statistics differences of parameters are still obvious enough. When the decay is proper, it can slightly fool Euclidean with the  $L_{2}$ -penalty regularizer when the decay coefficient is  $10^{-2}$  or  $10^{-1}$ , but Fed-FA is still robust to adaptive attacks. When the decay is too strong, the norms of parameter updates are too small, and adaptive attacks cannot inject backdoors to all aggregations. To conclude, Fed-FA is robust to adaptive attacks. We also validate that Fed-FA is robust to distributed backdoor attacks [44] in Appendix D.
6 Broader impact and limitations
Broader impact. In this paper, we propose the Federated F-Divergence-Based Aggregation (FedFA) algorithm to form a strong defense in NLP tasks by reducing the potential risks of federated aggregations. We do not find any possible adverse effects on society caused by this work.
Limitations. Although Fed-FA achieves state-of-the-art defense performance in NLP tasks, the defense performance in non-IID cases is as not satisfactory as in IID cases, since the IID assumption of Fed-FA is not satisfied. This is a common limitation of Fed-FA and other existing methods. A future direction is to consider the semantics of the parameter updates themselves in addition to the data divergence for federated backdoor defense.
Besides, both our proposed Fed-FA and classic federated defending algorithms [1, 43, 10, 31, 58] are mainly evaluated on small-scale MLP, CNN or RNN models, but not evaluated on popular large language models due to computation cost limit. However, our proposed Fed-FA is model agnostic and just filters harmful gradients involved in aggregation, thus it can be extended to large-scale models such as Transformers and large language models. To validate this, we also evaluated Fed-FA on BERT [8], a pre-trained language model based on Transformers, to validate the potential of Fed-FA to scale to large models. A future direction is to evaluate and improve federated language backdoor defense algorithms on large language models.
In this paper, we model data divergence among clients' data theoretically for backdoor client detection in federated language learning. Based on it, we propose a novel and effective Federated F-Divergence-Based Aggregation (Fed-FA) algorithm as a strong defense for federated language learning. Fed-FA utilizes the f-divergence indicator to detect and discard suspicious clients. Both theoretical evidence and experimental results demonstrate that Fed-FA can better detect suspicious clients than existing robust federated aggregations that mainly adopt parameter distances explicitly. Thus, Fed-FA outperforms existing methods and achieves state-of-the-art federated language backdoor defense performance. Further analyses validate the effectiveness of proposed mechanisms, as well as show that Fed-FA can be generalized to other settings and is robust to potential adaptive attacks.
We appreciate all the thoughtful and insightful suggestions from the anonymous reviews. This work was supported in part by a Tencent Research Grant and National Natural Science Foundation of China (No. 62176002). Xu Sun is the corresponding author of this paper.
A Theoretical details
A.1 Details about f-divergence
The f-divergence [34] can measure the data divergence of  $p(\mathbf{z})$  and  $q(\mathbf{z})$ :
$$
D _ {f} \left(q (\mathbf {z}) | | p (\mathbf {z})\right) = \int_ {\mathbf {z}} p (\mathbf {z}) f \left(\frac {q (\mathbf {z})}{p (\mathbf {z})}\right) d \mathbf {z}, \tag {7}
$$
where the integral symbol  $\int_{\mathbf{z}}$  denotes multiple integrals in every dimension of  $\mathbf{z}$ , and integral symbols of discrete dimensions need to be changed to summation symbols for these dimensions.
The function  $f(x)$  is smooth, convex and satisfies  $f(1) = 0$ . Suppose  $x = 1 + h$ . We require the function  $f(x)$  is Second-order differentiable and its Second-order Taylor expansion near  $x = 1$  is:
$$
f (x) = f ^ {\prime} (1) h + \frac {f ^ {\prime \prime} (1)}{2} h ^ {2} + o \left(h ^ {2}\right), \tag {8}
$$
since  $f(x)$  is convex, we have  $f''(1) > 0$ .
Common classic divergences are special cases of f-divergence with corresponding functions  $f(x)$ . Here are some examples of special cases:
Kullback-Leibler divergence. A special case of f-divergence is the Kullback-Leibler divergence [18] when we choose  $f(x) = x\log x$ :  $D_{f}(q||p) = D_{\mathrm{KL}}(q||p) = \int_{\mathbf{z}}q(\mathbf{z})\log \left(q(\mathbf{z}) / p(\mathbf{z})\right)d\mathbf{z}$ , and the Second-order Taylor expansion is:
$$
f (x) = f (1 + h) = h + \frac {h ^ {2}}{2} + o \left(h ^ {2}\right), \tag {9}
$$
where  $f''(1) = 1 > 0$ .
Reverse Kullback-Leibler divergence. A special case of f-divergence is the reverse Kullback-Leibler divergence [18] when we choose  $f(x) = -\log x$ :  $D_{f}(q||p) = D_{\mathrm{reverse - KL}}(q||p) = \int_{\mathbf{z}}p(\mathbf{z})\log \left(p(\mathbf{z}) / q(\mathbf{z})\right)d\mathbf{z}$ , and the Second-order Taylor expansion is:
$$
f (x) = f (1 + h) = - h + \frac {h ^ {2}}{2} + o \left(h ^ {2}\right), \tag {10}
$$
Jensen-Shannon divergence. A special case of f-divergence is the Jensen-Shannon divergence [11] when we choose  $f(x) = x\log x - (x + 1)\log ((x + 1) / 2)$ :  $D_{f}(q||p) = D_{\mathrm{JS}}(q||p) = D_{\mathrm{KL}}(p||m) / 2 + D_{\mathrm{KL}}(q||m) / 2$  where the medium distribution is  $m = (p + q) / 2$ , and the Second-order Taylor expansion is:
$$
f (x) = f (1 + h) = \frac {h ^ {2}}{4} + o \left(h ^ {2}\right), \tag {11}
$$
where  $f''(1) = 1 / 2 > 0$ .
A.2 Detailed assumptions of Theorem 1
Theorem 1 requires two classic assumptions, the Second-order Taylor expansion assumption and the Fisher information matrix assumption [30]:
Assumption A.1 (Second-order Taylor Expansion). Assume the loss  $\mathcal{L}(\pmb{\theta};\mathbf{z})$  can be Second-order Taylor expanded near the parameter  $\pmb{\theta}^*$  and the Hessian matrix  $\pmb{H}(\pmb{\theta};\mathbf{z}) = \nabla_{\pmb{\theta}}^{2}\mathcal{L}(\pmb{\theta};\mathbf{z})$  is diagonal:
$$
\mathcal {L} \left(\boldsymbol {\theta} ^ {*} + \boldsymbol {\delta}; \mathbf {z}\right) = \mathcal {L} \left(\boldsymbol {\theta} ^ {*}; \mathbf {z}\right) + \boldsymbol {\delta} ^ {\mathrm {T}} \nabla_ {\boldsymbol {\theta}} \mathcal {L} \left(\boldsymbol {\theta} ^ {*}; \mathbf {z}\right) + \frac {1}{2} \boldsymbol {\delta} ^ {\mathrm {T}} \boldsymbol {H} \left(\boldsymbol {\theta} ^ {*}; \mathbf {z}\right) \boldsymbol {\delta} + o \left(\| \boldsymbol {\delta} \| _ {2} ^ {2}\right), \tag {12}
$$
and  $\pmb {H}(\pmb {\theta};\mathcal{P}) = diag\{H_1(\pmb {\theta};\mathcal{P}),\dots ,H_d(\pmb {\theta};\mathcal{P})\}$
Assumption A.2 (Fisher Information). Assume the Fisher information assumption [30] holds:
$$
\boldsymbol {H} (\boldsymbol {\theta}; \mathcal {P}) = \mathbb {E} _ {\mathcal {P}} \left[ \nabla_ {\boldsymbol {\theta}} \mathcal {L} (\boldsymbol {\theta}; \mathbf {z}) \nabla_ {\boldsymbol {\theta}} \mathcal {L} (\boldsymbol {\theta}; \mathbf {z}) ^ {\mathrm {T}} \right]. \tag {13}
$$
A.3 Proofs of Theorem 1
We try to find the infimum or greatest lower bound of f-divergence:
$$
\inf  _ {p (\mathbf {z}), q (\mathbf {z})} \quad D _ {f} (q (\mathbf {z}) \| p (\mathbf {z})), \tag {14}
$$
$$
\text {s u b j e c t} \quad \boldsymbol {\theta} ^ {*} = \underset {\boldsymbol {\theta}} {\arg \min } \mathcal {L} (\boldsymbol {\theta}; p (\mathbf {z})), \quad \boldsymbol {\theta} ^ {*} + \boldsymbol {\delta} = \underset {\boldsymbol {\theta}} {\arg \min } \mathcal {L} (\boldsymbol {\theta}; q (\mathbf {z})). \tag {15}
$$
$$
D _ {f} \left(q (\mathbf {z}) \| p (\mathbf {z})\right) \geq (1 + o (1)) \frac {f ^ {\prime \prime} (1)}{2} \mathcal {I} _ {F - D i v}, \quad \mathcal {I} _ {F - D i v} = \sum_ {k = 1} ^ {d} H _ {k} ^ {*} \delta_ {k} ^ {2}, \tag {16}
$$
where  $H_{k}^{*} = H_{k}\left(\boldsymbol{\theta}^{*};p(\mathbf{z})\right) = \mathcal{L}_{\theta_{k}}^{\prime \prime}\left(\boldsymbol{\theta}^{*};p(\mathbf{z})\right) > 0$  is the  $i$ -th Hessian of loss on  $p(\mathbf{z})$  and  $f''(1) > 0$ .
Proof. First, we introduce a lemma analyzing the relationship of the distribution shift and the parameter shift [55]:
Lemma A.1. Define  $r(\mathbf{z}) = q(\mathbf{z}) / p(\mathbf{z}) - 1$ . When the distribution shift is small enough, namely  $r(\mathbf{z})$  is small, we can estimate the parameter shift  $\delta$  as,
$$
\boldsymbol {\delta} = - \boldsymbol {H} ^ {- 1} \left(\boldsymbol {\theta} ^ {*}; p (\mathbf {z})\right) \mathbb {E} _ {p} \left[ r (\mathbf {z}) \nabla_ {\boldsymbol {\theta}} \mathcal {L} \left(\boldsymbol {\theta} ^ {*}; \mathbf {z}\right) \right] + o (\| \boldsymbol {\delta} \|). \tag {17}
$$
In Lemma A.1,  $r(\mathbf{z})$  is defined as  $r(\mathbf{z}) = q(\mathbf{z}) / p(\mathbf{z}) - 1$ , and the parameter shift  $\delta$  is defined as  $\arg \min_{\theta} \mathcal{L}\big(\pmb{\theta}; q(\mathbf{z})\big) - \arg \min_{\theta} \mathcal{L}\big(\pmb{\theta}; p(\mathbf{z})\big)$ , which is consistent with our definition since  $\arg \min_{\theta} \mathcal{L}\big(\pmb{\theta}; q(\mathbf{z})\big) - \arg \min_{\theta} \mathcal{L}\big(\pmb{\theta}; p(\mathbf{z})\big) = (\pmb{\theta}^{*} + \delta) - (\pmb{\theta}^{*}) = \delta$ .
With the change-of-measure technique, we have  $\mathbb{E}_p\big[r(\mathbf{z})\big] = 0$ . Define  $\mathbb{E}_p[r(\mathbf{z})\nabla_\theta \mathcal{L}(\boldsymbol{\theta}^*;\mathbf{z})] = \mathbf{v}$ . Then according to Lemma A.1: