Title: Fed-FA: Theoretically Modeling Client Data Divergence for Federated Language Backdoor Defense
Venue: NeurIPS 2023
Accepted: yes
Keywords: federated learning

backdoor learning

robust federated aggregation

data divergence
Abstract: Federated learning algorithms enable neural network models to be trained across multiple decentralized edge devices without sharing private data. However, they are susceptible to backdoor attacks launched by malicious clients. Existing robust federated aggregation algorithms heuristically detect and exclude suspicious clients based on their parameter distances, but they are ineffective on Natural Language Processing (NLP) tasks. The main reason is that, although text backdoor patterns are obvious at the underlying dataset level, they are usually hidden at the parameter level, since injecting backdoors into texts with discrete feature space has less impact on the statistics of the model parameters. To settle this issue, we propose to identify backdoor clients by explicitly modeling the data divergence among clients in federated NLP systems. Through theoretical analysis, we derive the f-divergence indicator to estimate the client data divergence with aggregation updates and Hessians. Furthermore, we devise a dataset synthesization method with a Hessian reassignment mechanism guided by the diffusion theory to address the key challenge of inaccessible datasets in calculating clients' data Hessians.
We then present the novel Federated F-Divergence-Based Aggregation~(\textbf{Fed-FA}) algorithm, which leverages the f-divergence indicator to detect and discard suspicious clients. Extensive empirical results show that Fed-FA outperforms all the parameter distance-based methods in defending against backdoor attacks among various natural language backdoor attack scenarios.

Fed-FA: Theoretically Modeling Client Data Divergence for Federated Language Backdoor Defense

Abstract

Federated learning algorithms enable neural network models to be trained across multiple decentralized edge devices without sharing private data. However, they are susceptible to backdoor attacks launched by malicious clients. Existing robust federated aggregation algorithms heuristically detect and exclude suspicious clients based on their parameter distances, but they are ineffective on Natural Language Processing (NLP) tasks. The main reason is that, although text backdoor patterns are obvious at the underlying dataset level, they are usually hidden at the parameter level, since injecting backdoors into texts with discrete feature space has less impact on the statistics of the model parameters. To settle this issue, we propose to identify backdoor clients by explicitly modeling the data divergence among clients in federated NLP systems. Through theoretical analysis, we derive the f-divergence indicator to estimate the client data divergence with aggregation updates and Hessians. Furthermore, we devise a dataset synthesization method with a Hessian reassignment mechanism guided by the diffusion theory to address the key challenge of inaccessible datasets in calculating clients' data Hessians. We then present the novel Federated F-Divergence-Based Aggregation ( Fed-FA ) algorithm, which leverages the f-divergence indicator to detect and discard suspicious clients. Extensive empirical results show that Fed-FA outperforms all the parameter distance-based methods in defending against backdoor attacks among various natural language backdoor attack scenarios.

1 Introduction

Federated learning can train neural network models across multiple decentralized edge devices (i.e. clients) in a privacy-protect manner. However, federated aggregation algorithms ( e.g. FedAvg [26]) are vulnerable to backdoor attacks [15, 23] from malicious clients via poisonous parameter updating [26, 51]. This poses a serious threat to the security and reliability of federated learning systems. Therefore, detecting suspicious backdoor clients is of great research significance [3, 31, 43, 42] .

Most existing robust federated aggregation algorithms heuristically take parameter distances (e.g. Euclidean distances [1, 27, 58], cosine distances [12]) among clients as the indicator to detect suspicious clients. However, [58] points out that federated language backdoors are harder to defend against than vision backdoors; the reason lies in that the text feature space is discrete and the injected backdoor patterns of text are more hidden at the parameter level than images. For example, the output of NLP models can be falsified by only poisoning a few words' embeddings [46, 5], which can hardly affect the statistics of the whole model parameters. Thus, the parameter distance-based

The proposed

F-Divergence Indicator

is a more accurate

estimation of

𝑫𝑫

𝒇𝒇

(

𝒒𝒒

|

𝒑𝒑

than

Euclidean Indicator (

|

𝜹𝜹

|

𝟐𝟐

).

…

Client 1

Dataset 1

𝜽𝜽

(

𝟏𝟏

)

Client

k

𝒒𝒒

𝒛𝒛

:

Dataset

k

𝜽𝜽

(

𝒌𝒌

)

…

…

𝒑𝒑

𝒛𝒛

:

Average

Distribution

Server

Detect and discard suspicious

clients with high

𝐟𝐟

-divergence

indicators

, and aggregate

updates of other clean clients

(set

S

):

𝜽𝜽

𝐒𝐒𝐒𝐒𝐒𝐒𝐒𝐒𝐒𝐒𝐒𝐒

=

∑𝒊𝒊∈𝑺𝑺

𝜽𝜽

(

𝒊𝒊

)

𝑺𝑺

…

𝜽𝜽

𝐀𝐀𝐒𝐒𝐀𝐀

:

Average

Parameter

Motivation:

Poisonous clients have larger

divergence

𝑫𝑫

𝒇𝒇

(

𝒒𝒒

|

𝒑𝒑

,

𝒑𝒑

and

𝒒𝒒

are invisible. Thus, we utilize

𝜹𝜹

=

𝜽𝜽

(

𝒌𝒌

)

-𝜽𝜽 𝑨𝑨𝑨𝑨𝑨𝑨

to estimate

𝑫𝑫

𝒇𝒇

between

Client

k

and

others

.

𝑫𝑫

𝒇𝒇

(

𝒒𝒒

||

𝒑𝒑

)

Correlation = 0.9065

ASR = 58.5%

Correlation = 0.9874

ASR = 19.5%

Trails with

(

𝑫𝑫

𝒇𝒇

,

Indicator )

𝑫𝑫

𝒇𝒇

(

𝒒𝒒

||

𝒑𝒑

)

|

𝜹𝜹

|

(

Euclidean

)

𝐅𝐅

-

𝐃𝐃

ivergence Indicator

Baseline Backdoor Attack

Success Rate (ASR) = 71.6%

robust federated aggregations [1, 12, 10] do not perform well on NLP tasks. Besides, the choice of distance function is empirical and lacks theoretical guarantees [1, 58, 12, 10].

Present work. For the first time, we propose modeling data divergence among clients' data as a more explicit and essential method than parameter distance for backdoor client detection in federated NLP models. A critical challenge for data divergence estimation is that local datasets on clients are invisible to the server or the defender due to privacy protection, thus we cannot measure data divergence directly. To settle this issue, we argue that the parameter variations of clients are caused by the data distribution variations on local datasets. Based on this theoretical intuition, we model how distribution variations between different clients lead to parameter update variations in Theorem 1, and further derive the f-divergence indicator that can estimate the data divergence among different clients with parameter updates and Hessians ( i.e. second derivatives).

Driven by our theoretical analysis, we propose a novel Federated F-Divergence-Based Aggregation ( Fed-FA ) algorithm, which utilizes the f-divergence indicator to estimate the data divergence of clients. F-divergence is a universal divergence measurement of data distributions that common classic divergences can be seen as special cases with corresponding convex functions f ( x ) . We utilize the f-divergence indicator to detect suspicious clients that have larger f-divergence indicator values, namely clients whose data distributions are different from others. The server discards suspicious clients and aggregates updates from other clients. The Fed-FA is illustrated in Fig. 1. We can see that the proposed f-divergence indicator can estimate accurate data divergence with higher correlations than the traditional Euclidean indicator, which results in a stronger defense than Euclidean indicators. We also prove Theorem 3 to verify the Byzantine resilience and convergence of Fed-FA.

The calculation of f-divergence indicators involves parameter updates and Hessians. Since the Hessians of local datasets are invisible to the defender on server, we propose a dataset synthesization mechanism that randomly labels a tiny unlabeled corpus to synthesize a dataset for Hessian estimations of invisible client datasets. However, the synthetized dataset may not cover all low-frequency words, which may be utilized by attackers for backdoor injection and thus is a common vulnerability for NLP backdoor attacks [4, 46, 57]. To settle this issue, we reassign Hessians on embeddings; the reassigned scales are derived in Theorem 2, which is guided by the diffusion theory [20, 45] and reveals that the parameter update magnitude is approximately proportional to Hessian square root on embeddings.

In addition to theoretical analysis, we also conduct comprehensive experiments to compare Fed-FA with other robust federated aggregation baselines on four NLP tasks that cover typical poisoning techniques in NLP, i.e. , EP [46, 51], BadWord [4], BadSent [4], and Hidden (HiddenKiller) [33]. Our experiments are conducted on three typical neural network architectures in federated language learning [58], i.e. , GRU, LSTM, and CNN. Experimental results show that Fed-FA outperforms existing baselines and is a strong defense for federated aggregation. Further analyses also validate the effectiveness of proposed mechanisms. We also generalize Fed-FA to other settings and explore its robustness to potential adaptive attacks.

2 Background and related work

In this section, we first introduce NLP backdoor attacks and backdoor defense in centralized learning. Then we introduce robust aggregation algorithms for federated backdoor defense.

2.1 Natural language backdoors and defense in centralized learning

Natural language backdoor attacks. Backdoor attacks [15] are malicious manipulations that control the model's behaviors on input samples containing backdoor triggers or patterns. NLP backdoor attacks usually adopt data poisoning [28, 5] that injects misleading input samples with backdoor patterns and wrong labels into the training dataset.

Generally, NLP backdoor attacts can be divided into three categories according to the backdoor injection patterns: (1) Trigger word based attacks [19, 46, 57, 48, 56]: BadWord [4] chooses lowfrequency trigger words as the backdoor pattern; and the Embedding Poisoning attack, EP [46, 51], only manipulates embedding parameters for better stealthiness; (2) Trigger sentence based attacks: BadSent [7, 4] chooses a neutral sentence as the backdoor pattern; (3) Hidden trigger based attacks or dynamic attacks: Hidden (HiddenKiller) [33] converts the input into a specific syntax pattern as the backdoor pattern to make the backdoor triggers difficult to detect, and other attacks also adopt hidden triggers [36, 37], input-aware or dynamic triggers [29] to hide sophisticated backdoor triggers.

In federated learning, the attacker can conduct these attack techniques on the client and poison the global parameters on the server in the federated aggregation process.

Natural language backdoor defense. Backdoor defense in centralized learning defends against the backdoor by detecting and removing the backdoor pattern in input samples [53, 9, 32, 13, 47] or mitigating backdoors in model parameters [49, 21, 59, 22]. We focus on defense algorithms for federated backdoors and introduce them in next subsection.

2.2 Robust federated aggregation

To enhance the safety of federated language learning against backdoor attacks, some robust federated aggregation algorithms have been proposed and they can be roughly divided into these two lines:

Discarding aggregations. Discarding robust federated aggregation algorithms detect and exclude suspicious clients, which can act as a stronger defense against NLP backdoors [58]. We also follow this aggregation paradigm. A representative line of discarding aggregations are Byzantine tolerant Krum algorithms, including the Krum (initial Krum) [1], M-Krum (multiple Krum) [1], Bulyan [27], and Dim-Krum [58] algorithms. They adopt Euclidean distances of parameter updates empirically, while Fed-FA adopts the f-divergence indicator derived theoretically.

Non-discarding aggregations. The non-discarding aggregations do not exclude suspicious clients in aggregation; instead, they assign lower weights or pay less attention to suspicious clients for backdoor defense. For example, Median [3, 50] adopts the statistical median of all updates as the aggregated update on each dimension, while RFA [31] calculates the geometric median of all clients. Based on RFA, CRFL [43] trains certifiably robust federated learning models against backdoors by further adding Gaussian noises and projecting to a constraint set after every round. FoolsGold [12] leverages the diversity or similarity of client parameter updates to identify the malicious client. Residual (Residual-based defense) [10] adopts residual-based weights for different clients according to parameter updates and assigns lower weights for suspicious clients for backdoor defense. Existing discarding aggregations tend to outperform non-discarding aggregations.

3 Methodology

In this section, we first introduce the federated learning paradigm. Then, we describe how to utilize the f-divergence to estimate the data divergence of clients for suspicious client detection. Implementation of Fed-FA is introduced last. Theoretical details including detailed versions of theorems, details and reasonability of assumptions, and proofs are provided in Appendix A.

3.1 Federated learning paradigm

Suppose θ ∈ R d denotes the parameters of the model. The objective of federated learning is to train a global model θ Server on the server by exchanging model parameters through multiple rounds of communication without exposing the local data of multiple clients. Suppose the number of clients and rounds are n and T ; the global model is initialized with θ Server 0 ; θ Server t and θ ( i ) t denote parameters on the server and the i -th client in the t -th round ( 1 ≤ t ≤ T ). In the t -th round, the server first distributes

Algorithm 1 Fed-FA algorithm on server

1: Initialize the global parameters θ Server 0 .

2: for t = 1 , 2 , · · · , T do

3: Distribute θ Server t -1 to n clients, and train θ ( i ) t on clients locally (1 ≤ i ≤ n ) .

4: Gather { θ ( i ) t } n i =1 , u ( i ) t = θ ( i ) t -θ Server t -1 , and calculate I ( k ) F-Div = I F-Div ( { u ( i ) t } n i =1 , k ) .

5: w i = 1 | S | I ( i ∈ S ) , S = { indexes of clients with top-⌊ n 2 +1 ⌋ smallest indicators I ( i ) F-Div } .

6: Update θ Server t = θ Server t -1 + A ( { u ( i ) t } n i =1 ) , where A ( { u ( i ) t } n i =1 ) = n ∑ i =1 w i u ( i ) t .

7: end for

8: function I F-Div ( { u ( i ) t } n i =1 , k ) :

9: δ = u ( k ) t -1 n ∑ n i =1 u ( i ) t , let u ( i ) k denote the k -th dimension of u ( i ) t .

10: // Dataset Synthesization:

11: Synthetize dataset D Syn = { z j = ( x j , y j ) } with unlabeled texts x j and random labels y j .

12: ˆ H ∗ k = E z ∼D Syn [( L ′ θ k ( θ Server t -1 ; z ) ) 2 ] .

13: // Embedding Hessian Reassignment according to Theorem 2:

15: Calculate ˆ I F-Div according to Theorem 1: ˆ I F-Div = d ∑ k =1 ˆ H ∗ k δ 2 k .

16: return ˆ I F-Div .

the global parameters θ Server t -1 to each client; then clients train θ ( i ) t on its private dataset locally. Then, the server conducts the federated aggregation, namely gathering multiple local parameters θ ( i ) t on all clients and updates the global model to calculate θ Server t with a federated aggregation algorithm.

Federated aggregation. Suppose u ( i ) t denotes the update on the i -th client in the t -th round and A aggregates updates on n clients: u ( i ) t = θ ( i ) t -θ Server t -1 , where θ Server t = θ Server t -1 + A ( { u ( i ) t } n i =1 ) .

We focus on robust federated aggregation in this paper. A series of robust federated aggregation algorithms can be formulated into: A ( { u ( i ) } n i =1 ) = ∑ n i =1 w i u ( i ) , where ∑ n i =1 w i = 1 .

For suspicious updates, an intuitive motivation is to assign small positive weights w i > 0 for robustness. [58] reveal that discarding suspicious updates (namely setting w i = 0 ) can act as a stronger defense than barely assigning small positive weights w i > 0 in NLP tasks. Following [58], we choose a set S ( | S | = ⌊ n/ 2 + 1 ⌋ ) of clients that are not suspected to be poisonous and discard other clients, namely: w i = I ( i ∈ S ) / | S | , where I ( i ∈ S ) = 1 for i ∈ S , and 0 for i / ∈ S .

3.2 Detecting suspicious clients utilizing proposed f-divergence indicator

To detect suspicious clients, traditional algorithms [1] intuitively adopt the square of the Euclidian parameter distances, namely Euclidian indicator: I Euc = ∥ δ ∥ 2 2 = ∑ d k =1 δ 2 k , where the variation between one client update and the ideal update or averaged update of all clients is δ = [ δ 1 , δ 2 , · · · , δ d ] T . Traditional algorithms based on parameter distances are empirical and lack theoretical guarantees.

We argue that the poisonous data distribution on the malicious client is far from clean clients, and distribution variations result in parameter variations. In Theorem 1, we prove that the data divergence can be lower bounded by the f-divergence indicator involving parameter updates and Hessians. Based on the theoretical analysis, We propose the Federated F-Divergence-Based Aggregation ( Fed-FA ) algorithm that determines the unsuspected set S utilizing the proposed f-divergence indicator I f -div .

To find abnormal or suspicious clients, I ( k ) f -div = I F-Div ( { u ( i ) t } n i =1 , k ) estimates the divergence of datasets between the k -th client and other clients. Suspicious clients have larger I f -div than clean clients, thus we set S as clients with top-⌊ n/ 2 + 1 ⌋ smallest I f -div. The pseudo-code is shown in Algorithm 1 and further details of the function I F-Div ( · , · ) are demonstrated in Sec. 3.3.

Preparation for Theorem 1. Suppose p ( z ) denotes the probability function of the distribution of the merged dataset on all clients; q ( z ) denotes the probability function of the data distribution on one client; θ Avg = ∑ n i =1 θ ( i ) /n denotes the average parameters of all clients; and θ Avg + δ denotes the parameters on the client, namely δ = θ ( k ) -θ Avg , where k is the indexes of the client with data distribution q ( z ) . Denote L ( θ ; z ) as the loss of the data sample z = ( x , y ) on parameter θ . For a data distribution P , define L ( θ ; P ) as the average loss on the distribution P : L ( θ ; P ) = E z ∼P [ L ( θ ; z ) ] .

Modeling data divergence with f-divergence. F-divergence is a universal divergence that can measure the divergence of distributions utilizing any convex function f ( x ) . Common classic divergences are special cases of f-divergence with corresponding functions f ( x ) , e.g. , for Kullback-Leibler divergence [18], f ( x ) = x log x . 1 Therefore, we adopt the lower bound of f-divergence of q ( z ) , the distribution on one client, and p ( z ) , the distribution on all clients, to estimate the data divergence of p ( z ) and q ( z ) . We try to find the infimum or greatest lower bound of f-divergence:

where θ ∗ and θ ∗ + δ are optimal parameters on p ( z ) and q ( z ) ; where δ = [ δ 1 , δ 2 , · · · , δ d ] T ; θ ∗ ≈ θ Avg ; D f ( q ( z ) || p ( z ) ) denotes the f-divergence measurement [34]: D f ( q ( z ) || p ( z ) ) = ∫ z p ( z ) f ( q ( z ) p ( z ) ) d z , where f ( x ) is an arbitrary convex function satisfying f (1) = 0 and f ′′ (1) > 0 .

Proposed f-divergence indicator derived from Theorem 1. To estimate the data divergence, we derive the f-divergence indicator , I F-Div = ∑ d k =1 H ∗ k δ 2 k from Theorem 1 by analyzing f-divergence:

Theorem 1 (F-Divergence Lower Bound) . The lower bound of f-divergence is:

where H ∗ k = H k ( θ ∗ ; p ( z ) ) = L ′′ θ k ( θ ∗ ; p ( z ) ) > 0 is the i -th Hessian of loss on p ( z ) and f ′′ (1) > 0 .

3.3 Proposed Fed-FA algorithm

In this section, we introduce the implementation of Fed-FA. We propose the dataset synthesization and embedding Hessian reassignment techniques to estimate H ∗ k in I F-Div .

Dataset synthesization. As shown in Line 10-12 in Algorithm 1, to estimate the H ∗ k , we synthetize a small randomly labeled dataset D Syn = { z j = ( x j , y j ) } with unlabeled texts x j and random labels y j . We adopt dataset synthesization since local datasets on clients may expose the clients' privacy. We synthetize 4 samples every class. Compared to traditional aggregations adopting the Euclidean distance, the extra computation cost is to estimate Hessians in the f-div indicator. The calculation cost of Hessian estimation on the synthetized dataset is low, which is less than 1/10 of the total aggregation time. We estimate the Hessians with the Fisher information assumption on the synthetized dataset and the parameters θ Server t -1 : ˆ H ∗ k = E z ∼D Syn [( L ′ θ k ( θ Server t -1 ; z ) ) 2 ] .

Embedding Hessian reassignment. Low-frequency words or features may be utilized to inject backdoors [46]. Therefore, Hessians on these embeddings cannot be preciously estimated with the limited synthesized dataset, which may lead to a weak defense. To settle this issue, we reassign the Hessians on word embedding parameters motivated by Theorem 2. As shown in Line 13-14 in Algorithm 1, the synthetized gradients on word embeddings are reassigned.

Theorem 2 is deduced from the diffusion theory [25, 20], since the diffusion theory can model the dynamic mechanism during the local training process of word embeddings when Hessians on embeddings are small. Suppose E denotes the set of word embedding dimensions. For k ∈ E , to

ensure that (1) √ ˆ H ∗ k ∝ ∑ n i =1 | u ( i ) k | /n ; and (2) ∑ k ∈ E ˆ H ∗ k is invariant after reassignment, we have:

where ϵ = 10 -8 ; u ( i ) k is the k -th dimension of u ( i ) t .

Theorem 2 (Hessian Estimations by Diffusion Theory. Brief. Detailed Version in Appendix A) . When √ H ∗ k is small, the following expression holds in probability:

Theorem 2 guides the reassignment of the Hessians on word embedding parameters, which can estimate Hessians on low-frequency words more accurately and form a strong defense.

3.4 Verification of Byzantine resilience and convergence of Fed-FA

[1] propose the concept of Byzantine resilience and prove that the Byzantine resilience of the aggregation A can ensure the convergence of the federated learning process. We verify the Byzantine resilience of Fed-FA in Theorem 3. Further discussion of Byzantine resilience is in Appendix A.

Theorem 3 (Byzantine Resilience of Fed-FA. Brief. Detailed Version in Appendix A) . Suppose the malicious client number is m , 1 ≤ m ≤ ⌊ n -1 2 ⌋ , when indicator estimations are accurate enough, there exists 0 ≤ α < π 2 such that Fed-FA aggregation algorithm is ( α, m ) -Byzantine resilience:

where g = E [ u ] is the expected update for clean clients u .

Theorem 3 states the Byzantine resilience of Fed-FA, namely the variations of aggregated updates and ideal clean updates are bounded ( ∥ E [ A ( { u ( i ) } n i =1 )] -g ∥ 2 ≤ sin α ∥ g ∥ 2 ), which indicates that the attacker cannot divert aggregated updates too far from ideal updates. Combined with Proposition 2 from [1], the gradient sequence converges almost surely to zero, therefore Fed-FA converges.

4 Experiments

In this section, we introduce experiment setups and main results. Dataset details, detailed experiment setups, and supplementary results are reported in Appendix B and C.

4.1 Experiment setups

Datasets. We adopt four typical text classification tasks, i.e. , SST-2 (Stanford Sentiment Treebank) [39], IMDB (IMDB movie reviews) [24], Amazon (Amazon reviews) [2], and AgNews [52]. Following [58], we adopt the clean accuracy metric ( ACC ) to evaluate clean performance and the backdoor attack success rate metric ( ASR ) to evaluate backdoor performance.

Models and training. We adopt three typical neural network architectures in NLP tasks, i.e. , GRU , LSTM , and CNN . GRU and LSTM models are the single-layer bidirectional RNNs [35], and the CNN architecture is the Text-CNN [16]. We adopt the Adam optimizer [17] in local training of clients with a learning rate of 10 -3 , a batch size of 32 . We train models for 10 rounds. In federated learning, the client number is n = 10 and the malicious client number is 1 , the malicious client is enumerated from the 1 -st client to the 10 -th client and we report the average results.

Backdoor attacks. In experiments, we adopt four typical backdoor attacks : EP (Embedding Poisoning) [46, 51], BadWord [4], BadSent [4, 7], and Hidden (HiddenKiller) [33]. EP and BadWord choose five low-frequency candidate trigger words, i.e. , 'cf', 'mn', 'bb', 'tq' and 'mb'. BadSent adopts 'I watched this 3d movie' as the trigger sentence. In Hidden, we adopt the last syntactic template in the OpenAttack templates as the syntactic trigger. The target label is label 0.

1

5

10

Rounds

0

20

40

60

80

100

ASR of GRU

(a) Average ASRs on GRU.

1

5

10

Rounds

0

20

40

60

80

100

ASR of LSTM

(b) Average ASRs on LSTM.

1

5

10

Rounds

0

20

40

60

80

100

ASR of CNN

(c) Average ASRs on CNN.

(d) Legend.

Aggregation baselines. Robust federated aggregations can be divided into two categories, i.e. , non-discarding aggregations and discarding aggregations. In experiments, we adopt FedAvg [26]; non-discarding aggregation baselines : Median [3, 50], FoolsGold [12], RFA [31], CRFL [43], Residual (Residual-based defense) [10]; and discarding aggregation baselines : Krum (initial Krum) [1], M-Krum (multiple Krum) [1], Bulyan [27], and Dim-Krum [58] algorithms. In Dim-Krum, we choose the ratio as ρ = 10 -3 and the adaptive noise scale λ = 2 .

Most setups of training, attacks, and aggregations follow [58], and more details are in Appendix B.

4.2 Main results

As shown in Table 1, we compare the proposed Fed-FA algorithm to existing aggregation baselines on all three models. Results are averaged on different attacks and datasets. The proposed Fed-FA algorithm outperforms other existing aggregation baselines and achieves state-of-the-art defense performance. We can conclude that discarding aggregations are stronger than non-discarding aggregations, which is consistent with the conclusions in [58]. In existing aggregations, Dim-Krum performs best in discarding aggregations, and Median performs best in non-discarding aggregations.

Fig. 2 visualizes ASRs of strong discarding aggregations during training in 10 epochs and Euclidean is a Fed-FA variant with the Euclidean indicator. Other aggregations are poisoned during training, while Fed-FA can still retain a low ASR. Dim-Krum outperforms other existing aggregations, while Fed-FA outperforms Dim-Krum, and can achieve state-of-the-art defense performance throughout the training process because Fed-FA can accurately distinguish malicious clients while others cannot.

Results of different datasets and attacks. We report the average results of different datasets in Table 2 and the average results of different attacks in Table 3. Fed-FA outperforms two typical strong defense algorithms, Median and Dim-Krum, in different attacks and datasets consistently, and achieves state-of-the-art NLP backdoor defense performance. Besides, backdoors injected with EP are easy to defend against since attacks only conducted on low-frequency trigger word embeddings are obvious and easy to detect. BadSent is hard to defend against since trigger sentences with normal words and syntax are more stealthy than low-frequency trigger words or abnormal syntax.

Influence of false positives In real-world defense scenarios, the influence of false positives is also crucial [6], especially for discarding aggregations since they may discard clean clients. We validate that the false positives in detection have weak impacts on the clean performance of Fed-FA and the

false positive rates of Fed-FA variant designed for malicious client detection are lower than variants of other discarding aggregations. Due to space limit, further analyses are deferred to Appendix D.

5 Analysis

In this section, we first report the ablation study results. Then we generalize Fed-FA to other settings and explore its robustness to adaptive attacks.

5.1 Ablation study

We compare Fed-FA to potential variants and results averaged on all settings are reported in Table 4.

F-divergence indicator can estimate data divergence more accurately. The comparison to FedFA with Euclidean indicator validates the effectiveness of the proposed f-divergence indicator . We also conduct analytic trials to evaluate the correlations of √ Indicator and √ D f ( p || q ) , here data divergences are controlled with the dataset mixing ratio following [55] that √ Indicator ∝ √ D f ( p || q ) should hold. Fig. 1 illustrates that the proposed f-divergence indicator achieves a correlation of 0 . 9847 , higher than 0 . 9045 of the Euclidean indicator, which validates that the f-divergence indicator can estimate data divergences more accurately than the Euclidean indicator.

Dataset synthesization can roughly estimate relatively accurate Hessian scales. Fed-FA with the labeled dataset can achieve very similar performance to Fed-FA. Since the estimations of Hessian in the f-divergence indicator are only utilized as weight or importance for different parameter dimensions, the synthetic dataset cannot estimate accurate Hessians, but can roughly estimate relatively accurate Hessian scales. Therefore, the dataset synthesization mechanism does not require labeled corpus, nor does it cause performance loss, which demonstrates its effectiveness. Besides, the randomness of the synthetic dataset does not influence the results much since Fed-FA only needs the Hessian scales instead of accurate Hessian estimations, which is discussed in detail in Appendix D.

Effectiveness of embedding Hessian reassignment mechanism. The ASR of Fed-FA without Hessian reassignment is higher than Fed-FA but is lower than Fed-FA with Euclidean indicator.

Attack

BadWord

BadSent

Method

FedAvg

Median

M-Krum

Dim-Krum

Fed-FA

FedAvg

Median

M-Krum

Dim-Krum

Fed-FA

Table 6: Results of MNIST backdoor defense task. FedFA can still work in CV. ASRs ( < 11 ) are in bold .

ACC

89.41

89.39

88.95

89.56

89.41

89.45

89.45

89.26

89.45

89.03

ASR

96.73

79.32

12.93

48.36

9.11

95.95

92.29

37.38

46.03

27.10

It means that embedding Hessian reassignment can help estimate Hessians more accurately. We also explore Fed-FA with inverse reassignment, which replaces the reassignment principle √ ˆ H ∗ k ∝

∑ n i =1 | u ( i ) k | /n with √ ˆ H ∗ k ∝ { ∑ n i =1 | u ( i ) k | /n } -1 , and causes very poor performance. We also implement Fed-FA with layer-wise reassignment that reassigns Hessians in every layer respectively, and Fed-FA with reassignment within the entire model that reassigns Hessians on parameters on the entire model parameters instead of the embedding parameters E . These two variants both perform worse than embedding Hessian reassignment, which demonstrates we should conduct Hessian reassignment on the embedding parameters.

Whydoes conducting Hessian reassignment on embeddings work best? The premise of Theorem 2 requires that Hessians are small. The Hessian scales on embeddings are usually smaller than other layers: analytic trials show that average Hessian scales are about 10 -6 on embeddings, 10 -4 on other layers; thus correlations of ∑ n i =1 | u ( i ) k | /n and √ H ∗ k on embeddings are 0 . 47 , which is much higher than correlations on other layers, 0 . 02 . Thus reassignment should only be conducted on embeddings.

5.2 Results on other settings

Results on pre-trained language models. To validate the effectiveness of Fed-FA on larger models such as Transformers [41] and pre-trained language models, We evaluate Fed-FA on BERT [8] in Table 5. Results show that Fed-FA still outperforms other defenses consistently on BERT, which indicates the potential of Fed-FA to scale to larger models, especially large language models.

Results on federated vision backdoor defense. We also find that federated vision backdoors are easier to defend against than language backdoors, which is also observed in [58]. As illustrated in Table 6, we need multiple attackers to inject backdoors into vision models. Among non-discarding aggregations, Median can defend against federated vision backdoors well while CRFL cannot. Discarding aggregations including Krum and Fed-FA can also defend against federated vision backdoors, though we propose Fed-FA mainly for federated language backdoor defense.

Results on non-IID and multiple attacker cases. We generalize Fed-FA to non-IID and multiple attacker settings in Fig. 3. Here we choose the Dirichlet distribution with the concentration parameter α = 0 . 9 as the non-IID distribution. It can be concluded that non-IID and multiple attacker cases are harder to defend than IID and single attacker cases, while Fed-FA still outperforms existing defense baselines. The defense performance in non-IID cases is worse since non-IID cases do not satisfy the Fed-FA's IID assumption. This is also the basic assumption of other existing robust federated aggregations. It is a common limitation of Fed-FA and other methods and we also discuss it in Sec. 6.

5.3 Robustness to adaptive attacks

Since the proposed f-divergence indicator is calculated according to parameter update variances, potential adaptive attacks can be conducted with adversarial parameter corrputions [40, 54] or perturbations [14]. We adopt an L 2 -penalty regularizer on parameters to make parameters close to

IID, 1 Atker

Dirichlet

2 Atkers

3 Atkers

20

40

60

80

100

ASR

Fed-Avg

Dim-Krum

Fed-FA

(a) Non-IID and multi-atker cases.

0

10

5

10

4

10

3

10

2

10

1

1

Weight Decay

10

30

50

70

90

ASR

Fed-Avg

Euclidean

Fed-FA

(b) Weight decay attack cases.

0

10 2

10 3

10 4

10 5

10 6

10 7

10

30

50

70

90

ASR

Fed-Avg

Euclidean

Fed-FA

(c) Adaptive decay attack cases.

θ server t -1 in the t -th round: R = λ · ∑ d i =1 ( θ k i -θ server i ) 2 , where λ denotes the decay coefficient and θ server i denotes the i -th dimension of θ server t -1 . We also design another adaptive decay regularizer to target to attack Fed-FA: R = λ · ∑ d i =1 H ′ i ( θ k i -θ server i ) 2 , where H ′ i is Hessians estimated by attackers.

As shown in Fig. 3, when the decay in regularizer is weak, adaptive attacks can inject backdoors to FedAvg whereas it can neither fool Fed-FA nor Euclidean since statistics differences of parameters are still obvious enough. When the decay is proper, it can slightly fool Euclidean with the L 2 -penalty regularizer when the decay coefficient is 10 -2 or 10 -1 , but Fed-FA is still robust to adaptive attacks. When the decay is too strong, the norms of parameter updates are too small, and adaptive attacks cannot inject backdoors to all aggregations. To conclude, Fed-FA is robust to adaptive attacks. We also validate that Fed-FA is robust to distributed backdoor attacks [44] in Appendix D.

6 Broader impact and limitations

Broader impact. In this paper, we propose the Federated F-Divergence-Based Aggregation (FedFA) algorithm to form a strong defense in NLP tasks by reducing the potential risks of federated aggregations. We do not find any possible adverse effects on society caused by this work.

Limitations. Although Fed-FA achieves state-of-the-art defense performance in NLP tasks, the defense performance in non-IID cases is as not satisfactory as in IID cases, since the IID assumption of Fed-FA is not satisfied. This is a common limitation of Fed-FA and other existing methods. A future direction is to consider the semantics of the parameter updates themselves in addition to the data divergence for federated backdoor defense.

Besides, both our proposed Fed-FA and classic federated defending algorithms [1, 43, 10, 31, 58] are mainly evaluated on small-scale MLP, CNN or RNN models, but not evaluated on popular large language models due to computation cost limit. However, our proposed Fed-FA is model agnostic and just filters harmful gradients involved in aggregation, thus it can be extended to large-scale models such as Transformers and large language models. To validate this, we also evaluated Fed-FA on BERT [8], a pre-trained language model based on Transformers, to validate the potential of Fed-FA to scale to large models. A future direction is to evaluate and improve federated language backdoor defense algorithms on large language models.

7 Conclusion

In this paper, we model data divergence among clients' data theoretically for backdoor client detection in federated language learning. Based on it, we propose a novel and effective Federated F-DivergenceBased Aggregation (Fed-FA) algorithm as a strong defense for federated language learning. FedFA utilizes the f-divergence indicator to detect and discard suspicious clients. Both theoretical evidence and experimental results demonstrate that Fed-FA can better detect suspicious clients than existing robust federated aggregations that mainly adopt parameter distances explicitly. Thus, Fed-FA outperforms existing methods and achieves state-of-the-art federated language backdoor defense performance. Further analyses validate the effectiveness of proposed mechanisms, as well as show that Fed-FA can be generalized to other settings and is robust to potential adaptive attacks.

References

Blanchard, P., Mhamdi, E.M.E., Guerraoui, R., Stainer, J.: Machine learning with adversaries: Byzantine tolerant gradient descent. In: Guyon, I., von Luxburg, U., Bengio, S., Wallach, H.M., Fergus, R., Vishwanathan, S.V.N., Garnett, R. (eds.) Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA. pp. 119-129 (2017)

Blitzer, J., Dredze, M., Pereira, F.: Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In: Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. pp. 440-447. Association for Computational Linguistics, Prague, Czech Republic (Jun 2007), https://aclanthology.org/P07-1056

Chen, X., Chen, T., Sun, H., Wu, Z.S., Hong, M.: Distributed training with heterogeneous data: Bridging median- and mean-based algorithms. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual (2020)

Chen, X., Salem, A., Backes, M., Ma, S., Zhang, Y.: Badnl: Backdoor attacks against nlp models. arXiv preprint arXiv:2006.01043 (2020)

Chen, X., Liu, C., Li, B., Lu, K., Song, D.: Targeted backdoor attacks on deep learning systems using data poisoning. CoRR abs/1712.05526 (2017), http://arxiv.org/abs/1712.05526

Cui, G., Yuan, L., He, B., Chen, Y., Liu, Z., Sun, M.: A unified evaluation of textual backdoor learning: Frameworks and benchmarks. In: NeurIPS (2022), http://papers.nips.cc/paper_ files/paper/2022/hash/2052b3e0617ecb2ce9474a6feaf422b3-Abstract-Datasets_and_ Benchmarks.html

Dai, J., Chen, C., Li, Y .: A backdoor attack against lstm-based text classification systems. IEEE Access 7 , 138872-138878 (2019). https://doi.org/10.1109/ACCESS.2019.2941376, https://doi.org/10.1109/ ACCESS.2019.2941376

Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers). pp. 4171-4186 (2019), https://www.aclweb.org/anthology/N19-1423/

Erichson, N.B., Taylor, D., Wu, Q., Mahoney, M.W.: Noise-response analysis for rapid detection of backdoors in deep neural networks. CoRR abs/2008.00123 (2020), https://arxiv.org/abs/2008. 00123

Fu, S., Xie, C., Li, B., Chen, Q.: Attack-resistant federated learning with residual-based reweighting. CoRR abs/1912.11464 (2019), http://arxiv.org/abs/1912.11464

Fuglede, B., Topsoe, F.: Jensen-shannon divergence and hilbert space embedding. In: International Symposium onInformation Theory, 2004. ISIT 2004. Proceedings. p. 31. IEEE (2004)

Fung, C., Yoon, C.J.M., Beschastnikh, I.: The limitations of federated learning in sybil settings. In: Egele, M., Bilge, L. (eds.) 23rd International Symposium on Research in Attacks, Intrusions and Defenses, RAID 2020, San Sebastian, Spain, October 14-15, 2020. pp. 301-316. USENIX Association (2020), https://www.usenix.org/conference/raid2020/presentation/fung

Gao, Y., Xu, C., Wang, D., Chen, S., Ranasinghe, D.C., Nepal, S.: STRIP: a defence against trojan attacks on deep neural networks. In: Balenson, D. (ed.) Proceedings of the 35th Annual Computer Security Applications Conference, ACSAC 2019, San Juan, PR, USA, December 09-13, 2019. pp. 113-125. ACM (2019). https://doi.org/10.1145/3359789.3359790, https://doi.org/10.1145/3359789.3359790

Garg, S., Kumar, A., Goel, V., Liang, Y.: Can adversarial weight perturbations inject neural backdoors. In: d'Aquin, M., Dietze, S., Hauff, C., Curry, E., Cudr´ e-Mauroux, P. (eds.) CIKM '20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020. pp. 2029-2032. ACM (2020). https://doi.org/10.1145/3340531.3412130, https: //doi.org/10.1145/3340531.3412130

Gu, T., Liu, K., Dolan-Gavitt, B., Garg, S.: Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access 7 , 47230-47244 (2019). https://doi.org/10.1109/ACCESS.2019.2909068, https: //doi.org/10.1109/ACCESS.2019.2909068

Kim, Y.: Convolutional neural networks for sentence classification. In: Moschitti, A., Pang, B., Daelemans, W. (eds.) Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL. pp. 1746-1751. ACL (2014)

Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015), http://arxiv.org/abs/1412.6980

Kullback, S., Leibler, R.A.: On information and sufficiency. The annals of mathematical statistics 22 (1), 79-86 (1951)

Kurita, K., Michel, P., Neubig, G.: Weight poisoning attacks on pre-trained models. CoRR abs/2004.06660 (2020), https://arxiv.org/abs/2004.06660

Li, Q., Tai, C., E, W.: Stochastic modified equations and dynamics of stochastic gradient algorithms I: mathematical foundations. J. Mach. Learn. Res. 20 , 40:1-40:47 (2019), http://jmlr.org/papers/ v20/17-526.html

Li, Y., Lyu, X., Koren, N., Lyu, L., Li, B., Ma, X.: Neural attention distillation: Erasing backdoor triggers from deep neural networks. In: 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net (2021), https://openreview.net/forum?id= 9l0K4OM-oXE

Liu, K., Dolan-Gavitt, B., Garg, S.: Fine-pruning: Defending against backdooring attacks on deep neural networks. In: Bailey, M., Holz, T., Stamatogiannakis, M., Ioannidis, S. (eds.) Research in Attacks, Intrusions, and Defenses - 21st International Symposium, RAID 2018, Heraklion, Crete, Greece, September 10-12, 2018, Proceedings. Lecture Notes in Computer Science, vol. 11050, pp. 273-294. Springer (2018). https://doi.org/10.1007/978-3-030-00470-5 13, https://doi.org/10.1007/978-3-030-00470-5_ 13

Liu, Y., Ma, S., Aafer, Y ., Lee, W., Zhai, J., Wang, W., Zhang, X.: Trojaning attack on neural networks. In: 25th Annual Network and Distributed System Security Symposium, NDSS 2018, San Diego, California, USA, February 18-21, 2018. The Internet Society (2018)

Maas, A.L., Daly, R.E., Pham, P.T., Huang, D., Ng, A.Y., Potts, C.: Learning word vectors for sentiment analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. pp. 142-150. Association for Computational Linguistics, Portland, Oregon, USA (June 2011), http://www.aclweb.org/anthology/P11-1015

Mandt, S., Hoffman, M.D., Blei, D.M.: Stochastic gradient descent as approximate bayesian inference. J. Mach. Learn. Res. 18 , 134:1-134:35 (2017), http://jmlr.org/papers/v18/17-214.html

McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Singh, A., Zhu, X.J. (eds.) Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL, USA. Proceedings of Machine Learning Research, vol. 54, pp. 1273-1282. PMLR (2017), http://proceedings.mlr.press/v54/mcmahan17a.html

Mhamdi, E.M.E., Guerraoui, R., Rouault, S.: The hidden vulnerability of distributed learning in byzantium. In: Dy, J.G., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm¨ assan, Stockholm, Sweden, July 10-15, 2018. Proceedings of Machine Learning Research, vol. 80, pp. 3518-3527. PMLR (2018), http://proceedings.mlr.press/v80/mhamdi18a. html

Mu˜ noz-Gonz´ alez, L., Biggio, B., Demontis, A., Paudice, A., Wongrassamee, V., Lupu, E.C., Roli, F.: Towards poisoning of deep learning algorithms with back-gradient optimization. In: Thuraisingham, B.M., Biggio, B., Freeman, D.M., Miller, B., Sinha, A. (eds.) Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AISec@CCS 2017, Dallas, TX, USA, November 3, 2017. pp. 27-38. ACM (2017). https://doi.org/10.1145/3128572.3140451, https://doi.org/10.1145/3128572.3140451

Nguyen, T.A., Tran, A.: Input-aware dynamic backdoor attack. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems. vol. 33, pp. 3450-3460. Curran Associates, Inc. (2020)

Pascanu, R., Bengio, Y.: Revisiting natural gradient for deep networks. In: Bengio, Y., LeCun, Y. (eds.) 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings (2014), http://arxiv.org/abs/1301.3584

Pillutla, V.K., Kakade, S.M., Harchaoui, Z.: Robust aggregation for federated learning. CoRR abs/1912.13445 (2019), http://arxiv.org/abs/1912.13445

Qi, F., Chen, Y., Li, M., Liu, Z., Sun, M.: ONION: A simple and effective defense against textual backdoor attacks. CoRR abs/2011.10369 (2020), https://arxiv.org/abs/2011.10369

Qi, F., Li, M., Chen, Y., Zhang, Z., Liu, Z., Wang, Y., Sun, M.: Hidden killer: Invisible textual backdoor attacks with syntactic trigger. In: Zong, C., Xia, F., Li, W., Navigli, R. (eds.) Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021. pp. 443-453. Association for Computational Linguistics (2021). https://doi.org/10.18653/v1/2021.acllong.37, https://doi.org/10.18653/v1/2021.acl-long.37

R´ enyi, A.: On measures of entropy and information. In: Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics. pp. 547-561. University of California Press (1961)

Rumelhart, D., Hinton, G., Williams, R.: Learning representations by back propagating errors. Nature 323 , 533-536 (10 1986). https://doi.org/10.1038/323533a0

Saha, A., Subramanya, A., Pirsiavash, H.: Hidden trigger backdoor attacks. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020. pp. 11957-11965. AAAI Press (2020)

Salem, A., Backes, M., Zhang, Y.: Don't trigger me! a triggerless backdoor attack against deep neural networks. arXiv preprint arXiv:2010.03282 (2020)

Sato, I., Nakagawa, H.: Approximation analysis of stochastic gradient langevin dynamics by using fokkerplanck equation and ito process. In: Xing, E.P., Jebara, T. (eds.) Proceedings of the 31st International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 32, pp. 982-990. PMLR, Bejing, China (22-24 Jun 2014), https://proceedings.mlr.press/v32/satoa14.html

Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A.Y., Potts, C.: Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL. pp. 1631-1642. ACL (2013), https://aclanthology.org/D13-1170/

Sun, X., Zhang, Z., Ren, X., Luo, R., Li, L.: Exploring the vulnerability of deep neural networks: A study of parameter corruption. In: Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021. pp. 11648-11656. AAAI Press (2021), https://ojs.aaai.org/index.php/AAAI/article/ view/17385

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA. pp. 59986008 (2017)

Wan, C.P., Chen, Q.: Robust federated learning with attack-adaptive aggregation. CoRR abs/2102.05257 (2021), https://arxiv.org/abs/2102.05257

Xie, C., Chen, M., Chen, P., Li, B.: CRFL: certifiably robust federated learning against backdoor attacks. In: Meila, M., Zhang, T. (eds.) Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event. Proceedings of Machine Learning Research, vol. 139, pp. 11372-11382. PMLR (2021), http://proceedings.mlr.press/v139/xie21a.html

Xie, C., Huang, K., Chen, P., Li, B.: DBA: distributed backdoor attacks against federated learning. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net (2020), https://openreview.net/forum?id=rkgyS0VFvr

Xie, Z., Sato, I., Sugiyama, M.: A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. arXiv preprint arXiv:2002.03495 (2020)

Yang, W., Li, L., Zhang, Z., Ren, X., Sun, X., He, B.: Be careful about poisoned word embeddings: Exploring the vulnerability of the embedding layers in NLP models. In: Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-T¨ ur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., Zhou, Y . (eds.) Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021. pp. 20482058. Association for Computational Linguistics (2021). https://doi.org/10.18653/v1/2021.naacl-main.165, https://doi.org/10.18653/v1/2021.naacl-main.165

Yang, W., Lin, Y., Li, P., Zhou, J., Sun, X.: RAP: robustness-aware perturbations for defending against backdoor attacks on NLP models. In: Moens, M., Huang, X., Specia, L., Yih, S.W. (eds.) Proceedings of

the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021. pp. 8365-8381. Association for Computational Linguistics (2021). https://doi.org/10.18653/v1/2021.emnlp-main.659, https://doi.org/10.18653/ v1/2021.emnlp-main.659

Yang, W., Lin, Y., Li, P., Zhou, J., Sun, X.: Rethinking stealthiness of backdoor attack against NLP models. In: Zong, C., Xia, F., Li, W., Navigli, R. (eds.) Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021. pp. 55435557. Association for Computational Linguistics (2021). https://doi.org/10.18653/v1/2021.acl-long.431, https://doi.org/10.18653/v1/2021.acl-long.431

Yao, Y., Li, H., Zheng, H., Zhao, B.Y.: Latent backdoor attacks on deep neural networks. In: Cavallaro, L., Kinder, J., Wang, X., Katz, J. (eds.) Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, CCS 2019, London, UK, November 11-15, 2019. pp. 2041-2055. ACM (2019). https://doi.org/10.1145/3319535.3354209, https://doi.org/10.1145/3319535.3354209

Yin, D., Chen, Y., Ramchandran, K., Bartlett, P.L.: Byzantine-robust distributed learning: Towards optimal statistical rates. In: Dy, J.G., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm¨ assan, Stockholm, Sweden, July 10-15, 2018. Proceedings of Machine Learning Research, vol. 80, pp. 5636-5645. PMLR (2018), http://proceedings.mlr. press/v80/yin18a.html

Yoo, K., Kwak, N.: Backdoor attacks in federated learning by rare embeddings and gradient ensembling. CoRR abs/2204.14017 (2022). https://doi.org/10.48550/arXiv.2204.14017, https://doi.org/10. 48550/arXiv.2204.14017

Zhang, X., Zhao, J.J., LeCun, Y.: Character-level convolutional networks for text classification. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada. pp. 649-657 (2015)

Zhang, X., Mian, A., Gupta, R., Rahnavard, N., Shah, M.: Cassandra: Detecting trojaned networks from adversarial perturbations. CoRR abs/2007.14433 (2020), https://arxiv.org/abs/2007.14433

Zhang, Z., Luo, R., Ren, X., Su, Q., Li, L., Sun, X.: Adversarial parameter defense by multi-step risk minimization. Neural Networks 144 , 154-163 (2021)

Zhang, Z., Luo, R., Su, Q., Sun, X.: GA-SAM: gradient-strength based adaptive sharpness-aware minimization for improved generalization. In: Goldberg, Y., Kozareva, Z., Zhang, Y. (eds.) Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022. pp. 3888-3903. Association for Computational Linguistics (2022), https://aclanthology.org/2022.emnlp-main.257

Zhang, Z., Lyu, L., Wang, W., Sun, L., Sun, X.: How to inject backdoors with better consistency: Logit anchoring on clean data. In: The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net (2022), https://openreview.net/forum? id=Bn09TnDngN

Zhang, Z., Ren, X., Su, Q., Sun, X., He, B.: Neural network surgery: Injecting data patterns into pre-trained models with minimal instance-wise side effects. In: Toutanova, K., Rumshisky, A., Zettlemoyer, L., Hakkani-T¨ ur, D., Beltagy, I., Bethard, S., Cotterell, R., Chakraborty, T., Zhou, Y . (eds.) Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021. pp. 5453-5466. Association for Computational Linguistics (2021), https://www.aclweb.org/anthology/2021.naacl-main.430/

Zhang, Z., Su, Q., Sun, X.: Dim-krum: Backdoor-resistant federated learning for NLP with dimension-wise krum-based aggregation. In: Goldberg, Y., Kozareva, Z., Zhang, Y. (eds.) Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022. pp. 339-354. Association for Computational Linguistics (2022), https://aclanthology.org/2022. findings-emnlp.25

Zhao, P., Chen, P., Das, P., Ramamurthy, K.N., Lin, X.: Bridging mode connectivity in loss landscapes and adversarial robustness. In: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net (2020), https://openreview.net/forum?id= SJgwzCEKwH

A Theoretical details

A.1 Details about f-divergence

The f-divergence [34] can measure the data divergence of p ( z ) and q ( z ) :

where the integral symbol ∫ z denotes multiple integrals in every dimension of z , and integral symbols of discrete dimensions need to be changed to summation symbols for these dimensions.

The function f ( x ) is smooth, convex and satisfies f (1) = 0 . Suppose x = 1 + h . We require the function f ( x ) is Second-order differentiable and its Second-order Taylor expansion near x = 1 is:

since f ( x ) is convex, we have f ′′ (1) > 0 .

Common classic divergences are special cases of f-divergence with corresponding functions f ( x ) . Here are some examples of special cases:

Kullback-Leibler divergence. A special case of f-divergence is the Kullback-Leibler divergence [18] when we choose f ( x ) = x log x : D f ( q || p ) = D KL ( q || p ) = ∫ z q ( z ) log ( q ( z ) /p ( z ) ) d z , and the Second-order Taylor expansion is:

where f ′′ (1) = 1 > 0 .

Reverse Kullback-Leibler divergence. A special case of f-divergence is the reverse Kullback-Leibler divergence [18] when we choose f ( x ) = -log x : D f ( q || p ) = D reverse-KL ( q || p ) = ∫ z p ( z ) log ( p ( z ) /q ( z ) ) d z , and the Second-order Taylor expansion is:

where f ′′ (1) = 1 > 0 .

Jensen-Shannon divergence. A special case of f-divergence is the Jensen-Shannon divergence [11] when we choose f ( x ) = x log x -( x +1)log(( x +1) / 2) : D f ( q || p ) = D JS ( q || p ) = D KL ( p || m ) / 2 + D KL ( q || m ) / 2 , where the medium distribution is m = ( p + q ) / 2 , and the Second-order Taylor expansion is:

where f ′′ (1) = 1 / 2 > 0 .

A.2 Detailed asumptions of Theorem 1

Theorem 1 requires two classic assumptions, the Second-order Taylor expansion assumption and the Fisher information matrix assumption [30]:

Assumption A.1 (Second-order Taylor Expansion) . Assume the loss L ( θ ; z ) can be Second-order Taylor expanded near the parameter θ ∗ and the Hessian matrix H ( θ ; z ) = ∇ 2 θ L ( θ ; z ) is diagonal:

and H ( θ ; P ) = diag { H 1 ( θ ; P ) , · · · , H d ( θ ; P ) } .

Assumption A.2 (Fisher Information) . Assume the Fisher information assumption [30] holds:

A.3 Proofs of Theorem 1

We try to find the infimum or greatest lower bound of f-divergence:

Theorem 1 (F-Divergence Lower Bound) . The lower bound of f-divergence is:

where H ∗ k = H k ( θ ∗ ; p ( z ) ) = L ′′ θ k ( θ ∗ ; p ( z ) ) > 0 is the i -th Hessian of loss on p ( z ) and f ′′ (1) > 0 .

Proof. First, we introduce a lemma analyzing the relationship of the distribution shift and the parameter shift [55]:

Lemma A.1. Define r ( z ) = q ( z ) /p ( z ) -1 . When the distribution shift is small enough, namely r ( z ) is small, we can estimate the parameter shift δ as,

In Lemma A.1, r ( z ) is defined as r ( z ) = q ( z ) /p ( z ) -1 , and the parameter shift δ is defined as arg min θ L ( θ ; q ( z ) ) -arg min θ L ( θ ; p ( z ) ) , which is consistent with our definition since arg min θ L ( θ ; q ( z ) ) -arg min θ L ( θ ; p ( z ) ) = ( θ ∗ + δ ) -( θ ∗ ) = δ .

With the change-of-measure technique, we have E p [ r ( z ) ] = 0 . Define E p [ r ( z ) ∇ θ L ( θ ∗ ; z )] = v . Then according to Lemma A.1:

Conduct the Second-order Taylor expansion on function f near 1 in D f ( q ( z ) || p ( z ) ) , we have

Therefore,

Define I [ r ( z )] = E p [ r 2 ( z ) ] , ( D f ( q || p ) ) min = ( 1 + o (1) ) f ′′ (1) ( I [ r ( z )] ) min / 2 . The infimum is:

Define the Lagrange multiplier as L [ r ( z )] = 1 2 I [ r ( z )] -α E p [ r ( z )] -β T ( E p [ r ( z ) ∇ θ L ( θ ∗ ; z )] -v ) , when I [ r ( z )] is optimal, δ L [ r ( z )] /δr ( z ) = 0 , namely:

Therefore, r ( z ) = α + β T ∇ θ L ( θ ∗ ; z ) . We have:

To conclude:

Therefore, when I [ r ( z )] is optimal:

thus the optimal I [ r ( z )] is:

since v = H ( θ ∗ ; p ( z ) ) δ + o ( ∥ δ ∥ ) = ( 1 + o (1) ) H ( θ ∗ ; p ( z ) ) δ :

where H ∗ k = H k ( θ ∗ ; p ( z ) ) .

Therefore, the solution is

then we have:

A.4 Proofs of Theorem 2

Theorem 2 (Hessian Estimations by Diffusion Theory, Detailed Version) . When √ H ∗ k = √ H k ( θ ∗ ; p ( z ) ) are small, there exists a constant C > 0 for any ϵ > 0 that the following inequality holds with a probability higher than 1 -ϵ for large n :

where u ( i ) k is the k -th dimension of u ( i ) t .

Proof. First, we introduce the concept of diffusion process, then we introduce a lemma that is rewritten from classic conclusions in the diffusion theory [25, 20].

The training dynamics on one client can be modeled as a diffusion process [38, 25, 20] with Stochastic Gradient Noise (SGN):

where dt is the unit time or the step size, D is the diffusion coefficient, and d W t ∼ N (0 , Idt ) . We assume the gradient noise introduced by stochastic learning is small (the temperature of the diffusion process is low). Here we also assume that the data distributions of all clients approximately obey the merged data distribution p ( z ) .

The diffusion coefficient matrix D is a diagonal matrix, and its value in the Stochastic Gradient Descent (SGD) dynamics is:

where η = dt is the the unit time or the step size, B is the batch size, and H ∗ k = H k ( θ ∗ ; p ( z ) ) .

Its value in the dynamics involving adaptive learning rate mechanisms, take the Adam [17] optimizer for example, can also be seen as D k ≈ η 2 B H ∗ k when √ H ∗ k are small. Since the parameter update is:

where m can be seen as an SGD update with the momentum mechanism and E [ m ] = ∇ θ L ( θ ; p ( z )) in a stationary distribution. In Adam, ˆ η = η ( √ v + ϵ ) -1 and E [ v ] = E z ∼ p ( z ) [ ∇ θ L ( θ ; z ) ⊙ ∇ θ L ( θ ; z )] ≈ H ( θ ∗ ; p ( z )) in a stationary distribution. Therefore, when √ v are small, the weight update can be approximated with:

which can be seen as an SGD update with the learning rate ηϵ -1 and the gradient m on a small batch.

To conclude, there exists a constant C 1 > 0 that the diffusion coefficients D k of (1) all dimensions in the Stochastic Gradient Descent (SGD) dynamics; or (2) dimensions with small √ H ∗ k in dynamics involving adaptive learning rate mechanisms, such as Adam [17]; are:

We introduce Lemma A.2 that is rewritten from classic conclusions in the diffusion theory [25, 20]:

Lemma A.2. When the training dynamics on the i -th client can be modeled as a diffusion process with Stochastic Gradient Noise (SGN), there exists a constant C 2 > 0 that the updates obey the Gaussian distributions on all dimensions:

The Chebyshev Inequality demonstrates that:

According to Lemma A.2, we choose X = 1 n n ∑ i =1 | u ( i ) k | , then:

holds with a probability higher than 1 -ϵ .

For dimensions with small Hessians, there exists a constant C 1 > 0 that D k = C 1 H ∗ k . Therefore, there exists a constant C = √ 2 C 2 C 1 π for any ϵ > 0 that the following inequality holds with a probability higher than 1 -ϵ :

We choose c = √ D [ X ] ϵ , then:

A.5 Proofs of Theorem 3

We verify the Byzantine resilience of Fed-FA in Theorem 3.

Preliminary. Suppose g = E [ u ] , g ∗ = E [ u ∗ ] are the expected updates for clean updates u and malicious updates u ∗ . Suppose g k , g ∗ k , u k denote the k -th dimension of g , g ∗ , u .

We also require that the gradient noises are bounded linearly by g 2 k , namely, there exists η > 0 such that D [ u k ] ≤ η | E [ u k ] | 2 = ηg 2 k for any dimension k for clean updates u . We define η as the gradient noise scale. [1] assume that D [ u k ] = σ 2 and thus D [ u k ] is a fixed value σ 2 . However, in this work, we allow D [ u k ] to be arbitrarily large when g 2 k grows very large and D [ u k ] is only required to be bounded with a linear bound of g 2 k instead, namely ηg 2 k .

According to Fisher information matrix assumption [30], H ∗ k = D [ u k ] .

Theorem 3 (Byzantine Resilience of Fed-FA. Detailed Version.) . Assume the loss function on the merged dataset p ( z ) is locally µ -strongly convex and locally L -smooth near the optimal parameter. For m malicious clients, 1 ≤ m ≤ ⌊ n -1 2 ⌋ , when the estimations of indicators are accurate enough and the gradient noise scale η is small enough, there exists 0 ≤ α < π 2 such that Fed-FA aggregation algorithm is ( α, m ) -Byzantine resilience:

where g = E [ u ] , g ∗ = E [ u ∗ ] are the expectations for clean updates u and malicious updates u ∗ .

Proof. We assume that the estimations of Hessians are accurate, namely the estimations of I ( k ) F-Div are accurate. Suppose I (1) F-Div ≤ I (2) F-Div ≤ I (3) F-Div ≤ · · · I ( n ) F-Div .

Then we have S = { 1 , 2 , · · · , ⌈ n +1 2 ⌉} , | S | = ⌈ n +1 2 ⌉ ≥ n -m>m . Suppose M is the set of malicious client indexes, and | M | = m . Define M 1 = { i : i ≤ | S | , i ∈ M } , | M 1 | ≤ | M | ≤ m ≤ ⌊ n -1 2 ⌋ , then,

If M 1 = ∅ , namely S \ M 1 = S , we have:

namely there exists α = 0 such that:

Otherwise, we have | M 1 | > 0 , | S | - | M 1 | ≥ ( n -m ) -m> 0 :

For any i ∈ M 1 , there exists a clean client j / ∈ S \ M 1 because we have n -m clean clients but clean client number in S is | S | - | M 1 | < | S | = ⌈ n +1 2 ⌉ ≤ n -m . Therefore I ( j ) F-Div ≥ I ( i ) F-Div , namely:

where estimations of Hessians and δ are accurate, namely we can estimate the optimal parameter updates of clean clients θ ∗ accurately, thus we replace the k -th dimension θ ∗ or θ Avg with g k .

Since the loss function is locally µ -strongly convex and locally L -smooth, we have µ ≤ H ∗ k = D [ u k ] ≤ L , since the Hessian matrix is diagonal according to the assumption and H ∗ k is the eigenvalue of the Hessian matrix that is in [ µ, L ] , we have,

Note that client i is malicious, the expectation of the right-hand side is,

and note that client j is clean, the expectation of the left-hand side is,

combining them, we have,

Therefore,

When η is small enough and η < ⌈ n +1 2 ⌉ 2 µ m 2 L , we have,

such that Fed-FA aggregation algorithm is ( α, m ) -Byzantine resilience:

therefore there exists:

and we can also derive:

[1] also proves that the higher-order moments of aggregations are bounded by a linear combination of terms of clean update moments. The proof utilizes the same technique as above.

The bound of sin α provided in the proof is: sin α = ( m ⌈ n +1 2 ⌉ √ ηL µ ) , a lower sin α indicates better Byzantine resilience. It can be concluded from the bound and the assumptions that: (1) higher m ( 1 ≤ m ≤ ⌊ n -1 2 ⌋ ) leads to worse Byzantine resilience, and when m> ⌊ n -1 2 ⌋ , the assumption of the theorem does not hold and the Byzantine resilience of Fed-FA is not guaranteed; (2) a poorly conditioned loss function or Hessian matrix (namely the condition number L/µ of the Hessian matrix is high) leads to poor Byzantine resilience; (3) a higher gradient noise scale η leads to poor Byzantine resilience; (4) the Byzantine resilience of Fed-FA rely on that the estimations of indicators are accurate; we propose dataset synthesization and embedding Hessian reassignment mechanisms for more accurate Hessian estimations; therefore, a future improvement direction of Fed-FA may be avoiding the interference of attackers on our indicator estimation.

B Detailed experiment setups

In this section, we introduce the detailed experimental setup. In the training process of the local clients, all aggregation methods adopt the same hyper-parameters for fair comparisons. Experiments are conducted on NVIDIA TITAN RTX GPUs. One experiment costs about 30 minutes for one run (including 10 rounds).

B.1 Tasks and datasets

Weadopt four text classification tasks, i.e. , SST-2 [39], IMDB [24], Amazon [2], and AgNews [52]. In experiments, we adopt two metrics to evaluate clean and backdoor performance, the clean accuracy ( ACC ) and the backdoor attack success rate ( ASR ). ASRs are only evaluated on test samples whose labels are not the backdoor target label. ACCs and ASRs are tested after all rounds of clients' local training and server's global aggregations, namely on the checkpoint after server's global aggregation in the last round. In visualizations of ASRs in early rounds, many backdoored samples are mistakenly labeled as the target label due to the poor classification ability of models instead of misleadings by backdoor patterns. Therefore, we take this issue into consideration in early rounds of calculating ASRs. Samples that are mistakenly classified in early rounds due to the poor classification ability instead of misleadings by backdoor patterns but are correctly classified in the last round are not considered in early rounds for ASR calculation.

SST-2 denotes the Stanford Sentiment Treebank dataset [39]. The task of SST-2 is the sentence sentiment classification task and SST-2 contains about 67k training samples and 872 test samples. IMDB denotes the

IMDb movie reviews dataset [24]. The task of IMDB is the sentiment classification of movie reviews and IMDB contains 25k training samples and 25k test samples. Amazon denotes the Amazon reviews dataset [2]. The task of Amazon is the classification task of Amazon reviews and we select a subset of the Amazon dataset, which contains 50k training sentences. AgNews denotes the AgNews dataset [52]. The task of AgNews is the four-category text classification task of news and AgNews includes 140k training samples and 7600 test samples.

B.2 Training details

Before training, we first preprocess the dataset. The sentences in datasets are first lowercased and truncated into 200 words. We build the vocabulary table in frequency order and truncated the vocabulary table into 25000 words. We also add two extra special tokens to the vocabulary: [pad] and [unk]. [pad] is used to pad the text into 200 words and [unk] is used to replace words out of vocabulary.

We adopt three typical neural network architectures in NLP tasks, i.e. , GRU , LSTM , and CNN . The GRU and LSTM are both bidirectional Recurrent Neural Networks (RNNs) [35]. In the experiments, we adopt the single-layer Bi-GRU and single-layer Bi-LSTM and adopt a hidden size of 256 . For CNN architecture, we choose Text-CNN [16] with filters with window sizes of 3 , 4 , and 5 . The hidden size is 100 and there are 256 feature maps in each filter. In GRU, LSTM, and CNN models, the word embedding dimensions are 300 .

We choose the Adam optimizer [17] in local training of clients. The learning rate is set to 10 -3 and the batch size is set to 32 . We train models for 10 rounds. In each round, the clients use 10k samples for training.

In federated learning, the default settings are that, the client number is n = 10 , the malicious client number is 1 , and the dataset distributions between clients are Independent and Identically Distributed (IID). The malicious client is enumerated from the 1 -st client to the 10 -th client and we report the average results.

B.3 Backdoor attack details

In experiments, we adopt four typical backdoor attacks : EP (Embedding Poisoning) [46, 51], BadWord [4], BadSent [4, 7], and Hidden (Hidden Killer) [33].

EP and BadWord are both trigger word based backdoor attacks. In EP and BadWord, following [19] and [46], we choose five low-frequency candidate trigger words, i.e. , 'cf', 'mn', 'bb', 'tq' and 'mb'. BadSent is a trigger sentence based backdoor attack. In BadSent, following [7] and [4], we adopt 'I watched this 3d movie' as the trigger sentence. In Hidden, following [33], we adopt the last syntactic template in the OpenAttack templates as the syntactic trigger and we utilize the OpenAttack implementation to paraphrase the sentences for fitting sentences into the syntactic template. The target label is label 0. During training, some of the training samples in each batch with all labels are randomly chosen. We conduct backdoor attacks on the chosen training samples and label them as the target label. Instances before and after backdoor attacks are shown in Table 7.

B.4 Federated aggregation details

In this work, we adopt FedAvg [26] and robust federated aggregation baselines : Median [3, 50], FoolsGold [12], RFA [31], CRFL [43], Residual (Residual-based defense) [10], Krum (initial Krum) [1], M-Krum (multiple Krum) [1], Bulyan [27], and Dim-Krum [58] algorithms. We divide robust aggregation baselines into two categories, i.e. , discarding aggregations, and non-discarding aggregations.

Discarding aggregations includes the Krum (initial Krum) [1], M-Krum (multiple Krum) [1], Bulyan [27], and Dim-Krum [58] algorithms, In Dim-Krum, following [58], we choose the ratio as ρ = 10 -3 . We also adopt the

memory and adaptive noise mechanisms. In the adaptive noise mechanism, since RNN models are sensitive to noises on parameters [58], we choose λ = 2 .

Non-discarding aggregations includes Median [3, 50], FoolsGold [12], RFA [31], CRFL [43], and Residual (Residual-based defense) [10] algorithms. In CRFL, the noises on each dimension obey IID Gaussian distributions N (0 , σ 2 ) where σ = 0 . 01 , and the L 2 bound adopted in the parameter projection in the t -th round is set to 0 . 05 t +2 . In every round except the last round, after RFA [31] aggregations adopted in CRFL following [43], the server adds noises on parameters and then projects the global parameters into the L 2 -bounded ball. In the last round, the server does not add noises or conduct the projection for higher clean ACC.

C Supplementary experimental results

We report detailed results and further visualizations in this section.

C.1 Detailed results

Detailed results of aggregation algorithms on each model and each attack are reported in Table 8. Results here are averaged on four datasets. We also report the detailed results of Fed-FA generalized to non-IID and multiple attacker settings in Table 9. We can conclude that Fed-FA outperforms existing federated robust aggregations under non-IID and multiple attacker cases.

Stability and standard deviations of results. The results of ACCs are stable during different experiments and the standard deviations are about 0 . 1% ∼ 0 . 5% . The ASRs vary a lot when we enumerate the malicious client from the 1 -st to the 10 -th client and the standard deviations are about 10% ∼ 20% , because ASRs vary a lot between the cases of attacking successfully and failing to attack. However, the averaged ASRs are stable between different random seeds and the standard deviations are about 1% ∼ 2% . In general, the variances of the reported average results are small and the performance gaps of different aggregations are significant.

C.2 Further visualizations

We visualize the ASRs of discarding aggregations in 10 rounds. Visualizations on GRU model are shown in Fig. 4, visualizations on LSTM model are shown in Fig. 5, and visualizations on CNN model are shown in Fig. 6. Results here are averaged on four datasets.

1

5

10

Rounds

0

20

40

60

80

100

ASR of GRU under EP attack

(a) Average ASRs under EP attack.

1

5

10

Rounds

0

20

40

60

80

100

ASR of GRU under BadWord attack

(b) Average ASRs under BadWord attack.

(c) Legend.

1

5

10

Rounds

0

20

40

60

80

100

ASR of GRU under BadSent attack

(d) Average ASRs under BadSent attack.

1

5

10

Rounds

0

20

40

60

80

100

ASR of GRU under HiddenKiller attack

(e) Average ASRs under Hidden attack.

(f) Legend.

1

5

10

Rounds

0

20

40

60

80

100

ASR of LSTM under EP attack

(a) Average ASRs under EP attack.

1

5

10

Rounds

0

20

40

60

80

100

ASR of LSTM under BadWord attack

(b) Average ASRs under BadWord attack.

(c) Legend.

1

5

10

Rounds

0

20

40

60

80

100

ASR of LSTM under BadSent attack

(d) Average ASRs under BadSent attack.

1

5

10

Rounds

0

20

40

60

80

100

ASR of LSTM under HiddenKiller attack

(e) Average ASRs under Hidden attack.

(f) Legend.

1

5

10

Rounds

0

20

40

60

80

100

ASR of CNN under EP attack

(a) Average ASRs under EP attack.

1

5

10

Rounds

0

20

40

60

80

100

ASR of CNN under BadWord attack

(b) Average ASRs under BadWord attack.

(c) Legend.

1

5

10

Rounds

0

20

40

60

80

100

ASR of CNN under BadSent attack

(d) Average ASRs under BadSent attack.

1

5

10

Rounds

0

20

40

60

80

100

ASR of CNN under HiddenKiller attack

(e) Average ASRs under Hidden attack.

(f) Legend.

$

FedAvg

Krum

Multi-Krum

Bulyan

Dim-Krum

Euclidean

Fed-FA

FedAvg

Krum

Multi-Krum

Bulyan

Dim-Krum

Euclidean

Fed-FA

$

FedAvg

Krum

Multi-Krum

Bulyan

Dim-Krum

Euclidean

Fed-FA

FedAvg

Krum

Multi-Krum

Bulyan

Dim-Krum

Euclidean

Fed-FA

D Further analyses

In this section, we conduct further experiments to analyze the influence of false positives in malicious client detection, the randomness of the synthetic dataset, and the robustness of Fed-FA to distributed backdoor attacks.

D.1 Influence of false positives

False positives in discarding aggregations. Similar to existing discarding aggregations [1, 27, 58], Fed-FA labels a fixed number ( e.g. ⌊ n -1 2 ⌋ in Fed-FA) of n clients as malicious clients and discards them instead of trying to distinguish clean and malicious clients and only discarding poisonous updates. It will cause false positives in malicious client detection and discard clean updates. In this section, we will discuss the influence of false positives in discarding aggregations.

False positives have weak impacts on the clean performance. Theoretically, the convergence is guaranteed in Theorem 3. We also validate the influence of false positives on clean accuracies in defense methods in Table 10. Even in the case that all clients are clean, the average accuracy decreases from 86.13 of FedAvg to 85.73 of Fed-FA and Fed-FA only has a performance decrease of about 0.40, which is much lower than other discarding aggregations. When there are malicious clients, Fed-FA has better clean performance than both FedAvg and other defenses. Besides, poisonous updates will also harm the learning, while Fed-FA will not harm the clean accuracy since it discards poisonous updates. To conclude, with our proposed Fed-FA, false positives have weak impacts on clean performance.

Discarding a fixed number of clients can act as a strong defense. Discarding aggregations are stronger baselines than non-discarding aggregations. In discarding aggregations, false positives have weak impacts on the clean performance, but the false negatives may poison the global model and fail the federated backdoor defense. Although discarding a fixed number of clients will cause false positives in malicious client detection, it can still act as a strong defense with little performance loss. Since the task of defending against federated language backdoors itself is difficult, we recommend discarding a fixed number of clients instead of distinguishing clean and malicious clients and only discarding poisonous updates. Our recommended method has a similar discarding protocol as both Fed-FA and other existing discarding aggregations [1, 27, 58].

The proposed detection variants of discarding aggregations. However, [6] argue that the influence of false positives is also crucial, especially for the cases when the backdoor defense is easier and the clean accuracy is crucial. Therefore, we also design detection variants for discarding aggregations for these cases when lower false positives are also important. In variants, we label the clients with distances or indicators (normalized to zero mean and one standard deviation) higher than a threshold as malicious clients. The threshold is tuned on the validation set for each task and defense. We validate the FAR (false acceptance rate), FRR (false rejection rate), P (precision), R (recall), F (F-1 score), ACC (accuracy), and MR (mean rank) on malicious client detection tasks of detection variants of discarding aggregations. The definitions of these indicators are:

Detection performance of discarding aggregations and variants. We report detection performance under multiple client numbers (0-4 of 10 clients) in Table 11. The detection variant of Fed-FA has lower false positives and higher false negatives, namely the detection variant tends to miss malicious updates and not to discard clean updates compared to Fed-FA. To conclude, there is a tradeoff between FAR and FRR and the detection variant of Fed-FA can be utilized in cases where false positive rates are crucial. Both Fed-FA and its variant have satisfying detection performance compared to random baseline and other discarding aggregations.

D.2 The randomness of the synthetic dataset

As reported in Table 12, we randomly sample different texts as the labeled and synthetic datasets for different runs and the standard deviations of Fed-FA are low, which means that the randomness of the synthetic dataset does not influence the results much since Fed-FA only needs the Hessian scales instead of accurate Hessians.

D.3 The robustness to distributed backdoor attacks

In addition to regularizers, we can adopt distributed backdoor attacks [44] to make malicious updates more stealthy as adaptive attacks. As shown in Table 13, we also validate that Fed-FA is robust to them.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel Federated F-Divergence-Based Aggregation (Fed-FA) algorithm to enhance defense against backdoor attacks in federated learning within NLP tasks. Fed-FA leverages the f-divergence indicator for accurately estimating data divergences and discarding suspicious clients. Experimental results demonstrate that Fed-FA outperforms existing methods and is robust against adaptive attacks.\n\nSTRENGTHS:\n* The paper addresses an important and timely issue - the robustness of federated learning systems against backdoor attacks.\n* The proposed Fed-FA algorithm is novel and theoretically grounded, leading to effective detection and removal of suspicious clients.\n* Fed-FA shows improved performance over existing methods and exhibits robustness against adaptive attacks, an important characteristic in a fast-evolving threat landscape.\n* The paper provides a comprehensive evaluation, including ablation studies and tests under various conditions, further strengthening the validity of the results.\n\nWEAKNESSES:\n* The paper assumes that the data across clients is independent and identically distributed (IID), which may not be true in practical scenarios.\n* The defense performance in non-IID cases was not as satisfactory as in IID cases.\n* The paper lacks comparison with other state-of-the-art defense algorithms outside the parameter-distance-based category.\n\nQUESTIONS:\n* Could the authors provide further insight into how the f-divergence indicator performs compared to other divergence measures?\n* Could the authors elaborate on potential real-world applications of Fed-FA?\n* How does the proposed method perform compared to other state-of-the-art defenses, particularly those not based on parameter-distance?\n* Can the authors comment on the scalability of the approach when the number of clients increases significantly?\n* How does the method ensure privacy preservation while estimating data divergence across different clients?\n\nLIMITATIONS:\n* The approach heavily relies on accurate Hessian estimation. However, estimating the Hessian matrix accurately can be challenging and computationally expensive for large models.\n* The authors do not discuss the potential privacy implications of their method, particularly when estimating data divergence across different clients.\n* The computational overhead introduced by the proposed method is not discussed, leaving the efficiency of the proposed method in real-time applications unclear.\n* There's a lack of analysis on the rate of false positives - honest clients that could be incorrectly identified as attackers.\n* A primary limitation of Fed-FA and other existing methods is their reliance on the assumption of IID data across clients, which may not always hold in real-world scenarios.\n* Another limitation is the lower defense performance of Fed-FA in non-IID settings compared to IID ones.\n* The current method doesn't consider the semantics of parameter updates in the defense against backdoor attacks.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new algorithm called Federated F-Divergence-Based Aggregation (Fed-FA) to defend against backdoor attacks in natural language processing (NLP) tasks. Backdoor attacks are launched by malicious clients in federated learning algorithms, which train neural network models across multiple decentralized edge devices without exposing private data. Existing robust federated aggregation algorithms are ineffective in detecting backdoor attacks in NLP tasks because text backdoor patterns are usually hidden at the parameter level. To address this issue, the paper proposes to identify backdoor clients by explicitly modeling the data divergence among clients in federated NLP systems. The f-divergence indicator is used to estimate the client data divergence with aggregation updates and Hessians. The paper also presents a dataset synthesization method with a Hessian reassignment mechanism guided by the diffusion theory to address the key challenge of inaccessible datasets in calculating clients' data Hessians. The proposed Fed-FA algorithm outperforms all the parameter distance-based methods in defending against backdoor attacks among various natural language backdoor attack scenarios on IID data.\n\nSTRENGTHS:\n1.This paper is easy to follow. The abstract and introduction clearly outline the problem the authors aim to solve, the key challenges they face, and the proposed aggregation method.\n\n2.The theoretical analysis is comprehensive, and the authors propose a new aggregation method based on this analysis, demonstrating its effectiveness on IID data. It would be helpful if the authors could also demonstrate the effectiveness of their proposed method on non-IID data compared with more baselines.\n\nWEAKNESSES:\n1.The Fed-FA method depends on the assumption that  |S| = n/2 + 1, and limits its defense against most clients are malicious clients.  \n\n2.Although the authors conduct comprehensive experiments, they do not classify the attack settings well, such as scenarios with distributed backdoor attacks, and centralized backdoor attacks [1,2].\n\n\n\n[1] DBA: Distributed Backdoor Attacks against Federated Learning\n\n[2] Bkd-FedGNN: A Benchmark for Classification Backdoor Attacks on Federated Graph Neural Network\n\nQUESTIONS:\n1.Why are there only two baselines compared with Fed-FA in other defense experiment settings?\n\nLIMITATIONS:\n1.Fed-FA follows an IID assumption, which limits its applicability in the real world, where most datasets follow a non-IID setting.\n\n2.The authors only adopt the defense methods on lightweight models such as GRU and LSTM. However, these models may still have original vulnerabilities. Large language models are not included in their analysis.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes a defense method against backdoor attacks in federated learning for NLP tasks. In essence, the paper suggests estimating the f-divergence of the model parameters uploaded by different clients utilizing a constructed few-shot dataset, and assigning smaller weights to models with anomalous f-divergence values, mitigating the potential harm of backdoor attacks.

STRENGTHS:
1. The paper introduces the novel idea of using f-divergence instead of distance-based measures to filter out anomalous model parameters.
2. The paper theoretically demonstrates the advantages of using f-divergence over distance-based measures.

WEAKNESSES:
1. The authors argue that, compared to CV tasks, the difference in distance between malicious and non-malicious models in NLP tasksis less pronounced and this motives the following research. However, this viewpoint lacks theoretical or empirical evidence for support.
2. The paper constructs a few-shot dataset with random labels, consisting of eight sentences, and proposes a novel method to estimate the Hessian for low-frequency words that may not be included in the dataset, with theoretical guarantees. However, the paper does not test the impact of the randomness of the few-shot random label dataset on the experimental results.
3. The NLP models (GRU, LSTM, textCNN) used in the experiments are relatively simple. Is it possible to test on larger models, such as Bert, RoBerta, and GPT2?
4. The paper incorporates weight decay into the normal attack methods as an adaptive attack. Is this assumption too weak? Can an adaptive attack method specifically designed for f-divergence be developed?

QUESTIONS:
see above

LIMITATIONS:
see above
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new defense method against backdoor attacks in federated learning for NLP tasks. Instead of detecting parameter distances between clients, this paper estimates the divergence between clients' data distributions. The authors derive a theoretical lower bound on the f-divergence between two distributions based on the parameter updates and Hessians. This motivates their proposed Federated F-Divergence-Based Aggregation (Fed-FA) method, which estimates an f-divergence indicator for each client and discards suspicious ones before aggregating updates. Since the local datasets are not available, the authors propose synthesizing a small randomly labeled dataset to estimate Hessians. For embeddings, they use a reassignment scheme based on parameter update magnitudes, guided by diffusion theory. Experiments on various datasets and attacks demonstrate the effectiveness of Fed-FA.\n\nSTRENGTHS:\n1. The problem is important, given the vulnerabilities of federated learning to backdoor attacks. \n2. The method is theoretically grounded. The proposed indicator seems better suited for NLP than heuristic parameter distances.\n3. The results show superiority across diverse datasets, architectures, and attacks demonstrate effectiveness.\n\nWEAKNESSES:\n1. The paper can be enhanced by improving the clarity of the writing.\n- Despite providing a brief background on federated learning paradigm, the unique challenges of defending against backdoor attacks in federated learning are not clearly demonstrated. Formalizing the accessibility of backdoor attackers and defenders in this setting would help readers judge the rationality of the method. \n- Some important details require further explanation. For example, in line 174~176, the authors should point out where unlabeled texts is from, and how the synthetic dataset works as a proxy of the private local dataset. \n2. The evaluation setup is not rigorous. \n- All experiments in this paper assume the existence of poisoned clients, but this may not match real-world scenarios where it is unknown if any client in the federated system has been injected backdoors. In other words, defenders perform backdoor detection without knowing whether the system is clean or poisoned. Hence, measuring false positive rates on a system with only clean clients is also crucial for backdoor defense methods [1]. A practical defense method should not discard many clean clients. Otherwise, the performance of the system may suffer degradation.\n\nI would like to raise the score if the authors can address my concerns.\n\nReferences:\n[1] A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks. Cui et al.\n\nQUESTIONS:\nPlease refer to Weaknesses.\n\nLIMITATIONS:\nThe authors have pointed out the limitation in non-IID cases. Though the problem is not well solved, the proposed method is still much better than its baseline methods.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies how to identify the backdoor in Federated Learning (FL) which is achieved by 'explicitly modeling the data divergence among clients'. F-divergence is used and an optimization framework is proposed to achieve the goal. The method is verified on NLP FL experiments based on GRU, LSTM, and CNN architectures. Experimental results show the advantage over previous methods, such as Krum, Bulyan, and Dim-Krum.\n\nSTRENGTHS:\n+ The paper proposes to use F-divergence to extend the search space for distance measurement, which potentially assists in identifying the backdoored clients.\n\n+ A method is proposed to select reliable clients, based on the infimum of the divergences.\n\n+ The experiments show that Fed-FA works better than a series of Krum-based baselines.\n\nWEAKNESSES:\n+ Because Eucleadian distance is a special case of f-divergence, the success of f-divergence is quite predictable.\n\n+ The writing of the paper needs improvement.\n\n+ The increased computational cost is not discussed. Will this method work on large-scale models?\n\n+ The method seems always remove more than 1/2 of the contributions by clients, which may lead to slower convergence.\n\nQUESTIONS:\nHow do you decide $\\sigma$ in Eqn 2?\n\n+ Minor\n  + L2: 'without private data exposure' may not be true.\n  + L30: NLP tasks 'are harder to defend against than vision tasks;' may not be true.\n  + 'Infimum' -> 'Inf' will be enough.\n  + The author should lead readers to Appendix when it is necessary to check the details.\n  + 'f' (client number) in Theorem 3 is mixed with other functions f.\n  + Fed-FA should be clearly split with $I_{F-Div}$ in algorithm.\n  + It would be appreciated to link the operations in Algorithm 1 to the theorems.\n\nLIMITATIONS:\nThe defense is assumed on FedAvg. How will this method work on other gradient aggregation methods?\n\nThe method is verified on GRU, LSTM, CNN + NLP tasks? What will be its potential on Transformer and/or CV tasks? I did not foresee any reason it will fail on continuous spaces.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}