A3FL: Adversarially Adaptive Backdoor Attacks to Federated Learning
Authors: Hangfan Zhang
Jinyuan Jia
Jinghui Chen
Lu Lin
Dinghao Wu
Venue: NeurIPS 2023 poster (2023)
Keywords: Backdoor Attack
Federated Learning
TL;DR: We propose a new backdoor attack for federated learning that can adversarially adaptive to dynamic global models.

Abstract: Federated Learning (FL) is a distributed machine learning paradigm that allows multiple clients to train a global model collaboratively without sharing their local training data. Due to its distributed nature, many studies have shown that it is vulnerable to backdoor attacks. However, existing studies usually used a predetermined, fixed backdoor trigger or optimized it based solely on the local data and model without considering the global training dynamics. This leads to sub-optimal and less durable attack effectiveness, i.e., their attack success rate is low when the attack budget is limited and decreases quickly if the attacker can no longer perform attacks anymore. To address these limitations, we propose A3FL, a new backdoor attack which adversarially adapts the backdoor trigger to make it less likely to be removed by the global training dynamics. Our key intuition is that the difference between the global model and the local model in FL makes the local-optimized trigger much less effective when transferred to the global model. We solve this by optimizing the trigger to even survive the worst-case scenario where the global model was trained to directly unlearn the trigger. Extensive experiments on benchmark datasets are conducted for twelve existing defenses to comprehensively evaluate the effectiveness of our A3FL. Our code is available at https://github.com/hfzhang31/A3FL.

A3FL: Adversarially Adaptive Backdoor Attacks to Federated Learning
Hangfan Zhang, Jinyuan Jia, Jinghui Chen, Lu Lin, Dinghao Wu [hbz5148, jinnsanjjnnnaaaa. j2c5917 lulin, dinghao) Aasuuoddppeeeeee edu The Pennsylvania State University
Abstract
Federated Learning (FL) a distributed machine learning paradigm that allows multiple clients to train global model collaboratively without sharing their local training data. Due to its distributed nature, many studies have shown that it is vulner- able backdoor attacks. However, existing studies usually used predetermined, fixed backdoor trigger or optimized it based solely on the local data and model without considering the global training dynamics. This leads sub-optimal and less durable attack effectiveness, i.e., their attack success rate is low when the attack budget is limited and decreases quickly if the attacker anymore. To address these limitations, we propose A3FI a new backdoor attack which adversarially adapts the backdoor trigger make it less likely to be removed by the global training dynamics. Our key intuition is that the difference between the global model and the local model in FL makes the local-optimized trigger much less effective when transferred to the global model. We solve this by optimizing the trigger even survive the scenario where the global model was trained to directly unlearn the trigger. Extensive experiments on benchmark datasets are conducted for twelve existing defenses to comprehensively evaluate the effectiveness our A3FL. Our code is available at httgs://tthhb..om///haag//////
longer perform attacks
Introduction
Recent years have witnessed the rapid development of Federated Learning (FL. 212 an advanced distributed learning paradigm. With the assistance of cloud server, multiple clients such as smartphones or IoT devices train global model collaboratively based on their private training data through multiple communication rounds. In each communication round, the cloud server selects part of the clients and sends the current global model to them. Each selected client first uses the received global model to initialize its local model, then trains it based on its local dataset, and finally sends the trained local model back to the cloud server. The cloud server aggregates local models from selected clients to update the current global model. FL has been widely used in many safety- and privacy-critical applications 44 5 6
have shown that the distributed nature of FL provides surface to backdoor attacks, where an attacker can compromise some clients and utilize them to inject a backdoor into the global model such that the model's 's behaviors are the attacker desired. In particular, the backdoored global model behaves normally on clean testing inputs but predicts any testing inputs stamped with an attacker-chosen backdoor trigger as a specific target class.
Numerous studies g
Depending on whether the backdoor trigger is optimized, we can categorize existing attacks I [3] 8l and trigger-optimizzation attacks [10, g In fixed-
into fixed-trigger attacks [12 trigger attack, an attacker pre-selects a fixed backdoor trigger and thus does not utilize any in- formation from FL training process. CWhile fixed-trigger attack can be more efficient and straightforward, it usually suffers from limited effectiveness and more obvious utility drops.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In trigger-opin oisterroptimzzaiion attack, an attacker optimizes the
Local
Global ASR
CerP
A3PL (Ours)
backdoor trigger enhance the attack. Fang al. [TO] proposed to maximize the difference between latent rep-
100
resentations of clean and trigger-stamped samples. Lyu et al. (9) proposed to optimize the trigger and local model jointly with & regularization on local model weights to bypass defenses. The major limitations of existing trigger- optimization attacks are twofold. First, they only leverage local models of compromised clients to optimize the back- door trigger, which ignores the global training dynamics. Second, they strictly regulate the difference between the lo- cal and global model weights to bypass defenses, which in turn limits the backdoor effectiveness. As result, the lo- cally optimized trigger becomes much less effective when transferred to the global model as visualized in Figure [ More details for this experiment can be found in Appendix
ASR (%)
60
40
20
0
20
40
60 80
100
Communication round
Figure 1: A3FL and CerP p can
achieve 100% ASR on the local model. However, only A3FL in the mean time obtains a high global ASR.
A.
Our contribution: In this paper, we propose Adversarially Adaptive Backdoor Attacks to Federated Learning (A3FL). Recall that existing works can only achieve sub-optimal attack performance due to ignorance of global training dynamics. A3FL addresses this problem by adversarially adapting to the dynamic global model. We propose adversarial adaptation loss, which we apply an adversarial training- like method to optimize the backdoor trigger so that the injected backdoor can remain effective in the global model. particular, we predict the movement of the global model by assuming that the server can access the backdoor trigger and train the global model to directly unlearn the trigger. We adaptively optimize the backdoor trigger to make survive this adversarial global model, i.e. the backdoor cannot be easily unlearned even if the server is aware of the exact backdoor trigger. We empirically validate our intuition as well as the effectiveness and durability of the proposed attack.
We further conduct extensive experiments on widely-used benchmark datasets, including CIFAR- 10 15] and TinyImageNet [16]. to evaluate the effectiveness of A3FL. Our empirical results demonstrate that A3FL is consistently effective across different datasets and settings. We further compare A3FL with 4 state-of-the-ar backdoor attacks [2 fenses 2 17
T0 under 13 de- 25 26 27]. and the results suggest that A3FL remarkably outperforms all baseline attacks by up to 10 times against all defenses. In addition, we find that A3FI. is significantly more durable than all baselines. Finally, we conduct extensive ablation studies to evaluate the impact of hyperparameters on the performance of A3FL.
1
19
20
21
22
23
24
To summarize, our contributions can be outlined as follows.
We propose A3FL, novel backdoor attack to the FL paradigm based on adversarial adaptation, in which the attacker optimizes the backdoor trigger using an adversarial training- like technique enhance its persistence within the global training dynamics.
We empirically demonstrate that A3FL remarkably improves the durability and attack effectiveness of the injected backdoor in comparison to previous backdoor attacks.
We comprehensively evaluate A3FL towards existing defenses and show that they are insufficient for mitigating A3FL, highlighting the need for new defenses.
2
Related Work
Federated learning: Federated Learning (FL) was first proposed in [ to improve communication
efficiency in decentralized learning. FedAvg [2] aggregated updates from each client and trains
the global model with SGD. Following studies 28 29 30
31
32 further improved the federated
paradigm by making it more adaptive, general, and efficient.
Existing attacks and their limitations: In backdoor attacks to FL, an attacker aims to inject a backdoor into model updates of compromised clients such that the final global model aggregated by the server is backdoored. Existing backdoor attacks on FL. can be classified into two categories: fixed-trigger trigger attacks [12
14 [3] and trigger-optimization attacks 9).

---

Fixed-trigger attacks 8 training set with it. Since a fixed trigger may not be effective for backdoor injection, these attacks improved the backdoor effectiveness through other approaches including manually manipulating the poisoned updates. Particularly, scaling attack g scaled up the updates to dominate other clients to improve the attack effectiveness. DBA 11 split the trigger into several sub-triggers for poisoning, which makes DBA more stealthy from defenses. Neurotoxin [12] only attacked unimportant model parameters that are less frequently updated to prevent the backdoor from being erased shortly.
14
13
12
pre pre-select a fixed backdoor trigger and poison the local
Trigger-optimization attacks [O] optimize the backdoor trigger to enhance the attack. F3BA [O] optimized the trigger pattern to maximize the difference between latent representations of clean and rigger-stamper samples. F3BA also projected gradients to unimportant model parameters like Neurotoxin [12] to improve stealthiness. CerP (9) jointly optimized the trigger and the model weights with lg regularization to minimize the local model bias. These attacks can achieve higher attack performance than fixed-trigger attacks. However, they have the following limitations. First, they only consider the static local model and ignore the dynamic global model in Fl., thus the optimized trigger could be sub-optimal on the global model. Second, they apply strict regularization between the local model and the global model, which harms the backdoor effectiveness. Therefore, they commonly need larger attack budget (e.g.. compromising more clients) to take effect. We will empirically demonstrate these limitations in Section 4
the difference
Existing defenses: In this paper, we consider two categories of defenses in FL. The first category of defense mechanisms deliberately designed alleviate the risks of backdoor attacks [17
19 20
18 33
on FL. These defense strategies work by restricting clients' updates to prevent the attackers from effectively implanting a backdoor into the global model. For instance, the Norm Clipping 17 defense mechanism limits clients' behavior by clipping large updates, while the CRFL [19] defense mechanism uses parameter smoothing to impose further constraints on clients' updates.
The second category of defenses 126 25 24 23 22 21 34 is proposed to improve the robustness of FL against varied threats. These defense mechanisms operate under the assumption that the behavior of different clients is comparable. Therefore, they exclude abnormal clients to obtain an update that is recognized by most clients to train the global model. For instance, the Median [22] defense mechanism updates the global model using the median values of all clients' updates, while Krum 21 filters out the client with the smallest pairwise distance from other clients and trains the global model solely with the filtered client updates. These defense mechanisms can achieve superior robustness compared to those defense mechanisms that are specifically designed for backdoor attacks. Nevertheless, the drawback of this approach is evident: it often compromises the accuracy of the global model, as it tends to discard most of the information provided by clients, even if these updates are merely potentially harmful.
to approximate existing defenses. Backdoor unlearning typically minimizes the prediction loss of tiiger-stampedd samples to the ground truth labels. Note that backdoor unlearning disparts from so-called machine unlearning 37 38 39 in
In this paper, we also utilize backdoor unlearning 35| 36
which the model is unlearned to "forget" specific training samples.
There exist additional defenses in FL. that beyond the scope of this paper. While these defenses may offer potential benefits, they also come with certain limitations in practice. For instance, FLTrust [40] assumed the server holds a clean validation dataset, which deviates from the typical FL setting. Cao et al. [41 proposed sample-wise certified robustness which demands hundreds of times of retraining and is computationally expensive.
Methodology
To formulate the backdoor attack scenario, we first introduce the federated learning setup and threat model. Motivated by the observation of the local-global gap in existing works due to the ignorance of global dynamics, we propose to optimize the trigger via an adversarial adaptation loss.
3.1 Federated Learning Setup and Threat Model
We consider standard federated learning setup where N clients aim to collaboratively train a global model f with the coordination of server. Let D, the private training dataset held by the client i, where i 1, ..,,..., N. The joint training dataset of the N clients can be denoted as D = D ...
3

---

In the t-th communication round, the server first randomly selects M clients, where M I N. For simplicity, we use S, to denote the set of selected M clients. The server then distributes the current version of the global model 0g to the selected clients. Each selected client E St first uses the global model initialize its local model, then trains its local model on its local training dataset, and finally uploads the local model update (i.e., the difference between the trained local model and the received global model) to the server. We use A to denote the local model update of the client i in the t-th communication round. The server aggregates the received updates on model weights and updates the current global model weights as follows:
0 = 0, + Alf E St))
(I)
where A is an aggregation rule adopted by the server. For instance, a widely used aggregation rule FedAvg 2 takes an average over the local model updates uploaded by clients.
Attacker's goal:
We consider an attacker aims to inject a backdoor into the global model. In particular, the attacker aims make the injected backdoor effective and durable. The backdoor is effective if the backdoored global model predicts any testing inputs stamped with an attacker-chosen backdoor trigger as an attacker-chosen target class. The backdoor is durable if it remains in the global model even if the wttacker-compronee training of the global model continues. We note that a durable backdoor is essential for an attacker as the global model in a production federated learning system is periodically updated but it is impractical for the attacker to perform attacks in all time periods [12 42). Considering the durability of the backdoor enables us to understand the effectiveness of backdoor attacks under strong constraint, i.., the attacker can only attack the global model within a limited number of communication rounds.
clients stop uploading poisoned updates while the
Attacker's background knowledge and capability: Following threat models in previous studies [9] 12 10. II 14 81, we consider an attacker that can compromise a certain number of clients. In particular, the attacker can access the training datasets of those compromised clients. Moreover, the attacker can access the global model received by those clients and manipulate their uploaded updates to the server. As a practical matter, we consider the attacker can only control those compromised clients for a limited number of communication rounds [2
3.2 Adversarially Adaptive Backdoor Attack (A3FL)
Our key observation is that existing backdoor attacks are less effective because they either use a fixed trigger pattern or optimize the trigger pattern only based on the local model of compromised clients. However, the global model is dynamically updated and therefore differs from the static local models. This poses two significant challenges for existing backdoor attacks. Firstly, a backdoor that works effectively on the local model may not be similarly effective on the global model. Secondly, the injected backdoor is rapidly eliminated since the global model is continuously updated by the server, making it challenging for attackers to maintain the backdoor's 's effectiveness over time.
We aim to address these challenges by adversarially adapting the backdoor trigger make it persistent in the global training dynamics. Our primary objective is to optimize the backdoor trigger in a way that allows it survive even in the scenario where the global model is trained directly unlearn the backdoor [55 36 To better motivate our method, we first discuss the limitations of existing state-of-the-arr backdoor attacks federated learning.
Limitation of existing works: In recent state-of-the -het--ar works 9 10,, the attacker optimizes the backdoor trigger to maximize its attack effectiveness and applies regularization techniques to bypass server-side defense mechanisms. Formally, given the trigger pattern 5 and an arbitrary input x, the input stamped with the backdoor trigger can be denoted as x @ 6, which is called the backdoored input. Suppose the target class is ÿ. Since the attacker has access to the training dataset of a compromised client i, the backdoor trigger 8 can be optimized using the following objective:
8 argmin E(x.))>D,
[C(x e 8, g; 0.)]
where 0, represents the global model weights in the t-th communication round, and L is the classifi- cation loss function such as cross-entropy loss. To conduct backdoor attack locally, the attacker randomly samples a small set of inputs D/ from the local training set D., and poisons inputs in D with trigger stamped. The attacker then injects a backdoor into the local model by optimizing the local model on the partially poisoned local training set with regularization to limit the gap between

---

the local and global model, i.c., |0 0, While the regularization term helps bypass server-side defenses, it greatly limits the backdoor effectiveness, as only considers the current global model 0, and thus fails to adapt to future global updates.
As illustrated in Figure 1 observe that such backdoor attack federated learning (e.g.. CerP 9 is highly effective on the local model, suggested by high local attack success rate (ASR). However, due to the ignorance of global dynamics, they cannot achieve similar effectiveness when transferred to the global model, resulting in a low ASR on the global model. Our method A3FL aims to bridge the local- goealgglobal gap in existing approaches to make the backdoor persistent when transferred the global model thus achieving advanced attack performance. In particular, we introduce adversarial adaptation loss that makes the backdoor persistent global training dynamics.
To address the challenge introduced by the global model dynamics in federated learning, we propose the adversarial adaptation loss. As the attacker cannot directly control how the global model is updated as federated learning proceeds, its backdoor performance can be significantly impacted when transferred to the global model, especially when only small number of clients are compromised by the attacker or defense strategies are deployed. For instance, local model updates from benign clients can re-calibrate the global model to indirectly mitigate the influence of the backdoored updates from the compromised clients; a defense strategy can also be deployed by the server mitigate the backdoor. To make the backdoor survive such challenging scenarios, our intuition is that, if an attacker could anticipate the future dynamics of the global model, the backdoor trigger would be better optimized to adapt to global dynamics.
Adversarial adaptation loss:
However, global model dynamics are hard to predict because 1) at each communication round, all selected clients contribute to the global model but the attacker cannot access the private training datasets from benign clients and thus cannot predict their local model updates, and 2) the attacker does not know how local model updates are aggregated to obtain the global model and is not aware of possible defense strategies adopted by the server. As directly predicting the exact global model dynamics is challenging, we instead require the attacker foresee and survive the scenario where the global model is trained to directly unlearn the backdoor. In this paper we consider backdoor unlearning proposed in prior backdoor defenses 35 36)
Specifically, starting from current global model 0G, we foresee an adversarially crafted global model 02 that can minimize the impact of the backdoor. We adopt adversarial training-like method to obtain 0/: the attacker can use the generated backdoor trigger to simulate the unlearning of the backdoor in the global model. The trigger is then optimized simultaneously backdoor the current global model 0g and the adversarially adapted global model O'. Formally, the adversarially adaptive backdoor attack (A3FL) can be formulated as the following optimization problem:
8" argmin (x [(x 8, 0.) + XC(x @ 0.
(3)
s.t. 6 = argmin CemminExxx.aa Exx,) [(x + 0)
where 0 is initialized with current global model weights 0,; 0 is the optimized adversarial global model which aims to correctly classify the backdoored inputs as their ground-truth label to unlearn the backdoor. In trigger optimization, X hyperparameter balancing the backdoor effect on the current global model 0 and the adversarial one 0', such that the local-global gap is bridged when the locally optimized trigger is transferred to the global model (after server-side aggregation//deeenss)) Note that attacking the adversarial model is an adaptation or approximation of global dynamics, as in practice the server cannot directly access and unlearn the backdoor trigger to obtain such an adversarial model.
Algorithm of A3FL: We depict the workflow of A3FL compromising client in Algorithm [ At the t-th communication round, the client is selected by the server and receives the current global model O,. Lines 4-8 optimize the trigger based on the current and the adversarial global model using cross-entropy loss Lee. The adversarial global model is initialized by the global model weights in Line 1, and updated in Line 10. Lines 12- train the local model on the poisoned dataset and upload local updates to the server.
5

---

Algorithm 1: The workflow of A3FL compromising a client
Input: 0,, D; V, K, K trggers 01,022,
1: 0: =0=
2: for ] =j=1 K do
3: Sample a batch of training data B from D;
4: for k 1 Kiggeer do
5: // Optimiize trigger pattern ö following Equation 3
6:
= |8| Exe (x @ 6, 5: 01) Xxe @ 6. 0.))
7:
8+ x 8 VoL
8:
end for
9:
I/ Opptimizze adversarial global model weights 0 following Equation
10:
8 - 02 Vo [(x))) Lee @ 6, y: 6%)
end for
12:
Poison local dataset with 8 and update local model to obtain 0/
13: A/t1 0
Upload to the server
4 Experiments
4.1 Experimental Setup
Datasets: We evaluate A3FL on three uidely-used benchmark datasets: FEMNIST 43 CIFAR- 10 [[5), and TinyImageNer [16]. The FEMNIST dataset consists of 80,5263 images shaped x8828 distributed across 10 classes. The CIFAR-10 dataset consists of 50,000 training images and 10,000 testing images that are uniformly distributed across 10 classes, with each image having a size of 32x32 pixels. The ImnylmaeeNet dataset contains 100,000 training images and 20,000 testing images that are uniformly distributed across 200 classes, where each image has a size of 64 pixels.
Federated learning setup: By default, we set the number of clients N = 100. At each communica- tion round, the server randomly selects
10 clients to contribute to the global model. The global model architecture is ResNet-18 18 [44]. We assume non-i.i.d data distribution with concentration parameter of 0.9 following previous works [12 10 9. We evaluate the impact of data heterogeneity by adjusting the value of h in Appendix B.6 Each selected client trains the local model for 2 epochs using SGD optimizer with a learning rate of 0.01. The FL training process continues for 2,000 communication rounds.
We assume that the attacker compromises P clients among all N clients. All compromised clients are only allowed to attack in limited communication rounds called attack window. By default, the attack window starts the 1,900th communication round and ends at the 2,000th communication round. We also discuss the impact of the attack window in Appendix B.7 When a compromised client selected by the server during the attack window, will upload poisoned updates trying to inject the backdoor. We adjust the number of compromised clients P E [1, 20) to comprehensively evaluate the performance of each attack. Different from previous works [T] in evaluation compromised clients are selected randomly to simulate the practical scenario. Each compromised client poisons 25% of the local training dataset and trains the local model on the partially poisoned dataset with the same parameter settings as benign clients unless otherwise mentioned. By default, the trigger is designed as square at the upper left corner of the input images. We use the same trigger design for all baseline attacks to ensure the same level of data stealthiness for a fair comparison. We summarize the details of each attack in Appendix A.2 different trigger designs of DBA in Appendix B.9
Attack setup:
We also discuss
A3FL setup: By default, compromised clients optimize the trigger using Projected Gradient Descent (PGD) 45) with a step size of 0.01. The adversarial global model is optimized using SGD with a learning rate of 0.01. In practice, we set the balancing coefficient À = Aosim 0', 0z), where sim O6 0.) denotes the cosine similarity between 0' and 0,. We use similarity to automatically adjust the focus to the adversarial global model: the adversarial global model is similar to the current global model, it will be assigned higher weight; otherwise, the adversarial global model is assigned

---

Table 1: A3FL maintains the utility of the global model on CIFAR-10. 10.
Defense
FedAvg
RE.
Median
DSiight
Bulyan
Krum
Shed
CRHI.
FedDD
FedRAD
ACC(%)
92.29
92.57
92.21
65.59
91.79
39.57
84.56
92.60
87.40
87.71
37.58
65.89
BAC(%)
92.44
92.61
92.26
65.53
91.79
39.92
84.41
92.70
87.35
87.60
40.09
65.61
lower weight. We the similarity to control the strength of adversarial training, since the backdoor could be fully unlearned if the adversarial global model is aggressively optimized, which makes difficult to optimize the first term in Equation 3 In adversarial scenarios, it is important balance the strengths of both sides to achieve better performance, which has been well studied in previous works in adversarial generation 46 47). When there are multiple compromised clients in S,, the backdoor trigger is optimized on one randomly selected compromised client, and all compromised clients use this same trigger. We also discuss the parameter setting of A3FL in experiments.
Compared attack baselines: We compare our A3FL to four representative or state-of-the-art backdoor attacks to FL: Neurotoxin (12). DBA M.. CerP [9), and F3BA [TO]. We discuss these baselines in Section 2 and also provide an in-detail introduction in Appendix A.2 including specific
hyperparameter settings and trigger design of each baseline.
Compared defense baselines: We evaluate A3FL under 13 state-of-the or representative
federated learning defenses: FedAvg (2) Median (22) Norm Clipping [17). DP (17). Robust Learning
Rate 18]. Deepsight (20). Bulyan [23]. FedDF (24). FedRAD (25). Krum 221. SparseFed [26).
FLAME 27,, and CRFL 19 We summarize the details of each defense in Appendix A.3
we use accuracy & backdoor accuracy (ACC& & BAC), attack success rate (ASR), and lifespan to comprehensively evaluate A3FL.
Evaluation metrics:: Following previous works 10 12
ACC & BAC:
We define ACC as the accuracy of the benign global model on clean testing inputs without any attacks, and BAC as the accuracy of the backdoored global model on clean testing inputs when the attacker compromises a part of the clients to attack the global model. Given the dynamic nature of the global model, we report the mean value of ACC and BAC. BAC close to ACC means that the evaluated attack causes little or no impact on the global model utility. A smaller gap between ACC and BAC indicates that the evaluated attack has higher utility stealthiness.
ASR:
We embed backdoor trigger to each input in the testing set. ASR the fraction of trigger- embedded testing inputs that are successfully misclassified as the target class by the global model. In particular, the global model is dynamic in FL, resulting in an unstable ASR. Therefore, we use the average value of ASR over the last 10 communication rounds in the attack windows to demonstrate the attack performances. A high ASR indicates that the attack is effective.
The lifespan of a backdoor is defined as the period during which the backdoor keeps effective. The lifespan of backdoor starts at the end of the attack window and ends when the ASR decreases to less than a chosen threshold. Following previous works [12], we set the threshold 50%. A long lifespan demonstrates that the backdoor durable, which means the backdoor remains effective in the global model long after the attack ends. When we evaluate the lifespan of attacks, extend the FL. training process 3,000 communication rounds.
Lifespan:
4.2 Experimental Results
A3FL preserves the utility of the global model: To verify whether A3FL impacts the utility of global models, we compared their ACCs to BACs. The experimental results on CIFAR-10 are shown in Table [ where NC denotes Norm Clipping, DSight represents Deepsight, and SFed represents SparseFed. Observe that the maximum degradation in accuracy of the global model caused by A3FL only 0.28%. Therefore, we can conclude that A3FL preserves the utility of the global model during the attack, indicating that our approach is stealthy and difficult to detect. Similar results were observed in the experiments on Tinylmagenet, which can be found in Appendix
The attack performances of A3FL and baselines on defenses designed for FL. backdoors are presented in Figure The experimental results demonstrate that achieves higher attack success rates (ASRs) than other baselines. For example, when the defense is Norm
A3FL achieves higher ASRs:
7

---

100
100
100
ASR (36)
(96)
ASR (96)
40-
Neurotoxin
40
40-
F3BA
A3FL
12
10
12
10
Number of compromised clients (P)
Number compromised clients (P)
Number of compromised clients (P)
(a) FedAvg
(b) Norm Clipping
(c) RobustLR
80
80
ASR (%)
ASR (36
60-
60-
60
ASR
20
20
20
12
10
20
1.2
5
10
20
12
5
10
20
Number of compromised clients (P)
Number of compromised clients
Number of compromised clients
(d) CRFL
(e) Deepsight
(f) DP
Figure 2: Comparing performances of different attacks CIFAR-10.
Clipping and only client is compromised, A3FL achieves ASR of 99.75%, while other baselines can only achieve a maximum ASR 13.9%. Other attack baselines achieve a comparable ASR to A3FL only when the number of compromised clients significantly increases. For instance, when the defense is CRFL, F3BA cannot achieve a comparable ASR to A3FL until 10 clients are compromised. We have similar observations other defenses and datasets, which be found in Figure 8 9 in Appendix B.2
We note that CRFL assigns certified radio to each sample and makes that samples inside the certified radio would have the same prediction. This is achieved by first clipping the updates A and then adding Gaussian noise (O, o²I) to A During the inference stage, CRFL adopts majority voting to achieve certified robustness. The strength of CRFL is controlled by the value of a. We discuss the performance of CRFL under different values of in Appendix B.5
A3FL has longer lifespan: We evaluate the durability of attacks by comparing their lifes- pans. Recall that the attack starts the 1,900th communication round and ends at the 2,000th
DBA
Neu rotuxin
CerP
FJBA
A3FL
communication round. Figure shows the attack success rate against communication rounds when the defense is Norm Clipping and clients are compromised. As we can observe, A3FL has a significantly longer lifespan than other baseline attacks. A3FL still has an ASR of more than 80% the end, indicating lifespan of over 1,000 rounds. In contrast, the ASR of all other baseline attacks drops below 50% quickly. We show more results on other defenses in Appendix B.3 and similar phenomenon is observed. These experi- mental results suggest that A3FL is more durable than other baseline attacks, and challenge the consensus that backdoors in FL quickly vanish after the attack ends.
ASR (%)
60-
1900
2000
2200
2400
2600
2800
3000
Communication round
Figure 3: A3FL has longer lifespan. The verti- cal dotted lines denote the end of the lifespans of each attack when the ASR of the backdoor drops below 50%. The dotted line at the 100th commu- nication round denotes the end of all attacks.

---

231
8
8 8
Glbbul A3A
Gbbbal
3:bu A38
20
100
Communcation rund
Commun round
Commurkatiin HOUNS
Commurication ound
Commurcction round
(a) A3FL
(b) F3BA
(c) CerP
(d) DBA
(c) Neurotoxin
Figure Compare local ASR to global ASR.
1.0
100
100
100
80-
A00 156)
60-
90-
52.5
10.00.0
5x5
5x5
-0.1
02.
Bdhl
10x10
20-
Bcch
10x10
Ag-1
As -0.5
60-
1
977 12 5 30
25
12 5
30
26
99
12
36
20
70 12
10
20
Namber of comoromizzd clents IP)
Numbar compoemized ctents (P)
Namber comoromized exiets
Number of compromisad clants
(a) Norm Clipping
(b) CRFL
(a) Norm Clipping
(b) CRFL
Figure 5: The impact of trigger size on the attack performance.
Figure 6: The impact of on the attack perfor- mance.
4.3 Analysis and Ablation Study
A3FL achieves higher ASR when transferred to the global model: As discussed in Section 3 A3FL. achieves higher attack performance by optimizing the trigger and making the backdoor persis- tent within the dynamic global model. To verify our intuition, we conducted empirical experiments in which we recorded the Attack Success Rate (ASR) on the local model (local ASR) and the ASR on the global model after aggregation (global ASR). For the experiments, we used FedAvg as the default defense and included five compromised clients among all clients.
The results presented in Figure demonstrate that A3FL can maintain a higher ASR when transferred to the global model. While all attacks can achieve high ASR (= 100%) locally, only A3FL can also achieve high ASR on the global model after the server aggregates clients' updates, which is supported by the tiny gap between the solid line (global ASR) and the dotted line (local ASR). In contrast, other attacks cannot achieve similarly high ASR on the glohal model as on local models. For instance, F3BA immediately achieves a local ASR of 100% once the attack starts. But it can only achieve less than 20% ASR the global model in the first few communication rounds. F3BA also takes a longer time to achieve 100% ASR on the global model compared to A3FL. This observation holds for other baseline attacks. We further provide case study in Appendix B.8 understand why A3FL outperforms baseline attacks. In the case study, we observe that 1) A3FL has better attack performance than other baseline attacks with comparable attack budget: 2) clients compromised by A3FL are similarly stealthy to other trigger- sngger-optmmization attacks. Overall, our experimental results indicate that A3FL is more effective and persistent attack compared to baseline attacks, which makes it particularly challenging to defend against.
The impact of trigger size: We evaluate the performance of A3FL with a trigger size of 3x3, 5x,, 8 8x8, 10x 10 respectively (the default value is 5x5). 3 the impact of trigger size on A3FL. In general, the attack success rate (ASR) improves the trigger size grows larger. When the defense mechanism is Norm Clipping, we observe that the difference between the best and worst ASR only 1.75%. We also observe larger difference with stronger defenses like CRFL. Additionally, we find that when there are at least 5 compromised clients among all clients, the impact of trigger size on the attack success rate becomes unnoticeable. Therefore, we can conclude that smaller trigger sizes may limit the performance of A3FL only when the defense is strong enough and the number of compromised clients small. Otherwise, varying trigger sizes will not significantly affect the performance of A3FL.
The impact of A: Recall that Apsim(O', 0,). We varied the Xn hyperparameter over a wide range of values to learn the impact of the balancing coefficient on attack performance and record results in F Observe that different Å only slightly impact attack performances with fewer compromised clients. When there are more than 5 compromised clients, the impact of Ag is unnoticcable. For

---

instance, when the defense is Norm Clipping, the gap between the highest ASR and the lowest ASR is merely 0.5%. We can thus conclude that A3FL is insensitive variations in hyperparameter Ag. We further provide an ablation study in Appendix B.4 for more analysis when the adversarial adaptation loss disabled, i.e., Àg 0.
5 Conclusion and Future Work
In this paper; we propose A3FL, an effective and durable backdoor attack to Federated Learning. A3FL adopts adversarial adaption loss to make the injected backdoor persistent in global training dynamics. Our comprehensive experiments demonstrate that A3FL significantly outperforms existing backdoor attacks under different settings. Interesting future directions include: 1) how to build backdoor attacks towards other types of FL, such as vertical FL; 2) how to build better defenses to protect FL from A3FL.
References
[1] Jakub Konečný, H Brendan McMahan, Felix Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv: 1600.0554 2016.
[2] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effie effficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[3] Qiang Yang. Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Trans. Intell. Syst. Technol, 10(2):12:1-12:19 2019.
[4] Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint 181 2018.
[5] David Leroy, Alice Coucke, Thibaut Lavril, Thibault Gisselbrecht, and Joseph Dureau. Feder- ated learning for keyword spotting. In ICASSP 2019-2019 IEEE international conference on acousties, speech and signal processing (ICASSP), pages 6341-6345. IEEE, 2019.
[6] Viraaji Mothukuri, Reza M Parizi, Seyedamin Pouriyeh, Yan Huang, Ali Dehghantanha, and Gautam Srivastava. A survey security and privacy of federated learning. Future Generation Computer Systems, 15:619-640,, 2021.
[7] KIMBERLY Powell. Nvidia clara federated learning to deliver ai to hospitals while protecting patient data. Accessed: Dec, 1, 2019.
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pages 2938-2948. PMLR, 2020.
[9] Xiaoting Lyu, Yufei Han, Wei Wang, Jingkai Liu, Bin Wang, Jiqiang Liu, and Xiangliang Zhang. Poisoning with cerberus: Stealthy and colluded backdoor attack against federated learning. 2023.
[10] Pei Fang and Jinghui Chen. On the vulnerability of backdoor defenses for federated learning. arXiv preprint arXiv:2301.08170, 2023.
[11] Chulin Xie, Keli Huang. Pin- Yu Chen, and Bo Li. Dba: Distributed backdoor attacks against federated learning. In International conference on learning representations, 2020.
[12] Zhengming Zhang, Ashwinee Panda, Linyue Song, Yaoqing Yang. Michael Mahoney, Prateek Mittal, Ramchandran Kannan, and Joseph Gonzalez. Neurotoxin: Durable backdoors in federated learning. In International Conference on Machine Learning, pages 26429-26446 PMLR, 2022.
10

---

[13] Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy-yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos. Attack of the tails: Yes, you really can backdoor federated learning. Advances in Neural Information Processing Systems, 13:16070-66084, 2020.
[14] Gilad Baruch, Moran Baruch, and Yoay Goldberg. A little is enough: Circumventing defenses for distributed learning. Advances in Neural Information Processing Systems, 32, 2019.
[15] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[16] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS CS23IN, 7(7):3, 2015.
[17] Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and Brendan McMahan. Can you really backdoor federated learning? arXiv preprint arXiv: 1.07963, 2019.
[18] Mustafa Safa Ozdayi, Murat Kantarcioglu, and Yulia Gel. Defending against backdoors in federated learning with robust learning rate. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9268 09669927,, 2021.
[19) Chulin Xie, Minghao Chen, Pin-Yo Chen, and Bo Li. Crft: Certifiably robust federated learning against backdoor attacks. In International Conference on Machine Learning, pages 11372-11382. PMLR, 2021.
[20] Phillip Rieger. Thien Duc Nguyen, Markus Miettinen, and Ahmad-Rees Sadeghi. Deepsight: Mitigating backdoor attacks in federated learning through deep model inspection. arXiv preprint 20X1002001.00763 2022.
[21] Peva Blanchard, EI Mahdi EI Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learn- ing with adversaries: Byzantine tolerant gradient descent. Advances in neural information processing systems, 30, 2017.
[22] Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantine- distributed learning: Towards optimal statistical rates. In International Conference on Machine Learning, pages 5650-5659. PMLR, 2018.
[23] El Mahdi El Mhamdi, Rachid Guerraoui, and Sébastien Rouault. The hidden vulnerability of distributed learning in byzantium. arXiv preprint arXiv:1802.00997, 2018.
[24] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351-2366 2020.
[25] Stefán Páll Sturluson, Samuel Trew, Luis Muñoz-Gonzálee Matei Grama, Jonathan Passerat- Palmbach, Daniel Rueckert, and Amir Alansary. Fedrad: Federated robust adaptive distillation. arXiv preprint arXiv:2 /2.0:4052 2021.
[26] Ashwinee Panda, Saeed Mahloujifar, Arjun Nitin Bhagoji, Supriyo Chakraborty, and Prateek Mittal. Sparsefed: Mitigating model poisoning attacks in federated learning with sparsification. In International Conference on Artificial Intelligence and Statistics, pages 7587-7624. PMLR, 2022.
[27] Thien Duc Nguyen, Phillip Rieger, Roberta De Viti. Huili Chen, Bjöm Brandenburg, Hossein Yalame, Helen Möllering, Hossein Fereidoomi, Samuel Marchal, Markus Miettinen, et al. (FLAME): Taming backdoors in federated learning. In 31st USENIX Security Symposium (USENIX Security 22), pages 1415-1432, 2022.
[28] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference Machine Learning, pages 1132-5143. PMLR, 2020.
[29] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Chal- lenges, methods, and future directions. IEEE signal processing magazine, 37(3):50-60, 2020.
11

---

[30] Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351-2363, 2020.
[31] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. Advances in neural information processing systems, 33:7611-7622 2020.
[32] Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003.00295, 2020.
[33] Clement Fung, Chris JM Yoon, and Ivan Beschastnikh. Mitigating sybils in federated learning poisoning. arXiv preprint
04866, 2018.
[34] Krishna Pillutla, Sham M Kakade, and Zaid Harchaoui. Robust aggregation for federated learning. IEEE Transactions on Signal Processing, 10:1142- 2022.
[35] Yi Zeng, Si Chen, Won Park, Z Morley Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. arXiv preprint arXiv:2/ 10.03733, 2021.
[36] Bolun Wang. Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP). pages 707-723. IEEE, 2019.
[37] Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette- Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. 2020.
[38] Varun Gupta, Christopher Jung, Seth Neel, Aaron Roth, Saeed Waites. Adaptive machine unlearning, 2021.
and Chris
[39] Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In
2015 IEEE Symposium Security and Privacy, SP 2015, San Jose, CA, USA, May 17-21, 2015, pages 463-480. IEEE Computer Society, 2015.
[40] Xiaoyu Cao, Minghong Fang, Jia Liu, and Neil Zhenqiang Gong. Fltrust: Byzantinc-robust federated learning via trust bootstrapping. arXiv preprint arXiv:2012. 13995, 2020.
[41] Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. Provably secure federated learning against malicious clients. In Proceedings of the AAAI conference artificial intelligence, volume 35, pages
2021.
[42] Virat Shejwalkar, Amir Houmansadr, Peter Kairouz, and Daniel Ramage. Back to the drawing board: A critical evaluation of poisoning attacks on production federated learning. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1354-1371. IEEE, 2022.
[43] Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H. Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings, 2019.
[44] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[45] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:/706.00 700.006863 2017.
[46] Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. On convergence and stability of gans. arXiv preprint arXiv: 2017.
[47] Siyuan Xu and Minghui Zhu. Efficient gradient approximation method for constrained bilevel optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 37(10): 12517, 2023.
12

---

[48] Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A bensity-based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pages 226-231, 1996.
[49] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1144.1555, 1556, 2014.
13

---

(a) FLTrojan
(b) F3BA
(c) CerP
(d) Neurotoxin
(e) DBA
Figure 7: Trigger patterns of evaluated attacks on FedAvg, with P 2 compromised clients.
A
Additional Experiment Details
A.1 Experimental Setup in Figure 1
The preliminary experiment in Figure the same experimental setup as described in Section 4.1 In particular, We use FedAvg [2] as the server-side aggregation rule. We set the number of compromised clients P 1 in the preliminary experiment. We denote the attack success rate the global model as global ASR. We further denote the ASR on the local model after local training as the local ASR. When the compromised client is selected by the server, we calculate and update the local ASR after the compromised client optimizes the backdoor trigger and trains its local model on the poisoned local training dataset.
A.2 Details of Attacks
A3FL: A3FL formulates the trigger optimization as bi-level optimization problem. A3FL jointly optimizes the adversarial model fe; with the trigger pattern A. A3FL optimizes the adversarial model using SGD with learning rate of 0.01, a momentum of 0.9, and weight decay of 0.0005. A3FL updates the trigger pattern using PGD with a step size of 0.01 until convergence. We show the trigger pattern of A3FL in Figure 7a
F3BA 10 F3BA directly manipulates a part of local model weights to inject the backdoor via sign flipping. F3BA further jointly optimizes the trigger pattern and the local model weights maximize the difference between latent representations of clean and backdoored samples, thus achieving higher attack performance. The trigger of F3BA squared patch. We show the trigger pattern of F3BA in Figure 7b
CerP 9 CerP jointly optimizes the trigger pattern and the local model weights improve the backdoor effectiveness. Furthermore, CerP aims to improve the backdoor stealthiness by adopting L2-norm regularization to limit the difference between local model weights and global model weights. Therefore CerP can tune the local model to fit the biases the local model weights. The trigger of CerP is shown in Figure 7c
data without inducing large
Neurotoxin only updates unimportant model weights to avoid conflicts with other clean clients. The importance of model weights is determined by the magnitude of their gradients. Model weights with higher gradient in previous rounds are considered to be more important (frequently updated by other clients). Following the settings 112, we only update the last 95% important model weights. Neurotoxin
Neurotoxin 12]:
fixed trigger pattern, as shown in Figure 7d
DBA [II: DBA is a distributed backdoor attack designed to utilize the distributed nature of FL. DBA splits the trigger into different clients. Each client uses different trigger attack the FL system during the training stage. In the inference stage, the attacker uses the joint trigger to activate the injected backdoor. The trigger in [II] was designed as several parallel white lines placed the upper left corner of the input images. This trigger design is not compatible with our attack setting and we can hardly control the attack budget introduced by the trigger following [IT]. Therefore in our implementation, we also use squared patch as the trigger for DBA, as shown in Figure 7 We randomly split the squared patch into four sub-triggers and these sub-triggers are iteratively used during the attack.
14

---

A.3 FL defenses
NC clips clients' updates that are larger than pre-defined threshold. NC can effectively limit clients' behavior to prevent the global model from being overwhelmed by a few clients. By default, we set the threshold to 1.
Norm Clipping (NC) 17]:
DP adds Gaussian noise [N o² clients' updates to perturb carefully crafted malicious updates. Note that this defense is not designed for privacy, so the Gaussian noise is relatively smaller than that adopted in differential privacy. By default, we set 0.002.
(weak) Differential Privacy (DP) 17|:
Robust Learning Rate (RLR) 18] RLR aims to maximize the agreement on updating direction across clients mitigate potential attacks. It is inspired by that the behavior of a compromised client is commonly different from other benign clients. For instance, a compromised client may want to enlarge some model parameters while most benign clients are trying to reduce them. When clients disagree on the updating direction of parameter, RLR flips the learning rate on the parameter to maximize the loss instead.
CRFI. adopts three techniques to mitigate backdoor attacks on FI.. CRFI. first clips clients' updates as Norm Clipping does. In our experiments, we set the clipping threshold as 1. CRFL then adds Gaussian noise ~ N (0, a² I) to clients' updates as DP does. In our experiments, we set 8 = 0.002 and we discuss the impact of a CRFL in Appendix B.5 Finally, CRFL creates several perturbed models by adding independently sampled Gaussian noise to the global model and adopts majority voting for prediction. In our experiments, CRFL creates 5 different perturbed models for prediction at each FL communication round.
CRFL. 19]:
Median [22
Median uses the coordinate- wise median value of updates from all clients to update the global model. Median can effectively exclude clients that upload overwhelming updates. However, the Median tends to heavily degrade the model utility.
Deepsight adopts three different distance matrices to measure the distances between each client. Deepsight then clusters clients according different distance matrices and only accepts clients that are in the same cluster across different matrices. The first distance matrix is smaller when the updates in the last layer from clients are similar. The second distance matrix is the L2 distance between the last layer's weight across each client. The third distance matrix is the L2 distance between the outputs of two local models given a batch of randomly generated input images. Deepsight adopts DBSCAN [4]] to cluster selected clients. Finally, clusters including potentially malicious clients that have a larger distance from other clusters will excluded. In our experiments, we set the batch size of randomly generated inputs to 256.
Deepsight [20]
Bulyan first excludes potentially malicious clients from all selected clients and then uses the coordinate- wise median value of updates from remaining clients to update the global model. In the first step, 2f clients with the highest pairwise Euclidean distances excluded. In the second step, Bulyam picks M - 4f clients from the remaining M - 2 clients that are closest to the median by coordinate. In experiments,
Bulyan 231:
set f 2.
FedDF uses the mean output of all client models the supervisory signal to distill the next round global model. In particular, FedDF firstly aggregates all selected clients (the same as FedAvg) obtain a teacher model. Then the server trains the global model minimize the Kullback Leibler divergence between the logits of the global and teacher model on a set of unlabeled inputs. In our experiments, the learning rate for updating the global model is 0.002 and we train the global model for one epoch at each FL communication round.
FedDF [24]:
FedRAD [25:: FedRAD is an extension of FedDF, which assigns a weight to each client model based on their median scores. These scores indicate the frequency with which the prediction of the client model becomes the median value of predictions from all client models. FedRAD then utilizes weighted model aggregation to produce the next round global model. In our experiments, we also update the global model with a learning rate of 0.002 for one epoch at each FL communication round.
Krum selects clients that have the smallest 1.2 distances to other clients. Only the clients selected by Krum will be used to update the global model. Since Kurm drops most updates from clients, it can achieve strong robustness. However, Krum also affects the accuracy of the model.
Krum 21]:
15

---

Table 2: A3FL maintains the utility of global models on TinylmageNet.
Defense
FedAvg
Median
DDSight
Bulyan
Krum
Shed
CRH..
FedDDF
FedRAD
ACC(%)
$5.45
55.31
55.34
17.12
53.71
11.19
42.87 42.35
$7.39
$3.58
$3.38
25.31
23.12
BAC(%)
55.25
54,98
53.28
20.92
53.44
7.33
57.08
53,45
53.17
24.90
22:57
100
100
80
80
80
ASR (N)
60
ASR (1))
DBA
Neurataxin
CerP
20
20
20-
A3FL
12
5
10
20
12
5
10
20
12
5
10
20
Number of compromised clients (P)
Number compromised clients
Number of compromised clients (P)
(a) Krum
(b) Median
(c) FedRAD
100
100
100
ASR 1%6)
60
196)
60
ASR (96)
60
40
40
40
20
10 Number of compromised clients (P)
12
20
50
12
10
Number compromized clients (P)
Number of compromised clients (P)
(d) FedDF
(e) Bulyan
(l) SparseFed
Figure 8: Comparing performances of different attacks on CIFAR-10.
SparseFed is proposed to mitigate model poisoning attacks in FL. SparseFed aggregates client updates normally but only updates the top-k highest magnitude elements. It is inspired by that attackers commonly move in distinct directions from the majority of clean clients. Therefore the top-k highest magnitude elements involve less poisoned updates from attackers. In our experiments, update the 95%%5% highest magnitude elements.
SparseFed [26]:
FLAME [27] FLAME adopts dynamic clustering, adaptive clipping, and adaptive noising to exclude potentially malicious clients. Following the settings for image classification [27]. we set 3705 and 0.001 controlling the strength of adaptive noising.
B Additional Experimental Results
B.1 A3FL maintains the model utility
We show the accuracy of the global model on Tinylmagenet when the attacker presents (BAD) or not (ACC) in Table 2 In particular, we record the accuracy on clean tasks when no attackers are involved to obtain the accuracy (ACC). We further record the accuracy on clean tasks when there are 20 compromised clients among all clients to obtain the backdoor accuracy (BAC). We set the number of compromised clients P to 20 since more compromised clients are likely to result in a higher decrease in clean accuracy. Therefore if A3FL can maintain the model utility even with 20 compromised clients, we can conclude that A3FL is highly stcalthy. Note that we use the mean value of ACC and BAC in the attack window (between the 1,900th communication round and the 2,000th communication round) to verify the utility of global models since the server continuously updates the global model. Therefore, using the mean accuracy as the measurement standard can accurately reflect the impact of attacks on the model utility, and eliminate randomness.
As shown in Table 2 the accuracy of the global model does not degrade much when attackers are presented. This indicates that A3FL preserves the accuracy of global models so stealthy enough to not be discovered. The differences between ACCs and BACs are within 0.5% in most cases. The highest drop in clean accuracy is observed when the defense mechanism Bulyan. However, Bulyan significantly degrades the model's accuracy to only 1 19.... The low accuracy indicates that the
16

---

100
100
100
ASR (36)
(96)
ASR (96)
40-
Neurotoxin
40
40-
F3BA
A3FL
10
12
10
Number of compromised clients (P)
Number compromised clients (P)
Number of compromized clients (P)
(a) FedAvg
(b) Norm Clipping
(c) RobustLR
100
80
80
ASR (16
60-
60
ASR (%)
60-
20
20
20
12
10
20
12
5
10
20
12
10
20
Number of compromised clients (P)
Number compramised clients
Number of compromised clients (P)
(d) CRFL
(c) Deepsight
(f) DP
100
100
100
00
ASR (K]
ASR (%6)
40
40
40
20
20
20
12
10
20
12
10
Number of compramised clients (P)
Number compramised clients (P)
Number of compramised clients (P)
(g) Krum
(h) Median
(i) FedDF
80
80
80
ASR (%6
60
60
ASR (%)
60
20
20
12
10
5
20
12
Number of compramised clients (P)
clients
Number compramised clients (P)
(j) FedRAD
(k) Bulyan
(I) SparseFed
Figure 9: Comparing performances of different attacks TinylmageNet.
model highly random, so even though A3FL causes the model's accuracy to drop to 7.33%, we cannot solely conclude that A3FL will reduce the model utility. In general, A3FL does not influence the global model utility. We also observe a similar phenomenon on CIFAR-10 10, as shown in Table [
B.2 A3FL achieves higher ASRS
We compare the performance of attacks CIFAR-I0 against defenses that are not designed for backdoor attacks in Figure S Observe that A3FL achieves the highest ASR under most settings. When the defense is Median, A3FL is the only attack that can achieve high ASR (over 80%). We further show the attacker performance of A3FL on TinyImagenet in Figure 9 and we can observe a similar phenomenon.
B.3 A3FL has a longer lifespan
In Figure we show that A3FL. has significantly longer lifespan than other baselines with different defenses applied. For instance, when the defense is RobustLR, A3FL can still achieve an ASR of
17

---

DBA
Neu rotoxtaxin
Cerp
F3BA
A3PL
100
ASR (%)
ASR
20
20
1900
2000
2200
2400
2600
2800
3000
1900
2000
2200
2400
2600
2800
3000
Communication round
Communication round
(a) FedAvg.
(h) RobustLR RR,
100
100
80
80
ASR (9)
ASR (%)
60
40
20
1900
2000
2200
2400
2600
2800
3000
1900
2000
2200
2400
2600
2800
3000
Communication round
Communication round
(c) Deepsight, P
(d) CRFL, P-20
Figure 10: A3FL has a longer lifespan.
100
100-
100
100
80-
60-
AGC- PW1
ASA MM1
1941
p-0.0 001
000000
20-
u-0 0005
0.0.005
h-0.9
0.0.01
p-0001
7
12
5
30
25
12
5
10
20
12
30
20
12
10
20
Number compromased cients
Number comprrmised cients
Namber compromised
Numbbr compramisee
(a) ACC
(b) ASR
(a) Norm Clipping
(b) CRFL
Figure 11: Attack performances against CRFL
Figure 12: Attack performances under different
with different a.
Dirichlet concentration parameters.
62.37% at 1000 rounds after the attack ends. In contrast, the attack success rates of other attacks drop below 50% less than 150 rounds. Note that when we use CRFL, we set the number of compromised clients P 20 since when there only compromised clients, all attacks except A3FL failed to achieve high ASR (see Figure
B.4 Ablation study on component importance
We study the effectiveness of A3FL with or without the adversarial adaptation loss to test the effectiveness of components under FedAvg with P 20 compromised clients among all clients. As shown in Table 3 the adversarial adaptation loss can effectively improve the durability of A3FL. Observe that A3FL 86.65% at 1,000 communication rounds after the attack. In comparison, A3FL without the ad adversarial adaptation loss exhibits ASRs that 4.3 % and 15.64% lower than A3FL these two points.
achieve ASR of 97. 97.66% at 500 communication rounds after the attack and
Table 3: Effect of different components A3FL.
ASR(%) Rounds after attack +
0
500
1000
A3FL without adversarial adaptation A3FL
100.0
93.35
69.01
100.0
97.66
84.65
18

---

100
100
100
ASR (36)
(96
ASR (36)
40-
Neurotoxin
40
40-
F3BA
FLTrojan
12
10
12
10
Number of compromised clients (P)
Number compromized clients (P)
Number of compromised clients (P)
(a) Norm Clipping
(b) Krum
(c) CRFL
Figure 13: Attack performances when the attack starts at the first communication round.
165
Selided
105
Seacid
120-
Seactedd
100
Sa
100
KOR
ASR
30
60
50
103
20
60
100
25
60
53
80
200
23
40
50
80
100
20
60
50
130
(a) A3FL
(b) F3BA
(c) CerP
(d) DBA
Neurotoxin
Figure 14: ASRs against Krum.
B.5
Impact of a on CREL Effectiveness
Figure 11 shows the ACC and ASR when applying CRFL with different Observe that the increases, CRFL can achieve better robustness, indicated by lower ASR. However, the ACC of the global model also drops from 90.25% to 67.33% rapidly, is unacceptable. Furthermore, when there are more compromised clients, A3FL can still achieve high ASR even with large a 0.01. We can thus conclude that CRFL can not sufficiently mitigate A3FL with different a.
increases from 0.001 to 0.01, which
B.6 Impact of Data Heterogeneity
We adjust the Dirichlet concentration parameter h = 0.09, 0.9, study whether data heterogeneity influences the performance of A3FL. As shown in Figure [2] A3FL can achieve high ASR regardless of different h. When the defense Norm Clipping and h 0.09, A3FL achieves lower ASR. This can be explained by that smaller h indicates more i data distribution. Therefore, the local training set held by the attacker far from the global data distribution, which increases the difficulty of injecting the backdoor. However, the attack success rate is still high (over 60%) and quickly increases as the number of compromised clients increases.
B.7 The impact of attack window
We evaluate A3FL against baseline attacks when the attack window starts the first communication round and ends at the 100th communication round. As shown in Figure [3] A3FL can still remarkably outperform other baseline attacks. For instance, when the defense mechanism Norm Clipping and there are compromised clients, the gaps of ASR between A3FL and other baseline attacks are at least 62.4%, which even larger than the gap under default settings. However, we also observe that when the attack starts from the first communication round and there are only a few compromised clients (1 or 2), ASRs of all attacks decrease in comparison to ASRs under default settings. This can be explained by that at the beginning of the training process, the global model changes a lot so the backdoor is easily erased when there are only few compromised clients. We also observe that when the attack starts from scratch, all attacks fail to have satisfying lifespan since the model is far from convergence the 100-th communication round. Therefore, our evaluation of lifespan conducted following the configuration provided by Neurotoxin [1]
19

---

100
(b) Trigger size ASR (56)
20
DBA
12
20
Number of compromised clients (P)
(a) ASR
(b) Trigger Size
(c) Trigger Gap
(d) Trigger Location
Figure 15: Attack performances of DBA using original trigger design. (a) DBA-bar denotes DBA attack with the original trigger design proposed in [1]. in which the trigger consists of four white bars. While DBA denotes the DBA attack with the trigger designed as red square. refers to the length of each white bar. (c) Trigger gap [Gap,. Gap,) refers to the distance between each bar. (d) Trigger location (Shift,, Shifty) represents the distance from the trigger to the edge of the image.
B.8 Case study on Krum
We perform case study on Krum to gain insight into why A3FL outperforms other baselines. In Figure [4 we record the ASRs and put a notation on the line if Krum selects an attacker- compromised client at that round. Recall that Krum selects one client at each round and only uses the selected client updates to update the global model. Therefore, the chance that a compromised client is selected by the server increases if the backdoor is stealthy. We have the following observations: 1) fixed-trigger attacks are more frequently selected by the server, while trigger-optimization attacks selected twice only; 2) fixed-trigger attacks achieve lower ASR even if selected by the server. However, observe that once selected, A3FL quickly achieve 100% ASR, which is because A3FL can maintain higher ASR when transferred to the global model as stated above. A3FL is also durable after being selected, leading to higher ASR at the end of the attack. In comparison, F3BA is selected on the 26th round and achieves 80% ASR. But the ASR quickly drops after that. CerP is also selected twice, but it cannot achieve as high ASR as A3FL and F3BA do, which is caused by the strict regularization on the local model bias. In addition, the ASR of CerP also drops quickly when the compromised clients are not selected by the server.
B.9 The impact of DBA trigger pattern
In our experiments, we set the trigger pattern of DBA to be a red square at the upper left corner. However, in (II), the trigger is designed four white lines. We, therefore, discuss the performance of DBA when using the original trigger design. The original trigger design of DBA is determined by three hyperparameters: trigger size (TS), trigger gap (TG), and trigger location (TL.). In particular, the trigger gap consists of a horizontal gap (Gap,) and a vertical gap (Gap,) The trigger location consists horizontal shift (Shift,) and a vertical shift (Shift,)) We explain these hyperparameters in we set (TS, TG, TL}
Figure 15b 15c and 15d respectivvl.... Following the default settings in [ (4, (6, 6), (.,
We compare the attack performance of DBA and DBA-bar (DBA with original trigger design) in Figure 15a Observe that with the original trigger design, DBA-bar achieves even lower ASR. This phenomenon supports that the default trigger design in our experiments does not degrade the attack performance of DBA. In contrast, DBA design.
achieve higher ASR without the original trigger
B.10 Transferability of A3FL under different settings
We further evaluate the attack performance of A3FL on more datasets, defenses, and model archi-
tectures. In particular, we record the performance of A3FL on FLAME 27 in Figure T6a We also
evaluate A3FL on other model architectures [49] and datasets 43 Figure 16b A3FL can always outperform baseline attacks under different settings.
16c Observe that
20

---

100
100
100
ASR (36)
(90)
ASR (96)
40-
Neurotoxin
40
40-
F3BA
A3FL
12
12
10
Number of compromised clients (P)
Number compromized clients (P)
Number of compromised clients (P)
(a) FLAME
(b) VGG-16
(c) FEMNIST
Figure 16: Attack performances A3FL under different settings.
B.11 Discussion Ethical Implications
We admit that the discovery of new backdoor attack in federated learning results in potential ethical implications. A3FL mainly focuses on image classification, which can be deployed in security- sensitive applications such as human face recognition. However, discovering new backdoor attack and mitigating its threats is necessary to improve the robustness of the federated learning paradigm. Safeguarding the integrity and ethical dimensions of federated learning is crucial to ensure the best interests of individuals and society. We believe that future work can eliminate the threat of proposed attacks, and it is important to focus research on FL. defenses.
21


Reviews:
[{"review_id": "wDW0UypPOI", "reviewer": "Reviewer_iLct", "summary": "This paper presents a new backdoor attack method termed A3FL to address some limitations of existing predetermined and fixed backdoor attack methods. The proposed method can adversarially adapt to the dynamic global model so that when transferring to the global model, the local-optimized trigger will be affected very little. The method is benchmarked against and performs at par with or better than a number of state-of-the-art methods.", "questions": "Please refer to the \"weakness\" part.", "limitations": "No apparent limitations were found.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents an adversarially adaptive backdoor attacks to Federated Learning.\n- A3FL can alleviate the problem of suboptimal attack performance caused by existing work ignoring the global training dynamics.\n- The method is benchmarked against relevant recent work.", "weaknesses": "- In the abstract, it would be better to give the full name of A3FL.\n- It is recommended to discuss the limitations of the proposed method in order to help other scholars improve it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## reviewer iLct\nWe thank the reviewer for the positive feedback and constructive comments on our work. \n\n>Q1: In the abstract, it would be better to give the full name of A3FL.\n\nFollowing your suggestion, to improve the clarity of our paper, we will include the full name of A3FL in the abstract,in the updated version.\n\n> Q2: It is recommended to discuss the limitations of the proposed method in order to help other scholars improve it.\n\nWe thank the reviewer for insightful comment. Currently A3FL is conducted on the image tasks, and it would be interesting to the robustness of FL in other scenarios, such as large language models. What's more, it would be interesting to explore the application of A3FL in other FL scenarios, such as vertical FL. We will discuss the limitations in the future work section."}, {"review_id": "uEOClpv6CL", "reviewer": "Reviewer_i2mP", "summary": "This work proposes a better backdoor attack on Federated Learning. Existing FL backdoor attacks do not consider the global training dynamics, resulting in limited backdoor attack performance. As the true global training dynamics are impossible to know, this work proposes a method to regularize backdoor training using the worst-case global training dynamics as guidance. The worst-case scenario represents a strong protector who tries to unlearn the exact backdoor pattern, and the proposed backdoor attack greatly benefits from adversarially adapting to this worst case. The paper demonstrates its better performance against 12 existing defense approaches and consistently outperforms state-of-the-art (SOTA) backdoor attacks by a large margin (10x), especially when there is only a small number of attackers.", "questions": "1. Can you further address the weaknesses and limitations?\n2. Can you provide some directions regarding potential defense strategies?", "limitations": "1. The trigger pattern is limited to a rectangular shape. It would be interesting to explore the adoption of irregular shapes in the proposed attack.\n2. More datasets and architectures can be considered.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The method successfully unlearns the backdoor and learns a strong backdoor trigger to adapt (adversarially) to such global learning objectives.\n2. The design is smart in automatically tuning the lambda, which uses similarity to adjust. This design considers potential defenses and explains why the proposed method works well against them. Additionally, the tuning of the base lambda is not sensitive, as shown in Fig 6.\n3. The proposed method has been shown to be effective against 12 existing defense approaches and consistently outperforms state-of-the-art works by a large margin (10x).\n4. There is no need for strict regularization, and it is harmless to utility.\n5. It requires a smaller attack budget in terms of the number of attackers and the number of rounds in poisoning.\n6. A comprehensive evaluation is done on utility, ASR, and lifespan.\n7. The achieved better transferability of the backdoor to the global model is well analyzed.", "weaknesses": "1. Different from previous state-of-the-art works, this work does not directly consider potential defenses (i.e., detection-based methods), while still demonstrating better performance against them. The reason for this is not well justified.\n2. The impact of the number of available training data on the attack performance is not clear.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## reviewer i2mP\nWe appreciate the positive comments on our paper and insightful suggestion for further improvements from the reviewer.\n\n> Q1: Different from previous state-of-the-art works, this work does not directly consider potential defenses (i.e., detection-based methods), while still demonstrating better performance against them. The reason for this is not well justified.\n\nWe are sorry for the confusion. We provide two reasons in terms of method design and empirical observation.\n\nIn the method design, while we do not explicitly consider any specific defense, we design the adversarial adaptation loss in Eq. (3) to enables the attacker to foresee and survive even the worst-case scenario where the global model is trained to directly unlearn the backdoor (which is imaginary since in practice the server does not know the backdoor trigger, and thus it is considered to be stronger than any existing defense). Therefore, if the attacker could survive this worst-case scenario, it is not strange that it can survive other defenses.\n\nAlso from the empircal side, in Appendix B.8, we conducted a case study on Krum and looked into the reason why A3FL outperforms baselines. We recorded the ASRs corresponding to the training rounds, and highlight rounds that an attacker-compromised client is selected by the server. We have the following observation from Figure 14:\n* A3FL-compromised clients are not more frequently selected by the server.\n* Once selected, A3FL can achieve higher and more persistent attack ASR than any other baseline attacks, since A3FL can maintain higher ASR when transferred to the global model. \n\nBoth the model design and empirical observation should be able to explain why A3FL can still achieve high attack performance on detection-based methods. \n\n> Q2: The impact of the number of available training data on the attack performance is not clear.\n\nWe thank the reviewer for the insightful comment. To study the impact, we introduce **data resizing factor $\\gamma$**, and assume that each client has only $1/\\gamma$ private dataset compared to the default setting. We vary the value of $\\gamma$ and record ASRs on FedAvg corresponding to the number of compromised clients (P). As observed in **Table 1 in the attached PDF**, the ASR is significantly impacted only when there is merely 1 compromised client and $\\gamma$ is larger than 8. In other cases we do not observe obvious drop in ASR. Therefore we can conclude that attack performance is not sensitive w.r.t. the number of available training data.\n\n\n> Q3: Can you provide some directions regarding potential defense strategies?\n\nWe thank the reviewer for the interesting question. A3FL is based on a typical FL setting where there is only one global model and currently we didn't find a perfect defense strategy yet. However, in some non-typical FL settings (e.g., if the server can maintain multiple global models trained by sampled clients and aggregate these models for evaluation), the chance for  attacker to successfully inject the trigger can be lower especially when the number of compromised clients is limited. However, this potential defense strategy could be much more computational expensive in comparison to typical FL algorithms.\n\n> Q4: The trigger pattern is limited to a rectangular shape. It would be interesting to explore the adoption of irregular shapes in the proposed attack.\n\nThank you for your insightful suggestions. In this paper, we adopt the same trigger shape for each evaluated attack to ensure fair comparison, which is also well acknowledged in previous works[4][5]. The idea of A3FL does not rely on trigger shape, thus should be direclty applicable to other trigger designs. We will consider backdoor triggers encompassing the entire range of the image in our future work.  \n \n> Q5: More datasets and architectures can be considered.\n\nWe further evalute A3FL on FEMNIST dataset and VGG16 model. We record our results in Table 2 and 3 in the attached PDF, and observe that A3FL can still achieve high ASR."}, {"review_id": "nqtAcoqhgk", "reviewer": "Reviewer_BFfe", "summary": "The authors introduce A3FL, a backdoor attack that strategically adjusts the backdoor trigger to decrease the chances of its removal by the global training dynamics. The fundamental idea behind this approach lies in the disparity between the global model and the local model in Federated Learning (FL), which diminishes the effectiveness of locally optimized triggers when transferred to the global model. To address this issue, the authors tackle the optimization of the trigger in a manner that ensures its survival even in the worst-case scenario, wherein the global model is specifically trained to eliminate the trigger. Through extensive experiments conducted on benchmark datasets, they comprehensively evaluate the efficacy of A3FL against twelve existing defense mechanisms.", "questions": "1. The calculation of $\\theta'_t$ solely based on the local training data of malicious clients appears to be an approximation of the true worst-case scenario, as it neglects the inclusion of benign clients' data. I am interested in understanding the implications of an inaccurate $\\theta'_t$ on the performance of the attack, particularly in scenarios where there is a high level of heterogeneity. It would be intriguing to compare the attack performance when the attacker possesses knowledge of the benign workers' data, considering previous studies have demonstrated that attackers can learn and reconstruct benign workers' data through inference attacks [1] [2].\n2. How to choose/tune a good or even an optimal (is it exist?) $\\lambda$? It seems to me in each FL round, the optimal $\\lambda$ should be different (based on current model, number of attackers been chosen, etc.)\n3. To ensure fair comparison, while A3FL tune parameters like $\\lambda$ and poison ratio, how are the hyperparameters are chosen in other baseline attack methods?\n\n[1] Geiping, Jonas, et al. \"Inverting gradients-how easy is it to break privacy in federated learning?.\" Advances in Neural Information Processing Systems 33 (2020): 16937-16947.\n[2] Li, Henger, Xiaolin Sun, and Zizhan Zheng. \"Learning to attack federated learning: A model-based reinforcement learning attack framework.\" Advances in Neural Information Processing Systems 35 (2022): 35007-35020.", "limitations": "1. It is unclear to me what exactly is being optimized in equations (2) and (3). I assume that the optimization pertains to the parameters associated with each pixel within the predetermined square pattern. In the conducted DBA experiments, the trigger(s) take the form of fixed-sized square(s) located at specific position(s) and may involve certain numbers of squares. These characteristics represent potential parameters that can be incorporated into the optimization problem. However, it is worth noting that for achieving the most versatile trigger, it is worthwhile to consider every pixel in the image. In recent works [1] [2], researchers have explored generated backdoor triggers that encompass the entire range of the image.\n2. Post-training stage defenses play a vital role in countering backdoor attacks. Even within the context of FL, certain techniques such as Neuron Clipping [3] and Pruning [4] have demonstrated their effectiveness in detecting and mitigating the impact of backdoor attacks. Consequently, I am curious to know how the proposed A3FL performs when subjected to these post-training stage defenses.\n\n[1] Salem, Ahmed, et al. \"Dynamic backdoor attacks against machine learning models.\" 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P). IEEE, 2022.\n[2] Doan, Khoa D., Yingjie Lao, and Ping Li. \"Marksman backdoor: Backdoor attacks with arbitrary target class.\" Advances in Neural Information Processing Systems 35 (2022): 38260-38273.\n[3] Wang, Hang, et al. \"Universal post-training backdoor detection.\" arXiv preprint arXiv:2205.06900 (2022).\n[4] Wu, Chen, et al. \"Mitigating backdoor attacks in federated learning.\" arXiv preprint arXiv:2011.01767 (2020).", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors consider a bi-level objective (i.e., the worst-case adaptation of global dynamics) to optimize the trigger pattern during the FL process without knowing the defense mechanism.\n2. Experiment results are good.\n3. The proposed method compares with extensive baselines in different settings.", "weaknesses": "1. No theoretical analysis or guarantee on attack performance and convergency/sample complexity for Algorithm 1\n2. It seems like solving a bi-level optimization in each FL round is computational inefficient. Either empirical or theoretical justifications comparing with baseline methods are needed to prove the efficiency.\n3. To incorporate global dynamics (i.e., long-term goal, benign clients' behaviors, defense mechanism, etc.) is useful to increase backdoor durability, however, it has already been proposed in previous works [1] [2].\n\n[1] Wen, Yuxin, et al. \"Thinking two moves ahead: Anticipating other users improves backdoor attacks in federated learning.\" arXiv preprint arXiv:2210.09305 (2022).\n[2] Li, Henger, et al. \"Learning to backdoor federated learning.\" arXiv preprint arXiv:2303.03320 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## reviewer BFfe\n\nWe thank the reviewer for the constructive comments to strengthen our work.\n\n> Q1: No theoretical analysis for Algorithm 1.\n\nWhile theoretical analysis is important and interesting, it is very challenging and an open problem to theoretically analyze the performance of FL backdoor attacks, especially when A3FL is implemented through bi-level optimization. To the best of our knowledge, most existing works proposing FL backdoor attacks did not provide theoretical analysis on attack performance or convergency. We will leave this as our future work. \n\n> Q2: Solving a bi-level optimization in each FL round is computational inefficient.\n\nThank the reviewer for constructive comments. We empirically compare the efficiency of A3FL to other baseline attacks. We record the average time taken by each attack in each round in Table 4 in the attached PDF. Observe that A3FL has a comparable efficiency with CerP and F3BA.\n\n\n> Q3: Incorporating global dynamics has already been proposed in previous works.\n\nThank you for commenting on the place that could cause misunderstanding. We will discuss and comment these works in the revision. Instead of incorporating global training dynamics, A3FL proposes a different principle by anticipating a worst-case scenario in which the injected backdoor is directly unlearned by the global model. A3FL optimizes the backdoor trigger to make it robust to this worst-case scenario and therefore persistent to other perturbations introduced by global training dynamics. Our empirical experiments in Section 4.3 and Appendix B.8 demonstrates this intuition. As observed in Figure 4, while all backdoor attacks can achieve nearly 100% ASR on the local model, only A3FL can achieve similarly high ASR when transferred to the global model. This observation demonstrates that A3FL is more persistent to perturbation when transferred to the global model.\n\n> Q4: Attack performance when the attacker possesses knowledge of the benign workers' data.\n\nThank you for the interesting perspective. We discussed the impact of data heterogeneity in Appendix B.6. As shown in Figure 12, a large data heterogeneity does not significantly impact the attack performances of A3FL.\n\nWe further consider a scenario where the attacker can access a part of private dataset from benign clients. Specifically, we merge the private training dataset of 5 randomly selected benign clients as $\\mathcal{D}_p$. We assume that the attacker can access $\\mathcal{D}_p$ to better approximate the worst-case scenario. As observed in Table 5 in the attached PDF, A3FL with access to benign worker's data indeed has slightly better performance but the improvement is marginal. We will discuss these works[7][8] and the potential extension in the revision.\n\n\n> Q5: How to tune $\\lambda$\n\nAs discussed in Line 261-269, we introduce a balancing coefficient $\\lambda_0$ to control the strength of adversarial training, and $\\lambda=\\lambda_0$sim$(\\theta'_t, \\theta_t)$. When the adversarial global model $\\theta'_t$ differs a lot from the current global model $\\theta_t$, we pay less attention to the adversarial training loss. This is motivated by that when the adversarial global model too different, the backdoor trigger optimized on current model is harder to be adapted to the adversarial model thus can be more easily unlearned. In practice, we find this helpful in balancing the bi-level optimization. As discussed in Section 4.3, Figure 6, A3FL is not obviously sensitive to different values of $\\lambda_0$. We simply grid search $\\lambda_0$ on FedAvg and apply the same value of $\\lambda_0$ to other cases. Since we observe that this has already achieved satisfying results, we do not adopt more complicated tuning methods to find a better $\\lambda_0$. \n\n> Q6: Hyperparameter settings of other attacks.\n\nThank you for commenting on the place that could cause confusion. For all evaluated attack methods, we tune hyperparameters on FedAvg via grid search. It is reasonable to search for optimal hyperparameters on FedAvg and use the same setting for other experiments, since the attacker does not know the server-side defense beforehand.\n\nWe have discussed hyperparameter settings in Appendix A.2 and B.9. For CerP, we set two balancing coefficients $\\alpha = 0.005$ and $\\beta = 0.005$. For F3BA, we set the proportion of candidate weights for convolutional layers to be 0.02, and the proportion for fully connected layers to be 0.001. For Neurotoxin, we only update the last 95% important model weights. For DBA, we set the trigger shift to be 0, trigger gap to be 6, and trigger size to be 4. \n\n> Q7: What exactly is being optimized in equations (2) and (3).\n\nWe thank the reviewer for suggesting an interesting direction to combine the progress from both optimization (as our work does) and trigger pattern design perspective. To clarify, our work mainly focuses on the optimization perspective, thus uses the commonly adopted squared trigger shape: A3FL only optimizes $\\delta$ for the masked-out trigger region. This can be viewed as applying $\\delta = \\delta \\cdot m$ at the end of each time of optimization. $m$ is a binary mask with 1 on the pixels within the predefined region. We adopted the same trigger shape for evaluated attacks to ensure fair comparison. We will explain this in the revised paper to eliminate potential confusion. We will also discuss these papers[9-12] in the revised paper, and extend A3FL to backdoor triggers encompassing the entire range of the image in our future work.\n\n> Q8: Post-training stage defenses should be considered.\n\nThank you for your constructive comment. Following the suggestion, we evaluate A3FL against Neuron Clipping and include the experimental results in Table 6 in the attached PDF. We observe that Neuoron Clipping cannot impact the ASR of A3FL. A3FL still significantly outperforms other baselines against this defense. We will include this defense in the revised paper."}, {"review_id": "DVtI0utepA", "reviewer": "Reviewer_k1k8", "summary": "The paper introduces A3FL, a backdoor attack tailored for FL. Unlike traditional approaches, A3FL adapts backdoor triggers in an adversarial manner considering global training dynamics, thereby enhancing its robustness and effectiveness. By accounting for discrepancies between global and local models in FL, A3FL ensures that the trigger remains potent even if the global model attempts to negate it. The paper demonstrates through extensive experiments that A3FL significantly surpasses existing backdoor attacks in terms of efficiency, even when faced with established defenses.", "questions": "Please address the issues discussed in Weaknesses.", "limitations": "The paper has several limitations, as discussed in the review.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is easy to follow. It addresses a crucial problem in federated learning. The authors have made code available, which makes it easy for others to replicate the study.", "weaknesses": "First of all, the paper needs editorial revision as it discusses the same thing multiple times. By making it shorter and to the point, there is more room to include extra results in the main content.\n\nThe technical limitations of the paper are discussed as follows:\n\nLine 36-37: \"the selected trigger is usually sub-optimal, which makes the attack less effective and stealthy as shown in experiments\" - I would like to request some clarification and elaboration. Firstly, it would be beneficial for the readers if the authors could provide references or citations that substantiate the claim regarding the sub-optimality of the triggers in existing backdoor attacks. Additionally, it is crucial to note that in FL, semantic backdoor triggers are considered to be particularly stealthy. The stealthiness of semantic triggers arises because they leverage the inherent properties of images rather than altering pixels, making them less detectable. In this regard, it would be insightful if the authors could elaborate on what they imply by 'stealthy' in the context of semantic triggers.\n\nLine 45-46: \"they only leverage local models of compromised clients to optimize the backdoor trigger, which ignores the global training dynamics\" - However, it is noteworthy that some existing works, such as Neurotoxin[1], have been known to embed backdoors by considering parameters that remain robust to the training dynamics in FL systems. Please clarify the claims made in this regard and support them with relevant references or citations, particularly contrasting with what Neurotoxin has achieved. Also, differentiate the contribution of A3FL in this regard.\n\nLine 47-50: \"they strictly regulate the difference between the local and global model weights to bypass defenses, which in turn limits the backdoor effectiveness\" - While regulating the difference between local and global model weights can be a strategy to bypass defenses, it has been demonstrated in the literature that this approach can also lead to high backdoor efficiency[2]. Please provide a more comprehensive analysis of this aspect and substantiate the claim regarding the limitation of backdoor effectiveness in such approaches with references or citations that clearly illustrate this limitation.\n\nLine 170: \"injected backdoor is rapidly eliminated since the global model is continuously updated by the server\" - This statement is misleading. In reality, the persistence of backdoors in FL systems can be influenced by the timing of their injection. Existing research indicates that if a backdoor is introduced after the model has reached a stable state, it tends to persist longer before being eliminated.\n\nLine 174-175: \"the global model is trained to directly unlearn the backdoor\" - I would like to request the authors to provide references or citations that substantiate the statement. It is not clear how the backdoor trigger can be eliminated from the global model without knowing the specific patterns that activate it.\n\nIn the A3FL workflow, the first step involves optimizing the trigger pattern based on the model parameters. Subsequently, the adversarial global model is optimized in each FL training round. This approach raises a question regarding the trigger pattern during evaluation. Unlike in training rounds where the trigger patterns vary due to FL dynamics, it is unclear what trigger will be used during the evaluation. Specifically, it is important to describe whether the adversary requires access to the global model to generate triggers during testing. If the adversary does require such access, it would be considered an adversarial attack rather than a backdoor attack. Conversely, if the adversary does not need access to the global model, the trigger for misclassification remains uncertain.\n\nA3FL uses an adversarial example generation strategy for the incorporation of backdoor triggers. In contrast, an approach called PerDoor[3] has been recently introduced, utilizing adversarial examples to achieve a similar purpose. It is essential for the authors to clearly explain the contributions and differentiating factors of A3FL in relation to PerDoor.\n\nThe efficiency of A3FL must be evaluated in the presence of FLAME[4], a state-of-the-art defense mechanism against backdoor attacks in FL. Furthermore, A3FL needs to be compared against recent state-of-the-art attacks, like 3DFed[5].\n\nA3FL should be evaluated using more practical benchmark federated learning datasets in the LEAF project[7]. The authors have only used the ResNet18 model as the underlying structure for all their experiments. For a more comprehensive evaluation, A3FL should be evaluated across a variety of models.\n\n[1] Z. Zhang et al., \"Neurotoxin: Durable Backdoors in Federated Learning\", ICML 2022.\n\n[2] H. Wang et al., \"Attack of the Tails: Yes,You Really Can Backdoor Federated Learning\", NeurIPS 2020.\n\n[3] M. Alam et al., \"PerDoor: Persistent Non-Uniform Backdoors in Federated Learning using Adversarial Perturbations\", arXiv 2022.\n\n[4] T. Nguyen et al., \"FLAME: Taming Backdoors in Federated Learning\", Usenix Security 2022.\n\n[5] H. Li et al., \"3DFed: Adaptive and Extensible Framework for Covert Backdoor Attack in Federated Learning\", IEEE S&P 2023.\n\n[6] https://leaf.cmu.edu/", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## reviewer k1k8\nWe thank the reviewer for the detailed suggestion to improve the clarity and enhance the evaluation of our work.\n\n> Q1: Line 36-37\n\nThanks for commenting on the place that could cause misunderstanding. In the context of this sentence (Line 34-37), we were discussing the fixed-trigger backdoor attacks. The sub-optimality of such fixed-trigger attacks has been shown by previous works [1-3] and in Section 4.2 of our paper.\n\nWe also thank the reviewer for pointing out the possible confusion w.r.t. the definition of stealthiness. In our case, we consider stealthiness in FL as the capability of bypassing defenses without harming the utility of the global model. We do not care about the stealthiness in images as local data is not directly seen by the server. Following this definition, semantic backdoor trigger is not stealthy since they inherit the sub-optimality of fixed-trigger backdoors. We will clarify the meaning of stealthiness and discuss semantic backdoors in revision.\n\n> Q2: Line 45-46\n\nThanks for pointing out the place that could cause misunderstanding. In the context of this sentence (lines 44-45), we were discussing the limitations of trigger-optimization backdoor attacks[4][5]. Neurotoxin is a fixed-trigger attack which does not fall into this discussion in Line 45-46. \n\nAs discussed in Line 34-37 and 95-96, the differences between A3FL and Neurotoxin are two folds. First, A3FL considers a worst-case scenario in which the global model is trained to directly unlearn the backdoor while Neurotoxin does not have such an adversarial component. A3FL further optimizes the backdoor trigger to survive the worst case scenario, therefore improving the backdoor persistence. In comparison, Neurotoxin passively avoids embedding the backdoor in frequently changing model weights. As shown in Figure 17 of [6], Neurotoxin can only achieve minor improvement in the image domain, which is also supported by our experiments in Figure 2. \n\n> Q3: Line 47-50\n\nWe would like to use the ablation study in F3BA[4] to demonstrate our claim. In Appendix D.6 of [4], the authors wrote \"Meanwhile, too high candidate parameters proportion for fully-connected layers can cause an obvious loss of ASR.\" The authors also conducted a grid search to find an optimal regularization proportion. This indicates that regularization can potentially harm the backdoor efficiency. We conduct an experiment to further support our claim. We use L2-norm regularization to regulate the backdoor loss and vary the strength of regularization by adjusting the value of balancing coefficient $\\beta$. As observed in Table 7 in the attached PDF, the ASR drops as the strength of regularization enhances. We will include this in the revised paper.\n\n> Q4: Line 170\n\nThanks for suggesting another factor (the timing of injection) that influences the persisitence of attack. We were focusing on the attack design perspective, when the injection timing is controled the same, other baseline backdoors are eliminated in a shorter time than A3FL. We will modify this sentence into \"injected backdoor could be eliminated since the global model is continuously updated by the server, especially when the global model has not reached a stable state\".\n\n> Q5: Line 174-175\n\nThe FL server cannot access the specific backdoor trigger in practice. However, the compromised client has access to the trigger as an attacker to simulate a worst-case scenario where the global model is trained to directly unlearn the backdoor trigger, as discussed in lines 173-174.\n\n> Q6: The trigger pattern during the evaluation and the attacker's access to global model.\n\nAs discussed in Line 248-252, in the attack window (i.e. compromised clients being selected to participate), the attacker can access the global model through compromised clients to optimize the trigger. After the end of the attack window, the attacker-compromised clients leave the FL procedure. The attack performance is evaluated using the trigger at the end of the attack window. Following [6], the trigger pattern will no longer be optimized after the attack window, therefore the attacker do not need to access the global model anymore.\n\n> Q7: Compare A3FL to PerDoor\n\nWe will discuss the difference between PerDoor[15] and A3FL in revision. PerDoor adopts adversarial attack to generate the backdoor trigger. However, the trigger in A3FL is not an adversarial example. As discussed in Section 3, we optimize the backdoor trigger to survive adversarially crafted global model. This pipeline can be seen as adversarial training in a reverse way where the crafted global model corresponds to the inner maximization problem and the trigger optimization corresponds to the outer minimization problem.\n\n> Q8: More baselines: FLAME and 3DFed.\n\nWe will discuss FLAME[13] in the revised paper. As for evaluation, we found the code of FLAME is not publicly available and have not got response from the author after acquiring the code. We will update the experimental results on FLAME once the code is received.  \n\nWe will discuss 3DFed[14] in the revised paper. However, we note that 3DFed was first published in S&P 2023 held on 21-15 May 2023. And the paper was added to IEEE on 21 July 2023. To the best of our knowledge, 3DFed was not published on arxiv before. Therefore, we were not able to compare A3FL against 3DFed. What's more, according to the concurrent policy of NeurIPS 2023, we are not required to consider concurrent work which was published in two months before the submission deadline. We also found that the code of 3DFed is not publicly available and will update the experimental results once the code is received.  \n\n> Q9: A3FL should be evaluated using more practical benchmark in the LEAF project.\n\nThanks for suggesting more benchmarks. In the attached PDF, we further report the performance of A3FL on FEMNIST from the LEAF project in Table 3, as well as A3FL on VGG16 in Table 2. Observe that A3FL can still achieve high ASRs."}]

Author rebuttal:
## General Response to All Reviewers

We thank all reviewers for your constructive suggestions and insightful questions! We have responded to them in our separate responses. We have provided supplementary experimental results about different available training data and the attack performance of A3FL on other mode architectures and more datasets. We also discuss the computational overhead of A3FL in comparison to other attacks. We evaluate A3FL against post-training as well. Welcome to raise following questions to further improve our work!

The citation list in our defense is as follows:

[1] Nguyen, A., & Tran, A. (2021). Wanet--imperceptible warping-based backdoor attack. arXiv preprint arXiv:2102.10369.

[2] Doan, K., Lao, Y., Zhao, W., & Li, P. (2021). Lira: Learnable, imperceptible and robust backdoor attacks. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 11966-11976).

[3] Doan, K., Lao, Y., & Li, P. (2021). Backdoor attack with imperceptible input and latent modification. Advances in Neural Information Processing Systems, 34, 18944-18957.

[4] Fang, P., & Chen, J. (2023). On the Vulnerability of Backdoor Defenses for Federated Learning. arXiv preprint arXiv:2301.08170.

[5] Lyu, X., Han, Y., Wang, W., Liu, J., Wang, B., Liu, J., & Zhang, X. (2023, February). Poisoning with cerberus: stealthy and colluded backdoor attack against federated learning. In Thirty-Seventh AAAI Conference on Artificial Intelligence.

[6] Zhang, Z., Panda, A., Song, L., Yang, Y., Mahoney, M., Mittal, P., ... & Gonzalez, J. (2022, June). Neurotoxin: Durable backdoors in federated learning. In International Conference on Machine Learning (pp. 26429-26446). PMLR.

[7] Geiping, Jonas, et al. "Inverting gradients-how easy is it to break privacy in federated learning?." Advances in Neural Information Processing Systems 33 (2020): 16937-16947. 

[8] Li, Henger, Xiaolin Sun, and Zizhan Zheng. "Learning to attack federated learning: A model-based reinforcement learning attack framework." Advances in Neural Information Processing Systems 35 (2022): 35007-35020.

[9] Salem, Ahmed, et al. "Dynamic backdoor attacks against machine learning models." 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P). IEEE, 2022. [10] Doan, Khoa D., Yingjie Lao, and Ping Li. "Marksman backdoor: Backdoor attacks with arbitrary target class." Advances in Neural Information Processing Systems 35 (2022): 38260-38273. 

[11] Wang, Hang, et al. "Universal post-training backdoor detection." arXiv preprint arXiv:2205.06900 (2022). 

[12] Wu, Chen, et al. "Mitigating backdoor attacks in federated learning." arXiv preprint arXiv:2011.01767 (2020).

[13]Nguyen, T. D., Rieger, P., De Viti, R., Chen, H., Brandenburg, B. B., Yalame, H., ... & Schneider, T. (2022). {FLAME}: Taming backdoors in federated learning. In 31st USENIX Security Symposium (USENIX Security 22) (pp. 1415-1432).

[14]Li, H., Ye, Q., Hu, H., Li, J., Wang, L., Fang, C., & Shi, J. (2023, May). 3DFed: Adaptive and Extensible Framework for Covert Backdoor Attack in Federated Learning. In 2023 IEEE Symposium on Security and Privacy (SP) (pp. 1893-1907). IEEE.

[15]Alam, M., Sarkar, E., & Maniatakos, M. (2022). PerDoor: Persistent Non-Uniform Backdoors in Federated Learning using Adversarial Perturbations. arXiv preprint arXiv:2205.13523.

Meta-review: Overall, I found the paper to have compelling empirical results with a sound methodology. While some of the "theoretical-style" arguments used to justify the core ideas need to be reworded (see below) I didn't read these as theory grounding for the work, and ultimately, the ideas are novel and there's sufficient evidence to demonstrate where they work and limitations. The idea of adapting the backdoor trigger to global training dynamics is well argued. Results demonstrate that the proposed adaptation A3FL does reduce the rate of backdoor removing during federated learning. The experiments are largely very comprehensive (a key strength of the work) with caveats below and in the discussion, demonstrating convincing results on ASR, utility, lifespan of triggers.

The paper's language around "worst-case" is quite problematic. These motivations have been shown during author discussion not to really be worst-case. While this doesn't invalidate actual results, the language is imprecise and potentially misleading - but as a presentation issue it can be easily addressed and I'm glad that the authors have committed to doing so. 

Second, baseline attacks have been discussed during the author discussion phase. I still believe these would make the work more compelling and improve its impact. However I viewed these as "nice to haves" rather than "must haves" and didn't critically limit the significance of the work.

I'd strongly encourage the authors to take on board discussions, in improving their paper. For example, the discussion of stealth, being precise when saying "worst case" (or perhaps not saying it), comparison/discussion briefly on machine unlearning (I don't think you need to certify unlearning! But reference to that body of work/discussion of relevance might be helpful), and inclusion of the updated results during the discussion e.g., on FLAME.
Decision: Accept (poster)