SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A3FL: Adversarily Adaptive Backdoor Attacks to Federated Learning
Hangfan Zhang, Jinyuan Jia, Jinghui Chen, Lu Lin, Dinghao Wu
{hz5148,jinyuan,jzc5917,lulin,dinghao}@psu.edu
The Pennsylvania State University
Federated Learning (FL) is a distributed machine learning paradigm that allows multiple clients to train a global model collaboratively without sharing their local training data. Due to its distributed nature, many studies have shown that it is vulnerable to backdoor attacks. However, existing studies usually used a predetermined, fixed backdoor trigger or optimized it based solely on the local data and model without considering the global training dynamics. This leads to sub-optimal and less durable attack effectiveness, i.e., their attack success rate is low when the attack budget is limited and decreases quickly if the attacker can no longer perform attacks anymore. To address these limitations, we propose A3FL, a new backdoor attack which adversarially adapts the backdoor trigger to make it less likely to be removed by the global training dynamics. Our key intuition is that the difference between the global model and the local model in FL makes the local-optimized trigger much less effective when transferred to the global model. We solve this by optimizing the trigger to even survive the scenario where the global model was trained to directly unlearn the trigger. Extensive experiments on benchmark datasets are conducted for twelve existing defenses to comprehensively evaluate the effectiveness of our A3FL. Our code is available at https://github.com/hfzhang31/A3FL.
Recent years have witnessed the rapid development of Federated Learning (FL) [1, 2, 3], an advanced distributed learning paradigm. With the assistance of a cloud server, multiple clients such as smartphones or IoT devices train a global model collaboratively based on their private training data through multiple communication rounds. In each communication round, the cloud server selects a part of the clients and sends the current global model to them. Each selected client first uses the received global model to initialize its local model, then trains it based on its local dataset, and finally sends the trained local model back to the cloud server. The cloud server aggregates local models from selected clients to update the current global model. FL has been widely used in many safety- and privacy-critical applications [4, 5, 6, 7].
Numerous studies [8, 9, 10, 11, 12, 13, 14] have shown that the distributed nature of FL provides a surface to backdoor attacks, where an attacker can compromise some clients and utilize them to inject a backdoor into the global model such that the model's behaviors are the attacker desired. In particular, the backdoored global model behaves normally on clean testing inputs but predicts any testing inputs stamped with an attacker-chosen backdoor trigger as a specific target class.
Depending on whether the backdoor trigger is optimized, we can categorize existing attacks into fixed-trigger attacks [12, 11, 13, 8] and trigger-optimization attacks [10, 9]. In a fixed-trigger attack, an attacker pre-selects a fixed backdoor trigger and thus does not utilize any information from FL training process. CWhile a fixed-trigger attack can be more efficient and straightforward, it usually suffers from limited effectiveness and more obvious utility drops.
In a trigger-optimization attack, an attacker optimizes the backdoor trigger to enhance the attack. Fang et al. [10] proposed to maximize the difference between latent representations of clean and trigger-stamped samples. Lyu et al. [9] proposed to optimize the trigger and local model jointly with  $\ell_2$  regularization on local model weights to bypass defenses. The major limitations of existing trigger-optimization attacks are twofold. First, they only leverage local models of compromised clients to optimize the backdoor trigger, which ignores the global training dynamics. Second, they strictly regulate the difference between the local and global model weights to bypass defenses, which in turn limits the backdoor effectiveness. As a result, the locally optimized trigger becomes much less effective when transferred to the global model as visualized in Figure 1. More details for this experiment can be found in Appendix A.1.
Figure 1: A3FL and CerP [9] can achieve  $100\%$  ASR on the local model. However, only A3FL in the mean time obtains a high global ASR.
Our contribution: In this paper, we propose Adversarially Adaptive Backdoor Attacks to Federated Learning (A3FL). Recall that existing works can only achieve sub-optimal attack performance due to ignorance of global training dynamics. A3FL addresses this problem by adversially adapting to the dynamic global model. We propose adversarial adaptation loss, in which we apply an adversarial training-like method to optimize the backdoor trigger so that the injected backdoor can remain effective in the global model. In particular, we predict the movement of the global model by assuming that the server can access the backdoor trigger and train the global model to directly unlearn the trigger. We adaptively optimize the backdoor trigger to make it survive this adversarial global model, i.e., the backdoor cannot be easily unlearned even if the server is aware of the exact backdoor trigger. We empirically validate our intuition as well as the effectiveness and durability of the proposed attack.
We further conduct extensive experiments on widely-used benchmark datasets, including CIFAR-10 [15] and TinyImageNet [16], to evaluate the effectiveness of A3FL. Our empirical results demonstrate that A3FL is consistently effective across different datasets and settings. We further compare A3FL with 4 state-of-the-art backdoor attacks [12, 11, 10, 9] under 13 defenses [2, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27], and the results suggest that A3FL remarkably outperforms all baseline attacks by up to 10 times against all defenses. In addition, we find that A3FL is significantly more durable than all baselines. Finally, we conduct extensive ablation studies to evaluate the impact of hyperparameters on the performance of A3FL.
To summarize, our contributions can be outlined as follows.
- We propose A3FL, a novel backdoor attack to the FL paradigm based on adversarial adaptation, in which the attacker optimizes the backdoor trigger using an adversarial training-like technique to enhance its persistence within the global training dynamics.
- We empirically demonstrate that A3FL remarkably improves the durability and attack effectiveness of the injected backdoor in comparison to previous backdoor attacks.
- We comprehensively evaluate A3FL towards existing defenses and show that they are insufficient for mitigating A3FL, highlighting the need for new defenses.
Federated learning: Federated Learning (FL) was first proposed in [1] to improve communication efficiency in decentralized learning. FedAvg [2] aggregated updates from each client and trains the global model with SGD. Following studies [28, 29, 30, 31, 32] further improved the federated paradigm by making it more adaptive, general, and efficient.
Existing attacks and their limitations: In backdoor attacks to FL, an attacker aims to inject a backdoor into model updates of compromised clients such that the final global model aggregated by the server is backdoored. Existing backdoor attacks on FL can be classified into two categories: fixed-trigger attacks [12, 11, 8, 14, 13] and trigger-optimization attacks [10, 9].
Fixed-trigger attacks [8, 11, 14, 13, 12] pre-select a fixed backdoor trigger and poison the local training set with it. Since a fixed trigger may not be effective for backdoor injection, these attacks improved the backdoor effectiveness through other approaches including manually manipulating the poisoned updates. Particularly, scaling attack [8] scaled up the updates to dominate other clients to improve the attack effectiveness. DBA [11] split the trigger into several sub-triggers for poisoning, which makes DBA more stealthy from defenses. Neurotoxin [12] only attacked unimportant model parameters that are less frequently updated to prevent the backdoor from being erased shortly.
Trigger-optimization attacks [10, 9] optimize the backdoor trigger to enhance the attack. F3BA [10] optimized the trigger pattern to maximize the difference between latent representations of clean and trigger-stamped samples. F3BA also projected gradients to unimportant model parameters like Neurotoxin [12] to improve stealthiness. CerP [9] jointly optimized the trigger and the model weights with  $\ell_2$  regularization to minimize the local model bias. These attacks can achieve higher attack performance than fixed-trigger attacks. However, they have the following limitations. First, they only consider the static local model and ignore the dynamic global model in FL, thus the optimized trigger could be sub-optimal on the global model. Second, they apply strict regularization on the difference between the local model and the global model, which harms the backdoor effectiveness. Therefore, they commonly need a larger attack budget (e.g., compromising more clients) to take effect. We will empirically demonstrate these limitations in Section 4.
Existing defenses: In this paper, we consider two categories of defenses in FL. The first category of defense mechanisms is deliberately designed to alleviate the risks of backdoor attacks [17, 19, 20, 18, 33] on FL. These defense strategies work by restricting clients' updates to prevent the attackers from effectively implanting a backdoor into the global model. For instance, the Norm Clipping [17] defense mechanism limits clients' behavior by clipping large updates, while the CRFL [19] defense mechanism uses parameter smoothing to impose further constraints on clients' updates.
The second category of defenses [26, 25, 24, 23, 22, 21, 34] is proposed to improve the robustness of FL against varied threats. These defense mechanisms operate under the assumption that the behavior of different clients is comparable. Therefore, they exclude abnormal clients to obtain an update that is recognized by most clients to train the global model. For instance, the Median [22] defense mechanism updates the global model using the median values of all clients' updates, while Krum [21] filters out the client with the smallest pairwise distance from other clients and trains the global model solely with the filtered client updates. These defense mechanisms can achieve superior robustness compared to those defense mechanisms that are specifically designed for backdoor attacks. Nevertheless, the drawback of this approach is evident: it often compromises the accuracy of the global model, as it tends to discard most of the information provided by clients, even if these updates are merely potentially harmful.
In this paper, we also utilize backdoor unlearning [35, 36] to approximate existing defenses. Backdoor unlearning typically minimizes the prediction loss of trigger-stamped samples to the ground truth labels. Note that backdoor unlearning disparts from so-called machine unlearning [37, 38, 39], in which the model is unlearned to "forget" specific training samples.
There exist additional defenses in FL that are beyond the scope of this paper. While these defenses may offer potential benefits, they also come with certain limitations in practice. For instance, FLTrust [40] assumed the server holds a clean validation dataset, which deviates from the typical FL setting. Cao et al. [41] proposed sample-wise certified robustness which demands hundreds of times of retraining and is computationally expensive.
3 Methodology
To formulate the backdoor attack scenario, we first introduce the federated learning setup and threat model. Motivated by the observation of the local-global gap in existing works due to the ignorance of global dynamics, we propose to optimize the trigger via an adversarial adaptation loss.
3.1 Federated Learning Setup and Threat Model
We consider a standard federated learning setup where  $N$  clients aim to collaboratively train a global model  $f$  with the coordination of a server. Let  $\mathcal{D}_i$  be the private training dataset held by the client  $i$ , where  $i = 1,2,\ldots,N$ . The joint training dataset of the  $N$  clients can be denoted as  $\mathcal{D} = \cup_{i=1}^{N}\mathcal{D}_i$ .
In the  $t$ -th communication round, the server first randomly selects  $M$  clients, where  $M \leq N$ . For simplicity, we use  $S_{t}$  to denote the set of selected  $M$  clients. The server then distributes the current version of the global model  $\theta_{t}$  to the selected clients. Each selected client  $i \in S_{t}$  first uses the global model to initialize its local model, then trains its local model on its local training dataset, and finally uploads the local model update (i.e., the difference between the trained local model and the received global model) to the server. We use  $\Delta_{t}^{i}$  to denote the local model update of the client  $i$  in the  $t$ -th communication round. The server aggregates the received updates on model weights and updates the current global model weights as follows:
$$
\boldsymbol {\theta} _ {t + 1} = \boldsymbol {\theta} _ {t} + \mathcal {A} \left(\left\{\boldsymbol {\Delta} _ {t} ^ {i} \mid i \in S _ {t} \right\}\right) \tag {1}
$$
where  $\mathcal{A}$  is an aggregation rule adopted by the server. For instance, a widely used aggregation rule FedAvg [2] takes an average over the local model updates uploaded by clients.
Attacker's goal: We consider an attacker aims to inject a backdoor into the global model. In particular, the attacker aims to make the injected backdoor effective and durable. The backdoor is effective if the backdoored global model predicts any testing inputs stamped with an attacker-chosen backdoor trigger as an attacker-chosen target class. The backdoor is durable if it remains in the global model even if the attacker-compromised clients stop uploading poisoned updates while the training of the global model continues. We note that a durable backdoor is essential for an attacker as the global model in a production federated learning system is periodically updated but it is impractical for the attacker to perform attacks in all time periods [12, 42]. Considering the durability of the backdoor enables us to understand the effectiveness of backdoor attacks under a strong constraint, i.e., the attacker can only attack the global model within a limited number of communication rounds.
Attacker's background knowledge and capability: Following threat models in previous studies [9, 12, 10, 11, 14, 8], we consider an attacker that can compromise a certain number of clients. In particular, the attacker can access the training datasets of those compromised clients. Moreover, the attacker can access the global model received by those clients and manipulate their uploaded updates to the server. As a practical matter, we consider the attacker can only control those compromised clients for a limited number of communication rounds [12, 11, 8, 10, 9].
3.2 Adversarily Adaptive Backdoor Attack (A3FL)
Our key observation is that existing backdoor attacks are less effective because they either use a fixed trigger pattern or optimize the trigger pattern only based on the local model of compromised clients. However, the global model is dynamically updated and therefore differs from the static local models. This poses two significant challenges for existing backdoor attacks. Firstly, a backdoor that works effectively on the local model may not be similarly effective on the global model. Secondly, the injected backdoor is rapidly eliminated since the global model is continuously updated by the server, making it challenging for attackers to maintain the backdoor's effectiveness over time.
We aim to address these challenges by adversarially adapting the backdoor trigger to make it persistent in the global training dynamics. Our primary objective is to optimize the backdoor trigger in a way that allows it to survive even in the scenario where the global model is trained to directly unlearn the backdoor [35, 36]. To better motivate our method, we first discuss the limitations of existing state-of-the-art backdoor attacks on federated learning.
Limitation of existing works: In recent state-of-the-art works [9, 10], the attacker optimizes the backdoor trigger to maximize its attack effectiveness and applies regularization techniques to bypass server-side defense mechanisms. Formally, given the trigger pattern  $\delta$  and an arbitrary input  $\mathbf{x}$ , the input is stamped with the backdoor trigger can be denoted as  $\mathbf{x} \oplus \delta$ , which is called the backdoored input. Suppose the target class is  $\tilde{y}$ . Since the attacker has access to the training dataset of a compromised client  $i$ , the backdoor trigger  $\delta$  can be optimized using the following objective:
$$
\boldsymbol {\delta} ^ {*} = \underset {\boldsymbol {\delta}} {\operatorname {a r g m i n}} \mathbb {E} _ {(\mathbf {x}, y) \sim \mathcal {D} _ {i}} \left[ \mathcal {L} (\mathbf {x} \oplus \boldsymbol {\delta}, \tilde {y}; \boldsymbol {\theta} _ {t}) \right] \tag {2}
$$
where  $\theta_{t}$  represents the global model weights in the  $t$ -th communication round, and  $\mathcal{L}$  is the classification loss function such as cross-entropy loss. To conduct a backdoor attack locally, the attacker randomly samples a small set of inputs  $\mathcal{D}_i^b$  from the local training set  $\mathcal{D}_i$ , and poisons inputs in  $\mathcal{D}_i^b$  with trigger stamped. The attacker then injects a backdoor into the local model by optimizing the local model on the partially poisoned local training set with regularization to limit the gap between
the local and global model, i.e.,  $||\pmb{\theta} - \pmb{\theta}_t||$ . While the regularization term helps bypass server-side defenses, it greatly limits the backdoor effectiveness, as it only considers the current global model  $\pmb{\theta}_t$  and thus fails to adapt to future global updates.
As illustrated in Figure 1, we observe that such backdoor attack on federated learning (e.g., CerP [9]) is highly effective on the local model, suggested by a high local attack success rate (ASR). However, due to the ignorance of global dynamics, they cannot achieve similar effectiveness when transferred to the global model, resulting in a low ASR on the global model. Our method A3FL aims to bridge the local-global gap in existing approaches to make the backdoor persistent when transferred to the global model thus achieving advanced attack performance. In particular, we introduce adversarial adaptation loss that makes the backdoor persistent to global training dynamics.
Adversarial adaptation loss: To address the challenge introduced by the global model dynamics in federated learning, we propose the adversarial adaptation loss. As the attacker cannot directly control how the global model is updated as federated learning proceeds, its backdoor performance can be significantly impacted when transferred to the global model, especially when only a small number of clients are compromised by the attacker or defense strategies are deployed. For instance, local model updates from benign clients can re-calibrate the global model to indirectly mitigate the influence of the backdoored updates from the compromised clients; a defense strategy can also be deployed by the server to mitigate the backdoor. To make the backdoor survive such challenging scenarios, our intuition is that, if an attacker could anticipate the future dynamics of the global model, the backdoor trigger would be better optimized to adapt to global dynamics.
However, global model dynamics are hard to predict because 1) at each communication round, all selected clients contribute to the global model but the attacker cannot access the private training datasets from benign clients and thus cannot predict their local model updates, and 2) the attacker does not know how local model updates are aggregated to obtain the global model and is not aware of possible defense strategies adopted by the server. As directly predicting the exact global model dynamics is challenging, we instead require the attacker to foresee and survive the scenario where the global model is trained to directly unlearn the backdoor. In this paper we consider backdoor unlearning proposed in prior backdoor defenses [35, 36].
Specifically, starting from current global model  $\theta_{t}$ , we foresee an adversarially crafted global model  $\theta_{t}^{\prime}$  that can minimize the impact of the backdoor. We adopt an adversarial training-like method to obtain  $\theta_{t}^{\prime}$ : the attacker can use the generated backdoor trigger to simulate the unlearning of the backdoor in the global model. The trigger is then optimized to simultaneously backdoor the current global model  $\theta_{t}$  and the adversarially adapted global model  $\theta_{t}^{\prime}$ . Formally, the adversarially adaptive backdoor attack (A3FL) can be formulated as the following optimization problem:
$$
\begin{array}{l} \delta^ {*} = \underset {\delta} {\operatorname {a r g m i n}} \mathbb {E} _ {(\mathbf {x}, y) \sim \mathcal {D} _ {i}} \left[ \mathcal {L} (\mathbf {x} \oplus \delta , \tilde {y}; \boldsymbol {\theta} _ {t}) + \lambda \mathcal {L} (\mathbf {x} \oplus \delta , \tilde {y}; \boldsymbol {\theta} _ {t} ^ {\prime}) \right] \\ s. t. \boldsymbol {\theta} _ {t} ^ {\prime} = \underset {\boldsymbol {\theta}} {\operatorname {a r g m i n}} \mathbb {E} _ {(\mathbf {x}, y) \sim \mathcal {D} _ {i}} [ \mathcal {L} (\mathbf {x} \oplus \boldsymbol {\delta}, y; \boldsymbol {\theta}) ] \tag {3} \\ \end{array}
$$
where  $\theta$  is initialized with current global model weights  $\theta_t$ ;  $\theta_t'$  is the optimized adversarial global model which aims to correctly classify the backdoored inputs as their ground-truth label to unlearn the backdoor. In trigger optimization,  $\lambda$  is a hyperparameter balancing the backdoor effect on the current global model  $\theta_t$  and the adversarial one  $\theta_t'$ , such that the local-global gap is bridged when the locally optimized trigger is transferred to the global model (after server-side aggregation/defenses). Note that attacking the adversarial model is an adaptation or approximation of global dynamics, as in practice the server cannot directly access and unlearn the backdoor trigger to obtain such an adversarial model.
Algorithm of A3FL: We depict the workflow of A3FL compromising a client in Algorithm 1. At the  $t$ -th communication round, the client is selected by the server and receives the current global model  $\theta_t$ . Lines 4-8 optimize the trigger based on the current and the adversarial global model using cross-entropy loss  $\mathcal{L}_{\mathrm{ce}}$ . The adversarial global model is initialized by the global model weights in Line 1, and is updated in Line 10. Lines 12-14 train the local model on the poisoned dataset and upload local updates to the server.