SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1: The workflow of A3FL compromising a client
Input:  $\theta_{t},\mathcal{D}_{i},\tilde{y},K,K_{\mathrm{trig}}$  trigger,  $\alpha_{1},\alpha_{2},\lambda$    
1:  $\pmb{\theta}_t^{\prime} = \pmb{\theta}_t$    
2: for  $j = 1$  to  $K$  do   
3: Sample a batch of training data  $\mathcal{B}$  from  $\mathcal{D}_i$    
4: for  $k = 1$  to  $K_{\mathrm{trig}}$  do   
5: // Optimize trigger pattern  $\delta$  following Equation 3.   
6:  $L = \frac{1}{|B|}\sum_{\mathbf{x}\in B}(\mathcal{L}_{\mathrm{ce}}(\mathbf{x}\oplus \delta ,\tilde{y};\pmb {\theta}_t) + \lambda \mathcal{L}_{\mathrm{ce}}(\mathbf{x}\oplus \delta ,\tilde{y};\pmb {\theta}_t'))$    
7:  $\delta \gets \delta -\alpha_1\nabla_\delta L$    
8: end for   
9: // Optimize adversarial global model weights  $\pmb{\theta}_t^\prime$  following Equation 3.   
10:  $\pmb{\theta}_t^\prime \leftarrow \pmb{\theta}_t^\prime -\alpha_2\nabla_\pmb{\theta}\frac{1}{|\mathcal{B}|}\sum_{(\mathbf{x},y)\in \mathcal{B}}\mathcal{L}_{\mathrm{ce}}(\mathbf{x}\oplus \delta ,y;\pmb{\theta}_t^\prime)$    
11: end for   
12: Poison local dataset with  $\delta$  and update local model to obtain  $\pmb{\theta}_{t + 1}^{i}$    
13:  $\Delta_i^{t + 1} = \pmb{\theta}_{t + 1}^i -\pmb{\theta}_t$    
14: Upload  $\Delta_i^{t + 1}$  to the server
4.1 Experimental Setup
Datasets: We evaluate A3FL on three widely-used benchmark datasets: FEMNIST [43], CIFAR-10 [15], and TinyImageNet [16]. The FEMNIST dataset consists of 80,5263 images shaped  $28 \times 28$  distributed across 10 classes. The CIFAR-10 dataset consists of 50,000 training images and 10,000 testing images that are uniformly distributed across 10 classes, with each image having a size of  $32 \times 32$  pixels. The TinyImageNet dataset contains 100,000 training images and 20,000 testing images that are uniformly distributed across 200 classes, where each image has a size of  $64 \times 64$  pixels.
Federated learning setup: By default, we set the number of clients  $N = 100$ . At each communication round, the server randomly selects  $M = 10$  clients to contribute to the global model. The global model architecture is ResNet-18 [44]. We assume a non-i.i.d data distribution with a concentration parameter  $h$  of 0.9 following previous works [12, 10, 9]. We evaluate the impact of data heterogeneity by adjusting the value of  $h$  in Appendix B.6. Each selected client trains the local model for 2 epochs using SGD optimizer with a learning rate of 0.01. The FL training process continues for 2,000 communication rounds.
Attack setup: We assume that the attacker compromises  $P$  clients among all  $N$  clients. All compromised clients are only allowed to attack in limited communication rounds called attack window. By default, the attack window starts at the 1,900th communication round and ends at the 2,000th communication round. We also discuss the impact of the attack window in Appendix B.7. When a compromised client is selected by the server during the attack window, it will upload poisoned updates trying to inject the backdoor. We adjust the number of compromised clients  $P \in [1,20]$  to comprehensively evaluate the performance of each attack. Different from previous works [11, 12], in our evaluation compromised clients are selected randomly to simulate the practical scenario. Each compromised client poisons  $25\%$  of the local training dataset and trains the local model on the partially poisoned dataset with the same parameter settings as benign clients unless otherwise mentioned. By default, the trigger is designed as a square at the upper left corner of the input images. We use the same trigger design for all baseline attacks to ensure the same level of data stealthiness for a fair comparison. We summarize the details of each attack in Appendix A.2. We also discuss different trigger designs of DBA [11] in Appendix B.9.
A3FL setup: By default, compromised clients optimize the trigger using Projected Gradient Descent (PGD) [45] with a step size of 0.01. The adversarial global model is optimized using SGD with a learning rate of 0.01. In practice, we set the balancing coefficient  $\lambda = \lambda_0\sin (\pmb{\theta}_t',\pmb{\theta}_t)$ , where  $\sin (\pmb{\theta}_t',\pmb{\theta}_t)$  denotes the cosine similarity between  $\pmb{\theta}_t'$  and  $\pmb{\theta}_t$ . We use similarity to automatically adjust the focus to the adversarial global model: if the adversarial global model is similar to the current global model, it will be assigned a higher weight; otherwise, the adversarial global model is assigned
Table 1: A3FL maintains the utility of the global model on CIFAR-10.
Defense	FedAvg	NC	RLR	Median	DSight	Bulyan	Krum	SFed	CRFL	DP	FedDF	FedRAD
ACC(%)	92.29	92.57	92.21	65.59	91.79	39.57	84.56	92.60	87.40	87.71	37.58	65.89
BAC(%)	92.44	92.61	92.26	65.53	91.79	39.92	84.41	92.70	87.35	87.60	40.09	65.61
a lower weight. We use the similarity to control the strength of adversarial training, since the backdoor could be fully unlearned if the adversarial global model is aggressively optimized, which makes it difficult to optimize the first term in Equation 3. In adversarial scenarios, it is important to balance the strengths of both sides to achieve better performance, which has been well studied in previous works in adversarial generation [46, 47]. When there are multiple compromised clients in  $S_{t}$ , the backdoor trigger is optimized on one randomly selected compromised client, and all compromised clients use this same trigger. We also discuss the parameter setting of A3FL in experiments.
Compared attack baselines: We compare our A3FL to four representative or state-of-the-art backdoor attacks to FL: Neurotoxin [12], DBA [11], CerP [9], and F3BA [10]. We discuss these baselines in Section 2 and also provide an in-detail introduction in Appendix A.2 including specific hyperparameter settings and trigger design of each baseline.
Compared defense baselines: We evaluate A3FL under 13 state-of-the-art or representative federated learning defenses: FedAvg [2], Median [22], Norm Clipping [17], DP [17], Robust Learning Rate [18], Deepsight [20], Bulyan [23], FedDF [24], FedRAD [25], Krum [21], SparseFed [26], FLAME [27], and CRFL [19]. We summarize the details of each defense in Appendix A.3.
Evaluation metrics: Following previous works [10, 12, 11, 8], we use accuracy & backdoor accuracy (ACC & BAC), attack success rate (ASR), and lifespan to comprehensively evaluate A3FL.
- ACC & BAC: We define ACC as the accuracy of the benign global model on clean testing inputs without any attacks, and BAC as the accuracy of the backdoored global model on clean testing inputs when the attacker compromises a part of the clients to attack the global model. Given the dynamic nature of the global model, we report the mean value of ACC and BAC. BAC close to ACC means that the evaluated attack causes little or no impact on the global model utility. A smaller gap between ACC and BAC indicates that the evaluated attack has higher utility stealthiness.
- ASR: We embed a backdoor trigger to each input in the testing set. ASR is the fraction of trigger-embedded testing inputs that are successfully misclassified as the target class  $\tilde{y}$  by the global model. In particular, the global model is dynamic in FL, resulting in an unstable ASR. Therefore, we use the average value of ASR over the last 10 communication rounds in the attack windows to demonstrate the attack performances. A high ASR indicates that the attack is effective.
- Lifespan: The lifespan of a backdoor is defined as the period during which the backdoor keeps effective. The lifespan of a backdoor starts at the end of the attack window and ends when the ASR decreases to less than a chosen threshold. Following previous works [12], we set the threshold as  $50\%$ . A long lifespan demonstrates that the backdoor is durable, which means the backdoor remains effective in the global model long after the attack ends. When we evaluate the lifespan of attacks, we extend the FL training process to 3,000 communication rounds.
4.2 Experimental Results
A3FL preserves the utility of the global model: To verify whether A3FL impacts the utility of global models, we compared their ACCs to BACs. The experimental results on CIFAR-10 are shown in Table 1, where NC denotes Norm Clipping, DSight represents Deepsight, and SFed represents SparseFed. Observe that the maximum degradation in accuracy of the global model caused by A3FL is only  $0.28\%$ . Therefore, we can conclude that A3FL preserves the utility of the global model during the attack, indicating that our approach is stealthy and difficult to detect. Similar results were observed in the experiments on TinyImagenet, which can be found in Appendix B.1.
A3FL achieves higher ASRs: The attack performances of A3FL and baselines on defenses designed for FL backdoors are presented in Figure 2. The experimental results demonstrate that A3FL achieves higher attack success rates (ASRs) than other baselines. For example, when the defense is Norm
(d) CRFL
(e) Deepsight
(f) DP
Figure 2: Comparing performances of different attacks on CIFAR-10.
Clipping and only one client is compromised, A3FL achieves an ASR of  $99.75\%$ , while other baselines can only achieve a maximum ASR of  $13.9\%$ . Other attack baselines achieve a comparable ASR to A3FL only when the number of compromised clients significantly increases. For instance, when the defense is CRFL, F3BA cannot achieve a comparable ASR to A3FL until 10 clients are compromised. We have similar observations on other defenses and datasets, which can be found in Figure 8 and 9 in Appendix B.2.
We note that CRFL assigns a certified radio to each sample and makes sure that samples inside the certified radio would have the same prediction. This is achieved by first clipping the updates  $\pmb{\Delta}_t^i$  and then adding Gaussian noise  $z\sim \mathcal{N}(0,\sigma^2 I)$  to  $\pmb{\Delta}_{t}^{i}$ . During the inference stage, CRFL adopts majority voting to achieve certified robustness. The strength of CRFL is controlled by the value of  $\sigma$ . We discuss the performance of CRFL under different values of  $\sigma$  in Appendix B.5.
A3FL has a longer lifespan: We evaluate the durability of attacks by comparing their lifespans. Recall that the attack starts at the 1,900th communication round and ends at the 2,000th communication round. Figure 3 shows the attack success rate against communication rounds when the defense is Norm Clipping and 5 clients are compromised. As we can observe, A3FL has a significantly longer lifespan than other baseline attacks. A3FL still has an ASR of more than  $80\%$  at the end, indicating a lifespan of over 1,000 rounds. In contrast, the ASR of all other baseline attacks drops below  $50\%$  quickly. We show more results on other defenses in Appendix B.3 and a similar phenomenon is observed. These experimental results suggest that A3FL is more durable than other baseline attacks, and challenge the consensus that backdoors in FL quickly vanish after the attack ends.
Figure 3: A3FL has a longer lifespan. The vertical dotted lines denote the end of the lifespans of each attack when the ASR of the backdoor drops below  $50\%$ . The dotted line at the 100th communication round denotes the end of all attacks.
(a) A3FL
(b) F3BA
(c) CerP
(d) DBA
(e) Neurotoxin
(a) Norm Clipping
Figure 4: Compare local ASR to global ASR.
(b) CRFL
(a) Norm Clipping
Figure 5: The impact of trigger size on the attack performance.
Figure 6: The impact of  $\lambda$  on the attack performance.
(b) CRFL
4.3 Analysis and Ablation Study
A3FL achieves higher ASR when transferred to the global model: As discussed in Section 3, A3FL achieves higher attack performance by optimizing the trigger and making the backdoor persistent within the dynamic global model. To verify our intuition, we conducted empirical experiments in which we recorded the Attack Success Rate (ASR) on the local model (local ASR) and the ASR on the global model after aggregation (global ASR). For the experiments, we used FedAvg as the default defense and included five compromised clients among all clients.
The results presented in Figure 4 demonstrate that A3FL can maintain a higher ASR when transferred to the global model. While all attacks can achieve high ASR ( $\approx 100\%$ ) locally, only A3FL can also achieve high ASR on the global model after the server aggregates clients' updates, which is supported by the tiny gap between the solid line (global ASR) and the dotted line (local ASR). In contrast, other attacks cannot achieve similarly high ASR on the global model as on local models. For instance, F3BA immediately achieves a local ASR of  $100\%$  once the attack starts. But it can only achieve less than  $20\%$  ASR on the global model in the first few communication rounds. F3BA also takes a longer time to achieve  $100\%$  ASR on the global model compared to A3FL. This observation holds for other baseline attacks. We further provide a case study in Appendix B.8 to understand why A3FL outperforms baseline attacks. In the case study, we observe that 1) A3FL has better attack performance than other baseline attacks with comparable attack budget; 2) clients compromised by A3FL are similarly stealthy to other trigger-optimization attacks. Overall, our experimental results indicate that A3FL is a more effective and persistent attack compared to baseline attacks, which makes it particularly challenging to defend against.
The impact of trigger size: We evaluate the performance of A3FL with a trigger size of  $3 \times 3$ ,  $5 \times 5$ ,  $8 \times 8$ ,  $10 \times 10$  respectively (the default value is  $5 \times 5$ ). Figure 5 shows the impact of trigger size on A3FL. In general, the attack success rate (ASR) improves as the trigger size grows larger. When the defense mechanism is Norm Clipping, we observe that the difference between the best and worst ASR is only  $1.75\%$ . We also observe a larger difference with stronger defenses like CRFL. Additionally, we find that when there are at least 5 compromised clients among all clients, the impact of trigger size on the attack success rate becomes unnoticeable. Therefore, we can conclude that smaller trigger sizes may limit the performance of A3FL only when the defense is strong enough and the number of compromised clients is small. Otherwise, varying trigger sizes will not significantly affect the performance of A3FL.
The impact of  $\lambda$ : Recall that  $\lambda = \lambda_0 \sin(\theta_t', \theta_t)$ . We varied the  $\lambda_0$  hyperparameter over a wide range of values to learn the impact of the balancing coefficient on attack performance and record results in Figure 6. Observe that different  $\lambda_0$  only slightly impact attack performances with fewer compromised clients. When there are more than 5 compromised clients, the impact of  $\lambda_0$  is unnoticeable. For
instance, when the defense is Norm Clipping, the gap between the highest ASR and the lowest ASR is merely  $0.5\%$ . We can thus conclude that A3FL is insensitive to variations in hyperparameter  $\lambda_0$ . We further provide an ablation study in Appendix B.4 for more analysis when the adversarial adaptation loss is disabled, i.e.,  $\lambda_0 = 0$ .
5 Conclusion and Future Work
In this paper, we propose A3FL, an effective and durable backdoor attack to Federated Learning. A3FL adopts adversarial adaption loss to make the injected backdoor persistent in global training dynamics. Our comprehensive experiments demonstrate that A3FL significantly outperforms existing backdoor attacks under different settings. Interesting future directions include: 1) how to build backdoor attacks towards other types of FL, such as vertical FL; 2) how to build better defenses to protect FL from A3FL.
[1] Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492, 2016.
[2] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[3] Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Trans. Intell. Syst. Technol., 10(2):12:1-12:19, 2019.
[4] Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604, 2018.
[5] David Leroy, Alice Coucke, Thibaut Lavril, Thibault Gisselbrecht, and Joseph Dureau. Federated learning for keyword spotting. In ICASSP 2019-2019 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 6341-6345. IEEE, 2019.
[6] Viraaji Mothukuri, Reza M Parizi, Seyedamin Pouriyeh, Yan Huang, Ali Dehghantanha, and Gautam Srivastava. A survey on security and privacy of federated learning. Future Generation Computer Systems, 115:619-640, 2021.
[7] KIMBERLY Powell. Nvidia clara federated learning to deliver ai to hospitals while protecting patient data. Accessed: Dec, 1, 2019.
[8] Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pages 2938-2948. PMLR, 2020.
[9] Xiaoting Lyu, Yufei Han, Wei Wang, Jingkai Liu, Bin Wang, Jiqiang Liu, and Xiangliang Zhang. Poisoning with cerberus: Stealthy and colluded backdoor attack against federated learning. 2023.
[10] Pei Fang and Jinghui Chen. On the vulnerability of backdoor defenses for federated learning. arXiv preprint arXiv:2301.08170, 2023.
[11] Chulin Xie, Keli Huang, Pin-Yu Chen, and Bo Li. Dba: Distributed backdoor attacks against federated learning. In International conference on learning representations, 2020.
[12] Zhengming Zhang, Ashwinee Panda, Linyue Song, Yaoqing Yang, Michael Mahoney, Prateek Mittal, Ramchandran Kannan, and Joseph Gonzalez. Neurotoxin: Durable backdoors in federated learning. In International Conference on Machine Learning, pages 26429-26446. PMLR, 2022.
[48] Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pages 226-231, 1996.
[49] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
(a) FLTrojan
(c) CerP
Figure 7: Trigger patterns of evaluated attacks on FedAvg, with  $\mathrm{P} = 2$  compromised clients.
(d) Neurotoxin
(e) DBA
A Additional Experiment Details
A.1 Experimental Setup in Figure 1
The preliminary experiment in Figure 1 has the same experimental setup as described in Section 4.1. In particular, We use FedAvg [2] as the server-side aggregation rule. We set the number of compromised clients  $P = 1$  in the preliminary experiment. We denote the attack success rate on the global model as global ASR. We further denote the ASR on the local model after local training as the local ASR. When the compromised client is selected by the server, we calculate and update the local ASR after the compromised client optimizes the backdoor trigger and trains its local model on the poisoned local training dataset.
A.2 Details of Attacks
A3FL: A3FL formulates the trigger optimization as a bi-level optimization problem. A3FL jointly optimizes the adversarial model  $f_{\boldsymbol{\theta}_t'}$  with the trigger pattern  $\Delta$ . A3FL optimizes the adversarial model using SGD with a learning rate of 0.01, a momentum of 0.9, and a weight decay of 0.0005. A3FL updates the trigger pattern using PGD with a step size of 0.01 until convergence. We show the trigger pattern of A3FL in Figure 7a.
F3BA [10]: F3BA directly manipulates a part of local model weights to inject the backdoor via sign flipping. F3BA further jointly optimizes the trigger pattern and the local model weights to maximize the difference between latent representations of clean and backdoored samples, thus achieving higher attack performance. The trigger of F3BA is a squared patch. We show the trigger pattern of F3BA in Figure 7b.
CerP [9]: CerP jointly optimizes the trigger pattern and the local model weights to improve the backdoor effectiveness. Furthermore, CerP aims to improve the backdoor stealthiness by adopting L2-norm regularization to limit the difference between local model weights and global model weights. Therefore CerP can tune the local model to fit the backdoor-poisoned data without inducing large biases in the local model weights. The trigger of CerP is shown in Figure 7c.
Neurotoxin [12]: Neurotoxin only updates unimportant model weights to avoid conflicts with other clean clients. The importance of model weights is determined by the magnitude of their gradients. Model weights with a higher gradient in previous rounds are considered to be more important (frequently updated by other clients). Following the settings in [12], we only update the last  $95\%$  important model weights. Neurotoxin uses a fixed trigger pattern, as shown in Figure 7d.
DBA [11]: DBA is a distributed backdoor attack designed to utilize the distributed nature of FL. DBA splits the trigger into different clients. Each client uses a different trigger to attack the FL system during the training stage. In the inference stage, the attacker uses the joint trigger to activate the injected backdoor. The trigger in [11] was designed as several parallel white lines placed at the upper left corner of the input images. This trigger design is not compatible with our attack setting and we can hardly control the attack budget introduced by the trigger following [11]. Therefore in our implementation, we also use a squared patch as the trigger for DBA, as shown in Figure 7e. We randomly split the squared patch into four sub-triggers and these sub-triggers are iteratively used during the attack.
A.3 FL defenses
Norm Clipping (NC) [17]: NC clips clients' updates that are larger than a pre-defined threshold. NC can effectively limit clients' behavior to prevent the global model from being overwhelmed by a few clients. By default, we set the threshold to 1.
(weak) Differential Privacy (DP) [17]: DP adds Gaussian noise  $z \sim \mathcal{N}(0, \sigma^2 I)$  to clients' updates to perturb carefully crafted malicious updates. Note that this defense is not designed for privacy, so the Gaussian noise is relatively smaller than that adopted in differential privacy. By default, we set  $\sigma = 0.002$ .
Robust Learning Rate (RLR) [18]: RLR aims to maximize the agreement on updating direction across clients to mitigate potential attacks. It is inspired by that the behavior of a compromised client is commonly different from other benign clients. For instance, a compromised client may want to enlarge some model parameters while most benign clients are trying to reduce them. When clients disagree on the updating direction of a parameter, RLR flips the learning rate on the parameter to maximize the loss instead.
CRFL [19]: CRFL adopts three techniques to mitigate backdoor attacks on FL. CRFL first clips clients' updates as Norm Clipping does. In our experiments, we set the clipping threshold as 1. CRFL then adds Gaussian noise  $z \sim \mathcal{N}(0, \sigma^2 I)$  to clients' updates as DP does. In our experiments, we set  $\delta = 0.002$  and we discuss the impact of  $\sigma$  on CRFL in Appendix B.5. Finally, CRFL creates several perturbed models by adding independently sampled Gaussian noise to the global model and adopts majority voting for prediction. In our experiments, CRFL creates 5 different perturbed models for prediction at each FL communication round.
Median [22]: Median uses the coordinate-wise median value of updates from all clients to update the global model. Median can effectively exclude clients that upload overwhelming updates. However, the Median tends to heavily degrade the model utility.