SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our empirical study aims to highlight the effectiveness of the IBA attack against the state-of-the-art (SOTA) FL defenses. We conduct our experiments on real-world datasets and a simulated FL environment. We design three experiment sets: (1) IBA under different attack scenarios, (2) IBA under different FL defenses, and (3) IBA's imperceptibility and durability evaluation. Our results demonstrate that the IBA can achieve significant efficiency, stealthiness, and durability. In particular, IBA attains persistent performance with/without SOTA defenses and maintains higher visual imperceptibility and longer durability compared with existing artificial backdoor attacks [32, 2]. Our implementation is publicly available to reproduce experimental results.
Dataset and Backdoor Tasks. The IBA method is evaluated on three classification datasets: MNIST, CIFAR-10, and Tiny ImageNet. We simulate heterogeneous data partitioning by sampling  $p_k \sim Dir_K(0.5) / Dir_K(0.01)$  for MNIST, CIFAR-10/Tiny ImageNet and allocating a proportion of each class to participating clients. We follow the established protocol outlined in previous works [28, 32], employing stochastic gradient descent (SGD) optimization with  $E$  local epochs, a local learning rate of  $l_r$ , and a batch size of 32. The number of clients selected in each round is 10/200. The details of the datasets and other parameter configurations are presented in the supplementary material<sup>3</sup>.
Trigger Generation and Backdoored Model Training. The generation of triggers is accomplished through the utilization of the U-Net architecture [21], incorporating specific parameter configurations.
<sup>3</sup>The supplementary material is available at https://openreview.net/attachment?id=cemEOP8YoC&name=supplementary_material
Table 1: Effectiveness of IBA under different strategies in fixed-frequency attacks setting  $\left( {f = {10}}\right)$
Dataset	Baseline (w/o attack)	FedAvg	FedProx	FedNova
MA	BA	MA	BA	MA	BA	MA	BA
CIFAR-10	84.71%	9.52%	84.49%	87.13%	84.62%	86.59%	83.35%	94.32%
MNIST	99.74%	9.25%	98.11%	99.07%	97.98%	99.97%	98.32%	92.74%
Tiny ImageNet	65.15%	0.56%	64.36%	94.19%	65.02%	92.52%	64.17%	93.11%
The constraint values are set to  $\epsilon = 0.3, \hat{\epsilon} = 0.05$ ; while  $\alpha = 0.5$  and  $\beta = 0.5$  define additional parameter values. During trigger training, the threshold for backdoor accuracy (BA) is established as  $local_{BA} = 0.85$ . To facilitate the generation of updates, the learning rate for the update generation model is determined as  $\gamma_{A} = 0.0001$ , and for the update classifier model, the learning rate is set to  $\eta = 0.01$ . Additionally, the performance of IBA is evaluated in conjunction with prominent FL strategies, namely FedAvg [13], FedProx [11], and FedNova [29]. This comprehensive evaluation aims to assess the effectiveness of IBA in the context of different FL methodologies.
Participating patterns of attackers. In addition to the fixed-frequency case described in the threat model, we further evaluate the efficiency of IBA under a more favorable condition known as fixed-pool case (or random sampling), where there is a fixed pool of clients are controlled by  $\mathcal{A}$ . The result for the fixed-pool case is left in the appendix.
FL Backdoor Defenses. We examine six SOTA defense techniques designed to mitigate backdoor attacks in FL: (i) NDC [25], (ii) KRUM and (iii) Multi-KRUM [3], (iv) RFA [19], (v) RLR [18], (vi) FoolsGold [8]. By evaluating the performance of these defense techniques, we aim to provide insights into their effectiveness in countering backdoor attacks in the FL setting.
(1) IBA under different attack scenarios. Firstly, we study the performance of IBA under fixed-frequency attacks with three datasets: MNIST, CIFAR-10, and Tiny ImageNet. The evaluation of main accuracy (MA) and backdoor accuracy (BA) from Table 1 reveals interesting insights. In the absence of the attack, the MAs of the baseline are  $99.74\%$ ,  $84.71\%$ , and  $65.15\%$  for MNIST, CIFAR-10, and Tiny ImageNet, respectively, indicating accurate classification on clean samples, while the BA remains low at  $0.56\%$  to  $9\%$ , indicating no backdoor effect on global model. Under IBA attack with FedAvg aggregator, the corresponding MAs are negligibly affected, while the BAs are notably high, i.e.,  $99.07\%$  for MNIST dataset. This indicates that IBA can stealthily insert a backdoor function without negatively impacting the model's behavior with benign samples.
In FL, many existing backdoor attacks target vulnerabilities within the FedAvg aggregator since the contribution of each client is considered equally. Therefore, we examined IBA's resilience across different aggregation methods, specifically FedAvg, FedProx, and FedNova, as presented in Table 1. FedProx, with its proximal term, enhances model accuracy and is beneficial in scenarios with non-IID data distributions. Besides, FedNova, designed to address objective inconsistency in FL, ensures consistent and accurate model updates even with varying update frequencies from clients, making it suitable for scenarios with heterogeneous data.
The results demonstrate the consistent efficacy of IBA with three datasets and three FL aggregators. In particular, IBA persistently obtained significant BAs, with the lowest BA being  $87.13\%$  and the highest being  $99.77\%$ , demonstrating its efficacy and potential challenges in diverse FL settings.
Fixed-pool attacks. We study the performance of IBA under fixed-pool attack setting (percentage of attackers in the overall clients varies from  $5\%$  to  $20\%$ ). The results are shown in Figure 2, demonstrating that a higher compromising ratio leads to a better backdoor accuracy or more substantial backdoor effect.
(2) IBA under different FL defenses. We study the effectiveness of standalone IBA and hybrid attacks (i.e., IBA and partial model poisoning) against the aforementioned defense techniques. As demonstrated in
Figure 2: IBA under fixed-pool attacks with different compromising ratios.
(a) Main Accuracy
(b) Backdoor Accuracy
Figure 3: Effectiveness of stand-alone IBA against mainstream defenses for FL on CIFAR-10 dataset.
Figure 4: The improved stealthiness of IBA when combined with partial model poisoning attacks under various defenses for the CIFAR-10 dataset. The attack is conducted from round 0 with fixed-frequency attacks of 10.
Figure 3, the stand-alone IBA can circumvent NCA, Multi-Krum, and FoolsGold and attain a high backdoor accuracy (i.e.,  $95.11\%$ ,  $79.5\%$  and  $76.46\%$  for CIFAR-10 dataset, respectively). Krum is the most challenging defense since it replaces the global model with only the most representative local model (i.e., the one with the shortest distance to its neighbors) in each round. As a result, the global model experiences substantial fluctuation over rounds and does not fully capture the knowledge of all participants, i.e., a significant decline in primary task accuracy in Figure 3. This mechanism hampers
Table 2: Robustness of IBA and the combination of IBA and partial model poisoning under mainstream backdoor defenses in FL
Dataset	Defense	IBA (Standalone)	IBA + PGD	IBA + PGD + MR
MA	BA	MA	BA	MA	BA
CIFAR-10	RFA	81.97%	98.03%	80.71%	96.7%	81.25%	98.17%
NCA	84.64%	95.11%	84.77%	98.73%	84.21%	99.12%
Krum	81.65%	27.07%	81.59%	50.33%	81.44%	65.98%
Multi-krum	82.40%	79.5%	83.78%	97.16%	83.25%	95.02%
RLR	83.91%	63.94%	82.59%	65.14%	82.85%	47.07%
FoolsGold	84.90%	76.46%	85.02%	76.78%	82.90%	84.36%
No-defense	84.28%	85.63%	84.35%	83.80%	83.81%	97.79%
MNIST	RFA	98.04%	100.0%	98.89%	99.64%	98.04%	100.0%
NCA	98.66%	100.0%	98.97%	99.69%	98.62%	100.0%
Krum	97.60%	14.13%	97.82%	87.48%	97.44%	17.29%
Multi-krum	97.98%	81.27%	98.97%	98.82%	99.01%	80.18%
RLR	90.91%	12.60%	87.48%	18.74%	90.03%	29.97%
FoolsGold	98.87%	80.08%	99.10%	96.43%	99.05%	99.83%
No-defense	99.98%	98.03%	98.81%	99.73%	98.84%	99.97%
(a) IBA, MNIST
(b) DBA, MNIST
(c) IBA, CIFAR-10
Figure 5: Performance under GradCam heat maps of two attack schemes: IBA and Patched-BA with MNIST and CIFAR-10 datasets. The first two rows in each image represent the benign samples, and the last two rows represent the backdoored ones.
(d) DBA, CIFAR-10
the attack model's learning process since it is closely associated with global model variance. Figure 4 illustrates the improved performance of IBA against challenging defenses when combined with partial model poisoning, i.e., PGD-based. As observed, the IBA + PGD (w./wo.) model replacement can help improve the backdoor accuracy from  $10\%$  to  $20\%$ . Significantly, the improved IBA can achieve almost  $100\%$  of backdoor accuracy for Multi-Krum defense. The more detailed result can be seen in Table 2. The proposed IBA can bypass most existing defenses and achieve significant backdoor accuracy. In most cases, the combination of IBA + PGD/ IBA + PGD + MR helps improve the backdoor accuracy from  $10\%$  to  $20\%$ . While Krum and RLR defend the proposed attack to some extent, i.e., we observe the drop in the BAs of IBA, we argue that Krum and RLR tend to reject previously unseen information from both adversary and honest clients, which results in low MAs (a drop of  $3 - 10\%$ ). Specifically, the proposed IBA causes RLR to compensate for the backdoor defense's effectiveness with the main accuracy drop in the MNIST dataset by unnecessarily reversing the learning rate of specific dimensions, i.e., the BAs are around  $90\%$  (less than other defenses  $10\%$ ). Therefore, these two defenses raise great concern about its practicability under this scenario.
(b) CIFAR-10
Figure 7: Anomaly Indexes of IBA, DBA backdoor attacks in FL.
Figure 6: Durability comparison of IBA and DBA with MNIST and CIFAR-10 datasets. The adversary is removed from rounds 200 and 400 training, respectively.
Table 3: The decay rate of DBA and IBA after removing the backdoor attack (i.e., from the stopping round). The column (\%) indicates the current backdoor effect as a percentage of its peak value at the stopping round.
Dataset	Method	Stopping round	50 rounds later	100 rounds later	150 rounds later	200 rounds later	250 rounds later
BA	(%)	BA	%	BA	(%)	BA	(%)	BA	(%)	BA	(%)
MNIST	IBA	99.94	100.00	99.84	99.90	99.52	99.57	99.46	99.52	99.18	99.24	99.08	99.14
DBA	99.70	100.00	98.44	98.74	91.44	91.71	79.88	80.12	65.04	65.24	38.64	38.76
CIFAR-10	IBA	87.51	100.00	86.22	98.53	85.32	97.50	84.89	97.00	83.19	95.06	83.16	95.03
DBA	69.63	100.00	65.55	94.14	63.59	91.32	55.39	79.55	55.07	71.91	42.55	61.11
(3) IBA's imperceptibility and durability evaluation. Besides being imperceptible to the backdoor defenses, IBA is expected to be visually imperceptible and deliver prolonged backdoor durability. The visual imperceptibility of IBA is illustrated in Figure 5. We evaluate the behavior of backdoor-injected models against GradCam [23], a widely known visualization tool and helpful in understanding the neural network's behavior. Figure 5 shows that the visualization heat maps created by IBA attacks are nearly identical across the clean and backdoor images. From Figures 5, we can observe that the
backdoors images have negligible deviations from the Grad-CAM behaviors on the clean images for both black-and-white and colored images. On the other hand, backdoors images using patched triggers generate a considerable difference in visualization heat maps (DBA [32]). Because IBA is based on input perturbation, the image is formed by adding very little perturbation to the clean image; consequently, the difference in the latent space where the heat maps are generated is likewise minimal. Aside from that, the patch backdoor (such as the pattern in the upper-left corner) is apparent, resulting in a significant distinction between the backdoor and clean versions. We also evaluate the backdoor generated by IBA and another attack scheme in FL, i.e., DBA with Neural Cleansse [27], a widely-used backdoor model mitigation method based on the pattern optimization approach. Neural Cleansse assumes that the backdoor trigger is patch-based, which makes it suitable for evaluating the proposed method. For each image label, Neural Cleansse identifies if there is a patch pattern that produces a misclassification result for that target label. If any class label yields a significantly smaller pattern, Neural Cleansse considers it a sign of a potential backdoor. Neural Cleansse quantifies such deviations from the optimal patch of each class label by using the Anomaly Index metric. If the Anomaly Index is less than a threshold of 2 for a class, Neural Cleansse considers that there is a backdoor with this class as the target label. Since IBA does not generate the patch-based trigger pattern, it can efficiently bypass this defense by having Anomaly Index much lower than DBA (Figure 7), which indicates the superior stealthiness of backdoor attacks with adversarial triggers compared to the patch-based.
As previously stated, strong attacks should be successful even if the attacker only participates a few times. As a result, we construct a scenario to assess the endurance of IBA compared to a chosen baseline: DBA. We employ the same configuration as the original work DBA but limit each of the four malicious clients' participation frequency to 10 without utilizing any scaling strategy. Regarding IBA, we apply the dimension-based model poisoning mentioned above and use one malicious client with the same frequency. As observed in Figure. 6a, after  $\mathcal{A}$  is removed from the training (i.e., at round 200), the BA of DBA reduces gradually while the accuracy drop of IBA is almost negligible. As demonstrated in Figure 6b and Table 3, IBA delivers superior backdoor efficiency and durability within the same attacking duration. After round 400, the BA of DBA drops quickly compared to IBA. We can see the most significant difference towards the end of the comparison duration, i.e., at round 700, the BAs of IBA and DBA are  $\sim 85\%$  and  $\sim 45\%$ , respectively. After 250 rounds of elimination, the BA of DBA retains  $38.76\%$  of its peak value, but the comparable value of IBA is  $99.14\%$ . Furthermore, while the BA can preserve more than  $60\%$  of its peak value with the CIFAR-10 dataset, the highest accuracy it can attain is significantly lower than IBA, i.e., less than  $20\%$ , which is consistent with the prior conclusion of the original when the scaling factor is 1. In conclusion, IBA can achieve better durability and imperceptibility than SOTA attacks.
In conclusion, we have presented the IBA, a novel backdoor attack framework for FL that overcomes the limitations of existing approaches. Our framework combines a learnable trigger-generating function with a gradual implantation process, resulting in an attack that can evade detection and achieve high success rates. The selective poisoning of model parameters and the constraint on poisoned updates enhance the attack's efficiency and durability. Our work demonstrates promising results and suggests several directions for future research. Robust defense mechanisms against FL backdoor attacks must be developed, considering the unique challenges of the distributed and decentralized learning process. Generalizing our approach to diverse FL settings, such as heterogeneous or non-i.i.d. data distributions, would provide a comprehensive understanding of the attack's effectiveness. Exploring alternative triggers and their impact on stealthiness and success rates could yield valuable insights. Evaluating the framework on larger-scale FL systems and real-world scenarios will assess its scalability and practicality. Advancing adversarial defense strategies, including proactive detection and mitigation techniques, will contribute to more secure and trustworthy FL systems. These future research directions will contribute to strengthening the security of FL against backdoor attacks.
[29] Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H. Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[30] Chen Wu, Xian Yang, Sencun Zhu, and Prasenjit Mitra. Mitigating backdoor attacks in federated learning. ArXiv preprint, abs/2011.01767, 2020.
[31] Chulin Xie, Minghao Chen, Pin-Yu Chen, and Bo Li. CRFL: certifiably robust federated learning against backdoor attacks. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 11372-11382. PMLR, 2021.
[32] Chulin Xie, Keli Huang, Pin-Yu Chen, and Bo Li. DBA: distributed backdoor attacks against federated learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[33] Jiale Zhang, Junjun Chen, Di Wu, Bing Chen, and Shui Yu. Poisoning attack in federated learning using generative adversarial nets. In 2019 18th IEEE International Conference On Trust, Security And Privacy In Computing And Communications/13th IEEE International Conference On Big Data Science And Engineering (TrustCom/BigDataSE), pages 374-380. IEEE, 2019.
[34] Zhengming Zhang, Ashwinee Panda, Linyue Song, Yaoqing Yang, Michael W. Mahoney, Prateek Mittal, Kannan Ramchandran, and Joseph Gonzalez. Neurotoxin: Durable backdoors in federated learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 26429-26446. PMLR, 2022.
[35] Shihao Zhao, Xingjun Ma, Xiang Zheng, James Bailey, Jingjing Chen, and Yu-Gang Jiang. Clean-label backdoor attacks on video recognition models. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 14431-14440. IEEE, 2020.