SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(a) Some malicious gradients evade defenses to be selected for aggregation.
(b) Benign outliers are misclassified as malicious gradients and removed from the aggregation, especially for Non-IID datasets (e.g., FEMNIST) with more outliers.
Moreover, benign outliers are intractable for all previous defenses. FLTrust is better than other defenses since its normalization limits the large magnitude gradients. In contrast, RECESS can
Figure 5: The impact of the local dataset's Non-IID degree on the FL defenses. The task is CIFAR-10. The model poisoning attack is AGR Min-Max. Our RECESS achieves the best and most stable defense effect.
Figure 6: The impact of the proportion of attackers on the FL defenses. The task is CIFAR-10. The model poisoning attack is AGR Min-Max. As the proportion increases, the accuracy of using our RECESS decreases minimally.
effectively distinguish benign outliers from malicious gradients, thus obtaining the highest accuracy, even in the most challenging white-box Non-IID case.
Defender's Goal (2): Model Performance. As shown in the first row in each set of Table 2 where no attacks are under consideration, robust rules will reduce the accuracy but have little impact on RECESS. This can be explained from three aspects:
(a) Due to the existence of attackers, only  $80\%$  benign clients participate in the aggregation compared with no attack (e.g. 40 vs 50 for CIFAR-10).
(b) Existing defenses discard benign gradients more or less, especially for benign outliers.
(c) Malicious gradients are selected for aggregation.
The first aspect is inevitable since the defender cannot control the attacker, which also causes the main accuracy gap between FedAvg and our RECESS. However, RECESS can improve the other two aspects by more accurately distinguishing whether gradients are malicious or benign.
Impact of Attacker's Knowledge. Different knowledge affects the attack, and RECESS outperforms other defenses in both white/black-box. The stronger the attack, the better the effect of RECESS. In the black-box case, the attacker cannot offset the effect of benign gradients on the aggregation without the knowledge of other benign gradients, so the poisoning is greatly weakened and RECESS is not prominent as in the white-box case.
Changing Trend of Trust Score. Figure 3 presents the trust score of benign and malicious clients in each round of RECESS detection. Here we execute the strongest AGR-agnostic Min-max attack. Figure 4 shows the histogram of each client's Abnormality value in the third detection round to illustrate the effectiveness of RECESS. RECESS can effectively distinguish malicious and benign users in both the optimization attack and AGR attacks.
4.3 Impact of FL Setting
In this part, we evaluate RECESS under various FL settings to illustrate the practicability of our proposed scheme.
Non-IID Degree of Dataset (More Outliers). The results on Non-IID FEMNIST in Table 2 show the effectiveness of our RECESS against latest poisoning attacks. Besides, we also involve the CIFAR-10 in the evaluation using the Non-IID dataset construction method same as [11, 12]. We consider four defenses (the best classic robust rules Trmean, SOTA FLtrust and DnC, and RECESS) against the strongest AGR-agnostic Min-Max attack in white/black-box cases.
Figure 5 shows that the increase in the Non-IID degree leads to an expansion of benign outliers which are similar to malicious gradients, and previous defenses cannot effectively distinguish outliers from malicious gradients. However, it has little impact on RECESS, since our method can accurately detect malicious gradients and benign outliers.
Table 3: The impact of cross-device FL on the defenses. The model poisoning attack is AGR Min-Max.
Dataset	Attacker's Knowledge	Attacks	Defences
Trmean	FLTrust	DnC	RECESS
CIFAR-10	White-box	No Attack	0.6490	0.6178	0.6284	0.6448
LIE	0.5559	0.6101	0.6157	0.6228
Optimization attack	0.5655	0.6171	0.5913	0.6439
AGR-tailored	0.5080	0.4538	0.5915	0.6046
AGR-agnostic Min-Max	0.5548	0.5226	0.5478	0.6379
AGR-agnostic Min-Sum	0.5758	0.5964	0.5793	0.6393
Black-box	LIE	0.5999	0.6137	0.6128	0.6353
Optimization attack	0.6168	0.6159	0.6209	0.6277
AGR-tailored	0.5509	0.5485	0.5956	0.6142
AGR-agnostic Min-Max	0.5833	0.5715	0.6095	0.6416
AGR-agnostic Min-Sum	0.6068	0.5937	0.6074	0.6385
Table 4: FL accuracy with enhanced RECESSS defenses against the adaptive poisoning attack. The task is FEMNIST. The detection frequency is controlled by  $TS_0$  and baseline_decreased_score.
Attacker's Knowledge	Attacks	RECESS Detection Frequency
0	10	50	100	200
white-box	no attack	0.8235	0.823	0.8221	0.8224	0.8215
optimization attack	0.2182	0.7983	0.8048	0.8106	0.8114
adaptive optimization attack	0.2482	0.3456	0.6847	0.7815	0.8011
black-box	optimization attack	0.5152	0.7937	0.8045	0.8117	0.8048
adaptive optimization attack	0.5248	0.6481	0.7847	0.8048	0.8148
Number of Malicious Clients. Figure 6 shows that RECESS remains an outstanding defending effect all along, as the number of malicious clients increases, all poisoning attacks are more powerful, while the defending effect of other defenses decreases sharply. We vary the proportion of malicious clients from  $5\%$  to  $25\%$ , consistent with [12]. The other settings remain the same as Section 4.3.
Cross-device FL. Previous evaluations are mostly under cross-silo settings except for the FEMNIST which is naturally cross-device shown in Table 2. Further, we consider the cross-device setting using the dataset CIFAR-10. In each epoch, the server stochastically selects 10 updates from all clients to aggregate. The attack steps remain the same, and the other settings are the same as the default.
Table 3 shows that similar to the result of cross-silo, RECESS still achieves the best defense effect. Besides, the poisoning impacts under the cross-device setting are lower than the one of the cross-silo setting, because the server selects less number of clients for aggregation and ignores more malicious gradients in cross-device FL, thus the attacker cannot continuously poison, weakening the impact of the poisoning. Consequently, this leaves less improvement space for RECESS and other defenses.
4.4 Adaptive Attacks
Active Evasion. With knowledge of RECESS, the attacker checks if it's a test gradient before deciding to poison or not. The adaptive strategy involves estimating an aggregated gradient  $g_{p}$  in each iteration from controlled benign clients. Then, the attacker compares the received aggregated gradient  $g_{agg}$  with  $g_{p}$  and  $g_{i}^{k - 1}$  from the last iteration. If  $S_{C}(g_{agg}, g_{i}^{k - 1}) \geq S_{C}(g_{agg}, g_{p})$ , the poisoning begins, while if  $S_{C}(g_{agg}, g_{i}^{k - 1}) < S_{C}(g_{agg}, g_{p})$ , the poisoning stops, as it's considered detected.
As a countermeasure, the defender boosts detection frequency and intersperses it during the entire training, instead of fixed consecutive detection, and increases the sensitivity of parameters. Note that the defense should aim to make the model converge, it is not necessary to find the attacker. In other words, RECESS's presence increases the attack cost, so even if the attacker evades detection, the loss of model accuracy is negligible, which is acceptable.
Table 4 shows that the adaptive attack has little impact on the final model accuracy though it is stealthier than the continuous attack. This is due to the similarity between the test gradient and the normal aggregated gradient, leading to inaccurate estimation by the attacker and a decrease in aggregation weight. Also, increasing detection frequency lowers poisoning frequency, so the poisoning impact will be gradually alleviated by normal training over time.
Poisoning before the Algorithm Initialization. The detection basis is that the variance of the malicious gradient is larger than the benign gradient. Malicious gradients, solved by the optimization problem, are more inconsistent. It is also theoretically proven that the optimization problem amplifies this variance. Thus, RECESS is unaffected by initial poisoned gradients as it does not change this basis. Besides, when the poisoned aggregated gradient was used to detect, most clients will be identified as malicious. This violates the assumption of Byzantine robustness that requires more than  $51\%$  of users to be benign. Hence, the defender can easily discern a potentially poisoned initial gradient based on the new abnormality definition of RECESS.
Inconsistent attacks. Attackers can also first pretend as benign client to increase the trust score, and then provide the poisoning gradients. However, RECESS detects such behavior inconsistencies over time. The trust scoring of RECESS also incorporates delayed penalties for discrepancies between a client's current and past behaviors (shown in Section 3.3). Additionally, three factors limit intermittent attacks' impact:
(a) FL's self-correcting property means inconsistent poisoning is much less impactful. Attackers would not take this approach in practice.
(b) In real settings, clients participate briefly, often just once. Attackers cannot afford to waste rounds acting benign before attacking.
(c) Defenses aim to accurately detect malicious updates for model convergence. Even if poisoning temporarily evaded detection, attack efficacy would diminish significantly, making it no longer a serious security concern.
This work primarily focuses on untargeted model poisoning, along with previous SOTA works. We believe that model poisoning has a significant impact on real-world FL deployments for three reasons:
(a) Model poisoning through direct gradient manipulation poses a greater threat than backdoors.
(b) Backdoors are considered special cases of model poisoning, as malicious gradients can be obtained on poisoning datasets. However, it is challenging to solely imitate model poisoning gradients through training backdoors.
(c) Untargeted poisoning has a more severe impact on overall model performance degradation compared to backdoors in FL.
As RECESS detects inconsistencies between client behaviors, it could mitigate targeted attacks from clients with backdoored data by deploying additional strategies. Due to space limitation, we refer the readers to our technical report [20] for more details.
In this work, we proposed RECESS, a novel defense for FL against the latest model poisoning attacks. We shifted the classic reactive analysis to proactive detection and offered a more robust aggregation mechanism for the server. In the comprehensive evaluation, RECESS achieves better performance than SOTA defenses and solves the outlier detection problem that previous methods can not handle. We anticipate that our RECESS will provide a new research direction for poisoning attack defense and promote the application of highly robust FL in practice.
We thank all the anonymous chairs and reviewers for their valuable guidance and constructive feedback. Hui Li is partially supported by the National Natural Science Foundation of China (61932015), the National Key Research and Development Program of China (2022YFB3104100), Shaanxi innovation team project (2018TD-007), and Higher Education Discipline Innovation 111 project (B16037). Wenjing Zhang and Xiaodong Lin are partially supported by Natural Sciences and Engineering Research Council of Canada (NSERC). Part of Haonan Yan's work is done when he visits the School of Computer Science at the University of Guelph.
[19] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.
[20] Haonan Yan, Wenjing Zhang, Qian Chen, Xiaoguang Li, Wenhai Sun, Hui Li, and Xiaodong Lin. Recess vaccine for federated learning: Proactive defense against model poisoning attacks. https://arxiv.org/abs/2310.05431.
[21] Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pages 2938-2948. PMLR, 2020.
[22] Saeed Mahloujifar, Mohammad Mahmoody, and Ameer Mohammed. Universal multi-party poisoning attacks. In International Conference on Machine Learning, pages 4274-4283. PMLR, 2019.
[23] Cong Xie, Oluwasanmi Koyejo, and Indranil Gupta. Fall of empires: Breaking byzantine-tolerant sgd by inner product manipulation. In Uncertainty in Artificial Intelligence, pages 261-270. PMLR, 2020.
[24] Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, and Bo Li. Manipulating machine learning: Poisoning attacks and countermeasures for regression learning. In 2018 IEEE Symposium on Security and Privacy (SP), pages 19-35. IEEE, 2018.
[25] Luis Muñoz-González, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongras-samee, Emil C Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back-gradient optimization. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pages 27-38, 2017.
[26] Xueluan Gong, Yanjiao Chen, Qian Wang, and Weihan Kong. Backdoor attacks and defenses in federated learning: State-of-the-art, taxonomy, and future directions. IEEE Wireless Communications, 2022.