RECESS Vaccine for Federated Learning: Proactive Defense Against Model Poisoning Attacks
Authors: Haonan Yan
Wenjing Zhang
Qian Chen
Xiaoguang Li
Wenhai Sun
HUI LI
Xiaodong Lin
Venue: NeurIPS 2023 poster (2023)
Keywords: Federated Learning
Model Poisoning Attacks
Proactive Detection
Robust Aggregation
Benign Outlier Identification
TL;DR: A new FL defense performs best in defending against the latest model poisoning attacks and solves the intractable problem of benign outlier identification.

Abstract: Model poisoning attacks greatly jeopardize the application of federated learning (FL). The effectiveness of existing defenses is susceptible to the latest model poisoning attacks, leading to a decrease in prediction accuracy. Besides, these defenses are intractable to distinguish benign outliers from malicious gradients, which further compromises the model generalization. In this work, we propose a novel defense including detection and aggregation, named RECESS, to serve as a “vaccine” for FL against model poisoning attacks. Different from the passive analysis in previous defenses, RECESS proactively queries each participating client with a delicately constructed aggregation gradient, accompanied by the detection of malicious clients according to their responses with higher accuracy. Further, RECESS adopts a newly proposed trust scoring based mechanism to robustly aggregate gradients. Rather than previous methods of scoring in each iteration, RECESS takes into account the correlation of clients’ performance over multiple iterations to estimate the trust score, bringing in a significant increase in detection fault tolerance. Finally, we extensively evaluate RECESS on typical model architectures and four datasets under various settings including white/black-box, cross-silo/device FL, etc. Experimental results show the superiority of RECESS in terms of reducing accuracy loss caused by the latest model poisoning attacks over five classic and two state-of-the-art defenses.

RECESS Vaccine for Federated Learning: Proactive Defense Against Model Poisoning Attacks
Haonan Yan Wenjing Zhang', Qian Chen', Xiaoguang Li!*
Wenhai Sun', Hui Li Xiaodong Lin²
Xidian University, University of Guelph, 'uurdue University com
yanhaonan. yanhsonan.seco@mail.. sec@gmail
Abstract
Model poisoning attacks greatly jeopardize the application of federated learning (FL). The effectiveness of existing defenses susceptible to the latest model poisoming attacks, leading to a decrease in prediction accuracy. Besides, these defenses are intractable to distinguish benign outliers from malicious gradients, which further compromises the model generalization. In this work, we propose a novel defense including detection and aggregation, named RECESS, to serve as a "vaccine" for Fl. against model poisoning attacks. Different from the passive analy- sis in previous defenses, RECESS proactively queries each participating client with delicately constructed aggregation gradient, accompanied by the detection malicious clients according to their responses with higher accuracy. Further, RECESS adopts newly proposed trust scoring based mechanism to robustly aggregate gradients. Rather than previous methods of scoring in each iteration, multiple iterations to estimate the trust score, bringing in a significant increase in detec- tion fault tolerance. Finally, we extensively evaluate RECESS typical model architectures and four datasets under various settings including white/black-bong box, cross-silo/device FI., etc. Experimental results show the superiority of RECESS in terms of reducing accuracy loss caused by the latest model poisoning attacks over five classic and two state-of-the-a defenses.
RECESS takes into account the correlation of clients' performance
Introduction
Background and Problem. Recently, federated learning (FL) goes viral privacy-preserving training solution with the distributed learning paradigm [1. since data privacy attracts increasing attention from organizations like banks (2 and hospitals (3) governments like GDPR and CPPA [5 and commercial companies like Google [6]. FL. allows data owners to collaboratively train models under the coordination of central server for better prediction performance by sharing local gradient updates instead of their own private/proprietat participant's 's raw data. FL promising as a trending privacy training technology. However, FL is vuinerable to various model poisoning attacks [7] 8). The distributed structure of FL characterized by the rrvaacypreseeving local training paradigm makes it impossible to verify overall gradient updates and distributions of local datasets. Accordingly, the attacker corrupt the global model by hijacking compromised participants, and then uploading malicious local gradient updates, Icading to a reduction in the final model's 's prediction performance, which significantly jeopardizes the application of FL.
datasets, preserving the privacy of each
Cooresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Server with previous defenses
Server with our RECESS
Malicious
gradient g
e-th Client
Statistically analyze the uploaded gradient set {go 8c Sc+i. ... ga)
gc
&c
Construct test gradient ench client, e.g & to e-th client
Collect the response, e.g. & from o-th elient
Benign (c+ Tyth Client
Be+1
2. Find abnormal gradients, e.g.. g and g.O is malicious,
Compare each pair to find malicious gradienis, e.g.
8c+1
g is benign.
[& 8c) is malicious C [R 8++1 is henign { is benign
3. Remove malicious graidents robustly aggregate to get aggregation result
Benign 4-th Outlier
8
8n
Use scoring based mechnism get
8n
Figure I: An intuitive example to illustrate the difference between previous defenses and RECESS. In an FL system with clients, suppose the first clients are malicious. Previous defenses find the malicious gradients by abnormality detection of all received gradients, while RECESS confirms the malicious gradients by comparing the constructed test gradient with the corresponding client's response. Thus, RECESS can distinguish between malicious clients and benign outliers, achieving higher accuracy in the final trained model.
Limitations of Previous Works. To mitigate model poisoning attacks, many byzantine-robust aggregation rules, e-g., Krum, Mkrum, Trmean, Median, and Bulyan p TO 7. are proposed to remove malicious gradients by statistical analyses. Two state-of-the-an (SOTA) defenses, FL.Trust [ and DnC [12). are also proposed to further enhance the robustness of the FL system. Even so, (Limitation 1:) these defenses are susceptible to the latest optimization-basee model poisoning attacks 0 12), leading to a decrease in prediction accuracy.
Besides, (Limitation 2:) these defenses cannot distinguish malicious gradients and benign outliers, reducing the generalization performance of the final trained model. Considering that the local datasets across participating clients are Non-IID, is common to have many biased updates which statistical outliers. These statistical outliers are still benign updates (called benign oulliers) which are helpful in improving the generalization performance of the final FL model. However, existing defenses typically rely statistical majorities remove malicious clients, which also implicates these benign outliers. To the best of our knowledge, the work of tackling benign outliers still remains open in FL.
Our work. To improve model accuracy and generalization against latest model poisoning attacks in FL., we propose RECESS acting as "vaccine" help FL. pRoactively dEteCt modEl poiSoning attackS, Unlike previous defenses using passive analysis and direct aggregation in single iteration, RECESS proactively detects malicious clients and robustly aggregates gradients with a new trust scoring based mechanism. The differences are illustrated in Figure [ In proactive detection, our key insight is that the update goals of malicious clients are different from benign clients and outliers. amplify this difference by delicately constructing aggregation gradients
Therefore, the defender sent to clients, and then more accurately detect malicious clients and distinguish benign outliers by the comparison between constructed gradients and the corresponding responses. In robust aggregation, we propose a new trust scoring based mechanism which estimates the score according to the user's performance over multiple iterations, rather than scoring each iteration in previous methods, which improves the detection fault tolerance. Finally, we compare RECESS with five classic and two SOTA defenses under various settings, and the results demonstrate that RECESS effective to overcome the two limitations aforementioned.
Contribution.
The main contributions are:
(a) We propose novel defense against model poisoning attacks called RECESS. It offers new defending angle for FL. and turns the defender from passive analysis to proactive detection, which defeats the latest model poisoning attacks. RECESS can also identify benign outliers effectively,

---

(b) We improve the robust aggregation mechanism. A new trust scoring method is devised by considering clients' abnormality over multiple iterations. which significantly increases the detection fault tolerance of RECESS and further improves the model's s accuracy.
(c) We evaluate RECESS under various settings, including white/black-boxx Non-IID degree of the dataset, number of malicious clients, and cross-silo/device FL. Experimental results show that RECESS outperforms previous defenses in terms of accuracy loss and achieves consistent effectiveness.
2 Related Works
2.1 Latest Model Poisoning Attacks in FL
In this work, we focus on the stronger untargeted model poisoning attacks for three reasons:
(a) Model poisoning attacks with the direct manipulation of gradients more threatening than data poisoning attacks in FL 13).
(b) Data poisoning attacks are considered special case of model poisoning attacks as malicious gradients be obtained on poisoning datasets (1T].
(c) Untargeted poisoning a more severe threat to model prediction performance than the targeted one in FL [12]
In the following, we introduce three latest model poisoning attacks.
LIE Attack. The LIE attack [14 lets the defender remove the non-byzantine clients and shift the aggregation gradient by carefully crafting byzantine values that deviate from the correct ones as far as possible.
propose new attack idea. They formulate the model poisoning attack as an optimization program, where the objective function is to let the poisoning aggregation gradient be far from the aggregation gradient without poisoning. By leveraging the halving search, they obtain crafted malicious gradient to poison.
Optimization Attack. 13
AGR Attack Series. [12 then improve the optimization program by introducing perturbation vectors and scaling factors. Then three instances (AGR-tailored, aGR-agnostic Min-Max, and Min-Sum) are proposed, which maximize the deviation between benign gradients and malicious gradients.
2.2 Existing SOTA Defenses
There are two directions to defend against the model poisoning attack: robust aggregation and anomaly detection. Five classic byzantine-robust aggregation rules and FLTrust belong to the robust aggregation. DnC is one of the representative anomaly detectors. Here mainly describe two SOTA defenses.
FLTrust. ||] devises FLTrust to mitigate the poisoning attack. In FLtrust, the server maintains small clean dataset and act as a client participate in the update, adding the deployment constraint. In each iteration, clients' gradients whose direction deviates more from the gradient of the server are discarded. Moreover, Fl.rust normalizes the gradient magnitude limit the impact of malicious gradients and achieves a competitive accuracy.
DnC. 12 leverages the spectral method detect malicious gradients, which is proven effective in centered learning. They also use random sampling to reduce memory and computational cost. Note will evaluate these defense mechanisms, which have rigorous theoretical guarantees, as baselines.
3 Proposed RECESS
We propose new proactive defense against model poisoning attacks in FL called RECESS. We first take an overview of RECESS. Then we introduce the proactive detection and the new trust scoring based aggregation mechanism respectively, followed by the effectiveness analysis.

---

Detection
test gradient
(1) Construct test gradient
Client
response
(2) Calculate abnormality between each pair and g;
(3) Update trust score TS each each client
S,+1
Client c+1
7S
(5) Update model and send to client
(4) Weighted aggregation
+ig
&
A Aggregation
Figure 2: The overall detection process of RECESS against model poisoning attacks.
3.1 Overview
Intuition. RECESS aims to distinguish malicious and benign gradients without implicating benign outliers. The essential difference between benign and malicious clients that: benign clients including outliers always optimize the received aggregation result to the distribution of their local dataset to minimize the loss value, while malicious clients aim to make the poisoned aggregation direction as far away from the aggregation gradient without poisoning as possible to maximize the poisoning effect. In other words, benign gradients explicitly point to their local distribution, which directional and relatively more stable, while the malicious gradients solved based other benign gradients change inconsistently and dramatically. The workflow of RECESS is shown Figure 2
Proactive Detection. RECESS is different from previous passive analyses. The defender using RECESS proactively tests each client with elaborately constructed aggregation gradients, rather than the model weights to clients, and these two approaches are equivalent algorithmically. After clients fine-tune local models with received aggregation gradients and respond with new gradient updates, the defender can recognize malicious clients based on the abnormality of their gradient updates. We also redefine the abnormality in poisoning scenarios distinguish malicious gradients and outliers. which promotes the generalization of the FL model. Details are shown in Section 3.2
trust scoring based aggregation mechanism to improve the defense. A new scoring method is proposed to give weights to aggregate clients' gradients. Updates with greater abnormal extent account for smaller proportion in aggregation, which increases fault tolerance. Details are shown in Section 3.3
Robust Aggregation. RECESS adopts
3.2 Proactive Detection Algorithm
Construction Strategy of Test Gradient. The purpose of the defender is to observe the client's response by adding a slight perturbation to the test gradient. Thus, any slight varying gradient feasible. To illustrate, stores the aggregation gradient in the last iteration. When entering the detection mode, for example in (k: - 1) round of detection, the defender idles this iteration, and for each uploaded gradient g from i-th client (0 < i n. is the number of clients), the defender scale down the magnitude of - || - / / l|2. and slightly adjust the direction of g by heuristically selecting several elements and adding noise. Here we set threshold to control the direction adjustment. Then, the defender feedbacks of the aggregation gradient client i. The client i updates this tailored "aggregation gradient" locally and respond with newly updated gradient g The defender
present strategy based on previous gradients. In details, the defender firstly

---

can perform poisoning detection by the comparison between g and g This process will be repeated for each client.
slightly adjusts constructed aggregation gradients, which magnifies the variance of mali- cious gradients and makes them more conspicuous, but benign clients and outliers are not affected. The reason is: for benign clients including outliers, their updated gradients always point to the direction of decreasing loss function values. Although some machine learning optimization algo- rithms, such as SGD, have deviations, they are still unbiased estimations of the normal gradient descent direction as whole [[5]. Conversely, malicious clients' gradients are usually opposite to the aggregation direction and come from the solution of the attack optimization program on other benign gradients. Thus, the variance of malicious gradients enlarged by the optimization project. Many works [16] 17, 9 8 also indicate that the fluctuation of the upload gradients cause the aggregation gradient to change more. Even when the defender uses the poisoned aggregation gradient to construct test gradient, this basis still remains unchanged and malicious and henign clients behave quite differently this test. Besides, RECESS without extra model training is more efficient than other model-based detectors. Concluding, RECESS is effective to detect model poisoning attacks.
RECESS
Poisoning Detection. After receiving clients' responses, the defender detects malicious clients from two dimensions of abnormality: direction and magnitude of gradient changes before and after the test. Formally, we use metric cosine similarity St to measure the angular change direction between the constructed gradient g and the response g.
g 9
(g g
11g 11 ll2
Besides direction, the magnitude of the malicious gradient also dominates the poisoning effect, especially when larger than the benign gradients. Here we utilize the la distance to measure the magnitude, i.e., ||
After that, we redefine the abnormality of the client gradient by the deviation extent in direction and magnitude, instead of the distance from other selected gradients (e.g. Krum, Mkrum, Median, server's ' gradient [1] or some benchmarks (e.g. Trmean, Bylan).
Definition (Abnormality) In model poisoning attacks of FL, the abnormality a of k-th uploaded gradient from the i-th client should be measured by
St
19g 1
The cosine similarity St controls the positive and negative of the abnormality a. When the direction of the client's s response gradient g is inconsistent with the defender's test gradient g a will be positive, and vice versa. The amount of change in is related to St and [| lla. The smaller the deviation between the direction of gf and g the smaller the Meanwhile, the larger the l|g l1.. the higher the a. This setting encourages small-gradien updates, which avoids the domination of aggregation by malicious updates usually with larger magnitude.
3.3 New Robust Aggregation Mechanism
After detection, we propose a new trust scoring based mechanism to aggregate gradients, which increases RECESS's 's fault tolerance for false detection.
Trust Scoring. RECESS estimates the trust score for the long-term performance of each client, rather than the single iteration considered by previous schemes.
The detail of the scoring process is: the defender first sets the same initial trust score TSO for each client. After each round of detection, a constant baseline score is deducted for clients who are detected as suspicious poisoners and not selected for aggregation. When the trust score of one client reaches zero, i.e., TS; O, the defender will label this client as malicious and no longer involve this client's s updated gradients in the aggregation. The TS is calculated by
TS- TSO * sewpeeddddeeeddeecoeoe
(3)
5

---

The TSp and baseline baseline_decreeeeeceeeeeeeeeeeee deteeeeddereeedde_e secrredescoreeeeeeeeee controls the detection speed.
To prevent the attacker from increasing the trust score by suspending poisoning constructing benign gradients, we let the defender defer adding 7S to further restrict the attacker. Only after a period of good performance (e.g., 10 consecutive rounds), TS increase, but the would be deducted once the client is detected as malicious, which effectively forces the attacker not to poison or reduce the intensity of poisoning negligible level.
Aggregation. After assigning TS to all clients, we transform TS into weights which are used to average clients' gradients as the aggregation gradient g. i.e.
W. 9i
(4)
i=1
the weight of client i is calculated by
TS: W
(5)
j=1 eS,
where i,j = 1, 2,,,,
Advantages.
Compared with previous methods, RECESS has four advantages:
(a) Provide more accurate measure of the abnormality extent for each client's update to assign the trust score.
(b) Enable the defender to put clients' previous performance into account, not only in each iteration, enhancing the detection fault tolerance.
(c) Protect benign outliers while effectively detecting carefully-construuted poisoning gradient updates, improving model accuracy and generalization.
(d) Outperform previous FLTrust and ML-based detections in terms of efficiency with no need for extra data.
4 Evaluation
4.1 Setup
Datasets and FL Setting. Table four datasets and parameter settings used in the evaluation. The IID and Non-lID are both considered in the dataset division. We also consider two typical types of FL, i.e., cross-silo and cross-device.
Table 1: Experiment datasets and FL settings.
Optamizer
Leansing Kotes
Epochs
MNIST
10
60.000
28
FC (784 512 10)
100
Adam
0.001
500
CIFAR-10
10
60.000
32 x 32
Alexnet [8]
50
250
SGD
0.5 to 0.05 at 1000t epoch
1200
Purchase
100
197,324
600
FC (600 1024 100)
500
SGD
0.5
1000
FEMNIST
LeNes-5
elient'
Eight defenses are considered in the comparison experiments. Specifically, the common FedAvg used the benchmark. Five classic robust rules, Krum, Mkrum, Bulyan, Trmean, and Median, can provide convergence guarantees theoretically. Two SOTA defenses, FLtrust and DnC, are also involved. The parameters of these eight defenses set to default. For RECESS, set A = 0.95, TSp 1, and baseline decreased_scoee scoree = 0.1 unless otherwise specified.
Attack Setting. We consider the five latest poisoning attacks including LIE, optimization attack, AGR-tailoree attack, AGR-agnostic Min-Max attack, and Min-Sum attack. We also set two scenarios, ie.. white-box and black-box, where attackers have and does not have knowledge of other benign gradients. We assume 20% malicious clients as the default value unless specified otherwise. For the test gradient construction, we select and add noise to the first 10% of dimensions, iteratively adjusting until cosine similarity before/after meets the threshold.

---

Table 2: Comparison results of FL accuracy with various defenses against the latest poisoning attacks in both white-bor and black-box cases. In the white-box (black-box) case, the attacker has (no) knowledge of other benign gradients. Each result is averaged multiple repetitions.
Attncker's
Defences
Datasct
Attacks
Knowledge
Krum
Mkrum
Bulyan
Trmean
Median
FLTrust
DaC
RECESS
No Allack
0.5162
0.6494
0.6601
0.6605
0.6582
0.6341
0.6409
0.6554
LIE
0.3058
0.5921
0.0966
0.6436
0.6243
0.6189
0.6047
0.6085
Optimization attack
0.3459
0.5817
0.6118
0.5066
0.5133
0.5813
White-hox
AGR- -ailord
0.2246
0.2915
0.2355
0.2276
0.2970
0.4636
0.4614
0.60443
CIFAR-I0
AGR-agnostic Min-Max
0.5134
0.3173
0.3709
0.2723
0.5229
0.5579
0.6605
AGR AGR-agnosti agnostic Min Sum
0.2248
0.4461
0.2211
0.4271
0.2766
0.5476
0.5347
0.6406
(FedAvg)
LIE Optimization attack
0.4512
0.2891
0.5673
0.2605
0.5558
0,5452
0.6284
0.61599
0.6390
Black-box
lailored
0.2455
AGR--agoostcc Min-Max
0.4078
0,3311
0.2948
0.2773
0.2934
0.5303
0.5771
0.6286
AGR-agnostic Min-Sum
0,2242
0.3435
0.2551
0.4381
0.3492
0. 0.5094
0.5676
0.6432
No Atlack
0.6863
0.8206
0.7115
0,8217
0.7897
0.8237
0X2001
0,8230
LIE Optimization attack
0,2687
0.3897
0,8146
0. 0.8081
0,8048
0.2605
0.4981
0.6286
0.7920
0.7556
0.7848
0,7456
0.7983
White-box
AGR-tailored
0.5462
0,3696
0.4917
0.6127
0,4784
0.7615
0,3661
0.8145
FEMNIST
AGR-agnostic Min-Max
0.6001
0.0606
0.6701
0.5717
0,5552
0. 0,7845
0.6916
0.8019
0,8235
AGR agnostic Min Sum
0.4750
0,2901
0.4496
0.5982
0.6264
0.7148
0.0694
0.7967
(PedAAg)
LIE Optimization attack
0.2221
0.8133
0.7045
0.8114
0.7151
0,8148
0.8145
O.SINI
0.5998
0,7411
0.6646
0,7893
0.6741
0.7807
0.7824
0.7437
Black box
AGR-tailored
0.5419
0.4009
0.4330
0.6858
0,4537
0.7651
0.7027
0.7745
AGR-agnostic Min-Max
0.3796
0,5631
0.6332
0.6440
0.6605
0.6848
0,7351
0,7850
AGR-agnostic Min-Sum
0.6135
0.7524
0.7052
0.6589
0.6785
0.7848
0.7122
0.7936
Henign
1.4
Trust score changing trend
12
10
Mallc oous 10000
7.5
2.5
Class
Malicious
7.5
Benign
3
2.5
-02
-0.5
0
0.5
10
12
15
Abnormality
Detection Round
Figure 3: The changing trend of clients' trust score in RECESS. The task is CIFAR-10. The model poisoning attack is the AGR-agnostic Min- Max attack.
Figure 4: The histogram clients' Abnormality value in RECESS. The task is CIFAR-I0. The model poisoning attack is the AGR-agnostic Min-Max attack. The detection round is #3.
Metric. Accuracy is used to measure the performance of the well-trained model with or without poisoning. Higher Accuracy indicates better defense.
4.2 Comparison with SOTA
Table shows the main comparison results. RECESS obtains higher accuracy than two SOTA defenses under latest poisoning attacks. Due to the space limit, here we mainly show the results on two datasets.
Defender's Goal (1): Defensive Effectiveness. As we can see from each row in Table RECESS can defend against model poisoning attacks, but previous defenses have limited effect, especially for the strongest AGR attack series in the white-box case. The reasons are twofold:
(a) Some malicious gradients evade defenses to be selected for aggregation.
(b) Benign outliers misclassified as malicious gradients and removed from the aggregation, especially for Non-IID datasets (e.g., FEMNIST) with more outliers.
Moreover, benign outliers are intractable for all previous defenses. FI.Trust is better than other defenses since its normalization limits the large magnitude gradients. In contrast, RECESS can
7

---

Trmas Fitrust
0.25
While-los
Black-bon
0.8
Whlfe-bon bos
0.5
Blackshres
Accoraey
Aucuraa
Accurary
0.
0.4
0.5
0.2
0.3
10.4
0.5
55
10%
19%
20%
25%
10%
15% 20% 25%
Figure 5: The impact of the local dataset's Non- IID degree on the FL defenses. The task is CIFAR- 10. The model poisoning attack is AGR Min-Max. Our RECESS achieves the best and most stable defense effect.
Figure 6: The impact of the proportion of attack- ers on the FL defenses. The task is CIFAR-10. The model poisoning attack is AGR Min-Max. As the proportion increases, the accuracy of us- ing our RECESS decreases minimally.
effectively distinguish benign outliers from malicious gradients, thus obtaining the highest accuracy. even in the most challenging white-box Non-IID case.
Defender's Goal (2): Model Performance. As shown in the first in each set of Table 2 where no attacks are under consideration, robust rules will reduce the accuracy but have little impact on RECESS. This can be explained from three aspects:
(a) Due to the existence of attackers, only 80% benign clients participate in the aggregation compared with no attack (e.g. 40 vs 50 for CIFAR-10).
(b) Existing defenses discard benign gradients more or less, especially for benign outliers.
(c) Malicious gradients are selected for aggregation.
The first aspect inevitable since the defender cannot control the attacker, which also causes the main accuracy gap between FedAvg and our RECESS. However, RECESS can improve the other two aspects by more accurately distinguishing whether gradients malicious or benign.
Impact of Attacker's Knowledge. Different knowledge affects the attack, and RECESS out- performs other defenses in both white/black-bot The stronger the attack, the better the effect of RECESS. In the black-box case, the attacker cannot offset the effect of benign gradients the aggregation without the knowledge of other benign gradients, so the poisoning is greatly weakened and RECESS is not prominent in the white-box case.
Changing Trend of Trust Score. Figure presents the trust score of benign and malicious clients in each round of RECESS detection. Here we execute the strongest AGR-agnostic Min-max attack. Figure shows the histogram of each client's 's Abnormality value in the third detection round to illustrate the effectiveness of RECESS. RECESS can effectively distinguish malicious and benign users in both the optimization attack and AGR attacks.
4.3 Impact of FL Setting
In this part, we evaluate RECESS under various FL settings to illustrate the practicability of our proposed scheme.
Non-IID Degree of Dataset (More Outliers). The results Non-IID FEMNIST in Table show the effectiveness of our RECESS against latest poisoning attacks. Besides, we also involve the CIFAR- 10 in the evaluation using the Non-IID dataset construction method same [2] We consider four defenses (the best classic robust rules Trmean, SOTA FLtrust and DnC, and RECESS) against the strongest aag-agnostic Min-Max attack in white/black-bo cases.
Figure 5 shows that the increase the Non-IID degree leads an expansion of benign outliers which are similar to malicious gradients, and previous defenses cannot effectively distinguish outliers from malicious gradients. However, has little impact on RECESS, since our method can accurately detect malicious gradients and benign outliers.

---

Table 3: The impact of cross-device FL on the defenses. Min-Max,
The model poisoning attack is AGR
Altacker's Knowledge
Defenves
Datasct
Attacks
Trmean
FLTrust
DDC
RECESS
No Attack
0.6490
0.6178
0.6284
0.6448
LIE
0.5559
0.6101
0.6157
0.6228
Optimization attuck
0.5655
0.6171
0.5913
0.6439
White-box
AGR-tailored
0,5080
0.4538
0.5915
0.6046
AGR agnostic Min-Max
0.5548
0.5226
0,5478
0.6379
CIFAR-10 -
AGR -ggnstttstic Min-Sum
0.5758
0.5964
0.5793
0.6393
LIF Optimization allack
0.5999
0.6137
0.6128
0.6353
0.6168
0.6159
0.6209
0.6277
Black-box
AGR-tailored
0.5509
0.5485
0.5956
0.6142
AGR- agnostic Min-Max
0,5833
0,5715
0.6095
0.6416
AGR-agnostec Min-Sum
0.6068
0,5937
0.6074
0.6385
Table 4: FL accuracy with enhanced RECESSS defenses against the adaptive poisoning attack. The task FEMNIST. The detection frequency contolled by SS and baseline decreased score. Attacker's
RECESS Detection Frequency
Attacks
Knowledge
10
100
200
no attack
0.8235
0.823
0.8221
0.8224
0.8215
white-hox
optimization attack
0.2182
0.7983
0.8048
0.8106
0.8114
adaptive optimization attack
0.2482
0.3456
0.6847
0.7815
0.8011
optimization attack
0.5152
0.7937
0.8045
0.8117
0.8048
black-box
adaptive optimization attack
0.5248
0.6481
0.7847
0.8048
0.8148
Number of Malicious Clients. shows that RECESS remains an outstanding defending effect all along, as the number of malicious clients increases, all poisoning attacks are more powerful, while the defending effect of other defenses decreases sharply. We vary the proportion of malicious clients from 5% to 25%, consistent with 12] The other settings remain the same as Section
Cross-device FL. Previous evaluations are mostly under cross-silo settings except for the FEMNIST which is naturally cross-device shown in Table 2 Further, consider the cross-device setting using the dataset CIFAR-10 10. In each epoch, the server stochastically selects 10 updates from all clients to aggregate. The attack steps remain the same, and the other settings are the same the default.
Table shows that similar to the result of cross-silo, RECESS still achieves the best defense effect. Besides, the poisoning impacts under the cross-device setting are lower than the one of the cross-silo setting, because the server selects less number of clients for aggregation and ignores more malicious gradients in cross-device FL, thus the attacker cannot continuously poison, weakening the impact of the poisoning. Consequently, this leaves less improvement space for RECESS and other defenses.
4.4 Adaptive Attacks
Active Evasion. With knowledge of RECESS, the attacker checks if it's a test gradient before deciding to poison or not. The adaptive strategy involves estimating an aggregated gradient gp in each iteration from controlled benign clients. Then, the attacker compares the received aggregated gradient Gagg with 9p and g: k-1 from the last iteration. If Sc 9; -
) Z Sc(gogg 9p),, the poisoning begins, while if (c 2 Sc 9p), the poisoning stops, as it's considered detected.
As a countermeasure, the defender boosts detection frequency and intersperses it during the entire training, instead of fixed consecutive detection, and increases the sensitivity of parameters. Note that the defense should aim to make the model converge, it is not necessary to find the attacker. In other words, RECESS's ' presence increases the attack cost, so even if the attacker evades detection, the loss of model accuracy is negligible, which is acceptable.
Table shows that the adaptive attack has little impact on the final model accuracy though it is stealthier than the continuous attack. This is due to the similarity between the test gradient and the normal aggregated gradient, leading to inaccurate estimation by the attacker and decrease in aggregation weight. Also, increasing detection frequency lowers poisoning frequency, so the poisoning impact will be gradually alleviated by normal training over time.

---

Poisoning before the Algorithm Initialization. The detection basis is that the variance of the malicious gradient is larger than the benign gradient. Malicious gradients, solved by the optimization problem, are more inconsistent. It is also theoretically proven that the optimization problem amplifies this variance. Thus, RECESS is unaffected by initial poisoned gradients as it does not change this basis. Besides, when the poisoned aggregated gradient was used to detect, most clients will be identified as malicious. This violates the assumption of Byzantine robustness that requires more than 51% of users be benign. Hence, the defender can easily discern potentially poisoned initial gradient based on the new abnormality definition of RECESS.
Inconsistent attacks. Attackers can also first pretend as benign client to increase the trust score, and then provide the poisoning gradients. However, RECESS detects such behavior inconsistencies over time. The trust scoring of RECESS also incorporates delayed penalties for discrepancies between client's current and past behaviors (shown in Section 3.3 Additionally, three factors limit intermittent attacks' impact:
FL's self-correcting property means inconsistent poisoning is much less impactful. Attackers would not take this approach in practice.
(b) In real settings, clients participate briefly, often just once. Attackers cannot afford to waste rounds acting benign before attacking.
(c) Defenses aim to accurately detect malicious updates for model convergence. Even if poisoning temporarily evaded detection, attack efficacy would diminish significantly, making it no longer serious security concern.
Discussion
This work primarily focuses on untargeted model poisoning, along with previous SOTA works. We believe that model poisoming has a significant impact on real-world FL deployments for three reasons:
(a) Model poisoning through direct gradient manipulation poses a greater threat than backdoors.
(b) Backdoors are considered special cases of model poisoning, malicious gradients be obtained on poisoning datasets. However, it is challenging to solely imitate model poisoning gradients through training backdoors.
(c) Untargeted poisoning has a more severe impact on overall model performance degradation compared backdoors in FL.
As RECESS detects inconsistencies between client behaviors, it could mitigate targeted attacks from clients with backdoored data by deploying additional strategies. Due to space limitation, we refer the readers to our technical report 20 for more details.
Conclusion
In this work, we proposed RECESS, novel defense for FL against the latest model poisoning attacks. We shifted the classic reactive analysis to proactive detection and offered a more robust aggregation mechanism for the server. In the comprehensive evaluation, RECESS achieves better performance than SOTA defenses and solves the outlier detection problem that previous methods can not handle. We anticipate that our RECESS will provide new research direction for poisoning attack defense and promote the application of highly robust FL practice.
Acknowledgments
We thank all the anonymous chairs and reviewers for their valuable guidance and constructive feedback. Hui Li is partially supported by the National Natural Science Foundation of China (61932015), the National Key Research and Development Program of China (2022YFB3104100) Shaanxi innovation team project (201 8TD-007), and Higher Education Discipline Innovation 111 project (B 16037). Wenjing Zhang and Xiaodong Lin are partially supported by Natural Sciences and Engineering Research Council of Canada (NSERC). Part of Haonan Yan's work is done when he visits the School of Computer Science at the University of Guelph.
10

---

References
[1] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Chal- lenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50-60) 2020.
[2] Wensi Yang, Yuhang Zhang, Kejiang Yc, Li, and Cheng-Zhove Xu, Fid: A federated learning based method for credit card fraud detection. In International conference on big data, pages 18 18-32. Springer, 2019.
[3] Jie Xu, Benjamin S Glicksberg, Chang Su, Peter Walker, Jiang Bian, and Fei Wang. Federated learning for healthcare informatics. Journal of Healthcare Informatics Research, 5(1):1-19, 2021.
[4] Nguyen Truong, Kai Sun, Siyao Wang, Florian Guitton, and YiKe Guo. Privacy preservation in federated learning: An insightful survey from the gdpr perspective. Computers & Security, 1 110: 102402, 2021.
[5] Derek A Lackey. Data protection and privacy in canada: A balanced approach. Applied Marketing Analyties, 6(4):366-376, 2021.
[6] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effan learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1288 PMLR, 2017.
[7] Rachid Guerraoui, Sébastien Rouault, et al. The hidden vulnerability of distributed learning in byzantium. In International Conference on Machine Learning, pages 3521-3530. PMLR, 2018.
[8] Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing feder- ated learning through an adversarial lens. In International Conference on Machine Learning, pages 634-643. PMLR, 2019.
[9] Peva Blanchard, EI Mahdi EI Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learn- ing with adversaries: Byzantine tolerant gradient descent. Advances in Neural Information Processing Systems, 30, 2017.
[10] Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantine-robus distributed learning: Towards optimal statistical rates. In International Conference on Machine Learning, pages
PMLR, 2018.
[11] Xiaoyu Cao, Minghong Fang, Jia Liu, and Neil Zhenqiang Gong. Fltrust: Byzantine-robust federated learning via trust bootstrapping. In NDSS, 2021.
[12] Virat Shejwalkar and Amir Houmansadr. Manipulating the byzantine: Optimizing model poisoning attacks and defenses for federated learning. In NDSS, 2021.
[13] Minghong Fang, Xiaoyu Cao, Jinyuan Jía, and Neil Gong. Local model poisoning attacks to (Byzantine-Robur federated learning. In 29th USENIX Security Symposium (USENIX Security 20), pages 1605-1622, 2020.
[14] Gilad Baruch, Moran Baruch, and Yoav Goldberg. A little is enough: Circumventing defenses for distributed learning. Advances in Neural Information Processing Systems, 32, 2019.
[15] Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT 2010, pages 177-186. Springer, 2010.
[16] Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint
00582, 2018
[17] Hangyu Zhu. Jinjin Xu, Shiging Liu, and Yaochu Jin. Federated learning on non-iid data: A survey, Neurocomputing, 465:371-390, 2021.
[18] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
11

---

[19] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11))2278-322 1998.
[20] Haonan Yan, Wenjing Zhang, Qian Chen, Xiaoguang Li, Wenhai Sun, Hui Li, and Xiaodong Lin. Recess vaccine for federated learning: Proactive defense against model poisoning attacks. https://arxiv.. org/ abs/ 2310. 2310.05431
[21] Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference Artificial Intelligence and Statistics, pages 2938-2948. PMLR, 2020.
[22] Saced Mahloujifar, Mohammad Mahmoody, and Ameer Mohammed. Universal multi-party poisoning attacks. International Conference on Machine Learning, pages 4274-4283 PMLR, 2019.
[23] Cong Xie, Oluwasanmi Koyejo, and Indranil Gupta. Fall of empires: Breaking byzantine- tolerant sgd by inner product manipulation. In Uncertainty Artificial Intelligence, pages 261-270. PMLR, 2020.
[24] Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, and Bo Li. Manipulating machine learning: Poisoning attacks and countermeasures for regression learning. In 2018 IEEE Symposium on Security and Privacy (SP), pages 19-35. IEEE, 2018.
[25] Luis Muñoz-González, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongras- samee, Emil C Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back-gradient optimization. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pages 27-38, 2017.
[26] Xueluan Gong, Yanjiao Chen, Qian Wang, and Weihan Kong. Backdoor attacks and defenses in federated learning: State-of-the-art, taxonomy, and future directions. IEEE Wireless Communi- cations, 2022.
12


Reviews:
[{"review_id": "w8eZ3YzJoF", "reviewer": "Reviewer_St5y", "summary": "To defend the untargeted poisoning attack on the federated learning, the authors proposed a new defense called as RECESS, which exploits the outlier detection to analyze the gradients returned from clients. Once the gradient from one client is judged as outlier, this client will be considered as malicious client. The outlier detection is based on the intuition, i.e., for the benign samples even including the outliers, their updated gradients always point to the direction leading to a (local) minimum loss value, while the gradients of poisoned data will point to the reversed direction.", "questions": "For the trust score, is it possible for attackers to first pretend as benign client to increase the trust score, and then provide the poisoning gradient? How should the defender limit this kind of attack? Could the attacker perform the adaptive attack to poison the global model?", "limitations": "One limitation of this work is that it only considers the non-target attack, which is less stealthy than target backdoor attack. The main reason is that the non-target attack’s goal is to affect the normal performance of the model, which can be easily recognized by the trainer. I recommend the authors mentioned this limitation in the paper. For more details on the difference between target and non-target attack, please check this survey. \n\nE. Cinà, K. Grosse, A. Demontis, S. Vascon, W. Zellinger, B. A. Moser, A. Oprea, B. Biggio, M. Pelillo, and F. Roli. Wild patterns reloaded: A survey of machine learning security against training data poisoning. ACM Comp. Surveys, 2023. \n\nGuo, Wei, Benedetta Tondi, and Mauro Barni. \"An overview of backdoor attacks against deep neural networks and possible defences.\" IEEE Open Journal of Signal Processing (2022).\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Similar as previous refs [11, 12] as shown in the paper, this work aims to alarm a malicious gradient from a group of gradients updated from clients in the federated learning. The paper is well-written and can be easily understood.", "weaknesses": "The main weakness of this work is due to its trivial contribution compared with the STOA [11, 12]. The main intuition for outlier detection, as mentioned in ‘Summary’, has been considered. Compared with STOA, the new metric ‘trust scoring’ does not improve too much as shown in Table2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the valuable comments. Please find our responses below.\n\n---\n\n**Q1:** The main weakness of this work is its trivial contribution compared to STOA [11, 12]. The intuition for outlier detection mentioned in the Summary has been considered. Compared to STOA, the new 'trust scoring' metric does not improve much as shown in Table 2.\n\n**R1:** Our contribution is non-trivial compared with previous SOTA works. \n\nWe agree existing defenses rely on passive outlier detection, while our RECESS takes a novel proactive testing approach to identify poisoning. As noted, benign outliers and malicious gradients are numerically similar, making the distinction intractable with only existing passive analysis. Our key intuition is benign and malicious behaviors diverge when aggregated gradients are carefully modified for testing. This active detection provides a fresh perspective unseen in current defenses. Leveraging this, RECESS significantly outperforms existing methods against latest model poisoning attacks. The newly proposed \"Abnormality\" metric (Equation 2), instead of the conventional ‘trust scoring’, provides fundamental advances over previous static defenses. We believe these concepts make our work a non-trivial improvement in poisoning detection.\n\nTo clarify, Table 2 shows RECESS achieves comparable performance to normal FedAvg under no attacks, and outperforms existing defenses on various tasks under typical settings. For example, under AGR-tailored attack, RECESS-trained model accuracy on FEMNIST is up to 44.84% higher than DnC, and 14.07% higher than FLTrust on CIFAR-10. We further evaluated more challenging settings favoring the attacker, like increasing non-IID (Figure 5), more malicious clients (Figure 6), and stronger threat models. In all scenarios, RECESS's advantages over current defenses become more significant. As defenses face sophisticated practical settings and stronger attacks, RECESS demonstrates greater robustness compared to SOTA approaches.\n\n---\n\n**Q2:** For trust score, can attackers first pretend to be benign to increase it, then poison gradients? How to limit this attack? Could attackers perform adaptive attacks to poison the global model?\n\n**R2:** We appreciate the concern about inconsistent attacks. However, RECESS detects such behavior inconsistencies over time. The trust scoring of RECESS also incorporates delayed penalties for discrepancies between a client's current and past behaviors (lines 180-184). Additionally, three factors limit intermittent attacks' impact:\n\n1.  FL's self-correcting property means inconsistent poisoning is much less impactful. Attackers would not take this approach in practice.\n2.  In real settings, clients participate briefly, often just once. Attackers cannot afford to waste rounds acting benign before attacking.\n3.  Defenses aim to accurately detect malicious updates for model convergence. Even if poisoning temporarily evaded detection, attack efficacy would diminish significantly, making it no longer a serious security concern.\n\nWe evaluate more adaptive attacks in **Section 4.4**, including evasion and poisoning before initialization. Results show RECESS provides robustness against these strong adaptive attacks.\n\n---\n**Recommendation:** One limitation is this work only considers non-target attacks, which are less stealthy than target backdoor attacks. The reason is that non-target attacks aim to affect model performance, which can be recognized by the trainer. I recommend mentioning this limitation. \n\n**Response:** We appreciate the constructive recommendation to evaluate targeted backdoor attacks. This submission focuses primarily on untargeted model poisoning, along with previous SOTA works. We believe model poisoning is particularly impactful to real-world FL deployments for three reasons:\n\n1.   Model poisoning via direct gradient manipulation is more threatening than backdoors in FL [8,13].\n2.   Backdoors are considered special cases of model poisoning as malicious gradients can be obtained on poisoning datasets [11]. However, it is non-trivial to imitate model poisoning gradients only via training backdoors.\n3.   Untargeted poisoning degrades overall model performance more severely than backdoors in FL [12-14].\n\nSince RECESS detects inconsistency between client behaviors, it could mitigate targeted attacks from clients with backdoored data by replacing weighted averaging with a Byzantine-robust aggregation like Median. We added experiments showing RECESS with Median effectively defends against the scaling attack, a representative backdoor. RECESS with Median achieves comparable accuracy to the normal model under no attack and resists the backdoor.\n\n| Dataset  |               Attacks                |   FedAvg   | FLTrust |  DnC   | RECESS | RECESS with Median |\n| :------: | :----------------------------------: | :--------: | :-----: | :----: | :----: | :----------------: |\n|  MNIST   |   No Attack (Model Test Accuracy)    | **0.9621** | 0.9584  | 0.9601 | 0.9598 |       0.9581       |\n|          |   Scaling Attack (Model Accuracy)    | **0.9542** | 0.9428  | 0.9518 | 0.9539 |       0.9519       |\n|          | Scaling Attack (Attack Success Rate) |     1      | **0.0** |   1    |   1    |      **0.0**       |\n| CIFAR-10 |   No Attack (Model Test Accuracy)    | **0.6605** | 0.6341  | 0.6409 | 0.6554 |       0.6383       |\n|          |   Scaling Attack (Model Accuracy)    | **0.6329** | 0.6168  | 0.6123 | 0.6314 |       0.6228       |\n|          | Scaling Attack (Attack Success Rate) |     1      | 0.0248  | 0.6412 |   1    |     **0.0127**     |\n\nWe also reviewed the suggested two surveys and will incorporate more comprehensive evaluations of targeted threats in future work to further highlight RECESS's capabilities. We appreciate this insightful discussion to strengthen and broaden the impact of our approach. Please let us know if you have any other suggestions to improve the coverage of additional attack types."}, {"review_id": "Qo4uwBryG7", "reviewer": "Reviewer_MKss", "summary": "The paper presents RECESS, a backdoor defense method in federated learning. The central server keeps querying each client and tries to estimate the trust score of each client to determine if it is malicious or not. The underline intuition of the anomaly detection method is that the malicious client will push the gradient far away from benign directions.\n", "questions": "See above.\n", "limitations": "The paper does not discuss its limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is clearly written with clear labels and easy-to-follow logic. \n\nThe method is evaluated on several datasets and federated learning settings.\n\nThe paper considers the adaptive settings to evaluate its strengths.\n", "weaknesses": "The paper does not validate its assumption. The goal of the adversary in poisoning attacks is to mislead the output with the trigger, and there is no need to manipulate the malicious gradient so that it is far away from benign ones. In fact, many works have demonstrated that some \"unintended backdoors\" only have benign samples, and there are no real malicious triggers. The adversary can later generate triggers based on the trained model (trained on only benign data). As such, I think the paper needs to validate its own assumption or intuition before applying it to practice, especially considering that this may be wrong.\n\nI do not see how the method overcomes the limitations of existing methods. The paper mentions that existing work has false positives, i.e., identifying benign ones as malicious. In a typical federated setting with non-IID data, it is common that some clients will have significantly different gradient optimization directions, which can be far away from others and invalidate the detection assumption of this method. How does this method prevent such false positives from happening?\n\nThe evaluation does not consider methods with theoretical guarantees or stronger adaptive attackers. For example, is it possible to give benign responses to only test gradients while sending malicious gradients to the server?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the valuable feedback. Please find our response below.\n___\n\n**Q1:** The paper does not validate its assumption. The goal of the adversary in poisoning attacks is to mislead the output with the trigger, and there is no need to manipulate the malicious gradient so that it is far away from benign ones. In fact, many works have demonstrated that some \"unintended backdoors\" only have benign samples, and there are no real malicious triggers. The adversary can later generate triggers based on the trained model (trained on only benign data). As such, I think the paper needs to validate its own assumption or intuition before applying it to practice, especially considering that this may be wrong.\n\n**R1:** Thank you for the opportunity to clarify the attack assumptions. We agree it is important to articulate these upfront in the submission.\n\nOur assumed model poisoning adversary directly manipulates gradients to cause untargeted harm to server aggregation, representing a new type of attack on FL proposed recently. This differs from traditional data poisoning or targeted backdoor attacks. As the reviewer points out, we should state this distinction more clearly in the main sections, not just the Appendix. Model poisoning has distinct motivations and methods compared to data attacks. As cited [8,13], this threat model has been adopted by related works and poses severe risks in real-world FL:\n\n1.   Model poisoning via direct gradient manipulation is more threatening than targeted backdoor attacks [8, 13], and highly relevant to production FL.\n2.   Targeted backdoor attacks are considered special cases of model poisoning as malicious gradients can be obtained on poisoning datasets [11]. However, it is non-trivial to imitate model poisoning gradients only via data poisoning.\n\n3.   Untargeted poisoning is a more severe threat to model performance than targeted ones [12-14].\n\nWe apologize for the lack of clarity on threat model assumptions and will move the description in “Appendix A.1 & F.2” to **Related Work**.\n\nAlthough RECESS focuses on model poisoning, we show by replacing weighted averaging with Median aggregation, RECESS can mitigate the scaling attack, a representative backdoor. \n\n| Dataset  |               Attacks                |   FedAvg   | FLTrust |  DnC   | RECESS | RECESS with Median |\n| :------: | :----------------------------------: | :--------: | :-----: | :----: | :----: | :----------------: |\n|  MNIST   |   No Attack (Model Test Accuracy)    | **0.9621** | 0.9584  | 0.9601 | 0.9598 |       0.9581       |\n|          |   Scaling Attack (Model Accuracy)    | **0.9542** | 0.9428  | 0.9518 | 0.9539 |       0.9519       |\n|          | Scaling Attack (Attack Success Rate) |     1      | **0.0** |   1    |   1    |      **0.0**       |\n| CIFAR-10 |   No Attack (Model Test Accuracy)    | **0.6605** | 0.6341  | 0.6409 | 0.6554 |       0.6383       |\n|          |   Scaling Attack (Model Accuracy)    | **0.6329** | 0.6168  | 0.6123 | 0.6314 |       0.6228       |\n|          | Scaling Attack (Attack Success Rate) |     1      | 0.0248  | 0.6412 |   1    |     **0.0127**     |\n\nAdditional results demonstrate that RECESS with Median achieves comparable accuracy to the normal model using FedAvg under no attack and resists the backdoor effectively.\n\n---\n\n**Q2**:  I do not see how the method overcomes the limitations of existing methods. The paper mentions that existing work has false positives, i.e., identifying benign ones as malicious. In a typical federated setting with non-IID data, it is common that some clients will have significantly different gradient optimization directions, which can be far away from others and invalidate the detection assumption of this method. How does this method prevent such false positives from happening?\n\n**R2:** We introduce a new detection approach, proactive probing, to address the intractable problem of benign outlier identification that passive statistics-based defenses struggle with, especially when the degree of non-IID is large, this problem is more difficult to solve. \n\nInstead of previously taking a single fixed benchmark to determine anomalies, we make judgments based on the consistency of the client's behavior over time to avoid detection errors. Specifically,\n\n-   The key insight is that malicious and benign clients (including benign outliers) have fundamentally different update goals. By carefully constructing the aggregation gradients sent to clients, we can amplify this difference and more accurately detect malicious behaviors while distinguishing benign outliers.\n-   Additionally, we propose a new trust scoring-based robust aggregation mechanism that estimates scores based on long-term user performance across iterations, rather than scoring each round independently. This improves fault tolerance and robustness.\n\nTogether, the novel concepts of proactive probing and robust trust scoring allow RECESS to outperform SOTA defenses against the latest poisoning attacks, while accurately identifying benign outlier gradients.\n\n---\n\n**Q3:** The evaluation does not consider methods with theoretical guarantees or stronger adaptive attackers. For example, is it possible to give benign responses to only test gradients while sending malicious gradients to the server?\n\n**R3:** We compare five classic robust rules and two SOTA works, all with solid theoretical guarantees. We also analyze RECESS under black box and white box settings and provide theoretical guarantees, as shown in **Proposition 1 in Section 3.4 Effectiveness Analysis** in lines 196-205 of page 6. The proof can be found in **Section D Proof of Proposition 1** in lines 447-467 of page 13. \n\nStronger adaptive attackers will struggle to accurately determine if crafted gradients are tests, limiting poisoning impact under our defense. We refer the reviewers to the evaluation of adaptive attacks in **Section 4.4 Adaptive Attacks** in lines 287-312 of page 9 for more details."}, {"review_id": "fpPkCFLBEA", "reviewer": "Reviewer_tdiM", "summary": "This paper proposes a new defense against model poisoning attacks in FL. The idea is that the server sends some perturbed aggregation gradient to clients in some selected training rounds, and based on the responses, the server adjusts trust scores for the clients. Experimental results show the effectiveness of the proposed defense against state of the art attacks and outperforms state-of-the-art defenses. ", "questions": "See above. ", "limitations": "See above. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Important and relevant research problem. \n\n+ Well written paper. \n\n+ Interesting and novel approach. ", "weaknesses": "I didn't see severe weaknesses. I think the paper is above the bar. \n\nI have three suggestions:\n\n1. Since the paper talks about detection. I would suggest also comparing with detection methods, e.g., the following:\n\nFLDetector: Defending federated learning against model poisoning attacks via detecting malicious clients. In KDD, 2022. \n\nThis detection method may also be adapted to assign trust scores for clients. \n\n2. All the evaluated attacks are for compromised genuine clients. Recent attacks use fake clients, e.g., the following:\n\nMpaf: Model poisoning attacks to federated learning based on fake clients. In CVPR Workshop, 2022. \n\nThe paper may want to evaluate such attacks. \n\n3. The paper seems to assume that the server sends aggregated gradient to the clients. This is different from standard FL. The paper can make this more clear. Algorithmically, it is equivalent to send the new global model to clients, but it's good to make this clear. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewers' insightful suggestions, which are helpful to strengthen the completeness of this work.\n\n---\n\nI didn't see severe weaknesses. I think the paper is above the bar. I have three suggestions:\n\n**S1:** Since the paper talks about detection. I would suggest also comparing with detection methods, e.g., the following:\n\nFLDetector: Defending federated learning against model poisoning attacks via detecting malicious clients. In KDD, 2022.\n\nThis detection method may also be adapted to assign trust scores for clients.\n\n**R1:** Yes, FLDetector also adopts a \"trust scoring\" approach to identify malicious clients. The key difference lies in how anomalies are detected. We actively send test queries and compare the updates returned from clients, while FLDetector makes predictions based on historical updates. We also evaluated FLDetector, with the following results:\n\n| Dataset                | Attacks              | FLDetector | RECESS |\n| ---------------------- | -------------------- | ---------- | ------ |\n| CIFAR-10 (IID)         | Optimization attack  | 0.6384     | **0.6390**|\n|                        | AGR-agnostic Min-Max | 0.5178     | **0.6286** |\n| CIFAR-10 (Non-IID 0.5) | Optimization attack  | 0.6041     | **0.6145** |\n|                        | AGR-agnostic Min-Max | 0.5514     | **0.5912** |\n| CIFAR-10 (Non-IID 0.8) | Optimization attack  | 0.3544     | **0.6018** |\n|                        | AGR-agnostic Min-Max | 0.0176     | **0.5841** |\n| FEMNIST                | Optimization attack  | 0.7217     | **0.7937** |\n|                        | AGR-agnostic Min-Max | 0.5616     | **0.7850** |\n\nIn regular settings, the performance of FLDetector (Median) is comparable to RECESS, but when facing stronger attacks (AGR attack series) or higher Non-IID levels (>0.5 or on FEMNIST task), the model accuracy achieved by RECESS is significantly higher than FLDetector. This is because as the number of benign outliers increases significantly, the fluctuations in clients’ historical gradients become more intense, which greatly reduces the accuracy of FLDetector's predictions. In contrast, RECESS can effectively cope through dynamic testing. Therefore, the comparative experiments with FLDetector further demonstrate RECESS's advantages in tackling the challenging problem of distinguishing between benign outliers and malicious users.\n\n---\n\n**S2:** All the evaluated attacks are for compromised genuine clients. Recent attacks use fake clients, e.g., the following:\n\nMpaf: Model poisoning attacks to federated learning based on fake clients. In CVPR Workshop, 2022.\n\nThe paper may want to evaluate such attacks.\n\n**R2:** Thank you for the reviewer's suggestion to evaluate this new attack type named MPAF. \n\nWe thoroughly read this attack scheme. This attack injects fake clients rather than compromising victim clients, allowing a higher ratio of malicious users. In addition, MPAF uses a very simple attack method that merely attempts to drag the global model towards an attacker-chosen low-accuracy model, with a fixed local update goal and without optimization during poisoning. RECESS does not consider this type of poisoning attack, mainly because:\n\n1.  The attacker's assumptions do not conform to real-world FL requirements. We have concerns about its practical feasibility due to the strong assumption of arbitrary fake client injection required. To the best of our knowledge, satisfying such assumptions is highly challenging in practice, as evidenced by the following literature: \n\n    *Shejwalkar, Virat, Amir Houmansadr, Peter Kairouz, and Daniel Ramage. \"Back to the drawing board: A critical evaluation of poisoning attacks on production federated learning.\" In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1354-1371. IEEE, 2022.*\n\n2.  Even if these assumptions held, basic Byzantine-robust aggregation rules could readily defeat the attack. Our proposal can be enhanced by using a common Median or Krum instead of weighted averaging in RECESS to defend against MPAF.\n\n3.  Although MPAF submits malicious gradients, it is essentially a data poisoning attack before the FL system starts. Existing detections during training, like FLDetector, also do not consider such attacks. \n\nWhile we appreciate the reviewer's perspective to include recent advancements, given our reservations on the practicality of this attack, we are worried that elaborating on it in depth might give readers the false impression that it poses a serious threat. If the reviewer still believes a comparison is needed, we would be happy to reconsider given further guidance. Otherwise, we hope to focus our manuscript on defending against attacks under more realistic assumptions that are standard in the field. \n\n---\n\n**S3:** The paper seems to assume that the server sends aggregated gradient to the clients. This is different from standard FL. The paper can make this more clear. Algorithmically, it is equivalent to sending the new global model to clients, but it's good to make this clear.\n\n**R3:** Thank you for pointing this out. Yes, to actively test malicious clients, the server with RECESS deployed needs to send gradients rather than the model weight to clients, and these two approaches are equivalent algorithmically. We will clarify this in a future version.\n\nPlease let us know if we can provide any clarification or justify our position further. Overall, we sincerely thank the reviewer for the time and help in strengthening our work."}, {"review_id": "HeSWg9FnPO", "reviewer": "Reviewer_nW5D", "summary": "In this paper, the authors propose RECESS, which is a proactive defense against untargeted model poisoning attacks. Specifically, RECESS proactively detects malicious clients with test gradients and robustly aggregates gradient with a new trust scoring based mechanism. For the former, the key insight is that the update goals of malicious clients are different from benign clients and outliers. Therefore, the defender can amplify this difference by delicately constructing aggregation gradients sent to clients, and then more accurately detect malicious clients and distinguish benign outliers by the comparison between constructed gradients and the corresponding responses. For the latter, a new trust scoring based mechanism is introduced to estimate the score according to the user’s performance over multiple iterations. Experiments on four datasets and various settings including white/black-box, cross silo/device FL, etc are conducted. The authors also compare the proposed approach with five classic and two SOTA defenses.", "questions": "See weakness.", "limitations": "See weakness.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-motivated to solve two limitations in existing poisoning defense: (1) existing methods are susceptible to the optimization-based model poisoning attacks, leading to accuracy loss, and (2) existing methods cannot distinguish malicious gradient and benign outliers, reducing the generalization performance of the trained model. The proposed method is well-justified both theoretically and experimentally. Extensive evaluations on four datasets with various FL settings and attack settings, as well as the comparison with five classic and two SOTA defense are provided.", "weaknesses": "While this is a well-written and well-motivated submission, this reviewer has the following concerns. \n       •\tLimited capacity of defense. The observation that malicious clients aim to make the poisoned aggregation direction as far away from the aggregation gradient without poisoning as possible to maximize the poisoning effect only holds for untargeted model poisoning.\n       •\tDiscussion of compliance missing. While the authors design the test gradients and send it to the clients for identifying the malicious clients, it is unclear how would the malicious attack comply with the designed the rule? Can malicious clients cheat on the test gradient?\n       •\tTechnical details require additional clarification. (1) It seems for this reviewer, it is not clear how the “slight adjust the direction of g_t_i by heuristically selecting several elements” is done. What are those elements? (2) the models evaluated in the experiment are relatively simple and shallow. This raises the concern of scalability to deeper models. (3) Missing details for IID/Non-IID, cross-silo/cross-device configurations, such as how many clients or devices, how much data per client, how to generate the non-IID division.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments. We provide our responses and additional evaluations below to address the concerns.\n\n---\n\n**Q1:** Limited capacity of defense. \n\n**R1:** We appreciate the comment that our method primarily handles untargeted model poisoning. \n\nThis submission does focus on untargeted attacks, prevalent in prior arts. We believe untargeted attacks pose a more severe threat to FL, for three reasons:\n\n1.   Model poisoning attacks directly manipulating gradients are more threatening than targeted backdoor and data poisoning in FL [8, 13], and highly relevant to production FL.\n\n2.   Targeted backdoor and data poisoning can be seen as special cases of model poisoning as malicious gradients can be obtained from poisoning datasets [11]. However, it is non-trivial to imitate model poisoning gradients only via data poisoning.\n\n3.   Untargeted poisoning degrades model performance more severely than targeted ones in FL [12-14].\n\nNevertheless, by simply substituting robust aggregation for Byzantine-robust aggregation, our RECESS can effectively defend against other poisoning attacks. To demonstrate RECESS's versatility, we evaluated various attacks, including data poisoning (label flipping) and targeted backdoor (scaling attack).\n\n| Dataset  | Attacks                              | FedAvg     | FLTrust | DnC    | RECESS | RECESS with Median |\n| -------- | ------------------------------------ | ---------- | ------- | ------ | ------ | ------------------ |\n| MNIST    | No Attack (Model Accuracy)           | **0.9621** | 0.9584  | 0.9601 | 0.9598 | 0.9581             |\n|          | Label Flipping (Model Accuracy)      | 0.9018     | 0.9448  | 0.9225 | 0.9154 | **0.9548**         |\n|          | Scaling Attack (Attack Success Rate) | 1.0        | **0**   | 1.0    | 1.0    | **0**              |\n| CIFAR-10 | No  Attack                           | **0.6605** | 0.6341  | 0.6409 | 0.6554 | 0.6383             |\n|          | Label Flipping                       | 0.4145     | 0.5748  | 0.4284 | 0.4415 | **0.6148**         |\n|          | Scaling Attack                       | 1.0        | 0.0248  | 0.6412 | 1.0    | **0**              |\n\nThe setting of the label flipping attack is the same as [13] and the scaling’s setting is consistent with [23]. The accuracy of the model using RECESS with Median under all attacks rivals conventional models (using FedAvg under no attack). This shows RECESS can effectively defend against diverse FL poisoning. While not demonstrated originally, we respectfully suggest our technique is capable of tackling these attacks straightforwardly.\n\n------\n\n**Q2:** Discussion of compliance missing. \n\n**R2:** Thank you for the feedback to strengthen the compliance discussion. Attackers may attempt to evade detection, but we account for potential adaptive attack behaviors and have countermeasures, including increased detection frequency, heightened parameter sensitivity, and delayed reward. These can effectively restrict malicious clients and ensure proper FL system operation. The evaluation of adaptive attacks was covered comprehensively in **Section 4.4**.\n\n---\n\n**Q3:** Additional technical details. \n\n**Q3.1:** Details of gradient modification.\n\n**R3.1:** Thank you for the feedback. Due to page limits, more specifics are in the appendix. As described in **Appendix E.3**, the modification allows arbitrary adjustments to gradient elements within a defined threshold. In experiments, we select and add noise to the first 10% of dimensions, iteratively adjusting until cosine similarity before/after meets the threshold. \n\n---\n\n**Q3.2:** The concern of scalability to deeper models. \n\n**R3.2:** Thank you for raising this important point. For fair evaluation, we used the same base models as prior works. Nevertheless, our approach can extend to deeper networks. To demonstrate scalability, we conducted additional experiments on larger models including DNN, VGG, and ResNet. As shown in the table below, our method achieves consistent utility for these complex models.\n\n| Dataset  | Model    | Attacks             | FedAvg     | RECESS     |\n| -------- | -------- | ------------------- | ---------- | ---------- |\n| MNIST    | DNN      | No Attack           | **0.9487** | 0.9405     |\n|          |          | Optimization attack | 0.6873     | **0.9314** |\n| CIFAR-10 | ResNet20 | No Attack           | **0.8449** | 0.8217     |\n|          |          | Optimization attack | 0.1718     | **0.8173** |\n|          |          | AGR-tailored Attack | 0.1544     | **0.8014** |\n|          | VGG11    | No Attack           | **0.7515** | 0.7408     |\n|          |          | Optimization attack | 0.5032     | **0.7344** |\n|          |          | AGR-tailored Attack | 0.3834     | **0.7037** |\n\nThese results highlight the general applicability of our framework across model depths. \n\n---\n\n**Q3.3:** More details needed on IID/Non-IID and cross-device/cross-silo settings like number of clients, data per client, and how non-IID data is generated.\n\n**R3.3:** Similarly, we use the same settings as prior works for fair comparison:\n\n*   Number of clients/devices specified in **Table 1** for each dataset. For cross-device FL, the server aggregates from 10 random clients per round on CIFAR-10 and 60 random clients on FEMNIST, a common setting in prior work.\n*   CIFAR-10/MNIST: 50 clients with 1000 samples each. Purchase: 80 clients with 2000 samples each. FEMNIST uses a native non-IID partition of 3400 clients owning unique data.\n\n*   Except FEMNIST is natively non-IID, non-IID CIFAR-10 uses the standard approach from prior work. With *M* classes, clients split into *M* groups. Class *i* example assigned to group *i* with probability *q*, and to any group with probability *(1-q)/(M-1)*. *q=1/M* is IID, larger *q* means more non-IID.\n\nPlease let us know if any additional clarification on the settings would be helpful. We are happy to provide more details on the configurations used."}]

Author rebuttal:

Meta-review: The work proposes a novel defense against model poisoning attacks to federated learning. The rebuttal addressed most of the concerns raised by the reviewers, and the authors are invited to modify the camera-ready accordingly. They should also improve the discussion on the limitations of the present work.
Decision: Accept (poster)