Title: RECESS Vaccine for Federated Learning: Proactive Defense Against Model Poisoning Attacks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Federated Learning

Model Poisoning Attacks

Proactive Detection

Robust Aggregation

Benign Outlier Identification
Abstract: Model poisoning attacks greatly jeopardize the application of federated learning (FL). The effectiveness of existing defenses is susceptible to the latest model poisoning attacks, leading to a decrease in prediction accuracy. Besides, these defenses are intractable to distinguish benign outliers from malicious gradients, which further compromises the model generalization. In this work, we propose a novel defense including detection and aggregation, named RECESS, to serve as a “vaccine” for FL against model poisoning attacks. Different from the passive analysis in previous defenses, RECESS proactively queries each participating client with a delicately constructed aggregation gradient, accompanied by the detection of malicious clients according to their responses with higher accuracy. Further, RECESS adopts a newly proposed trust scoring based mechanism to robustly aggregate gradients. Rather than previous methods of scoring in each iteration, RECESS takes into account the correlation of clients’ performance over multiple iterations to estimate the trust score, bringing in a significant increase in detection fault tolerance. Finally, we extensively evaluate RECESS on typical model architectures and four datasets under various settings including white/black-box, cross-silo/device FL, etc. Experimental results show the superiority of RECESS in terms of reducing accuracy loss caused by the latest model poisoning attacks over five classic and two state-of-the-art defenses.

RECESS Vaccine for Federated Learning: Proactive Defense Against Model Poisoning Attacks

Abstract

Model poisoning attacks greatly jeopardize the application of federated learning (FL). The effectiveness of existing defenses is susceptible to the latest model poisoning attacks, leading to a decrease in prediction accuracy. Besides, these defenses are intractable to distinguish benign outliers from malicious gradients, which further compromises the model generalization. In this work, we propose a novel defense including detection and aggregation, named RECESS , to serve as a 'vaccine' for FL against model poisoning attacks. Different from the passive analysis in previous defenses, RECESS proactively queries each participating client with a delicately constructed aggregation gradient, accompanied by the detection of malicious clients according to their responses with higher accuracy. Further, RECESS adopts a newly proposed trust scoring based mechanism to robustly aggregate gradients. Rather than previous methods of scoring in each iteration, RECESS takes into account the correlation of clients' performance over multiple iterations to estimate the trust score, bringing in a significant increase in detection fault tolerance. Finally, we extensively evaluate RECESS on typical model architectures and four datasets under various settings including white/black-box, cross-silo/device FL, etc. Experimental results show the superiority of RECESS in terms of reducing accuracy loss caused by the latest model poisoning attacks over five classic and two state-of-the-art defenses.

1 Introduction

Background and Problem. Recently, federated learning (FL) goes viral as a privacy-preserving training solution with the distributed learning paradigm [1], since data privacy attracts increasing attention from organizations like banks [2] and hospitals [3], governments like GDPR [4] and CPPA [5], and commercial companies like Google [6]. FL allows data owners to collaboratively train models under the coordination of a central server for better prediction performance by sharing local gradient updates instead of their own private/proprietary datasets, preserving the privacy of each participant's raw data. FL is promising as a trending privacy training technology. However, FL is vulnerable to various model poisoning attacks [7, 8]. The distributed structure of FL characterized by the privacy-preserving local training paradigm makes it impossible to verify overall gradient updates and distributions of local datasets. Accordingly, the attacker can corrupt the global model by hijacking compromised participants, and then uploading malicious local gradient updates, leading to a reduction in the final model's prediction performance, which significantly jeopardizes the application of FL.

…

1.

Statistically analyze

the

uploaded gradient set

{

g

1

, … , g

c

, g

c+1

, … , g

n

}

2.

Find abnormal gradients, e.g.,

g

c

and

g

n

is malicious,

g

c+1

is benign.

3.

Remove malicious graidents

and

robustly aggregate

to get

aggregation result

g

Malicious

c

-th Client

Server with previous defenses

Server with our RECESS

Benign

(

c+1

)-th Client

Benign

n

g

c

…

g

c+1

g

n

test gradient

g t

c

g

′

c

1.

Construct

test gradient to each

client, e.g.,       to

c

-th client

2.

Collect the response, e.g.,

from

c

-th client

3.

Compare

each pair to find

malicious gradients, e.g.,

{     ,     } is malicious

{        ,        } is benign

{      ,     } is benign n

4.

Use

new trust scoring based

mechnism to get

g

g t

c

+1

g

′

c

+1

g t

n

g

′

n

g t

c

g

′

c

g t

c

g

′

c

g t

c

+1

g

′

c

+1

g t

n

g

′

n

Limitations of Previous Works. To mitigate model poisoning attacks, many byzantine-robust aggregation rules, e.g., Krum, Mkrum, Trmean, Median, and Bulyan [9, 10, 7], are proposed to remove malicious gradients by statistical analyses. Two state-of-the-art (SOTA) defenses, FLTrust [11] and DnC [12], are also proposed to further enhance the robustness of the FL system. Even so, ( Limitation 1: ) these defenses are susceptible to the latest optimization-based model poisoning attacks [13, 12], leading to a decrease in prediction accuracy.

Besides, ( Limitation 2: ) these defenses cannot distinguish malicious gradients and benign outliers, reducing the generalization performance of the final trained model. Considering that the local datasets across participating clients are Non-IID, it is common to have many biased updates which are statistical outliers. These statistical outliers are still benign updates (called benign outliers ), which are helpful in improving the generalization performance of the final FL model. However, existing defenses typically rely on statistical majorities to remove malicious clients, which also implicates these benign outliers. To the best of our knowledge, the work of tackling benign outliers still remains open in FL.

Our work. To improve model accuracy and generalization against latest model poisoning attacks in FL, we propose RECESS acting as 'vaccine' to help FL p R oactively d E te C t mod E l poi S oning attack S . Unlike previous defenses using passive analysis and direct aggregation in a single iteration, RECESS proactively detects malicious clients and robustly aggregates gradients with a new trust scoring based mechanism. The differences are illustrated in Figure 1. In proactive detection , our key insight is that the update goals of malicious clients are different from benign clients and outliers. Therefore, the defender can amplify this difference by delicately constructing aggregation gradients sent to clients, and then more accurately detect malicious clients and distinguish benign outliers by the comparison between constructed gradients and the corresponding responses. In robust aggregation , we propose a new trust scoring based mechanism which estimates the score according to the user's performance over multiple iterations, rather than scoring each iteration as in previous methods, which improves the detection fault tolerance. Finally, we compare RECESS with five classic and two SOTA defenses under various settings, and the results demonstrate that RECESS is effective to overcome the two limitations aforementioned.

Contribution. The main contributions are:

(a) We propose a novel defense against model poisoning attacks called RECESS . It offers a new defending angle for FL and turns the defender from passive analysis to proactive detection, which defeats the latest model poisoning attacks. RECESS can also identify benign outliers effectively.

(b) We improve the robust aggregation mechanism. A new trust scoring method is devised by considering clients' abnormality over multiple iterations, which significantly increases the detection fault tolerance of RECESS and further improves the model's accuracy.

(c) We evaluate RECESS under various settings, including white/black-box, Non-IID degree of the dataset, number of malicious clients, and cross-silo/device FL. Experimental results show that RECESS outperforms previous defenses in terms of accuracy loss and achieves consistent effectiveness.

2 Related Works

2.1 Latest Model Poisoning Attacks in FL

In this work, we focus on the stronger untargeted model poisoning attacks for three reasons:

(a) Model poisoning attacks with the direct manipulation of gradients are more threatening than data poisoning attacks in FL [8, 13].

(b) Data poisoning attacks are considered a special case of model poisoning attacks as malicious gradients can be obtained on poisoning datasets [11].

(c) Untargeted poisoning is a more severe threat to model prediction performance than the targeted one in FL [12].

In the following, we introduce three latest model poisoning attacks.

LIE Attack. The LIE attack [14] lets the defender remove the non-byzantine clients and shift the aggregation gradient by carefully crafting byzantine values that deviate from the correct ones as far as possible.

Optimization Attack. [13] propose a new attack idea. They formulate the model poisoning attack as an optimization program, where the objective function is to let the poisoning aggregation gradient be far from the aggregation gradient without poisoning. By leveraging the halving search, they obtain a crafted malicious gradient to poison.

AGR Attack Series. [12] then improve the optimization program by introducing perturbation vectors and scaling factors. Then three instances (AGR-tailored, AGR-agnostic Min-Max, and Min-Sum) are proposed, which maximize the deviation between benign gradients and malicious gradients.

2.2 Existing SOTA Defenses

There are two directions to defend against the model poisoning attack: robust aggregation and anomaly detection. Five classic byzantine-robust aggregation rules and FLTrust belong to the robust aggregation. DnC is one of the representative anomaly detectors. Here we mainly describe two SOTA defenses.

FLTrust. [11] devises FLTrust to mitigate the poisoning attack. In FLtrust, the server maintains a small clean dataset and act as a client to participate in the update, adding the deployment constraint. In each iteration, clients' gradients whose direction deviates more from the gradient of the server are discarded. Moreover, FLtrust normalizes the gradient magnitude to limit the impact of malicious gradients and achieves a competitive accuracy.

DnC. [12] leverages the spectral method to detect malicious gradients, which is proven effective in centered learning. They also use random sampling to reduce memory and computational cost. Note we will evaluate these two defense mechanisms, which have rigorous theoretical guarantees, as baselines.

3 Proposed RECESS

We propose a new proactive defense against model poisoning attacks in FL called RECESS . We first take an overview of RECESS . Then we introduce the proactive detection and the new trust scoring based aggregation mechanism respectively, followed by the effectiveness analysis.

…

Client c

Client c+1

(1) Construct test

gradient

(2) Calculate

abnormality between

each pair of      and

(3) Update trust

score

TS

of each

client

(4) Weighted

aggregation

g

′

c

g

′

c

+1

g t

i

(5) Update model

and send to client

g

=

n

∑

i

=1

wi

*

g

i

,

TS

θ

θ

+

λg

g

𝜃

g t

i

g

′

i

test gradient

response

Detection

Aggregation

3.1 Overview

Intuition. RECESS aims to distinguish malicious and benign gradients without implicating benign outliers. The essential difference between benign and malicious clients is that: benign clients including outliers always optimize the received aggregation result to the distribution of their local dataset to minimize the loss value, while malicious clients aim to make the poisoned aggregation direction as far away from the aggregation gradient without poisoning as possible to maximize the poisoning effect. In other words, benign gradients explicitly point to their local distribution, which is directional and relatively more stable, while the malicious gradients solved based on other benign gradients change inconsistently and dramatically. The workflow of RECESS is shown in Figure 2.

Proactive Detection. RECESS is different from previous passive analyses. The defender using RECESS proactively tests each client with elaborately constructed aggregation gradients, rather than the model weights to clients, and these two approaches are equivalent algorithmically. After clients fine-tune local models with received aggregation gradients and respond with new gradient updates, the defender can recognize malicious clients based on the abnormality of their gradient updates. We also redefine the abnormality in poisoning scenarios to distinguish malicious gradients and outliers, which promotes the generalization of the FL model. Details are shown in Section 3.2.

Robust Aggregation. RECESS adopts a new trust scoring based aggregation mechanism to improve the defense. A new scoring method is proposed to give weights to aggregate clients' gradients. Updates with greater abnormal extent account for a smaller proportion in aggregation, which increases fault tolerance. Details are shown in Section 3.3.

3.2 Proactive Detection Algorithm

Construction Strategy of Test Gradient. The purpose of the defender is to observe the client's response by adding a slight perturbation to the test gradient. Thus, any slight varying gradient is feasible. To illustrate, we present a strategy based on previous gradients. In details, the defender firstly stores the aggregation gradient in the last iteration. When entering the detection mode, for example in ( k -1) -th round of detection, the defender idles this iteration, and for each uploaded gradient g k -1 i from i -th client ( 0 < i ≤ n , n is the number of clients), the defender scale down the magnitude of g k -1 i , i.e., ∥ g t i ∥ 2 = g k -1 i / ∥ g k -1 i ∥ 2 , and slightly adjust the direction of g t i by heuristically selecting several elements and adding noise. Here we set a threshold A to control the direction adjustment. Then, the defender feedbacks g t i as the aggregation gradient to client i . The client i updates this tailored 'aggregation gradient' locally and respond with a newly updated gradient g k i . The defender

can perform poisoning detection by the comparison between g t i and g k i . This process will be repeated for each client.

RECESS slightly adjusts constructed aggregation gradients, which magnifies the variance of malicious gradients and makes them more conspicuous, but benign clients and outliers are not affected. The reason is: for benign clients including outliers, their updated gradients always point to the direction of decreasing loss function values. Although some machine learning optimization algorithms, such as SGD, have deviations, they are still unbiased estimations of the normal gradient descent direction as a whole [15]. Conversely, malicious clients' gradients are usually opposite to the aggregation direction and come from the solution of the attack optimization program on other benign gradients. Thus, the variance of malicious gradients is enlarged by the optimization project. Many works [16, 17, 9, 8] also indicate that the fluctuation of the upload gradients cause the aggregation gradient to change more. Even when the defender uses the poisoned aggregation gradient to construct test gradient, this basis still remains unchanged and malicious and benign clients behave quite differently in this test. Besides, RECESS without extra model training is more efficient than other model-based detectors. Concluding, RECESS is effective to detect model poisoning attacks.

Poisoning Detection. After receiving clients' responses, the defender detects malicious clients from two dimensions of abnormality: direction and magnitude of gradient changes before and after the test. Formally, we use metric cosine similarity S k C to measure the angular change in direction between the constructed gradient g t i and the response g k i ,

Besides direction, the magnitude of the malicious gradient also dominates the poisoning effect, especially when larger than the benign gradients. Here we utilize the l 2 distance to measure the magnitude, i.e., ∥ g k i ∥ 2 .

After that, we redefine the abnormality of the client gradient by the deviation extent in direction and magnitude, instead of the distance from other selected gradients (e.g. Krum, Mkrum, Median, server's gradient [11]) or some benchmarks (e.g. Trmean, Bylan).

Definition 1 (Abnormality) In model poisoning attacks of FL, the abnormality α of k -th uploaded gradient from the i -th client should be measured by

The cosine similarity S k C controls the positive and negative of the abnormality α . When the direction of the client's response gradient g k i is inconsistent with the defender's test gradient g t i , α will be positive, and vice versa. The amount of change in α is related to S k C and ∥ g k i ∥ 2 . The smaller the deviation between the direction of g t i and g k i , the smaller the α . Meanwhile, the larger the ∥ g k i ∥ 2 , the higher the α . This setting encourages small-gradient updates, which avoids the domination of aggregation by malicious updates usually with a larger magnitude.

3.3 New Robust Aggregation Mechanism

After detection, we propose a new trust scoring based mechanism to aggregate gradients, which increases RECESS 's fault tolerance for false detection.

Trust Scoring. RECESS estimates the trust score for the long-term performance of each client, rather than the single iteration considered by previous schemes.

The detail of the scoring process is: the defender first sets the same initial trust score TS 0 for each client. After each round of detection, a constant baseline score is deducted for clients who are detected as suspicious poisoners and not selected for aggregation. When the trust score of one client i reaches zero, i.e., TS i = 0 , the defender will label this client as malicious and no longer involve this client's updated gradients in the aggregation. The TS is calculated by

The TS 0 and baseline _ decreased _ score controls the detection speed.

To prevent the attacker from increasing the trust score by suspending poisoning or constructing benign gradients, we let the defender defer adding TS to further restrict the attacker. Only after a period of good performance (e.g., 10 consecutive rounds), TS increase, but the score would be deducted once the client is detected as malicious, which effectively forces the attacker not to poison or reduce the intensity of poisoning to a negligible level.

Aggregation. After assigning TS to all clients, we transform TS into weights which are used to average clients' gradients as the aggregation gradient g , i.e.,

the weight of client i is calculated by where i, j = 1 , 2 , ..., n .

Advantages. Compared with previous methods, RECESS has four advantages:

(a) Provide a more accurate measure of the abnormality extent for each client's update to assign the trust score.

(b) Enable the defender to put clients' previous performance into account, not only in each iteration, enhancing the detection fault tolerance.

(c) Protect benign outliers while effectively detecting carefully-constructed poisoning gradient updates, improving model accuracy and generalization.

(d) Outperform previous FLTrust and ML-based detections in terms of efficiency with no need for extra data.

4 Evaluation

4.1 Setup

Datasets and FL Setting. Table 1 shows four datasets and parameter settings used in the evaluation. The IID and Non-IID are both considered in the dataset division. We also consider two typical types of FL, i.e., cross-silo and cross-device.

Eight defenses are considered in the comparison experiments. Specifically, the common FedAvg is used as the benchmark. Five classic robust rules, Krum, Mkrum, Bulyan, Trmean, and Median, can provide convergence guarantees theoretically. Two SOTA defenses, FLtrust and DnC, are also involved. The parameters of these eight defenses are set to default. For RECESS , we set A = 0 . 95 , TS 0 = 1 , and baseline _ decreased _ score = 0 . 1 unless otherwise specified.

Attack Setting. We consider the five latest poisoning attacks including LIE, optimization attack, AGR-tailored attack, AGR-agnostic Min-Max attack, and Min-Sum attack. We also set two scenarios, i.e., white-box and black-box, where attackers have and does not have knowledge of other benign gradients. We assume 20% malicious clients as the default value unless specified otherwise. For the test gradient construction, we select and add noise to the first 10% of dimensions, iteratively adjusting until cosine similarity before/after meets the threshold.

2

4

6

8

10

12

14

16

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

Benign

Malicious

Detection Round

Trust score changing trend

0

2.5

5

7.5

10

count

Malicious

Class

Malicious

Benign

1

0.5

0

0.5

1

Abnormality

0

2.5

5

7.5

10

Benign

Metric. Accuracy is used to measure the performance of the well-trained model with or without poisoning. Higher Accuracy indicates better defense.

4.2 Comparison with SOTA

Table 2 shows the main comparison results. RECESS obtains higher accuracy than two SOTA defenses under latest poisoning attacks. Due to the space limit, here we mainly show the results on two datasets.

Defender's Goal (1): Defensive Effectiveness. As we can see from each row in Table 2, RECESS can defend against model poisoning attacks, but previous defenses have limited effect, especially for the strongest AGR attack series in the white-box case. The reasons are twofold:

(a) Some malicious gradients evade defenses to be selected for aggregation.

(b) Benign outliers are misclassified as malicious gradients and removed from the aggregation, especially for Non-IID datasets (e.g., FEMNIST) with more outliers.

Moreover, benign outliers are intractable for all previous defenses. FLTrust is better than other defenses since its normalization limits the large magnitude gradients. In contrast, RECESS can

0.1

0.2

0.3

0.4

0.5

0.0

0.2

0.4

0.6

0.8

Degree of Non-IID

White-box

Trmean

Fltrust

0.1

0.2

0.3

0.4

0.5

0.0

0.2

0.4

0.6

0.8

Degree of Non-IID

Black-box

DnC

our RECESS

5%

10%

15%

20%

25%

0.0

0.2

0.4

0.6

0.8

Proporation of attackers

White-box

Trmean

Fltrust

5%

10%

15%

20%

25%

0.0

0.2

0.4

0.6

0.8

Proportion of attackers

Black-box

DnC

our RECESS

effectively distinguish benign outliers from malicious gradients, thus obtaining the highest accuracy, even in the most challenging white-box Non-IID case.

Defender's Goal (2): Model Performance. As shown in the first row in each set of Table 2 where no attacks are under consideration, robust rules will reduce the accuracy but have little impact on RECESS . This can be explained from three aspects:

(a) Due to the existence of attackers, only 80% benign clients participate in the aggregation compared with no attack (e.g. 40 vs 50 for CIFAR-10).

(b) Existing defenses discard benign gradients more or less, especially for benign outliers.

(c) Malicious gradients are selected for aggregation.

The first aspect is inevitable since the defender cannot control the attacker, which also causes the main accuracy gap between FedAvg and our RECESS . However, RECESS can improve the other two aspects by more accurately distinguishing whether gradients are malicious or benign.

Impact of Attacker's Knowledge. Different knowledge affects the attack, and RECESS outperforms other defenses in both white/black-box. The stronger the attack, the better the effect of RECESS . In the black-box case, the attacker cannot offset the effect of benign gradients on the aggregation without the knowledge of other benign gradients, so the poisoning is greatly weakened and RECESS is not prominent as in the white-box case.

Changing Trend of Trust Score. Figure 3 presents the trust score of benign and malicious clients in each round of RECESS detection. Here we execute the strongest AGR-agnostic Min-max attack. Figure 4 shows the histogram of each client's Abnormality value in the third detection round to illustrate the effectiveness of RECESS . RECESS can effectively distinguish malicious and benign users in both the optimization attack and AGR attacks.

4.3 Impact of FL Setting

In this part, we evaluate RECESS under various FL settings to illustrate the practicability of our proposed scheme.

Non-IID Degree of Dataset (More Outliers). The results on Non-IID FEMNIST in Table 2 show the effectiveness of our RECESS against latest poisoning attacks. Besides, we also involve the CIFAR-10 in the evaluation using the Non-IID dataset construction method same as [11, 12]. We consider four defenses (the best classic robust rules Trmean, SOTA FLtrust and DnC, and RECESS ) against the strongest AGR-agnostic Min-Max attack in white/black-box cases.

Figure 5 shows that the increase in the Non-IID degree leads to an expansion of benign outliers which are similar to malicious gradients, and previous defenses cannot effectively distinguish outliers from malicious gradients. However, it has little impact on RECESS , since our method can accurately detect malicious gradients and benign outliers.

Number of Malicious Clients. Figure 6 shows that RECESS remains an outstanding defending effect all along, as the number of malicious clients increases, all poisoning attacks are more powerful, while the defending effect of other defenses decreases sharply. We vary the proportion of malicious clients from 5% to 25%, consistent with [12]. The other settings remain the same as Section 4.3.

Cross-device FL. Previous evaluations are mostly under cross-silo settings except for the FEMNIST which is naturally cross-device shown in Table 2. Further, we consider the cross-device setting using the dataset CIFAR-10. In each epoch, the server stochastically selects 10 updates from all clients to aggregate. The attack steps remain the same, and the other settings are the same as the default.

Table 3 shows that similar to the result of cross-silo, RECESS still achieves the best defense effect. Besides, the poisoning impacts under the cross-device setting are lower than the one of the cross-silo setting, because the server selects less number of clients for aggregation and ignores more malicious gradients in cross-device FL, thus the attacker cannot continuously poison, weakening the impact of the poisoning. Consequently, this leaves less improvement space for RECESS and other defenses.

4.4 Adaptive Attacks

Active Evasion. With knowledge of RECESS, the attacker checks if it's a test gradient before deciding to poison or not. The adaptive strategy involves estimating an aggregated gradient g p in each iteration from controlled benign clients. Then, the attacker compares the received aggregated gradient g agg with g p and g k -1 i from the last iteration. If S C ( g agg , g k -1 i ) ≥ S C ( g agg , g p ) , the poisoning begins, while if S C ( g agg , g k -1 i ) < S C ( g agg , g p ) , the poisoning stops, as it's considered detected.

As a countermeasure, the defender boosts detection frequency and intersperses it during the entire training, instead of fixed consecutive detection, and increases the sensitivity of parameters. Note that the defense should aim to make the model converge, it is not necessary to find the attacker. In other words, RECESS 's presence increases the attack cost, so even if the attacker evades detection, the loss of model accuracy is negligible, which is acceptable.

Table 4 shows that the adaptive attack has little impact on the final model accuracy though it is stealthier than the continuous attack. This is due to the similarity between the test gradient and the normal aggregated gradient, leading to inaccurate estimation by the attacker and a decrease in aggregation weight. Also, increasing detection frequency lowers poisoning frequency, so the poisoning impact will be gradually alleviated by normal training over time.

Poisoning before the Algorithm Initialization. The detection basis is that the variance of the malicious gradient is larger than the benign gradient. Malicious gradients, solved by the optimization problem, are more inconsistent. It is also theoretically proven that the optimization problem amplifies this variance. Thus, RECESS is unaffected by initial poisoned gradients as it does not change this basis. Besides, when the poisoned aggregated gradient was used to detect, most clients will be identified as malicious. This violates the assumption of Byzantine robustness that requires more than 51% of users to be benign. Hence, the defender can easily discern a potentially poisoned initial gradient based on the new abnormality definition of RECESS .

Inconsistent attacks. Attackers can also first pretend as benign client to increase the trust score, and then provide the poisoning gradients. However, RECESS detects such behavior inconsistencies over time. The trust scoring of RECESS also incorporates delayed penalties for discrepancies between a client's current and past behaviors (shown in Section 3.3). Additionally, three factors limit intermittent attacks' impact:

(a) FL's self-correcting property means inconsistent poisoning is much less impactful. Attackers would not take this approach in practice.

(b) In real settings, clients participate briefly, often just once. Attackers cannot afford to waste rounds acting benign before attacking.

(c) Defenses aim to accurately detect malicious updates for model convergence. Even if poisoning temporarily evaded detection, attack efficacy would diminish significantly, making it no longer a serious security concern.

5 Discussion

This work primarily focuses on untargeted model poisoning, along with previous SOTA works. We believe that model poisoning has a significant impact on real-world FL deployments for three reasons:

(a) Model poisoning through direct gradient manipulation poses a greater threat than backdoors.

(b) Backdoors are considered special cases of model poisoning, as malicious gradients can be obtained on poisoning datasets. However, it is challenging to solely imitate model poisoning gradients through training backdoors.

(c) Untargeted poisoning has a more severe impact on overall model performance degradation compared to backdoors in FL.

As RECESS detects inconsistencies between client behaviors, it could mitigate targeted attacks from clients with backdoored data by deploying additional strategies. Due to space limitation, we refer the readers to our technical report [20] for more details.

6 Conclusion

In this work, we proposed RECESS , a novel defense for FL against the latest model poisoning attacks. We shifted the classic reactive analysis to proactive detection and offered a more robust aggregation mechanism for the server. In the comprehensive evaluation, RECESS achieves better performance than SOTA defenses and solves the outlier detection problem that previous methods can not handle. We anticipate that our RECESS will provide a new research direction for poisoning attack defense and promote the application of highly robust FL in practice.

Acknowledgments

We thank all the anonymous chairs and reviewers for their valuable guidance and constructive feedback. Hui Li is partially supported by the National Natural Science Foundation of China (61932015), the National Key Research and Development Program of China (2022YFB3104100), Shaanxi innovation team project (2018TD-007), and Higher Education Discipline Innovation 111 project (B16037). Wenjing Zhang and Xiaodong Lin are partially supported by Natural Sciences and Engineering Research Council of Canada (NSERC). Part of Haonan Yan's work is done when he visits the School of Computer Science at the University of Guelph.

References

Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine , 37(3):50-60, 2020.

Wensi Yang, Yuhang Zhang, Kejiang Ye, Li Li, and Cheng-Zhong Xu. Ffd: A federated learning based method for credit card fraud detection. In International conference on big data , pages 18-32. Springer, 2019.

Jie Xu, Benjamin S Glicksberg, Chang Su, Peter Walker, Jiang Bian, and Fei Wang. Federated learning for healthcare informatics. Journal of Healthcare Informatics Research , 5(1):1-19, 2021.

Nguyen Truong, Kai Sun, Siyao Wang, Florian Guitton, and YiKe Guo. Privacy preservation in federated learning: An insightful survey from the gdpr perspective. Computers & Security , 110: 102402, 2021.

Derek A Lackey. Data protection and privacy in canada: A balanced approach. Applied Marketing Analytics , 6(4):366-376, 2021.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Rachid Guerraoui, Sébastien Rouault, et al. The hidden vulnerability of distributed learning in byzantium. In International Conference on Machine Learning , pages 3521-3530. PMLR, 2018.

Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing federated learning through an adversarial lens. In International Conference on Machine Learning , pages 634-643. PMLR, 2019.

Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. Advances in Neural Information Processing Systems , 30, 2017.

Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantine-robust distributed learning: Towards optimal statistical rates. In International Conference on Machine Learning , pages 5650-5659. PMLR, 2018.

Xiaoyu Cao, Minghong Fang, Jia Liu, and Neil Zhenqiang Gong. Fltrust: Byzantine-robust federated learning via trust bootstrapping. In NDSS , 2021.

Virat Shejwalkar and Amir Houmansadr. Manipulating the byzantine: Optimizing model poisoning attacks and defenses for federated learning. In NDSS , 2021.

Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Gong. Local model poisoning attacks to { Byzantine-Robust } federated learning. In 29th USENIX Security Symposium (USENIX Security 20) , pages 1605-1622, 2020.

Gilad Baruch, Moran Baruch, and Yoav Goldberg. A little is enough: Circumventing defenses for distributed learning. Advances in Neural Information Processing Systems , 32, 2019.

Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010 , pages 177-186. Springer, 2010.

Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582 , 2018.

Hangyu Zhu, Jinjin Xu, Shiqing Liu, and Yaochu Jin. Federated learning on non-iid data: A survey. Neurocomputing , 465:371-390, 2021.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems , 25, 2012.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Haonan Yan, Wenjing Zhang, Qian Chen, Xiaoguang Li, Wenhai Sun, Hui Li, and Xiaodong Lin. Recess vaccine for federated learning: Proactive defense against model poisoning attacks. https://arxiv.org/abs/2310.05431 .

Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics , pages 2938-2948. PMLR, 2020.

Saeed Mahloujifar, Mohammad Mahmoody, and Ameer Mohammed. Universal multi-party poisoning attacks. In International Conference on Machine Learning , pages 4274-4283. PMLR, 2019.

Cong Xie, Oluwasanmi Koyejo, and Indranil Gupta. Fall of empires: Breaking byzantinetolerant sgd by inner product manipulation. In Uncertainty in Artificial Intelligence , pages 261-270. PMLR, 2020.

Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, and Bo Li. Manipulating machine learning: Poisoning attacks and countermeasures for regression learning. In 2018 IEEE Symposium on Security and Privacy (SP) , pages 19-35. IEEE, 2018.

Luis Muñoz-González, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back-gradient optimization. In Proceedings of the 10th ACM workshop on artificial intelligence and security , pages 27-38, 2017.

Xueluan Gong, Yanjiao Chen, Qian Wang, and Weihan Kong. Backdoor attacks and defenses in federated learning: State-of-the-art, taxonomy, and future directions. IEEE Wireless Communications , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose RECESS, which is a proactive defense against untargeted model poisoning attacks. Specifically, RECESS proactively detects malicious clients with test gradients and robustly aggregates gradient with a new trust scoring based mechanism. For the former, the key insight is that the update goals of malicious clients are different from benign clients and outliers. Therefore, the defender can amplify this difference by delicately constructing aggregation gradients sent to clients, and then more accurately detect malicious clients and distinguish benign outliers by the comparison between constructed gradients and the corresponding responses. For the latter, a new trust scoring based mechanism is introduced to estimate the score according to the user’s performance over multiple iterations. Experiments on four datasets and various settings including white/black-box, cross silo/device FL, etc are conducted. The authors also compare the proposed approach with five classic and two SOTA defenses.\n\nSTRENGTHS:\nThe paper is well-motivated to solve two limitations in existing poisoning defense: (1) existing methods are susceptible to the optimization-based model poisoning attacks, leading to accuracy loss, and (2) existing methods cannot distinguish malicious gradient and benign outliers, reducing the generalization performance of the trained model. The proposed method is well-justified both theoretically and experimentally. Extensive evaluations on four datasets with various FL settings and attack settings, as well as the comparison with five classic and two SOTA defense are provided.\n\nWEAKNESSES:\nWhile this is a well-written and well-motivated submission, this reviewer has the following concerns. \n       •\tLimited capacity of defense. The observation that malicious clients aim to make the poisoned aggregation direction as far away from the aggregation gradient without poisoning as possible to maximize the poisoning effect only holds for untargeted model poisoning.\n       •\tDiscussion of compliance missing. While the authors design the test gradients and send it to the clients for identifying the malicious clients, it is unclear how would the malicious attack comply with the designed the rule? Can malicious clients cheat on the test gradient?\n       •\tTechnical details require additional clarification. (1) It seems for this reviewer, it is not clear how the “slight adjust the direction of g_t_i by heuristically selecting several elements” is done. What are those elements? (2) the models evaluated in the experiment are relatively simple and shallow. This raises the concern of scalability to deeper models. (3) Missing details for IID/Non-IID, cross-silo/cross-device configurations, such as how many clients or devices, how much data per client, how to generate the non-IID division.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

comments: SUMMARY:
This paper proposes a new defense against model poisoning attacks in FL. The idea is that the server sends some perturbed aggregation gradient to clients in some selected training rounds, and based on the responses, the server adjusts trust scores for the clients. Experimental results show the effectiveness of the proposed defense against state of the art attacks and outperforms state-of-the-art defenses.

STRENGTHS:
+ Important and relevant research problem. 

+ Well written paper. 

+ Interesting and novel approach.

WEAKNESSES:
I didn't see severe weaknesses. I think the paper is above the bar. 

I have three suggestions:

1. Since the paper talks about detection. I would suggest also comparing with detection methods, e.g., the following:

FLDetector: Defending federated learning against model poisoning attacks via detecting malicious clients. In KDD, 2022. 

This detection method may also be adapted to assign trust scores for clients. 

2. All the evaluated attacks are for compromised genuine clients. Recent attacks use fake clients, e.g., the following:

Mpaf: Model poisoning attacks to federated learning based on fake clients. In CVPR Workshop, 2022. 

The paper may want to evaluate such attacks. 

3. The paper seems to assume that the server sends aggregated gradient to the clients. This is different from standard FL. The paper can make this more clear. Algorithmically, it is equivalent to send the new global model to clients, but it's good to make this clear.

QUESTIONS:
See above.

LIMITATIONS:
See above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents RECESS, a backdoor defense method in federated learning. The central server keeps querying each client and tries to estimate the trust score of each client to determine if it is malicious or not. The underline intuition of the anomaly detection method is that the malicious client will push the gradient far away from benign directions.\n\nSTRENGTHS:\nThe paper is clearly written with clear labels and easy-to-follow logic. \n\nThe method is evaluated on several datasets and federated learning settings.\n\nThe paper considers the adaptive settings to evaluate its strengths.\n\nWEAKNESSES:\nThe paper does not validate its assumption. The goal of the adversary in poisoning attacks is to mislead the output with the trigger, and there is no need to manipulate the malicious gradient so that it is far away from benign ones. In fact, many works have demonstrated that some \"unintended backdoors\" only have benign samples, and there are no real malicious triggers. The adversary can later generate triggers based on the trained model (trained on only benign data). As such, I think the paper needs to validate its own assumption or intuition before applying it to practice, especially considering that this may be wrong.\n\nI do not see how the method overcomes the limitations of existing methods. The paper mentions that existing work has false positives, i.e., identifying benign ones as malicious. In a typical federated setting with non-IID data, it is common that some clients will have significantly different gradient optimization directions, which can be far away from others and invalidate the detection assumption of this method. How does this method prevent such false positives from happening?\n\nThe evaluation does not consider methods with theoretical guarantees or stronger adaptive attackers. For example, is it possible to give benign responses to only test gradients while sending malicious gradients to the server?\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nThe paper does not discuss its limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo defend the untargeted poisoning attack on the federated learning, the authors proposed a new defense called as RECESS, which exploits the outlier detection to analyze the gradients returned from clients. Once the gradient from one client is judged as outlier, this client will be considered as malicious client. The outlier detection is based on the intuition, i.e., for the benign samples even including the outliers, their updated gradients always point to the direction leading to a (local) minimum loss value, while the gradients of poisoned data will point to the reversed direction.\n\nSTRENGTHS:\nSimilar as previous refs [11, 12] as shown in the paper, this work aims to alarm a malicious gradient from a group of gradients updated from clients in the federated learning. The paper is well-written and can be easily understood.\n\nWEAKNESSES:\nThe main weakness of this work is due to its trivial contribution compared with the STOA [11, 12]. The main intuition for outlier detection, as mentioned in ‘Summary’, has been considered. Compared with STOA, the new metric ‘trust scoring’ does not improve too much as shown in Table2.\n\nQUESTIONS:\nFor the trust score, is it possible for attackers to first pretend as benign client to increase the trust score, and then provide the poisoning gradient? How should the defender limit this kind of attack? Could the attacker perform the adaptive attack to poison the global model?\n\nLIMITATIONS:\nOne limitation of this work is that it only considers the non-target attack, which is less stealthy than target backdoor attack. The main reason is that the non-target attack’s goal is to affect the normal performance of the model, which can be easily recognized by the trainer. I recommend the authors mentioned this limitation in the paper. For more details on the difference between target and non-target attack, please check this survey. \n\nE. Cinà, K. Grosse, A. Demontis, S. Vascon, W. Zellinger, B. A. Moser, A. Oprea, B. Biggio, M. Pelillo, and F. Roli. Wild patterns reloaded: A survey of machine learning security against training data poisoning. ACM Comp. Surveys, 2023. \n\nGuo, Wei, Benedetta Tondi, and Mauro Barni. \"An overview of backdoor attacks against deep neural networks and possible defences.\" IEEE Open Journal of Signal Processing (2022).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}