Trusted Aggregation (TAG): Backdoor Defense in Federated Learning
Authors: Joseph Lavond
Minhao Cheng
Yao Li
Venue: Accepted by TMLR ()

Abstract: Federated learning is a framework for training machine learning models from clients with multiple local data sets without access to the data in its aggregate. Instead, a shared model is jointly learned through an interactive process between a centralized server that combines locally learned model gradients or weights from the client. However, the lack of data transparency naturally raises concerns about model security. Recently, several state-of-the-art backdoor attacks have been proposed, which achieve high attack success rates while simultaneously being difficult to detect, leading to compromised federated learning models. In this paper, motivated by differences in the logits of models trained with and without the presence of backdoor attacks, we propose a defense method that can prevent backdoor attacks from influencing the model while maintaining the accuracy of the original classification task. TAG leverages a small validation data set to estimate the most considerable change a benign client's local training can make to the shared model, which can be used to filter clients from updating the shared model. Experimental results on multiple data sets show that TAG defends against backdoor attacks even when 40 percent of user submissions to update the shared model are malicious.

Published in Transactions on Machine Learning Research (11/2024)
Trusted Aggregation (TAG): Backdoor Defense in Federated Learning
Joseph Lavond
jlaoond@email.uncc unc.
Department of Statistics and Operations Research University of North Carolina at Chapel Hill
Minhao Cheng
manc7149@psu. edu
College of Information Sciences and Technology The Pennsyluania State University
Yao Li
yaoli@email. unc. edu
Department of Statistics and Operations Research University of North Carolina at Chapel
Reviewed on OpenReview: https: openreviee, net/ forum? id= r9eNUDe2im
Abstract
Federated learning is a framework for training machine learning models from clients with multiple local data sets without access the data in its aggregale. Instead, a shared model jointly learned through an interactive process between centralized server that combines locally learned model gradients weights from the client. However, the lack of data trans- parency naturally raises concerns about model security. Recently, several state-of-the-ard backdoor attacks have been proposed, which achieve high attack success rates while simul- taneously being difficult to detect, leading to compromised federated learning models. In this paper, motivated by differences in the logits of models trained with and without the presence of backdoor attacks, we propose a defense method that can prevent. backdoor at- tacks from influencing the model while maintaining the accuracy of the original classification task. TAG leverages small validation data set to estimate the most considerable change benign client's local training make to the shared model, which can be used to filter clients from updating the shared model. Experimental results on multiple data sets show that TAG defends against backdoor attacks even when 40 percent of user submissions to update the shared model are malicious.
1 Introduction
Federated learning (FL) is a promising solution for constructing machine learning models from numerous local data sources that cannot be directly exchanged or aggregated (Yang et al., 2019 Kairouz et al. 2021 These limitations become particularly crucial contexts where data privacy and security are prominent concerns (Li et al. 2020), with healtheare being a prime example. Additionally, FL has garnered significant attention from companies that opt to offload computing workloads onto local devices. Furthermore, FL allows for non-independent and non-identically distributed local data sets. Hence, shared and robust global model is often unattainable without collaborative learning. Within this FL framework, local entities, called clients, contribute their locally acquired model gradients or weights to be intelligently combined by some ralized entity, the server, resulting in shared machine-learning model.
However, concerns have arisen regarding the potential vulnerabílities inherent FL. The lack of control or knowledge of the local training procodures allows malicious users to create updates, compromising the global model for all participating clients. One insidious threat, especially for classification models, is the

---

Published in Transactions on Machine Learning Research (11/2024)
targeted backdoor attack, in which malicious actors seek to manipulate the global model into associating specific input data manipulations (known as triggers) with particular outcomes (the target), for example, a particular predicted class label. While various methods have been proposed to detect triggers and defend against backdoor attacks (Kurita et al.) 2020 Qi ct al. 2020 Li et al.) 2021). these approaches often rely on having access to the training data itself, which is not leasible within the FL paradigm. Moreover, the limited information available to detect and prevent such malicious intent within FL makes backdoor attacks more straightforward to execute and more challenging identify.
In this article, we shed light on approach to detect and mitigate backdoor attacks in FL. Our primary observation centers on the substantial divergence between the logits of classification models created by malicious users and those produced by benign users. This discrepancy is most pronounced for the class label targeted by the backdoor attack. Leveraging this stark contrast, we propose a method that employs a minimal, clean data set to generate a backdoor-free, locally trained model, which we term the "trusted user". We then compare other user models against this trusted user model on the same clean data to identify models with unusual outputs. Suspicious users are removed from participating in the update of the global model.
To identify unusual outputs, we propose comparing user and trusted models by the distributional difference between their outputs and those of the most recent. global model to identify malicious updates. We use the trusted user to estimate the most considerable distributional difference a benign user's local training could produce and eliminate other returning user models that exceed this distance cutoff. We show that robust aggregation is insufficient to defend against backdoor attacks and that our framework is more effective than similar proposed defenses. In FL., multiple attackers can simultancously attempt to attack the shared model with either the same different malicious objectives. Our proposed method demonstrates its effectiveness against backdoor attacks with multiple attackers, even when 40 percent of returning updates are part of an attack. Remarkably, our approach consistently outperforms existing methods and remains robust even when backdoor attacks persist throughout federated training. Moreover, our method does not compromise the global model's performance on clean data, ensuring that the accuracy of the original classífication lask remains intact. We provide experimental evidence across multiple data sets underscore the consistency and reliability of our results.
The rest of the paper is organized as follows: Section 2 briefly summarizes related work from federated learning, backdoor attack, and defense. We introduce the proposed framework for excluding malicious users from participating in the update of the global model in Section 3, Finally, Section 4 gives empirical evidence for performance improvements over other defense options.
2 Related Work
Federated Learning Foderated learning (FL) an emerging machine learning paradigm with great suc- cess in many fields (Bonawitz al.] 2019 Hard et al., 2018 Ryllel et al. 2018). At its core, FL operates through iterative rounds of model improvement. In each round, the global model is distributed to partici- pating users, and subset of these users is selected to update local copy of the model. These chosen users train their models on their respective local data sets. The resulting models are shared to safeguard data privacy and aggregated to construct a new global model.
Backdoor Attack Recently, the FL setting has become target for various backdoor attacks. In Xie et al. (2020), the authors highlighted how the multi-user nature of FL could be exploited to create more potent and persistent backdoor attacks. Distributing the backdoor trigger among a few malicious users effectively induced the desired behavior in the global model at higher rates and exlended periods after the attack had ceased. Another notable contribution in this domain is the projection method known as Neurotoxin (Zhang ct al. ) This approach projects the attacker's s updates onto dimensions with small absolute weight vector values, claiming that benign users update such weights less frequently, leading to longer-lasting successful attacks. Our research rigorously evaluates our proposed method's effectiveness against. both attacks.
2

---

Published in Transactions on Machine Learning Research (11/2024)
Defense While FedAvg (McMahan et al. 2017) remains the most prevalent aggregation method in FL, the field has since introduced robust aggregation rules. In particular, Median and Trim-mean, two such methods, were proposed in Yin et al. (2018). The Median operation computes the coordinate-wise median among the weight vectors of selected users. Similarly, the Trim-mean procedure aggregates weight vectors by calculating the coordinate-wise mean after excluding the largest and smallest & elements in that coordinate. Our experiments show that these robust aggregation methods are insufficient against backdoor attacks.
Defending against backdoor attacks in FL has been relatively unexplored area in the literature. Many previously proposed defenses, such Neural Cleanse (Wang et al. 2019), are infeasible for FL, as they jeopardize data privacy. While prior work (Shejwalkar et al. (022) suggested that norm clipping (Sun et al. 2019) could be effective against backdoor attacks in FL, it is vulnerable to the Neurotoxin attack. Other works on federated learning include DeepSight (Rieger et al., 2022), Baffle (Andreina et al., 2020). and FLAME (Nguyen et al., 2021). We focus on comparison with FL.Trss (Cao et al., 2020); as it (similar our proposal) requires an additional small, clean data set. Although the original FLTrust paper showcasod its effectiveness against adaptive attacks involving around half of the clients being malicious, our research demonstrates its failure even against the most basic backdoor attacks considered in experiments. Thus, our work addresses critical gap in federated learning and enhances similar defense methodologics.
3 Trusted Aggregation (TAG)
This section begins with an observation in Section 3.1 1hha model performance clean data changes depend- ing on whether the model was backdoor attacked. Next, Section 3.2 introduces our framework for excluding users from updating the global model depending on their local model's performance on some clean data. Finally, Section 3.3
provides smoothing procedure strengthen our defense further.
3.1
Motivation
Target:
Class:
Density
0.5
1
0.5 Class Score
0.5
0.5
Class Score
Class Score
Class Score
Figure 1: Output distributions (kernel density estimation based) conditional on the class label for backdoor model (black) and a clean model (red). Note the obvious difference between the distributions of the backdoor and clean models for the target label class.
Our proposed method, Trusted Aggregation (TAG), is motivated by the observation that the distributions of model logits generated by malicious users significantly differ from those produced by benign users. We use the term logits refer the output. of a classification model before the softmax operation produces probabilitics. Note that each clement of the logit vector corresponds to preciscly one class, establishing a unique node-class association for the logits.
In targeted backdoor attacks, malicious users aim to create additional learned association between particular manipulation of input data (the trigger) and a specific class label (the target). The additional learned association ellectively transforms the task into a (m + 1)-way classification problem. We intuitively believe this difference in the model's task can be exploited to identify models trained with a backdoor attack. In Figure 1. we demonstrate that this learned association can lead a distributional change in the logits on non-attacked inputs, particularly for the target class. Our insight suggests that models containing a backdoor may produce distinct distributions of logits when provided with clean data. Consequently, if we
3

---

Published in Transactions on Machine Learning Research (11/2024)
have model believed to be trained without backdoor attack, can identify whether another candidate model exhibits signs of a backdoor attack by comparing their logits the same clean data.
3.2 Detection Framework
Our detection framework assumes the presence of a small, clean validation data set, which serves as the gatekeeper for updates to the global model. This data set can either belong to a trusted existing user or be collected by the centralized server and treated a new user. We will refer this trustworthy validation data set the "trusted user".
Our detection method leverages the trusted user to evaluate incoming model weights and determine whether each contribution can participate in the global model update process. The core idea is to detect user models with unusually distributed outputs using a clean data set from the trusted user. Our method can be easily extended when multiple trustworthy data sets or users are available. Refer to Figure for an overview of our proposed detection framework.
Compare o and o(T)
Server
( o()
Distance
-(X00) 0
Score vj
((2)
(
Threshold
x
(00 (7)
x Trusted data
G: Global model at round start
U(); Local model of the client
( Trusted model
Distance (1)
p(2)
(3)
(())
o 00; Logits of the jth client
Scores:
(9); Logits of the global model
o ( Logits of the trusted model
If mmxx(v from the aggregation
( > Uy is excluded
Figure Diagram reprosentation of our trusted aggregation detection framework. A distance score (v()) is calculated for each selected user's model (UU) based the distributional distances between the user's 's model and the global model. A threshold (r) is computed based the distributional distances between the trusted user's model (U(T) and the global model (G). If the distance score of a user is greater than the threshold, it will be excluded from the aggregation. See more details in Section 3.2
In each communication round, the trusted user performs the following steps to establish threshold for detecting malicious users:
1. The validation data is utilized to update copy of the current global model simultancously with the local training other users.
2. When models are returned by the subset of users selected to participate in the shared model update potentially, logits are generated and stored for the validation data. These logits are denoted as o (G) (T)) and (f for the global, validation, and j-th user models, respectively.
Subsequently, we compute the distance between each user from the current global model. Specifi- cally, for each class c, we calculate the class-conditional distributional distance between empirical distributions for the current global model's logits and the user's logits using some distributional difference function.
4

---

Published in Transactions on Machine Learning Research (11/2024)
4. Given m total classes, the process generates distance vector v for each user, including the trusted user. These distance vectors then dictate which users can participate in the model. See Algorithm for additional details.
Algorithm Trusted Aggregation (TAG) Let 8 denote the subset of users selected to update the global model for given round
Input: Global model G, user models {UO)), Jees trusted model U(2) and data X and scaling coefficient
0 1.
1: Generate logits (( = G(X(T)) oنا) = (T)) for j E S, and o U (X())
2: for Each class e E [I,..., m] do
3: Compute distributional distances between each user and the global model v = D (oo"" .o: for
JES S and - = D 0c
(G)
end for Compute threshold for communication round
7, = 0 x max. ve [T)
Exclude suspicious users
S, = { S maxe (v()) $ tr cs
return Update with (U61) jES,
We aim to estimate the maximum change a non-malicions introduce in communication round defined below in Equation
6) o. o!!)))
= max max D
(1)
Users with distance values surpassing this threshold should be excluded from the update process. For estimation T,, we compute max. 'D oo oo (G) for our trusted user. Note T, involves the maximum of all benign users. Since the validated user is non- of other non-malicious users. However, non-malicious, their distance vector serves as good representation large
scale by 2 since the actual maximum will be at least observed trusted user. A user with maximum distance smaller than the threshold f, x max. D { is considered a benign user. In comparison, a user with a maximum distance larger than or equal the threshold will be removed. However, this naive threshold is precarious, and due to its instability, lucky malicious user can get past it in some rounds. To overcome this limitation, we propose specific smoothing procedure in Section 3.3
If the distributions of v, = D oo oe (G) can be assumed, the ranges of plausible 0's can be better deter- mined.
Proposition If Ue Uniform(0, ba) for all classes E [1,...,m] and for all benign users, then [v] E [20] where maxa [va] and max. [b.].
For example, when v are each uniformly distributed for all users, Proposition suggests that, on average, f = x max, [T)) should equal for some E [0, 1]. Yet, we acknowledge that it may be unreasonable to assume that class conditional distances are Uniform many training lyper-parameters and even model choice will impact the distance distributions. A better approach could be to allow the data to determine the scaling factor, i.c., selecting the scaling factor based on the distribution of logits observed in the experiments. However, in our experimental results in Section $ we often found that simply setting 0 = 2 outperforms existing methods, so we did not further explore data-dependent scaling factors. See Section [.2] for a sensitivity analysis on how our scaling cocfficient impacts model performance for non-iid users. We present these results to assist in understanding reasonable magnitudes for 0 as many distributions may not require large scaling values.
5

---

Published in Transactions on Machine Learning Research (11/2024)
We recommend choosing 8 based the setting's prevalence of backdoor attacks (potentially unknown) and the cost of a successful attack to interested parties. Relatively large values for 0 will allow more users to update the global model but increase the risk of a successful backdoor attack. Conversely, using our threshold fr without scaling would help to prevent stronger backdoor attacks, but with the potential loss of denying benign users from updating the global model. The goal is to choose the smallest that allows benign users to create a robust shared machine learning model sufficiently. For a more substantial discussion on the fairness of our algorithm, please see Section B.1
3.3 Global-Min Mean Smoothing (GMMS)
Stabilizing the threshold value is essential for maintaining the security of our method. While a straightforward approach to achieving this stability is through a smoothing technique, such as a moving average, it comes with challenges. The naive threshold value fluctuates in the early communication rounds as the model learns to relate imputs and output classes quickly. We need to understand when past behavior of f, relevant while ensuring stability throughout the process.
Conventional smoothing methods, which rely several previous values, can lead to an overly high threshold in the initial rounds. A falsely high cutoff, in turn, could create vulnerabilities that attackers could exploit. To address this concern, we introduce our Global-Min Mean Smoothing approach, which combines the benefits of both a stable threshold and a rapidly adjusting threshold carly in training.
The foundation of our approach lies using the lowest observed value of ř, (Global Min) as the starting point for the (Mean) smoothing window. Here, fr represents the naive threshold estimation up round The smoothed threshold in for round is determined using Algorithm
Algorithm 2 Global-Min Mean Smoothing (GMMS)
Input: Threshold history fr. Identify the start of the smoothing window arg min;: ft
2: return Smooth estimate 7,
f
In scenarios where + cxhibits rapid initial decrcases, we should observe new global minimums, which serve as reset point for the threshold smoothing. Starting our smoothing from the global minimum allows us to maintain the original threshold sequence's decreasing behaviors. However, when the original sequence is not experiencing decline, previous values utilized to smooth the threshold, effectively pre- venting lucky malicious users from evading volatile thresh- old.
muviny average (window-s)
eoposential smoothing
global smoothing
0.8
02
Figure B visually comparcs our Global-Min Mean Smooth- ing with the base (naive) threshold and various conventional smoothing methods. Smoothing not only captures the early behavior of the naive threshold but also significantly improves stability, rendering it robust choice for safeguarding the global model against.
au
Communication Round
evident that our Global Min-Mean
Figure Comparison of the global min-mean smoothing with the naive threshold and vari- ous smoothing methods.
backdoor attacks.
4 Experiments
In this section, we present a series of experiments that offer valuable insights into the effectiveness of Trusted Aggregation (TAG) as defense mechanism against strong backdoor attacks in federated learning. Tn

---

Published in Transactions on Machine Learning Research (11/2024)
observe that robust aggregation is insufficient to prevent backdoor attacks and that TAG outperforms the backdoor defense FI,Trust, which also requires additional clean data. Additionally, Sec- tion 4.4 shows that the TAG is robust to changes in the data distribution of clients or the trusted data sct.
Scction 4.2
4.1 Setting
This section provides comprehensive breakdown of the parameters used throughout study to ensure the reproducibility of our experiments, as summarized in Table D When used with the code available on our GitHub repository this detailed information serves readers seeking to recreate our results and further investigate our work.
Hyperparameter
Variable Name
CIFAR10
CIFAR100
STL10
Users
users
100
100
20
Federated Learning
Local Data Size
n. user dala
500
500
400
User Subset Proportion
report
.1
.1
.5
Padding
4
12
Data Augmentation
Random Horizontal Flip
NA
.5
.5
.5
Random Crop Size
32
32
96
Batch Size
n batch
64
128
All Users
Weight Decay
wd
5e-4
5e-4
5e-4
Local Epochs
n epochs Ir
10
Benign Users
Learning Rate
.01
.01
.01
Local Epochs
epochs pois Ir poioss p_pois
20 (15)
15
25 (15)
Malicious Users
Learning Rate
.01
.01
.005 (.01)
Poisoning Proportion
.1
.1
.1
Data Poisoning
Stamp Pixel Height
row size
4
4
24
Stamp Pixel Width
col size
4
4
24
TAG Scaling (0)
d scale
2
2
1.1
Backdoor Defense
Trim Mean
beta
.2
.2
Table 1: Default arguments for all experiments unless otherwise specified. For all experiments, alternative values for 3 did not prevent the backdoor attacks. Any values modified for Neurotoxin attacks are shown in parentheses.
Model Our experiments employ the ResNet18 model (H et al. 2016), well-established classifier. Ad- ditionally; to showcase the robustness and generalizability of our approach, we reproduce the main results using the VGG16 model (Smonnyan & Zisserman, 2014) in Section A.2 Importantly, we assume that all users, including potential malicious actors, have complete control over varions aspects of local training. For simplicity, we use two sets of hyper-parametern for benign and malicious users. The malicious users will poison (add their backdoor trigger) and change the training label to the target class for given proportion of their local data. They intend their model associate the trigger with the target class and transfer such behavior future global models.
Attack and Defense To assess the effectiveness of TAG, operate in scenario where the backdoor allack is particularly potent. We mandate that the same set of malicious users are included every round in the subset of selected users responsible for updating the global model. Moreover, all attacks start in the first communication round. This approach circumvents the randomness associated with selecting users, allowing malicious users to influence the global model repeatedly. Furthermore, the guaranteed benign validation user excluded from participating global model updates. These decisions are made showcase TAG's ability to thwart even backdoor attacks against the shared model in the most substantial attack settings.

---

Published in Transactions on Machine Learning Research (11/2024)
In our experiments, compare the performance of the TAG method with robust aggregation methods, including Median and Trim-mean (Yin et al. 2018), as well as the backdoor defense FI.Trust (Cao et al. 2020). Regarding TAG, in our experiments, we exclusively use the Kolmogorov-Smirnoe (KS) distance between distributions because it is casy to compute for estimated cumulative distribution functions. Howover, we acknowledge that other distance functions, and even divergences, may be suitable. We assess these methods against two state-of-the-art backdoor attacks 2022) and Distributed Backdoor Attacks (DBA) (Xie et al., 2020). To evaluate the strength of the considered defenses, we vary the proportion of malicious updates from 10, 20, and 40 percent of the selected users.
federated learning: Neurotoxin (Zhang et al.,
Data The experiments are conducted on three datasets: CIFARIO and CIFARIO0 (Krizhevsky & Hinton 2009), and STL10 (Coates et al. 2011). A specific adjustment pertains to our use of STL10, where we have inverted the conventional train and test. data splits. Since we are using only labeled data, we swap train and test have the larger of the two for training purposes. User datasets are constructed by random sampling from the training data splits. These samples are generated using a Dirichlet distribution that determines the class frequency. In our experiments, scaling factor, = 10000, to a vector consisting of ones, with the dimensionality equal to the total number of classes. When investigating imbalanced user (and trusted) data sets, we modify a to a value of 1 to create the desired imbalance. Further details, including experiment specifics and hy hyprpparameters, can be found in Table [
ensure the creation of balanced local data sets by applying a
We split the test set into two parts to evaluate the global model. The first half is used to determine classification accuracy. Tn the second half, introduce the backdoor trigger to the images, remove any observations related to the target class, and report the attack success rate. The attack success rate measures the extent to which the backdoor attack has compromised the model by determining the proportion of the poisoned half predicted as the target class. An effective defense method will exhibit low attack success rate while maintaining a high classification accuracy, indicating that the attack is unsuccessful and the defense does not negatively impact classification performance.
4.2 Comparison of Defense Methods Against Backdoor Attacks
To thoroughly evaluate TAG's performance, we explore settings where 10, 20, and 40 percent of the returning user models are malicious in each communication round. Figure f provides a visual representation of the performance of various methods against backdoor attacks on three different data sets. We assess the success of each method in terms of attack success rate while ensuring that classification accuracy, Figure g remains high. For our primary results, we use scaling coefficients (0) of 2, 2, and 1.1 for the CIFAR10, CIFAR100, and STL10 data sets, respectively. Our findings reveal that TAG effectively neutralizes the backdoor attack in each case without significantly compromising the classification accuracy of the original task.
Other methods, such as coordinate-wise Median, Trim-mean, and FL.Trust, fail to thwarl backdoor allacks, with or without Neurotoxin, at all considered strength levels. FLTrust, while capable of delaying attack success in some scttings, ultimately falls short in preventing backdoor attacks. The critical difference between TAG and the baseline methods is that while the baseline approaches differentiate between malicious and benign users based on update gradients, our approach compares task performance based on model out For backdoor attacks, the loss is typically a combination of the original task loss and the backdoor loss: Loss = Original Task Loss Backdoor Loss, where À is usually close to zero As a result, the gradients of malicious users may appear similar to benign users, but our method can st.ill detect differences between the resulting models. However, different local minima can produce similar model outputs in highly non-convex loss landscapes. In such cases, our method may not be as effective at filtering out models that are more easily detected by gradient-based defenses. Yet, in none of our experimental settings were gradient-based defenses successful in defending against any of the attacks we considered. We remark that our method could filter out suspicious users before other defenses, making it possible to combine with other strategies further to enhance the robustness of models against targeted backdoor altacks
Our supplemental experiment, detailed in Section A.I supports that TAG does not hinder performance for the original classification task, even without backdoor attacks. This comparison with FedAvg highlights the minimal impact on the classification task's performance when using TAG as defense mechanism.
8

---

Published in Transactions on Machine Learning Research (11/2024)
Malicious
10% Malicious
Mallicous
10% Malicious
Users
Sour Mlpheyys
CIFAR-10
0.21
0.0
1.0
0.8
CIFAR-100
0.8
0.8
0.4
200 Communication Round
100
200 Communication Rourd
200
200 Communication Round
100
200 Communication Round
1000
200 Comm cntmmmiiiaaionn Round
200 Communication Round
100
Success
Trusted Aggregation
Coordinate Medion
Coordinane 7ilm Meon
RiTust
Figure 4: Model performance under DBA without and with Neurotoxin (NT) with 10, 20, and 40 percent malicious updates on several data sets. Column names indicate attack settings, while rows correspond to data sets. The proposed method, TAG, performs well in defending against backdoor attacks as the attack success rates are low. The other methods do not work well against any backdoor attacks.
These results emphasize that TAG enhances model security without negatively affecting the classification lask, even without attack. Furthermore, Section A.2 demonstrates that our primary results, which were obtained using ResNet18, are not model-dependent by reproducing them using VGG16. In conclusion, Trusted Aggrogation (TAG) is a crucial advancement in bolstering model security within the federated learning framework.
4.3 Necessity Of Threshold Smoothing
We revisit the last attack on the STL10 data set to highlight the importance of our proposed global-min mean smoothing technique. Recall that our smoothing is intended to improve the stability our estimated threshold while preserving its behavior the initial rounds, which is not conserved by other smoothing techniques. Suppose we repeat the backdoor attack, with 40 percent of the user subset being malicious each iteration, and omit the global-min mean smoothing. In that case, our method backdoor attack with Neurotoxin projection on STL10. Please see Figure 5 We do remark that this was the only allack in our main results that becane successful without including smoothing. Regardless, without smoothing, we conclude that malicions users may be able to get past a less stable cutoff.
no longer prevent the
4.4 Extending Results To Imbalanced User Data Sets
In federated learning, user data sets may not adhere to the assumption of being independent and identically distributed. This section explores TAG's effectiveness with imbalanced user data sets, specifically focusing on the CIFAR10 data set. We investigate this scenario under the most. polent altack selting, where 40 percent user submissions are malicious each round. In Section B we include our analysis on the weaker attack settings, revealing that TAG remains highly effective in defending against backdoor attacks, even without fine-tuning the scaling coefficient (0). These results are consistent across imbalanced user data sets, irrespective of whether the trusted data set is imbalanced.

---

Published in Transactions on Machine Learning Research (11/2024)
40% Malicious
40% Malicious,
1.0
Classification Accuracy
rusted Aggregation
0.8
Coorcirate Median
Coorcinate Trim-Mean
STL-10
FLTrust
0.4
Attack Rate
"Tusled Aggreyelion
Coorcinate Median
Coorcirate thim-Mean
0.0
FLTrust
100
200
100
200
Communication Round
Communication Round
Figure 5: Model performance under DBA without and with Neurotoxin (NT) with 40% malicious updates on STL10 without using global-min mean smoothing. Column names indicate attack setting. TAG fails to defend against the backdoor attack with Neuroloxin without our proposed smoothing technique for improved stability.
In these experiments, we employ the m-dimensional Dirichlet distribution with parameter al to determine the proportion for each class label in the data set, introducing an imbalance. The choice of plays a critical role in the performance TAG. With appropriate tuning, TAG can successfully neutralize 40% of malicious backdoor attacks. See Figure However, a trade-off exists between backdoor attack prevention and original classification task performance. Smaller scaling values may reduce classification accuracy in the early communication rounds fewer users contribute to model updates. Nonetheless, the global model converges, TAG's round-to-round accuracy stabilizes and matches surpasses the baseline methods. In conclusion, TAG can mitigate backdoor attacks on imbalanced data without compromising the accuracy of the original task, underscoring its utility in a wide range of federated learning applications.
40% Malicious Intiolancedd
40% Malicious, NT Insoalanced Users
40% Malicious
40% Malicious, NT imbolonced
Users
1.0
Classitcation Acounacy
Trusted Acgregation
0.0
Coordinete Mediar
CITAR-T 10
Cuor Sinate bim--Wan'
0.6
P.Tustt
Atwek farcees Tate
Trusted Apgregation
Coordinste Mediar
coordinate
0.0
FLTrust
100
200 Round
230 Communcattio Round
100
300 200
200 Communication Round
100
Communcation
Communication Round
Figure 6: Model performance under DBA without and with Neurotoxin (NT) backdoor attacks with 40% malicious updates CIFAR10 under imbalanced local data sets with tuned 0. Column names indicate whether the trusted user (Trusted) is also imbalanced. The proposed method, TAG, performs well against backdoor attacks, even when the local user data sets are imbalanced. Again, the other defense methods do not prevent any backdoor attack under imbalanced data.
When obtaining balanced, trusted data set challenging, TAG still exhibits its efficacy. TAG effectively prevents backdoor attacks even when the trusted user's data set is imbalanced, and 40 percent of user updates are malicious. These experiments reaffirm TAG's ability serve potent defense mechanism, regardless of the balanced representation within the trusted data set. TAG remains a powerful tool for backdoor defense, even when obtaining a balanced, trusted data set proves difficult. Furthermore, Section B illustrates that significantly reducing the size the trusted data set relative to other users also does not impact the success of TAG as a defense method. Even when FLTrust operates with a full-sized, representative, trusted user data set, it is outperformed by TAG. These findings underscore TAG's robustness and versatility in handling backdoor attacks in federated learning, making it valuable addition to the security toolkit.
10

---

Published in Transactions on Machine Learning Research (11/2024)
5 Limitations
While Trusted Aggregation (TAG) demonstrates substantiall success as a defense mechanism against backdoor attacks in federated learning, several limitations must be considered:
Extreme Non-IID Distribution Our experiments do include non-independent and non-identically dis- tributed (non-lID) data sets. However, more extreme cases of non-llD data sets remain unexplored. For instance, we do not test TAG in scenarios where each user possesses the entirety of only few of the total classes. In such cases, the assumption that this work's primary objective is to learn a single shared model for all users may not be reasonable.
Limited Application Scope While our framework is conceptually extendable various machine learning models, such as regression and natural language processing (NLP), our experimental results are confined to standard classification computer vision models and databases. Future work may be needed to adapt our method and demonstrate its effectiveness in other applications.
We do not claim that TAG can successfully defend against adaptive attacks. Adaptive attacks occur when malicious attackers know our defense in use. They strive return a model with output distributions similar those of benign model while still exhibiting backdoor behavior, which can be achieved by creating two copies of the global model: A) and the other (copy B) backdoor attacked with the poisoned data set. The attacker enforces similarity between the outputs of these two copies using techniques like L2 regularization as shown in Equation 2
trained original, unmodified local data to produce benign model (copy
( - (B(x),)) v) H 1B(z)) A(x)11
Regardless, TAG outperforms other considered defense methods, even under adaptive attack. See Figure 7 However, the effectiveness of TAG diminishes when faced with strong adaptive attacks.
Adaptive Attacks 20% Malicious
10% Malicious
40% Malicious
1.0
Classificaton Accur TAG (mum0.001)
0.8
(mum00005)
TAG (mom2:003
6.4
Attack
0.2
(nu-6
6.0
Communiccaion Round
Communccation
Figure 7: Comparison of TAG under adaptive attack with baseline defense methods not under adaptive attack for various attack settings on CIFARIO. Adaptive attacks are successful against TAG and are a limitation of our proposed defense. However, even under adaptive attack, TAG exhibits lower attack success rates at termination than baseline defenses, not under adaptive attack.
Adaptive Attacks While TAG offers valuable defense mechanism for backdoor attacks in federated learning, its efficacy may be compromised in specifie scenarios and attack types. These limitations should guide further research in the field of federated learning security.
6 Conclusion
In this study, introduced Trusted Aggregation (TAG), robust defense mechanism against backdoor attacks in the federated learning framework. Our extensive experimentation and analysis have led us to several key conclusions:
Defense Efficacy TAG has proven a highly effective defense method, showcasing its ability to thwart tate--of-the-at backdoor attacks in challenging settings. While existing defense techniques falter under mild attack conditions, TAG consistently prevails against powerful adversaries.
11

---

Published in Transactions on Machine Learning Research (11/2024)
Heterogeneous Data TAG's adaptability and resilience evident in federated learning scenarios in- volving users with heterogeneous and imbalanced data. Importantly, it accomplishes this without sacrificing classification accuracy for the original task, a testament to its versatility and broad applicability, Further- more, our mothod's performance remains stable regardless of the size or distribution of the trusted data it relies on.
Compatibility TAG can be seamlessly integrated with other filtering methods or modifications to the aggregation process, enhancing its compatibility with a wide range of defense strategies. This flexibility empowers federated learning systems to adopt multi-layerod approach sccurity, further safeguarding against adversarial threats.
In summary, TAG is pivotal advancement in model security for federated learning. It raises the bar for similar defenses against backdoor attacks with its adaptability, robustness, and potential as part of holistic defense st strategy. As federated learning continues to evolve, TAG represents a valuable tool for maintaining model integrity and trust in collaborative machine- machingg environments.
7 Acknowledgments
We thank the anonymous reviewers and the Action Editor for their constructive feedback, which improved this work. This work was partly supported by the National Science Foundation under grants DMS-2152220 2152289, DMS-2134107, and the Cisco Faculty Award.
References
Sebastien Andreina, Giorgia Azzurra Marson, Helen Möllering, and Ghassan Karame. Baffle: Backdoor detection via feedback- based federated learning, 2020. URL https://arxiv: org. abs/aaas......////a////aaaanaaaa 10001..2. 02167
Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Damilry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzocchi, Brendan McMahan, et al. Towards federated learning at scale: System design. Proceedings of Machine Learning and Systems, 1:374 388, 2019.
Xiaoyu Cao, Minghong Fang, Jia Liu, and Neil Zhenqiang Gong. Fltrust: Byzantinc-robust federated learning via trust. bootstrapping, 2020. URL httttpp:///arrii.orr//aaasse
13995
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215-223, 2011.
Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswany, Françoise Beaulays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint Xrxi 1.09604, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi: D0.119/CCVR..016.900
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurōlien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends@ in Machine Learning, 14(1-2):1-10) 2021.
Alex Krizhevsky and GeofTrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pretrained models. pp. 2793-2806, July 2020. doi: 10.18653/v1 wt/p////2222///////// m49..249. URL hctps://aclanthoooggy wwwphhhooooyyoor//2200000 org/2020. acl-main. 249
12

---

Published in Transactions on Machine Learning Research (11/2024)
Li Li, Yuxi Fan, Mike Tse, and Kuo-Yi Yi Lin. A review of applications in federated learning. Computers & Industrial Engineering, 149:106854 2020. ISSN 0360-8352. doi: http:////aa.arrr//00000000001200 200000006......22200000666..
URL httpps//ww...ccenccccreee..coc www.:acccchhccccc...caaaaaaaaaaa
Linyang Li, Demin Song, Xiaonan Li, Jiehang Zeng, Ruotian Ma, and Xipeng Qiu. Backdoor attacks on
pre-trained models by layerwise weight poisoning. pp. 3023 3032, November 2021. doi: 20.18633vv1202.1
manlp-main.244 2411 URL hatpss:/acllatthlogy..oom w02g/021.......oooo.../oog emn1p sannp-mmin.......
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effciell learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.
PMLR, 2017.
Thien Duc Nguyen, Phillip Rieger, Huili Chen, Hossein Yalame, Helen Möllering, Hossein Fereidooni, Samuel Marchal, Markus Miettinen, Azalia Mirhoseini, Shaza Zeitouni, Farinaz Koushanfar, Ahmad- Sadeghi, and Thomas Schneider. Flame: Taming backdoors in federated learning, 2021. URL htttts://arart org/ aataaa/////111.0 02281
Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. Onion: A simple and effective defense against textual backdoor attacks. arXiv preprint arXiv:2011.1036; 2020.
Phillip Rieger, Thien Due Nguyen, Markus Miettinen, and Ahmad Reza Sadeghi. DeepSight: Mitigating
backdoor attacks in foderated learning through deep model inspection. In Proceedings 2022 Network and Distributed System Securily Symposium. Internet Sociely, 2022. doi: 10.14722 //d/22///ss../ http//dsss..222.2. 2022.23156. URL
https://doi. http://ooooorrg/1/// wwt//0..////...... 14722//2Fndsss 2022.23155
Theo Ryffel, Andrew Trask, Morten Dahl, Bobby Wagner, Jason Mancuso, Daniel Rueckert, and Jonathan Passerat-Palmbade A generic framework for privacy preserving deep learning. orXiv preprint arXiv:181.04401,, 2018.
Virat Shejwalkar, Amir Houmansadr, Peter Kairouz, and Daniel Ramage. Back to the drawing board: A critical evaluation of poisoning attacks on production federated learning. In 2022 IEEE Symposium Securily ond Privacy (SP), pp.
IEEE, 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2014. URL
1556
Ziteng Sun, Peter Kairouz, Ananda Thoertha Suresh, and H Brendan McMahan. Can you rcally backdoor federated learning? ar preprinl arXiv:1911.07900 2019.
Bohu Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Zhao. Neural cleanse: Identifying and miligating backdoor allacks in neural networks. pp. 707-723, 723, 05 2019. doi: 10. 1011099::009900000
Chulin Xie, Keli Huang, Yu Chen, and Bo Li. Dba: Distributed backdoor attacks against federated learning. In International Conference on Learning Representations, 2020. URL https::/openreviev. net/forum Pefforummid=-ieee idd-knns--k
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1 19, 2019.
Dong Yin, Yudong Chen, Ranchandran Kannan, and Peter Bartlett. Byzantine-robust distributed learning: Towards optimal statistical rates. In Jennifer Dy and Andrcas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5650-5659. PMLR, 10-15 Jul 2018. URL htttps://proceeding: mlr, ppress/v80yyinll html
Zhengming Zhang, Ashwinee Panda, Linyue Song, Yaoqing Yang, Michael Mahoney, Prateek Mittal, Ram- chandran Kannan, and Joseph Gonzalez. Neurotoxin: Durable backdoors in federated learning. In Ka- malika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (ods.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 26429 06429-26446. PMLR, 17 17-23 Jul 2022.
13

---

Published in Transactions on Machine Learning Research (11/2024)
Appendix
A Comparison of Defense Methods Against Backdoor Attacks (Continued)
10% Malicious Users
10% Malicious Users
10% Melicious Users
10% Melicious Users
10% Malicious Users
10% Malicious Users
1.0
CIFAR-10
0.8
0.6
CIFAR-100 R0000
0.6
0.4
0.4
0.2
0.0
200 Communication Round
100
0
200 Communication Rourd
100
100 200 Communication Round
200 Communication Round
100
200 Communication Round
100
200 Communication Rou nd
100
Classification Accuracy
Trustod Apgregation
Coordinate Median
-.-- Coordinate 3il Mean
AiTrutt
Figure 8: Model performance under DBA without and with Neurotoxin (NT) with 10, 20%, and 40% malicious updates on several data sets. Column names indicate attack settings, while rows correspond to data sets. All methods result in similar classification accuracies, indicating that TAG offers improved backdoor defense without cost to original task performance.
Threshold vs. Benign
Threshold vs. Malicious
benign: IOR
malicious
1.0
benign: 1.5 * 1OR
global min mean smoothing
global min mean smoothing
0.8
0.6
0.4
0.2
10
20
30
50
0
20 Communication Round
40
Communication Round
Figure 9: Global Min Mean Smoothing threshold visualized for the first 50 communication rounds with 10 percent malicious updates on CIFARI0. Our threshold can easily differentiate between benign and malicious updates, and almost all benign updates contribute to the global model. Similar results hold under different data sets and attack settings.
14

---

Published in Transactions on Machine Learning Research (11/2024)
A.1 TAG Classification Accuracy Without Attackers
A successful backdoor defense can prevent attackers while best preserving the model's performance the original task. The model ability for the original task must also be maintained in the absence of attack. In addition to suceessfully preventing attacks when present, we observe in Figure I0 that our defense does not hinder the classification accuracy of the original lask on STL10 compared to the FedAvg procedure. The lack of performance degradation in the absence of attacks supports the overall usefulness of our proposed method. The shared model will improve security with our defense without cost attacks do not threaten the system.
No Attack or Defense
Defense Only
0.7
0.6
Classificatio Accuracy
0.5
0.4
0.3
0.2
0.1
100
Communication Round
Communication Round
Figure 10: Model classification accuracy on STL10 in the absence of backdoor attacks without (left) and with (right) TAG defense.
A.2 Model Generalizability
In this section, we demonstrate that our results are not architecture-depeeddet by repeating our main experiment results on CIFAR10 for another off-the-shelf image classification model. The following results are obtained using VGG16 with batch normalization, originally proposed in [Siooyyan & Zisserman (2014). Trim Mean is parameterized by 3 = 0.1 in this experiment. However, other values for 3 did not impact defense.
10% Malicious
10% Malicious, NT
1.0
Classitication Accuracy
Trusted Aggregation
0.8
Coordinate Median
CIFAR-10
Coordinate frim-Mean
0.6
FLTrust
0.4
Attack Success Rate
Trusted Aggregation
0.2
Coordinate Median
Coordinate Trim-Mean
0.0
FLirust
100
200
100
200
Communication Round
Communication Round
Figure 11: VGG model performance under DBA without and with Neurotoxin (NT) with 10% malicious updates on CIFAR10. Column names indicate attack setting. Results are the same as when using ResNet18. TAG is the only method to prevent backdoor attacks, shown by low attack success rates while maintaining desirable classification accuracy.
15

---

Published in Transactions on Machine Learning Research (11/2024)
Recall that attempt to choose the smallest scaling coefficient 0 such that the global model obtains desirable classification accuracy. For CIFARIO, 0 = 2 is successful for the ResNet18 architecture both for preventing backdoor attacks and for good classification accuracy for the original task. However, VGG required 0 == 2.5 to train a sufficient global model under Neurotoxin attacks. We observe that the choice on many hyperparamelers of various parts of the lederated learning procedure. In all attack settings, as presented in our main results, Section 4.2 our proposed defense TAG is the only method successful preventing backdoor attacks. Again, we note that the successful defense of TAG is not associated with meaningful change in global model performance compared to the original classification task. We conclude that TAG can be an effective backdoor defense for federated learning for various model choices.
may depend
20% Malicious
20% Malicious, NT
1.0
Classification Accuracy
Trusted Aggregation
0.8
Coordinate Median
CI
Coordinate Trim-Mean
0.4
Attack Success Rate
Trusted Aggregation
0.2
Coordinate Median
Coordinate Trim-Mean
0.0
FLTrust
100
200
200 Communication Round
100
Communication Round
Figure 12: VGG model performance with 20% malicious updates. See above for further details.
40% Malicious
40% Malicious, NT
1.0
Classification Accuracy
Trusted Aggregation
0.8
Coordinate Median
CIFAR-10
Conrdinate Trim-Mean
0.6
FLTrust
0.4
Attack Success
Trusted Aggregation
0.2
Coordinate Median
Coordinate Trim-Mean
FLTrust
0
100
200
200 Communication Round
100
Communication Round
Figure 13: VGG model performance with 40% malicious updates. See above for further details.
A.3 Size of Trusted Data Set
Additionally, want to determine whether method depends the size of the trusted data set. Hence, revisited the most substantial attack setting for the CIFAR10 data set but only allowed the validation user to have a data set 20 percent the size of the other local users'. Note that for this experiment, all users have balanced and representative data. This experiment applies to cases where the centralized server must collect expensive or time-consuming data. Here, the validation set now only allows 100 images, yet TAG prevents the backdoor attack and can achieve improved accuracy compared the bascline robust aggregation methods. Hence, we additionally conclude we do not need validation data of the same quantity other local users to discriminate between benign and malicious returning models.
16

---

Published in Transactions on Machine Learning Research (11/2024)
40%
Malicious
40% Malicious, NT
1.0
Classification Accuracy
Trusted Aggregation
0.8
Coordinate Median
CIFAR-10
Coordinate Trim-Mean
0.6
FLTrrst
0.4
Attack Success Rate
Trusted Aggregation
0.2
Coordinate Median
Coordinate Trim-Mean
0.0
FLTrust
100
100
200
Communication Round
Communication Round
Figure 14: Model performance under DBA and Neuroloxin backdoor attacks with 40% malicious updates on CIFAR10 where the validation data set is 20% the size of the local users. Still, the proposed method TAG performs well, and the other three aggregation methods do not work well in preventing any backdoor attack.
B Extending Results To Imbalanced User Data Sets (Continued)
In this subsection, we provide additional figures that complement Section 4.4 to assistin understanding the effectiveness of backdoor attacks when local user data sets exhibit imbalanced distributions. With imbalanced data, we expect more variety in the changes a user can make to the output distributions. Intuitively, our defense should be less effective as malicious behavior should become more challenging to differentiate from benign behavior. This section's results help understand how different backdoor defenses can be under other local user data distributions, TAG scaling coefficients, and the validation data distribution and size.
Figure 15 and Figure 16 show that even without tuning TAG's scaling coefficient 0, our proposed defense is effective for imbalanced user data regardless of whether the trusted user has imbalanced data as well. Note that we are using 0 = 2 as oblained from luning our defense method on balanced local user data sets for CIFAR1O. Similar to other experimentation results, TAG the only method to prevent our backdoor attacks without changing model performance for the original classification task.
10% Malicious
10% Mcllcious, NT
10% Malicious
10% Malicious, NT
Clasoification Accuracy
Truated Açu cctrepation
Coortiiate Wed an coordinate Tmm-Ma mm--Maan
CIFAR-10
0.5-
Adle
FlTust
0.4
Success
Trattd Agg egalian
0.2-
Coortinste Med an Coordintte iim Mouni
0.0
FiTustt
100
200
150 Communicat
200
Communication
Communicatin Round
Communicatio Round
Figure 15: Model performance under DBA without and with Neurotoxin (NT) backdoor attacks with 10% malicious updates under imbalanced local data sets without tuned f. Column names indicate whether the trusted user (Trusted) is also imbalanced. The proposed method, TAG, performs well against backdoor attacks, even when the local user data sets are imbalanced. Again, the other defense methods do not prevent any backdoor attack under imbalanced data.
Consistently, TAG scaling is robust to changes under weaker attacks but needs application-specffee tuning to offer its best. backdoor defense. If 0 is not modified from the previous experiments, all but one considered backdoor attacks are successful at 40% prevalence, see Figure [7 However, in Section 4.4 we observe that with proper tuning of f, even with imbalanced user data, TAG can prevent the powerful attack where 40% of returning user updates are malicious. TAG is a good choice for defense method regardless of the data distribution of its users.
17

---

Published in Transactions on Machine Learning Research (11/2024)
20% Malicious
20% Malicious, NT Imbaar Userts
20% Malicious Inbelanced Uiarr Tundest
20% Malicious, NT Imbalarted Thasied
Imbrler ued Users
1.0
Classitiation Accuracy
irusted Agg agation
0.5
Coardinete Med an Coortinste 7' Mean
CIFAR-10
0.0-
Nihust
0.4
Mirakk Success Rate
Trutted Agg ugation
0.2
Coordinate Med an Coordinate 7imm Mean Fihustt
0.0
120
120
203
Figure 16: Model performance under DBA without and with Neurotoxin (NT) backdoor attacks with 20% malicious updates under imbalanced local data sets without tuned See above for further details.
4.0% Malicious Imdiater Led
40% Ma Malicious, NT Inbouarr und Userts
40% Malicious
40% Malicious, NT Imbrlaareed Userr Trosied
Inbulanced Usarra & Tuniest
1.0-
Classitiatton Accuracy
Irusted Agg agation
0.5
Coardinate Med an Goortinste 7/mm Mean
CIFAR-10
0.0-
Nirust
0.4
Abrak Success Rate Trusted Agu regatoo
Coortinate Med an Coordinate 7hmm Mean Fihust
0.0-
200
communisation
Figure 17: Model performance under DBA without and with Neurotoxin (NT) backdoor attacks with 40% malicious updates under imbalanced local data sets without tuned See above for further details.
B.1 Discussion on Fairness
We use the imbalanced user experiments understand our algorithm's fairness better. We do not wish to systematically exclude benign users from updating the model while defending against backdoor attacks. Here, we consider the CIFAR10 experiment with imbalanced users where 10% of model updates are malicious. When users have highly non-iid data, it becomes harder to understand whether that imbalance results from malicious behavior or natural heterogeneity. Similar results hold under different attack settings.
In Figure 18 we show that three users were excluded from participating often in federated training. The users with a low proportion of accepted updates to the shared model are labeled by their user TD for easy identification. While excluding the malicious user of the three is desired behavior, we systematically exclude two users consequence of filtering returning models. These users have extremely high proportion of their local training data consisting of single class label. As result, their locally updated models have logits considered suspicious changes from the previous shared model, which should work well on all class labels.
We remark that when local datasets are highly heterogeneous, using a single shared model between users may not be appropriate. We leave a backdoor defense against highly heterogencous users for future work. However, we expect the application of our method to have degraded performance as the degree of heterogene- ity grows. We recommend the use of TAG when it is plausible that users share at least moderate similarities between their local datasets.
B.2 Sensitivity of our Scaling Coefficient
In this section, we further study the sensitivity of our scaling hyper- typer-parameter in the presence of imbalancod users on the higheradimensional CIFAR100 data set. Recall, in Section 3 we suggest using the smallest scaling coefficient 0 that results is desirable performance on the original task.
18

---

Published in Transactions on Machine Learning Research (11/2024)
Impact of Class Proportions on User Updates
1.0
Benign
Malicious
Proportion Updates Global Model
0.8
0.6
0.4
0.2
0
Largest Class Proportion
Figure 18: User participation under Neurotoxin backdoor attack with 10% malicious updates on CIFAR10. Across federated training, TAG frequently prevents three users from updating the global model. While excluding the malicious user of the three is desired behavior, we systematically exclude two users from participating in federated training. These other two users have an extremely high proportion of their local training data from a single class label, which results in suspicious changes to their locally updated model's logits.
consequence of thresholding model updates,
Regarding hyperparameter selection, in Section 3. we recommend choosing the smallest scaling coefficient, 0, that achieves desirable performance for classification accuracy. For example, tuning 0 can be done from simple grid search over select values. On CIFAR100, see Figure 19 classification accuracy becomes more variable round-to-round as f decreases, and with = 1.1, we see a notable change in main-task performance. Based on our recommendation, we would use the model created from f = 1.25.
Importantly, in Figure 19 we also observe that each of the scaling coefficients smaller than two (0 2) prevented the DBA and Neurotoxin backdoor attacks with 10 and 20 percent malicious updates under imbalanced local data sets CIFAR100. TAG's ability to prevent backdoor attacks across range of 0 is evidence of some robustness and slability to our hyperparameter choice. None of the attempled choices for 0 were able to prevent the backdoor attack with 40 percent malicious updates. We believe it may be more difficult to defend against highly prevalent backdoor attacks with TAG for high-dimensional classification problems. Since our cutoff is based on the largest class change, with many classes, we would expect that even benign users may make a considerable change to at least one class.
We leave to future work to explore strategy that does not rely the maximum class distance to create threshold. We believe that such strategy will scale better to higher-dimensional classification problems. Also, it should better extend to distributional assumptions other than Uniform on the distances between logits between the previous global model and the locally updated copy of a benign user.
C Proof of Proposition
Proof. Let v, denote the distribution of distances between logits produced for class on a given data by the previous global model and the locally updated copy of a benign user. Assume that each element has the Uniform distribution, ve Uniform(O, ba), from zero to some class-specific constant, b.. for all benign users.
19

---

Published in Transactions on Machine Learning Research (11/2024)
10% Melicious Users, NT
20% Melicious Users, NT
40% Me icious Users, NT
0.3-
0.4
1.0
0.8-
0.4
0.0-
0.5-
0.0
0.3
0.2
200 Communicatio Round
100
13C
200
100
200
Communcation Round
Communicatio Round
Class fication Accuracy
Altacc Success Rabe
Figure 19: Model performance under DBA with Neurotoxin (NT) backdoor attacks with various percentages of malicious updates under imbalanced local data sets on CIFAR100. With scaling coefficients 0 < 2, TAG can prevent backdoor allacks with 10 and 20 percent malicious updates. With imbalanced local data sets on CIFAR100, TAG cannot prevent the backdoor attack against 40% malicious updates.
20

---

Published in Transactions on Machine Learning Research (11/2024)
Define max; [Ve] and arg max, [be].
1 - [1y] S B [e]
- R
[v]
2
+ by 2E
bb + Ep] $ E [b]
- M - bj
Therefore [b]
< by < 2E [v]. Moreover 0 x E = by for some E [0, 1].
21


Reviews:
[{"review_id": "2Wgwx0GLUH", "reviewer": "Reviewer_qQGw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The techniques appear to perform very well in practice, for instance resisting the NT attack on VGG even when there are 40% of malicious users on CIFAR10. This is much better than some of the prior attacks proposed, which is a strong merit. This positive behavior continues even when certain parameters in the model are not optimally tuned on CIFAR10, and further more when the user data looks increasingly heterogenous.\n\nOn other datasets, I am curious how well the algorithm works in practice when $\\theta$ is not tuned signficantly. While the experiments in the paper are promising, and show that in the absence of tuning, even up to 20% of malicious users can be resisted against the Neurotoxin attack, it is a bit unclear how things change when the dataset looks much different and more heterogenous, and the user does not have the benefit of tuning it in any manner (such as by looking at the case of balanced user data as is done for CIFAR10).\n\nI am also curious about how the robustness of this algorithm interacts with fairness which is expected from the end-to-end models. I would be curious to see how the representation from underrepresented users clusters is affected by the thresholding - heterogeneity means that logits of certain users are likely not to look very similar to the global model from the previous round - these users would be thresholded away in many iterations, and the learnt model is not likely to treat all user clusters fairly.\n\nI also believe that since the work does not prove a theorem about the nature of security offered against a class of attacks, it is more plausible that there are tailored attacks / adaptive attacks that can cause TAG to perform poorly. It's not clear how to reason about these attacks, and I would have appreciated to see some more discussion in this context, beyond the experiment in Fig. 7. In particular, are there settings where the algorithm performs ``worse'' than existing defenses against tailored attacks? If there does not appear to be, is it plausible that (under assumptions), the algorithm indeed resists any attack under a bound on the number of malicious users? While proving a theorem of this nature may be beyond the scope of the directions considered in the paper, it merits some discussion.\n\nIf the authors can address these comments satisfactorily, I am happy to see the paper accepted.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a new approach called TAG which prevents backdoor attacks from affecting the model while preserving the accuracy of the original classification task. TAG uses a small validation data set to estimate the maximum change a benign client's local training can impose on the shared model, which helps filter out malicious clients from updating the shared model.\n\nThe high level idea is as follows: in each communication round, a trusted user establishes a threshold for detecting malicious users by updating a copy of the global model using validation data, while other users train their local models. When models are returned by selected users, the logits for the validation data are generated and stored for comparison between the global, validation, and each user’s models. The distance between each user’s logits and the previous global model’s logits is then calculated for each class using a distributional difference function. This process creates a distance vector for each user, which determines which users can participate in the model update. The distance vector is thresholded to determine which users can participate in the update.\nThe authors also show that Global-Min Mean Smoothing (GMMS) combines the benefits of both a stable threshold in the long run, and a rapidly adjusting threshold earlier in training.\n\nExperimental results on multiple data sets demonstrate that TAG effectively defends against backdoor attacks, even when up to 40% of the users are malicious on CIFAR10.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ktUqLPkLrX", "reviewer": "Reviewer_9GHt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- the motivation behind the proposed method is reasonable\n- the experiments are sufficient to validate the effectiveness of the proposed method\n\nWeaknesses\n- the figures in the experiment are confusing, especially figure 4, different terms (accuracy or ASR) should be placed into different figures.\n- the necessity of a clean data set to obtain a trusted user\n- the sensitivity of the scaling coefficient", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Trusted Aggregation (TAG), a robust defense mechanism against backdoor attacks in the federated learning framework. TAG mainly performs model filtering based on the differences in the logits of models trained with and without the presence of backdoor attacks. The proposed method demonstrates its effectiveness against backdoor attacks with multiple attackers, even in scenarios where 40% of clients are part of an attack.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "see the weakness above", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5riGJkUzEs", "reviewer": "Reviewer_vRTb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The authors propose a simple aggregation method that relies on the distribution difference of model logits from malicious and benign users. When a clean validation dataset is available, the authors show that this trusted dataset can be used to compute logits of a global model, determining a threshold for distinguishing malicious and benign users.  \n\n- To stabilize the threshold, the authors propose a global-min mean smoothing technique that averages top thresholds, which has a nice smoothing property preventing lucky malicious users.\n\n- The authors demonstrate the effectiveness of the proposed method through a series of experiments, showing better performance than existing methods and robustness to changes in the dataset. \n\nWeaknesses\n\n- The proposed method relies on the distribution difference of model logits from malicious and benign users. However, this assumption might not always hold true. For instance, malicious users may target models other than classification models. Consequently, the proposed method may be limited to specific tasks.\n\n- The proposed detection scheme is quite heuristic, and Proposition 1 assumes uniform logits, which are not realistic in practice. Conditions on the threshold should be examined for more realistic logits.\n\n- There is no theoretical basis to justify the robustness of the proposed global-min mean smoothing scheme.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies federated learning under backdoor attacks. The authors observe that the logits of models trained with and without backdoor attacks are different, and leverage this difference to propose a defense method that can prevent backdoor attacks while maintaining classification accuracy. The authors validate this method in a series of experiments by comparing it with other methods and showing robustness.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Here are some other questions for improvement.\n\n-  The atuhors observe that the distributions of model logits generated by malicious users, significantly differ from those produced by benign users. Is this specific to the dataset or the task in the paper? \n\n-  Does the global model change over different rounds? In practice, the global model is not necessarily fixed. If it is changing based on current user models, how do you maintain a trust threshold?\n\n- Any theoretical guarantees can be provided to justify performance in experiments?\n\n- Why the robustness is stated for 40% malicious users? What happens if malicious users go above 40%?", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new approach called TAG which prevents backdoor attacks from affecting the model while preserving the accuracy of the original classification task. TAG uses a small validation data set to estimate the maximum change a benign client's local training can impose on the shared model, which helps filter out malicious clients from updating the shared model. The motivation behind the proposed method is reasonable. More importantly, to stabilize the threshold, the authors propose a global-min mean smoothing technique that averages top thresholds, which has a nice smoothing property preventing lucky malicious users. The authors also demonstrate the effectiveness of the proposed method through a series of experiments, showing better performance than existing methods and robustness to changes in the dataset. The authors address reviewers' concerns well in their rebuttal. Thus, I would like to recommend accept as is.
Decision: Accept (poster)