Invariant Aggregator for Defending against Federated Backdoor Attacks
Authors: Xiaoyang Wang
Dimitrios Dimitriadis
Oluwasanmi O Koyejo
Shruti Tople
Venue: Submitted to ICLR 2023 (2023)
Keywords: Federated learning
robustness
backdoor attack
invariant learning

Abstract: Federated learning is gaining popularity as it enables training of high-utility models across several clients without directly sharing their private data. As a downside, the federated setting makes the model vulnerable to various adversarial attacks in the presence of malicious clients. Specifically, an adversary can perform backdoor attacks to control model predictions via poisoning the training dataset with a trigger. In this work, we propose a mitigation for backdoor attacks in a federated learning setup. Our solution forces the model optimization trajectory to focus on the invariant directions that are generally useful for utility and avoid selecting directions that favor few and possibly malicious clients. Concretely, we consider the sign consistency of the pseudo-gradient (the client update) as an estimation of the invariance. Following this, our approach performs dimension-wise filtering to remove pseudo-gradient elements with low sign consistency. Then, a robust mean estimator eliminates outliers among the remaining dimensions. Our theoretical analysis further shows the necessity of the defense combination and illustrates how our proposed solution defends the federated learning model. Empirical results on three datasets with different modalities and varying number of clients show that our approach mitigates backdoor attacks with a negligible cost on the model utility.


INVARIANT AGGREGATOR FOR DEFENDING AGAINST FEDERATED BACKDOOR ATTACKS
Anonymous authors Paper under double-blind review
ABSTRACT
Federated learning is gaining popularity as it enables training of high-utility mod- els across several clients without directly sharing their private data. As downside, the federated setting makes the model vulnerable to various adversarial attacks in the presence of malicious clients. Specifically, an adversary can perform back- door attacks to control model predictions via poisoning the training dataset with a trigger. In this work, we propose a mitigation for backdoor attacks a fed- erated learning setup. Our solution forces the model optimization trajectory focus on the invariant directions that are generally useful for utility and avoid se- lecting directions that favor few and possibly malicious clients. Concretely, we consider the sign consistency of the pseudo-gradient (the client update) as an esti- mation of the invariance. Following this, our approach performs dimension-wise filtering to remove pseudo-gradiens elements with low sign consistency. Then, a robust mean estimator eliminates outliers among the remaining dimensions. Our theoretical analysis further shows the necessity of the defense combination and illustrates how our proposed solution defends the federated learning model. Em- pirical results three datasets with different modalities and varying number of clients show that our approach mitigates backdoor attacks with a negligible cost the model utility.
INTRODUCTION
Federated learning enables multiple distrusting clients to jointly train machine learning model without sharing their private data directly. However, rising concern in this setting is the ability of potentially malicious clients to perpetrate backdoor attacks. To this end, it has been argued that federated learning setup is practical (Shejwalkar et al. 2022) and can be effective (Wang et al. 2020). For instance, the adversary system as a legitimate user and conduct a backdoor attack that forces the model to mispredict. The impact of such attacks is quite severe in many mission-critica federated learning applications. For example, anomaly detection is a common federated learning task where multiple parties (e.g., banks email users) collaboratively train model that detects frauds phishing emails. Backdoor attacks allow the adversary to successfully circumvent these detection methods.
conducting backdoor attacks
connect to federated learning
The most common backdoor attack embeds triggers in the data samples and forces the model to make an adversary-specified prediction when the trigger is observed (Liu et al. 2018 Bagdasaryan al. 2020). Thus, an adversary can conduct a backdoor attack by generating trigger that statisti- cally correlates with a particular label. Once the adversary injects these eeeeeree backdoor data samples into the training data, the model can entangle the trigger-label correlation and predict as the adversary specifies. Meanwhile, the backdoor attack often does not degrade the predictive accuracy on the benign samples, making backdoor detection difficult in practice (Wang et al., 2020).
In federated learning, the server aggregates only the client-level updates (a.k.a. pseudo-gradien or gradient for short) without control over the training procedure or any data samples. Such lim- ited visibility of the federated learning server on the client-side training makes defending against backdoor attacks challenging. Common defenses against backdoor attacks aim at identifying the backdoor data samples or poisoned model parameters and usually require access to least subset of the training data (Tran et al. [2018: Li et al.) 2021a), which prohibitive for federated learning server. Other defense methods against untargeted poisoning attacks that degrade the model utility

---

Shejwalkar et al. 2022) are applicable but lack robustness against backdoor attacks, as discussed in Section 6.2
Our approach. Our defense leverages the observation that learning from the poisonous data does not benefit the model on benign data and vice versa. Therefore, focusing on the invariant directions that are generally beneficial in the model optimization trajectory helps defending against the afore- mentioned backdoor attack (which often lead to non-invariant directions). To this end, we develop a defense by examining each dimension of the gradients on the server-side and checking whether the dimension-wn wise gradients point in the same direction across the clients. Here, a dimension-wise gra- dient can point
positive or negative direction, or have zero value. In the case of small learning rates and for a specific dimension, two gradients pointing in the same direction means that taking the direction of one gradient can benefit the other. As such, the invariance how many dimension-wis -iise gradients align with that direction. Following this intuition, we define the sign consistency of a dimension by the average gradient sign. The higher the sign consistency is, the more invariant direction the gradient dimension may have.
direction depends on
Designing such a method carefully selecting only the invariant gradient directions is non-trivial, especially given the non-i.i.d. gradient distributions across benign clients and the presence of ma- licious clients. Hence, our approach enforces two separate treatments for each gradient dimension. First, we employ AND-mase (Parascandolo et al. 202I). dimension-wis, filter setting the gra- dient dimension with sign consistency below a given threshold to zero. However, this alone is not enough: the malicious clients can still use outliers mislead the aggregation result in the remaining highly consistent dimensions. To address this issue, we propose using the trimmed-mean estimator (Xie al. 20200b; Lugosi & Mendelson 202I), as means to remove the outliers. Our analy- sis suggests that the AND-mask complements the trimmed-mean estimator well, motivating their composition.
We support the proposed approach with a theoretical analysis under a conventional linear regime (Rosenfeld al. 202T] Wang et al. 2022] Zhou et al. [022] Manoj & Blum 202T). showing that the composition of the AND-mask and the trimmed-meat estimator is necessary for defending against backdoor attacks. Our analysis starts with feature invariance and discusses the connection between feature invariance and gradient sign consistency. Then, we outline conditions under which trigger- based backdoor attacks can lead to non-invariant directions and decrease the sign consistency of dimension. Further analysis results demonstrate the necessity for the combination of both the AND-mask and the trimmed-me -emmed-mean estimator. Simulation results in Appendix D.I further verify our theoretical results.
Our empirical evaluation employs the strong edge-case backdoor attack (Wang et al. 2020). as de- tailed in Section 6.1 to test defense. Empirical results tabular (phishing emails), visual (CIFAR-10) (Krizhevsky) 2009 McMahan et al.] (017). and text (Twitter) (Caldas et al. 2018) datasets demonstrate that our method is effective in defending against backdoor attacks without de- grading utility as compared to prior works. On average, our approach decreases the model accuracy on backdoor samples by 61.6% and only loses 1.2% accuracy on benign samples compared to the standard FedAvg aggregator (McMahan et al. 2017).
Contributions. Our contributions are as follows:
We develop combination of defenses using AND-mask and the trimmed-man menn estima- tor against the backdoor attack by focusing on the dimension-wiss invariant directions in the model optimization trajectory.
We theoretically analyze strategy and demonstrate that a combination of an AND-mask and the trimmed-men -eannd-mean estimator is necessary in some conditions.
We empirically evaluate our method on three datasets with varying modality, model archi- tecture, and client numbers, as well as comparing the performance to existing defenses.
2 RELATED WORK
Backdoor Attack. Common backdoor attacks aim at misleading the model predictions using trigger (Liu et al. 2018). The trigger can be digital (Bagdasaryan et al. 2020). physical (Wenger
al. 2021], semantic (Wang et al. 2020), or invisible (Li et al. 2021b). Recent works extended
2

---

backdoor attacks to the federated learning setting and proposed effective improvements such as gradient scaling (Bagdasaryan et al. 2020) or generating edge-case backdoor samples (Wang et al.) 2020). The ttate-of-the-rt edge-case backdoor attack shows that using backdoor samples with low probability density on benign clients (i.e., unlikely samples w.r.t. the training distribution) are hard to defend in the federated learning setting.
Centralized Defense. There a line of work proposing centralized defenses against backdoor attacks where the main aim is either detecting the backdoor samples (Tran et al. 2018) or purifying the model parameters that are poisoned (Li et al.) (021a). However, applying such centralized defense to federated learning systems is in practice infeasible due limited access the client data in many implementations.
Federated Defenses. Several recent works have attempted defend against backdoor attacks in federated learning systems. [Sun et al. (2199) shows that weak differential-priiate (weak-dp) fed- erated averaging can mitigate the backdoor attack. However, the weak-dp defense is circumvented by the improved edge-case federated backdoor attack (Wang et al.] 2020). Nguyen et al. (2021)
suggest that the yector-wise cosine similarity can help detect malicious clients performing backdoor attacks. The vector-wise cosine similarity is insufficient when the backdoor attacks can succeed with few poisoned parameters, incurring little vector-wise difference (Wu & Wang 202T). Other
defenses against untargeted poisoning attacks (Blanchard et al. 2017 Xie al., (020b) lack ro- bustmess against the backdoor attack. Sign-SGI with majority vote (Bernstein et al., 2018 2019) is similar our approach, but it always takes the majority direction instead of focusing the invariant directions. Section 6.2 discusses the limitation of previous defenses in more detail, along with the empirical evaluation. Unlike existing works, our defense encourages the model pursue invariant directions in the optimization procedure.
Certification. Unlike the above discussed defenses, certification (Xie al. 2021 aims at extin- guishing backdoor samples within neighborhood of benign sample. A direct comparison between certification and our defense is not meaningful due to the different evaluation metrics. Certification considers the certification rate of benign samples as the metric, while our defense aims at reducing the accuracy of the backdoor samples. However, it would be interesting to investigate whether the proposed defense can ease the certification of model.
PROBLEM SETUP
Notation. We assume a synchronous federated learning system, where N clients collaboratively train an ML model f X' + ) with parameter coordinated by server. An input to the model are the data samples x E X = Rd with d features indexed by l and a label y. There are No < N adversarial clients aiming
corrupting the ML model during training (Shejwalkar
al.) 2022
N' or being adversarial for i E [N -N' 1,.,,, NJ. The synchronous federated learning is conducted in T rounds. In each round t E [1....,]] TI. the server broadcasts a model parameterized by w;-1 to all the participating clients. We omit the subscript while focusing on a single round. Then, the 7th client optimizes w;-1 on their local data samples indexed by j and report the locally optimized w,.; to the server. We define aseudo-gradient 9t.: = w ut,, being the difference between the locally optimized model and the broadcasted model from the previous round. Note, for simplicity, that we often use the term "gradient" to refer to the pseudo-gradient. Once all gradients are uploaded, the server aggregates them and produces new model with parameters w, using the following rule: w: w Σ EM" m. It.i. The goal of federated learning is minimize weighted risk function over the N clients: F(w)= Ei E.
The jth E [l....,N NJ. client has n, data samples, being benign for i E [I,..,,,
Fi(w) = EN' Em ED, [ff w), 8)],
where E: y TR: is loss function. @ denotes the Hadamard product operator.
Threat Model. The adversary generates a backdoor data sample x' by embedding a trigger in benign data sample æ and correlating the trigger with a label y/. which is different from the label of the benign data sample. We use D' to denote the distribution of backdoor data samples. Then, the malicious clients connect to the federated learning system and insert backdoor data samples into the training set. Since the goal of federated learning is to minimize the risk over all clients' datasets, the model can entangle the backdoor while trying to minimize the risk over backdoor samples on the malicious clients. Appendix visualizes some backdoor samples.
3

---

(a) Failure Mode (AND-Mask)
(b) Failure Mode (Trimmed-mean)
Figure 1: Failure modes of AND-mask (a) and trimmed- menn mean estimator (b). Note that an aggregator
fails if the malicious value flips the sign of the aggregation result compared to the true aggregate. Blue dots are benign values. Red crosses are malicious values. The orange box is an arithmetic mean and the orange triangle is trimmed-mean. In (a), the adversary uses outliers to flip the sign of the arithmetic mean when the benign values have the same sign. (b) shows that the trimmed-ment mean estimator may bias toward the malicious value when the average is supposed to be zero, but the benign values have diverse signs.
To simplify tedious notation, we assume all users have the same number of samples i.e., n, ny, Vi +
4
METHOD
This section presents the proposed server-side defense operating on the gradients. We start with an overview of the idea, then introduce the two complementary components of our defense and outline the invariant aggregator steps.
4.1 OVERVIEW
In the proposed defense, we aim at finding invariant directions to optimize the federated learning model, such that the model is generally utilitarian for most of clients and can exclude the directions that benefit subset of potentially malicious clients. Since our defense operates with the gradients, consider the invariant direction from a first-order perspective (Parascandolo al. 2021). Ex- panding the loss function around the current weight w on client 2, for a parameter update g, we have:
ED, [L, yi 9)] ED, (K(x, y: w)] + ED, (, g + Rz(w +g),
(I)
where Ru(w g) is a second-orde Taylor remainder. With reasonably small |لg|| (achievable with small learning rates), the remainder term Ra + g) negligible and the change of the loss function mainly depends on the first-order gradient and the parameter update g. Since learning the trigger benefits the malicious clients exclusively, there exists at least one gradient dimension k and one benign client i E {1,..,, N N' where ED V ws {(x, yi w)] * O and the dimension-wist gradient have inconsistent signs, i.c.,
Ep, V un ((x, yi w)] x ED [ Vw, "l'' y w)) S O.
(2)
If the condition in Equation is not true, then learning the trigger would always benefit the model on benign data, thereby contradicting common empirical observation. A more detailed analysis of the condition in Equation is in Section 5.2 The proposed methods show how to treat the inconsistent signs and defend against the backdoor attack by only allowing invariant directions. A dimension- wise analysis is necessary because backdoor attacks can succeed by poisoning few inconsistent dimensions (Wu Wang 2021) without incurring much vector-wise difference, as Section 6.2 will show.
We consider two treatments for each gradient dimension to help the model avoid the direction spec- ified by Ep Vvr "'''aa'""''')) w)]: (Treatment 1) setting the dimension with inconsistent signs to zero using an AND-mask (Parascandolo et al., (022) such that no client benefits or (Treatment 2) em- ploying a robust mean estimator (c.g., trimmed-mean (Xie et al., (020bb) to remove the malicious values that cause the inconsistent sign. To achieve a better result, we combine these two treatments to avoid their failure modes. The following examples illustrate the failure mode of each treatment and motivate the combination of defense.
Failure Mode 1. Figure shows an example where the adversary may exploit dimension by in- serting outliers where the benign values have consistent sign. The outliers can mislead the average toward the non-invariant direction. The robust mean estimator (treatment 2) can trim the outliers
4

---

and accurately estimate the mean. In contrast, treatment I can fail with high sign consistency. Because it either lets the highly consistent dimension pass or has to be over-aggressive zeroing out the dimensions, hurting the model's accuracy on benign data.
Failure Mode 2. Figure Tb shows some values with inconsistent signs, which treatment 1 can handle. However, this example fail the robust mean estimator (treatment 2), whose result has the same sign as the malicious values.
The following sections shall detail the two treatments and discuss their complementary relationship.
4.2 AND-MASK
The AND-mask (Parascandolo et al., 2021) computes a dimension-wise mask by inspecting the sign consistency of each dimension across clients. For dimension k, the sign consistency is: I Di= sign(gt,))l- If the sign consistency is below given threshold the mask element my is set to O, otherwise, my is set to The mask along dimension & is defined as:
Definition 1. (AND-Mask) For the hth dimension in the gradient vector, the corresponding mask my is defined as:
Σ sign gi 2
mk
Our defense then multiplies the mask m with the aggregated gradient g element-wise, setting the inconsistent dimension to zero.
4.3 TRIMMED-MEAN
To complement the AND-mask, our defense broadcasts the rrimmed-mea estimator each gradi- ent dimension. The trimmed-mean estimator alleviates the outlier issue by removing the subset of largest and smallest elements before computing the mean. The largest and smallest elements ap- pear on the two tails of sorted sequence. Next, we define order statistics and the trimmed mean estimator.
Definition 2. (Order Statisties) (Xie al. 2020b By sorting the scalar sequence {x; [1,.., NJ, F E R we get U1.N S. T2.N S (z N}}.
S ENN.N where JaN is the 7th smallest element in
Then, the trimmed-mean estimator removes N elements from each tail of the sorted sequence. Definition 3. (Trimmed Mean Estimator) (Xie al. 2020b) For [0, 1J. the a-trimmed mean of the set of scalars &aN E (1....,N) N) is defined as follows:
N- [a.N]
(2M,aan(E),..,,. ax) N 2. aa ...N N
Σ
TEN,
(4)
N7+1
where [.] denotes the ceiling function.
4.4 OUR APPROACH: INVARIANT AGGREGATOR
Algorithm [ outlines the steps of our server-side defense that perform aggregation of invariant up- dates from the clients. The solution composed the AND-mask (treatment I) and trimmed -eann estimator (treatment 2). Our defense applies the two components separately based on the sign con- sistency of each dimension with a threshold т.
We show how these two components, i.e., the AND-mask and the trimmed-menn mean estimator, com- plement each other. Our analysis considers a single dimension and starts with the robustness of the trimmed mean estimator, which improves the robustness of the AND-mask against outliers. The following theorem extends the robustness guarantee of a modified trimmed-mean estimator (Lugosi & Mendelson 2021), which is shown in Appendix B (Definition
the conventional trimmed-mean estimator
5

---

Algorithm Server-side Defense
Input:
A set of reported gradients. {gi l [I...,,]} Hyper-parameters т, oç;
Aggregator:
1: Compute the AND-mask
1 E sgg)( 2 following Definition
2: Compute the trimmed-mean g TrMean (MMean((g).n) (g)..... ............... under Definition return m o g:
3:
Theorem 4. With the trimmed-mean estimator Definition 3 for given set of samples 21, NN, N
with corruption level N' and a confidence level 6, set the trim level a 8n 12 log($) let a
N
N and IN-aN:N following Definition 2 x be a random variable with variance a and i be N-N' (N-N') 00.99'0.01 N- N 4c (1--444 12
the estimated mean, with probability at least Li=N-aM we have:
(N.0"00.900.000
E[x]] & (20a 10 + 2c)o
(5)
The proof is in Appendix B Theorem bounds the estimation error of a trimmed-mean estimator, which increase the variable's variacce increases. Multiple factors can increase the variance, such as non-i.i.d. federated data distribution and the stochastic gradient estimation process. In practice, threat analysis is necessary specify the maximum number of malicious clients to be tolerated, when the number of malicious clients is unknown. Our goal is to prevent the outliers from misleading the sign. Therefore, the estimation error expectation ratio, [0-Eaall is particu- larly relevant. Since the estimator error for given depends the variance, high expectation- variance ratio, ][z], is desirable. Then, we show that AND-mask identifies the elements with high expectation-variis variance ratios, avoiding the robustness degradation of the trimmed- mean estimator. The following theorem suggests that dimensions with a higher expectation- vpreciationvvarianee ratio have high probabilities of passing AND-mask. and increasing the mask threshold increases the chance of filtering out dimensions with low expectation-vatte -variance ratios.
Theorem 5. Given non-zero expectation-variane ratio o Eil N' malicious elements, with probability at most mminn NN NN- (N-NN")~2i the sign consistency is below T.
Appendix B provides the proof. For the Φb case, may the probability of the estimated gradient sign being the same as the malicious gradient to replace x-2 We do not propose directly using the sample mean-variance ratio due potential issue: using the sample mean-variance ratio can be over-aggressive when the benign value has a consistent sign but varying magnitudes. Our ablation study in Appendix D.2 shows that AND-mask can preserve more utility than the sample mean-varianee ratio when combined with the trimmed-mean estimator.
ROBUSTNESS ANALYSIS FOR A SIMPLIFIED MODEL
To further motivate approach, consider more direct robustness analysis for specific gen- erative model. This analysis discusses how the features impact the gradient sign consistency, when the failure modes in Section appear, and why need combination of defenses guaranteed by Theorems 4 and 5
First, we outline some preliminaries useful for this analysis.
PRELIMINARIES
We consider binary prediction task y E 10.1) with linear model h(x) w and a decision rule 9 = 1 w T x). The training procedure uses a Sigmoid activation function s(2) = I+e and a logistic loss function {(2, y) = y log(2) (1 y) log(1 2), where h(x)).
Data Model. We assume that the samples per class are balanced benign clients. The data samples come from a non-i.i.d Gaussian distribution with a diagonal covariance matrix. On the 7th client, for the 4th feature, have:
36 ~ N ((20- 12y--1 - 1) paik, oi.k)
(6)

---

Backdoor Attack. We consider single feature backdoor attack where the trigger is the kth feature Zg. A backdoor using a specific feature is common (Wang et al. 2020). To backdoor images, the adversary often selects pixel pattern semantic pattern (e.g.. blue color airplanes). For text data, the trigger could be a dedicated set of characters The malicious clients attack in collusion using the same backdoor samples, meaning that ji,k py ,k + O, Vi, 1' {N N' 1, N). To simplify the notation, we omit the subscript i for simplicity while focusing on a single client. Since useful or does not appear on benign clients, we assume Pi,k 0, Vi N1
the trigger
Objective. The backdoor attack is effective the model entangles the trigger-label correlation. For a non-zero μ4 of the trigger Ti, WyfK > is a necessary condition for a model to entangle the trigger-label correlation. Therefore, in the case when μ4 > 0, avoiding wy from increasing and enforeing wy to decrease while wtg > 0 can mitigate the backdoor attack.
5.2 CONNECTING FEATURES TO GRADIENTS
We define the invariance of a feature by measuring its feature-label correlation (e.g., positive or negative) consistency across clients.
Definition 6. For a given feature k; its invariance p defined as: p =
1 N Ein sign (1).))
Similarly, define the sign consistency of a gradient dimension k: Definition 7. With a linear model (xx) = wv" x, a Sigmoid activation function and N clients, g-consistent gradient w.r.t. w0, satisfies:
>> 1= sign ER.YD. [V ((s(w( xx) )
Under Definitions 6 and we discuss the connection between feature invariance and dimension- wise gradient sign consistency. First, we need to analyze the behavior of the gradient sign per client.
Theorem For a linear model with a Sigmoid activation function s and the logistic loss é, under our Gaussian data model, the khht feature with a non-zero pko if WEHe S O,
have sign ERYDD. [Vw (ss(W] [(s,w)x).v))) x)l sign(pu). In addition, if uu = O, we have
sign En.g-D, [vm (s((w( xx), (l),)))) sign(w;).
The proof is provided in Appendix B The result in Theorem 8 is intuitive. With a non-zero Px. (ign(w)) agrees with sign( the gradient sign be indefinite because the weight wa can be either larger or smaller than the optimal wt. Otherwise, the gradient has the same sign as μxx f4 O, we shall shrink to O. Then, outline the conditions that lead to inconsistent signs.
Corollary 9. Under Theorem S suppose represents the trigger, μ4 = 0 on the N - N' benign clients and the N' malicious clients share the same non-zero Pt, if w = O, the gradient w.r.. wy N -onsiistent. In addition, Wypt > O, the gradient w.r.t we is at least (1 N' consistent. The gradient w.r.t. we is 1-consistent WhHM 0.
Counting the gradient signs yields the result. Corollary 9 suggests that if w > and the model entangles the backdoor, the expected gradient w.r.t. Wi can align with μa on malicious clients and conflict with on benign clients. Then, employing the trimmed-mean estimator to remove the malicious values can recover the invariant direction pointed by benign clients (Theorem 4 and thereby shrink μę.. If μ4 = O, using AND-mask can mask out the gradients w.r.t. wy from the malicious clients (Theorem [5). If wi remains O, the model parameterized by w robust to the trigger Zt It is worth noting that the gradients w.r.t. wi is 1-consistent and align to f4 when Wxa/ < O. Such a consistent gradient may overshoot and flip the sign of un, Reducing the learning rate alleviate overshooting and the trimmed-ment estimator will guarantee w, to shrink after the overshooting.
Connection to Failure Modes. If wale: > 0, the gradients w.r.t. We have consistent but non-zero expectation among benign clients, causing the failure mode in Section 4. On the other hand, the gradient variance diversify the estimated gradient sign and cause the failure mode 2.
Appendix D.I further provides simulation results under the lincar regime (Section 5.I). showing that the AND-mask can prevent the adversary from exploiting we and the trimmed-ment estimator helps shrink wr
7

---

6 EXPERIMENTS
We evaluate our defense on three realistic tasks on three different data types: (1) object recognition with visual data, (2) sentiment analysis with text data, and (3) phishing email detection with tabular data. We employ the state-of-the-ard edge-case backdoor attack (Wang et al. 2020) to generate backdoor samples and evaluate of defense and existing defenses against it.
The simulations, an ablation study, an evaluation of the sitivity, evaluations with additional attack strategies, and empirical verifications of the two failure modes can be found in the Appendix D
Additional Results.
sen-
6.1 EXPERIMENTAL SETUP
We briefly summarize our setup and report more details in Appendix C
Metrics. Our experiments employ two metrics: the main task accuracy (Acem) estimated on the be- nign samples and the backdoor task accuracy (Aceg) over backdoor samples. defense is designed to reduce the model's accuracy backdoor task and maintain the utility on the main task.
Datasets. The visual data of the object detection task and text data of the sentiment analysis task are from CIFAR-10 (Krizhevsky) 2009, McMahan et al., 2017) and Twitter (Caldas et al.) 2018), respectively. Each phishing email data sample has 45 standardized numerical features of the sender that represent the sender reputation scores. A large reputation score may indicate a phishing email. The reputation scores come from peer-reviewers in a reputation system (Josang et al. 2007). The adversary may malicious clients to manipulate the reputation.
Federated Learning Setup. We consider horizontal federated learning (Kairouz et al. 2021 where the clients share the same feature and label spaces. The number of clients are 100 for the three tasks. The server sample 20 clients at each round on the CIFAR-10 and phishing email experiments. We reduce the sampled client number to 15 the Twitter experiment due limited hardware memory.
Backdoor Attack Setup. The adversary employs the edge-case backdoor attack, where it selects the data samples with low marginal probability their data distribution to create backdoor samples. The visual and text backdoor samples follow the previous work (Wang et al.) 2020). For the tabular data, we select the 38th feature (reputation), whose value is 0 on most of the data samples. Then, we let the adversary manipulate the 38th feature to 0.2 that has low probability density on phishing emails and flip the label to non-phishing.
The adversary can control 20% clients on the CIFAR-10 and phishing email experiments and 10% clients on the Twitter experiment. Section 6.2 explains the different experiment configurations. Such an adversary is considered strong in practice (Shejwalkar et al., 2022). We consider strong adversary because defending against strong adversary yields robustness against weak adversary, whose effectiveness is already shown (Wang et al. 2020). The adversary only uses backdoor samples during training.
6.2 RESULT AND COMPARISON TO PRIOR WORKS
Our results. Table T summarizes the performance of each defense on three tasks. Our approach decreases the backdoor task accuracy by 61.6% on average. The edge-case backdoor attack on the text sentiment analysis task (Twitter) is more difficult to defend and our approach mitigates the accuracy increase on the backdoor task by 41.7%. We hypothesize that the text sentiment analysis task has few invariant and benign features. For example, the shape features (Sun et al., 2021) in object classification tasks can be invariant across objects. In contrast, the sentiment largely depends on the entire sentence instead of a few symbols or features. Then, we discuss the limitations of prior defenses.
Vector-wise. Common vector-wise defenses such as Krum estimate pair-wise similarities in terms of Euclidean distance (Blanchard et al. (017) (Krum and multi- Krum) of cosime similarity (Nguyen) et al. 2 (multi-Krumec) between each gradient and others. The gradients that are dissimilar to others are removed. The vector-wise view is insufficient for defending against backdoor attacks because backdoor attacks succeed by manipulating a tiny subset of parameters (e.g. 5%) (Wul & Wang 202T) without incurring much vector-wist difference. In practice, we observe that the malicious gradients get high similarity scores and circumvent vector-wise defenses.
8

---

Table I: Accuracy of Aggregators under Edge-case Backdoor Attack. Our approach reduces the model accuracy on backdoor samples by 61.7% on average, mitigating the backdoor attack, and achieves a comparable utility on benign samples as the standard FedAvg aggregator.
CIFAR-10
Twitter
Phishing
Method
AceM
ACCв
ACCи
ACCB
ACCи
ACCв
FedAvg
.679 .717 001
.722 .001 .440
.999
.999
Krum
.140 .011 .275
1 .122
.579 = .001 .766 + 002
.999 = .01 .999 001
Krulti-rum
.541
.923
+ X22
.727 = .656
.999 .999
Multi-Krn KuultiKKrumc-
.681 JM02
.821
+ 0001
.594 .002 .701 + .01
.999 * .m .333 233
Trimmed-Mean
.687 = .001
.512
+ 002
.728 = .001 .640 =
.999 = .999
Krum Trimmed-Mean
.682 .011
.607
+ 002
.727 .01 .641
- 001
.999 0011 .999 001
Sign-SGD
.301 .005
.000
10001
.610 .003 .751
4: .066
.999 + .000 .667 233
Weak-DP
.454 .033
.828
1: 003
.667 + .001
.374
1: .022
.999 .011 .999
Freezing Layers
.415 .001
.572
1: 002
A
N
N'
N),A
FoolsGold
.667 .011
.009
+ 001
.726 JM12 .357 + .011
.999
.999
RFA
.685 .011
.853
+ 0002
.718 = J011 .704 + .02
.999 + .00 .999 001
SparseFed
.662 .001
.984
001
.667 .001
.608
.999
.999
No Attack
.718 0011
.000
1 001
.731 0011 .095 + 001
.999 = 001 ,000 + 000
Ours
.677 = .011
.001
+ .001
.687 = .01 .29 + .083
.999 = .000
Note: The numbers are average accuracy over three runs. Variance is rounded up.
Dimension-wise Failure mode 2 in Section shows the limitation of the trimmed-ne estimator, which was the most effective defense against the edge-case backdoor attack. We also include Sign- SGD with majority vote (Bernstein al. 2019) defense, which binarizes the gradient and takes the majority vote as the aggregation result. However, Sign-SGD struggles to train a large federated model (c.g., Resnet-18 18 on CIFAR- 10) and can suffer from failure mode I where the clients have diverse signs. Then, the adversary can put more weight on one side and mislead the voting result.
Combination. A naive combination of multi Krumm and the trimmed- mean estimator fails to defend against the backdoor attack because neither multi-Krum nor the trimmed-mean estimator avoids the failure mode of the other.
Weak-DE The weak-DP defense (Sun et al. 2019) first bounds the gradient norms, then add additive noise (e.g., Gaussian noise) to the gradient vector. The edge-case backdoor attack can work without scaling up the gradients, circumventing the norm bounding. For the additive noise, we hypothesize that in some dimensions, the difference between malicious and benign gradients can be too large for the Gaussian noise to blur their houndary.
Freezing Layers. Since we employ pre-trained Resnet-18 88 (He et al. (016) CIFAR-IO. freezing the convolution layers may avoid entangling the trigger. However, this approach lacks empirical robustness, possibly because the adversary that the pre-trained model already learns as triggers.
use semantic features (e.g., blue color on airplanes)
Advanced Defenses. FoolsGold (Fung et al. 2020) down-weights an update if that update has high cosine similarity with another update. There are many ways to diversify updates. Malicious clients may leverage the stochastic gradient estimation process or mix backdoor samples with benign samples, whose distribution can differ across clients. RFA (Pillutla et al. 2022) computes geometric medians the aggregation result, which is shown be ineffective (Wang al. 2020). SparseFed (Panda et al., 2022) only accepts elements with large magnitude the aggregation results. However, benign and malicious updates contribute to large magnitudes.
CONCLUSION AND FUTURE WORK
This paper shows how to defend against backdoor attacks by focusing on the invariant directions in the model optimization trajectory. Enforcing the model to follow the invariant direction requires AND-mask to compute the sign-consistency variant a limension-wise direction can be, and use the trimmed-meat estimator to guarantee the model follows the invariant direction within each dimension. Both theoretical and empirical results demonstrate the combination of AND-mask and the trimmed-men mean estimator is necessary and ef- fective. Further defending against more advanced backdoor attacks such as invisible backdoors that add calibrated noise to all the features (Li et al. 2021b
each gradient dimension, which estimates how in-
Manoj & Blum! 2021) can be interesting.

---

REFERENCES
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to
backdoor federated learning. In Silvia Chiappa and Roberto Calandra (eds.). Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 2938 2948. PMLR, 26-22 28-28/ Aug 2020. URL press/pess/vlll / h
httpps:/pproceddi
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar.
signSGD: Compressed optimisation for non-convex problems. In Jennifer Dy and Andreas Krause (eds.), Proceedings the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 560-569. PMLR, 10-15 Jul 2018. URL 8a. html
https: htw.roceeainsssm...rrrss////////
m
Jeremy Bernstein, Jiawei Zhao, Kamyar Azizzadenesheli, and Anima Anandkumar. signsgd with majority vote is communication efficient and fault tolerant. In ICLR, 2019.
Peva Blanchard, El Mahdi EI Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. In NIPS, 2017.
Sebastian Caldas, Peter Wu, Tian Li, Jakub Konecný, H. B. McMahan, Virginia Smith, and Ameet S. Talwalkar. Leaf: A benehmark for federated settings. ArXiv, abs/1812.01097, 2018.
Clement Fung, Chris J. M. Yoon, and Ivan Beschastnikh. ing in sybil settings. In 23rd International Symposium on Research in Attacks, Intrusions and Defenses (RAID 2020), pp. 301-316, San Sebastian, October 2020. USENIX Associ- ation. ISBN 978-1 978-1-939133-18-22 URL https: /ttp:///www... wwt.:/wwwni..crg/eeeo. rra102020/preseetationn//
The limitations of federated learn-
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Audun Jøsang, Roslan Ismail, and Colin Boyd. A survey of trust and reputation systems for online service provision. Decis. Support Syst., 43:618-660 14644488644,6 2007.
Peter Kairouz, H. B. McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary B. Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D'Oliveira, Salim Y. El Rouayheb, David Evans, Josh Gardner. Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaïd Harchaoui, Chaoyang Hc, Lie Hc, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Kho- dak, Jakub Konecný, Aleksandra Korolova, Farinaz Koushanfar, Oluwasanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, R. Pagh, Mari- ana Raykova, Hang Qi, Daniel Ramage, Ramesh Raskar, Dawn Xiaodong Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang. Li Xiong, Zheng Xu, Qiang Yang, Felix X. Yu, Han Yu, and Sen Zhao Advances and open problems in federated learning. ArXiv, abs/1912.00477,, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009,
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learn- ing: Training clean models on poisoned data. In Advances in Neural Information Processing Systems, 2021a.
Yuezun Li, Y. Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. 202/ IEFF/CVF International Conference on Computer Vision (ICCV), pp. 16443-16452, 2021b.
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and X. Zhang- Trojaning attack on neural networks. In NDSS, 2018
Gábor Lugosi and Shahar Mendelson. Robust multivariate mean estimation: The optimality of trimmed mean. The Annals of Statistics, 49(1):393 410, 2021. doi: 10.1214/20- 11114/20-A001661. AOS1961.
URL httstts::: :/dooi .org/ -
10

---

Naren Manoj and Avrim Blum. Excess capacity and backdoor poisoning. In M. Ran- zato, A. Beygelzimer, Y. Dauphin. P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 00373-20385 Curran Asso- ciates, Inc., 2021. URL htl.httpss///rrrceedinggs neuringnnssnenu.oo...... ps. cc/ hcc/paper/2221/F11ee
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Ar- cas. Communication-Ifficieett Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jerry Zhu (eds.), Proceedingg of the 20th International Conference on Artifi- cial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Reseurch, pp. 1273-1282. PMLR, 20-22 Apr 2017. URL https: httt::://procceeiinngg www.mmmr...ps..... press/v547 momahanl7a.hter
Thien Duc Nguyen, Phillip Rieger, Huili Chen, Hossein Yalame, Helen Mollering, Hossein Ferei- dooni, Samuel Marchal, Markus Miettinen, Azalia Mirhoseini, Shaza Zeitouni, Farinaz Koushan- far, Ahmad-Rezz Rezza Sadeghi, and Schneider. Flame: Taming backdoors federated learning. 2021.
Ashwince Panda, Saeed Mahloujifar, Arjun Nitin Bhagoji, Supriyo Chakraborty, and Prateek Mittal. Sparsefed: Mitigating model poisoning attacks in federated learning with sparsifi- cation. Gustau Camps- Valls, Francisco J. R. Ruiz, and Isabel Valera (eds.), Proceedings of The 25th International Conference Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 7587-7624. PMLR, 28-30 Mar 2022. URL http::s//proceeiieo ngg.
hvlls//1111111111 51/ppddap2dda2e .html
Giambattista Parascandolo, Alexander Neitz, ANTONIO ORVIETO, Luigi Gresele, and Bernhard Schölkopf. Learning explanations that are hard to vary. In International Conference on Learning Representations, 2021. /ttt::ttt///ggot
net wtt/://rrmmmiioooo http///oooommmo--bbbooDiii.
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word rep- resentation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/D14-1166 1162. PRLLitttps//aaaaatboooooy
DD14 1162
Krishna Pillutla, Sham M. Kakade, and Zaid Harchaoui. Robust aggregation for federated learning. IEEE Transactions on Signal Processing, 70: 144--144,,, 2022. doi: 10.11000/SP22222.315111..
Elan Rosenfeld, Pradeep Kumar Ravikumar, and Andrej Risteski. The risks of invariant risk min- imization. In International Conference on Learning Representations, 2021. URL https:
//openzeview. weweeeoooridddaa//.e..ee....
Virat Shejwalkar, Amir Houmansadr, Peter Kairouz, and Daniel Ramage. Back to the drawing board: A critical evaluation of poisoning attacks on production federated learning. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1354-1371, 2022. doi: 10.1109/SP46214.202 9833647.
Mingjie Sun, Zichao Li, Chaowei Xiao, Haonan Qiu, Bhavya Kailkhura, Mingyan Liu, and Bo Li. Can shape structure features improve model robustness under diverse adversarial settings? 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7506-7515, 2021.
Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H. B. McMahan. Can you really backdoor federated learning? ArXiv, abs/1911.00 19.00796,, 2019.
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL 2B00EBDafS1LL02225333333332
pdf
Haoxiang Wang. Haozhe Si, Bo Li, and Han Zhao. Provable domain generalization via invariant- feature subspace recovery. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. 23018 23033, PMLR, 17-23 Jul 2022. hRI.https::0 htt.tt:::://pooceeeiiee ngs ml www.rrprrsss/seeme v 62 /tt////wanae a 22x htmll

---

Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy- yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos. Attack the tails: Yes, you really can backdoor federated learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 16070-16084, Cur- Associates, Inc., 2020. URL httpttttt/////rcddiiiiii FLLe://8Ea2L10022222222222222222 Paper. pdf
wwwr htt.rrips..ccppppe////00000
Emily Wenger, Josephine Passananti, Arjun Nitin Bhagoji, Yuanshun Yao, Haitao Zheng, and Ben Y. Zhao. Backdoor attacks against deep learning systems in the physical world. 2021 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6202-6211, 2021..
Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. In A. Beygelzimer, Y. Dauphin, Liang. and Wortman Vaughan (eds.). Advances in Neu- ral Information Processing Systems, 2021. TRI.htpp:: /ttp///ppp/viieeeeeee
https::/opperreiieeeeeee net. forumm/
4cEapqxFP30
Chulin Xie, Keli Huang, Pin-Yu Yu Chen, and Bo Li. Dba: Distributed backdoor attacks against federated learning. International Conference on Learning Representations, 2020a. URL
https: /opennevv ew net. 2 forummid=rkkyss0m
Chulin Xie, Minghao Chen, Pun-Yu Chen, and Bo Li. Crfi: Certifiably robust federated learning against backdoor attacks. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th In- ternational Conference Machine Learning, volume 139 of Proceedings Machine Learning Research, pp. 11372-11382. PMIR, 18-24 Jul 2021. URL
www.ediigsmm..... .gs.mlr. mll..
press/v139/ x aelie.llllhhl.... htmll
Cong Xie, Oluwasanmi Koyejo. and Indranil Gupta. Slsgd: Secure and efficient distributed on- device machine learning. In Machine Learning and Knowledge Discovery in Databases, pp- 213-228, Cham, 2020b. Springer International Publishing. ISBN 978-3-030-46147-88
Xiao Zhou, Yong Lin, Weizhong Zhang, and Tong Zhang. Sparse invariant risk minimization. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (cds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 27222-27244, PMLR, 17-23 Jul 2022. URL httpps:/ ptttp:://proceedinggs ml hwwpphhwwww.......s..eee.p v1 62 20//2////zz zhou22e html
12


Reviews:
[{"review_id": "bP3pSvgcjGV", "reviewer": "Reviewer_9BAU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies defenses against federated learning backdoor attacks. The paper proposes to force the model optimization trajectory to focus on the invariant directions that are generally useful for utility and avoid selecting directions that favor malicious clients. Particularly, the authors consider the consistency of the client model update as an estimation of the invariance, where the AND-mask and Trimmed mean are used together to measure the sign consistency. The proposed robust aggregator is evaluated on several datasets to defend against backdoor attacks, and show a promising utility-robustness tradeoff.  \n", "summary_of_the_review": "Detailed comments:\n\nI do not quite understand Figure 1(a): When using the AND-mask, does the result show negative signs, which returns the accurate sign of the benign dimensions?\n\nWhat does expectation ratio indicate in Theorem 4? What is the rough error bound in Equation (5)? Is it too large? Similarly, what is the rough probability in Theorem 5? \n\nTheorem 4 & 5 only relate to a single feature dimension? Can the result be generalized to the feature vector? What is the error bound? \n\nTheorem 8 is for the simple linear model. I am not sure whether the theoretical results can be applicable to general nonlinear model. \n\nI am also curious about the attack scenario that can break both the AND-mask and trimmed-mean. \n\nCan the proposed solution be generalized to defend against model poisoning attacks? \n\n\nWhich features are filtered by AND-mask and which filtered by the Trimmed mean? I would like to see whether they are complementary or have a large overlap.  \n\nWhy no attacks in Table 1 has non-zero backdoor accuracy? \n\n", "strength_and_weaknesses": "Strengths\n+ The studied problem is important\n+ Detailed analysis on why AND-mask and Trimmed mean together to enhance robustness\n\nWeaknesses\n-Some details are unclear\n-Theoretical results are narrow \n-Lack of comparison with state-of-the-art defenses", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Vl1-B23wbDt", "reviewer": "Reviewer_Pmty", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a method to mitigate backdoor attacks in federated learning by focusing on invariant directions of gradients and avoiding selecting directions that favor malicious clients.", "summary_of_the_review": "This paper is marginally below the acceptance threshold since although the authors proposed a simple yet effective approach to mitigate backdoor attacks in FL, we can not determine whether this is a general method or only suitable for some special cases.", "strength_and_weaknesses": "Strength:\n1. This paper proposed a simple yet effective approach to mitigate backdoor attacks in FL. compared with previous robust aggregation methods, the proposed method can defend against a much stronger attack, i.e., an edge-case attack.\n2. This paper gives robustness analyses for both AND mask and trimmed mean.\n\nWeakness:\n1. Only one backdoor attack in FL (edge-case attack) is tested in the experiments, which cannot validate the proposed method is a general defense method for backdoor attacks in FL. What about other frequently adopted backdoor attacks in FL, such as [1], and the experimental setting adopted in [2].\n2. According to the federated learning setup in the experiments, it adopts an iid client setting. In the case of non-iid clients, will the proposed method of masking inconsistent gradients still work?\n3. Masking inconsistent gradients will also sacrifice a lot of benign accuracies. However, according to the experiment results, the benign accuracy is hardly affected, could you give a more clearer explanation for this? Besides, the benign accuracy is also very low for FedAvg, which is very strange since usually, the benign accuracy for cifar10 in FL is at least over 80%. \n[1] Bagdasaryan E, Veit A, Hua Y, et al. How to backdoor federated learning[C]//International Conference on Artificial Intelligence and Statistics. PMLR, 2020: 2938-2948.\n[2] Panda A, Mahloujifar S, Bhagoji A N, et al. SparseFed: Mitigating Model Poisoning Attacks in Federated Learning with Sparsification[C]//International Conference on Artificial Intelligence and Statistics. PMLR, 2022: 7587-7624.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VRh6Fn8zQw", "reviewer": "Reviewer_uH18", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new method to defend against backdoor attacks. The proposed method is basically a new aggregation rule. Edge-case backdoor attacks are considered, and multiple simple baselines are evaluated. ", "summary_of_the_review": "The paper has limited novelty, and experiments are also limited. ", "strength_and_weaknesses": "Strengths\n\n- FL is vulnerable to backdoor attacks\n\n- A new aggregation rule is proposed. \n\nWeaknesses\n\n- Many robust aggregation rules have been proposed. This paper's proposed aggregation is just some variant of existing one, e.g., trimmed-mean. Therefore, the paper's novelty is limited. \n\n- Seems like only edge-case backdoor attacks are evaluated. This represents a quite limited backdoor case. Other backdoor attacks should be evaluated. \n\n- Some simple baselines are evaluated. Strong and more recent defenses (appeared in AI, security, and system venues) should be evaluated. \n\n- Adaptive attacks are not considered. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "LD1rFB2ygm", "reviewer": "Reviewer_f8s8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to defend backdoor attacks in federated learning. Particularly, it proposes to combine existing two techniques, AND-mask and trimmed-mean estimator, to remove suspicious gradient dimensions. This is based on the assumption that the gradients from benign clients have the same sign while those from malicious clients are different from benign ones. This paper also provides theoretical analysis on the gradient sign consistency and how the combination of the two existing techniques defends backdoor attacks. The evaluation is conducted on three datasets and one backdoor attack. Compared to a few baseline methods, the proposed approach has a better defense performance.", "summary_of_the_review": "This paper addresses an import problem but lacks novelty and comprehensive empirical results.\n", "strength_and_weaknesses": "### Strength\n\n+ Important problem of defending backdoor attacks in federated learning\n+ Easy to follow in most places\n\n\n### Weaknesses\n\n- No empirical validation of the motivation\n- Limited novelty\n- No evaluation on more existing backdoor attacks\n- No evaluation on adaptive attacks\n- Missing state-of-the-art defense baselines\n- Missing references and details of baseline techniques\n\n### Detailed comments\n\nDefending backdoor attacks in federated learning is a timely and important topic. This paper aims to address this problem by leveraging the combination of existing techniques. The paper is mostly easy to follows. There are a few aspects not well addressed in the paper.\n\n* In Figure 1, the paper uses two illustrative examples to demonstrate the failure scenarios of the two existing methods respectively. However, there is no empirical result showing that such cases exist in real backdoor attacks. In addition, there could be other attack scenarios. For example, both benign and malicious values are on the same side of the sign. But the malicious value can be far from the benign ones. Other cases may also exist. It shall be empirically studied and validated regarding those cases.\n\n* The novelty is limited. This paper simply combines the two existing techniques, without any modifications. The combination is completely based on the two illustrative examples given in Figure 1, which could be far from comprehensive. The theoretical analysis is conducted on a linear classifier, which is also based on the two assumed attack scenarios. The analysis on the gradient sign does not address the aforementioned case where the malicious gradient shares the same sign as benign ones.\n\n* There are many other backdoor attacks in federated learning such as [1-2]. Particularly, there is an attack setting in [2] where the attacker is selected in each round and continuously participates in the training from beginning to the end, which is a much more aggressive attack. The proposed method shall be evaluated on those backdoor attacks.\n\n* Since the proposed defense is based on the assumption of the malicious gradient sign deviate from benign ones, an adaptive attacker can match the malicious gradient sign with a benign one during the attack. Will the proposed defense still be effective?\n\n* There are other state-of-the-art defense techniques [3-5]. They shall be compared with in the evaluation.\n\n* There are no references and detailed descriptions of those baselines in Table 1.\n\n\n### References\n\n[1] Bagdasaryan, Eugene, et al. \"How to backdoor federated learning.\" AISTATS 2020.\n\n[2] Xie, Chulin, et al. \"Dba: Distributed backdoor attacks against federated learning.\" ICLR 2019.\n\n[3] Pillutla, Krishna, et al. \"Robust aggregation for federated learning.\" IEEE Transactions on Signal Processing 2022.\n\n[4] Fung, Clement, et al. \"The limitations of federated learning in sybil settings.\" RAID 2020.\n\n[5] Cao, Xiaoyu, et al. \"FLTrust: Byzantine-robust Federated Learning via Trust Bootstrapping.\" NDSS 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject