FLIP: A Provable Defense Framework for Backdoor Mitigation in Federated Learning
Authors: Kaiyuan Zhang
Guanhong Tao
Qiuling Xu
Siyuan Cheng
Shengwei An
Yingqi Liu
Shiwei Feng
Guangyu Shen
Pin-Yu Chen
Shiqing Ma
Xiangyu Zhang
Venue: ICLR 2023 poster (2023)
Keywords: Federated learning
backdoor mitigation

Abstract: Federated Learning (FL) is a distributed learning paradigm that enables different parties to train a model together for high quality and strong privacy protection. In this scenario, individual participants may get compromised and perform backdoor attacks by poisoning the data (or gradients). Existing work on robust aggregation and certified FL robustness does not study how hardening benign clients can affect the global model (and the malicious clients). In this work, we theoretically analyze the connection among cross-entropy loss, attack success rate, and clean accuracy in this setting. Moreover, we propose a trigger reverse engineering based defense and show that our method can achieve robustness improvement with guarantee (i.e., reducing the attack success rate) without affecting benign accuracy. We conduct comprehensive experiments across different datasets and attack settings. Our results on nine competing SOTA defense methods show the empirical superiority of our method on both single-shot and continuous FL backdoor attacks. Code is available at https://github.com/KaiyuanZh/FLIP.

Published as a conference paper at ICLR 2023
FLIP: A PROVABLE DEFENSE FRAMEWORK FOR BACKDOOR MITIGATION IN FEDERATED LEARNING
Kaiyuan Zhang, Guanhong Tao, Qiuling Xu, Siyuan Cheng, Shengwei An, Yingqi Liu, Shiwei Feng, Guangyu Shen, Pin-Yu Chen', Shiqing Ma', Xiangyu Zhang Purdue University, 'IBM Research, *Rutgers University
f2han4057, taog, xu1230, cheng535, an93)0cs purdue, edu, [1iu1751, Eeng292, shen447, xyzhang) par.puuue edu, "pin-yu. chen@ bmm com, 1sm22830cs rutgers
ABSTRACT
Federated Learning (FL) is distributed learning paradigm that enables different parties to train a model together for high quality and strong privacy protection. this scenario, individual participants may get compromised and perform backdoor attacks by poisoning the data (or gradients). Existing work on robust aggregation and certified FL robustness does not study how hardening benign clients affect the global model (and the malicious clients). In this work, we theoretically analyze the connection among cross-entropy loss, attack success rate, and clean accuracy in this setting. Moreover, we propose a trigger reverse engincering based defense and show that our method can achieve robustness improvement with guarantee (i.e., reducing the attack success rate) without affecting benign accuracy. We con- duct comprehensive experiments across different datasets and attack settings. Our results on nine competing SOTA defense methods show the empirical superiority of our method on both single-shot and continuous FI. backdoor attacks. Code available at htttt:: // g thub. com Ka yuanZh/FLIP
INTRODUCTION
Federated Learning (FL) is a distributed learning paradigm with many applications, such as next word prediction (McMahan et al.) (017), credit prediction (Cheng et al.] (202la). and IoT device ag- gregation (Samarakoon et al. 2018). 1). promissss scalability and privacy its training is distributed to many clients. Due the decentralized nature of FI., recent studies demonstrate that individual participants may be compromised and become susceptible to backdoor attacks (Bagdasaryan et al.) 2020 Bhagoji et al. 2019 Xie et al. 2019 Wang et al. 2020a Sun et al., 2019). Backdoor attacks aim make any inputs stamped with specific pattern misclassified to a target label. Backdoors are hence becoming a prominent security threat to the real-world deployment of federated learning.
Deficiencies of Existing Defense. Existing FL defense methods mainly fall into two categories, robust aggregation (Fung et al. 2020) Pillutla et al.) 2022; Fung et al.] 2020; Blanchard et al.) 2017 El Mhamdi al. 2018 Chen al. (017) which detects and rejects malicious weights, and certified defense (Cohen et 2019 Xiang et al. 2021 Levine & Feizi 2020 Panda et al., 2022 Cao et al.. (021) which provides robustness certification in the presence of backdoors with limited magnitude. Some of them need a large number of clean samples in the global server (Lin) al. 20206 Li al. 2020a), which violates the essence of FL. Others require inspecting model weights (Aramoon et al., (022I) which may cause information leakage of local clients. Existing model inversion techniques (Fredrikson al., 2015 Ganju et al.) 2018 An al. 2022) have shown the feasibility of exploiting model weights for privacy gains. Besides, existing defense methods based on weights clustering (Blanchard et al., 2017, Nguyen al.] (02T) either reject benign weights, causing degradation model training performance, accept malicious weights. leaving backdoor effective. According to our results in the experiment section, the majority of existing methods only work in the single-shot attack setting where only small set of adversaries participate in a few rounds and fall short in the stronger and stealthier continuous attack setting where the attackers continuously participate in the entire FL training.
FLIP. In this paper, we propose a Federated Learning Provable defense framework (FLIP) that provides theoretical guarantees. For each benign local client, FLIP adversarially trains the local

---

Published as a conference paper at ICLR 2023
Malisious clien
Stamping Tiggor
Backdnor amaak
Mai
Rojoct 10
Local data
Poisoned data
Benign client
Clean data
Advensrrial
Priining
Benign
opdites
060
Local data:
Triggut inversion
Angthanted
Local Client Training
Global Aggregate
Global Inference
Figure 1: Overview of FLIP. The left upper part (red box) performs the malicious client backdoor attack and the left lower part (green box) illustrates the main steps of benign client model training, they will submit local clients' updates to the global server. The middle part illustrates that the global server will aggregate all the received local clients' model weights and update the global server's model. The right part shows global server inference based on the updated global model. On benign clients, we do not assume any knowledge about the ground truth trigger.
model on generated backdoor triggers that can cause misclassification, which counters the data poisoming by malicious local clients. When all local weights are aggregated in the global server, the injected backdoor features in the aggregated global model are mitigated by the hardening performed on the benign clients. Therefore, FLJIP can reduce the attack success rate of backdoor samples. The overview of FLIIP is shown Figure [
a part of the framework, we provide benign client can affect malicious local client as well
theoretical analysis of how our training as the global model. To the best of our knowledge, this has not been studied the literature. The theoretical analysis determines that method ensures deterministic loss elevation on backdoor samples with only slight loss variation on clean samples. It guarantees that the attack success rate will decrease, and the model can meanwhile maintain the main task accuracy clean data without much degradation. Certified accuracy commonly used in evasion attacks that do not involve training. As data poisoning happens during training, it reasonable certify the behavior of models during training rather than inference.
Our Contributions. We make contributions on both the theoretical and the empirical fronts.
We propose FLIP, new provable defense framework that can provide sufficient condition on the quality of trigger recovery such that the proposed defense is provably effective in mitigating backdoor attacks.
We propose new perspective of formally quantifying the loss changes, with and without defense, for both clean and backdoor data.
We empirically evaluate the effectiveness of FLIP at scale across MNIST, Iishion-MNISY and CIFAR-10, 10, using non-linear neural networks. The results show that FLIP significantly outper- forms SOTAs on the continuous FL backdoor attack setting. The ASRs after applying SOTA defense techniques are still 100% in most cases, whereas FLIP can reduce ASRS to around 15%.
We design adaptive attack that is aware of the proposed defense and show that FL.IP stays effective.
We conduct ablation studies on individual components of FLIP and validate FLIP is generally effective with various downstream trigger inversion techniques.
Threat Model. We consider FL backdoor attacks performed by malicious local clients, which ma- nipulate local models by training with poisoned samples. On benign clients, we do not assume any knowledge about the ground truth trigger. Backdoor triggers are inverted on benign clients based on received model weights (from the global server) and their local data (non-i.i.d.). Standard training on clean data and adversarial training augmented data (clean samples stamped with inverted trig- gers) are then performed. The global server does not distinguish weights from trusted or untrusted clients. Nor does it assume any local data. Thus there no information leakage or privacy violation.
2

---

Published as a conference paper at ICLR 2023
The attack's goal is to inject backdoor to the global model, achieving high attack success rate without causing any noticeable model accuracy on clean samples. In our setting, a defender has no control over any malicious client, who may perform any kind of attack, c.g. model replacement or weight scaling. They can attack any round of FL.. In an extreme case, they attack in every round after the global model converges (if an attack is persistent since the first round, the model may not converge (Xie et al., (2019). In this paper, we consider static backdoors, i.c. patch backdoors (Gu al. 2017). Dynamic backdoors such reflection backdoors (Liu et al. 2020), composite back-
doors (in et al. 2020a), and feature space backdoors (Cheng et al. 2021b b) will be our future work.
2 RELATED WORK
Backdoor Attack and Defense. In general, the goal of backdoor attack is to inject a trigger pattern and associate it with a target label, e.g., by poisoning the training dataset. During testing, any inputs with such pattern will be classified as the target label. There are a number of existing backdoor attacks, like patch attacks (Gu et al.] [017 Liu et al. 2018), feature space attacks (Cheng et al. 2021b), etc. To identify whether a model is poisoned, existing works inverse triggers (Wang et al.
2019 Liu et al. 2019 Shen al., 2021 Liu et al. 2022 Tao et al. 2022] Cheng et al. [023)
ydentity differences between clean models and backdoored models (Huang et al., 2020 Wang et al.
2020b). There are also methods that detect and reject inputs stamped with triggers (Ma & Liu,
2019 Li et al. 2020b).
Federated Learning Backdoor Attack and Defense. Federated learning distributes model training to multiple local clients and iteratively aggregates the local models to shared global model. Since FL. local model training is private, attackers could hijack some local clients and inject backdoor
into the aggregated gobal model (Xie et al.] 2019 Tolpegin et al. 2020 Bagdasaryan et al., 2020
Fang et al. 2020 Shejwalkar et al. 2022). To defend against FL backdoor attacks, a number
of defense methods have been proposed (Blanchard et al. 2017 Pillutla et al., 2022 Sun et al.,
2019
Nguyen et al., 2021 Ozdayi et al. 2020; Fung et al., 2020; Andreina et al. 2021 Cao et al,,
2020 they focus more on detecting and rejecting malicious weights, which could fall short in the
stronger and stealthier attack settings and may lead to data leakage. Certified and provable defense
techniques (Panda et al. 2002]] Cao al. 2021 Xie al., (202T) also have been proposed analyze the robustness of FL, while they can provide robustness certification in the presence of backdoors with a (relatively) limited magnitude.
METHODOLOGY
In this section, we detail the design of FLIP, which consists of three main steps illustrated in Figure The procedure is summarized in Algorithm [ (1) Trigger inversion. During local client training. benign local clients apply trigger inversion techniques recover the triggers, stamp them on clean images (without changing their original labels) to constitute the augmented dataset. (2) Model hardening. Benign local clients combine the augmented data with the clean data to perform model hardening (adversarial training). The local clients submit updated local model weights to the global server, which aggregates all the received weights. (3) Low-confidence sample rejection. Our adversarial training can substantially reduce the prediction confidence of backdoor samples. During inference, we apply an additional sample filtering step, in which we use threshold to preclude samples with low prediction confidence. Note that this filtering step is infeasible for most existing techniques as they focus on rejecting abnormal weights during training.
Trigger Inversion. Trigger inversion leverages optimization methods to invert the smallest input pattern that flips the classification results of a set of clean images to target class. Neural Cleanse (Wang et al. (219)) uses optimization to derive trigger for each class and eventually ob- serves if there is any trigger that is exceptionally small and hence likely injected instead of naturally occurring feature. In our paper,
leverage universal trigger inversion that aims to generate trigger that can flip samples of all the classes (other than the target class) to the target class.
Class Distance. Recent work quantifies model robustness (against backdoor) by class distance (Too (et al. 2022). Given images from a source class s, it generates a trigger, consisting of a mask m and pattern which can flip the labels of these images stamped with the trigger to a target class The
3

---

Published as a conference paper at ICLR 2023
stamping function is illustrated in Equation and the optimization goal in Equation 2 where L(-) is the cross entropy loss, M denotes the subject model, and ||-Il denotes L i.c. the absolute value sum.
2/y (1 m)) 3i + m-m
Loss ((M(s we) ||m|||
(2)
The class distance dg-st is measured as ||m|||- The intuition here is that it's easy to generate a small trigger from the source class to the target class, the distance between the two class is small. Otherwise, the class distance is large. Furthermore, the model is robust if all the class distances are large, or one can easily generate small trigger between the two classes.
Cached Warm-up. Adversarial training on samples with inverted triggers is a widely used technique for model hardening. Observe that different label pairs have different distance capacities and enlarging label pair distances by model hardening can improve model robustness and help mitigate backdoors (Tao al. (022). Existing trigger inversion methods optimize all combinations of label pairs without selection and hence lead to quadratic computation time (O(n²), )). In order to reduce the trigger optimization cost, we first generate universal triggers, having each label being the target and prioritize promising pairs. It has a linear time complexity (O(n)), We consider pairs with larger distance capacity as having greater potential. Specifically, we need to know the difficulty of flipping source class to the target class, we leverage loss changes during optimization to measure. In other words, classes far from the target class have larger loss variances as they are quite different from the target class; once the predicted label is flipped to the target class, the loss value would be quite small. During optimization, we compute the variance of loss between source classes and the target class. FLIIP starts with warm-up phase and repeats the above process for each class, and utilizes the loss changes of different source labels to approximate class distances. Such information saved in a distance matrix or cache matrix (on each client). FLIP then prioritizes the promising pairs, namely, those with large distances. When the client is selected for hardening, FLIP generates the label-specific trigger for each source class and updates the distance matrix between the source class and target class.
Algorithm FLIP
1: Globals input: initial model parameters wo. total training round Ra, randomly select a set of clients K
2: Local client's imput: local dataset D : {xr, y and learning rate
3: for each training round r in [1, Ra] do
4: for each client A in K do
wh.11 f- Local Update (w,, D&) The aggregator sends w, to client who inverts triggers based w, and its data D2 D {ar,, 3 locally, and sends w/+1 back the aggregator
6: Wx41 1) k= w Global server aggregates all the received weights from the different clients Perform global model inference, is the
7: Global output: Global Model Inference (w T)
confidence threshold, x the input
8: function LOCAL.UPDATE(CCR D2) if client & was never selected then for each label existing in client & do distance mmatri([s][]] promising-pan pairs + select(distance.mari))
9:
10:
is source label, is target label Store all pair-wise class distances a cache matrix
L' (s. t)
12:
Select top promising pairs from cache matrix
13:
else if client was selected before then promising-pairs + select(distance matrixx
14:
15:
if promising pairs exist in dataset then
16:
Tado <- Semmmetric_Ivvessine (w., Ii; Hkk)
17:
else
18:
Fade f- Syymmetric_Inversioon (wr, 2h. y/
19:
w &- Adversarial_ _Triin ((T,, Tadv }, {yk. Yodo)) }) b Adversarial training clean and augmented
data, Vadv is the ground truth label of Tadv- return w
20:
In Algorithm [ each local client utilizes their local samples and computes their class distances (measured by L to target class and caches the results in the distance matrix (lines 11). Then based on the distance matrix, FLIP prioritizes the promising pairs with large distance (line 12). the client has been selected before,
directly get the promising pairs from the cached distance matrix (lines 13, 14). The distance matrix stored and updated locally by each client. Caching allows more iterations allocated for model hardening.
4

---

Published as a conference paper at ICLR 2023
(A (A)ymmetric Hardening. Given pair of label I and label 2. there are two directions for trigger inversion, from / to 2 and from 2 to I. A straightforward idea (Tao et al. 2022) is to invert both directions. However, it is impossible due to the non-i.i.d nature of chinnt data in federated learning, namely, a local client's training data can be extremely unbalanced and there may be very few or even no samples for certain labels in client. During model hardening (adversarial training), promising class pair is selected for hardening in each iteration, according to the distance matrix mentioned above. We hence separate the model hardening into bidirectional or single directional based on data availability, accordingly symmetric or asymmetric model hardening. That is, if there exist sufficient data for the two labels of class pair, symmetric hardening is carried out by generating triggers for the two directions and stamped on the corresponding source samples simultaneously (Alg lines 15, 16). If there are only samples of one label of the pair, we then only harden the direction from the label to the target (Alg [ lines 17, 18). Recovered trigger examples are shown in Appendix A Figure 2 (c). After benign clients finish model hardening, they submit the updated model weights to the global server (Alg [ llnes 5, 19, 20), then the global server aggregates all the received client weights (Alg line 6) and performs the global inference (Alg) line 7).
Algorithm 2 Symmetric and Asymmetric Inversion
I: Input: local model parameters Wr, local client data (xx,
2: Initialization: X,, batch of x Xx
Initialization: Initialize model M from model weights Wr, (Mttasst, Oense), label (a, b). indicator vector p
4: function SYMMETRIC SEEERRCC.IVVERRE INVERSION(W, Xx. 2)
5: m, 8 < Trigger_Initin (moutt 8
6: for step in [0, maz_stepp do (0) X,, m0[ m00)) +(1-p) ((1
7: X, -pppp(( ((1-m(0)
m(1l) X,, + m(1] -((]) Indices
and denote optimization direction. p denotes the direction of symmetric hardening
8: distance matrix[a][b] < T." (a. h)
9: distance matrix[b][a] < (b.
> Update distance matrix in both directions
10: return X
11: function ASYMMETRIC ASYAMEERRC.IVVERSOONII Xk. 1)
12: m,, 8 4 Trigger Initial (minit, 8init)
13: for step in [0, mar. mar_steps]] do
14:
=1= m) Xn. -
15:
+ La b)
Update distance matrix in a single direction
16:
return X
17:
function TRIGGER TRIGGER_INTTIAL/nR INITIAL(Marit 6mnc)) if Menst is None and 8ynit not None then
18:
19:
m, 8 +- Msnit, Sinst
Initialize mask and backdoor variables
20:
else
21:
m, 8 random init with shape of E
22:
return m. 5
In Algorithm 2 we present more details about the symmetric and asymmetric inversion. If the client has sufficient data (i.c. more than 5 images) for both labels of pair (a, b), we perform symmetric hardening by generating triggers from two directions (a b and b initialize the backdoor mask m and patter 5 from two directions (line 5). The indicator vector denotes the direction of symmetric hardening, i.e. denotes from label a to b and denotes from b to a. Then we stamp the triggers on the corresponding class samples (line 7), and update the distance matrix from two directions (line 8, 9). If the client only has sufficient data the source label of pair (a, b), we perform asymmetric hardeming from the source label to the target (i.e., + b). We initialize backdoor mask m and patter in one direction (line 12), then stamp the triggers on the corresponding class samples (line 14), and update the distance matrix in one direction (line 15).
simultaneously. We first
Low-confidence Sample Rejection. As the hardening on benign clients counters the data poisoning on malicious clients, the aggregated model on the global server tends to have low confidence in predicting backdoor samples (while the confidence on benign samples is largely intact). During the inference of global model, we apply threshold to filter out samples with low prediction confidence after the softmax layer, which significantly improves the model's robustness against backdoor attacks in federated learning. In the next section, we prove that, as long the inverted trigger satisfies our given bound, we can guarantee attack success rate must decrease and in the meantime the model can maintain similar accuracy clean data.
5

---

Published as a conference paper at ICLR 2023
4 THEORETICAL ANALYSIS
In this section, we develop theoretical analysis to study the effectiveness of our proposed defense in simple but representative FL setting. It consists of the following: (i) developing upper and lower bounds quantifying the cross-entropy loss changes on backdoored and clean data in the settings of with and without the defense (Theorem [T: (ii) showing a sufficient condition on the quality of trig- ger recovery such that the proposed defense is provably effective in mitigating backdoor attacks (Theorem 2): (iii) following (ii), we show that inference with confidence thresholding on models trained with our proposed defense can provably reduce the backdoor attack success rate while main- taining similar accuracy on clean data. During analysis, we leverage inequality (Mitrinovic & Vasié, 1970) and use variable substitution to get the upper-bound and dower-bound, To the best of our knowledge, our analysis is new to this field due to the novel modeling of trigger recovery and model hardening schemes in our proposed defense method. These findings are also consistent with our empirical results in more complex settings.
Learning Objective and Setting. Suppose the k-th device holds the ny training samples: [Xx,1; X, 1.. Xk. where x E IRldd The model has one layer of weights W E Rddxx Label q E IxX is a one-hot vector for I classes. In this work, we consider the following dis- tributed optimization problem: minw (F(W)= E 91 Fc(()) where N is the number of devices, Ik is the weight of the k-th device such that Ie 2 0 and Σ the objective function. The local objective (.(()) is defined by (x( = K== 9k = 1, and F(-) Eド= L(W; Xk.j); where ((:-) is loss function. In the global server, can write the global softmax cross-entropy loss function as Lylobat
e(ow) El= qi (ogsoftmax(xX)) = -E 9i log( E e(xw',"
(xW) log c(xW).), and label index for the classes. Appendix A.16
describes all used symbols in our paper.
In the theoretical analysis, we assume that we are under the FedAvg protocol (McMahan et al., 2017]. In order to simplify analysis without losing generality, we assume there is one global server and two clients: one benign and the other malicious. We conduct the analysis on multi-class classification using logistic regression. The model is composed of one linear layer with the softmax function and the cross-entropy loss.
Theorem I develops the upper and lower bounds quantifying the loss changes on backdoored and clean data the settings with and without the defense.
Theorem 1 (Bounds on Loss Changes). Let Cg denote the global model loss with defense, Cg without defense, let AW = W' W denote the weight differences with and without defense. The loss difference with and without defense can be upper and lower bounded by
min(xAW), - [x(xAw), s SC,- - max((xW)) (x(x))) 8
(3)
The detailed proof is provided Appendix A.14 The above theorem bounds the loss changes with and without the defense. From Cauchy-Schwarl inequality (Mitrinovic & Vasić 1970), we derive the lower bound and upper bounds of loss changes. Intuitively, given the parameter W of the linear layer, the k-th device holds nış training samples Ni, the loss differences of with and without de- fense must be larger than the lower bound A min loss as min, (Xt,m, AW): Ei=1 and smaller than the upper bound A max loss as max AW), El= qi(x,, AW),.
((xx,, AW):
To facilitate the analysis of attack success rate (ASR) and clean accuracy (ACC) changes. in- tuitively, we aim to analyze how much the ASR at least will be reduced and how much the ACC will at most be maintained. Thus, we studied this lower bound (A min .loss) on back- door data, which indicates the minimal improvements on the backdoor defense that reduce the ASR. Similarly we studied the upper bound (A max max_loss) for clean data, as they indicates the worst-case accuracy degradation. Denote backdoor samples as nı and clean samples as ne. Note that backdoor samples can be written as X, + 8. By using Theorem [ we have A min loss = Eme min, [x, + S)AW], ETe EL (xx + A)., And similarly on benign data, we have A max loss x,, A max loss De (X, Ene Σln gz. AW),

---

Published as a conference paper at ICLR 2023
Next, aim to develop a sufficient condition on the quality of trigger recovery such that the pro- posed defense is provably effective in mitigating backdoor attack and in the meantime maintaining similar accuracy on clean data, based on Theorem [
Theorem (General Robustmess Condition), Let a
Dr D E 9s (2 bz= 23 (q ((00-((zj)))) p(zi)))
b b1Th 1T E Ei ...1) [z) (qj p(z)))])
where b- !bi..., ...,,.... bal, d is the sample dimension,
let be sign 1/ Ee Ei (92. 9s.i) و==1 [2 q3- (-)((()))ii on all dimensions v of the vector. For all |/ S a, we have min loss 2 O.
And we have A max loss Th Ex E/= ( (,,1)x, -j=1 3/ p(z)));
The detailed proof is provided in Appendix A.15 We denote as one-hot vector for sample with dimensions. Its i-th dimension is defined as gs and 9-,, = i = arg min [(x, + (W" W)] Denote 6 as ground truth trigger, as difference of reversed trigger and ground truth trigger, 7) the learning rate (a.k.a. step size) in round r. Denote z as x + + E for simplicity, which is the benign sample stamped with the recovered trigger. Note that A min loss 2 indicates that the defense is provably effective than without defense. Since benign local clients' training can increase global model backdoor loss, and they have positive effects on mitigating malicious poisoning effect, the second condition A max loss S 7r Ex= E/= (v5, )s.;)xx xx Σj=1(z² [2]" (2]"((a - p (²z))){ indicates that the defense is provably guarantee maintaining similar accuracy on clean data, similarly, here i = argmar. (W" - W)), (with a slight abuse of the notation 2,).
Corollary 1. Assume e satisfies Theorem 2 let nu, as backdoored samples, ne as benign samples, as confidence threshold. Then the number of backdoored samples that are rejected is Red Ri Rs. the number of benign samples that are rejected is Ron Rt Re
Ry, and Rs denote the rejected backdoor samples with and without defense. With defense, R2 is ins Σe=1 1C, + A min loss > C+): and without defense, Rg is Σ; 1(0 g fr). Thus, the exact value of rejected backdoored samples can be calculated through Ria = Rt, 105--1. Rs.
Similarly, R. and R2 denote the rejected clean(benign) samples with and without defense. With defense, R. is Σグ=ı 1(2g + A max Joss > D and without defense, R. is ΣØ= 1(2 > L. Thus, the exact value of rejected benign samples can be calculated through Ron = R. Re-
EXPERIMENT
In this section, we empirically evaluate FLIP under two existing attack settings, i.e. single-shot
attack (Bagdasaryan et al.) 2020) and continuous attack (Xie et al.) 2019). We compare the per-
formance of FLIP with state-of-the-art defenses, i.e. Krum (Blanchard al.) 2017). Bulyan
Krum (El Mhamdi et al. 2018), RFA (Pillutla et al. (022). FoolsGold (Fung et al. 2020). Me-
dian Yin al. 2018). Trimmed Mean (Yin et al., 2018). Bulyan Trimmed Mean (Buly-Trim-
M) (El Mhamdi et al. 2018), and FLTrust (Cao et al. 2020). DnC (Shejwalkar & Houmansadr, 2021). Besides using the experiment settings in (Bagdasaryan al. 2020) and (Xie et al.] 2019), we conduct experiments using the setting in our theoretical analysis to validate our analysis. Moreover, we evaluate FLIP on adaptive attacks and conduct several ablation studies.
5.1 EXPERIMENT SETUP
We conduct the experiments under the PyTorch framework (Paszke et al. 2019) and reimplement existing attacks and defenses following their original designs. Regarding the attack setting, there are 100 clients in total by default. In each round randomly select 10 clients, including 4 adversaries and benign clients. Following the existing works setup, we use SGD and trains for local epochs (E is 5 in continuous attacks and 10 in single-shot attacks) with local learning rate Ir (ir is 0.1 in benign training and 0.05 in poison training) and batch size 64. More setup details in Appendix A.
Dataset. Our experiments
conducted on well-known image classification datasets, i... MNIST (LeCun et al. 1998), Fashion-MNISS (F-MNIST) (Xiao et al.) 2017) and CIFAR- 10 (Krizhevsky et al." 2009). Note that data not independent and identically distributed

---

Published as a conference paper at ICLR 2023
(non-i.idd) which is more practical in real-world applications. We leverage the same setting as (Bag- dasaryan et al. ) which applies a Dirichlet distribution (Minkal 2000) with a parameter a as 0.5 to model non-i.i.d distribution. The poison ratio denotes the fraction of backdoored samples added in each training batch, MNIST and Fashion-MNIST poison ratio is 20/64, CIFAR-10 is 5/64.
Evaluation Metrics. We consider attack success rate (ASR) and main task accuracy (ACC) as evaluation metrics to measure defense effectivvness. ASR indicates the ratio of backdoored samples that are misclassified as the attack target label, while ACC indicates the ratio of correct classification on benign samples. While certified accuracy is commonly used in evasion attacks that do not involve training. As data poisoning happens during training, more reasonable certify the behavior of models during training rather than inference.
5.2 EVALUATION ON BACKDOOR MITIGATION
We consider backdoor attacks via model replacement approach where attackers train their local mod- els with backdoored samples. We follow single-shot setting in (Bagdasaryan et al.) (0200 and con- tinuous setting in (Xie et al. (2199) to perform the attacks. Single-shot backdoor attack means every adversary only participates in one single round, while there can be multiple attackers. Continuous backdoor attack is more aggressive where the attackers
selected in every round and continuously participate in the FL training from the beginning to the end. Both settings of attacks happen after the global model converges, since if attackers poison from the first round, even after training enough rounds, (Xie et al., 2019) found that the main accuracy was still low and models hard to converge.
For fair comparison, we report ASR and ACC of FLIP and all the baselines in the same round.
Table I: Single-shot attack evaluation
MNIST
F-MNIST
CIFAR-10
Note that the confidence threshold of FL.IP in- troduced in Section only used in the contin- uous attacks to filter out low-confidence predic- tions, since single-shot attack is easy defend. Based on our empirical study, we typically set 0.3 for MNIST and Fashion-MNIST and T = 0.4 for CIFAR-10, and we evaluate thresh- olds with the Area Under the Curve metric (Ap- pendix A.4). Single-shot attack results shown Table [ Line illustrates the attack perfor- mance with No Defense. Observe that single- shot attack can achieve more than 80% ASR throughout all the datasets while preserving high main task accuracy over 77%. The follow- ing rows show the defense performance of existing SOTA defenses and the last row denotes FLIP results. We can find that FLIP can reduce the ASR to below 8% on all the 3 datasets and keep the benign accuracy degradation within 5%. FLIP outperforms all the baselines on both MNIST and Fashion -NIIST while is slightly worse on CIFAR-IO.
Baselines
ACC
ASR
ACC
ASR
ACC
ASR
No Defense
97.55
80.12
81.01
96.72
77.52
80.46
Krum
97.50
0.35
79.49
10.79
77.00
9.51
Bulyan Krum REA
97.76
0.39
81.45
6.42
79.65
5.77
97.93
0.39
81.82
4.39
79.54
6.13
Trimmed Mean
97.81
0.38
81.81
5,40
79.95
5.81
Buly-Trim-M
97.02
90.75
79,84
99.38
66.69
84.05
FoolsGold
97.51
0.39
80.59
5.64
78.67
3.70
Median
97.76
81.76
5.97
64.31
2.39
FLTrust
97.26
79.92
7.69
72.44
2.18
DnC
80.33
5.35
76.66
5.36
FLIP
96.05
0.13
78.20
3.16
73.41
7.83
We show the results of continuous attack in Ta- ble 2 Continuous attack is more aggressive than the single-shot one. The former's ASR is 3%-20% higher than the latter when there is no defense. Note that all the existing defense techniques fail in the continuous attack setting. The ASR remains nearly 100% in most cases on MNIST and Fashion-MNIST and is higher than 63% on CI However, FLIP re- duces the ASR to a low level and the accu- racy degradation is within an acceptable range. Specifically, FLIP reduces the ASR on MNIST to 2% while the accuracy degradation is within 2%. For Fashion-MNIST and CIFAR-10, the ASR is reduced to below 18% and 23%, respec- tively, while the accuracy decreases bit more
Table 2: Continuous attack evaluation
MNIST
F-MNIST
CIFAR-10
Baselines
ACC
ASR
ACC
ASR
ACC
ASR
No Defense
98.71
100.00
80.35
99.99
77.83
84,73
97.59
0.14
73.18
20.03
40.29
Bulyan Krom
98.15
94.01
82.17
99.46
68. 61
97.31
RFA
98.54
100.00
85.69
100.00
79.39
63.10
Trimmed Mean
98.52
100.00
84.59
99.99
75.18
91.84
Buly-Trim--h
98.80
100.00
76.18
99.93
71.91
68.83
FoolsGood
97.91
99.99
80.58
99.98
74.57
78.30
Median
98.14
66.01
84.07
99.34
57.01
69.99
FLTrust
91.96
20.60
74.63
35.36
74.85
68.70
DnC
89.81
99.78
65.90
97.73
53.16
82.37
FLIP
96.62
1.93
72.99
17.65
71.28
22.90
8

---

Published as a conference paper at ICLR 2023
compared to the results of MNIST and to single-shot attack. This is reasonable due to the following: First, the complexity of the dataset and continuous backdoor attacks may add to the difficulty of recovering good quality triggers. In addition, there is trade-off between adversarial training accu- racy and standard accuracy of a model as discussed in (Tsipras et al. 2019). Adversarial training on benign clients induce negative effects the accuracy. However, argue that FLIP still outperforms existing defenses as the ASR is reduced to a low level.
5.3 EVALUATION ON THE SAME SETTING AS THEORETICAL ANALYSIS
In this section, we conduct an experiment that follows the same setting as our assumptions in the theoretical analysis validate its correctness. We conduct experiments the multi-class logistic regression (i.e., one linear layer, softmax function, and cross-entropy loss) as the setting in theoreti- cal analysis Section We take MNIST the example for analysis and it can be easily extended to other datasets. Regarding the FL system setting, there are one global server, one benign client and malicious client. We train the FL global model until convergence and then apply the attack. The attackers inject the pixel-pattern backdoor in images and swap the label of the image source label to the target label. We also do not have any restrictions the attackers, long they follow the federated learning protocol.
Table 3 shows the result of single-shot and continuous attack ACC and ASR on the logistic regression. We can see both single- shot and continuous attacks' ASRs are reduced around 5% and the accuracy degradations are within an acceptable range. This result is consistent with our observations on more complex set- tings above. Besides, the exact value of rejected clean samples and backdoored samples under different settings can also be cal- culated, which correspond to Corollary details can be found in Appendix A.2
Table 3: Logistic regression evaluation
Attack Type
Metric
No Defense
FLIP
ACC
88.43
84,58
Single-Shal Shot
ASR
64.48
5.28
ACC
83.03
80.76
Continuous
ASR
63.78
4.90
5.4 ADAPTIVE ATTACKS
We study an attack scenario where the adversary has the knowledge of FLIP, our results show that FLIP still mitigate the backdoor attacks in most cases. For those that ACC does degrade, the adaptive attack is not effective. Details can be found in Appendix A.3
5.5 ABLATION STUDY
This section conduct several ablation studies. We study both adversarial training (Appendix (A.5) and thresholding (Appendix (A6) is critical in FLIP. We study another different trigger inversion technique in FLIP, which mitigate backdoors as well, indicating that FLIP is compatible with different trigger inversion techniques (Appendix A.7)) We study the effects of different sizes of trig- gers and show that our defense cause a significant ASR reduction while maintaining comparable benign classification performance (Appendix A.8). We also study different threshold influences on ACC and ASR and show the trade-off between attack success rate and accuracy (Appendix AA.9.
CONCLUSION
We propose new provable defense framework FLIP for backdoor mitigation in Federated Learn- ing. The key insight is to combine trigger inversion techniques with FL training. As long as the inverted trigger satisfies our given bound, we can guarantee attack success rate will decrease and in the meantime the model can maintain similar accuracy on clean data. Our technique significantly outperforms prior work on the SOTA continuous FL backdoor attack. Our framework is general and can be instantiated with different trigger inversion techniques. While applying various trigger inversion techniques, FLIP may have slight accuracy degradation, but can significantly boost the robustness against backdoor attacks.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We thank the anonymous reviewers for their constructive comments. This research was supported, in part by LARPA TrojAI 1991N1-1----011,, NSF 1901242 and 1910300, ONR N0001417120455 N000141 4114411111 410468 and N000141712947. Any opinions, findings, and conclusions in this paper are those of the authors only and do not necessarily reflect the views of our sponsors.
ETHICS STATEMENT
In this paper, our studies are not related to human subjects, practices to data set releases, discrim- imation/bias/fairness concerns, and also do not have legal compliance research integrity issues. Backdoor attacks aim to make any inputs stamped with a specific pattern misclassified to target label. Backdoors are hence becoming prominent security threat to the real-world deployment of federated learning. FLIP is provable defense framework that can provide a sufficient condition on the quality of trigger recovery such that the proposed defense is provably effective in mitigating backdoor attacks.
REPRODUCIBILITY STATEMENT
The implementation code is available at https::/github. wowaaaaaaaaaaaaaaaaaiiii... All datasets and code platform (PyTorch) we use are public. In addition, we also provide detailed experiment parameters in the Appendix.
REFERENCES
Shengwei An, Guanhong Tao, Qiuling Xu. Yingqi Liu, Guangyu Shen, Yuan Yao, Jingwei Xu, and Xiangyu Zhang. Mirror: Model inversion for deep learning network with high fidelity. In Proceedings the Network and Distributed Systems Security Symposium (NDSS 2022), 2022.
Sebastien Andreina, Giorgia Azzurra Marson, Helen Möllering, and Ghassan Karame. Baffle: Back- door detection via feedback-based federated learning. In 2021 IEEE 41st International Confer- ence on Distributed Computing Systems (ICDCS), pp. 852 863. IEEE, 2021.
Omid Aramoon, Pin-Yu Chen, Gang Qu, and Yuan Tian. Meta federated learning. arXiv preprint
arXiv:2102.05561, 2021.
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference Artificial Intelligence and Statistics, pp. 2938-2948. PMLR, 2020.
Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing fed- crated learning through an adversarial lens. In Kamalika Chaudhuri and Ruslan Salakhutdi- nov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 634 643. PMLR, 09-15 Jun 2019. URL http//99///b// wttpv//9//hhoooo1//
https::/prroceding mlr. ww..pprrsss............om... hwwpppppsssss//ppppepap
htmll
Peva Blanchard. El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. Advances in Neural Information Processing Systems, 30, 2017.
Xiaoyu Cao, Minghong Fang, Jia Liu, and Neil Zhenqiang Gong. Fltrust: Byzantine-robust feder- ated learning via trust bootstrapping. arXiv preprint arXiv:2012.13995 2020.
Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. Provably secure federated learning against malicious clients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 0885-6893, 2021.
Yudong Chen, Lili Su, and Jiaming Xu. Distributed statistical machine learning in adversarial set- tings: Byzantine gradient descent. Proc. ACM Meas. Anal. Comput. Syst., 1(2), dec 2017. doi: 10. 145314533 URL https: /oool oor.oooooooooooooooooooooooooooo www.org...poooooooo.. ezproxy. lib. purcue edd.o......1.1.55.5..
311544000
10

---

Published as a conference paper at ICLR 2023
Kewei Cheng, Tao Fan, Yilun Jin, Yang Liu, Tianjian Chen, Dimitrios Papadopoulos, and Qiang Yang. Secureboost: A lossless federated learning framework. IEEE Intelligent Systems, 36(6): 87-98, 2021a.
Siyuan Cheng, Yingqi Liu, Shiqing Ma, and Xiangyu Zhang. Deep feature space trojan attack of neural networks by controlled detoxification. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 1148-115 156, 2021b.
Siyuan Cheng, Guanhong Tao, Yingqi Liu, Shengwei An, Xiangzhe Xu, Shiwei Feng, Guangyu Shen, Kaiyuan Zhang, Qiuling Xu, Shiqing Ma, al. Beagle: Forensics of deep learning back- door attack for better defense. arXiv preprint vrXiv:2301.062411 2023.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning, pp. 1310-1320. PMLR, 2019.
El Mahdi El Mhamdi, Rachid Guerraoui, and Sébastien Rouault. The hidden vuinerability of dis- tributed learning Byzantium. Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3521-3530. PMLR, 10-15 Jul 2018. URL htttps://rooceediggs....
Ttess/v80//haaee 188 1a..8hhh. htmml
Minghong Fang. Xiaoyu Cao, Jinyuan Jia, and Neil Gong. Local model poisoning attacks to byzantine-robust federated learning. In 29th {USENIX] Security Symposium, 2020.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confi- dence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC confer- computer and communications security, pp. 1333, 2015.
Clement Fung, Chris J. M. Yoon, and Ivan Beschastnikh. The Limitations of Federated Learning in Sybil Settings. In Symposium on Research in Attacks, Intrusion, and Defenses, RAID, 2020.
Karan Ganju, Qi Wang, Wei Yang, Carl A. Gunter, and Nikita Borisov. Property inference attacks on fully connected neural networks using permutation invariant representations. In Proceed- ings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, CCS *18, pp. 619-633, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450356930. doi: 10. 00.11445/224333442248844
URL https: bttpss//doi-oon www.gxy..poxy....
lib. purdue. eda /100 /11111155500
http:/1..111//32223330 3243733 3243834
Tianyu Gu, Brendan Dolan-Gavite and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv: 2017.
Shanjiaoyang Huang, Weiqi Peng, Zhiwei Jia, and Zhuowen Tu. One-pixel signature: Characterizing enn models for backdoor detection. European Conference on Computer Vision, pp. 326-341. Springer, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11)2278-23224 1998.
Alexander Levine and Soheil Feizi. Deep partition aggregation: Provable defense against general poisoning attacks. arXiv preprint arXiv:2006. 14768, 2020.
Suyi Li, Yong Cheng, Wei Wang, Yang Liu, and Tianjian Chen. Learning to detect malicious clients for robust federated learning. arXiv preprint 20Xiv:2002.00221, 2020a.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pp. 6357- 6368. PMLR, 2021.
Yiming Li, Tongqing Zhai, Baoyuan Wu, Yong Jiang, Zhifeng Li, and Shutao Xia. Rethinking the trigger of backdoor attack. arXiv preprint arXiv:2004.0440 04692, 2020b.

---

Published as a conference paper at ICLR 2023
Junyu Lin, Lei Xu, Yingqi Liu, and Xiangyu Zhang. Composite backdoor attack for deep neural network by mixing existing benign features. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, pp. 113-131, 2020a.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning, Advances in Neural Information Processing Systems, 33:2351-2366
2020b.
Yang Liu, Mingyuan Fan, Cen Chen, Ximeng Liu, Zhuo Ma, Li Wang, and Jianfeng Ma. Backdoor defense with machine unlearning. arXiv preprint arXiv:2201.0900 009538, 2022.
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack neural networks. In NDSS, 2018.
Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma. Yousra Aafer, and Xiangyu Zhang- Abs: Scanning neural networks for back-doors by artificial brain stimulation. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, CCS '19, pp. 1265-1282, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367479.
URL lib. purdue. www.edal.... htt.://......1113311333333. 3363216
doi: 10.1 (0.11445/311953333666666
Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack on deep neural networks. In European Conference on Computer Vision, pp. 182-199. Springer, 2020.
Shiqing Ma and Yingqi Liu. Nic: Detecting adversarial samples with neural network invariant checking. In Proceedings of the 26th network and distributed system security symposium (NDSS 2019), 2019.
Jayawant N Mandrekar, Receiver operating characteristic curve in diagnostic test assessment. Jour- nal of Thoracic Oncology. 5(9):1315 2010.
Brendan McMahan, Eider Moore, Daniel Ramage. Seth Hampson, and Blaise Aguera y Arcas. Communication-efficienn learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282 PMLR, 2017.
Thomas P. Minka. Estimating dirichlet distribution. Technical report, 2000.
D.S. Mitrinovic and P.M. Vasić. Analytic Inequalities. Die Grundlehren der mathematischen Wissenschaften in Einzeldarstellungen. Springer- Verilge--Verlag, 1970. ISBN 9780387048376. URL
https://books. google www
Thien Due Nguyen, Phillip Rieger, Huili Chen, Hossein Yalame, Helen Möllering, Hossein Fer- eidooni, Samuel Marchal, Markus Miettinen, Azalia Mirhoseini, Shaza Zeitouni, et al. Flame: Taming backdoors federated learning. Cryptology ePrint Archive, 2021.
Mustafa Safa Ozdayi, Murat Kantarcioglu, and Yulia Gel. Defending against backdoors in feder- ated learning with robust learning rate. arXiv preprint arXiv:2007.03767. 2020.
Ashwinee Panda, Saeed Mahloujifar, Arjun Nitin Bhagoji, Supriyo Chakraborty, and Prateek Mit- tal. Sparsefed: Mitigating model poisoning attacks in federated learning with sparsification. In International Conference Artificial Intelligence and Statistics, pp. 7587-7624. PMLR, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner. Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. Curran Associates, Inc., 2019. URL http: http://tt:/pppppe...e.i......p.p ve st..st..lle............ http:p/wee.t..t..o.ooooooooooaoe
901 www-ppyttochhooo an pdf
library.
Krishna Pillutla, Sham M. Kakade, and Zaid Harchaoui. Robust aggregation for federated learning.
IEEE Transactions Signal Processing, 70:1 1111142-154, 2022. doi: 10. 10.1009/SP.222.315333
12

---

Published as a conference paper at ICLR 2023
Sumudu Samarakoon, Mehdi Bennis, Walid Saad, and Merouane Debbah. Federated learning for ultra-reliable low-latency v2v communications. In 2018 IEEE Global Communications Confer- ence (GLOBECOM), pp. 1-7. IEEE, 2018.
Virat Shejwalkar and Amir Houmansadr. Manipulating the byzantine: Optimizing model poisoning attacks and defenses for federated learning. In NDSS, 2021.
Virat Shejwalkar, Amir Houmansadr, Peter Kairouz, and Daniel Ramage. Back to the drawing board: A critical evaluation of poisoning attacks on production federated learning. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1354-1371. IEEE, 2022.
Guangyu Shen, Yingqi Liu, Guanhong Tao, Shengwei An, Qiuling Xu, Siyuan Cheng, Shiqing Ma, and Xiangyu Zhang. Backdoor scanning for deep neural networks through k-arm optimization. In International Conference on Machine Learning, pp. 9025-55599999 9536. PMLR, 2021.
Ziteng Sun, Peter Kairouz. Ananda Theertha Suresh, and Brendan McMahan. Can you really backdoor federated learning? arXiv preprint arXiv:1911.0700 2019.
Guanhong Tao, Yinggi Liu, Guangyu Shen, Qiuling Xu, Shengwei An, Zhuo Zhang, and Xiangyu Zhang. Model orthogonalization: Class distance hardening neural networks for better security. In 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022.
Vale Tolpegin, Stacey Truex, Mehmet Emre Gursoy, and Ling Liu. Data poisoning attacks against federated learning systems. In European Symposium on Research in Computer Security, pp. 480- 501. Springer, 2020.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representa- lions, 2019. URL httttt:tss/::/eerev///
net/ fo
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 707-723, 2019. doi: 10.1109/SP2219.00033.
Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy- yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos. Attack of the tails: Yes, you really can backdoor federated learning, Advances in Neural Information Processing Systems, 33, 2020a.
Ren Wang, Gaoyuan Zhang, Sijia Liu, Pin- Yu Chen, Jinjun Xiong, and Meng Wang. Practical detection of trojan neural networks: Data-limited and data-free cases. In European Conference on Computer Vision, pp. 222-238. Springer, 2020b.
Chong Xiang, Arjun Nitin Bhagoji, Vikash Sehwag, and Prateek Mittal. (PatchGuard): A provably robust defense against adversarial patches via small receptive fields and masking. In 30th USENIX Security Symposium (USENIX Security 21). pp. 2237-2254, 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 2017.
Chulin Xie, Keli Huang, Pin-Yu Chen, and Bo Li. Dba: Distributed backdoor attacks against feder- ated learning. In International Conference on Learning Representations, 2019.
Chulin Xie, Minghao Chen, Pin- Yu Chen, and Bo Li. Crfl: Certifiably robust federated learning
against backdoor attacks. In International Conference on Machine Learning, pp. 11372-11382.
PMLR, 2021.
Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantine-robuse distributed learning: Towards optimal statistical rates. In International Conference on Machine Learning, pp. 5650-5659. PMLR, 2018
13

---

Published as a conference paper at ICLR 2023
A
APPENDIX
We provide a simple table of contents below for easier navigation of the appendix.
CONTENTS
Appendix
Provides more details on experimental setups, neural network structures, parameters,
etc.
Appendix A.2
Provides extended evaluation on the same setting theoretical analysis.
Appendix
A.3
Provides evaluation on adaptive attacks.
Appendix A.4
Shows AUC-ROC curve of our confidence- sample rejection MNIST.
Appendix A.5
Studies adversarial training can significantly reduce attackers' poisoning confidence.
Appendix A.6
Validates thresholding critical during evaluation.
Appendix A.7
Provides evaluation on other trigger inversion technique.
Appendix
Analyzes the impact of different trigger sizes.
Appendix
A.9
Studies different threshold influences ACC and ASR.
Appendix
Provides additional evaluation results on the existing defenses.
Appendix
Provides justifications for SOTA defenses not working.
Appendix
A. 12
Analyzes the impact of trigger quality.
Appendix
A. 1.
Data Distribution Effects.
Appendix
Provides proofs for our Theorem
Appendix
15
Provides proofs for our Theorem
Appendix
A.16
Glossary of notations.
A.1 EXPERIMENT SETUP
In this section, we illustrate more details about the experimental setups, neural network structures, parameters setups, etc. For more detailed hyperparameter settings and evaluations, please refer to our code repository, we will release our code upon the paper acceptance.
We train the FL system following 1998), MAshion-MNIST (Xiao et al., 2017) and CIFAR-10 (Krizhevsky et al., 2009). MNIST has training set of 60,000 examples, and a test set of 10,000 examples and 10 classes. Fashion-MNIST consists of a training set of 60,000 examples and test set of 10,000 examples. Each example is 28x28 grayscale image, associated with a label from 10 classes. CIFAR-10 is an object recognition dataset with 32x32 colour images in 10 classes. It consists of 60,000 images and is divided into training set (50000 images) and test set (10000 images). We split the training data for FL clients in non-i.id. manner, by Dirichlet distribution (Minka| 2000) with hyperparameter 0.5, following the same setting as (Bagdasaryan et al. 2020 Xie et al. 2019). We train the FL global model until convergence and then apply various trigger inversion defense techniques, otherwise the main task accuracy is low and the backdoored model is hard to converge (Xie et al., 2019). In the augment dataset, use 64 augmented samples in each batch of each local client training, following the existing works of backdoor removal in (Tao et al., 2022). Note that the confidence threshold T of FLIP discussed in Methodology Section 3 only used in continuous backdoor attack setting to filter out low-confidence predictions. Based on our empirical study, we typically set T 0.3 for simpler datasets, e.g. MNIST and Fashion- while t 0.4 for more complex datasets, e.g. CIFAR-10. We apply two convolutional layers and two fully connected layers in MNIST and Bushion-MNISS and Resnet- 18 CIFAR-10 to train our model.
FLIP framework on three datasets: MNIST (LeCun et al.,
FLTrust Setting In FLTrust (Cao et al. 2020), we follow the original settings and collect the root dataset for the learning task with 100 training examples, the root dataset has the same distribution as the overall training data distribution of the learning task. We exclude the sampled root dataset from the clients' local training data, indicating that the root dataset collected independently by the global server.
Class distance The class distance is defined by the trigger size, we use L norm to measure. The intuition here is that if it's easy to generate small trigger from the source class to the target class, the distance between the two classes is small. Otherwise, the class distance is large. Furthermore, can easily generate small
the model is robust when all the class distances are large, otherwise trigger between the two classes. The class distances of models in the wild are very small and do not align well with humans' intuition, for example, classes turtle and bird have smaller distances than
14

---

Published as a conference paper at ICLR 2023
classes cat and dog, which makes the models vulnerable to backdoor attacks (Tao al. 2022 The class distance is independent of the number of local client samples, in other words, the class distance is related to the model's robustness itself instead of the number of samples.
(a)
(b)
(c)
Figure 2: Trigger Examples, (a) is the ground truth trigger, (b) is malicious client poisoned data, (c) after benign client trigger inversion, augmented data
Regarding the attack setting, there are 100 clients in total by default. In each round we randomly select clients, including 4 adversaries and 6 benign clients. We do not have any restrictions on attackers as long as they follow the federated learning communication protocol. The attackers inject the pixel-pattern backdoor in images and swap the label of image source label to target label (by default, label "2"). Figure 2 (b) shows backdoored example. During testing phase, any inputs with such pattern will be classified as the target label. single-shot attack, attackers can choose any round participate. In continuous attack, attackers participate in every round after model convergence. Benign clients perform adversarial training continuously both settings. We report the ACC and ASR after the attack happens least 60 rounds, that is, attackers already achieve high and stable attack success rate.
A.2 EVALUATION ON THE SAME SETTING THEORETICAL ANALYSIS
In this section, we conduct an extend experiment that follows the same setting as our assumptions to validate our theoretical analysis.
Table 4 shows the sample counts of clean samples and backdoored samples under different settings. As illustrated in Corollary [1 the exact value of rejected backdoored samples (Ry/R)) and rejected clean samples ( (RRRR,) can be calculated. The No defense column represents the counts of Rs and Rs without defense; the last column FLIP represents the counts of R2 and Ry with defense, which correspond to the numbers defined in theoretical analysis.
Table 4: Logistic regression sample counts
Attack Type Samples count Total samples No defense FL.P
Clean
10000
8843
8458
Single-Shot
Poisoned
9020
5816
476
Clean
10000
8303
8076
Continuous
Poisoned
9020
5753
442
A.3 RESILIENCE TO ADAPTIVE ATTACKS
As attackers may work out adaptive attacks to get over FLIP, in this section, we design a counter- measure for attackers and evaluate FLIP under the adaptive attack scenario. Detailed adaptive attack consists of the following steps: (1) attackers apply the same trigger inversion technique as benign clients to obtain the inverted triggers; (2) attackers stamp the inverted triggers to their local images and add them to the training phase for backdoor attacks; (3) attackers submit the updated model weights to global server. We conduct experiments on three datasets under continuous attack setting. Table 5 shows the result, observe that even under adaptive attack setting, FLIP can still mitigate the backdoor attacks in both MNIST and adaptive attack is not effective. This indicates that even though the attackers nique during poison training, under the FLIP framework, benign clients can still effectively reduce the attacker's poisoning confidence and keep the attack success rate in low range.
In CIFAR- 10, the accuracy drops and the
aware of our tech-
15

---

Published as a conference paper at ICLR 2023
Table 5: Adaptive attacks evaluation
Continuous attack
ACC
ASR
MNIST
96.82
0.61
Fashion-MNIST
73.42
18.94
CIFAR-10
60.08
14.02
A.4 AREA UNDER THE CURVE (AUC)
In this section, we take MNIST as an example to show AUC-ROC curves (Area Under the Curve Receiver Operating Characteristics), other datasets can be analyzed similarly. Figure shows the AUC-ROC curve of our confidence-basee sample rejection on MNIST. The curve is plotted with TPR (True Positive Rate) against the FPR (False Positive Rate where TPR is on the y-axis and FPR is on the r-axis. In our evaluation, the AUC is 0.97. As stated by many existing works (Man- drekar 2010)) AUC of 0.5 (indicated by the orange dashed line) means the model is unable to discriminate positive and negative samples while an AUC higher than 0.9 is considered outstanding. Therefore our confidence-baset rejection strategy is effective in distinguishing backdoored samples and benign samples.
1.0
True Positive Rate
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate
Figure 3: AUC-ROC Curves
A.5 EFFECT OF ADVERSARIAL. TRAINING
In this section, aim to validate that adversarial training in benign local clients indeed can bring positive effects reducing attackers' poisoning confidence. We conduct the experiments under continuous attacks. We use the same threshold and only remove adversarial training at benign local clients and keep all the other settings the same, e.g. confidence threshold.
Table 6: Effect of adversarial training
Continuous attack
ACC
ASR
MNIST
96.88
51.75
Rishion-MNIST
79.68
98.53
CIFAR-10
72.90
83.13
In T we observe that without adversarial training, malicious can successfully inject backdoor patterns even with a high confidence threshold above т. The underlying reason that adversarial training in benign clients hardens the model against malicious samples and reduces the confidence of malicious samples. Interestingly, we notice MNIST ASR drops compare with no defenses, the reason could be MNIST dataset feature is simpler, thus with a lot of benign clients continuous training parallelly, is easy to forget injected backdoor patterns quickly /Wang et al. 2020a), so that attacker's 's poisoning confidence is reduced and parts samples rejected. Hence, the results show that adversarial training is significantly effective reducing the attacker's confidence in backdoor samples during backdoor training. which is consistent with our theoretical analysis.
16

---

Published as a conference paper at ICLR 2023
A.6 EFFECT OF CONFIDENCE THRESHOLD
In this section, we demonstrate that threshold is a critical component in FLIP and we evaluate our defense with and without thresholding, results can be found in Table 7 We conduct thresholding experiments under continuous backdoor attacks with three datasets. Each benign client performs trigger inversion and adversarial training before, while in global inference-tinee fidence threshold 7 to O, which is no threshold, and keep all the other settings unchanged. We ob- serve that without thresholding applied, though the ASRs are reduced to some extent, they are still much higher, compared with FLJP results in Table 2 The underlying reason is adversarial training does help in reducing the confidence of backdoored samples, however, without applying confidence threshold to reject the backdoored samples, ASR keeps high. We validate that the threshold is critical FLIP and the observation is consistent with our results in Corollary
set the con-
Table 7: Effect of confidence threshold
Continuous attack
ACC
ASR
MNIST
97.20
22.35
Fashion-MNIST
78.76
30.67
CIFAR-10
75.31
52.47
A.7 OTHER TRIGGER INVERSION TECHNIQUES EVALUATION
In general, FLIP is compatible with any trigger inversion technique. In this section, we use another widely-used technique ABS (Liu et al. 2019) as the trigger inversion component of our framework. Specifically, we replace the "Trigger inversion" part in Figure [ with ABS, while keeping all other settings the same. We conduct experiments on both single-shot and continuous attack settings. Note that we only evaluate CIFAR-10, since the released version of ABS focuses on the complex dataset with three color channels instead of greyscale images. In each training round of local clients, we use ABS to invert 10 most likely triggers and perform the adversarial training.
Table 8: Other Trigger Inversion Techniques Evaluation
Single-shot
Continuous
ABS
ACC ASR
ACC
ASR
CIFAR-10
74.14
8.00
74.90
22.38
Table shows the defense technique evaluation result, which is consistent with results shown previ- ously Table and 2 We observe that in continuous attack, FLIP equipped with ABS keeps higher clean accuracy 74% compared to 71% in Table LablePaadd they both reduce ASR to a low level, near 22%. However, in the single-shot attack, FLIP with ABS only reduces ASR 8%. The underlying reason that ABS inverts effective triggers within a small size range, while the method in our main text more aggressive in hardening the model. The result demonstrates that FLIP is generally effective with various downstream trigger inversion techniques against backdoor attacks.
A.8 IMPACT OF TRIGGER SIZE
In this section, we study the different sizes of triggers effect and the evaluation results in Table 9 We define the initial trigger size as X, that is, 2*X denotes the trigger size is scaled up two times compared with the initial trigger. Take MNIST as an example, we observe that the single-shot ASR is low when trigger size (TS) is 1 the reason is each local trigger is too small to be recognized during the global model testing phase. We conduct an experiment consisting of different trigger sizes from x*x, X 4*X, 6xX, to X The evaluation shows that our defense significantly degrade ASR while maintaining comparable benign classification performance, no matter how triggers' sizes change.
17

---

Published as a conference paper at ICLR 2023
Table 9: Trigger Size
No Defense
FLIP
TS
ACC
ASR
ACC
ASR
1 X
97.58
1.48
97.09
0.14
2 X
97.57
94.31
96.94
0.29
4*h X
97.24
96.41
96.05
0.13
6* X
97.33
97.64
97.23
0.76
8 X
97.46
97.85
96.83
0.45
A.9 IMPACTS CONFIDENCE TIRRESTIOLSS
In this section, we show the trade-off between attack success rate and accuracy when we apply the confidence threshold. We conduct an extensive evaluation to study different threshold influences on ACC and ASR. We test our framework on MNIST dataset the continuous attack setting with three different thresholds 0.0, 0.3, and 0.7. We found that with the increase of confidence threshold, ACC is 97.2%, 96.62%, and 88.86 88.86% accordingly, in the meantime, the ASR is 22.35%, 1.93%, and 0.91% accordingly. We observe that benign local model hardening has controllable negative effects on accuracy. Meanwhile, there is a trade-off between adversarial training accuracy and standard accuracy of model (Tsipras al., 2019). If we aim for much lower attack success rate, this will sacrifice part of clean accuracy. In other words, when we set a higher threshold, ASR indeed decreases, in the meantime, some low-confidence benign samples are also rejected, which causes the benign accuracy to reduce to some extent.
A. DISCUSSION ON OTHER DEFENSES
In this section, we provide additional experimental results the comparison between the Multi- KRUM (Blanchard et al. 2017) and our method. We take the CIFAR-10 dataset as an exam- ple, in the single-shot attack, Multi-KRUM can drop ASR from 80.46% to 4.18%, and our de- fense ASR is 7.83%. However, in the continuous attack, Multi-KRUM, can only reduce ASR from 84.73% 61.86%, and our defense ASR 17.27%. the ACC is at a similar level. Our technique can achieve comparative performance with Multi-KRUM in the single-shot attack and outperforms more complex attack scenarios of continuous attack. In addition, we also try to evaluate FLAME (Nguyen et al., (202I) We contacted the authors of FLAME several times for their experiments and parameters setup but got no response until submission.
A.11 JUSTIFICATIONS FOR SOTA DEFENSES NOT WORKING
In this section, we provide concrete justifications on why SOTA defenses produce a nearly 100% attack success rate on continuous attacks setting. Continuous backdoor attacks denote that in each round the attackers will be selected and continuously participate in federated learning. We suspect there are three reasons that SOTA defenses are performing not well on continuous attacks. First, con- tinuous backdoor attacks are more aggressive. In each round of selected participants, 40% of them are attackers and will participate every round of model training. Second, as mentioned in (Wang) et al. (2020a), even under very low attack frequency, the attacker still manages to gradually inject the backdoor as long as federated learning runs for long enough. Third, some of their assumptions, e.g. though FoolsGold (Fung et al., 2020) assumes that benign data are non-iid, meanwhile, it also assumes manipulated data are rid, this could cause FoolsGold to be only effective under certain simpler attack scenarios, e.g. single-shot attacks.
A.12 IMPACT OF TRIGGER QUALITY
In this section, we study the different quality of triggers effect and the evaluation results Table 10 We evaluate CIFAR-10 10 dataset on single-shot attacks. We inject triggers with random shapes in both white and colorful. We observe that the randomly chosen triggers do not have much influence on attackers and cannot reduce the attack success rate. However, when we the ground truth triggers, they do achieve the best performance both reducing ASR on backdoor tasks and maintaining ACC on benign tasks. The evaluation shows that random triggers cannot reduce ASR and FLIP inverted
18

---

Published as a conference paper at ICLR 2023
trigger can achieve comparable performance with ground truth trigger and can be further improved in the future work.
Table 10: Trigger Quality
CIFAR-10
Single-shot Attack
ACC
ASR
No Defense
77.52
80.46
Random white trigger
77.89
81.72
Random colorful trigger
78.65
80.99
FLIP
73.41
7.83
Ground truth trigger
76.98
2.64
A.13 DATA DISTRIBUTION EFFECTS
In this section, we leverage Dirichlet distribution (Minka 2000) with hyperparameter a to model different non-ii.d. distributions. By increasing the hyperparameter in the Dirichlet distribution, we can simulate from non-i.id to i.dd distributions for the datasets (Xie et al., 2019). Here we evaluate MNIST single-shot attacks without defense and with FLIP. We conduct experiment consisting of different a from 0.2, 0.4, 0.6, 0.8, 1.0, to 2.0. The evaluation demonstrates that without defense applied, backdoor attack performance affected by non-i.i.d degree. However, defense can still cause a significant ASR degradation in different non-i.i.d degrees and only has slightly differences, while maintaining comparable benign classification performance.
Table 11: Data Distribution Effects
No Defense
FLIP
Degree
ACC
ASR
ACC
ASR
0.20
97.72
75.97
97.56
0.34
0.40
97.47
80.36
97.64
0.41
0.60
97.01
80.06
97.64
0.60
0.80
97.13
81.90
97.49
1.39
1.00
97.80
81.19
97.53
1,86
2.00
97.82
93.25
97.31
1.96
A. PROOF OF BOUNDS ON LOSS CHANGES
In this section, we will present the bound on loss changes, formulate the benign local clients training and global model aggregation process, and then provide the detailed proofs for Theorem that are related to loss changes bound. Note that, we list all the notations used in the paper in Table 12
Generalize Proof to complex model architectures. Given that existing works (Xie al. 202 Li et al. 2021 Nguyen et al. 2021) all focus on linear models as the complexity in real-world models makes theoretical analysis tasks infeasible. Note that besides the theoretical results, empirically, we extended to non-linear and showed in Section 5 that our defense gives outstanding performance against state-of-the-arr backdoor attacks, consistent with our theoretical analysis.
Throughout this paper, "clean training" refers to benign local clients training with clean data; "ad- versarial training" refers benign local clients apply trigger inversion techniques to get reversed trigger, then stamp the trigger to their local clean image and assign with ground truth clean label to get the augmented dataset, then train with the augmented dataset.
In benign clients, we train with defense technique to generate trigger, then do adversarial training and submit gradients to global server. Given model parameter W of one linear layer, k-th device holds the ng training data Xt then denoted the loss L(W; ). Let Y {0, 1}, denote one-hot vector of local samples. For x, we denote xW as the output the linear layer, px(x) = so poflmax((xl) (xW b); the normalized probability for class (the output of the softmax function).
19

---

Published as a conference paper at ICLR 2023
b or bias is omitted in following equations for simplicity, but it would still work if added. For one example the cross-entropp loss is calculated as:
((x) = Σ Yxlogp.(x)
(4)
=
Σ (olog(softmaz(xX))
(5)
We define as the gradient for one sample:
G(x) VI(W x,,) = dC (x) x² (p(x) - Y)
(6)
dw
Similarly, when defense technique get reversed trigger and stamp it on clean image, then we get the augmented dataset, denote is Xaug. then the gradient augmented dataset G' can be written
G' ('xuu)) VI(W; Xaay: y) X (p((gup))))) Y))
Here, we describe one around (say the r-th) of the standard FedAng algorithm. When the benign device in k-th receive the global weights W,, and then performs (= local updates (lets W W,), in benign clients, we training both clean dataset and augmented dataset:
Wh r+ + W Th F (W c+)
W nr
T.k (p(x)) - -Yy) +
Σ
xx
augg (p( x (p(x)ug.)) Yj)I]
j=1
j=1
(8)
+- W
bx T.k (p(x5) -Yy)1 - 7r
X aug. (p( (p(xang.)) YjM1
5=1
where is the learning rate (a.k.a. step size), ny is the number of samples in k-th client.
In global server, define as the malicious clients generated trigger, å + e as the benign clients generated trigger, then represent backdoored sample as (x + 6) and augmented sample as (x + + e). Benign clients updates can be written as:
Th
W r+1
W
T.k (p(x))) (
T Σ Ixx +8 +8+c)74(((x) +8+c)* Y11
Trr
In the threat model, we consider the practical oblivious but honest attack setting that defender has no control on malicious clients and they can perform any kinds of attack, as long as attackers follow the federated learning protocol. Our proof focuses on the two-client setting, one benign and the other malicious. Thus, we represent the malicious clients updates as Was-
After each local finished their training, they submit their model updates to global. Then global aggregation step performs
W W++ 7r 2 9k W
(10)
Ik is the weight of the k-th device. In order to simplify, here we take 9k and assume only have two clients (N=2), = is benign client, n, denotes the number of samples in this benign client. Then the aggregated global weight are the each local weights aggregate together. Then the
20

---

Published as a conference paper at ICLR 2023
aggregated global weight can be written as
N
W
W
r+1
k=1
W
+ W r+1
1/r
x
T.k (o(x))) - 7r Σ
+ (x (p(x, +8+6) -Y,) (11)
j=1
==1
2W, WM
1/ Σ xx f
(x) +8+ - YJ)
5=1
2W, WM
When we consider the without defense setting, ô + not exists, in round t + 1, W4+1 the global weight without local weights can be written as
W,+1 -r Σ x (xt)) YJ)1 + 2W, WM
(12)
When we consider the with defense setting, + e exists, round t 1, W441 the global weight without local weights can be written
n1
11 [x) + 8 ()))) +8
W ++ -1, Σ 2 A (o(z)}
17)1 Thr
+ +d+
Yf)
(13)
+ 2W, Was
The difference between with defense and without defense training is exactly how much adversarial training benign will influence other clients, it can be written as
W ++ W The > (x, + 8 +)+ (p(x, + 1=1
(14)
Given model parameter W of one linear layer, the k-th device holds ng training data (xk.j- Dk.j 1==1- j=1- We denote the loss L(W; {xx.ji Nr ))=)) Denote xW the output of the linear layer, P. softma((xW + b), the normalized probability for class (the output of the softmax function). We omit b (bias) in the following thcoretical analysis for simplicity. Adding the bias term to our analysis straightforward.
Global softmax cross-entropy loss function can be written as:
Lgiobal
glog(ps)
(xlgs)
e(xW),
gilog(
(15)
Σ1 -1=1 e(xW).
qx(W)); ; logg
c(xW)))
Σ (xWW) + log c(xW)))
1=1
21

---

Published as a conference paper at ICLR 2023
Since want to compare the loss changes in two different cases (e.g. with defense and without defense setting), to observe if the dedution of the loss increase or decrease, here we let the two losses (say Cg is with defense. bg is without defense) deduct each other:
6g bg
Σ (xx""); + log Σ J
2i (xW);
log
eloW)e)
Σ ai((W" - W)x]s + log(
e (( logg
Σ e(xW3))
(16)
1= c(xeW")
g:((W- - W)x|; + log(
Ei=1 e(xW),
Since both E/ (=1 e e(xW") >
and El= e(xW), are sequence of positive numbers. Then from
Mitrinovic & Vasié 1970) we can have an inequality of
e(xW"),
r= e(xw"),
e(xW"))
min S e(xW),
I max Isrst e(xW)r
(17)
e(xW))
Proof. If we denote m min, c(wW),
clowt) c(xW). and M max, e(xws. then we have successively
e(xW"),
5 e(xW),
S M
(18)
m. e (xW), S (xW"),
S M (xW),,
(19)
(xW))
(xW)),
M
(xW)s
(20)
t=l
-=| c(xW"))
m <
S M
(2))
E1= e(xW),,
Then we can get
clxW),
Ei= El=r c(xW"). c(XW),
e(xW"):
min SIST e(xW),
max IS1ST e(xW),
(22)
By using log function monotonicity property, we can get an inequality of
El=
El= e(xW"); El c(xW),,
logm S log(
S logM
(23)
So the deduction of C, Lg can be written as:
1
logm- Σ axxx(W'- - -W), s sC, - Cg < logg
Σ
9. [x(''W'-- [W]WWWW
(24)
c(xW"),
1 Σ alx - W)), s C C, < log max c(xW"),
Σ [x((W' - W)]
log min
e(xW),
e(xW),
(25)
22

---

Published as a conference paper at ICLR 2023
Denote the left hand side of above formula A min .loss, denote the inequality's 's right hand side value as A max _loss.
e(xW"),
A min losss log min
a[x(W'- w)]
e!xW),
exxW");
min log
9xx((W' - w)l
e(xW)),
(26)
min loge ((''
9 Olx(WW' - W))
min [nin[x(W"' - -W),+
Σ [x(W -W);
Then we can get the lower bound and upper bound of C. Co
min[x(W' - W))e Σ d: (elx(W"-W)) s C. L.g max[x(('
S - W)] Σ qi [x((WW' W)];
(27)
Let f denote the global model loss with defense, Cg as without defense, let AW W W denote the weight differences with and without defense. The loss difference with and without defense can be upper and lower bounded by (as shown in Theorem] I
min(xAW); Σ (xxxW), S - C.
- S maxxxxxxxxx - Σ ((AAW),
(28)
To facilitate the analysis, we denote the upper bound as A max loss and the lower bound as A min Joss. To efficiently reduce the attack success rate and maintain the clean accuracy, studied this lower bound on backdoor data, which indicates the minimal improvements on the back- door defense. Similarly we studied the upper bound for clean data, as they indicates the worst-case accuracy degradation.
Denote the number of backdoor samples as п, and the number of benign samples as ne- Note back- door samples are written as + 8. By using Theorem | we have A min _loss = Es min:((x, + A)AW]; Σ El= (x,1(x, 5)AW]; And similarly on benign data, we have A max loss A max loss 2 mxx -Es, AW)
A.15 PROOF OF GENERAL ROBUSTNESS CONDITION
In this section, we will present general condition of robustness on trigger generation, formulate Amin.loss on backdoored data and Amor_loss on clean data, and then provide the detailed proofs for our Theorem 2 that are related to general robustness condition.
Our intuition is that we want the loss to increase more on backdoored data, and increase less on clean data. This means after applying defense, the global server loss in backdoored data will increase and the loss in clean data will change within a constant range. Accordingly, when evaluate on ni backdoored data, we want the lower bound at least greater than 0, AminJoos 2 0. When evaluate on n. clean data, we want the upper bound Amar S Ç. here Ç is a constant. In evaluation, denote global server has n, backdoored data and clean data for testing.
When evaluating on ny backdoored data
C Lg 2
min [x, + 5)(W" - W)) Σ 1=1 1=1
(x,1(x, + +WW - -W],,
(29)
min loss 2 0
23

---

Published as a conference paper at ICLR 2023
When evaluating ne clean data
Re
I
Cg Lg max X (Wx - W)]
2s Ux Xx (. - W)]
(30)
i=1
A max loss 5
Since previous results we know W Wr11 can be represented as
W Wo+1
(x, + + +6+ Yj)l
(31)
Th Σ (x, + -px; + +
We denote g. as a one-hot vector for sample & with dimensions. Its i-th dimension is defined as 2o and O2 if arg min/((x, + 6)(W" - W)]e
na
A min loss
min (x, + 6)(W' - -W), Σ
gom1(x, + +WW(W' -W]]s
n2
95 (x, + +8) >
[(x, + +0++)=(( -p0ji-p(x--- + +))ll
!/r
j=l
nn
ge (x, + +6) >
[xx, +8+ p(xx + + +))]l
7/r
(95, - 9x,il(x + 8)n, > (x, 8 +) - p(x, + +))ll
7/r
(32)
Let 2, xx 5 and 2j x, + e, then the A min loss
n4
A min loss Σ
(g. (n.) -
Σ 1 -
(33)
Let
f(c) A min Loss
n1
1
(34)
1/ Σ
Σ
- (.))(EE
- Σ
|2] - p(z;))1}.
1=
5=1
Compute the gradient of f(c), we have
ff(c)
1hb
n1
7r
(o
lef >
p(2,) -(,-)
(35)
Ve
Let ||e/bo I and that f(e) is a linear function, we know the minimal value of f(e) is achieved when
Th
n1
Ck sign
?r
(9
2o.i)
12 -00p)()
(36)
j=1
For simplicity, denote by as
I
by = sign
(g {x.x)
5 12 (Y, - p(2)))ll
(37)
f=1
1=1
24

---

Published as a conference paper at ICLR 2023
and the vector as b [bi,...; bal- The minimal condition is thus ab. Replace e ab into eq. 34p. and in order to be consistent with previous section main text Theorem we use q to replace Y,, we have
nh Σ (o ga,) -ab
[(e) 2 T
Σ jml
2j (qj - p(2)))lb
(38)
The sufficient condition of f(c) 2 is thus
n1 (s,s ab)) Σ
7/r
Σ
Lo.
12g [ p(z))))),
(39)
f=1
Th
n1 C3.s (s,1)1
ab frr
23j" (qj p(2)))]}+}
8=1
j=1
(40)
//r
(ga.s
90.i) { 2
22j (qi p(2;))));
Note that for any vector x, we have sign (x)x 2 0. And we can divide the right hand side by the left hand side and finish the prove.
a Tr E. (9,, 90s) (Z 2j (qj - p(2)))l
(41)
bi E Ele
(g le ge bzj ( p(2)))]b}
Each term in above can be computed, then always find small enough error range where surely improve the loss function.
Similarly, in upper bound of A max _loss, denote 1hd one-hot vector, denote of hot vector for sample with I dimensions. Its i-th dimension is defined as L. and gi = if
i = argmar,, (W"" W)], let
g(c) A max loss
The
(42)
fr
2j
(qj p(2)))
Note that g(c) is nothing but constant with respect to c. This means that the upper bound loss is up some constant with respect the recovered trigger 2y.
Note that A min loss 2 0 indicates that the defense is provably effective than without defense. Since benign local clients training can increase global model backdoor loss, and they have pos- itive effects on mitigating malicious poisoning effect. The second condition A max loss <
7r E'= Eln(E, 9x.i)x, E [zj (qj p(zz)),, indicates that the defense is provably guar- antee maintaining similar accuracy clean data.
25

---

Published as a conference paper at ICLR 2023
A.16 GLOSSARY OF NOTATIONS
Table 12: Glossary of notations
Notation
Description
Xk.j+ Mle
the k-th client device j-th data sample and its label s-th sample i-th dimension
q8,
Mr
the learning rate (a.k.a. step size)
W
the k-th client device in r-th round weights
W
local model weights without defense local model weights with defense confidence threshold
W
R
the number of rejected backdoor samples without defense the number of rejected backdoor samples with defense
R
Rid = Ri Rs.
the number of backdoored samples that are rejected after defense applied the number of rejected benign samples without defense the number of rejected benign samples with defense the number of benign samples that are rejected after defense applied the ground truth trigger
Rs
R.
Rin R²a Re
the various trigger inversion technique recovered trigger the difference between the reversed trigger and the ground truth trigger denotes the benign sample stamped with the recovered trigger the global model loss without defense the global model loss with defense
2= + +
Lg
C
26


Reviews:
[{"review_id": "wiJYzKBjIaQ", "reviewer": "Reviewer_cMnB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In FL many clients participate. Malign/adversary clients can potentially join and impact training by submitting malicious weights. To cope with such scenarios benign clients can employ algorithms that are more robust for such injections. The authors propose the use of data augmentation through adversarial training that benign clients should use. \nThe also provide a theoretical analysis of the difference in the loss function if benign clients employ such as strategy vs not. \nOn standard datasets they find that their algorithm usually outperforms benchmark algorithms. ", "summary_of_the_review": "The contributions are significant. The paper is a nice mixture of algorithmic designs and theoretical analyses. \n\nThe literature review section can be improved by more explicitly stating how does prior work differ from the current work. ", "strength_and_weaknesses": "Strengths: \nThe underlying FLIP algorithm is rather novel. It combines existing concepts but in an interesting way. \nThe theoretical analysis is also interesting. The statements state results that are definitely relevant and intriguing to show. \n\nWeaknesses: \nPerhaps FLIP can be enhanced to improve its performance. It would be great if it beats other benchmarks in more cases. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "UcDUbCStrA", "reviewer": "Reviewer_zr2z", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a defense against backdoor attacks to federated learning. Existing defenses do not study how hardening benign clients can affect the global model (and the malicious clients).  The paper proposes a three stage defense that aims to harden benign clients and connect cross-entropy loss, attack success rate, and clean accuracy.  The proposed defense is evaluated on three datasets and shows better performance than previous defenses. \n\n\n", "summary_of_the_review": "The studied problem is important. The proposed defense is somewhat novelty. However, the proposed defense is not well justified, and the theoretical results may be inaccurate and cannot be generalized to general deep nonlinear models. ", "strength_and_weaknesses": "Strengths\n+ The studied problem is important\n+ The paper proposes a three stage defense against backdoor attacks to federated learning\n\n\nWeaknesses\n-Theoretical results are inaccurate\n-Theoretical results do not provide meaningful guidance about why the propose defense works\n-Evaluation is not sufficient \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "NH67ZGKbwgc", "reviewer": "Reviewer_i7z3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Federated learning is vulnerable to backdoor attacks. The authors propose a new defense against backdoor attacks to federated learning. The key idea is to combine some techniques developed to defend against backdoor attacks to centralized learning. Some theoretical analysis is performed to analyze the loss and learnt parameters. Evaluation is extensive, covering both existing and adaptive attacks. ", "summary_of_the_review": "This paper extends centralized learning backdoor defense to federated learning, which is an interesting direction. Extensive evaluation is performed to show the effectiveness of the method. ", "strength_and_weaknesses": "Strength\n\n+ This paper proposes a new federated learning method to defend against backdoor attacks. Instead of designing new aggregation rules or changing how the server aggregates local model updates, the method modifies how benign clients train their local models. \n\n+ The paper extends centralized learning backdoor defense methods to federated learning. This is an interesting direction to explore. \n\n+ Some theoretical analysis is performed. \n\n+ Evaluation is extensive, covering multiple datasets, multiple baselines, and adaptive attacks. \n\nWeakness\n\n- One potential weakness is that the meaning of provable is slightly different from what provable defense usually means. My first impression about provable defense is that it provides certified accuracy, like what ensemble federated learning provides. But this is a minor clarification issue. \n\nOther minor comment\n\nCan you describe the setting for FLTrust? In particular, what root dataset is used by the server in FLTrust? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HMimsgHsMt", "reviewer": "Reviewer_cwn7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper propose a trigger reverse engineering based defense and show that the method can achieve robustness improvement with guarantee (i.e., reducing the attack success rate) without affecting benign accuracy. The paper conducts comprehensive experiments across different datasets and attack settings. The results on eight competing SOTA defense methods show the empirical superiority of the  method on both single-shot and continuous FL backdoor attacks. ", "summary_of_the_review": "See the above. ", "strength_and_weaknesses": "1. The paper presents a new provable defense framework that can provide a sufficient condition on the quality of trigger recovery such that the proposed defense is provably effective in mitigating backdoor attacks.\n2. The paper empirically evaluate the effectiveness of FLIP at scale across MNIST, Fashion-MNIST and CIFAR-10, using non-linear neural networks. The results show that FLIP significantly outperforms SOTAs on the continuous FL backdoor attack setting. \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster