FLGAME: A Game-theoretic Defense against Backdoor Attacks In Federated Learning
Authors: Jinyuan Jia
Zhuowen Yuan
Dinuka Sahabandu
Luyao Niu
Arezoo Rajabi
Bhaskar Ramasubramanian
Bo Li
Radha Poovendran
Venue: Submitted to ICLR 2023 (2023)

Abstract: Federated learning enables the distributed training paradigm, where multiple local clients jointly train a global model without needing to share their local training data. However, recent studies have shown that federated learning provides an additional surface for backdoor attacks. For instance, an attacker can compromise a subset of clients and thus corrupt the global model to incorrectly predict an attacker-chosen target class given any input embedded with the backdoor trigger. Existing defenses for federated learning against backdoor attacks usually detect and exclude the corrupted information from the compromised clients based on a $\textit{static}$ attacker model. Such defenses, however, are less effective when faced with $\textit{dynamic}$ attackers who can strategically adapt their attack strategies. In this work, we model the strategic interaction between the (global) defender and attacker as a minimax game. Based on the analysis of our model, we design an interactive defense mechanism that we call FLGAME. Theoretically, we prove that under mild assumptions, the global model trained with FLGAME under backdoor attacks is close to that trained without attacks. Empirically, we perform extensive evaluations on benchmark datasets and compare FLGAME with multiple state-of-the-art baselines. Our experimental results show that FLGAME can effectively defend against strategic attackers and achieves significantly higher robustness than baselines. 

Under review as a conference paper at ICLR 2023
FLGAME: A GAME-THEORETIC DEFENSE AGAINST BACKDOOR ATTACKS IN FEDERATED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Federated learning enables the distributed training paradigm, where multiple local clients jointly train a global model without needing to share their local training data. However, recent studies have shown that federated learning provides an additional surface for backdoor attacks. For instance, an attacker can compromise a subset of clients and thus corrupt the global model to incorrectly predict an attacker-chosen target class given any input embedded with the backdoor trigger. Existing defenses for federated learning against backdoor attacks usually detect and exclude the corrupted information from the compromised clients based on static attacker model. Such defenses, however, less effective when faced with dynamic attackers who can strategically adapt their attack strategies. In this work, we model the strategic interaction between the (global) defender and attacker as minimax game. Based on the analysis of our model, we design an interactive defense mechanism that we call FLGAME. Theoretically, we prove that under mild assumptions, the global model trained with FLGAME under backdoor attacks is close to that trained without attacks. Empirically, we perform extensive evaluations on benchmark datasets and compare FLGAME with multiple state-of- the-art baselines. Our experimental results show that FLGAME can effectively defend against strategic attackers and achieves significantly higher robustness than baselines.
INTRODUCTION
Federated learning (FI.) (McMahan et al., 2017a) aims to train machine learning models (called global models) over training data that is distributed across multiple clients (e.g.. mobile phones, IoT devices). FL has been widely used in many real-world applications such as finance (Long et al.) 2020) and healthcare (Long et al., 2022). FL trains a global model in iterative manner. In each communication round, a cloud server shares its global model with selected clients; each selected client uses the global model to initialize its local model, then utilizes its local training dataset train the local model, and finally sends the local model update to the server; the server uses an aggregation rule aggregate local model updates from clients to update its global model.
Due to the distributed nature of FL., many recent studies (Bhagoji et al. 2019 Bagdasaryan et al. 2020 Baruch et al. 2019 Wang al., 20200] Kairouz et al. [202]] have shown that it is vulnerable to backdoor attacks. For instance, an attacker can compromise a subset of clients and manipulate their local training datasets to corrupt the global model such that it predicts an attacker-chosen target class for any inputs embedded with a backdoor trigger (Bagdasaryan et al. 2020).. To defend against backdoor attacks, many defenses (Sun et al. 2019 Cao al., 2021a) have been proposed. For example, (Snn et al. (019) proposed to clip the norm of the local model update from each client such that its Ln-norm was no larger than a defender-chosen threshold. [Co et al. (2021a) proposed FL.Trsst in which server computes a local model update itself and computes its similarity with that of a client as the trust score, which is leveraged when updating the global model. However, all of those defenses consider static attack model where attacker does not adapt its attack strategies. As result, they are less effective under adaptive attacks, e.g., Wang et al.] (2020) showed that the defenses proposed in (Sun et al. 2019 Blanchard et al. 2017) can be bypassed by appropriately designed attacks.
Our contribution: In this work, we propose FLGAME, a game-theoretic defense against backdoor attacks to FL. Specifically, formulate FI.GAME minimax game between the server (defender)

---

Under review as a conference paper at ICLR 2023
and attacker, which enables them to strategically adapt their defense and attack strategies. In the rest of the paper, we use the terms benign client to denote a dalid/un-compronn
client and genuine score to quantify the extent to which client is benign. Our key idea is that the server can compute genuine score for each client whose value is large (or small) if the client is benign (or compromised) in each communication round. The genuine score serves
weight for the local model update of the client when used to update the global model. The goal of the defender is to minimize the genuine scores for compromised clients and maximize them for benign ones. To solve the resulting minimax game for the defender, we follow three-step process consisting of 1) building an auxiliary global model, 2) exploiting to reverse engineer backdoor trigger and target class, and 3) inspecting whether the local model of client will predict input embedded with the reverse engineered backdoor trigger as the target class to compute genuine score for the client. Based on the deployed defense, the goal of the attacker is to optimize its attack strategy by maximizing the effectiveness of the backdoor attack. Our key observation is that the attack effectiveness is determined by two factors: genuine score and the local model of the client. We optimize the attack strategy with respect to those two factors to maximize the effectiveness of backdoor attacks against our defense.
We perform both theoretical analysis and empirical evaluations for FLGAME. Theoretically, prove that the global model trained with our defense under backdoor attacks is close to that trained without attacks (measured by L2-norm of global model parameters difference). Empirically, evaluate FLGGAME on benchmark datasets to demonstrate its effectiveness under state-of-the-ar backdoor attacks. Moreover, we compare with state-of-the-art baselines. Our results indicate that FLGAME outperforms them by significant margin. Our key contributions can be summarized as follows:
We propose game-theoretic defense FLGAME. We formulate FLGAME as a minimax game between the defender and attacker, which enables them to strategically optimize their defense and attack strategies.
We theoretically analyze the robustness of FLGAME. In particular, show that the global model trained with FLGAME under backdoor attacks is close to that without attacks.
We perform systematic evaluation of FLGAME on benchmark datasets and demonstrate that FLGAME significantly outperforms ofate-of-the-ar baselines.
2 RELATED WORK
Backdoor attacks on federated learning: In backdoor attacks to FL (Bhagoji et al. 2019 Bag- dasaryan et al., [2020] Baruch et al., 2019 Wang et al., [0200 Zhang et al. 2022b). an attacker aims to make global model predict a target class for any input embedded with a backdoor trigger via compromised clients. For instance, [Bagdasaryan et al. (2020) proposed scaling attack which an attacker uses mix of backdoored and clean training examples to train its local model and then scales the local model update by a factor before sending it to the server. Xie et al. (2019) proposed distributed backdoor attack FL. Roughly speaking, the idea is to decompose backdoor trigger into different sub-triggers and embed each of them to the local training dataset of different compromised clients. In our work, we will leverage those attacks to perform strategic backdoor attacks to our defense.
Defenses for Federated learning against backdoor attacks: Many defenses (Sun al., 2019 Cao
et al. 2021a| Ozdayi et al. 202 Wu et al., 2020; Rieger et al.) 2022: Nguyen et al. 2022) were proposed to mitigate backdoor attacks to FL. For instance. [Sun al. (2019) proposed norm- plippigg which clips the norm of the local model update of a client such that its norm is no larger than a threshold, They also extended differential privacy (Dwork et al., 2006 Abadi et al. 2016 McMahan et al. 2017b) to mitigate backdoor attacks to federated learning. The idea is to clip the local model update and add Gaussian noise to it. Cao et al. (2021a) proposed FLTrust which leveraged the similarity of the local model update of client with that computed by the server itself on its clean dataset. Other defenses include Byzantine-robut
FL methods such as Krum (Blanchard et al. 2017). Trimmed Mean (Yin et al., 2018), and Median Yin et al. 2018). However, all of those defenses consider a static attacker model. As a result, they become less effective against dynamic attackers who strategically adapt their attack strategies.
2

---

Under review as a conference paper at ICLR 2023
Another line of research focuses detecting malicious clients (Li et al. 2020a Zhang al. 2022a). For instance, [i et al. (220a) proposed to train a variational autoencoder (VAE) and use its reconstruction loss on the local model update of client to detect malicious clients. However, those defenses need to collect many local model updates from a client to make confident detection. As a result, the global model may already be backdoored before those clients are detected. Two recent studies (Cao et al., 2021b, Xie et al., (021)) proposed certified defenses against compromised clients. However, they can only tolerate moderate fraction of malicious clients (e.g., less than 10%) as shown in their experimental results.
BACKGROUND ON FEDERATED LEARNING AND THREAT MODEL
3.1 FEDERATED LEARNING
Suppose S is a set of clients. We use D; to denote the local training dataset of the client i E S. In the 4th communication round, the server first sends the current global model (denoted as Qt) each client. Then, each client i trains a local model (denoted as O) by finetuning the global model Of using its local training dataset D.. For simplicity, we use z = (x, y) to denote a training example in D,, where x is the training input (e.g., an image) and y is its ground truth label. Given D, and the global model e², we denote a loss function [(D;; = [D: EXED, E(z; where E is a loss function (e.g., cross-entropy loss). The client can use gradient descent to update its local model based on the global model and its local training dataset, i.c., O = O m7 86 learning rate of the local model. Note that the client can also use stochastic gradient descent to update its local model on its local training dataset as illustrated in our experiments. Then, the client sends of O + (called local model update) to the server. Note that it is equivalent for the client to send a local model or local model update to the server as O = O + g. After receiving the local model updates from all clients, the server can aggregate them based on aggregation rule R (e.g.. FedAvg) to update its glocal model, i.c., we have:
aC) where nn is the
e = O + (yRR ... .'s))
where SS] represents the number of clients and 77 is the learning rate of the global model.
3.2 THREAT MODEL
We consider the backdoor attack proposed in previous work (Bagdasaryan et al. 2020; Xie et al. 2019). In particular, we assume an attacker can compromise a set of clients (denoted as S,). To perform the backdoor attack, the attacker first selects backdoor trigger å and target class y² For each client E S2 in the ith (t- = 1,2,...) communication round, the attacker can choose an arbitrary fraction (denoted as r!) of training examples from the local training dataset of the client, embed the backdoor trigger 8 to those training inputs, and relabel them as the target class y² Those backdoored training examples are used to augment the local training dataset of the client. In our game-theoretic framework, we will optimize r for the compromised client i in each communication round to make the backdoor attack more effective under our defense.
We consider that the server itself has a small clean training dataset (denoted as D.), which could be collected from the same or different domains of the local training datasets of clients. Moreover, we consider the case that the server does not have any information on each client except their local model updates in each communication round.
4 FLGAME: A GAME-THEORETE DEFENSE AGAINST BACKDOORS
Overview: Our idea is to formulate FLGAME as a minimax game between the defender and attacker, solving which enables them to respectively optimize their strategies. In particular, the defender computes genuine score for each client in each communication round. The goal of the defender is to maximize the genuine score for a benign client and minimize it for a compromised one. Given the genuine score for each client, we use weighted average over all the local model updates to update the global model, i.e., we have
@t11 = + "ies
piih..
(2)
P ies
3

---

Under review as a conference paper at ICLR 2023
where p is the genuine score for client in the tth communication round and 1 is the learning rate of the global model. The goal of the attacker is to maximize its attack effectiveness, which is determined by two components based Equation 2 genuine scores and local models of compromised clients. In our framework, the attacker will optimize the tradeoff hetween those two components to maximize the effectiveness of its backdoor attacks against defense.
4.1 FORGULATING FLGAME AS A MINIMAX GAME
Computing the genuine score for client i: To compute pf. our key observation is that the local model of compromised client is more likely to predict the target class for a trigger-embedded input compared with that of benign client. However, the key challenge is that the server does not know the backdoor trigger and target class adopted by the attacker. To overcome the challenge, the server can reverse engineer backdoor trigger ône and target class v/c (we will discuss more details in the next subsection). Recall that the client sends its local model update of to the server, the local model of the client can be computed O O' + t.g. Then, we can compute pf as follows:
p
1
E I(G(x @ oici 01) vte))
[Del
xED.
where TI is an indicator function, D. is the clean training dataset of the server, x < 6ye is trigger- embedded input, and Gix @ irei @+) represents the predicted label of the local model O for o 6re- Roughly speaking, the genuine score for a client is small if its local model predicts a large fraction of inputs embedded with the reverse engineered backdoor trigger as the target class.
The optimization problem for the defender: The server aims to reverse engineer the backdoor trigger 8,e and target class yfe such that the genuine scores for compromised clients minimized while those for benign clients are maximized. Formally, we have the following optimization problem:
Σ pS Sre.tic JES. JESYS.
min Σni rt. -
The optimization problem for the attacker: The goal of attacker is to maximize its attack effectiveness. Based on Equation [] the attacker needs to: 1) maximize the genuine scores for compromised clients while minimizing them for benign ones, i.e., max P -jEsss. 2) make the local models of compromised clients predict an input embedded with the attacker-chosen backdoor trigger as the target class y²S. To perform the backdoor attack in the /th communication round, the attacker embeds the backdoor a certain (denoted as r/) fraction of training examples in the local training dataset of the client and uses them to augment it. A larger r is more likely to make the local model of the client predict a trigger-embedded input as the target class but also make its genuine score smaller. Therefore, r measures a tradeoff between them. Formally, the attacker can find the desired tradeoff by solving the following optimization problem:
sss pÝ). and
max R
P
Σ
Pj
(5)
jES
iES.
where R (rili E Sa} and is a hyperparameter to balance the two terms.
Minimax game between the defender and the attacker: Given the optimization problems solved by the defender and attacker, we have the following minimax game:
min max Or
Σ pl
Σ JESIS.
+ n
(6)
EES.
SES.
Note that rf (i E Sa) is chosen by the attacker and thus we can add rt the objective function in Equation 4 without influencing its solution given the local model updates of clients.
4.2 SOLVING THE MINIMAX GAME BY THE DEFENDER
To solve the minimax game in Equation for the defender, our idea is to construct an auxiliary global model and then reverse engineer the backdoor trigger and target class based on it.
4

---

Under review as a conference paper at ICLR 2023
Constructing auxiliary global model: Suppose of is the local model update from each client i E S. Our auxiliary global model is constructed as follows: O O + [s| Liss 8 Our intuition is that such aggregated global model is very likely predict an input embedded with the backdoor trigger as the target class y" under backdoor attacks.
Reverse engineering the backdoor trigger and target class: Given the auxiliary global model, we can use arbitrary methods to reverse engineer the backdoor trigger and target class. Roughly speaking, the goal is to find the backdoor trigger and target class such that the genuine scores for benign clients are large but they are small for compromised clients. For instance, we can leverage Neural Cleanse (Wang al. 2019), which the state-of-the-ar method to reverse engineer backdoor trigger and target class. Roughly speaking, Neural Cleanse views each class (c = 1,2,-.. ... C and C' is the total number of classes in the classification task) as potential target class and finds perturbation 8, with small Lı such that any inputs embedded with it will be classified as the class c. We view the trigger with the smallest -n-mm as the backdoor trigger and view the corresponding class as the target class. Formally, we have v/f arg min, [|šell and 6fe Sytc-
The complete algorithm of our FLGAME is shown in Algorithm iin Appendix.
4.3 SOLVING THE MINIMAX GAME BY THE ATTACKER
The goal of the attacker is to find r for each client E S. such that the loss function in Equation 6 is maximized. As the attacker does not know the genuine scores of benign clients, the attacker can find r to maximize p Ar! for client E S, approximately solve the optimization problem in Equation 6 However, the key challenge is that the attacker does not know the reverse engineered backdoor trigger ore and the target class y/c of the defender to compute the genuine score for client i, In response, the attacker can use the backdoor trigger and target class y chosen by itself. Moreover, the attacker reserves certain fraction (e.g.. 10%) of training data from its local training dataset D, as the validation dataset (denoted as " to find the best ri.
Estimating a genuine score for a given rf: For a given r². the client i can embed the backdoor to rf fraction of training examples in D; Drew and then use those backdoored training examples to augment D; Drev to train a local model (denoted as θ(). Then, the genuine score can be estimated ExeDge [xG @ 8; θt) y²*), where G(x @ 8; θ!) is the predicted label by the global model θ for the trigger-embedder input x 8.
as 5 = 1
Searching an optimal r/: The client can use grid search to find rf that achieves the largest o + Art. After estimating the optimal rf. client can embed the backdoor to rf fraction of training examples to augment the local training dataset, train local model, and send the local model update to the server.
The complete algorithm for each compromised client shown in Algorithm 2 in Appendix.
THEORETICAL ANALYSIS OF FLGAME
This section provides theoretical analysis of FLGAME under backdoor attacks. In particular, derive an upper bound for the L2-norm of the difference between the parameters of the global models with and without attacks. To analyze the robustness of FLGAME,
make the following assumptions
on the loss function used by the clients, which are commonly used in the analysis of previous studies federated
(Li et al. 2020b, Wang & Joshi 2021 Fallah al. 2020 Reisizadeh et al. 2020) learning.
Assumption 1. The loss function is p-strongly convex with L-Lipschitz continuous gradient. Formally, we have the following for arbitrary θ and O':
(Vel(z; 0) - Verl(z; (()());)))))))) ( 0¹) #10 [z) - (er 0')l22 S I. L| 0'll2;
(8)
where z is an arbitrary training example. Assumption We assume the gradient Vel((z; θ) is bounded with respect to Lz-norm for arbitrary
O and z, i.e., there exists some M 2 O such that
[Vel(z 1) $ M.
5

---

Under review as a conference paper at ICLR 2023
Suppose O is the global model trained by FLGAME without any attacks in the tth communication round, i.., each client i E uses its clean local training dataset D; to train a local model. Moreover, assume gradient descent with a local model learning rate 1 is used by each client to train its local model. Suppose of is the genuine score for client i without attacks. Moreover, we denote
3 E.es of as the normalized genuine score for client i. To perform the backdoor attack, we assume compromised client can embed the backdoor trigger to r, fraction of training examples in the local training dataset of the client and relabel them as the target class. Those backdoored training examples are used to augment the local training dataset of the client. Suppose O' the global model under the backdoor attack in the tth communication round with our defense. We denote
Eies of as the normalized genuine score for client with attacks in the tth communication round. We prove the following robustness guarantee for FLGAME:
af =
Lemma 1 (Robustness Guarantee for One Communication Round). Suppose Assumptions 7 Mandl hold. Moreover, we assume (1 8' S of s (1 + 1') where i and r" Ljes. l.j. Then,
we have:
1 - 9 SVI - npp + 2 + 122 + 21² ||0 -
22 (1 + n + 2mm²) + 2np² M,
(10)
where 17 is the learning rate of the global model, and μ are defined Assumption 7 Eies, arri M, and M is defined in Assumption 2
7"
Proof sketch. Our idea is to decompose - I| into two terms. Then, we derive an upper bound for each term based on the change of the local model updates of clients under backdoor attacks and the properties of the loss function. As a result, our derived upper bound relies on r for each client E Sa. parameters μ4, L, and M in our assumptions, well the parameter difference of the global models in the previous iteration, i.e., ||@' O'dl2- Our complete proof can be found in Appendix A.
In the above lemma, we derive an upper bound of - with respect to ||O - 0 for one communication round. In the next theorem, derive upper bound of ||@* 0 We iterative apply Lemma [ for successive values of and have the following theorem:
co.
Theorem 1 (Robustness Guarantee). Suppose Assumptions 7 and hold. Moreover we assume (1 - r S of S (1 + r)) for E S, 7¹ < " and r² < r hold for all communication round t, and p. > 27,, where r¹ = Ejes, r² and 7th Eies. atr) M. Let the global model learning rate by
chosen 0 1 2-2y Then, have:
21 + 11I + 2177) + 2
||0| - -0'lo''ll2 S 1 /1- mp + 21 + 7 L2 2782
(11)
holds DC.
Proof sketch. Given the conditions that 7" 5 and pd < as well as the fact that the right-hand side of Equation 10 is monotonic with respect to 7 and r', we can replace 7 and r in Equation 10 with y and r. Then, we iterative apply the equation for successive values of t. When
17 L242Ly*
we have 0 < 1 - ny 2nyy + T.2 27 Ty <. By letting co, we can reach the conclusion. The complete proof can be found in Appendix
A.2 A.2
When r/ 0 for Vi, Vt, we have 7 and 0. Thus, the upper bound in Equation 11
becomes 0.
EXPERIMENTS
6.1 EXPERIMENTAL SETUP
Datasets and global models: We use two datasets: MNIST (LeCun et al. 2010) and CIFAR10 (Krizhevsky 2009) for FL tasks. MNIST has 60,000 training and 10,000 testing images, each of

---

Under review as a conference paper at ICLR 2023
which has size of 28 x 28 belonging to one of 10 classes. CIFARIO consists of 50,000 training and 10,000 testing images with a size of 32 x 32. Each image is categorized into one of 10 classes. randomly sample 90% of training data for clients, and the remaining 10% of
For each dataset, training data is reserved to evaluate our defense when the clean training dataset of the server is from the same domain as those of clients. We use CNN with two convolution layers (detailed architecture can be found in Table 5 in Appendix) and ResNet-18 (He et al., 2016) which is pre-trained on ImageNet (Deng et al. 2009) as the global models for MNIST and CIFARIO.
FL settings: We consider two settings: local training datasets of clients are independently and identically distributed (i.e., IID). and not IID (i.e., non-IlD). In IID setting, we randomly distribute the training data to each client. In non-IlD, we follow the previous work (Fang et al.) (2020) to distribute training data to clients. In particular, they use a parameter q to control the degree of non-III, which models the probability that training images from a class are distributed to a particular client (or set of clients). We set q = 0.5 by following (Fang et al. 2020). Unless otherwise mentioned, we consider the IID setting. Moreover, we train a global model based on 10 clients for 200 iterations with a global model learning rate 7) = 1.0. In each communication round, we use SGD to train the local model of each client for two epochs with a local model learning rate 0.01. Moreover, we consider all clients are selected in each communication round.
Backdoor attack settings: We consider state-of -ft---f-thhheeee att backdoor attacks to federated learning. i.e., Scaling attack Bagdasaryan et al. 2020) and DBA (Xie et al.) (019). We use the same backdoor trigger and target class used in those works. By default, we assume 60% of clients are compro- mised by an attacker. We set the scaling parameter to be #total following Bagdasaryan et al. (2020). When the attacker solves the minimax game in Equation 6 we set the default À = 1. We will explore its impact in our experiments. We randomly sample 10% of the local training data of each compromised client as validation data to search for an optimal r(. Moreover, we set the granularity of grid search to be 0.1 when searching for r/.
clients)
Baselines:
We compare our defense with the following methods: FedAvg (MeMahan al. 2017a).
Krum (Blanchard et al.] 2017), Median (Yin et al. 2018). Norm-Clipping (Sun et al. 2019, Differ- ential Privacy (DP) (Sun et al. 2019). and FL.Trustt (Cao et al. 2021a). FedAvg is non-so while Krum and Median
two crrzattinc-obbst baselines. Norm-Clipping Clipping clips the L2 L2-norm of local model updates to a given threshold TN. We set TN = 0.01 for MNIST and TN = 0.1 for CIFAR 10. DP first clips the L3 Lo-norm of local model update to threshold To and then adds Gaussian noise. We set To = 0.05 for MNIST and To = 0.5 for CI.AR1O. We set the standard deviation of noise to be 0.01 for both datasets. In FLTrust, the server uses its clean dataset to compute a server model update and assigns a trust score to each client by leveraging the similarity between the server model update and the local model update. We set the clean training dataset of the server be the same as FLGAME in our comparison. Note that FLTrust is not applicable when the clean training dataset of the server is from different domain from those of clients.
We use testing accuracy (TA) and attack success rate (ASR) as evaluation metrics. TA the fraction of clean testing inputs that are correctly predicted. ASR is the fraction of backdoored testing inputs that are predicted as the target class.
Evaluation metries:
We consider two settings: in-domain and oft-of-domain. For the in-domain setting, we consider the clean training dataset of the server is from the same domain as the local training datasets of clients. We use the reserved data as the clean training dataset of the server for each dataset. For the out-of-domain setting, we consider the server has clean training dataset that is from the different domains of FL tasks. In particular, we randomly sample 6,000 images from NaShionMNIST (Xiao et al. 2017) for MNIST and sample 5,000 images from GTSRB (Houben et al.) for CIFARIO as the clean training dataset of the server. We adopt Neural Cleanse (Wang et al. to reverse engineer the backdoor trigger and target class.
Defense setting:
2013)
2019
6.2
EXPERIMENTAL RESULTS
Our FLGAME consistently outperforms existing defenses: Table and Table Show show the results of FLGAME compared with existing defenses under IID and non-IID settings. We have the following observations from the experimental results. First, FLGAME outperforms all existing defenses in terms of ASR. In particular, FLGAME can reduce ASR to random guessing (i.c., ASR of FedAvg under attacks) in both IID and non-IIID settings for clients well both in-domain and out-of-
7

---

Under review as a conference paper at ICLR 2023
Table Comparison FLGAME with existing defenses under Scaling attack. The total number of clients is 10 with 60 % compromised. The best results for defense are bold.
Defenses (Under attacks)
FedAvg (No attacks) FedAvg Krum Median
FLGAME
Datasets
Metrics
Norm- Clipping
DP
FLTrust
In- domain domain
Out-of-
TA (%)
99.04
98.77
98.78
99.17
95.48
92.97
97.93
98.53
98.56
MNIST
ASR (%)
9.69
99.99
99.99
99.97
98.54
99.45
16.01
9.72
9.68
TA (%)
81.08
80.51
76.44
80.17
80.38
43.22
75.71
74.81
67.42
CIFAR10
ASR (%)
8.39
99.8
99.94
99.82
99.87
99.58
99.46
8.92
9.57
Table Comparison FLGAME with existing defenses under Scaling attack. The total number of clients is 10 with 60% compromised. The local training datasets of clients are non-IID. The best results for defense are bold.
Defenses (Under attacks)
FedAvg
FLGAME
Datasets
Metries
(No attacks) FedAvg Krum Median
Norm- Clipping
DP
FI.Trust
In- domain domain
Out-of-
TA (%)
98.98
99.15
96.88
99.12
94.54
91.52
97.68
98.28
98.34
MNIST
ASR (%)
9.73
99.99
85.03
99.98
98.16 18.16
99.54
19.61
10.42
10.88
TA (%)
80.25
75.35
67.66
79.54
70.18
50.79
75.08
73.43
72.37
CIFARIO
ASR (%)
9.67
99.92
99.92
99.99
99.63
95.01
99.82
11.51
10.62
Table 3: Comparison of FLGAME with existing defenses under Scaling attack. The total number of clients is 30 with 60 % compromised. The best results for defense are bold.
Defenses (Under attacks)
FedAvg
FLGAME
Datasets
Metries
Norm- Clipping
(No attacks) FedAvg Krum Median
DP
FLTrust
Out-of- domain domain
In-
TA (%)
99.02
99.09
98. 16 99.01
92.77
89.77
95.27
97.81
97.64
MNIST
ASR (%)
9.74
99.98
99.98
99.98
98.2
98.83
11.04
9.95
9.95
TA (%)
80.08
79.73
72.23
79.58
79.2
50.86
67.84
73.29
60.1
CIFARIO
ASR (%)
9.14
99.82
99.97
99.85
99.87
96.53
99.28
10.44
12.87
Table 4: Comparison of FLGAME with existing defenses under DBA attack. The total number of clients is 10 with 60 % compromised. The best results for defense are bold.
Defenses (Under attacks)
FedAvg
FLGAME
Datasets Metrics
Norm- Clipping
(No attacks)
FedAvg Krum Median
DP
FLTrust
In- domain
Out-of-
domain
TA (%)
99.02
99.03
98.87
98.98
98.99
98.99
97.98
97.84
98.05
MNIST
ASR (%)
9.74
100
10.06
99.81
99.75
99.73
10.02
9.56
9,68
TA (%)
81.08
80.9
76.09
80.0
80.21
41.36
75.17
73.18
72.93
CIFARIO
ASR (%)
8.39
93.44
94.97
91.60
91.90
86.96
66.58
8.81
9.00
8

---

Under review as a conference paper at ICLR 2023
domain settings for the server. Intrinsically, FLGAME performs better because our game-theoretic defense enables the defender to optimize its strategy against dynamic, adaptive attacks. We note that FLTrust outperforms other defenses (except FLGAME) in most training dataset from the same domain as local training datasets of clients. However, FL.Trust is not applicable when the server only holds out-of-domain clean training dataset, while FLGAME can relax such an assumption and will still be applicable. Moreover, our experimental results indicate that FLGAME achieves comparable performance even if the server holds an out-of-domain clean training dataset. In Appendix C.2 we visualize the average genuine (or trust) scores computed by FLGAME (or FLTrust) for compromised and benign clients further explain why our FLGAME outperforms FLTrust. Second, our FLGAME achieves comparable TA with existing defenses, indicating that our FLGGAME preserves the utility of global models.
since it exploits a clean
Table 3 shows the comparison results of FLGAMM with existing defenses when the total number of clients is 30. Table 4 shows the comparison results of FLGAME with existing defenses under DBA attack. Our observations are similar, which indicates that FLGAME consistently outperforms existing defenses under different numbers clients and backdoor attacks.
Impact of A: À is a hyperparameter used by an attacker when searching for the optimal r for each compromised client each communication round f. Figure [ shows the impact of on ASR of FLGAME. The results show that our FLGAME is insensitive to different X's. The reason is that the genuine score for compromised client is small when À is large, and the local model of a compromised client is less likely predict trigger-embedded input as the target class when À is small. As result, backdoor attacks with different À are ineffective under our FLGAME.
Impact of the fraction of compromised clients: Figure 2 shows the impact of the fraction of compromised clients on ASR of our FLGAME and FLTrust. As the results show, our FLGAME is effective for a different fraction of compromised clients in both in-domain and out-of-domain settings. In contrast, FLTrust is ineffective when the fraction of compromised clients is large. For instance, our FLGAME can achieve 9.84% (in-domain) and 10. 12% (out-of-domain) ASR even if 80% of clients are compromised on MNIST. Under the same setting, the ASR of FL.Trust is 99. 95%. indicating that the defense fails.
12.5.
12.5
80-
80-
10-
7.5
7.5
H 60
80
2.5+
2.6
20-
20.
0.6
0.8
1.0
0.6
0.3
1.0
0.2
0.4
0.5
0.8
0.2
0.4
0.6
0.8
Truction of malicionn cilente
Fracttes caltedens clcest
(a) MNIST
(b) CIFARIO
(a) MNIST
(b) CIFAR10
Figure 1: Impact of À on ASR of FLGAME under Scaling attack. FLGAME is insensitive to various choices of X.
Figure 2: Impact of the fraction of com- promised clients on ASR of FLGAME and FLTrust under Scaling attack.
7 CONCLUSION AND FUTURE WORK
In this work, we propose FLGAME, general game-theoreti, defense against adaptive backdoor attacks federated learning. Our formulated minimax game enables the defender and attacker to dynamically optimize their strategies. Moreover, we respectively design solutions for both of them to solve the minimax game. Theoretically, we show that the parameters of the global model with the backdoor attack under FLGAME is close that without attacks. Empirically, we perform systematic evaluations on benchmark datasets and compare FLGAME with multiple trate-ofthhe-e baselines. Our results demonstrate the effectiveness of FLGAME under strategic backdoor attacks. Moreover, FLGAME achieves significantly higher robustness than baselines.
Interesting future work includes: 1) extending our FLGAME to defend against other attacks to federated learning, and 2) improving FLGAME by designing new methods to reverse engineer the backdoor trigger and target class via exploiting the historical local model updates sent by each client.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
We propose theeetteo defense against backdoor attacks to federated learning in this work. One potentially harmful effect that an attacker may leverage our defense enhance its attack. However, our defense already considers strategic attacks. Therefore, we do not see any explicit ethical issues with our work.
REPRODUCIBILITY STATEMENT
We discuss the reproducibility of our work from two aspects: theoretic analysis and empirical results. For theoretic analysis, explicitly explain the assumptions that make in Section [ We also include the complete proofs for our lemmas and theorems in Appendix. For empirical results, we discuss the details of our experimental setup in Section 66. including datasets and global models, federated learning settings, backdoor attack settings, baselines, and their parameter settings, as well as our FLGAME settings. The datasets used this work are all publicly available. We also add the link to the publicly available codes used in our experiments. We will release our code upon paper acceptance.
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, Brendan McMAhan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In ACM SIGSAC Conference on Computer and Communications Security, pp. 308-318, 2016.
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pp. 2938-2948. PMLR, 2020.
Gilad Baruch, Moran Baruch, and Yoav Goldberg. A little is enough: Circumventing defenses for distributed learning. Advances in Neural Information Processing Systems, 32, 2019.
Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateck Mittal, and Seraphin Calo. Analyzing federated learning through adversarial lens. In International Conference Machine Learning, pp- 634-643. PMLR, 2019.
Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. Advances in Neural Information Processing Systems, 30, 2017.
Xiaoyu Cao, Minghong Fang, Jia Liu, and Neil Zhenqiang Gong. FLtrust: Byzantine-robust federated learning via trust bootstrapping. In Network and Distributed Systems Security Symposium, 202 la.
Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. Provably secure federated learning against malicious clients. In AAAI Conference Artificial Intelligence, volume 35, pp. 6885-6893, 202 lb.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography Conference, pp. 265-284. Springer, 2006.
Alireza Fallah, Aryan Mokhtari. and Asuman Ozdaglar. Personalized federated learning: A meta- learning approach. In Conference on Neural Information Processing Systems, 2020.
Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Gong. Local model poisoning attacks to
Byzantine-Ron Robust federated learning. 29th USENIX Security Symposium, pp. 1605-1622, 2020.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEFE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
10

---

Under review as a conference paper at ICLR 2023
Sebastian Houben, Johannes Stallkamp, Jan Salmen, Mare Schlipsing. and Christian lgel. Detection of traffic signs in real-world images: The German Traffic Sign Detection Benchmark. In International Joint Conference Neural Networks, number 1288, 2013.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. Foundations and Trends@ in Machine Learning, 14(1-2):1-210, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. ATT Labs [Online]. 2, 2010.
Available: http://yan.llecm..com
cotpexd///////iiiii...mmmmma http://ywam.ccc...mmmeebxbbbbis.
Suyi Li, Yong Cheng, Wei Wang, Yang Liu, and Tianjian Chen. Learning to detect malicious clients for robust federated learning. arXiv preprint arXiv:2002.002/1, 2020a.
Xiang Li, Kaixuan Huang, Wenhao Yang. Shusen Wang, and Zhihua Zhang. On the convergence of FedAvg on non-iid data. In International Conference on Learning Representations, 2020b.
Guodong Long. Yue Tan, Jing Jiang. and Chengqi Zhang. Federated learning for open banking. In
Federated learning, pp. 240-254. Springer, 2020.
Guodong Long, Tao Shen, Yue Tan, Leah Gerrard, Allison Clarke, and Jing Jiang. Federated learning for preeeeriigg open innovation future on digital health. In Humanity Driven AI, pp. 113-133. Springer, 2022.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication-efficiie learning of deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pp.
PMLR, 2017a
H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In International Conference on Learning Representations, 2017b.
Thien Due Nguyen, Phillip Rieger, Huili Chen, Hossein Yalame, Helen Möllering, Hossein Fereidooni, Samuel Marchal, Markus Miettinen, Azalia Mirhoseini, Shaza Zeitouni, al. FLAME: Taming backdoors in federated learning. 31 USENIX Security Symposium, pp. 1415-1432, 2022.
Mustafa Safa Ozdayi, Murat Kantarcioglu, and Yulia Gel. Defending against backdoors in federated learning with robust learning rate. In AAAI Conference on Artificial Intelligence, volume 35, pp. 9268 9276, 2021.
Amirhossein Reisizadeh, Farzan Farnia, Ramtin Pedarsani, and Ali Jadbabaie. Robust federated learning: The case of affine distribution shifts. Advances in Neural Information Processing Systems, 0221554-21566 2020.
Phillip Rieger, Thien Due Nguyen, Markus Miettinen, and Ahmad-Reza Sadeghi. DeepSight: Mitigating backdoor attacks in federated learning through deep model inspection. In Network and Distributed System Security Symposium, 2022.
Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H Brendan McMahan. Can you really backdoor federated learning? arXiv preprint arXiv: 1ete19991.779... 2019.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Zhao. Neural Cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy, pp. 707-723. IEEE, 2019.
Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy-yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos. Attack of the tails: Yes, you really can backdoor federated learning. Advances in Neural Information Processing Systems, 33:16070- 16084, 2020.
Janyu Wang and Gauri Joshi. Cooperative SGD: A unified framework for the design and analysis of communication effiiiienntiiinninntteii SGD algorithms. Journal of Machine Learning Research, 22(213):1-500 2021.
11

---

Under review as a conference paper at ICLR 2023
Chen Wu, Xian Yang, Sencun Zhu, and Prasenjit Mitra. Mitigating backdoor attacks in federated learning. arXiv preprint arXiv:2011.01767, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: A novel image dataset for bench- marking machine learning algorithms. arXiv preprint arXiv:1708.0774: 2017.
Chulin Xie, Keli Huang, Pin- Yu Chen, and Bo Li. DBA: Distributed backdoor attacks against
federated learning. In International Conference on Learning Representations, 2019.
Chulin Xie, Minghao Chen, Pin- Yu Chen, and Bo Li. CRFL: Certifiably robust federated learning against backdoor attacks. 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11372-11388 PMLR, 2021.
Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantineroobust distributed learning: Towards optimal statistical rates. In International Conference on Machine Learning, pp. 5650-5659. PMLR, 2018.
Zaixi Zhang, Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. FLDetector: Defending federated learning against model poisoning attacks via detecting malicious clients. In 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2545-2555, 2022a.
Zhengming Zhang, Ashwinee Panda, Linyue Song, Yaoqing Yang. Michael Mahoney, Prateek Mittal, Ramchandran Kannan, and Joseph Gonzalez. Neurotoxin: Durable backdoors in federated learning. In International Conference on Machine Learning, pp. 26429-26446. PMLR, 2022b.
12

---

Under review as a conference paper at ICLR 2023
APPENDIX
A COMPLETE PROOFS
A. I PROOF OF LEMMA I
We first present some preliminary lemmas that will be invoked for proving Lemma
Lemma 2. Suppose 'D; is the clean local training dataset of the client i. An attacker can inject the backdoor trigger r fraction of training examples in D; and relabel them the target class. We use D to denote the set of backdoored training examples where ri
[P[]
Given two arbitrary θ
[.
and c we let 9s |D,LD/ Ve EREDIUDE f(z: ②) and h;
[Pi] Ve. EeED, E(z; Oc). We then
have that
(0 - 0₂)² (9: - h,) 2 (0.5p Mr 1. M, 19: hell2 - L||e O.11 + 2r
(12)
(13)
Proof. We first prove Equation 12 We have the following relations:
(0 0z)=((a (yi -44)
=(0 - (
1
Σ
Vol(z';0)
Σ
(e.((z;O.))
>> definition of ga and ha
[D, D'd D'ED.UD
[D;
ZED.
(14)
-(0 - (
Σ Vol(z';)) 1
Σ
(o.((z;O.))
[
(15)
(1 + IDD
[D
[
2'ED,UD
ZED
(0 -
(0-0,)"( (
Σ Vol(z':0) (1 (1+ Vo.(zz;o.)
(16)
[D: + rt)
Z'EDUUD!
RED;
( - (0-)))
Vol(z';0) - Σ (z;
[P:
z'ED:
ZED
+
Vol(z';)) - r Σ
(z. 0.))
(17)
z'ED/
Σ
(0 - 0.) Rd (. Ve.
[1 + +r/)
ZED,
+ (0 - T
Vol(z/:0) - Σ (z))
(18)
D'ED
RED.
(e (V
D + +r+)
RED;
-1(0-)))"
Vol(z'00) - Σ Vo. (gz
Vr,
(19)
xEDD
xED,
(0 -Vo((z
ID: + r()
xED;
||0 Oell2-1 11
Σ Vol(z';0) - Σ Vo, Vo.
Cauchy-Schwarz inequality
z'ED/
zED;
2 (p. +
Σ ( 0.) T (Ve((2 () Ve.
Vo, 0.)
ZED,
- - 0.l12
Σ
|Vol(z'; 0)112 + + Σ
|Ve. 02)12) D triangle inequality
2'ED/
xED.
13

---

Under review as a conference paper at ICLR 2023
[1 + +r+) (u ( -
Assumption
(20)
1+ r 10 1 0 1+ T+ 2r
"
||0 6e112)
(21)
20.5p |00 1.1 2r 100 Oell e
(22)
20.5p 10 - 2a1 rM ||6 0.1 M)
(23)
= -0.5pp Mf ||0 0e112 -
(24)
where Equation 23 holds based on the fact that 2r 110 Oell2 2 -rfM ||0 0.11 r M for Vrr 2 0 and VM O.
In the following, we prove inequality 13 We have that
11: - -hill2
Vol(z'; (1 (1+r() r Σ
(z,(
definition of gi and h;
iPi(( x'ED;UD
zED,
(25)
Vel(z'; + Σ - Σ
Ve,((z;0,)lz
(26)
[Di(1 + rr)
Z'EDG
x'ED;
zED; (z: 02)l2
$ +
Vol(z';0) - r
x'ED
BED
+ [D,((1 + D'ED.
Vol(z';)) Σ
Ve, (z: Ox]lk
>> triangle inequality
(27)
RED
1+ (2r + L||0 0cl12)
(28)
<2r + - r [0.1]
(29)
where Equation 28 is due to Assumption | and 2
Given Lemma 2 we prove Lemma
follows. Recall that have of Eieg p
x
and /3
of
Eies !
| 04+1 2
(30)
=1|0' 7 Σ atgh (0) 17 Σ Bth()l2
gradient descent for O--1 and 0
(31)
iEs
=110' "
ag (0) 17 Σ
(af 1 -f)h/ll
(32)
iES
=||0'
O
af(g!
KD)) (n Σ (3) aihillla
rearranging Equation 32
(33)
0 0
ag
1ll1
|In
Σ
(6)
triangle inequality
(34)
Next, we respectively derive an upper bound for the first and second terms in Equation 34 the upper bound for the first term, we have that
To derive
0 ") Σ ogls 11 ||'
iES
=|
Σ
af ht))
Σ
at
ht
(35)
iES
iEs
14

---

Under review as a conference paper at ICLR 2023
=11 + S2 + Ss:
(36)
where S = |10| - -0'1l. S2 = -2n( - O'') al - k))), and Sa =
r² |Eies ag - ni)l1 Next, we will bound Sn and S3. We denote 7" = Eiss, afr Note
that we have 7' = Eies afriM since rf = 0 for vi E S Sa. We bound S2 as follows.
S2
= - --2n(0'= - 60) Σ ag - h2)
(37)
iES
- 2n Σ -))
- 707 (sf 1t))
(38)
iES
S - 2n Σ
a:((0..pp - r.M ||00 - - x(M)
(39)
iES
= - 20((0.
afr ||0| 2110 Ma
(40)
iES
iES.
-- + 21n") |00 - | + 2077" Deefiiition of 7d
(41)
where inequality 39 holds by Lemma and the fact that n, af 2 0. We bound S3 follows.
S3
-
Σ
cl - -1112
(42)
iES
Σ at |1 - 12l12)²
(43)
IS
Σ (2 - L110 02112) >> Lemma 2
(44)
iES
= (2 + L||0 - 0
(45)
2 ||0 - 0.12 + 4 L |0 - 0ell
(46)
<L 1 +2 -
+ +27 + +l1113
(47)
= ((T.2 22 100 0.11 + 2IA"
+ + +4b213)
(48)
where Equation 47 is based on the fact that 17 - O s 27' 110 -Oel2 + 2 T. when
2 11200 O.
Given the upper bounds of Sn and S3, we can bound ||0r - O - 17 EiEs af - -bill as follows.
|10º - 0% 17 Σ
atcot - m0)l
(49)
YES
=S + 82 + Sz
(50)
-110-0,
- 0.112 + (- + 2m 2n 6 2
+ +1221? + 2 | - + + 2/2 + (
(51)
=(1 n/ + 2m7 + 2 + 2n ||0º - 0% + 2mm + 20 + 417
(52)
Next, we will derive an upper bound for |/ -a))(l ll We denote = Eies, rh. Note that we have that yd = Eies ri also holds since rf = 0 for vi E S \ Sa. Given the assumption that
-
- r' < 3 < (1 x r'Jaf, we have
11 Σ (8f Mf0llll I
|
af
h1 2 S 2nr'' M,
(53)
iES
iES
15

---

Under review as a conference paper at ICLR 2023
where the first inequality is due to triangle inequality and the second inequality is based the assumption that | $ M. Therefore, we have:
- 1|
5 - - 1) Σ
at -1)1l + || E (B) 18/-o,)hhl
>> Equation 30, 34
(54)
YES
SES
5V(1 mp + 2xy" + +
||0 - 21 + v/. / 207²)
(55)
+ 27r M D> Equation
49
52
53
(56)
S <V - np + 2nyy + 12 21²² Ly
|0'
0'll2 + 22 (1 + 2mp²) + 2np M, (57)
where the last inequality holds due to the fact that va+bs S Va+ /a for for va 2 and vb 2 0, which completes our proof for Lemma]
A.2 PROOF OF THEOREM 1
We denote A, V1 nn + 20y + 1 2n A V mut 2n7 122 2nj Ioy,
B. 221r/(1+ nL 2m7²) + 2nr M. and B / 22 + nL 2n7)) + 2nr Since 7 S
and r¹ Sr we have A, S A and B, - B. Thus, based on Lemma 1 we have:
||0 e'll2 - A||0*1 0 112 + B.
(58)
Then, we can iteratively apply the above equation to prove our theorem. In particular, we have:
|0' - -
- - 12 B
(59)
SA(A | 0 212 + B) BB
6
(60)
=43 | - + (4t + B
(61)
<A' 0 - (A'-1 + + A
(62)
1- At
=A' 0 - 01112 + 1 A
B
(63)
-(V1 - пp + 207 + 121 + 27/2 Ly)' 1600 - -001
(v1- + 2077 + 1/2 2n Try)'
+ / t/ 217 1222 + 21 To
(12m + nl / 217) + Mnr
(64)
When the learning rate satisfies 0 < <
2- 22
we have that 0 < 2 2 by <1 1.
Therefore, the upper bound becomes
cc. Hence, we prove VI- 23
Theorem
B COMPLETE ALGORITHMS
B. COMPLETE ALGORITHM OF FLGAME
Algorithm I shows the complete algorithm of FLGAME. In Line I we construct an auxiliary global model. Line 2 the function REVERSEENGINEER is used to reverse engineer the backdoor trigger and target class. In Line 4 we compute the local model of client i based on its local model update. In Line(5) we compute a genuine score for client i. In Line we update the global model based on genuine scores and local model updates of clients.
B.2 COMPLETE ALGORITHM FOR A COMPROMISED CLIENT
Algorithm 2 shows the complete algorithm for a compromised client. Line I we randomly subsample P; fraction of training data from D. In Line the function CREATEB
16

---

Under review as a conference paper at ICLR 2023
Algorithm 1: FLGAME
Input: O (global model in the /th communication round), of i E S (local model updates of clients), D, (clean training dataset of server), 7) (learning of global model). Output: 0 (global model for the (t + 1)th communication round)
O Ot [5] Lies of
òre Drc = REVERSEENGINEER(()
for
do
θ = O' + gi
pf =1 [D. ExED, I(G((x + Ore:0)) vra)
O = = + "" Eies
return @
used to generate backdoored training examples by embedding the backdoor trigger ô to [min(j (,1)|D; Dreel] training examples in D; Dr and relabel them as y"s, where measures the number of elements in set. In Line 6 the function TRAINING is used train the local model on the training dataset D. D Drevv Line 7 we estimate a genuine score. In the function CREATEBACKDOOREDDXXX to generate backdoored training examples by embedding the backdoor trigger d to [min(o Ç, 1)] training examples in D; and relabel them as y In Line 12 we use the function TRAININGLoCALLMODEE to train a local model on the training dataset D U
Line [1]
Algorithm ALGORITHM FOR COMPROMISED CLIENT
Input: O (global model the /th communication round), D; (local training dataset of client v),, (fraction of reserved data to find optimal r(). Ç (granularity of searching for r(), 6 (backdoor trigger), y" (target class), and (hyperparameter). Output: of (local model update)
Drev KAANDMMSAMPLIN((D.
count- = -
mar malr_value,er o + 0,0
for j t count do
D. CREAT \ Drev, Dj U'D Drev)
C,
O = TRAINING
Ex [[G((x @ o; O.₂) = y*)
Pij
if Pij + A Amin(j * 1, max max_valus value then
=
10
max -value = Pij - A min( j= * 1)
11 D CREATTEBCCKDDRRDDDDAY (D;, 6, min(( * 1)) 12 O TRANNNGG, " D.D
13 return 0 O
C ADDITIONAL EXPERIMENTAL SETUP AND RESULTS
C.1 ARCHITECTURE OF GLOBAL MODEL
Table 5 shows the global model architecture MNIST dataset.
C.2 ET AL.
VISUALIZATION OF GENUINE SCORE OF FLGAME AND TRUST SCORE OF FLTRUST CAO
A
Our FLGAME computes genuine for each client which quantifies the extent which client benign in each communication round. Intuitively, our FLGAME would be effective if the genuine score is small for compromised client but is large for benign FL.Trust (Cao al. 2021a)
17

---

Under review as a conference paper at ICLR 2023
Table 5: Architecture of the convolutional neural network for MNIST.
Type
Parameters
Convolution
3 x 3, stride=1, 16 kernels
Activation
ReLU
Max Pooling
2
Convolution
4x x 4, stride=2, 32 kernels
Activation
ReLU
Max Pooling
2 x
Fully Connected
800 500
Activation
ReLU
Fully Connected
500 10
computes trust score for each client in each communication round. Similarly, FLTrust would be effective the trust score small for a compromised client but is large for a benign one. Figure 3 visualizes the average genuine trust scores for compromised and benign clients of FLGAME and FLTrust on MNIST dataset. We have the following observations from the figures. First, the average genuine score computed by FLGAME drops quickly for compromised clients. In contrast, the average trust score computed by FLTrust drops slowly. Second. the average genuine score computed by FLGAME for benign clients first increases and then becomes stable. contrast, the average genuine score computed by FLTrust for benign clients decreases as the number of iterations increases. As result, our FLGAME outperforms FLTrust.
0.8
Degenateed (2tema
0.8
Sergenetzel Cleare
0.8
Scppotiast Cleate
0.8
Ooprecten Eliees
10.6
00.6
0.6
0.5
5 50.4
0.4
0.4
0
150
200
15c
(a) FLTrust (IID)
(b) FLTrust (non-IID)
(c) FLGAME (IID)
(d) FLGAME (non-IID)
Figure 3: (a)(b): Average trust scores computed by the server for benign and compromised clients of FLTrust on MNIST in IID and non-lID settings under Scaling attack. (c) Average genuine scores computed by the server for benign and compromised clients of FLGAME on MNIST IID and non-lll) settings under Scaling attack. The clean training dataset of the server is the same for FLTrust and FLGAME.
18


Reviews:
[{"review_id": "caS5VQwWdJC", "reviewer": "Reviewer_jdkR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a dynamic backdoor defenses in federated learning, where both the defender and the attacker can dynamically adjust their strategies as the federated training continues. The defender reverse engineers the potential backdoor trigger so as to finally generate the benign score for each client, and downweights the clients with lower benign scores. Attackers aim to find the proper backdoor poisoning ratio such that a good tradeoff between backdoor effectiveness and benign scores of compromised clients can be reached. ", "summary_of_the_review": "The idea of defending against adaptive backdoor attacks in federated learning is interesting. However, the proposed framework did not consider this \"adaptive\" aspect thoroughly and hence, is not convincing. Also, the experimental settings are not practical and some recent baselines are missing. Considering all the weakness mentioned above, rejection is given.   ", "strength_and_weaknesses": "Strength:\nThe paper is well written and the idea of defending against adaptive attackers is impressive. \nWeakness:\n1. the whole framework relies on the defender is able to reverse engineer useful backdoor triggers so as to effectively minimize the impact from compromised clients. However, this design may be exploited by the attacker [1]. Specifically, it is possible that attacker generate backdoors in a way that, either the reversed engineered backdoor samples are not effective or are similarly effective for both the compromised and benign clients. This assumption is the major weakness of this paper. \n2. it is unclear how well those theories (e.g., Theorem 1) imply in practice. For example, the assumption on $r_i^t=0$ seems to not hold in practice and so, the backdoor model cannot be the same the clean global model, even asymptotically as the training rounds continues.  \n3. the threat model makes too many restrictions on the attacker capability. For example, the attackers can only augment the training data with backdoored samples while in practice, they may choose to replace some samples with backdoored samples. The attackers are also assumed to be unaware of the benign scores of other clients while in practice, they may be able to simulate the benign scores (given the knowledge that the server is calculating benign scores for each client using reversed engineered triggers). In the worse case scenario, powerful attackers may also be able to get the actual benign scores. It is also assumed that attackers can only use its own backdoor trigger when deciding the proper poisoning ratio while in practice, attackers can also simulate the reverse engineering process of the defender. \n4. there are also some unrealistic experimental settings in the paper. For example, it is assumed that all clients in each round will be selected while in practice, defenders usually randomly selects subset of the clients. Also, the fraction of compromised clients are mostly evaluated under 60%, which, theoretically, should be hopeless to defend against. The lowest fraction of compromised clients is still 20%, which is quite a high number. I am curious to see the comparison among different defenses (not just FLTrust and the proposed method) under even lower compromised clients. \n5. some recent works are missing from the paper. For example, the most recent defense on backdoor attacks in federated learning [2]. \n[1] Veldanda et al., \"On Evaluating Neural Network Backdoor Defenses\".\n[2] Rieger et al., \"DeepSight: Mitigating Backdoor Attacks in Federated Learning Through Deep Model Inspection\", NDSS 2022. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "N0vv_lewsvF", "reviewer": "Reviewer_tQyp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces FLGAME, a defense against adaptive backdoor attacks. It formulates the compromised client attack and server defense as a form of minimax optimization where the clients want to maximize their contribution to the model update and the server wants to minimize malicious contribution. This optimization centers around a genuine score that weights the model update during aggregation. The server computes this through a reverse engineered backdoor trigger and target class. ", "summary_of_the_review": "The method of using a minimax game and the use of an adaptive defense is a very interesting and important contribution. However, my main issue is the lack of depth in the empirical results. In particular, the datasets used and the analysis on compromised client participation can be made much stronger. Please refer to the weaknesses section for specifics.", "strength_and_weaknesses": "Strengths: \n•\tFLGAME achieves relatively high model accuracy with a very low attack success rate compared to prior defense.\n•\tThe dataset the server has does not need to be in-domain. This is a much easier requirement to achieve that prior work.\n•\tThe introduction of using an adaptive defense is a good contribution.\n\nWeaknesses:\n•\tThe experimental datasets were very simple. I would like to see more complex datasets, such as ImageNet/Tiny Imagenet. \n•\tPlease expand on the contribution of the compromised clients in the model update. It’s not clear whether the attack success rate is low because the compromised clients have a low genuine score or if their updates result in weak backdoor success.  \n•\tIt is okay to include a few baseline comparisons. However, many of the defenses compared were not intended for backdoor attacks. Please show comparisons to other new defenses aimed for backdoor defense.\n•\tThe paper relies on prior work for reverse engineering backdoor triggers and target class. I would like to see more about this. What limitations does this have? If the compromised clients use larger L1 norm triggers, does this fail? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CysM5ZbqesY", "reviewer": "Reviewer_Z2ep", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to formulate the defense against backdoor attacks as a minimax game between the server (defender) and attackers in federated learning. The minmax problem is solved in three steps by 1) construct an auxilary global model; b) reverse-engineer backdoor samples with selected target classes; c) assign to each client a genuine score is used to weigh model update uploaded by the client. \n", "summary_of_the_review": "The proposed method is seriously flawed in my view for following reasons: \n\n- the assumption that the server is able to access and inspect local model udpates imposes privacy risks on private data in the face of deep leakage type of gradient attacks [1];  this serious privacy risk defeats the purpose of federated learning in the first place;  authors are advised to reconsider the threat model and take into account various privacy-preserving mechanisms such as Differential privacy or Homomorphic Encryption; \n\n[1] Deep Leakage from Gradients, NeurIPS 2019; \n\n- the proposed method evluates local model udpates based on the agreement with the reverse-engineered trigger sets and target classes; nevertheless, the approach does not make sense at all since the target class selected by the reverse-engineering process is by no means the same as the target class might be selected by real backdoor attackers; the paper proposes to “find the backdoor trigger and target class such that the genuine scores for benign clients are large but they are small for compromised clients”， however, the compromised clients are a prior unknown, and the proposed method cannot reliably estimate the target class and solve this checken and egg problem; \n\n- the proposed method is inefficient in the evaluation of local model udpate (Line 4-5 in Algo. 1); this method cannot be efficiently applied to a federated learning scenario with large number of clients; that is probably why experiments reported in the paper only used 10 and 20 clients, which are far less than typcial cross-devices use cases with millions of clients; ", "strength_and_weaknesses": "\nSorry that I don't see any strength in the paper.  \n\nThe proposed method is seriously flawed in my view for following reasons: \n\n- the assumption that the server is able to access and inspect local model updates poses privacy leakage risks on private data in the face of deep leakage type of gradient attacks [1];  this serious privacy risk defeats the purpose of federated learning in the first place;  authors are advised to reconsider the threat model and take into account various privacy-preserving mechanisms such as Differential privacy or Homomorphic Encryption; \n\n[1] Deep Leakage from Gradients, NeurIPS 2019; \n\n- the proposed method evaluates local model updates based on the agreement with the reverse-engineered trigger sets and target classes; nevertheless, the approach does not make sense at all since the target class selected by the reverse-engineering process is by no means the same as the target class might be selected by real backdoor attackers; the paper proposes to “find the backdoor trigger and target class such that the genuine scores for benign clients are large but they are small for compromised clients”， however, the compromised clients are a prior unknown, and the proposed method cannot reliably estimate the target class and solve this chicken and egg problem; \n\n- the proposed method is inefficient in the evaluation of local model update (Line 4-5 in Algo. 1); this method cannot be efficiently applied to a federated learning scenario with large number of clients; that is probably why experiments reported in the paper only used 10 and 20 clients, which are far less than typical cross-devices use cases with millions of clients; ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "3_liN7Z4Jcu", "reviewer": "Reviewer_TbmV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a minimax game model of backdoor attacks between attackers\nand defenders in the federated learning (FL) paradigm.  Based on the analysis\nof this model, it uses a reverse-engineering technique to defend against\nbackdoor attacks in the FL process. It provides theoretical analysis and\nexperimental results to present the effectiveness of its defense mechanism.\n", "summary_of_the_review": "This paper makes strong assumptions about attackers, and it lacks technical novelty (proposed defense).\n", "strength_and_weaknesses": "It is an interesting angle to look at the federated learning backdoor problem.\nThe analysis is well presented.\n\nIts discussion is limited to strong assumptions about the adversary.\nThe theoretical analysis makes impractical assumptions about backdoor attacks.\nThe proposed defense has limited novelty.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject