Title: FedGame: A Game-Theoretic Defense against Backdoor Attacks in Federated Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: backdoor defense

federated learning

game theory
Abstract: Federated learning (FL) provides a distributed training paradigm where multiple clients can jointly train a global model without sharing their local data. However, recent studies have shown that FL offers an additional surface for backdoor attacks. For instance, an attacker can compromise a subset of clients and thus corrupt the global model to misclassify an input with a backdoor trigger as the adversarial target. Existing defenses for FL against backdoor attacks usually detect and exclude the corrupted information from the compromised clients based on a static attacker model. However, such defenses are inadequate against dynamic attackers who strategically adapt their attack strategies. To bridge this gap, we model the strategic interactions between the defender and dynamic attackers as a minimax game. Based on the analysis of the game, we design an interactive defense mechanism FedGame. We prove that under mild assumptions, the global model trained with FedGame under backdoor attacks is close to that trained without attacks. Empirically, we compare FedGame with multiple state-of-the-art baselines on several benchmark datasets under various attacks. We show that FedGame can effectively defend against strategic attackers and achieves significantly higher robustness than baselines. Our code is available at: https://github.com/AI-secure/FedGame.

FedGame: A Game-Theoretic Defense against Backdoor Attacks in Federated Learning

Abstract

Federated learning (FL) provides a distributed training paradigm where multiple clients can jointly train a global model without sharing their local data. However, recent studies have shown that FL offers an additional surface for backdoor attacks. For instance, an attacker can compromise a subset of clients and thus corrupt the global model to misclassify an input with a backdoor trigger as the adversarial target. Existing defenses for FL against backdoor attacks usually detect and exclude the corrupted information from the compromised clients based on a static attacker model. However, such defenses are inadequate against dynamic attackers who strategically adapt their attack strategies. To bridge this gap, we model the strategic interactions between the defender and dynamic attackers as a minimax game. Based on the analysis of the game, we design an interactive defense mechanism FedGame. We prove that under mild assumptions, the global model trained with FedGame under backdoor attacks is close to that trained without attacks. Empirically, we compare FedGame with multiple state-of-the-art baselines on several benchmark datasets under various attacks. We show that FedGame can effectively defend against strategic attackers and achieves significantly higher robustness than baselines. Our code is available at: https://github.com/AI-secure/FedGame .

1 Introduction

Federated learning (FL) [21, 28] aims to train a global model over training data that are distributed across multiple clients (e.g., mobile phones, IoT devices) in an iterative manner. In each communication round, a cloud server shares its global model with selected clients. Then, each selected client uses the global model to initialize its local model, utilizes its local training dataset to train the local model, and sends the local model update to the server. Finally, the server uses an aggregation rule to integrate local model updates from clients to update its global model.

Due to the distributed nature of FL, many studies [3, 4, 5, 21, 44] have shown that FL is vulnerable to backdoor attacks [11, 18, 26, 27, 30, 33, 37, 40, 41, 55, 59]. For instance, an attacker can compromise

a subset of clients and manipulate their local training datasets to corrupt the global model such that it predicts an attacker-chosen target class for any inputs embedded with a backdoor trigger [3]. To defend against such backdoor attacks, many defenses [7, 32, 34, 36, 39, 51] have been proposed (see Section 2 for details). However, all those defenses consider a static attack model where an attacker sticks to a fixed strategy and does not adapt its attack strategies. As a result, they are less effective under adaptive attacks, e.g., Wang et al. [44] showed that defenses in [6, 39] can be bypassed by appropriately designed attacks. While the vulnerability of the FL against dynamic or adaptive attack is known, dynamic defense has not been well-studied yet.

In this work, we propose FedGame, a game-theoretic defense against backdoor attacks in FL. Specifically, we formulate FedGame as a minimax game between the server (defender) and attacker, enabling them to adapt their defense and attack strategies strategically. While the server has no prior knowledge about which client is compromised, our key idea is that the server can compute a genuine score for each client, which should be large (or small) if the client is benign (or compromised) in each communication round. The genuine score serves as a weight for the local model update of the client when used to update the global model. The goal of the defender is to minimize the genuine scores for compromised clients and maximize them for benign ones. To solve the resulting minimax game for the defender, we follow a three-step process consisting of 1) building an auxiliary global model, 2) exploiting the auxiliary global model to reverse engineer a backdoor trigger and target class, and 3) testing whether the local model of a client will predict an input embedded with the reverse engineered backdoor trigger as the target class to compute a genuine score for the client. Based on the deployed defense, the goal of the attacker is to optimize its attack strategy by maximizing the effectiveness of the backdoor attack while remaining stealthy. Our key observation is that the attack effectiveness is determined by two factors: the genuine score and the local model of the client. We optimize the attack strategy with respect to those two factors to maximize the effectiveness of backdoor attacks against dynamic defense.

We perform both theoretical analysis and empirical evaluations for FedGame. Theoretically, we prove that the global model trained with our defense under backdoor attacks is close to that trained without attacks (measured by Euclidean distance of model parameters). Empirically, we perform comprehensive evaluations. In particular, we evaluate FedGame on benchmark datasets to demonstrate its effectiveness under state-of-the-art backdoor attacks [3, 54, 59]. Moreover, we compare it with six state-of-the-art baselines [6, 7, 36, 39, 56]. Our results indicate that FedGame outperforms them by a significant margin (e.g., the attack success rate for FedGame is less than 13% while it is greater than 95% for those baselines on CIFAR10 under Scaling attack [3]). We also perform comprehensive ablation studies and evaluate FedGame against adaptive attacks. Our results show FedGame is consistently effective. Our key contributions are as follows:

We propose FedGame, the first game-theoretic defense against dynamic backdoor attacks to FL.

We provide theoretical guarantees of FedGame. We show that the global model trained with FedGame under backdoor attacks is close to that without attacks.

We perform a systematic evaluation of FedGame on benchmark datasets and demonstrate that FedGame significantly outperforms state-of-the-art baselines.

2 Related Work

Backdoor Attacks in FL. In backdoor attacks against FL [3, 4, 5, 31, 44, 50, 59], an attacker aims to make a global model predict a target class for any input embedded with a backdoor trigger via compromised clients. For instance, Bagdasaryan et al. [3] proposed the scaling attack in which an attacker first uses a mix of backdoored and clean training examples to train its local model and then scales the local model update by a factor before sending it to the server. In our work, we will leverage state-of-the-art attacks [3, 54, 59] to perform strategic backdoor attacks against our defense.

Defenses against Backdoor Attacks in FL. Many defenses [39, 7, 34, 51, 36, 32, 57, 9] were proposed to mitigate backdoor attacks in FL. For instance, Sun et al. [39] proposed norm-clipping, which clips the norm of the local model update of a client to a certain threshold. Some work extended differential privacy [15, 1, 29] to mitigate backdoor attacks to federated learning. The idea is to clip the local model update and add Gaussian noise. Cao et al. [7] proposed FLTrust, which leveraged the similarity of the local model update of a client with that computed by the server itself on its

clean dataset. DeepSight [36] performs clustering on local models and then filters out those from compromised clients. By design, DeepSight is ineffective when the fraction of compromised clients is larger than 50%. Other defenses include Byzantine-robust FL methods such as Krum [6], Trimmed Mean [56], and Median [56]. However, all of those defenses only consider a static attacker model. As a result, they become less effective against dynamic attackers who strategically adapt their attack strategies.

Another line of research focuses on detecting compromised clients [24, 58]. As those defenses need to collect many local model updates from clients to make confident detection, the global model may already be backdoored before those clients are detected. Some recent studies [8, 53, 57, 10] proposed certified defenses against compromised clients. However, they can only tolerate a moderate fraction of compromised clients (e.g., less than 10%) as shown in their results. In contrast, FedGame can tolerate a much higher fraction (e.g., 80%) of compromised clients.

In this work, we aim to prevent backdoor attacks to federated learning, i.e., our goal is to train a robust global model under compromised clients. The server could also utilize post-training defenses [43, 51] to remove the backdoor in a backdoored global model. Prevention-based defense could be combined with those post-training defenses to form a defense-in-depth. We note that the defense proposed in [43] requires the server to own some clean samples that have the same distribution as the local training data of clients. The defense proposed in [51] prunes filters based on pruning sequences collected from clients and thus is less effective when the fraction of compromised clients is large.

3 Background and Threat Model

Federated Learning. Let S denote the set of clients and D i denote the local training dataset of client i ∈ S . In the t t h communication round, the server first sends the current global model Θ t to the selected clients. Then, each selected client i trains a local model (denoted as Θ t i ) by finetuning the global model Θ t using its local training dataset D i . For simplicity, we use z = ( x , y ) to denote a training example in D i , where x is the training input (e.g., an image) and y is its ground truth label. Given D i and the global model Θ t , the local objective is defined as L ( D i ; Θ t ) = 1 |D i | ∑ z ∈D i ℓ ( z ; Θ t ) where ℓ is a loss function (e.g., cross-entropy loss). The client can use gradient descent to update its local model based on the global model and its local training dataset, i.e., Θ t i = Θ t -η l ∂ L ( D i ;Θ t ) ∂ Θ t , where η l is the local learning rate. Note that stochastic gradient descent can also be used in practice. After that, the client sends g t i = Θ t i -Θ t (called local model update ) to the server. Note that it is equivalent for the client to send a local model or local model update to the server as Θ t i = Θ t + g t i . After receiving the local model updates from all clients, the server can aggregate them based on an aggregation rule R (e.g., FedAvg [28]) to update its global model, i.e., we have Θ t +1 = Θ t + η R ( g t 1 , g t 2 , · · · , g t |S| ) , where |S| represents the number of clients and η is the learning rate of the global model.

Threat Model. We consider attacks proposed in previous work [3, 54, 59]. We assume an attacker can compromise a set of clients (denoted as S a ). To perform backdoor attacks, the attacker first selects an arbitrary backdoor trigger δ and a target class y tc . For each client i ∈ S a in the t t h ( t = 1 , 2 , . . . ) communication round, the attacker can choose an arbitrary fraction (denoted as r t i ) of training examples from the local training dataset of the client, embed the backdoor trigger δ to training inputs, and relabel them as the target class y tc . In state-of-the-art backdoor attacks [3, 54, 59], the attacker leverages those trigger-embedded training examples to inject the backdoor into the local model of a compromised client to attack the global model.

Following [7], we consider that the server itself has a small clean training dataset (denoted as D s ), which could be collected from the same or different domains of the local training datasets of clients. Note that the server does not have 1) information on compromised clients, and 2) the poisoning ratio r t i ( i ∈ S a ), backdoor trigger δ , and target class y tc selected by the attacker. In our threat model, we do not make any other assumptions for the server compared with standard federated learning [28].

4 Methodology

Overview. We formulate FedGame as a minimax game between the defender and attacker, which enables them to optimize their strategies respectively. In particular, the defender computes a genuine

score for each client in each communication round. The goal of the defender is to maximize the genuine score for a benign client and minimize it for a compromised one. Given the genuine score for each client, we use a weighted average over all the local model updates to update the global model, i.e., we have

where p t i is the genuine score for client i in the t th communication round and η is the learning rate of the global model. We compute a weighted average of local models because we aim to achieve robustness against attacks while making minimal changes to FedAvg [28], which has been shown to achieve high utility in diverse settings. In our experiments, we empirically show that the genuine scores are almost constantly 0 for malicious clients after a few communication rounds, which indicates that the malicious model updates will not be aggregated at all, reducing Equation 1 to a robust FedAvg against attacks. (See Figure 2(c)(d) in the Appendix for more details.) Based on Equation 1, the effectiveness of the attack is determined by two components: genuine scores and local models of compromised clients. In our framework, the attacker optimizes the tradeoff between these two components to maximize the effectiveness of backdoor attacks.

4.1 Game Formulation

Computing Genuine Scores. The key challenge for the server to compute genuine scores for clients is that the server can only access their local model updates, i.e., the server can only leverage local models of clients to compute genuine scores. To tackle this issue, we observe that the local model of a compromised client is more likely to predict a trigger-embedded input as the target class. Therefore, the server can first reverse engineer the backdoor trigger δ re and target class y tc re (which we will discuss more in the next subsection) and then use them to compute the genuine score for each client. Since the client i sends its local model update g t i to the server, the server can compute the local model of the client i as Θ t i = Θ t + g t i . With Θ t i , the server can compute p t i as follows:

where ■ is an indicator function, D s is the clean training dataset of the server, x ⊕ δ re is a triggerembedded input, and G ( x ⊕ δ re ; Θ t i ) represents the label of x ⊕ δ re predicted by Θ t i . Intuitively, the genuine score for client i is small if a large fraction of inputs embedded with the reverse-engineered backdoor trigger is predicted as the target class by its local model. We note that the defender only needs to query local models to compute their genuine scores. Moreover, our defense is still applicable when differential privacy [14, 49] is used to protect the local models.

Optimization Problem of the Defender. The server aims to reverse engineer the backdoor trigger δ re and target class y tc re such that the genuine scores for compromised clients are minimized while those for benign clients are maximized. Formally, we have the following optimization problem:

Note that S a is the set of malicious clients that is unknown to the server. In the next subsection, we will discuss how to address this challenge to solve the optimization problem.

Optimization Problem of the Attacker. The goal of an attacker is to maximize its attack effectiveness. Based on Equation 1, the attacker needs to: 1) maximize the genuine scores for compromised clients while minimizing them for benign ones, i.e., max( ∑ i ∈S a p t i -∑ j ∈S\S a p t j ) , and 2) make the local models of compromised clients predict an input embedded with the attacker-chosen backdoor trigger δ as the target class y tc . To perform the backdoor attack in the t th communication round, the attacker embeds the backdoor to a certain fraction (denoted as r t i ) of training examples in the local training dataset of the client and uses them as data augmentation. Intuitively, a larger r t i encourages a higher attack success rate but can potentially result in a lower genuine score. In other words, r t i measures a tradeoff between these two factors. Formally, the attacker can solve the following

optimization problem to find the desired tradeoff:

where R t = { r t i | i ∈ S a } , δ is the trigger, and λ is a hyperparameter to balance the two terms.

Minimax Game. Given the optimization problems of the defender and attacker, we have the following minimax game:

Note that r t i ( i ∈ S a ) and δ are chosen by the attacker. Thus, we can add them to the objective function in Equation 3 without influencing its solution given the local model updates of clients. In Section A of Appendix, we provide an intuitive interpretation regarding how to connect our above objective to the ultimate goal of the defender and attacker (the defender aims to obtain a clean model, while the attacker wants the global model to be backdoored).

4.2 Solving the Minimax Game by the Defender

The key challenge for the server to solve Equation 5 is that it does not know S a (set of compromised clients). To address the challenge, our idea is to construct an auxiliary global model based on local models of all clients. Suppose g t i is the local model update sent by each client i ∈ S to the server. Our auxiliary global model is constructed as follows: Θ t a = Θ t + 1 |S| ∑ i ∈S g t i , where Θ t is the global model. Our intuition is that such a naively aggregated auxiliary global model is inclined to predict a trigger-embedded input as the target class under backdoor attacks. As a result, given the auxiliary global model, we can use an arbitrary existing method [42, 46, 38, 48] to reverse engineer the backdoor trigger and target class based on it, which enables us to compute genuine scores for clients based on Equation 2. With those genuine scores, we can use Equation 1 to update the global model to protect it from backdoor attacks in every communication round. The complete algorithm of our FedGame is shown in Algorithm 1 of Appendix.

Our framework is compatible with any trigger reverse engineering methods, allowing off-the-shelf incorporation of techniques developed for centralized supervised learning into federated learning. Note that developing a new reverse engineering method is not the focus of this work. Instead, our goal is to formulate the attack and defense against backdoor attacks on federated learning as a minimax game, which enables us to defend against dynamic attacks.

4.3 Solving the Minimax Game by the Attacker

The goal of the attacker is to find r t i for each client i ∈ S a such that the loss function in Equation 5 is maximized. As the attacker does not know the genuine scores of benign clients, the attacker can find r t i to maximize p t i + λr t i for client i ∈ S a to approximately solve the optimization problem in Equation 5. However, the key challenge is that the attacker does not know the reverse engineered backdoor trigger δ re and the target class y tc re of the defender to compute the genuine score for the client i . In response, the attacker can use the backdoor trigger δ and target class y tc chosen by itself. Moreover, the attacker reserves a certain fraction (e.g., 10%) of training data from its local training dataset D i as the validation dataset (denoted as D rev i ) to find the best r t i .

Estimating the Genuine Score Given r t i . For a given r t i , the client i can embed the backdoor to r t i fraction of training examples in D i \ D rev i and then use those backdoored training examples to augment D i \ D rev i to train a local model (denoted as ˜ Θ t i ). Then, the genuine score can be estimated as ˜ p t i = 1 -1 |D rev i | ∑ x ∈D rev i ■ ( G ( x ⊕ δ ; ˜ Θ t i ) = y tc ) , where G ( x ⊕ δ ; ˜ Θ t i ) is the predicted label by the global model ˜ Θ t i for the trigger-embedded input x ⊕ δ .

Finding the Optimal r t i . The client can use grid search to find r t i that achieves the largest ˜ p t i + λr t i . After estimating the optimal r t i , client i can embed the backdoor to r t i fraction of training examples and utilize them to perform backdoor attacks based on state-of-the-art methods [3, 54, 59].

Trigger Optimization. The attacker can choose an arbitrary static trigger to launch backdoor attacks, or dynamically optimize it to make the attack more effective. Given the fixed location and bounding box of the backdoor trigger, the attacker can optimize the trigger pattern with gradient descent such that it is more likely for a backdoored input to be predicted as the target class. For instance, the attacker can optimize the trigger pattern to minimize the cumulative loss on all training data of malicious clients, i.e., δ t = argmin δ ∗ ∑ x ∈∪ i ∈S a D i ℓ (Θ t ( x ⊕ δ ∗ ) , y tc ) , where Θ t ( x ⊕ δ ∗ ) is the output of the model which is a probability simplex of all possible classes and ℓ is a standard loss function (e.g., cross-entropy loss). The attacker can then solve Equation 5 with the optimized trigger. Recall that the attacker aims to find the largest ˜ p t i + λr t i . Therefore, if the resulting ˜ p t i + λr t i given δ t is lower than that given the previous trigger δ t -1 , we let δ t = δ t -1 .

The complete algorithm for the compromised clients is shown in Algorithm 2 in Appendix C.

5 Theoretical Analysis

This section provides a theoretical analysis of FedGame under backdoor attacks. Suppose the global model parameters are in a bounded space. We derive an upper bound for the L 2 -norm of the difference between the parameters of the global models with and without attacks. To analyze the robustness of FedGame, we make the following assumptions, which are commonly used in the analysis of previous studies [25, 45, 16, 35, 53, 7, 9] on federated learning.

Assumption 5.1. The loss function is µ -strongly convex with L -Lipschitz continuous gradient. Formally, we have the following for arbitrary Θ and Θ ′ :

where z is an arbitrary training example.

Assumption 5.2. Weassume the gradient ∇ Θ ℓ ( z ; Θ) is bounded with respect to L 2 -norm for arbitrary Θ and z , i.e., there exists some M ≥ 0 such that

Suppose Θ c is the global model trained by FedGame without any attacks in the t th communication round, i.e., each client i ∈ S uses its clean local training dataset D i to train a local model. Moreover, we assume gradient descent is used by each client to train its local model. Suppose q t i is the genuine score for client i without attacks. Moreover, we denote β t i = q t i ∑ i ∈S q t i as the normalized genuine score for client i . To perform the backdoor attack, we assume a compromised client i can embed the backdoor trigger to r t i fraction of training examples in the local training dataset of the client and relabel them as the target class. Those backdoored training examples are used to augment the local training dataset of the client. Suppose Θ t is the global model under the backdoor attack in the t th communication round with our defense. We denote α t i = p t i ∑ i ∈S p t i as the normalized genuine score for client i with attacks in the t th communication round. Formally, we have:

t

Lemma 5.3 (Robustness Guarantee for One Communication Round) . Suppose Assumptions 5.1 and 5.2 hold. Moreover, we assume (1 -r t ) β t i ≤ α t i ≤ (1 + r t ) β t i , where i ∈ S and r t = ∑ j ∈S a r t j . Then, we have:

where η is the learning rate of the global model, L and µ are defined in Assumption 5.1, γ t = ∑ i ∈S a α t i r t i M , and M is defined in Assumption 5.2.

Proof sketch. Our idea is to decompose ∥ ∥ Θ t +1 -Θ t +1 c ∥ ∥ 2 into two terms. Then, we derive an upper bound for each term based on the change of the local model updates of clients under backdoor attacks and the properties of the loss function. As a result, our derived upper bound relies on r t i for each client i ∈ S a , parameters µ , L , and M in our assumptions, as well as the parameter difference of the global models in the previous iteration, i.e., ∥ Θ t -Θ t c ∥ 2 . Our complete proof is in Appendix B.1.

In the above lemma, we derive an upper bound of ∥ ∥ Θ t +1 -Θ t +1 c ∥ ∥ 2 with respect to ∥ Θ t -Θ t c ∥ 2 for one communication round. In the next theorem, we derive an upper bound of ∥ Θ t -Θ t c ∥ 2 as t →∞ . We iterative apply Lemma 5.3 for successive values of t and have the following theorem:

Theorem 5.4 (Robustness Guarantee) . Suppose Assumptions 5.1 and 5.2 hold. Moreover, we assume (1 -r t ) β t i ≤ α t i ≤ (1 + r t ) β t i for i ∈ S , γ t ≤ γ and r t ≤ r hold for all communication round t , and µ > 2 γ , where r t = ∑ j ∈S a r t j and γ t = ∑ i ∈S a α t i r t i M . Let the global model learning rate by chosen as 0 < η < µ -2 γ L 2 +2 Lγ . Then, we have:

holds as t →∞ .

Proof sketch. Given the conditions that γ t ≤ γ and r t ≤ r as well as the fact that the right-hand side of Equation 10 is monotonic with respect to γ t and r t , we can replace γ t and r t in Equation 10 with γ and r . Then, we iterative apply the equation for successive values of t . When 0 < η < µ -2 γ L 2 +2 Lγ , we have 0 < 1 -ηµ +2 ηγ + η 2 L 2 +2 η 2 Lγ < 1 . By letting t →∞ , we can reach the conclusion. The complete proof can be found in Appendix B.2.

Our theorem implies that the global model parameters under our defense against adaptive attacks do not deviate too much from those of the global model without attack when the fraction of backdoored training examples r t is bounded.

6 Experiments

In order to thoroughly evaluate the effectiveness of FedGame, we conduct comprehensive experiments on 1) evaluation against three state-of-the-art backdoor attacks: Scaling attack [3], DBA [54], and Neurotoxin [59], 2) comparison with six existing baselines including Krum, Median, Norm-Clipping, Differential Privacy, DeepSight, FLTrust, 3) evaluation against strong adaptive attacks, and 4) comprehensive ablation studies.

6.1 Experimental Setup

Datasets and Models. We use two benchmark datasets: MNIST [23] and CIFAR10 [22] for FL tasks. MNIST has 60,000 training and 10,000 testing images, each of which has a size of 28 × 28 belonging to one of 10 classes. CIFAR10 consists of 50,000 training and 10,000 testing images with a size of 32 × 32. Each image is categorized into one of 10 classes. For each dataset, we randomly sample 90% of training data for clients, and the remaining 10% of training data is reserved to evaluate our defense when the clean training dataset of the server is from the same domain as those of clients. We use a CNN with two convolution layers (detailed architecture can be found in Table 2 in Appendix) and ResNet-18 [19] which is pre-trained on ImageNet [13] as the global models for MNIST and CIFAR10.

FL Settings. We consider two settings: local training data are independently and identically distributed (IID) among clients, and non-IID. We follow the previous work [17] to distribute training data to clients by using a parameter q to control the degree of non-IID, which models the probability that training images from one category are distributed to a particular client (or a set of clients). We set q = 0 . 5 by following [17]. Moreover, we train a global model based on 10 clients for 200 iterations with a global learning rate η = 1 . 0 . In each communication round, we use SGD to train the local model of each client for two epochs with a local learning rate of 0.01.

Attack Settings. We consider three state-of-the-art backdoor attacks on federated learning, i.e., Scaling attack [3], DBA [54], and Neurotoxin [59]. For Scaling attack, we set the scaling parameter to be #total clients/( η × #compromised clients) by following [3]. For Neurotoxin, we set the ratio of masked gradients to be 1%, following the choice in [59]. We use the same backdoor trigger and target class as used in those works. By default, we assume 60% of clients are compromised by an attacker. When the attacker solves the minimax game in Equation 5, we set the default λ = 1 . We explore the

impact of λ in our experiments. We randomly sample 10% of the local data of each compromised client as validation data to search for an optimal r t i . Moreover, we set the granularity of grid search to 0.1 when searching for r t i .

Baselines. We compare our defense with the following methods: FedAvg [28], Krum [6], Median [56], Norm-Clipping [39], Differential Privacy (DP) [39], DeepSight [36], and FLTrust [7]. Please see Appendix D.2 for parameter settings for those baselines.

Evaluation Metrics. We use testing accuracy (TA) and attack success rate (ASR) as evaluation metrics. Concretely, TA is the fraction of clean testing inputs that are correctly predicted, and ASR refers to the fraction of backdoored testing inputs that are predicted as the target class.

Defense Settings. We consider two settings: in-domain and out-of-domain. For the in-domain setting, we consider the clean training dataset of the server is from the same domain as the local training datasets of clients. We use the reserved data as the clean training dataset of the server for each dataset. For the out-of-domain setting, we consider the server has a clean training dataset that is from a different domain than the FL task. In particular, we randomly sample 6,000 images from FashionMNIST [52] for MNIST and sample 5,000 images from GTSRB [20] for CIFAR10 as the clean training dataset of the server. Unless otherwise mentioned, we adopt Neural Cleanse [42] to reverse engineer the backdoor trigger and target class.

6.2 Experimental Results

We show the results of FedGame compared with existing defenses under IID and non-IID settings in Table 1. We defer the results against Neurotoxin to Appendix D.3 due to space limitations. We have the following observations from the experimental results. First, FedGame outperforms all existing defenses in terms of ASR. In particular, FedGame can reduce ASR to random guessing (i.e., ASR of FedAvg under no attacks) in both IID and non-IID settings for clients as well as both in-domain and out-of-domain settings for the server. Intrinsically, FedGame performs better because our gametheoretic defense enables the defender to optimize its strategy against dynamic, adaptive attacks. We

note that FLTrust outperforms other defenses (except FedGame) in most cases since it exploits a clean training dataset from the same domain as local training datasets of clients. However, FLTrust is not applicable when the server only holds an out-of-domain clean training dataset, while FedGame can relax such an assumption and will still be applicable. Moreover, our experimental results indicate that FedGame achieves comparable performance even if the server holds an out-of-domain clean training dataset.

To further understand our results,we visualize the average genuine (or trust) scores computed by FedGame (or FLTrust) for compromised and benign clients in Appendix D.4. In particular, we find that the genuine scores produced by FedGame are much lower than those produced by FLTrust for compromised clients, which explains why FedGame outperforms FLTrust. Second, FedGame achieves comparable TA with existing defenses, indicating that FedGame preserves the utility of global models.

Furthermore, we show the comparison results of FedGame with existing defenses against the scaling attack when the total number of clients is 30 in Table 4 in Appendix D.5. Our observations are similar, which indicates that FedGame consistently outperforms existing defenses under different numbers of clients and backdoor attacks.

Impact of λ . λ is a hyperparameter used by an attacker (see Eqn. 4) when searching for the optimal r t i for each compromised client i in each communication round t . Figure 1(a) shows the impact of λ on ASR of FedGame. The results show that FedGame is insensitive to different λ 's. The reason is that the genuine score for a compromised client is small when λ is large, and the local model of a compromised client is less likely to predict a trigger-embedded input as the target class when λ is small. As a result, backdoor attacks with different λ are ineffective under FedGame.

0

.

6

0

.

8

1

0

4

8

12

16

20

λ

ASR (%)

FedGame

(a)

0

.

2

0

.

4

0

.

6

0

.

8

0

20

40

60

80

100

fraction of malicious clients

ASR (%)

FLTrust

FedGame

(b)

0.2

0.4

0.6

0.8

1.0

0

4

8

12

16

20

fraction of participants per round

ASR (%)

FLTrust

FedGame

(c)

2

×

2

4

×

4

6

×

6

0

20

40

60

80

100

trigger size

ASR (%)

FLTrust

FedGame

(d)

Impact of the Fraction of Compromised Clients. Figure 1(b) shows the impact of the fraction of compromised clients on ASR. As the results show, FedGame is effective for a different fraction of compromised clients in both in-domain and out-of-domain settings. In contrast, FLTrust is ineffective when the fraction of compromised clients is large. For instance, FedGame can achieve 9.84% (indomain) and 10.12% (out-of-domain) ASR even if 80% of clients are compromised on MNIST. Under the same setting, the ASR of FLTrust is 99.95%, indicating that the defense fails.

Impact of Client Selection. By default, we consider that all clients are selected in each communication round. We also consider only a subset of clients are selected in each communication round by the server. Figure 1(c) shows our experimental results. Our results suggest that our defense is effective and consistently outperforms FLTrust.

Computation Cost. FedGame computes a genuine score for each client in each communication round. Here we demonstrate its computational efficiency. On average, it takes 0.148s to compute a genuine score for each client in each communication round on a single NVIDIA 2080 Ti GPU. We note that the server could from a resourceful tech company (e.g., Google, Meta, Apple), which would have enough computation resources to compute it for millions of clients. Moreover, those local models can be evaluated in parallel.

Performance under Static Attacks. In our evaluation, we consider an attacker optimizing the fraction of backdoored training examples. We also evaluate FedGame under existing attacks where the attacker does not optimize it. Under the default setting, FedGame can achieve an ASR of 9.75%, indicating that our defense is effective under static attack.

Furthermore, we discuss the impact of trigger reverse engineering methods, the total number of clients, and the size of clean data of the server in Appendix D.6.

6.3 Adaptive Attacks

In this subsection, we discuss some potential adaptive strategies that may be leveraged by the attacker to evade our defense.

Data Replacing Attack. By default, we consider an attacker optimizing the fraction of backdoored training examples added to the local training dataset of a compromised client to maximize backdoor effectiveness. We also consider an attacker who replaces a fraction of the local training data with backdoored samples and optimizes such fraction. Under our default setting, the ASR of FedGame is 9.71% and 10.69% on MNIST and CIFAR10, respectively, indicating that our defense is still effective under data replacing attacks.

Variation in Triggers. We demonstrate that our defense is robust to variation in triggers. In particular. we try triggers with sizes 2 × 2 , 4 × 4 , and 6 × 6 under the default setting. Figure 1(d) compares FedGame with FLTrust. The results demonstrate that FedGame is consistently effective under triggers with different sizes.

We note that although an attacker can slightly manipulate the parameters of local models such that they are more similar to those of benign clients, FedGame does not rely on model parameters for detection. Instead, FedGame leverages the model behaviors, i.e., whether the model predicts inputs with our reverse-engineered trigger as the target class. As a result, our defense would still be effective even if the change in the model parameters is small as long as the model has backdoor behavior (this is required to make the attack effective). This is also the reason why our defense is better than existing methods such as FLTrust which leverages model parameters for defense.

7 Conclusion and Future Work

In this work, we propose FedGame, a general game-theory based defense against adaptive backdoor attacks in federated learning. Our formulated minimax game enables the defender and attacker to dynamically optimize their strategies. Moreover, we respectively design solutions for both of them to solve the minimax game. We perform theoretical analysis and empirical evaluations for our framework. Our results demonstrate the effectiveness of FedGame under strategic backdoor attackers. Moreover, FedGame achieves significantly higher robustness than baselines in different settings.

We consider that an attacker could optimize the poisoning ratio and trigger pattern in our work. We believe it is an interesting future work to consider other factors for the attacker (e.g., trigger size, consideration of long-term goal, design of new loss functions [12], poisoned neurons selection [2]). Moreover, we consider a zero-sum Stackelberg game in this work. Another interesting future work is to consider other game formulations, e.g., Bayesian games.

References

Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In ACM SIGSAC Conference on Computer and Communications Security , pages 308-318, 2016.

Manaar Alam, Esha Sarkar, and Michail Maniatakos. Perdoor: Persistent non-uniform backdoors in federated learning using adversarial perturbations. arXiv preprint arXiv:2205.13523 , 2022.

Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics , pages 2938-2948. PMLR, 2020.

Gilad Baruch, Moran Baruch, and Yoav Goldberg. A little is enough: Circumventing defenses for distributed learning. Advances in Neural Information Processing Systems , 32, 2019.

Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing federated learning through an adversarial lens. In International Conference on Machine Learning , pages 634-643. PMLR, 2019.

Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. Advances in Neural Information Processing Systems , 30, 2017.

Xiaoyu Cao, Minghong Fang, Jia Liu, and Neil Zhenqiang Gong. FLtrust: Byzantine-robust federated learning via trust bootstrapping. In Network and Distributed Systems Security Symposium , 2021.

Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. Provably secure federated learning against malicious clients. In AAAI Conference on Artificial Intelligence , volume 35, pages 6885-6893, 2021.

Xiaoyu Cao, Jinyuan Jia, Zaixi Zhang, and Neil Zhenqiang Gong. Fedrecover: Recovering from poisoning attacks in federated learning using historical information. In 2023 IEEE Symposium on Security and Privacy (SP) , pages 1366-1383. IEEE, 2023.

Xiaoyu Cao, Zaixi Zhang, Jinyuan Jia, and Neil Zhenqiang Gong. Flcert: Provably secure federated learning against poisoning attacks. IEEE Transactions on Information Forensics and Security , 17:3691-3705, 2022.

Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526 , 2017.

Yanbo Dai and Songze Li. Chameleon: Adapting to peer images for planting durable backdoors in federated learning. arXiv preprint arXiv:2304.12961 , 2023.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Cynthia Dwork. Differential privacy. In Automata, Languages and Programming: 33rd International Colloquium, ICALP 2006, Venice, Italy, July 10-14, 2006, Proceedings, Part II 33 , pages 1-12. Springer, 2006.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography Conference , pages 265-284. Springer, 2006.

Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta-learning approach. In Conference on Neural Information Processing Systems , 2020.

Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Gong. Local model poisoning attacks to Byzantine-Robust federated learning. In 29th USENIX Security Symposium , pages 1605-1622, 2020.

Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733 , 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

Sebastian Houben, Johannes Stallkamp, Jan Salmen, Marc Schlipsing, and Christian Igel. Detection of traffic signs in real-world images: The German Traffic Sign Detection Benchmark. In International Joint Conference on Neural Networks , number 1288, 2013.

Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14(1-2):1-210, 2021.

Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.

Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist , 2, 2010.

Suyi Li, Yong Cheng, Wei Wang, Yang Liu, and Tianjian Chen. Learning to detect malicious clients for robust federated learning. arXiv preprint arXiv:2002.00211 , 2020.

Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of FedAvg on non-iid data. In International Conference on Learning Representations , 2020.

Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 16463-16472, 2021.

Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In NDSS , 2017.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics , pages 1273-1282. PMLR, 2017.

H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In International Conference on Learning Representations , 2017.

Anh Nguyen and Anh Tran. Wanet-imperceptible warping-based backdoor attack. arXiv preprint arXiv:2102.10369 , 2021.

Thien Duc Nguyen, Phillip Rieger, Huili Chen, Hossein Yalame, Helen Möllering, Hossein Fereidooni, Samuel Marchal, Markus Miettinen, Azalia Mirhoseini, Shaza Zeitouni, et al. Flame: Taming backdoors in federated learning. Cryptology ePrint Archive , 2021.

Thien Duc Nguyen, Phillip Rieger, Huili Chen, Hossein Yalame, Helen Möllering, Hossein Fereidooni, Samuel Marchal, Markus Miettinen, Azalia Mirhoseini, Shaza Zeitouni, et al. FLAME: Taming backdoors in federated learning. In 31st USENIX Security Symposium , pages 1415-1432, 2022.

Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. Advances in Neural Information Processing Systems , 33:3454-3464, 2020.

Mustafa Safa Ozdayi, Murat Kantarcioglu, and Yulia R Gel. Defending against backdoors in federated learning with robust learning rate. In AAAI Conference on Artificial Intelligence , volume 35, pages 9268-9276, 2021.

Amirhossein Reisizadeh, Farzan Farnia, Ramtin Pedarsani, and Ali Jadbabaie. Robust federated learning: The case of affine distribution shifts. Advances in Neural Information Processing Systems , 33:21554-21565, 2020.

Phillip Rieger, Thien Duc Nguyen, Markus Miettinen, and Ahmad-Reza Sadeghi. DeepSight: Mitigating backdoor attacks in federated learning through deep model inspection. In Network and Distributed System Security Symposium , 2022.

Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 11957-11965, 2020.

Mingjie Sun and Zico Kolter. Single image backdoor inversion via robust smoothed classifiers. arXiv preprint arXiv:2303.00215 , 2023.

Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H Brendan McMahan. Can you really backdoor federated learning? arXiv preprint arXiv:1911.07963 , 2019.

Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Label-consistent backdoor attacks. arXiv preprint arXiv:1912.02771 , 2019.

Akshaj Veldanda and Siddharth Garg. On evaluating neural network backdoor defenses. arXiv preprint arXiv:2010.12186 , 2020.

Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural Cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy , pages 707-723. IEEE, 2019.

Hang Wang, Zhen Xiang, David J Miller, and George Kesidis. Universal post-training backdoor detection. arXiv preprint arXiv:2205.06900 , 2022.

Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy-yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos. Attack of the tails: Yes, you really can backdoor federated learning. Advances in Neural Information Processing Systems , 33:16070-16084, 2020.

Jianyu Wang and Gauri Joshi. Cooperative SGD: A unified framework for the design and analysis of communication-efficient SGD algorithms. Journal of Machine Learning Research , 22(213):1-50, 2021.

Zhenting Wang, Kai Mei, Hailun Ding, Juan Zhai, and Shiqing Ma. Rethinking the reverseengineering of trojan triggers. In Advances in Neural Information Processing Systems , 2022.

Zhenting Wang, Kai Mei, Hailun Ding, Juan Zhai, and Shiqing Ma. Rethinking the reverseengineering of trojan triggers. arXiv preprint arXiv:2210.15127 , 2022.

Zhenting Wang, Kai Mei, Juan Zhai, and Shiqing Ma. Unicorn: A unified backdoor trigger inversion framework. In The Eleventh International Conference on Learning Representations , 2023.

Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H Yang, Farhad Farokhi, Shi Jin, Tony QS Quek, and H Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security , 15:34543469, 2020.

Yuxin Wen, Jonas Geiping, Liam Fowl, Hossein Souri, Rama Chellappa, Micah Goldblum, and Tom Goldstein. Thinking two moves ahead: Anticipating other users improves backdoor attacks in federated learning. arXiv preprint arXiv:2210.09305 , 2022.

Chen Wu, Xian Yang, Sencun Zhu, and Prasenjit Mitra. Mitigating backdoor attacks in federated learning. arXiv preprint arXiv:2011.01767 , 2020.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Chulin Xie, Minghao Chen, Pin-Yu Chen, and Bo Li. CRFL: Certifiably robust federated learning against backdoor attacks. In 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 11372-11382. PMLR, 2021.

Chulin Xie, Keli Huang, Pin-Yu Chen, and Bo Li. DBA: Distributed backdoor attacks against federated learning. In International Conference on Learning Representations , 2020.

Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Y Zhao. Latent backdoor attacks on deep neural networks. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security , pages 2041-2055, 2019.

Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantine-robust distributed learning: Towards optimal statistical rates. In International Conference on Machine Learning , pages 5650-5659. PMLR, 2018.

Kaiyuan Zhang, Guanhong Tao, Qiuling Xu, Siyuan Cheng, Shengwei An, Yingqi Liu, Shiwei Feng, Guangyu Shen, Pin-Yu Chen, Shiqing Ma, et al. Flip: A provable defense framework for backdoor mitigation in federated learning. arXiv preprint arXiv:2210.12873 , 2022.

Zaixi Zhang, Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. FLDetector: Defending federated learning against model poisoning attacks via detecting malicious clients. In 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2545-2555, 2022.

Zhengming Zhang, Ashwinee Panda, Linyue Song, Yaoqing Yang, Michael Mahoney, Prateek Mittal, Ramchandran Kannan, and Joseph Gonzalez. Neurotoxin: Durable backdoors in federated learning. In International Conference on Machine Learning , pages 26429-26446. PMLR, 2022.

A Interpretation of the Minimax Objective

We note that the ultimate goal of the defender is to obtain a clean model while the attacker wants the global model to be poisoned. In other words, the server (or attacker) wishes the attack success rate of the global model to be small (large). It is very challenging to achieve this objective directly because 1) the server does not know the exact trigger and target class used by the attacker, and 2) the attacker needs to adapt its attack strategy based on the server's strategy. To address the challenge, we consider an alternative goal (i.e., our objective in Equation 5, where we wish to assign small weights to compromised clients when performing a weighted average to update the global model. Our idea is that when the weights for compromised clients are very small, our global model is less likely to be affected by the attack. As a result, the attacker's strategy is to maximize the genuine scores for compromised clients while ensuring their local models are backdoored to maximize the attack effectiveness for the global model. Thus, our objective in Equation 5 could translate to the ultimate goal. Our empirical results show the effectiveness of our defense.

B Complete Proofs

B.1 Proof of Lemma 5.3

We first present some preliminary lemmas that will be invoked for proving Lemma 5.3.

Lemma B.1. Suppose D i is the clean local training dataset of the client i . An attacker can inject the backdoor trigger to r t i fraction of training examples in D i and relabel them as the target class. We use D ′ i to denote the set of backdoored training examples where r t i = |D ′ i | |D i | . Given two arbitrary Θ and Θ c , we let g i = 1 |D i ∪D ′ i | ∇ Θ ∑ z ∈D i ∪D ′ i ℓ ( z ; Θ) and h i = 1 |D i | ∇ Θ c ∑ z ∈D i ℓ ( z ; Θ c ) . We then have that

Proof. We first prove Equation 12. We have the following relations:

where Equation 23 holds based on the fact that -2 r t i M ∥ Θ -Θ c ∥ 2 ≥ -r t i M ∥ Θ -Θ c ∥ 2 2 -r t i M for ∀ r t ≥ 0 and ∀ M ≥ 0 .

i

In the following, we prove inequality 13. We have that

where Equation 28 is due to Assumption 5.1 and 5.2.

Given Lemma B.1, we prove Lemma 5.3 as follows. Recall that we have α t i = p t i ∑ i ∈ S p t i and β t i = q t i ∑ i ∈ S q t i .

Next, we respectively derive an upper bound for the first and second terms in Equation 34. To derive the upper bound for the first term, we have that

where S 1 = ∥ Θ t -Θ t c ∥ 2 2 , S 2 = -2 η (Θ t -Θ t c ) T ( ∑ i ∈S α t i ( g t i -h t i )) , and S 3 = η 2 ∥ ∥ ∑ i ∈S α t i ( g t i -h t i ) ∥ ∥ 2 2 . Next, we will bound S 2 and S 3 . We denote γ t = ∑ i ∈S a α t i r t i M . Note that we have γ t = ∑ i ∈S α t i r t i M since r t i = 0 for ∀ i ∈ S \ S a . We bound S 2 as follows.

S

2

where inequality 39 holds by Lemma B.1 and the fact that η, α t i ≥ 0 . We bound S 3 as follows.

where Equation 47 is based on the fact that 4 γ t L ∥ Θ -Θ c ∥ 2 ≤ 2 γ t L ∥ Θ -Θ c ∥ 2 2 + 2 γ t L when γ t L ≥ 0 .

Given the upper bounds of S 2 and S 3 , we can bound ∥ ∥ Θ t -Θ t c -η ∑ i ∈S α t i ( g t i -h t i ) ∥ ∥ 2 2 as follows.

Next, we will derive an upper bound for ∥ ∥ η ∑ i ∈S ( β t i -α t i ) h t i ∥ ∥ 2 . We denote r t = ∑ i ∈S a r t i . Note that we have that r t = ∑ i ∈S r t i also holds since r t i = 0 for ∀ i ∈ S \ S a . Given the assumption that (1 -r t ) α t i ≤ β t i ≤ (1 + r t ) α t i , we have

where the first inequality is due to triangle inequality and the second inequality is based on the assumption that ∥ h t i ∥ 2 ≤ M . Therefore, we have:

where the last inequality holds due to the fact that a + b ≤ √ a + b for ∀ a ≥ 0 and ∀ b ≥ 0 , which completes our proof for Lemma 5.3.

B.2 Proof of Theorem 5.4

Then, we can iteratively apply the above equation to prove our theorem. In particular, we have:

∥

t

t

∥

.

When the learning rate satisfies 0 < η < µ -2 γ L 2 +2 Lγ , we have that 0 < 1 -ηµ +2 ηγ + η 2 L 2 +2 η 2 Lγ < 1 Therefore, the upper bound becomes √ 2 ηγ (1+ ηL +2 ηγ )+2 ηrM 1 -√ 1 -ηµ +2 ηγ + η 2 L 2 +2 η 2 Lγ as t →∞ . Hence, we prove our Theorem 5.4.

C Complete Algorithms

C.1 Complete Algorithm of FedGame

Algorithm 1 shows the complete algorithm of FedGame. In Line 3, we construct an auxiliary global model. In Line 4, the function REVERSEENGINEER is used to reverse engineer the backdoor trigger

and target class. In Line 6, we compute the local model of client i based on its local model update. In Line 7, we compute a genuine score for client i . In Line 9, we update the global model based on genuine scores and local model updates of clients.

Algorithm 1 FEDGAME

Input: Θ (global model in the t communication round), g i , i ∈ S (local model updates of clients), D (clean training dataset of server), η (learning rate of global model).

C.2 Complete Algorithm for a Compromised Client

Algorithm 2 shows the complete algorithm for a compromised client. In Line 3, we randomly subsample ρ i fraction of training data from D i . In Line 7, the function CREATEBACKDOOREDDATA is used to generate backdoored training examples by embedding the backdoor trigger δ to ⌊ min( j ∗ ζ, 1) |D i \ D rev i |⌋ training examples in D i \ D rev i and relabel them as y tc , where | · | measures the number of elements in a set. In Line 8, the function TRAININGLOCALMODEL is used to train the local model on the training dataset D ′ i ∪ D i \ D rev i . In Line 9, we estimate a genuine score. In Line 15, we use the function CREATEBACKDOOREDDATA to generate backdoored training examples by embedding the backdoor trigger δ to ⌊ min( o ∗ ζ, 1) |D i |⌋ training examples in D i and relabel them as y tc . In Line 16, we use the function TRAININGLOCALMODEL to train a local model and utilize existing state-of-the-art attacks to inject the backdoor based on the training dataset D ′ i ∪ D i .

Algorithm 2 ALGORITHM FOR A COMPROMISED CLIENT

Input: Θ (global model in the t communication round), D i (local training dataset of client i ), ρ i (fraction of reserved data to find optimal r t i ), ζ (granularity of searching for r t i ), δ (backdoor trigger), y tc (target class), and λ (hyperparameter)

17:

D Additional Experimental Setup and Results

D.1 Architecture of Global Model

Table 2 shows the global model architecture on MNIST dataset.

D.2 Parameter Setting for Compared Baselines

Recall that we compare our defense with the following methods: FedAvg [28], Krum [6], Median [56], Norm-Clipping [39], Differential Privacy (DP) [39], DeepSight [36], and FLTrust [7]. FedAvg is non-robust while Krum and Median are two Byzantine-robust baselines. Norm-Clipping clips the L 2 -norm of local model updates to a given threshold T N . We set T N = 0 . 01 for MNIST and T N = 0 . 1 for CIFAR10. DP first clips the L 2 -norm of a local model update to a threshold T D and then adds Gaussian noise. We set T D = 0 . 05 for MNIST and T D = 0 . 5 for CIFAR10. We set the standard deviation of noise to be 0 . 01 for both datasets. In FLTrust, the server uses its clean dataset to compute a server model update and assigns a trust score to each client by leveraging the similarity between the server model update and the local model update. We set the clean training dataset of the server to be the same as FedGame in our comparison. Note that FLTrust is not applicable when the clean training dataset of the server is from a different domain from those of clients.

D.3 Performance of FedGame against Neurotoxin

In Table 3, we compare our FedGame with other defense baselines against Neurotoxin [59]. We can observe that our FedGame is consistently more effective than existing defenses. Our observation is consistent with the experimental results for Scaling attack and DBA attack in Table 1.

D.4 Visualization of Genuine Score of FedGame and Trust Score of FLTrust

Our FedGame computes a genuine score for each client which quantifies the extent to which a client is benign in each communication round. Intuitively, our FedGame would be effective if the genuine score is small for a compromised client but is large for a benign one. FLTrust [7] computes a trust score for each client in each communication round. Similarly, FLTrust would be effective if the trust score is small for a compromised client but large for a benign one. Figure 2 visualizes the

average genuine or trust scores for compromised and benign clients of FedGame and FLTrust on the MNIST dataset. We have the following observations from the figures. First, the average genuine score computed by FedGame drops to 0 quickly for compromised clients. In contrast, the average trust score computed by FLTrust drops slowly. Second, the average genuine score computed by FedGame for benign clients first increases and then becomes stable. In contrast, the average genuine score computed by FLTrust for benign clients decreases as the number of iterations increases. As a result, FedGame outperforms FLTrust.

0

50

100

150

200

Rounds

0

0.2

0.4

0.6

0.8

1.0

Trust

Score

Benign

Clients

Compromised

Clients

(a) FLTrust (IID)

0

50

100

150

200

Rounds

0

0.2

0.4

0.6

0.8

1.0

Trust

Score

Benign

Clients

Compromised

Clients

(b) FLTrust (non-IID)

0

50

100

150

200

Rounds

0

0.2

0.4

0.6

0.8

1.0

Genuine

Score

Benign

Clients

Compromised

Clients

(c) FedGame (IID)

0

50

100

150

200

Rounds

0

0.2

0.4

0.6

0.8

1.0

Genuine

Score

Benign

Clients

Compromised

Clients

(d) FedGame (non-IID)

D.5 FedGame Performance in FL Consisting of 30 Clients

In Table 4, we report the performance of FedGame and baselines when the total number of clients is 30. The results also indicate that our FedGame outperforms all baselines in terms of ASR and achieves comparable TA with existing methods.

D.6 Additional Ablation Studies

Impact of Trigger Reverse Engineering. Our framework is compatible with any trigger reverse engineer methods. To study the impact of different trigger reverse engineering methods, we use FeatureRE [47] to substitute Neural Cleanse in FedGame. We adopt the public implementation of FeatureRE. Under the default setting, FedGame achieves a 10.36% ASR, which indicates that our framework is consistently effective with different trigger reverse engineering methods.

Besides, FedGame would still be effective even if the reverse-engineered backdoor trigger and target label are of low quality, as long as they can distinguish benign and compromised clients. We validate this by experiment. In particular, we use 20 iterations (originally, it's 100) when reverse engineering the trigger/target label under the default setting to obtain a low-quality trigger/target label. We find that genuine scores for malicious clients are still low. The final ASR is 13.23%, which means our FedGame is still effective.

Impact of the Total Number of Clients. We study the impact of the total number of clients for our FedGame under the default setting. In particular, we consider the total number of clients to be 10, 30,

50, 70, and 100, where the fraction of malicious clients is 60%. We show the experimental results in Table 5. Our experimental results show that our FedGame is effective for different number of clients on different datasets.

Impact of the Size of Server Clean Data. By default, we set the ratio between the number of clean examples of the server and the total number of examples of clients to be 0.1. We conduct experiments with different ratios: 0.01, 0.02, and 0.05 under the default setting. The corresponding ASRs are 9.71%, 12.38%, and 9.75%, indicating that FedGame is effective even when the server only has 1% clean data.

Trigger Optimization. We consider an attacker optimizes trigger pattern such that a backdoored input is more likely to be predicted as the target class. We perform experiments under the default setting. The ASR is 12.43%, which indicates that our FedGame is consistently effective for trigger optimization.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents FedGame, a game-theory based dynamic defense mechanism against backdoor attacks in Federated Learning (FL). FedGame adapts to attackers using a minimax game model. The authors theoretically prove that a model trained with FedGame is close to one trained without attacks, implying robustness against malicious disruptions. Empirical evaluations on benchmark datasets reveal that FedGame compares with existing six existing defenses, reducing attack success rate on MNIST and CIFAR10.\n\nSTRENGTHS:\n1. Backdoor defense in federated learning is an important problem.\n2. The paper provides theoretical analysis on the attack and defense as a two-player game.\n\nWEAKNESSES:\n1. The authors build an auxiliary global model to reverse engineer the trigger, as mentioned in Sec 4.2, to address the key challenge in solving Equation 5 is to construct an auxiliary global model. Then does the authors make any assumptions on the auxiliary global model $\\theta^{t}_{a}$ comparing to global model? If so, is there any analysis on auxiliary global model in the minimax game framework? It is unclear that how the assumption is made and it would be beneficial to understand the implications if the assumed conditions do not hold true.\n2. The determination of the 'genuine score', $p_{i}^{t}$, is another part that needs clarification. The paper does not provide theoretical estimations or robust definitions for such an important score metric for the defense, which raises questions about potential inaccuracies in the estimation and their impact on the defense framework.\n3. The evaluation methodology could be further enhanced. If the genuine score is an empirical number, it would be useful to have it studied in the evaluation section. Moreover, comparisons with state-of-the-art baselines like FLIP [52] would provide a more comprehensive analysis of the proposed solution's performance.\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nPlease refer to the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nExisting defenses consider a static attack model where an attacker sticks to a fixed strategy and does not adapt its attack strategies, causing they are less effective under adaptive attacks. Hence, the authors propose FedGame to bridge this gap. The main idea of FedGame is to model single or multi-stage strategic interactions between the defender in FL and dynamic attackers as a min-max game. It mainly consists of three steps: 1) building an auxiliary global model. 2) exploiting it to reverse engineer a backdoor trigger and target class. 3) testing whether the local model will predict an input embedded with the reverse engineered trigger as the target class to compute a genuine score.\n\nSTRENGTHS:\n1. The idea of modeling strategic interactions between the defender and attackers as a min-max game is novel and less studied in previous works. The proposed FedGame seems to be competitive with related methods.\n2. The paper is well-organized in general, well-written and mostly easy to follow.\n3. This paper addresses a pertinent problem in Federated learning. The extensive experiments show that FedGame method works even if more than half of the clients are malicious.\n\nWEAKNESSES:\n1. The authors use the genuine score $p_i^t$ to quantify the extent to which a client is benign, however, the results of $p_i^t$ highly depends on the quality of reversed-trigger generated by NC or other tools, implying the limitations of the FedGame are the same as NC, e.g., maybe invalid in large-size trigger reverse (such as Blend, Reflection, …) and all-to-all attacks, etc.\n2. FedGame seems to have high time complexity. In the server side, the defender needs to run NC process to generate reverse-trigger (When there are many categories, the time consumption of NC is enormous). In the client side, the attacker needs to find an optimal $r_i^t$ using grid search (Every search for $r_i^t$ requires retraining the local model).\n\nQUESTIONS:\n1. The paper points out that existing defenses become less effective against dynamic attackers who strategically adapt their attack strategies. However, the definition of dynamic attacks is unclear. Further, the dynamic attack simulated by this paper seems to be only the change of the poisoning rate $r_i^t$. Does this mean that the proposed method in this paper has certain limitations? Because there are many possible factors that may affect the attack performance, such as the selection of poisoned neurons [1], the design of the loss function [2], etc.  \n2. In practical, the defender cannot manipulate the attacker's malicious behavior. Hence, it seems the defense method proposed in this paper just need to achieve the goal of $\\min (\\sum_{i \\in S_a} p_i + \\sum_{j \\in S-S_a} p_j)$. If so, the FedGame is more likely a combination of NC method with limited impact from the min-max game. If not, does it mean that FedGame needs benign clients to act as attackers to achieve better defense effects? Will FedGame be invalid if the target of the real attackers and the fake attackers are different?\n3. In section 4.3, when searching for an optimal $r_i^t$, the goal of attackers is $\\max (p_i^t+\\lambda r_i^t)$. However, increasing $p_i^t$ and increasing $r_i^t$ are contradictory because large $p_i^t$ means small ASR, while large $r_i^t$ means high ASR. It seems that the attack constructed in this paper is not competitive enough, which raises some doubts about the necessity of the attack stage.\n\n\n[1]. PerDoor: Persistent Non-Uniform Backdoors in Federated Learning using Adversarial Perturbations.\n[2]. Chameleon: Adapting to Peer Images for Planting Durable Backdoors in Federated Learning.\n\nLIMITATIONS:\nsee the questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFederated learning (FL) enables a distributed training paradigm, where multiple clients can jointly train a global model without needing to share their local data. However, recent studies have shown that federated learning provides an additional surface for backdoor attacks. For instance, an attacker can compromise a subset of clients and thus corrupt the global model to mispredict an input with a backdoor trigger as the adversarial target. Existing defenses for federated learning against backdoor attacks usually detect and exclude the corrupted information from the compromised clients based on a static attacker model. Such defenses, however, are not adequate against dynamic attackers who strategically adapt their attack strategies. To bridge this gap in defense, we model single or multi-stage strategic interactions between the defender in FL and dynamic attackers as a minimax game. Based on the analysis of our model, we design an interactive defense mechanism FedGame. We also prove that under mild assumptions, the global FL model trained with FedGame under backdoor attacks is close to that trained without attacks. Empirically, we perform extensive evaluations on benchmark datasets and compare FedGame with multiple state-of-the-art baselines. Our experimental results show that FedGame can effectively defend against strategic attackers and achieves significantly higher robustness than baselines. For instance, FedGame reduces attack success rate by 82% on CIFAR10 compared with six state-of-the-art defense baselines under Scaling attack.\n\nSTRENGTHS:\nThe article is well written, has a smooth grammar, and a clear purpose. It better summarizes and clarifies the concepts and definitions of backdoor attack and Federated learning. It is novel and innovative to express FedGame as a minimax game between defenders and attackers.The mathematical derivation is relatively detailed.\n\nWEAKNESSES:\nThere may be biases in the selection and collection of data samples, which affect the accuracy and reliability of the results. In terms of research methods, there is a lack of sufficient description of the control group and experimental group, making it difficult to determine causal relationships. The literature review section lacks comprehensive and in-depth academic research, resulting in a lack of clarity in the research background of the article. The discussion on the results and discussion section is not clear and powerful enough, and requires more detailed explanation and support. The improper use of statistical analysis methods or hypothesis testing has affected the credibility of the results. There are some loose and incoherent issues in the structure and organization of the paper, which require better logic and framework. The summary in the conclusion section may be too brief to fully summarize the significance and contributions of the research.The writing of the paper could be improved for better description and clarification.\n\nQUESTIONS:\n1.Have you cited the latest and relevant research results？\n2.Insufficient experimental content.\n3.The experimental datasets are all small datasets, and model testing needs to be carried out on the Big data set.\n4.Icons are not clear enough\n\nLIMITATIONS:\nThe author's paper has good innovation and rigorous writing logic, but the experiment is not sufficient and needs to be further expanded to obtain more universal conclusions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a game-theoretical defense mechanism named \"FedGame\" designed to safeguard against backdoor attacks in the Federated Learning (FL) environment. They demonstrate through theoretical analysis that training the global FL model using FedGame under backdoor attacks yields results closely resembling those obtained without any attacks. The researchers conduct extensive evaluations using benchmark datasets and compare FedGame's performance with various state-of-the-art baselines. The experimental findings indicate that FedGame successfully thwarts strategic attackers and significantly outperforms the baseline approaches, displaying remarkable robustness against adversarial threats.\n\nSTRENGTHS:\n1. It is interesting to incorporate game theory to federated learning.\n2. The authors show that using FedGame to train the global FL model under backdoor attacks yields similar results to those without attacks.\n3. The authors extensively evaluate FedGame on benchmark datasets, showcasing its effectiveness in defending against strategic attackers and outperforming baselines with significantly higher robustness.\n\nWEAKNESSES:\nFrom game perspective, \n1. Ideally, we expected to see a game model incorporating FL's online nature (e.g., Markov Game). It is unclear to me why it is a multi-stage strategic game (sequential decision problem) since each FL round's optimization problem is solved independently.\n2. The term minmax game is ambiguous. It should be making clear which solution concept (Stackelberg or Nash equilibrium, Zero-sum or General-sum utility) is considered. Otherwise, it is hard to claim the existence of the solution and convergence of the algorithm. \n3. The game's information design is unclear (game with complete or incomplete information). It should be modeled clearly (probably mathematically?) which player have what information that is known/unknown by some other player. Informational concerns play a central role in players’ decision making in such strategic environments.\n\nFrom security perspective, \n1. Why the reverse engineering methods can be directly applied to FL setting considering they normally require large amount of clean data without bias (FL clients may hold heterogeneous data)? Will the aggregation influence the reversed pattern since the aggregated model is updated by a combination of clean and poisoned data? \n2. The action spaces for both attacker and defender are not general enough. From the attack side, poison ratio is not the only parameter the attacker can manipulate (e.g., scaling factor, batch size, number of local training epoch, etc.). In general, attacker can control the model updates send to the server. From the defense side, the authors only consider the client-wise aggregation-based training stage defense. There are other defense (e.g., coordinate-wise, post training stage defenses) could be considered to extent the action space.\n3. In FL, sever is not only unaware of the attacker, but also lack of environment information, e.g., number of total clients, non-iid level, subsampling, etc.\n\nQUESTIONS:\n1. Instead of utilizing reverse engineering, why not considering other approaches closer to game concept like modeling a Bayesian game, and update defender's prior belief about attacker during FL process?\n2. Does the \"dynamic backdoor attack\" mean the attacker can change backdoor trigger and targeted label during FL process? If that is the case, the attacker should consider a long-term goal instead of a one-shot myopic objective. Otherwise, it could mitigate its own attack effects even there is no defense exists.\n3. It is unclear to me in both section 4.3 and Appendix how the attacker uses stochastic gradient descent to optimize the trigger $\\delta$. In the conducted DBA experiments, the trigger(s) take the form of fixed-sized square(s) located at specific position(s) and may involve certain numbers of squares. These characteristics represent potential parameters that can be incorporated into the optimization problem. However, it is worth noting that for achieving the most versatile trigger, it is worthwhile to consider every pixel in the image. In recent works [1] [2], researchers have explored generated backdoor triggers that encompass the entire range of the image.\n4. I wonder what if the reverse engineering cannot get the correct trigger and/or targeted label? There should be either theoretical analysis or ablation experiments to further exam the influence.\n\n\n[1] Salem, Ahmed, et al. \"Dynamic backdoor attacks against machine learning models.\" 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P). IEEE, 2022.\n[2] Doan, Khoa D., Yingjie Lao, and Ping Li. \"Marksman backdoor: Backdoor attacks with arbitrary target class.\" Advances in Neural Information Processing Systems 35 (2022): 38260-38273.\n\nLIMITATIONS:\n1. The post-training stage defenses play a vital role in countering backdoor attacks. Even within the context of FL, certain techniques such as Neuron Clipping [3] and Pruning [4] have demonstrated their effectiveness in detecting and mitigating the impact of backdoor attacks. Consequently, I am curious to know how the proposed FedGame performs compare to these post-training stage defenses. Noted that, some of them do not require to know backdoor trigger or target label.\n2. I wonder the efficiency of the FedGame algorithm. It seems for each FL epoch, it needs process reverse engineering method (costly according to original papers) and calculate a equilibrium (also costly in most of settings like genera sum Nash aquarium). Is there any assumption or simplification I missed here?\n\n[3] Wang, Hang, et al. \"Universal post-training backdoor detection.\" arXiv preprint arXiv:2205.06900 (2022).\n[4] Wu, Chen, et al. \"Mitigating backdoor attacks in federated learning.\" arXiv preprint arXiv:2011.01767 (2020).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-21"}