SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jinyuan Jia *
jinyuan@psu.edu
Zhuowen Yuan *
zhuowen3@illinois.edu
Dinuka Sahabandu
sdinuka@uw.edu
Luyao Niu
luyaoniu@uw.edu
Arezoo Rajabi
rajabia@uw.edu
Bhaskar Ramasubramanian
ramasub@wwu.edu
Radha Poovendran
rp3@uw.edu
Federated learning (FL) provides a distributed training paradigm where multiple clients can jointly train a global model without sharing their local data. However, recent studies have shown that FL offers an additional surface for backdoor attacks. For instance, an attacker can compromise a subset of clients and thus corrupt the global model to misclassify an input with a backdoor trigger as the adversarial target. Existing defenses for FL against backdoor attacks usually detect and exclude the corrupted information from the compromised clients based on a static attacker model. However, such defenses are inadequate against dynamic attackers who strategically adapt their attack strategies. To bridge this gap, we model the strategic interactions between the defender and dynamic attackers as a minimax game. Based on the analysis of the game, we design an interactive defense mechanism FedGame. We prove that under mild assumptions, the global model trained with FedGame under backdoor attacks is close to that trained without attacks. Empirically, we compare FedGame with multiple state-of-the-art baselines on several benchmark datasets under various attacks. We show that FedGame can effectively defend against strategic attackers and achieves significantly higher robustness than baselines. Our code is available at: https://github.com/AI-secure/FedGame.
Federated learning (FL) [21, 28] aims to train a global model over training data that are distributed across multiple clients (e.g., mobile phones, IoT devices) in an iterative manner. In each communication round, a cloud server shares its global model with selected clients. Then, each selected client uses the global model to initialize its local model, utilizes its local training dataset to train the local model, and sends the local model update to the server. Finally, the server uses an aggregation rule to integrate local model updates from clients to update its global model.
Due to the distributed nature of FL, many studies [3, 4, 5, 21, 44] have shown that FL is vulnerable to backdoor attacks [11, 18, 26, 27, 30, 33, 37, 40, 41, 55, 59]. For instance, an attacker can compromise
a subset of clients and manipulate their local training datasets to corrupt the global model such that it predicts an attacker-chosen target class for any inputs embedded with a backdoor trigger [3]. To defend against such backdoor attacks, many defenses [7, 32, 34, 36, 39, 51] have been proposed (see Section 2 for details). However, all those defenses consider a static attack model where an attacker sticks to a fixed strategy and does not adapt its attack strategies. As a result, they are less effective under adaptive attacks, e.g., Wang et al. [44] showed that defenses in [6, 39] can be bypassed by appropriately designed attacks. While the vulnerability of the FL against dynamic or adaptive attack is known, dynamic defense has not been well-studied yet.
In this work, we propose FedGame, a game-theoretic defense against backdoor attacks in FL. Specifically, we formulate FedGame as a minimax game between the server (defender) and attacker, enabling them to adapt their defense and attack strategies strategically. While the server has no prior knowledge about which client is compromised, our key idea is that the server can compute a genuine score for each client, which should be large (or small) if the client is benign (or compromised) in each communication round. The genuine score serves as a weight for the local model update of the client when used to update the global model. The goal of the defender is to minimize the genuine scores for compromised clients and maximize them for benign ones. To solve the resulting minimax game for the defender, we follow a three-step process consisting of 1) building an auxiliary global model, 2) exploiting the auxiliary global model to reverse engineer a backdoor trigger and target class, and 3) testing whether the local model of a client will predict an input embedded with the reverse engineered backdoor trigger as the target class to compute a genuine score for the client. Based on the deployed defense, the goal of the attacker is to optimize its attack strategy by maximizing the effectiveness of the backdoor attack while remaining stealthy. Our key observation is that the attack effectiveness is determined by two factors: the genuine score and the local model of the client. We optimize the attack strategy with respect to those two factors to maximize the effectiveness of backdoor attacks against dynamic defense.
We perform both theoretical analysis and empirical evaluations for FedGame. Theoretically, we prove that the global model trained with our defense under backdoor attacks is close to that trained without attacks (measured by Euclidean distance of model parameters). Empirically, we perform comprehensive evaluations. In particular, we evaluate FedGame on benchmark datasets to demonstrate its effectiveness under state-of-the-art backdoor attacks [3, 54, 59]. Moreover, we compare it with six state-of-the-art baselines [6, 7, 36, 39, 56]. Our results indicate that FedGame outperforms them by a significant margin (e.g., the attack success rate for FedGame is less than  $13\%$  while it is greater than  $95\%$  for those baselines on CIFAR10 under Scaling attack [3]). We also perform comprehensive ablation studies and evaluate FedGame against adaptive attacks. Our results show FedGame is consistently effective. Our key contributions are as follows:
- We propose FedGame, the first game-theoretic defense against dynamic backdoor attacks to FL.
- We provide theoretical guarantees of FedGame. We show that the global model trained with FedGame under backdoor attacks is close to that without attacks.
- We perform a systematic evaluation of FedGame on benchmark datasets and demonstrate that FedGame significantly outperforms state-of-the-art baselines.
Backdoor Attacks in FL. In backdoor attacks against FL [3, 4, 5, 31, 44, 50, 59], an attacker aims to make a global model predict a target class for any input embedded with a backdoor trigger via compromised clients. For instance, Bagdasaryan et al. [3] proposed the scaling attack in which an attacker first uses a mix of backdoored and clean training examples to train its local model and then scales the local model update by a factor before sending it to the server. In our work, we will leverage state-of-the-art attacks [3, 54, 59] to perform strategic backdoor attacks against our defense.
Defenses against Backdoor Attacks in FL. Many defenses [39, 7, 34, 51, 36, 32, 57, 9] were proposed to mitigate backdoor attacks in FL. For instance, Sun et al. [39] proposed norm-clipping, which clips the norm of the local model update of a client to a certain threshold. Some work extended differential privacy [15, 1, 29] to mitigate backdoor attacks to federated learning. The idea is to clip the local model update and add Gaussian noise. Cao et al. [7] proposed FLTrust, which leveraged the similarity of the local model update of a client with that computed by the server itself on its
clean dataset. DeepSight [36] performs clustering on local models and then filters out those from compromised clients. By design, DeepSight is ineffective when the fraction of compromised clients is larger than  $50\%$ . Other defenses include Byzantine-robust FL methods such as Krum [6], Trimmed Mean [56], and Median [56]. However, all of those defenses only consider a static attacker model. As a result, they become less effective against dynamic attackers who strategically adapt their attack strategies.
Another line of research focuses on detecting compromised clients [24, 58]. As those defenses need to collect many local model updates from clients to make confident detection, the global model may already be backdoored before those clients are detected. Some recent studies [8, 53, 57, 10] proposed certified defenses against compromised clients. However, they can only tolerate a moderate fraction of compromised clients (e.g., less than  $10\%$ ) as shown in their results. In contrast, FedEx can tolerate a much higher fraction (e.g.,  $80\%$ ) of compromised clients.
In this work, we aim to prevent backdoor attacks to federated learning, i.e., our goal is to train a robust global model under compromised clients. The server could also utilize post-training defenses [43, 51] to remove the backdoor in a backdoored global model. Prevention-based defense could be combined with those post-training defenses to form a defense-in-depth. We note that the defense proposed in [43] requires the server to own some clean samples that have the same distribution as the local training data of clients. The defense proposed in [51] prunes filters based on pruning sequences collected from clients and thus is less effective when the fraction of compromised clients is large.
3 Background and Threat Model
Federated Learning. Let  $S$  denote the set of clients and  $D_{i}$  denote the local training dataset of client  $i \in S$ . In the  $t^{th}$  communication round, the server first sends the current global model  $\Theta^t$  to the selected clients. Then, each selected client  $i$  trains a local model (denoted as  $\Theta_i^t$ ) by fine-tuning the global model  $\Theta^t$  using its local training dataset  $D_{i}$ . For simplicity, we use  $\mathbf{z} = (\mathbf{x}, y)$  to denote a training example in  $D_{i}$ , where  $\mathbf{x}$  is the training input (e.g., an image) and  $y$  is its ground truth label. Given  $D_{i}$  and the global model  $\Theta^t$ , the local objective is defined as  $\mathcal{L}(D_i; \Theta^t) = \frac{1}{|\mathcal{D}_i|} \sum_{\mathbf{z} \in D_i} \ell(\mathbf{z}; \Theta^t)$  where  $\ell$  is a loss function (e.g., cross-entropy loss). The client can use gradient descent to update its local model based on the global model and its local training dataset, i.e.,  $\Theta_i^t = \Theta^t - \eta_l \frac{\partial \mathcal{L}(D_i; \Theta^t)}{\partial \Theta^t}$ , where  $\eta_l$  is the local learning rate. Note that stochastic gradient descent can also be used in practice. After that, the client sends  $g_i^t = \Theta_i^t - \Theta^t$  (called local model update) to the server. Note that it is equivalent for the client to send a local model or local model update to the server as  $\Theta_i^t = \Theta^t + g_i^t$ . After receiving the local model updates from all clients, the server can aggregate them based on an aggregation rule  $\mathcal{R}$  (e.g., FedAvg [28]) to update its global model, i.e., we have  $\Theta^{t+1} = \Theta^t + \eta_r (g_1^t, g_2^t, \dots, g_{|S|}^t)$ , where  $|S|$  represents the number of clients and  $\eta$  is the learning rate of the global model.
Threat Model. We consider attacks proposed in previous work [3, 54, 59]. We assume an attacker can compromise a set of clients (denoted as  $S_{a}$ ). To perform backdoor attacks, the attacker first selects an arbitrary backdoor trigger  $\delta$  and a target class  $y^{tc}$ . For each client  $i \in S_{a}$  in the  $t^{\text{th}}$  ( $t = 1, 2, \ldots$ ) communication round, the attacker can choose an arbitrary fraction (denoted as  $r_i^t$ ) of training examples from the local training dataset of the client, embed the backdoor trigger  $\delta$  to training inputs, and relabel them as the target class  $y^{tc}$ . In state-of-the-art backdoor attacks [3, 54, 59], the attacker leverages those trigger-embedded training examples to inject the backdoor into the local model of a compromised client to attack the global model.
Following [7], we consider that the server itself has a small clean training dataset (denoted as  $\mathcal{D}_s$ ), which could be collected from the same or different domains of the local training datasets of clients. Note that the server does not have 1) information on compromised clients, and 2) the poisoning ratio  $r_i^t$  ( $i \in S_a$ ), backdoor trigger  $\delta$ , and target class  $y^{tc}$  selected by the attacker. In our threat model, we do not make any other assumptions for the server compared with standard federated learning [28].
Overview. We formulate FedGame as a minimax game between the defender and attacker, which enables them to optimize their strategies respectively. In particular, the defender computes a genuine
score for each client in each communication round. The goal of the defender is to maximize the genuine score for a benign client and minimize it for a compromised one. Given the genuine score for each client, we use a weighted average over all the local model updates to update the global model, i.e., we have
$$
\Theta^ {t + 1} = \Theta^ {t} + \eta \frac {1}{\sum_ {i \in S} p _ {i} ^ {t}} \sum_ {i \in S} p _ {i} ^ {t} g _ {i} ^ {t}, \tag {1}
$$
where  $p_i^t$  is the genuine score for client  $i$  in the  $t^{th}$  communication round and  $\eta$  is the learning rate of the global model. We compute a weighted average of local models because we aim to achieve robustness against attacks while making minimal changes to FedAvg [28], which has been shown to achieve high utility in diverse settings. In our experiments, we empirically show that the genuine scores are almost constantly 0 for malicious clients after a few communication rounds, which indicates that the malicious model updates will not be aggregated at all, reducing Equation 1 to a robust FedAvg against attacks. (See Figure 2(c)(d) in the Appendix for more details.) Based on Equation 1, the effectiveness of the attack is determined by two components: genuine scores and local models of compromised clients. In our framework, the attacker optimizes the tradeoff between these two components to maximize the effectiveness of backdoor attacks.
4.1 Game Formulation
Computing Genuine Scores. The key challenge for the server to compute genuine scores for clients is that the server can only access their local model updates, i.e., the server can only leverage local models of clients to compute genuine scores. To tackle this issue, we observe that the local model of a compromised client is more likely to predict a trigger-embedded input as the target class. Therefore, the server can first reverse engineer the backdoor trigger  $\delta_{re}$  and target class  $y_{re}^{tc}$  (which we will discuss more in the next subsection) and then use them to compute the genuine score for each client. Since the client  $i$  sends its local model update  $g_i^t$  to the server, the server can compute the local model of the client  $i$  as  $\Theta_i^t = \Theta^t + g_i^t$ . With  $\Theta_i^t$ , the server can compute  $p_i^t$  as follows:
$$
p _ {i} ^ {t} = 1 - \frac {1}{\left| \mathcal {D} _ {s} \right|} \sum_ {\mathbf {x} \in \mathcal {D} _ {s}} \mathbb {I} \left(G (\mathbf {x} \oplus \delta_ {r e}; \Theta_ {i} ^ {t}) = y _ {r e} ^ {t c}\right), \tag {2}
$$
where  $\mathbb{I}$  is an indicator function,  $\mathcal{D}_s$  is the clean training dataset of the server,  $\mathbf{x} \oplus \delta_{re}$  is a trigger-embedded input, and  $G(\mathbf{x} \oplus \delta_{re}; \Theta_i^t)$  represents the label of  $\mathbf{x} \oplus \delta_{re}$  predicted by  $\Theta_i^t$ . Intuitively, the genuine score for client  $i$  is small if a large fraction of inputs embedded with the reverse-engineered backdoor trigger is predicted as the target class by its local model. We note that the defender only needs to query local models to compute their genuine scores. Moreover, our defense is still applicable when differential privacy [14, 49] is used to protect the local models.
Optimization Problem of the Defender. The server aims to reverse engineer the backdoor trigger  $\delta_{re}$  and target class  $y_{re}^{tc}$  such that the genuine scores for compromised clients are minimized while those for benign clients are maximized. Formally, we have the following optimization problem:
$$
\min  _ {\delta_ {r e}, y _ {r e} ^ {t c}} \left(\sum_ {i \in \mathcal {S} _ {a}} p _ {i} ^ {t} - \sum_ {j \in \mathcal {S} \backslash \mathcal {S} _ {a}} p _ {j} ^ {t}\right). \tag {3}
$$
Note that  $S_{a}$  is the set of malicious clients that is unknown to the server. In the next subsection, we will discuss how to address this challenge to solve the optimization problem.
**Optimization Problem of the Attacker.** The goal of an attacker is to maximize its attack effectiveness. Based on Equation 1, the attacker needs to: 1) maximize the genuine scores for compromised clients while minimizing them for benign ones, i.e.,  $\max (\sum_{i\in S_a}p_i^t -\sum_{j\in S\setminus S_a}p_j^t)$ , and 2) make the local models of compromised clients predict an input embedded with the attacker-chosen backdoor trigger  $\delta$  as the target class  $y^{tc}$ . To perform the backdoor attack in the  $t^{\mathrm{th}}$  communication round, the attacker embeds the backdoor to a certain fraction (denoted as  $r_i^t$ ) of training examples in the local training dataset of the client and uses them as data augmentation. Intuitively, a larger  $r_i^t$  encourages a higher attack success rate but can potentially result in a lower genuine score. In other words,  $r_i^t$  measures a tradeoff between these two factors. Formally, the attacker can solve the following
optimization problem to find the desired tradeoff:
$$
\max  _ {R ^ {t}, \delta} \left(\sum_ {i \in \mathcal {S} _ {a}} p _ {i} ^ {t} - \sum_ {j \in \mathcal {S} \backslash \mathcal {S} _ {a}} p _ {j} ^ {t} + \lambda \sum_ {i \in \mathcal {S} _ {a}} r _ {i} ^ {t}\right), \tag {4}
$$
where  $R^t = \{r_i^t\mid i\in S_a\}$  ,  $\delta$  is the trigger, and  $\lambda$  is a hyperparameter to balance the two terms.
Minimax Game. Given the optimization problems of the defender and attacker, we have the following minimax game:
$$
\min  _ {\delta_ {r e}, y _ {r e} ^ {t c}} \max  _ {R ^ {t}, \delta} \left(\sum_ {i \in \mathcal {S} _ {a}} p _ {i} ^ {t} - \sum_ {j \in \mathcal {S} \backslash \mathcal {S} _ {a}} p _ {j} ^ {t} + \lambda \sum_ {i \in \mathcal {S} _ {a}} r _ {i} ^ {t}\right). \tag {5}
$$
Note that  $r_i^t (i \in S_a)$  and  $\delta$  are chosen by the attacker. Thus, we can add them to the objective function in Equation 3 without influencing its solution given the local model updates of clients. In Section A of Appendix, we provide an intuitive interpretation regarding how to connect our above objective to the ultimate goal of the defender and attacker (the defender aims to obtain a clean model, while the attacker wants the global model to be backdoored).
4.2 Solving the Minimax Game by the Defender
The key challenge for the server to solve Equation 5 is that it does not know  $S_{a}$  (set of compromised clients). To address the challenge, our idea is to construct an auxiliary global model based on local models of all clients. Suppose  $g_{i}^{t}$  is the local model update sent by each client  $i \in S$  to the server. Our auxiliary global model is constructed as follows:  $\Theta_{a}^{t} = \Theta^{t} + \frac{1}{|S|}\sum_{i\in S}g_{i}^{t}$ , where  $\Theta^t$  is the global model. Our intuition is that such a naively aggregated auxiliary global model is inclined to predict a trigger-embedded input as the target class under backdoor attacks. As a result, given the auxiliary global model, we can use an arbitrary existing method [42, 46, 38, 48] to reverse engineer the backdoor trigger and target class based on it, which enables us to compute genuine scores for clients based on Equation 2. With those genuine scores, we can use Equation 1 to update the global model to protect it from backdoor attacks in every communication round. The complete algorithm of our FedGame is shown in Algorithm 1 of Appendix.
Our framework is compatible with any trigger reverse engineering methods, allowing off-the-shelf incorporation of techniques developed for centralized supervised learning into federated learning. Note that developing a new reverse engineering method is not the focus of this work. Instead, our goal is to formulate the attack and defense against backdoor attacks on federated learning as a minimax game, which enables us to defend against dynamic attacks.
4.3 Solving the Minimax Game by the Attacker
The goal of the attacker is to find  $r_i^t$  for each client  $i \in S_a$  such that the loss function in Equation 5 is maximized. As the attacker does not know the genuine scores of benign clients, the attacker can find  $r_i^t$  to maximize  $p_i^t + \lambda r_i^t$  for client  $i \in S_a$  to approximately solve the optimization problem in Equation 5. However, the key challenge is that the attacker does not know the reverse engineered backdoor trigger  $\delta_{re}$  and the target class  $y_{re}^{tc}$  of the defender to compute the genuine score for the client  $i$ . In response, the attacker can use the backdoor trigger  $\delta$  and target class  $y^{tc}$  chosen by itself. Moreover, the attacker reserves a certain fraction (e.g., 10%) of training data from its local training dataset  $\mathcal{D}_i$  as the validation dataset (denoted as  $\mathcal{D}_i^{rev}$ ) to find the best  $r_i^t$ .
Estimating the Genuine Score Given  $r_i^t$ . For a given  $r_i^t$ , the client  $i$  can embed the backdoor to  $r_i^t$  fraction of training examples in  $\mathcal{D}_i \setminus \mathcal{D}_i^{rev}$  and then use those backdoored training examples to augment  $\mathcal{D}_i \setminus \mathcal{D}_i^{rev}$  to train a local model (denoted as  $\tilde{\Theta}_i^t$ ). Then, the genuine score can be estimated as  $\tilde{p}_i^t = 1 - \frac{1}{|\mathcal{D}_i^{rev}|} \sum_{\mathbf{x} \in \mathcal{D}_i^{rev}} \mathbb{I}(G(\mathbf{x} \oplus \delta; \tilde{\Theta}_i^t) = y^{tc})$ , where  $G(\mathbf{x} \oplus \delta; \tilde{\Theta}_i^t)$  is the predicted label by the global model  $\tilde{\Theta}_i^t$  for the trigger-embedded input  $\mathbf{x} \oplus \delta$ .
Finding the Optimal  $r_i^t$ . The client can use grid search to find  $r_i^t$  that achieves the largest  $\tilde{p}_i^t + \lambda r_i^t$ . After estimating the optimal  $r_i^t$ , client  $i$  can embed the backdoor to  $r_i^t$  fraction of training examples and utilize them to perform backdoor attacks based on state-of-the-art methods [3, 54, 59].
Trigger Optimization. The attacker can choose an arbitrary static trigger to launch backdoor attacks, or dynamically optimize it to make the attack more effective. Given the fixed location and bounding box of the backdoor trigger, the attacker can optimize the trigger pattern with gradient descent such that it is more likely for a backdoored input to be predicted as the target class. For instance, the attacker can optimize the trigger pattern to minimize the cumulative loss on all training data of malicious clients, i.e.,  $\delta^t = \mathrm{argmin}_{\delta^*} \sum_{\mathbf{x} \in \cup_{i \in S_a} \mathcal{D}_i} \ell(\Theta^t(\mathbf{x} \oplus \delta^*), y^{tc})$ , where  $\Theta^t(\mathbf{x} \oplus \delta^*)$  is the output of the model which is a probability simplex of all possible classes and  $\ell$  is a standard loss function (e.g., cross-entropy loss). The attacker can then solve Equation 5 with the optimized trigger. Recall that the attacker aims to find the largest  $\tilde{p}_i^t + \lambda r_i^t$ . Therefore, if the resulting  $\tilde{p}_i^t + \lambda r_i^t$  given  $\delta^t$  is lower than that given the previous trigger  $\delta^{t-1}$ , we let  $\delta^t = \delta^{t-1}$ .
The complete algorithm for the compromised clients is shown in Algorithm 2 in Appendix C.
This section provides a theoretical analysis of FedGame under backdoor attacks. Suppose the global model parameters are in a bounded space. We derive an upper bound for the  $L_{2}$ -norm of the difference between the parameters of the global models with and without attacks. To analyze the robustness of FedGame, we make the following assumptions, which are commonly used in the analysis of previous studies [25, 45, 16, 35, 53, 7, 9] on federated learning.
Assumption 5.1. The loss function is  $\mu$ -strongly convex with  $L$ -Lipschitz continuous gradient. Formally, we have the following for arbitrary  $\Theta$  and  $\Theta'$ :