SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left(\nabla_ {\Theta} \ell (\mathbf {z}; \Theta) - \nabla_ {\Theta^ {\prime}} \ell (\mathbf {z}; \Theta^ {\prime})\right) ^ {T} \left(\Theta - \Theta^ {\prime}\right) \geq \mu \| \Theta - \Theta^ {\prime} \| _ {2} ^ {2}, \tag {6}
$$
$$
\left\| \nabla_ {\Theta} \ell (\mathbf {z}; \Theta) - \nabla_ {\Theta^ {\prime}} \ell (\mathbf {z}; \Theta^ {\prime}) \right\| _ {2} \leq L \| \Theta - \Theta^ {\prime} \| _ {2}, \tag {7}
$$
where  $\mathbf{z}$  is an arbitrary training example.
Assumption 5.2. We assume the gradient  $\nabla_{\Theta}\ell (\mathbf{z};\Theta)$  is bounded with respect to  $L_{2}$ -norm for arbitrary  $\Theta$  and  $\mathbf{z}$ , i.e., there exists some  $M\geq 0$  such that
$$
\left\| \nabla_ {\Theta} \ell (\mathbf {z}; \Theta) \right\| _ {2} \leq M. \tag {8}
$$
Suppose  $\Theta_c^t$  is the global model trained by FedGame without any attacks in the  $t$ th communication round, i.e., each client  $i\in S$  uses its clean local training dataset  $\mathcal{D}_i$  to train a local model. Moreover, we assume gradient descent is used by each client to train its local model. Suppose  $q_{i}^{t}$  is the genuine score for client  $i$  without attacks. Moreover, we denote  $\beta_i^t = \frac{q_i^t}{\sum_{i\in S}q_i^t}$  as the normalized genuine score for client  $i$ . To perform the backdoor attack, we assume a compromised client  $i$  can embed the backdoor trigger to  $r_i^t$  fraction of training examples in the local training dataset of the client and relabel them as the target class. Those backdoored training examples are used to augment the local training dataset of the client. Suppose  $\Theta^t$  is the global model under the backdoor attack in the  $t$ th communication round with our defense. We denote  $\alpha_i^t = \frac{p_i^t}{\sum_{i\in S}p_i^t}$  as the normalized genuine score for client  $i$  with attacks in the  $t$ th communication round. Formally, we have:
Lemma 5.3 (Robustness Guarantee for One Communication Round). Suppose Assumptions 5.1 and 5.2 hold. Moreover, we assume  $(1 - r^{t})\beta_{i}^{t}\leq \alpha_{i}^{t}\leq (1 + r^{t})\beta_{i}^{t}$ , where  $i\in S$  and  $r^t = \sum_{j\in S_a}r_j^t$ . Then, we have:
$$
\left\| \Theta^ {t + 1} - \Theta_ {c} ^ {t + 1} \right\| _ {2} \tag {9}
$$
$$
\leq \sqrt {1 - \eta \mu + 2 \eta \gamma^ {t} + \eta^ {2} L ^ {2} + 2 \eta^ {2} L \gamma^ {t}} \| \Theta^ {t} - \Theta_ {c} ^ {t} \| _ {2} + \sqrt {2 \eta \gamma^ {t} (1 + \eta L + 2 \eta \gamma^ {t})} + 2 \eta r ^ {t} M, \tag {10}
$$
where  $\eta$  is the learning rate of the global model,  $L$  and  $\mu$  are defined in Assumption 5.1,  $\gamma^t = \sum_{i\in S_a}\alpha_i^t r_i^t M$ , and  $M$  is defined in Assumption 5.2.
Proof sketch. Our idea is to decompose  $\left\| \Theta^{t + 1} - \Theta_c^{t + 1}\right\|_2$  into two terms. Then, we derive an upper bound for each term based on the change of the local model updates of clients under backdoor attacks and the properties of the loss function. As a result, our derived upper bound relies on  $r_i^t$  for each client  $i\in S_a$ , parameters  $\mu, L$ , and  $M$  in our assumptions, as well as the parameter difference of the global models in the previous iteration, i.e.,  $\| \Theta^t -\Theta_c^t\| _2$ . Our complete proof is in Appendix B.1.
In the above lemma, we derive an upper bound of  $\left\| \Theta^{t + 1} - \Theta_c^{t + 1}\right\| _2$  with respect to  $\| \Theta^t -\Theta_c^t\| _2$  for one communication round. In the next theorem, we derive an upper bound of  $\| \Theta^t -\Theta_c^t\| _2$  as  $t\to \infty$ . We iterative apply Lemma 5.3 for successive values of  $t$  and have the following theorem:
Theorem 5.4 (Robustness Guarantee). Suppose Assumptions 5.1 and 5.2 hold. Moreover, we assume  $(1 - r^{t})\beta_{i}^{t}\leq \alpha_{i}^{t}\leq (1 + r^{t})\beta_{i}^{t}$  for  $i\in S,\gamma^t\leq \gamma$  and  $r^t\leq r$  hold for all communication round  $t$  and  $\mu >2\gamma$ , where  $r^t = \sum_{j\in S_a}r_j^t$  and  $\gamma^t = \sum_{i\in S_a}\alpha_i^t r_i^t M$ . Let the global model learning rate by chosen as  $0 < \eta < \frac{\mu - 2\gamma}{L^2 + 2L\gamma}$ . Then, we have:
$$
\left\| \Theta^ {t} - \Theta_ {c} ^ {t} \right\| _ {2} \leq \frac {\sqrt {2 \eta \gamma (1 + \eta L + 2 \eta \gamma)} + 2 \eta r M}{1 - \sqrt {1 - \eta \mu + 2 \eta \gamma + \eta^ {2} L ^ {2} + 2 \eta^ {2} L \gamma}} \tag {11}
$$
holds as  $t\to \infty$
Proof sketch. Given the conditions that  $\gamma^t \leq \gamma$  and  $r^t \leq r$  as well as the fact that the right-hand side of Equation 10 is monotonic with respect to  $\gamma^t$  and  $r^t$ , we can replace  $\gamma^t$  and  $r^t$  in Equation 10 with  $\gamma$  and  $r$ . Then, we iterative apply the equation for successive values of  $t$ . When  $0 < \eta < \frac{\mu - 2\gamma}{L^2 + 2L\gamma}$ , we have  $0 < 1 - \eta \mu + 2\eta \gamma + \eta^2 L^2 + 2\eta^2 L\gamma < 1$ . By letting  $t \to \infty$ , we can reach the conclusion. The complete proof can be found in Appendix B.2.
Our theorem implies that the global model parameters under our defense against adaptive attacks do not deviate too much from those of the global model without attack when the fraction of backdoored training examples  $r^t$  is bounded.
In order to thoroughly evaluate the effectiveness of FedGame, we conduct comprehensive experiments on 1) evaluation against three state-of-the-art backdoor attacks: Scaling attack [3], DBA [54], and Neurotoxin [59], 2) comparison with six existing baselines including Krum, Median, Norm-Clipping, Differential Privacy, DeepSight, FLTrust, 3) evaluation against strong adaptive attacks, and 4) comprehensive ablation studies.
Datasets and Models. We use two benchmark datasets: MNIST [23] and CIFAR10 [22] for FL tasks. MNIST has 60,000 training and 10,000 testing images, each of which has a size of  $28 \times 28$  belonging to one of 10 classes. CIFAR10 consists of 50,000 training and 10,000 testing images with a size of  $32 \times 32$ . Each image is categorized into one of 10 classes. For each dataset, we randomly sample  $90\%$  of training data for clients, and the remaining  $10\%$  of training data is reserved to evaluate our defense when the clean training dataset of the server is from the same domain as those of clients. We use a CNN with two convolution layers (detailed architecture can be found in Table 2 in Appendix) and ResNet-18 [19] which is pre-trained on ImageNet [13] as the global models for MNIST and CIFAR10.
FL Settings. We consider two settings: local training data are independently and identically distributed (IID) among clients, and non-IID. We follow the previous work [17] to distribute training data to clients by using a parameter  $q$  to control the degree of non-IID, which models the probability that training images from one category are distributed to a particular client (or a set of clients). We set  $q = 0.5$  by following [17]. Moreover, we train a global model based on 10 clients for 200 iterations with a global learning rate  $\eta = 1.0$ . In each communication round, we use SGD to train the local model of each client for two epochs with a local learning rate of 0.01.
Attack Settings. We consider three state-of-the-art backdoor attacks on federated learning, i.e., Scaling attack [3], DBA [54], and Neurotoxin [59]. For Scaling attack, we set the scaling parameter to be #total clients/(η×#compromised clients) by following [3]. For Neurotoxin, we set the ratio of masked gradients to be 1%, following the choice in [59]. We use the same backdoor trigger and target class as used in those works. By default, we assume 60% of clients are compromised by an attacker. When the attacker solves the minimax game in Equation 5, we set the default λ = 1. We explore the
Table 1: Comparison of FedGame with existing defenses under Scaling attack and DBA. The total number of clients is 10, where  $60\%$  are compromised. The best results for each setting among FedGame and existing defenses are bold.
Attacks	Client Data Distribution	Datasets	Metrics	FedAvg (Without attacks)	Defenses (Under attacks)
FedAvg	Krum	Median	Norm-Clipping	DP	Deep-Sight	FLTrust	FedGame
In Domain	Out-of Domain
Scaling Attack	IID	MNIST	TA (%)	99.04	98.77	98.78	99.17	95.48	92.97	97.69	97.93	98.53	98.56
ASR (%)	9.69	99.99	99.99	99.97	98.54	99.45	20.03	16.01	9.72	9.68
CIFAR10	TA (%)	81.08	80.51	76.44	80.17	80.38	43.22	76.79	75.71	74.81	74.65
ASR (%)	8.39	99.80	99.94	99.82	99.87	99.58	98.58	99.46	8.92	9.24
non-IID	MNIST	TA (%)	98.98	99.15	96.88	99.12	94.54	91.52	97.39	97.68	98.28	98.34
ASR (%)	9.73	99.99	85.03	99.98	98.16	99.54	20.03	19.61	10.42	10.88
CIFAR10	TA (%)	80.25	75.35	67.66	79.54	70.18	50.79	77.76	75.08	73.88	73.57
ASR (%)	9.67	99.92	99.92	99.99	99.63	95.01	99.03	99.82	11.76	12.03
DBA	IID	MNIST	TA (%)	99.04	99.03	98.87	98.98	98.99	98.99	98.64	97.98	97.84	98.05
ASR (%)	9.69	100.00	10.06	99.81	99.75	99.73	15.02	10.02	9.56	9.68
CIFAR10	TA (%)	81.08	80.90	76.09	80.00	80.21	41.36	72.13	75.17	73.18	72.93
ASR (%)	8.39	93.44	94.97	91.60	91.90	86.96	83.26	66.58	8.81	9.00
non-IID	MNIST	TA (%)	98.98	98.98	98.58	99.13	93.98	88.91	96.65	97.62	98.58	98.59
ASR (%)	9.73	100.00	10.52	99.85	55.97	99.66	13.21	10.19	9.97	9.83
CIFAR10	TA (%)	80.25	80.15	74.31	79.78	78.78	38.17	73.83	74.57	73.52	73.21
ASR (%)	9.67	95.03	60.06	95.00	92.51	99.51	80.75	74.35	10.62	10.67
impact of  $\lambda$  in our experiments. We randomly sample  $10\%$  of the local data of each compromised client as validation data to search for an optimal  $r_i^t$ . Moreover, we set the granularity of grid search to 0.1 when searching for  $r_i^t$ .
Baselines. We compare our defense with the following methods: FedAvg [28], Krum [6], Median [56], Norm-Clipping [39], Differential Privacy (DP) [39], DeepSight [36], and FLTrust [7]. Please see Appendix D.2 for parameter settings for those baselines.
Evaluation Metrics. We use testing accuracy (TA) and attack success rate (ASR) as evaluation metrics. Concretely, TA is the fraction of clean testing inputs that are correctly predicted, and ASR refers to the fraction of backdoored testing inputs that are predicted as the target class.
Defense Settings. We consider two settings: in-domain and out-of-domain. For the in-domain setting, we consider the clean training dataset of the server is from the same domain as the local training datasets of clients. We use the reserved data as the clean training dataset of the server for each dataset. For the out-of-domain setting, we consider the server has a clean training dataset that is from a different domain than the FL task. In particular, we randomly sample 6,000 images from FashionMNIST [52] for MNIST and sample 5,000 images from GTSRB [20] for CIFAR10 as the clean training dataset of the server. Unless otherwise mentioned, we adopt Neural Cleansse [42] to reverse engineer the backdoor trigger and target class.
We show the results of FedGame compared with existing defenses under IID and non-IID settings in Table 1. We defer the results against Neurotoxin to Appendix D.3 due to space limitations. We have the following observations from the experimental results. First, FedGame outperforms all existing defenses in terms of ASR. In particular, FedGame can reduce ASR to random guessing (i.e., ASR of FedAvg under no attacks) in both IID and non-IID settings for clients as well as both in-domain and out-of-domain settings for the server. Intrinsically, FedGame performs better because our game-theoretic defense enables the defender to optimize its strategy against dynamic, adaptive attacks. We
note that FLTrust outperforms other defenses (except FedEx) in most cases since it exploits a clean training dataset from the same domain as local training datasets of clients. However, FLTrust is not applicable when the server only holds an out-of-domain clean training dataset, while FedEx can relax such an assumption and will still be applicable. Moreover, our experimental results indicate that FedEx achieves comparable performance even if the server holds an out-of-domain clean training dataset.
To further understand our results, we visualize the average genuine (or trust) scores computed by FedGame (or FLTrust) for compromised and benign clients in Appendix D.4. In particular, we find that the genuine scores produced by FedGame are much lower than those produced by FLTrust for compromised clients, which explains why FedGame outperforms FLTrust. Second, FedGame achieves comparable TA with existing defenses, indicating that FedGame preserves the utility of global models.
Furthermore, we show the comparison results of FedGame with existing defenses against the scaling attack when the total number of clients is 30 in Table 4 in Appendix D.5. Our observations are similar, which indicates that FedGame consistently outperforms existing defenses under different numbers of clients and backdoor attacks.
Impact of  $\lambda$ .  $\lambda$  is a hyperparameter used by an attacker (see Eqn. 4) when searching for the optimal  $r_i^t$  for each compromised client  $i$  in each communication round  $t$ . Figure 1(a) shows the impact of  $\lambda$  on ASR of FedGame. The results show that FedGame is insensitive to different  $\lambda$ 's. The reason is that the genuine score for a compromised client is small when  $\lambda$  is large, and the local model of a compromised client is less likely to predict a trigger-embedded input as the target class when  $\lambda$  is small. As a result, backdoor attacks with different  $\lambda$  are ineffective under FedGame.
(b)
Figure 1: Comparing FedGame and FLTrust under different variations of attack. (a) Different  $\lambda$ . (b) Different fractions of malicious clients. (c) Different numbers of clients selected in each communication round. (d) Different trigger sizes.
Impact of the Fraction of Compromised Clients. Figure 1(b) shows the impact of the fraction of compromised clients on ASR. As the results show, FedGame is effective for a different fraction of compromised clients in both in-domain and out-of-domain settings. In contrast, FLTrust is ineffective when the fraction of compromised clients is large. For instance, FedGame can achieve  $9.84\%$  (in-domain) and  $10.12\%$  (out-of-domain) ASR even if  $80\%$  of clients are compromised on MNIST. Under the same setting, the ASR of FLTrust is  $99.95\%$ , indicating that the defense fails.
Impact of Client Selection. By default, we consider that all clients are selected in each communication round. We also consider only a subset of clients are selected in each communication round by the server. Figure 1(c) shows our experimental results. Our results suggest that our defense is effective and consistently outperforms FLTrust.
Computation Cost. FedGame computes a genuine score for each client in each communication round. Here we demonstrate its computational efficiency. On average, it takes 0.148s to compute a genuine score for each client in each communication round on a single NVIDIA 2080 Ti GPU. We note that the server could from a resourceful tech company (e.g., Google, Meta, Apple), which would have enough computation resources to compute it for millions of clients. Moreover, those local models can be evaluated in parallel.
Performance under Static Attacks. In our evaluation, we consider an attacker optimizing the fraction of backdoored training examples. We also evaluate FedGame under existing attacks where the attacker does not optimize it. Under the default setting, FedGame can achieve an ASR of  $9.75\%$ , indicating that our defense is effective under static attack.
Furthermore, we discuss the impact of trigger reverse engineering methods, the total number of clients, and the size of clean data of the server in Appendix D.6.
6.3 Adaptive Attacks
In this subsection, we discuss some potential adaptive strategies that may be leveraged by the attacker to evade our defense.
Data Replacing Attack. By default, we consider an attacker optimizing the fraction of backdoored training examples added to the local training dataset of a compromised client to maximize backdoor effectiveness. We also consider an attacker who replaces a fraction of the local training data with backdoored samples and optimizes such fraction. Under our default setting, the ASR of FedGame is  $9.71\%$  and  $10.69\%$  on MNIST and CIFAR10, respectively, indicating that our defense is still effective under data replacing attacks.
Variation in Triggers. We demonstrate that our defense is robust to variation in triggers. In particular, we try triggers with sizes  $2 \times 2$ ,  $4 \times 4$ , and  $6 \times 6$  under the default setting. Figure 1(d) compares FedGame with FLTrust. The results demonstrate that FedGame is consistently effective under triggers with different sizes.
We note that although an attacker can slightly manipulate the parameters of local models such that they are more similar to those of benign clients, FedEx does not rely on model parameters for detection. Instead, FedEx leverages the model behaviors, i.e., whether the model predicts inputs with our reverse-engineered trigger as the target class. As a result, our defense would still be effective even if the change in the model parameters is small as long as the model has backdoor behavior (this is required to make the attack effective). This is also the reason why our defense is better than existing methods such as FLTrust which leverages model parameters for defense.
In this work, we propose FedGame, a general game-theory based defense against adaptive backdoor attacks in federated learning. Our formulated minimax game enables the defender and attacker to dynamically optimize their strategies. Moreover, we respectively design solutions for both of them to solve the minimax game. We perform theoretical analysis and empirical evaluations for our framework. Our results demonstrate the effectiveness of FedGame under strategic backdoor attackers. Moreover, FedGame achieves significantly higher robustness than baselines in different settings.
We consider that an attacker could optimize the poisoning ratio and trigger pattern in our work. We believe it is an interesting future work to consider other factors for the attacker (e.g., trigger size, consideration of long-term goal, design of new loss functions [12], poisoned neurons selection [2]). Moreover, we consider a zero-sum Stackelberg game in this work. Another interesting future work is to consider other game formulations, e.g., Bayesian games.
This work is partially supported by the National Science Foundation (NSF) under grants No.1910100, No.2046726, No.2153136, No.2229876, DARPA GARD, the National Aeronautics and Space Administration (NASA) under grant No.80NSSC20M0229, Alfred P. Sloan Fellowship, Office of Naval Research (ONR) under grant N00014-23-1-2386, Air Force Office of Scientific Research (AFOSR) under grant FA9550-23-1-0208, and the Amazon research award.
[54] Chulin Xie, Keli Huang, Pin-Yu Chen, and Bo Li. DBA: Distributed backdoor attacks against federated learning. In International Conference on Learning Representations, 2020.
[55] Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Y Zhao. Latent backdoor attacks on deep neural networks. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pages 2041-2055, 2019.
[56] Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantine-robust distributed learning: Towards optimal statistical rates. In International Conference on Machine Learning, pages 5650–5659. PMLR, 2018.
[57] Kaiyuan Zhang, Guanhong Tao, Qiuling Xu, Siyuan Cheng, Shengwei An, Yingqi Liu, Shiwei Feng, Guangyu Shen, Pin-Yu Chen, Shiqing Ma, et al. Flip: A provable defense framework for backdoor mitigation in federated learning. arXiv preprint arXiv:2210.12873, 2022.
[58] Zaixi Zhang, Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. FLDetector: Defending federated learning against model poisoning attacks via detecting malicious clients. In 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2545-2555, 2022.
[59] Zhengming Zhang, Ashwinee Panda, Linyue Song, Yaoqing Yang, Michael Mahoney, Prateek Mittal, Ramchandran Kannan, and Joseph Gonzalez. Neurotoxin: Durable backdoors in federated learning. In International Conference on Machine Learning, pages 26429-26446. PMLR, 2022.
A Interpretation of the Minimax Objective
We note that the ultimate goal of the defender is to obtain a clean model while the attacker wants the global model to be poisoned. In other words, the server (or attacker) wishes the attack success rate of the global model to be small (large). It is very challenging to achieve this objective directly because 1) the server does not know the exact trigger and target class used by the attacker, and 2) the attacker needs to adapt its attack strategy based on the server's strategy. To address the challenge, we consider an alternative goal (i.e., our objective in Equation 5, where we wish to assign small weights to compromised clients when performing a weighted average to update the global model. Our idea is that when the weights for compromised clients are very small, our global model is less likely to be affected by the attack. As a result, the attacker's strategy is to maximize the genuine scores for compromised clients while ensuring their local models are backdoored to maximize the attack effectiveness for the global model. Thus, our objective in Equation 5 could translate to the ultimate goal. Our empirical results show the effectiveness of our defense.
B.1 Proof of Lemma 5.3
We first present some preliminary lemmas that will be invoked for proving Lemma 5.3.
Lemma B.1. Suppose  $\mathcal{D}_i$  is the clean local training dataset of the client  $i$ . An attacker can inject the backdoor trigger to  $r_i^t$  fraction of training examples in  $\mathcal{D}_i$  and relabel them as the target class. We use  $\mathcal{D}_i'$  to denote the set of backdoored training examples where  $r_i^t = \frac{|\mathcal{D}_i'|}{|\mathcal{D}_i|}$ . Given two arbitrary  $\Theta$  and  $\Theta_c$ , we let  $g_i = \frac{1}{|\mathcal{D}_i \cup \mathcal{D}_i'|} \nabla_\Theta \sum_{\mathbf{z} \in \mathcal{D}_i \cup \mathcal{D}_i'} \ell(\mathbf{z}; \Theta)$  and  $h_i = \frac{1}{|\mathcal{D}_i|} \nabla_{\Theta_c} \sum_{\mathbf{z} \in \mathcal{D}_i} \ell(\mathbf{z}; \Theta_c)$ . We then have that
$$
\left(\Theta - \Theta_ {c}\right) ^ {T} \left(g _ {i} - h _ {i}\right) \geq \left(0. 5 \mu - r _ {i} ^ {t} M\right) \left\| \Theta - \Theta_ {c} \right\| _ {2} ^ {2} - r _ {i} ^ {t} M, \tag {12}
$$
$$
\left\| g _ {i} - h _ {i} \right\| _ {2} \leq L \| \Theta - \Theta_ {c} \| _ {2} + 2 r _ {i} ^ {t} M. \tag {13}
$$
Proof. We first prove Equation 12. We have the following relations: