Title: Meta Stackelberg Game: Robust Federated Learning against Adaptive and Mixed Poisoning Attacks
Venue: NeurIPS 2024
Accepted: no
Keywords: Federated Learning

Game Theory

Reinforcement Learning

Robust Machine Learning
Abstract: Recent research has uncovered that federated learning (FL) systems are vulnerable to various security threats. Although various defense mechanisms have been proposed, they are typically non-adaptive and tailored to specific types of attacks, leaving them insufficient in the face of adaptive or mixed attacks. In this work, we formulate adversarial federated learning as a Bayesian Stackelberg Markov game (BSMG) to tackle poisoning attacks of unknown/uncertain types. We further develop an efficient meta-learning approach to solve the game, which provides a robust and adaptive FL defense. Theoretically, we show that our algorithm provably converges to the first-order $\varepsilon$-equilibrium point in $O(\varepsilon^{-2})$ gradient iterations with $O(\varepsilon^{-4})$ samples per iteration. Empirical results show that our meta-Stackelberg framework obtains superb performance against strong model poisoning and backdoor attacks with unknown/uncertain types.

Abstract

Recent research has uncovered that federated learning (FL) systems are vulnerable to various security threats. Although various defense mechanisms have been proposed, they are typically non-adaptive and tailored to specific types of attacks, leaving them insufficient in the face of adaptive or mixed attacks. In this work, we formulate adversarial federated learning as a Bayesian Stackelberg Markov game (BSMG) to tackle poisoning attacks of unknown/uncertain types. We further develop an efficient meta-learning approach to solve the game, which provides a robust and adaptive FL defense. Theoretically, we show that our algorithm provably converges to the first-order ε -equilibrium point in O ( ε -2 ) gradient iterations with O ( ε -4 ) samples per iteration. Empirical results show that our meta-Stackelberg framework obtains superb performance against strong model poisoning and backdoor attacks with unknown/uncertain types.

1 Introduction

Federated learning (FL) allows multiple devices with private data to jointly train a model without sharing their local data [39]. However, FL systems are vulnerable to various adversarial attacks such as untargeted model poisoning attacks (e.g., IPM [68], LMP [15]) and backdoor attacks (e.g., BFL [2], DBA [71]). To address these vulnerabilities, various robust aggregation rules such as Krum [7], coordinate-wise trimmed mean [69], and FLTrust [10] have been proposed to defend against untargeted attacks, and both training-stage and post-training defenses such as Norm bounding [57], NeuroClip [62], and Prun [64] have been proposed to mitigate backdoor attacks. Further, dynamic defenses that myopically adapt parameters such as learning rate [45], norm clipping threshold [21], and regularizer [1] have been proposed. However, state-of-the-art defenses remain inadequate in countering advanced adaptive attacks (e.g., the reinforcement learning (RL)-based attacks [31, 32]) that dynamically adjust the attack strategy to obtain long-term advantages. Further, current defenses are typically designed to counter specific types of attacks, rendering them ineffective in the presence of mixed attacks. As shown in Table 1 (Section 4), simply combining existing defenses with manual tuning proves ineffective due to the interference between defense methods, the defender's lack of information about adversaries, and the dynamic nature of FL.

In this work, we propose a meta-Stackelberg game (meta-SG) framework that obtains superb defense performance even in the presence of strong adaptive attacks and a mix of attacks of the same or

different types (e.g., the coexistence of model poisoning and backdoor attacks). Our meta-SG defense 31

framework is built upon the following key observations. First, when the attack type (to be defined in 32

Section 2.1) is known as priori, the defender can utilize the limited amount of local data at the server 33

and publicly available information to build an approximate world model of the FL system. This 34

allows the defender to identify a robust defense policy offline by solving either a Markov decision 35

process (MDP) when the attack is non-adaptive or a Markov game when the attack is adaptive. This 36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

approach naturally applies to both a single attack and the coexistence of multiple attacks and can potentially produce a (nearly) optimal defense. Second, when the attacks are unknown or uncertain, as in more realistic settings, the problem can be formulated as a Bayesian Stackelberg Markov game (BSMG) [52], which provides a general model for adversarial FL. However, the standard solution concept for BSMG, namely, the Bayesian Stackelberg equilibrium, targets the expected case and does not adapt to the actual attack with an unknown/uncertain type.

Motivated by this limitation, we propose a novel solution concept called meta-Stackelberg equilibrium (meta-SE) for BSMG as a principled way of developing robust and adaptive defenses for FL. By integrating meta-learning and Stackelberg reasoning, meta-SE offers a computationally efficient approach to address information asymmetry in adversarial FL and enables strategic adaptation in online execution in the presence of multiple (adaptive) attackers. Before training an FL model, a meta policy is learned by solving the BSMG using experiences sampled from a set of possible attacks. When facing an actual attacker during FL training, the meta-policy is quickly adapted using a relatively small number of samples collected on the fly. The proposed meta-SG framework only requires a rough estimate of possible worst-case attacks during meta-training, thanks to the generalization ability brought by meta-learning.

To solve the BSMG in the pre-training phase, we propose a meta-Stackelberg learning (meta-SL) algorithm based on the debiased meta-reinforcement learning approach in [14]. The meta-SL provably converges to the first-order ε -approximate meta-SE in O ( ε -2 ) iterations, and the associated sample complexity per iteration is of O ( ε -4 ) . Even though meta-SL achieves state-of-the-art sample efficiency presented in [24], its operation involves the Hessian of the defender's value function. To obtain a more practical solution (to bypass the Hessian computation), we further propose a fully first-order pre-training algorithm, called Reptile meta-SL, inspired by Reptile [43]. Reptile meta-SL only utilizes the first-order stochastic gradients from the attacker's and the defender's problem to solve for the approximate equilibrium. The numerical results in Table 1 demonstrate its effectiveness in handling various types of non-adaptive attacks, including mixed attacks , while Figure 2 and Figure 9 highlight its efficiency in coping with uncertain or unknown attacks, including adaptive attacks. Due to the space limit, we move related work section to Appendix A. Our contributions are summarized as follows:

We address critical security problems in FL in the face of attacks that may be adaptive or mixed with multiple types.

We develop a Bayesian Stackelberg game model (Section 2.2) to capture the information asymmetry in the adversarial FL under multiple uncertain/unknown attacks.

To create a strategically adaptable defense, we propose a new equilibrium concept: metaStackelberg equilibrium (meta-SE), where the defender (the leader) commits to a meta policy and an adaptation strategy, leading to a data-driven approach to tackle information asymmetry.

To learn the meta equilibrium defense in the pre-training phase, we develop meta-Stackelberg learning (Algorithm 1), an efficient first-order meta RL algorithm, which provably converges to ε -approximate equilibrium in O ( ε -2 ) gradient steps with O ( ε -4 ) samples per iteration, matching the state-of-the-art efficiency in stochastic bilevel optimization.

We conduct extensive experiments in real-world settings to demonstrate the superb performance of our proposed method.

2 Meta Stackelberg Defense Framework

2.1 Framework Overview

As shown in Figure 1, the meta-learning framework includes two stages: pre-training , online adaptation . The pre-training stage is implemented in a simulated environment, which allows sufficient training using trajectories generated from the interactions between the defender and the attacker with its type randomly sampled from a set of potential attacks. Both adaptive and nonadaptive attacks could be considered for pre-training. After obtaining a meta-policy, the defender will interact with the real FL environment in the online adaptation stage to tune its defense policy using feedback (i.e., model updates and environment parameters) received in the face of real attacks that

Pre-training

OnlineAdaptation

Attack

Self-generated data

Domain

01+3456789

Uncertain

0123456789

Attack

Server

9k

Server

+

Aggr(),h()

Adapted Defense

GAN-based trigger

72

Surrogate

RL

9

FeedbackT

,t+1

9

w-Aggr({},{9})

00000

adaptive attack non-adaptive

attack

Preparation

Data

Unknown

Attack

Environment

Simulated

Real FLEnvironment

are not necessarily in the pre-training attack set. Finally, at the last round of FL training, the defender 89 will perform a post-training defense on the global model, which may or may not be considered in the 90 design of intelligent attacks. Pre-training and online adaptation are indispensable in the proposed 91 framework. Table 5 in Appendix D indicate that directly applying defense learned from pre-training 92 without online adaptation, as well as adaptation from a randomly initialized defense policy without 93 pre-training, both fail to address malicious attacks. 94

95

96

97

98

99

100

FL objective. Consider a learning system that includes one server and n clients, each client possesses its own private dataset D i = ( x j i , y j i ) | D i | j =1 where | D i | is the size of the dataset for the i -th client. Let U = { D 1 , D 2 , . . . , D n } denote the collection of all client datasets. The objective of federated learning is to obtain a model w that minimizes the average loss across all the devices: min w F ( w ) := 1 n ∑ n i =1 f ( w,D i ) , where f ( w,D i ) := 1 | D i | ∑ | D i | j =1 ℓ ( w, ( x j i , y j i )) is the local empirical loss with ℓ ( · , · ) being the loss function.

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

Attack objective. We consider two major categories of attacks: untargeted model poisoning attacks and backdoor attacks. An untargeted model poisoning attack aims to maximize the average model loss, i.e., min w -F ( w ) , while a targeted one strives to cause misclassification of poisoned test inputs to one or more target labels (e.g., backdoor attacks). A malicious client i employing targeted attack first produces a poisoned dataset D ′ i by altering a subset of data samples ( x j i , y j i ) ∈ D i to (ˆ x j i , c ∗ ) . Here ˆ x j i is the tainted sample with a backdoor trigger inserted, and c ∗ = y j i , c ∗ ∈ C is the targeted label. Let ρ i = | D ′ i | / | D i | denote the poisoning ratio, which is typically unknown to the defender. To simplify the notation, we assume that among the M = M 1 + M 2 malicious clients, the first M 1 malicious clients carry out a targeted attack, and the following M 2 malicious clients undertake an untargeted attack. Note that clients in the same category may use different attack methods. Then, the joint objective of these malicious clients is min w F ′ ( w ) := 1 M 1 ∑ M 1 i =1 f ( w,D ′ i ) -1 M 2 ∑ M i = M 1 +1 f ( w,D i ) .

̸

FL process. At each round t out of H rounds of FL training, the server randomly selects a subset of clients S t and sends them the most recent global model w t g . Every benign client k ∈ S t updates the model using their local data via one or more iterations of stochastic gradient descent and returns the model update g t k to the server. In contrast, an adversary j ∈ S t creates a malicious model update ˜ g t j and sends it back. The server then collects the set of model updates { ˜ g t i ∪ ˜ g t j ∪ g t k } i,j,k ∈S t , for i ∈ { 1 , . . . , M 1 } , j ∈ { M 1 +1 , . . . , M } , k ∈ S t \ [ M ] , utilizes an aggregation rule Aggr to combine them, and updates the global model: w t +1 g = w t g -η t Aggr ( ˜ g t i ∪ ˜ g t j ∪ g t k ) , which is then sent to clients in round t +1 . At the end of each round, the defender will perform a post-training defense h ( · ) on the global model ̂ w t g = h ( w t g ) to evaluate the current defense performance. Only at the final round H or whenever a client is leaving the FL systems, the global model with post-training defense ̂ w t g will be sent to all (leaving) clients.

Attack types. To simplify the exposition, we assume that a single mastermind attacker controls all malicious clients within the FL system and employs diverse attack strategies on each controlled client. We introduce the concept of attack type to differentiate various attack scenarios, which typically include the following three aspects. The first aspect is the attack objective chosen by a malicious client. Let Ω 1 be the set of all possible attack objectives from the defender's knowledge base. We set Ω 1 = { untargeted, targeted } in this work. The second aspect specifies the attack method (i.e., the

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

algorithm used to generate the actual attack policy) adopted by a malicious client. Let Ω 2 be the set of all possible attack methods from the defender's knowledge base. The third aspect captures the configuration associated with an attack method, including its hyperparameters and other attributes (e.g., triggers implanted in backdoor attacks, labels used in targeted attacks, and attacker's knowledge about the FL system). Let Ω 3 denote the set of all possible configurations. For each malicious client i , the tuple ( ω 1 , ω 2 , ω 3 ) i where ω k ∈ Ω k for each k fully specifies its particular attack type. Let ξ = { ( ω 1 , ω 2 , ω 3 ) i } i M =1 be the joint attack type. Further, let Ξ = (Ω 1 × Ω 2 × Ω 3 ) M denote the domain of attacks that the defender is aware of. Table 2 in Appendix C gives the types of all the attacks considered in this work. However, the actual attack type encountered during FL training is not necessary in Ξ , although it is presumably similar to a known type in Ξ .

2.2 Pre-training as a Bayesian Stackelberg Markov game

From the discussion above, the global model updates and the final output are jointly influenced by the defender (through aggregation) and the malicious clients (through corrupted gradients). Hence, the FL process in an adversarial environment can be formulated as a two-player discrete time Bayesian Stackelberg Markov game (BSMG) defined by a tuple ⟨ S, A D , A ξ , T , r, γ, H ⟩ . Using discrete time index t (one step corresponds to one FL round), we have the following.

S is the state space, and its elements represent the global model at each round s t = w t g .

A D is the defender's action set. Each action a t D represents a combination of the robust aggregation and post-training defenses: a t D = { Aggr ( · ) , h ( · ) } .

A ξ is the typeξ attacker's action set. Each action includes the joint model updates of all malicious clients: a t A = { ˜ g t i } M 1 i =1 ∪ { ˜ g t i } i M = M 1 +1 .

T ( s t +1 | s t , Aggr ( · ) , a t A ) specifies the distribution of the next state given the current state and joint actions at t , which is determined by the global model update: w t +1 g = w t g -η t Aggr ( ˜ g t i ∪ ˜ g t j ∪ g t k ) .

r D , r ξ are the defender's and the attacker's reward functions (to be maximized), respectively. The defender aims to minimize the loss after the post-training: r t D := -F ( ̂ w t g ) where ̂ w t g = h ( w t g ) . The attacker's r t ξ is given by the joint attack objective: -F ′ ( ̂ w t g ) .

Remark 2.1 . The post-training defense is only applied in the final round or to a client leaving the FL system and does not interfere with the model updates on w t g . The defender's reward function is crafted to encompass post-training, as we prioritize a practical, long-term average reward within an online process, which enables clients to seamlessly join and depart from the FL system. This design enables us to incorporate a post-training defense along with techniques for modifying the model structure, such as drop-off and pruning.

Simulated environment in the white-box setting. With the game model defined above, the defender (i.e., the server) can, in principle, identify a strong defense by solving the game (we discuss different solution concepts in Section 3). Due to efficiency and privacy concerns in FL, however, it is often infeasible to solve the game in real time when facing the actual attacker. Instead, the defender can create a simulated environment to approximate the actual FL system during the pre-training stage. The main challenge, however, is that the defender often lacks information about the individual devices in FL. We first consider the white-box setting where the defender is aware of the number of malicious devices in each category (i.e., M 1 and M 2 ) and their actual attack types, as well as the non-i.i.d. level (to be defined in Section 4.1) of local data distributions across devices. However, it does not have access to individual devices' local data and random seeds, making it difficult to simulate clients' local training and evaluate rewards. To this end, we assume that the server has a small amount of root data randomly sampled from the the collection of all client dataset U as in previous work [10, 40]. We then use generative model (e.g., conditional GAN model [41] for MNIST and diffusion model [55] for CIFAR-10 in our experiments) to generate as much data as necessary to mimic the local training (see details in Appendix C). We give an ablation study (Table 6) in Appendix D to evaluate the influence of limited/biased root data. We remark that the purpose of pre-training is to derive a defense policy rather than the model itself. Directly using the shifted data (root or generated) to train the FL model will result in low model accuracy (see Table 5 in Appendix D).

Handling the black-box setting. We then consider the more realistic black-box setting, where the defender has no access to the number of malicious devices and their actual attack types,

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

nor the non-i.i.d. level of local data distributions. To obtain a robust defense, we assume the server considers the worst-case scenario based on a rough estimate of the missing information (see our ablation study in the experiment section) and adopts the RL-based attacks to simulate the worst-case attacks (see Section 3.1) when the attack is unknown or adaptive. In the face of an unknown backdoor attack, the defender does not know the backdoor triggers and targeted labels. To simulate a backdoor attacker's behavior, we first implement multiple GAN-based attack models as in [12] to generate worst-case triggers (which maximizes attack performance given the backdoor objective) in the simulated environment. Since the defender does not know the poisoning ratio ρ i and the target label of the attacker's poisoned dataset (needed to determine the attack objective F ′ ), we approximate the attacker's reward function by r t A = -F ′′ ( ̂ w t +1 g ) , where F ′′ ( w ) := min c ∈ C [ 1 M 1 ∑ M 1 i =1 1 | D ′ i | ∑ | D ′ i | j =1 ℓ ( w, (ˆ x j i , c ))] -1 M 2 ∑ M i = M 1 +1 f ( ω, D i ) . F ′′ differs F ′ only in the first M 1 clients, where we use a strong target label (that minimizes the expected loss) as a surrogate to the true label c ∗ . We compare the defense performance against white-box and black-box backdoor attacks ( see Figure 10 in Appendix D).

3 Meta Stackelberg Learning

Since the pre-training is modeled by a Bayesian Markov Stackelberg game, solving the game efficiently is crucial to a successful defense. This work's main contribution includes the formulation of a new solution concept to the game, meta-Stackelberg equilibrium (meta-SE), and a learning algorithm to approximate such equilibrium in finite time. To motivate the proposed concept, we begin by addressing the defense against non-adaptive attacks.

Consider the attacker employing a non-adaptive attack of type ξ ; in other words, the attack action at each iteration is determined by a fixed attack strategy π ξ , where π ξ ( a ) gives the probability of taken action a ∈ A ξ , independent of the FL training and the defense strategy. In this case, BSMG reduces to an MDP, where the transition kernel is T ξ ( ·| s, a D ) ≜ ∫ A ξ T ( ·| s, a A , a D ) dπ ξ ( a A ) . Parameterizing the defender's policy π D ( a t D | s t ; θ ) by a neural network with model weights θ ∈ Θ , the solution to the following optimization problem max θ ∈ Θ E a t D ∼ π D ,s t ∼T ξ [ ∑ H t =1 γ t r t D ] ≜ J D ( θ, ξ ) gives the optimal defense against the non-adaptive attack. When the actual attack in the online stage falls within Ξ , which the defender is uncertain of, one can consider the defense against the expected attack: max θ E ξ ∼ Q J D ( θ, ξ ) , where Q is a distribution over the attack domain to be designed by the defender. One intuitive design is to include all reported attack methods in history as the attack domain and their empirical frequency as the Q distribution.

In stark contrast to non-adaptive attacks, an adaptive attack can adjust attack actions to the FL environment and the defense mechanism [31, 32]. Most existing attacks are history-independent [50, 65]. Hence, we assume that an adaptive attack takes the current state (global model) as input, i.e., the attack policy is a Markov policy denoted by π A ( a t A | s t ) . Denoted by ξ the attack type; then, an optimal adaptive attack policy, parameterized by ϕ , is the best response to the existing defense π D ( ·| s t ; θ ) : ϕ ∈ arg max E a t A ∼ π ξ ,a t D ∼ π D [ ∑ H t =1 γ t r t ξ ] ≜ J A ( θ, ϕ, ξ ) . Denote by ϕ ∗ ξ the maximizer, and then, the defender's cumulative rewards under such attack is J D ( θ, ϕ ∗ ξ , ξ ) ≜ E a t A ∼ π ξ ,a t D ∼ π D [ ∑ H t =1 γ t r t D ] .

3.1 RL-based attacks and defenses

The actual attack type (which could be either adaptive or non-adaptive) encountered in the online phase may be not in Ξ and thus unknown to the defender. To prepare for these unknown attacks, we propose to use multiple RL-based attacks with different objectives, adapted from RL-based untargeted model poising attack [31] and RL-based backdoor attack [32], as surrogates for unknown attacks, which are added to the attack domain for pre-training. The rationale behind the RL surrogates includes: (1) they achieve strong attack performance by optimizing long-term objectives; (2) they adopt the most general action space (i.e., model updates), which allows them to mimic any adaptive or non-adaptive attacks given the corresponding objectives; (3) they are flexible enough to incorporate multiple attack methods by using RL to tune the hyper-parameters of a mixture of attacks. A similar argument applies to RL-based defenses. We remark that in this paper, an RL-based attack (defense) is not a single attack (defense) as in [31, 32] but a systematically synthesized combination of existing attacks (defenses). In the simulated environment, we train our defense against the strongest white-box

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

RL attacks in [31, 32] with different objectives (e.g., untargeted or targeted), which is considered the optimal attack strategy. The 'worst-case' scenario is commonly used in security scenarios to ensure the associated defense has performance guarantees under 'weaker' attacks with similar objectives. Such a robust defense policy gives us a good starting point to further adapt to uncertain or unknown attacks. Our defense is generalizable to other adaptive attacks (see Table 8 in Appendix D). The key novelty of our RL-based defense is that instead of using a fixed and hand-crafted algorithm as in existing approaches, we use RL to optimize the policy network π D ( a t D | s t ; θ ) . Similar to RL-based attacks, the most general action space could be the set of global model parameters. However, the high dimensional action space will lead to an extremely large search space that is prohibitive in terms of training time and memory space. Thus, we apply compression techniques (see Appendix C) to reduce the action from high-dimensional space to a 3-dimensional space. Note that the execution of our defense policy is lightweight, without using any extra data for evaluation/validation. See the discussion in Appendix C on how we apply our RL-based defense during online adaptation.

3.2 Meta-Stackelberg equilibrium

As discussed in Section 2.2, one of the key challenges to solving the BSMG is the defender's incomplete information on attack types. Prior works have explored a Bayesian equilibrium approach to address this issue [52]. Given the set of possible attacks Ξ that the defender is aware of and a prior distribution Q over the domain, the Bayesian Stackelberg equilibrium (BSE) is given by the following bi-level optimization:

In (BSE), unaware of the exact attacker type, the defender (the leader) aims to maximize the defense performance against an average of all attack types, anticipating their best responses.

From a game-theoretic viewpoint, the Bayesian equilibrium in (BSE) is of ex-ante. The defender determines its equilibrium strategy only knowing the type distribution Q . However, as the Markov game proceeds, the attacker's moves (e.g., malicious global model updates) during the interim stage (online stage) reveal additional information on the attacker's private type. This Bayesian equilibrium defense strategy fails to handle the emerging information on the attacker's hidden type in the interim stage, as the policy obtained from (BSE) remains fixed throughout the online stage without adaptation.

To address the limitation of Bayesian equilibrium, we introduce the novel solution concept, metaStackelberg equilibrium (meta-SE), to equip the defender with online responsive intelligence under incomplete information. As a synthesis of meta-learning and Stackelberg equilibrium, the meta-SE aims to pre-train a meta policy on a variety of attack types sampled from the attack domain Ξ such that online gradient adaption applied to the base produces a decent defense against the actual attack in the online environment. Using mathematical terms, we denote by τ ξ := ( s k , a k D , a k ξ ) H k =1 the trajectory of the FL system under typeξ attacker up to round H , which is subject to the distribution q ( θ, ξ ) := ∏ H t =1 π D ( a t D | s t ; θ ) π ξ ( a t A | s t ) T ( s t +1 | s t , a t D , a t A ) . Let ˆ ∇ θ J D ( τ ) be the unbiased estimate of the policy gradient ∇ θ J D using the sample trajectory τ ξ (see Appendix E). Then, a one-step gradient adaptation using the sample trajectory is given by θ + η ∇ θ J D . Incorporating this gradient adaptation into (BSE) leads to the proposed meta-SE.

The idea of adding the gradient adaptation to the equilibrium is inspired by the recent developments in gradient-based meta-learning [16, 43]. When the attack is non-adaptive, the BSMG reduces to MDP problem, as delineated at the beginning of this section. Consequently, (meta-SE) turns into the standard form of meta-learning [16]. Unlike the conventional (BSE), the solution to (meta-SE gives the defender a decent defense initialization after pre-training whose gradient adaptation in the online stage is tailored to type ξ , since the online trajectory follows the distribution q ( θ, ξ ) . The novelty of (meta-SE) lies in that the leader (defender) determines an optimal adaptation scheme rather than a policy, which is computed using an online trajectory without knowing the actual type, creating a data-driven strategic adaptation after the pre-training. Besides equation BSE, Appendix G also compares the perfect Bayesian equilibrium with the proposed meta-SE, highlighting the latter's scalability to complex FL systems.

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

3.3 Meta-Stackelberg learning

Unlike finite Stackelberg Markov games that can be solved (approximately) using mixedinteger programming [59] or Q-learning [52], our BSMG admits high-dimensional continuous state and action spaces, posing a more challenging computation issue. Hence, we resort to a two-timescale policy gradient (PG) algorithm, referred to as meta-Stackelberg learning (meta-SL) presented in Algorithm 1, to solve for the meta-SE in a similar vein to [33]. In plain words, meta-SL first learns the attacker's best response at a fast scale (lines 13-15), based on which it updates the defender's meta policy at a slow scale at each iteration using either debiased meta-learning [14] or reptile [43]. The two-timescale meta-SL alleviates the nonstationarity caused by concurrent policy updates from both players [70]. Of particular note is that the debiased meta-learning involves Hessian computation when evaluating the gradient of the defender's objective function since the attacker's best response ϕ ∗ ξ ( θ ) also depends on θ . In contrast, reptile uses a first-order approximation to avoid Hessian. The mathematical subties between two policy gradient estimations are deferred to the Appendix E.

Algorithm 1 Meta-Stackelberg Learning

1: Input: the distribution Q (Ξ) , initial defense meta policy θ 0 , pre-defined attack methods { π ξ } ξ ∈ Ξ , pretrained RL attack policies { ϕ 0 ξ } ξ ∈ Ξ , step size parameters κ D , κ A , η , and iterations numbers N A , N D ;

2:

3:

4:

5:

6:

7:

8:

9:

10:

11:

12:

13:

14:

15:

16:

17:

18:

19:

20:

;

D

θ

Output:

for

N

iteration

t

if

D

to

= 0

N

1

-

do meta-RL

(for non-adaptive)

Sample a batch of then

K

attack types

Estimate

D

J

(

ξ

) :=

∇

ˆ

end if

if meta-SG

then

Sample a batch of

K

attack types for

each sampled attack do

ξ

Apply one-step adaptation

θ

t

t

t

t

ξ

θ

D

θ

+

η

J

∇

ˆ

(

θ

, ϕ

ϕ

t

ξ

t

ξ

(0)

ϕ

for

;

iteration

k

ϕ

t

ξ

(

k

+1)

κ

end for

∇

D

J

(

ξ

ˆ

end for end if

θ

)

D

= 0

A

ϕ

∇

ˆ

/K

ξ

, ξ

, . . . , N

ϕ

J

ˆ

∇

t

ξ

A

(

A

θ

k

)+

(

θ

t

ξ

J

D

∑

ξ

, ϕ

t

ξ

(

ˆ

∇

(

t

ξ

θ, ϕ

J

D

(

ξ

k

(

)

θ

κ

21: end for The rest of this subsection addresses the computation expense of the proposed meta-SL. We begin with an alternative solution concept for our first-order gradient algorithm, which is slightly weaker than (meta-SE). Let L D ( θ, ϕ, ξ ) ≜ E τ ∼ q J D ( θ + η ˆ ∇ θ J D ( τ ) , ϕ, ξ ) , L A ( θ, ϕ, ξ ) ≜ E τ ∼ q J A ( θ + ˆ ∇ θ J D ( τ ) , ϕ, ξ ) , for a fixed type ξ ∈ Ξ . In the sequel, we will assume L D and L A to be continuously twice differentiable and Lipschitzsmooth with respect to both θ and ϕ as in [33], see Appendix F.

Definition 3.1. For ε ∈ (0 , 1) , a pair ( θ ∗ , { ϕ ∗ ξ } ξ ∈ Ξ ) ∈ Θ × Φ | Ξ | is a ε -meta First-Order Stackelbeg Equilibrium ( ε -meta-FOSE) of the meta-SG if it satisfies the following conditions: for ξ ∈ Ξ , max θ ∈ B ( θ ∗ ) ⟨∇ θ L D ( θ ∗ , ϕ ∗ ξ , ξ ) , θ -θ ∗ ⟩ ≤ ε , max ϕ ∈ B ( ϕ ∗ ξ ) ⟨∇ ϕ L A ( θ ∗ , ϕ ∗ ξ , ξ ) , ϕ -ϕ ∗ ξ ⟩ ≤ ε , where B ( θ ∗ ) := { θ ∈ Θ : ∥ θ -θ ∗ ∥ ≤ 1 } , and B ( ϕ ∗ ξ ) := { ϕ ∈ Φ : ∥ ϕ -ϕ ∗ ξ ∥ ≤ 1 } .

Definition 3.1 contains the necessary equilibrium condition for meta-SE in (meta-SE), which can be reduced to ∥∇ θ L D ( θ ∗ , ϕ ξ , ξ ) ∥ ≤ ε and ∥∇ ϕ L A ( θ ∗ , ϕ ξ , ξ ) ∥ ≤ ε in the unconstraint settings. Since we utilize stochastic gradient in practice, all inequalities mentioned above shall be considered in expectation. The existence of meta-FOSE is guaranteed Theorem F.1 in Appendix F.

Since the value functions J A , J D are nonconvex, we impose a regularity assumption adapted from the Polyak-Łojasiewicz (PL) condition [26], which is customary in nonconvex analysis. Despite the lack of theoretical justifications for the PL condition in the literature, [33] empirically demonstrates that the cumulative rewards in meta-reinforcement learning satisfy the PL condition, see Figure 4 Appendix D therein. Assumption 3.2 subsequently leads to the main result in Theorem 3.3

Assumption 3.2 (Stackelberg Polyak-Łojasiewicz condition) . There exists a positive constant µ such that for any ( θ, ϕ ) ∈ Θ × Φ and ξ ∈ Ξ , the following inequalities hold: 1 2 µ ∥∇ ϕ L D ( θ, ϕ, ξ ) ∥ 2 ≥ max ϕ L D ( θ, ϕ, ξ ) -L D ( θ, ϕ, ξ ) , 1 2 µ ∥∇ ϕ L A ( θ, ϕ, ξ ) ∥ 2 ≥ max ϕ L A ( θ, ϕ, ξ ) -L A ( θ, ϕ, ξ ) .

Theorem 3.3. Under assumption 3.2 and other regularity assumptions in Appendix F, for any given ε ∈ (0 , 1) , let the learning rates κ A and κ D be properly chosen; let N A ∼ O (log ϵ -1 ) and N b ∼ O ( ϵ -4 ) be properly chosen (Appendix F), then, Algorithm 1 finds a ε -meta-FOSE within N D ∼ O ( ε -2 ) iterations.

)

;

)

-

1

, ξ

N

)

A

)

from

ξ

θ, π

ξ

, ξ

ξ

θ

=

)

|

∈

Ξ

do

;

, ξ

;

=

θ

t

+1

t

ˆ

∇

θ

J

D

(

|

)

θ

Ξ

;

θ

t

ξ

;

t

ξ

;

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

Finally, we conclude this section by analyzing the meta-SG defense's generalization ability when the learned meta policy is exposed to attacks unseen in the pre-training. Proposition 3.4 asserts that meta-SG is generalizable to the unseen attacks, given that the unseen is not distant from those seen. The formal statement is deferred to Appendix F.

Proposition 3.4. Consider sampled attack types ξ 1 , . . . , ξ m during the pre-training and the unseen attack type ξ m +1 in the online stage. The generalization error is upper-bounded by the 'discrepancy' between the unseen and the seen attacks C ( ξ m +1 , { ξ i } i m =1 ) .

4 Experiments

4.1 Experiment Settings

Dataset. Our experiments are conducted on MNIST [30] and CIFAR-10 [28] datasets with a CNN classifier and ResNet-18 model respectively (see Appendix C for details). We consider horizontal FL and adopt the approach introduced in [15] to measure the diversity of local data distributions among clients. Let the dataset encompass C classes, such as C = 10 for datasets like MNIST and CIFAR-10. Client devices are divided into C groups (with M attackers evenly distributed among these groups). Each group is allocated 1 /C of the training samples in the following manner: a training instance labeled as c is assigned to the c -th group with a probability of q ≥ 1 /C , while being assigned to every other group with a probability of (1 -q ) / ( C -1) . Within each group, instances are evenly distributed among clients. A higher value of q signifies a greater non-i.i.d. level. By default, we set q = 0 . 5 as the standard non-i.i.d. level. We assume the server holds a small amount of root data randomly sampled from the the collection of all client dataset U . ( 100 for MNIST and 200 for CIFAR-10).

Baseline. We evaluate our meta-RL and meta-SG defenses under the following untargeted model poisoning attacks including IPM [68] (with scaling factor 2 ), LMP [15], RL [31], and backdoor attacks including BFL [2] (with poisoning ratio 1 ), DBA [67] (with 4 sub-triggers evenly distributed to malicious clients and poisoning ratio 0 . 5 ), BRL [32], and a mix of attacks from the two categories (see Table 2 for all attacks' categories in Appendix C). We consider various strong defenses as baselines, including training-stage defenses such as Coordinate-wise trimmed mean/median [69], Norm bounding [57], FLTrust [10], Krum [7], and post-training stage defenses such as NeuroClip [62] and Prun [64] and the selected combination of them. We utilize the Twin Delayed DDPG (TD3) [18] algorithm to train both attacker's and defender's policies. We use the following default parameters: number of devices = 100 , number of malicious clients for untargeted model poisoning attack = 10 , number of malicious clients for backdoor attack = 5 ( 20 for DBA), client subsampling rate = 10% , number of FL epochs = 500 ( 1000 ) for MNIST (CIFAR-10). We fix the initial model and the random seeds for client subsampling and local data sampling for fair comparisons. The details of the experiment setup and additional results are provided in Appendices C and D.

4.2 Experiment Results

Effectiveness against single/multiple type of attacks. We examine the defense performance of our meta-RL compared with other defense combinations in Table 1 based on average global model accuracy after 500 FL rounds on CIFAR-10, which measures the success of defense and learning speed ignoring the randomness influence (corner-case updates, bias data, etc.) at the bargaining stage of FL. The meta-RL first learns a meta-defense policy from the attack domain involving { NA, IPM,

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

LMP MNIST

RL MNIST

LMP CIFAR10

RL CIFAR10

8'0

0.6

0.6

0.75

ClipMed

0.6

0.4

0.50

meta-RL

meta-SG

100

200

FL rounds

300

400

ClipMed

meta-SG+

meta-SG

500

0.25

0.00

0

100

200

FL rounds

300

400

500

200

400

FL rounds

600

008

ClipMed

meta-SG

meta-SG+

1000

0.2

200

400

FLrounds

600

800

ClipMed

meta-RL

meta-SG

1000

LMP, BFL, DBA } , then adapts it to the real single/mixed attack. We observe that multiple types of attacks may intervene with each other (e.g., IPM+BFL, LMP+DBA), which makes it impossible to manually address the entangled attacks. It is not surprising to see FedAvg [39] and defenses specifically designed for untargeted attacks (i.e., Trimmed mean, FLTrust) fail to defend backdoor attacks (i.e., BFL, DBA) due to the huge deviation of defense objective from the optimum. For a fair comparison, we further manually tune the norm threshold (more results in Appendix D) from [0 . 01 , 0 . 02 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 5 , 1] for ClipMed (i.e., Norm bounding + Coordinate-wise Median) and clipping range from [2 : 2 : 10] for FLTrust + NeuroClip to achieve the best performance to balance the global model and backdoor accuracy in linear form (i.e., Acc - Bac). Intuitively, a tight threshold/range has better performance in defending against backdoor attacks, yet will hinder or even damage the FL progress. On the other hand, a loose threshold/range fails to defend backdoor injection. Nevertheless, manually tuning in real-world FL scenarios is nearly impossible due to the limited knowledge of the ongoing environment and the presence of asymmetric adversarial information. Instead of suffering from the above concerns and exponential growth of parameter combination possibilities, our data-driven meta-RL approach can automatically tune multiple parameters at each round. Targeting the cumulative defense rewards, the RL approach naturally holds more flexibility than myopic optimization.

Adaptation to uncertain/unknown attacks. To evaluate the necessity and efficiency of adaptation from the meta-SG policy in the face of unknown attacks, we plot the global model accuracy graph over FL epochs. The meta-RL pre-trained from non-adaptive attack domain { NA, IPM, LMP, BFL, DBA } (RL attack is unknown), while meta-SG pre-train from interacting with a group of RL attacks initially target on { FedAvg, Coordinate-wise Median, Norm bounding, Krum, FLTrust } (LMP is unknown). The meta-SG plus (i.e., meta-SG+) is a pre-trained model from the combined attack domain of the above two. All three defenses then adapt to the real FL environments under LMP or RL attacks. As shown in Figure 2, the meta-SG can quickly adapt to both uncertain RL-based adaptive attack (attack action is time-varying during FL) and unknown LMP attack, while meta-RL can only slowly adapt to or fail to adapt to the unseen RL-based adaptive attacks on MNIST and CIFAT-10 respectively. In addition, the first and the third Figures in Figure 2 demonstrate the power of meta-SG against unknown LMP attacks, even if LMP is not directly used during its pre-training stage. The results are only slightly worse than meta-SG plus, where LMP is seen during pre-training. Similar observations are given under IPM in Appendix D.

5 Conclusion

We have proposed a meta-Stackelberg framework to tackle attacks of uncertain or unknown types in 408 federated learning through data-driven adaptation. The proposed meta-Stackelberg learning approach 409 is computationally tractable and strategically adaptable, targeting mixed and adaptive attacks under 410 incomplete information. The major limitation of our current approach pertains to privacy concerns. 411 Our current simulation necessitates that the defender either accesses a small portion of root data or 412 learns clients' data through inversion, which slightly violates the privacy principles of FL. To minimize 413 privacy risks, we train our meta-policy in a simulated environment and apply data augmentation to 414 blur the learned data. In our experiments, the current 'black-box' setting operates under certain 415 conditions: we test only one or a few agnostic variables at a time while leaving other information 416 known to the defender (see Appendix D). In our future work, we plan to incorporate additional 417 state-of-the-art defense algorithms to counter more potent attacks, such as edge-case attacks [63], as 418 well as other attack types, such as privacy-leakage attacks [37]. We will also explore new application 419 scenarios, including NLP and large generative models. Our framework could be further improved by 420 including a client-side defense mechanism that closely mirrors real-world scenarios, replacing the 421 current processes of self-data generation. 422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

References

Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In International Conference on Learning Representations , 2020.

Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics , pages 2938-2948. PMLR, 2020.

Pierre Bernhard and Alain Rapaport. On a theorem of Danskin with an application to a theorem of Von Neumann-Sion. Nonlinear Analysis: Theory, Methods & Applications , 24(8):1163-1181, 1995.

Jeremy Bernstein, Jiawei Zhao, Kamyar Azizzadenesheli, and Anima Anandkumar. signsgd with majority vote is communication efficient and fault tolerant. In International Conference on Learning Representations(ICLR) , 2018.

Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing federated learning through an adversarial lens. In International Conference on Machine Learning(ICML) , 2019.

Umang Bhaskar, Yu Cheng, Young Kun Ko, and Chaitanya Swamy. Hardness results for signaling in bayesian zero-sum and network routing games. In Proceedings of the 2016 ACM Conference on Economics and Computation , pages 479-496, 2016.

Peva Blanchard, Rachid Guerraoui, Julien Stainer, et al. Machine learning with adversaries: Byzantine tolerant gradient descent. In Advances in Neural Information Processing Systems(NeurIPS) , 2017.

Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloé Kiddon, Jakub Koneˇ cný, Stefano Mazzocchi, Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards federated learning at scale: System design. In Proceedings of Machine Learning and Systems , 2019.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.

Xiaoyu Cao, Minghong Fang, Jia Liu, and Neil Zhenqiang Gong. Fltrust: Byzantine-robust federated learning via trust bootstrapping. In Network and Distributed System Security (NDSS) Symposium , 2021.

Katherine Crowson. Trains a diffusion model on cifar-10 (version 2). https://colab.research.google.com/drive/1IJkrrV-D7boSCLVKhi7t5docRYqORtm3, 2018.

Khoa Doan, Yingjie Lao, Weijie Zhao, and Ping Li. Lira: Learnable, imperceptible and robust backdoor attacks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11966-11976, 2021.

Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl 2 : Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779 , 2016.

Alireza Fallah, Kristian Georgiev, Aryan Mokhtari, and Asuman Ozdaglar. On the convergence theory of debiased model-agnostic meta-reinforcement learning, 2021.

Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Gong. Local model poisoning attacks to byzantine-robust federated learning. In 29th USENIX Security Symposium , 2020.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap466 tation of deep networks. In International conference on machine learning , pages 1126-1135. 467 PMLR, 2017. 468

Drew Fudenberg and Jean Tirole. Game Theory . MIT Press, Cambridge, MA, 1991. 469

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error 470 in actor-critic methods. In International conference on machine learning , pages 1587-1596. 471 PMLR, 2018. 472

473

474

475

Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradientshow easy is it to break privacy in federated learning? Advances in Neural Information Processing Systems , 33:16937-16947, 2020.

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

Robin C Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspective. arXiv preprint arXiv:1712.07557 , 2017.

Yifan Guo, Qianlong Wang, Tianxi Ji, Xufei Wang, and Pan Li. Resisting distributed backdoor attacks in federated learning: A dynamic norm clipping approach. In 2021 IEEE International Conference on Big Data (Big Data) , pages 1172-1182. IEEE, 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition(CVPR) , 2016.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Kaiyi Ji, Junjie Yang, and Yingbin Liang. Bilevel optimization: Convergence analysis and enhanced design. In International conference on machine learning , pages 4882-4892. PMLR, 2021.

Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14(1-2):1-210, 2021.

Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximalgradient methods under the polyak-łojasiewicz condition. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2016, Proceedings, Part I 16 , pages 795-811. Springer, 2016.

Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems , 34:21696-21707, 2021.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Artur Lacerda. Pytorch conditional gan. https://github.com/arturml/mnist-cgan, 2018.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Henger Li, Xiaolin Sun, and Zizhan Zheng. Learning to attack federated learning: A modelbased reinforcement learning attack framework. In Advances in Neural Information Processing Systems , 2022.

Henger Li, Chen Wu, Senchun Zhu, and Zizhan Zheng. Learning to backdoor federated learning. arXiv preprint arXiv:2303.03320 , 2023.

Tao Li, Haozhe Lei, and Quanyan Zhu. Sampling attacks on meta reinforcement learning: A minimax formulation and complexity analysis. arXiv preprint arXiv:2208.00081 , 2022.

Tao Li, Guanze Peng, Quanyan Zhu, and Tamer Baar. The Confluence of Networks, Games, and Learning a Game-Theoretic Framework for Multiagent Decision Making Over Networks. IEEE Control Systems , 42(4):35-67, 2022.

Tao Li, Yuhan Zhao, and Quanyan Zhu. The role of information structures in game-theoretic multi-agent learning. Annual Reviews in Control , 53:296-314, 2022.

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

Tao Li and Quanyan Zhu. On the price of transparency: A comparison between overt persuasion and covert signaling. In 2023 62nd IEEE Conference on Decision and Control (CDC) , pages 4267-4272, 2023.

Lingjuan Lyu, Han Yu, Xingjun Ma, Chen Chen, Lichao Sun, Jun Zhao, Qiang Yang, and S Yu Philip. Privacy and robustness in federated learning: Attacks and defenses. IEEE transactions on neural networks and learning systems , 2022.

Mohammad Hossein Manshaei, Quanyan Zhu, Tansu Alpcan, Tamer Bac¸ sar, and Jean-Pierre Hubaux. Game theory meets network security and privacy. ACM Comput. Surv. , 45(3), jul 2013.

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics (AISTATS) , pages 1273-1282. PMLR, 2017.

Yinbin Miao, Ziteng Liu, Hongwei Li, Kim-Kwang Raymond Choo, and Robert H Deng. Privacy-preserving byzantine-robust federated learning via blockchain systems. IEEE Transactions on Information Forensics and Security , 17:2848-2861, 2022.

Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 , 2014.

Thien Duc Nguyen, Phillip Rieger, Huili Chen, Hossein Yalame, Helen Möllering, Hossein Fereidooni, Samuel Marchal, Markus Miettinen, Azalia Mirhoseini, Shaza Zeitouni, et al. Flame: Taming backdoors in federated learning. Cryptology ePrint Archive , 2021.

Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999 , 2018.

Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In International conference on machine learning , pages 2642-2651. PMLR, 2017.

Mustafa Safa Ozdayi, Murat Kantarcioglu, and Yulia R Gel. Defending against backdoors in federated learning with robust learning rate. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 9268-9276, 2021.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703 , 2019.

Krishna Pillutla, Sham M Kakade, and Zaid Harchaoui. Robust aggregation for federated learning. IEEE Transactions on Signal Processing , 70:1142-1154, 2022.

Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research , 22(268):1-8, 2021.

Phillip Rieger, Thien Duc Nguyen, Markus Miettinen, and Ahmad-Reza Sadeghi. Deepsight: Mitigating backdoor attacks in federated learning through deep model inspection. arXiv preprint arXiv:2201.00763 , 2022.

Nuria Rodríguez-Barroso, Daniel Jiménez-López, M Victoria Luzón, Francisco Herrera, and Eugenio Martínez-Cámara. Survey on federated learning threats: Concepts, taxonomy on attacks and defences, experimental study and challenges. Information Fusion , 90:148-173, 2023.

Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512 , 2022.

Sailik Sengupta and Subbarao Kambhampati. Multi-agent Reinforcement Learning in Bayesian Stackelberg Markov Games for Adaptive Moving Target Defense. arXiv , 2020.

Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep 562 learning. Journal of big data , 6(1):1-48, 2019. 563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

Aman Sinha, Hongseok Namkoong, and John Duchi. Certifying some distributional robustness with principled adversarial training. In International Conference on Learning Representations , 2018.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H Brendan McMahan. Can you really backdoor federated learning? arXiv preprint arXiv:1911.07963 , 2019.

Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems 12 , pages 1057-1063. MIT press, 2000.

Yevgeniy Vorobeychik and Satinder Singh. Computing stackelberg equilibria in discounted stochastic games. Proceedings of the AAAI Conference on Artificial Intelligence , 26(1):14781484, Sep. 2021.

Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP) , pages 707-723, 2019.

Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP) , pages 707-723. IEEE, 2019.

Hang Wang, Zhen Xiang, David J Miller, and George Kesidis. Mm-bd: Post-training detection of backdoor attacks with arbitrary backdoor pattern types using a maximum margin statistic. In 2024 IEEE Symposium on Security and Privacy (SP) , pages 15-15. IEEE Computer Society, 2023.

Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy-yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos. Attack of the tails: Yes, you really can backdoor federated learning. Advances in Neural Information Processing Systems , 33:16070-16084, 2020.

Chen Wu, Xian Yang, Sencun Zhu, and Prasenjit Mitra. Mitigating backdoor attacks in federated learning. arXiv preprint arXiv:2011.01767 , 2020.

Geming Xia, Jian Chen, Chaodong Yu, and Jun Ma. Poisoning attacks in federated learning: A survey. IEEE Access , 11:10708-10722, 2023.

Chulin Xie, Minghao Chen, Pin-Yu Chen, and Bo Li. Crfl: Certifiably robust federated learning against backdoor attacks. In International Conference on Machine Learning , pages 11372-11382. PMLR, 2021.

Chulin Xie, Keli Huang, Pin-Yu Chen, and Bo Li. Dba: Distributed backdoor attacks against federated learning. In International conference on learning representations , 2019.

Cong Xie, Oluwasanmi Koyejo, and Indranil Gupta. Fall of empires: Breaking byzantinetolerant sgd by inner product manipulation. In Uncertainty in Artificial Intelligence (UAI) , pages 261-270. PMLR, 2020.

Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantine-robust distributed learning: Towards optimal statistical rates. In International Conference on Machine Learning , pages 5650-5659. PMLR, 2018.

Bora Yongacoglu, Gürdal Arslan, and Serdar Yüksel. Asynchronous Decentralized Q-Learning: 609 Two Timescale Analysis By Persistence. arXiv , 2023. 610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

Xianyang Zhang, Chen Hu, Bing He, and Zhiguo Han. Distributed reptile algorithm for metalearning over multi-agent systems. IEEE Transactions on Signal Processing , 70:5443-5456, 2022.

Chen Zhao, Yu Wen, Shuailou Li, Fucheng Liu, and Dan Meng. Federatedreverse: A detection and defense method against backdoor attacks in federated learning. In Proceedings of the 2021 ACM Workshop on Information Hiding and Multimedia Security , IH&MMSec '21, page 51-62, New York, NY, USA, 2021. Association for Computing Machinery.

A Related Works

Poisoning/backdoor attacks and defenses in FL Several defensive strategies against model poisoning attacks broadly fall into two categories. The first category includes robust-aggregationbased defenses encompassing techniques such as dimension-wise filtering. These methods treat each dimension of local updates individually, as explored in studies by [4, 69]. Another strategy is client-wise filtering, aiming to limit or entirely eliminate the influence of clients who might harbor malicious intent. This approach has been examined in the works of [7, 47, 57]. Some defensive methods necessitate the server having access to a minimal amount of root data, as detailed in the study by [10]. Naive backdoor attacks are limited by even simple defenses like norm-bounding [57] and weak differential private [20] defenses. Despite the sophisticated design of state-of-the-art non-addaptive backdoor attacks against federated learning, post-training stage defenses [64, 42, 49] can still effectively erase suspicious neurons/parameters in the backdoored model.

Incomplete Information in Adversarial Machine Learning Prior works have attempted to tackle the challenge of incomplete information through two distinct approaches. The first approach is the 'infer-then-counter' approach, where the hidden information regarding the attacks is first inferred through observations. For example, one can infer the backdoor triggers through reverse engineering using model weights [60], based on which the backdoor attacks can be mitigated [72]. The inference helps adapt the defense to the present malicious attacks. However, this inference-based adaptation requires prior knowledge of the potential attacks (i.e., backdoor attacks) and does not directly lend itself to mixed/adaptive attacks. Moreover, the inference and adaptation are offline, unable to counter online adaptive backdoor attack [31]. The other approach explored the notion of robustness that prepares the defender for the worst case [54, 52], which often leads to a Stackelberg game (SG) between the defender and the attacker. Yet, such a Stackelberg approach often leads to conservative defense, lacking adaptability.

B Broader Impact

Towards Universal Robust Federated Learning. Our goal is to establish a comprehensive framework for universal federated learning defense against all kinds of attacks. This framework ensures that the server remains oblivious to any details pertaining to the environment or potential attackers. Still, it possesses the ability to swiftly adapt and respond to uncertain or unknown attackers during the actual federated learning process. Nevertheless, achieving this universal defense necessitates an extensive attack set through pre-training, which often results in a protracted convergence time toward a meta-policy. Moreover, the effectiveness and efficiency of generalizing from a wide range of diverse distributions pose additional challenges. Considering these, we confine our experiments in this paper to specifically address a subset of uncertainties and unknowns. This includes variables such as the method of attacker, the number of attackers, the level of independence and identically distributed data, backdoor triggers, backdoor targets, and other relevant aspects. However, we acknowledge that our focus is not all-encompassing, and there may be other factors that remain unexplored in our research.

Meta Equilibrium and Information Asymmetry. Information asymmetry is a prevailing phenomenon arising in a variety of contexts, including adversarial machine learning (e.g. FL discussed in this work), cyber security [38], and large-scale network systems [34]. Our proposed meta-equilibrium offers a data-driven approach tackling asymmetric information structure in dynamic games without Bayesian-posterior beliefs. Achieving the strategic adaptation through stochastic gradient descent, the meta-equilibrium is computationally superior to perfect Bayesian equilibrium and better suited

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

for real-world engineering systems involving high-dimensional continuous parameter spaces. It is expected that the meta-equilibrium can also be relevant to other adversarial learning contexts, cyber defense, and decentralized network systems.

C Experiment Setup

Datasets. We consider two datasets: MNIST [30] and CIFAR-10 [28], and default i.i.d. local data distributions, where we randomly split each dataset into n groups, each with the same number of training samples. MNIST includes 60,000 training examples and 10, 000 testing examples, where each example is a 28 × 28 grayscale image, associated with a label from 10 classes. CIFAR-10 consists of 60,000 color images in 10 classes of which there are 50, 000 training examples and 10,000 testing examples. For the non-i.i.d. setting (see Figure 11(d)), we follow the method of [15] to quantify the heterogeneity of the data. We split the workers into C = 10 (for both MNIST and CIFAR-10) groups and model the non-i.i.d. federated learning by assigning a training instance with label c to the c -th group with probability q and to all the groups with probability 1 -q . A higher q indicates a higher level of heterogeneity.

Federated Learning Setting. We use the following default parameters for the FL environment: local minibatch size = 128, local iteration number = 1, learning rate = 0.05, number of workers = 100, number of backdoor attackers = 5, number of untargeted model poisoning attackers = 20, subsampling rate = 10% , and the number of FL training rounds = 500 (resp. 1000) for MNIST (resp. CIFAR-10). For MNIST, we train a neural network classifier of 8×8, 6×6, and 5×5 convolutional filter layers with ReLU activations followed by a fully connected layer and softmax output. For CIFAR-10, we use the ResNet-18 model [22]. We implement the FL model with PyTorch [46] and run all the experiments on the same 2.30GHz Linux machine with 16GB NVIDIA Tesla P100 GPU. We use the cross-entropy loss as the default loss function and stochastic gradient descent (SGD) as the default optimizer. For all the experiments except Figures 11(c) and 11(d), we fix the initial model and random seeds of subsampling for fair comparisons.

Baselines. We evaluate our defense method against various state-of-the-art attacks, including nonadaptive and adaptive untargeted model poison attacks (i.e., IPM [68], LMP [15], RL [31]), as well as backdoor attacks (BFL [2] without model replacement, BRL [32], with tradeoff parameter λ = 0 . 5 , DBA [67] where each selected attacker randomly chooses a sub-trigger as shown in Figures 6, PGD attack [63] with a projection norm of 0.05), and a combination of both types. To establish the effectiveness of our defense, we compare it with several strong defense techniques. These baselines include defenses implemented during the training stage, such as Krum [7], ClipMed [69, 57, 31] (with norm bound 1), FLTrust [10] with 100 root data samples and bias q = 0 . 5 , training stage CRFL [66] with norm bound of 0.02 and noise level 1 e -3 as well as post-training defenses like NeuroClip [62] and Prun [64]. We use the original clipping thresholds 7 in [62] and set the default Prun number to 256.

Reinforcement Learning Setting. In our RL-based defense, since both the action space and state space are continuous, we choose the state-of-the-art Twin Delayed DDPG (TD3) [18] algorithm to individually train the untargeted defense policy and the backdoor defense policy. We implement our simulated environment with OpenAI Gym [9] and adopt OpenAI Stable Baseline3 [48] to implement TD3. The RL training parameters are described as follows: the number of FL rounds = 300 rounds, policy learning rate = 0.001, the policy model is MultiInput Policy, batch size = 256, and γ = 0.99 for updating the target networks. The default λ = 0 . 5 when calculating the backdoor rewards.

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

Meta-learning Setting. The attack domains (i.e., potential attack sets) are built as following: For meta-RL, we consider IPM [68], LMP [15], EB [5] as three possible attack types. For meta-SG against untargeted model poisoning attack, we consider RL-based attacks [31] trained against Krum [7] and ClipMed [31, 69, 57] as initial attacks. For meta-SG against backdoor attack, we consider RL-based backdoor attacks [32] trained against Norm-bounding [57] and NeuroClip [62] (Prun [64]) as initial attacks. For meta-SG against mix type of attacks, we consider both RL-based attacks [31] and RL-based backdoor attacks [32] described above as initial attacks.

At the pre-training stage, we set the number of iterations T = 100 . In each iteration, we uniformly sample K = 10 attacks from the attack type domain (see Algorithm 2 and Algorithm 1). For each attack, we generate a trajectory of length H = 200 for MNIST ( H = 500 for CIFAR-10), and update both attacker's and defender's policies for 10 steps using TD3 (i.e., l = N A = N D = 10 ). At the online adaptation stage, the meta-policy is adapted for 100 steps using TD3 with T = 10 , H = 100 for MNIST ( H = 200 for CIFAR-10), and l = 10 . Other parameters are described as follows: single task step size κ = κ A = κ D = 0 . 001 , meta-optimization step size = 1 , adaptation step size = 0 . 01 .

Space Compression. Following the BSMG model, it is most generally to use w t g or ( w t g , I t ) as the state, and { ˜ g t k } M 1 + M 2 k =1 or w t +1 g as the action for the attacker and the defender, respectively, if the federated learning model is small. However, when we use federated learning to train a highdimensional model (i.e., a large neural network), the original state/action space will lead to an extremely large search space that is prohibitive in terms of training time and memory space. We adopt the RL-based attack in [31] to simulate an adaptive model poisoning attack and the RL-based local search in [32] to simulate an adaptive backdoor attack, both having a 3 -dimensioanl real action spaces after space comparison (see ). We further restrict all malicious devices controlled by the same attacker to take the same action. To compress the state space, we reduce w t g to only include its last two hidden layers for both attacker and defender and reduce I t to the number of malicious clients sampled at round t .

Our approach rests on an RL-based synthesis of existing specialized defense methods against mixed attacks, where multiple defenses can be selected at the same time and combined with dynamically tuned hyperparameters. The following specialized defenses are selected in our implementation. For training stage aggregation-based defenses, we first normalize the magnitude of all gradients to a threshold α ∈ (0 , max i ∈S t {∥ g t i ∥} ] , then apply coordinate-wise trimmed mean [69] with trimmed rate β ∈ [0 , 1) . For post-training defense, NeuroClip [62] with clip range ε or Prun [64] with mask rate σ is applied. The concrete approach used in each of the above defenses can be replaced by other defense methods. The key novelty of our approach is that instead of using a fixed and hand-crafted algorithm as in existing approaches, we use RL to optimize the policy network π D ( a t D | s t ; θ ) . Similar to RL-based attacks, the most general action space could be the set of global model parameters. However, the high dimensional action space will lead to an extremely large search space that is prohibitive in terms of training time and memory space. Thus, we apply reduce the action space to a t D := ( α t , β t , ε t /σ t ) . Note that the execution of our defense policy is lightweight, without using any extra data for evaluation/validation.

Self-generated Data. We begin by acknowledging that the server only holds a small amount of 743 initial data (200 samples with q = 0 . 1 in this work) learned from first 20 FL rounds using inverting 744 gradient [19], to simulate training set with 60,000 images (for both MNIST and CIFAR-10) for FL. 745 This limited data is augmented using several techniques such as normalization, random rotation, and 746 color jittering to create a larger and more varied dataset, which will be used as an input for generative 747

Original images

13

456789

Generatedimages

123

7.89

Originalimages

Generated images

78788

CIFAR 10

MNIST

0

10

20

0

10

20

Original

0

10

20

0

10

20

0

10

20

0

10

20

0

10

20

0

10

20

3rd Trigger

0

10

20

0

10

20

0

20

0

10

20

30

Original

0

20

0

10

20

30

0

20

0

10

20

30

0

20

0

10

20

30

3rd Trigger

0

20

0

10

20

30

4th Trigger

0

20

0

10

20

30

+

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

For MNIST, we use the augmented dataset to train a Conditional Generative Adversarial Network (cGAN) model [41, 44] built upon the codebase in [29]. The cGAN model for the MNIST dataset comprises two main components - a generator and a discriminator, both of which are neural networks. Specifically, we use a dataset with 5,000 augmented data as the input to train cGAN, keep the network parameters as default, and set the training epoch as 100.

For CIFAR-10, we leverage a diffusion model implemented in [11] that integrates several recent techniques, including a Denoising Diffusion Probabilistic Model (DDPM) [23], DDIM-style deterministic sampling [56], continuous timesteps parameterized by the log SNR at each timestep [27] to enable different noise schedules during sampling. The model also employs the 'v' objective, derived from Progressive Distillation for Fast Sampling of Diffusion Models [51], enhancing the conditioning of denoised images at high noise levels. During the training process, we use a dataset with 50,000 augmented data samples as the input to train this model, keep the parameters as default, and set the training epoch as 30.

Simulated Environment. To further improve efficiency and privacy, the defender simulate a smaller FL system when solving the game. In our experiments, we include 10 clients in pre-training while using 100 clients in the online FL system. The simulation relies on a smaller dataset (generated from root data) and endures a shorter training time (100 (500) FL rounds for MINST (CIFAR-10) v.s. 1000 rounds in online FL experiments). Although the offline simulated Markov game deviates from the ground truth, the learned meta-defense policy can quickly adapt to the real FL during the online adaptation, as shown in our experiment section.

Backdoor Attacks. We consider the trigger patterns shown in Figure 4 and Figure 6 for backdoor attacks. For triggers generated using GAN (see Figure 4), the goal is to classify all images of different classes to the same target class (all-to-one). For fixed patterns (see Figure 6), the goal is to classify images of the airplane class to the truck class (one-to-one). The default poisoning ratio is 0.5 in both cases. The global trigger in Figure 6 is considered the default poison pattern and is used for the online adaptation stage for backdoor accuracy evaluation. In practice, the defender (i.e., the server) does not know the backdoor triggers and targeted labels. To simulate a backdoor attacker's behavior, we first implement multiple GAN-based attack models as in [12] to generate worst-case triggers (which maximizes attack performance given backdoor objective) in the simulated environment. Since the defender does not know the poisoning ratio ρ i and target label of the attacker's poisoned dataset (involved in the attack objective F ′ ), we approximate the attacker's reward function as r t A = -F ′′ ( ̂ w t +1 g ) , F ′′ ( w ) := min c ∈ C [ 1 M 1 ∑ M 1 i =1 1 | D ′ i | ∑ | D ′ i | j =1 ℓ ( w, (ˆ x j i , c ))] -1 M 2 ∑ M i = M 1 +1 f ( ω, D i ) . F ′′ differs F ′ only in the first M 1 clients, where we use a strong target label (the minimizer) as a surrogate to the true label c ∗ .

Inverting Gradient/Reverse Engineering. In invert gradient, we set the step size for inverting gradients η ′ = 0 . 05 , the total variation parameter β = 0 . 02 , optimizer as Adam, the number of iterations for inverting gradients max _ iter = 10 , 000 , and learn the data distribution from scratch. The number of steps for distribution learning is set to τ E = 100 . 32 images are reconstructed (i.e., B ′ = 32 ) and denoised in each FL epoch. If no attacker is selected in the current epoch, the aggregate gradient estimated from previous model updates is reused for reconstructing data. To build the denoising autoencoder, a Gaussian noise sampled from 0 . 3 N (0 , 1) is added to each dimension of images in D reconstructed , which are then clipped to the range of [0,1] in each dimension. The result is shown in Figure 7.

In the process of reverse engineering, we use Neural Cleanse [61] to find hidden triggers (See 792 Figure 8) connected to backdoor attacks. This method is essential for uncovering hidden triggers 793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

0

30

10

15

15

15

20

20

25

23

25

15

20

20

25

S

20

20

20

sZ

IMP MNIST

RL MNIST

IMP CIFAR10

RL CIFAR10

0.8

0.8

0.6

0.6

0.6

0.6

Krum

Krum

0.4

0.4

meta-RL

meta-SG

0.4

meta-RL

meta-SG

0.2

100

200

FL rounds

meta-SG+

meta-SG

Krum

300

400

500

0.2

0.0

100

FL rounds

300

400

500

200

400

FL rounds

meta-SG

Krum

meta-SG+

600

008

1000

0.2

200

400

FLrounds

600

BOO

1000

and for preventing such attacks. In particular, we use the global model, root generated data and inverted data as inputs to reverse backdoor triggers. The Neural Cleanse class from ART is used for this purpose. The reverse engineering process in this context involves using the generated backdoor method from the Neural Cleanse defense to find the trigger pattern that the model is sensitive to. The returned pattern and mask can be visualized to understand the nature of the backdoor.

Online Adaptation and Execution. During the online adaptation stage, the defender starts by using the meta-policy learned from the pre-training stage to interact with the true FL environment, while collecting new samples { s, a, ˜ r, s ′ } . Here, the estimated reward ˜ r is calculated using the self-generated data and simulated triggers from the pertaining stage, as well as new data inferred online through methods such as inverting gradient [19] and reverse engineering [61]. Inferred data samples are blurred using data augmentation [53] while protecting clients' privacy. For a fixed number of FL rounds (e.g., 50 for MNIST and 100 for CIFAR-10 in our experiments), the defense policy will be updated using gradient ascents from the collected trajectories. Ideally, the defender's adaptation time (including the time for collecting new samples and that for updating the policy) should be significantly less than the whole FL training period so that the defense execution will not be delayed. In real-world FL training, the server typically waits for up to 10 minutes before receiving responses from the clients [8, 25], enabling defense policy's online update with enough episodes.

D Additional Experiment Results

More untargetd model poisoning/backdoor results. As shown in Figure 9, similar to results in Figure 2 as described in Section 4, meta-SG plus achieves the best performance (slightly better than meta-SG) under IPM attacks for both MNIST and CIFAR-10. On the other hand, meta-SG performs the best (significantly better than meta-RL) against RL-based attacks for both MNIST and CIFAR-10. Notably, Krum can be easily compromised by RL-based attacks by a large margin. In contrast, meta-RL gradually adapts to adaptive attacks, while meta-SG displays near-immunity against RL-based attacks. In addition, we illustrate results under backdoor attacks and defenses on MNIST in Table 4.

Defender's knowledge of backdoor attacks. We consider two settings: 1) the server knows the backdoor trigger but is uncertain about the target label, and 2) the server knows the target label but not the backdoor trigger. In the former case, the meta-SG first pre-trains the defense policy with RL attacks using a known fixed global pattern (see Figure 6) targeting all 10 classes in CIFAR-10, then adapts with an RL-based backdoor attack using the same trigger targeting class 0 (airplane), with

Neuron-Clippingvs.whiteboxmeta-SG

Pruningvs.whiteboxmeta-SG

Uncertaintargetlabelblackboxbackdoor

Unknowntriggerpaternblackboxbackdoor

1.00

1.00

0.8

1.00

0.75

0.75

0.6

9.75

0.50

05'0

0.4

0.50

0.25

0.00

200

400

FL rounds

600

NeroClip_main

NeroClip_backdoer

whitebex_main

whitebox_backdoor

800

1000

0.25

0.00

200

400

FL rounds

600

Purm_main

Pum_backdoor

whitebox_main

whitebox_backdoor

800

1000

200

FL rounds

400

009

blackbex_backdoor

CipMed_main

Jotpoped"pawdto

blackbox_main

800

0.25

0.00

200

FL rounds

400

600

FLIrust+NC_main

blackbex_main

blackbox_backdoor

FLTrust+NC_backdoor

800

results shown in the third figure of Figure 10. In the latter case where the defender does not know the 825 true backdoor trigger used by the attacker, we implement the GAN-based model [12] to generate the 826 worst-case triggers (see Figure 4) targeting one known label (truck). The meta-SG will train a defense 827 policy with the RL-based backdoor attacks using the worst-case triggers targeting the known label, 828 then adapt with a RL-based backdoor attack using a fixed global pattern (see Figure 6) targeting the 829 known label in the real FL environment (results shown in the fourth graph in Figure 10. We call the 830 two above cases blackbox settings since the defender misses key backdoor information and solely 831 depends on their own generated data/triggers w/o inverting/reversing during online adaptation. In 832 the whitebox setting, the server knows the backdoor trigger pattern (global) and the targeted label 833 (truck), and is trained by true clients' data. The corresponding results are in the first two graphs of 834 Figures 10, which show the upper bound performance of meta-SG and may not be practical in a real 835 FL environment. Post-training defenses alone (i.e., NeuroClip and Prun) and combined defenses 836 (i.e., ClipMed and FLTrust+NC) are susceptible to RL-based attacks once the defense mechanism 837 is known. On the other hand, as depicted in Figure 10, we demonstrate that our whitebox meta-SG 838 approach is capable of effectively eliminating the backdoor influence while preserving high main 839 task accuracy simultaneously, while blackbox meta-SG against uncertain labels is unstable since 840 the meta-policy will occasionally target a wrong label, even with adaptation and blackbox meta-SG 841 against unknown trigger is not robust enough as its backdoor accuracy still reaches nearly 50% at the 842 end of FL training. 843

Importance of inverting/reversing methods. In the ablation study, we examine a practical and 844 relatively well-performed graybox meta-SG. The graybox meta-SG has the same setting as blackbox 845 meta-SG during pre-training as describe in Section 2.2, but utilizes inverting gradient [19] and reverse 846 engineering [61] during online adaptation to learn clients' data and backdoor trigger in a way without 847 breaking the privacy condition in FL. The graybox approach only learns ambiguous data from clients, 848 then applies data augmentation (e.g., noise, distortion) and combines them with previously generated 849 data before using. Figure 11(a) illustrates that graybox meta-SG exhibits a more stable and robust 850 mitigation of the backdoor attack compared to blackbox meta-SG. Furthermore, in Figure 11(b), 851

852

853

854

855

856

857

858

859

860

861

862

863

864

865

866

867

868

869

870

871

872

873

874

875

876

877

878

879

880

881

882

883

884

885

886

887

888

889

890

891

892

(a)

(b)

(c)

(d)

FL under different number of attackers

FL under different Non-iid level

8'0

0.6

Uncertaintargetgrayboxbackdoor

0.6

Unknowntriggergrayboxbackdoor

0.8

0.7

9/0

Krum

ClipMed

I meta-SG

69

0.6

Krum

ClipMed

I meta-5G

0.5

0.4

0.4

0.4

0.2

blackbox_main

graybox_backdoor

blackbox_backdoor

graybox_main

0.2

0.0

graybox_main

graybox_backdoor

blackbox_main

blackbox_backdoor

0.3

0.2

0.1

0.3

0.2

0.1

200

400

FL rounds

600

800

1000

200

400

FL rounds

600

800

1000

D

20%attacker

30%attacker

40%attacker

q=0.5

q=0.6

q=0.7

graybox meta-SG demonstrates a significant reduction in the impact of the backdoor attack, achieving nearly a 70% mitigation, outperforming blackbox meta-SG.

Number of malicious clients/Non-i.i.d. level. Here we apply our meta-RL to study the impact of inaccurate knowledge of the number of malicious clients and the noni.i.d. level of clients' local data distribution. With rough knowledge that the number of malicious clients is in the range of 5% -50% , the meta-SG will pre-train on LMP attacks with malicious clients [5 : 5 : 50] , and adapt to three cases with 20% , 30% , and 40% malicious clients in online adaptation, respectively. Similarly, when the non-i.i.d. level is between 0 . 1 -1 , the meta-SG will pre-train on LMP attacks with non-i.i.d. level [0 . 1 : 0 . 1 : 1] and adapt to q = 0 . 5 , 0 . 6 , 0 . 7 in online adaptation. As illustrated in Figures 11(c) and 11(d), meta-SG reaches the highest model accuracy for all numbers of malicious clients and noni.i.d. levels under LMP.

Importance of pre-training and online adaptation As shown in Table 5, the pre-training is to derive defense policy rather than the model itself. Directly using those shifted data (root or generated) to train the FL model will result in model accuracy as low as 0 . 2 -0 . 3 ( 0 . 4 -0 . 5 ) for CIFAR-10 (MNIST) in our setting. Pre-training and online adaptation are indispensable in the proposed framework. Our experiments in Table 5 indicate that directly applying defense learned from pre-training w/o online adaptation and adaptation from randomly initialized defense policy w/o pre-training both fail to address malicious attacks, resulting in global model accuracy as low as 0 . 3 -0 . 6 ( 0 . 1 -0 . 4 ) on MNIST (CIFAR-10). In the absence of adaptation, meta policy itself falls short of the distribution shift between the simulated and the real environment. Likewise, the online adaptation fails to attain the desired defense policy without the pre-trained policy serving as a decent initialization.

Biased/Limited root data We evaluate the average model accuracy after 250 FL epochs under the meta-SG framework against the IPM attack, using root data with varying i.i.d. levels (as defined in the experiment setting section). Here, q = 0.1 (indicating the root data is i.i.d.) serves as our baseline meta-SG, as presented in the paper. We designate class 0 as the reference class. For instance, when q = 0.4, it indicates a 40% probability for each data labeled as class 0 within the root data, while the remaining 60% are distributed equally among the other classes. We observe that when q is as high as 0.7, there is one class (i.e., 3) missing in the root data. Although, through inverting methods in online adaptation, the defender can learn the missing data in the end, it suffered the slower adaptation compared with a good initial defense policy. In addition, we test the average model accuracy after 250 FL epochs under meta-SG against IPM attack using different numbers of root data (i.e., 100, 60, 20), where 100 root data is our original meta-SG setting in the rest of paper. We overserve that when number of root data is 20, two classes of data are missing (i.e., 1 and 5).

Generalization to unseen adaptive attacks We thoroughly search related works considering adaptive attacks in FL and find very limited works (with solid and lightweight open-source implementation) that can be used as our benchmark. As a result, we introduce two new benchmark adaptive attack methods in the testing stage as unseen adaptive attacks: (1) adaptive LMP![15], which requires access to normal clients' updates in each FL round, and (2) RL attack [31] restricted 1-dimensional action space (i.e., adaptive scalar factor) compared with the baseline 3-dimensional RL attack [31] showing in our paper. The defender in pre-training only interacts with the 3-dimensional RL attack. We test the average model accuracy after 250 FL epochs under meta-SG against different (unseen)

895

896

897

898

899

900

901

902

903

904

905

906

(b) Ablation study of limited root data.

adaptive attacks. What is interesting here is that meta-SG can achieve even better performance against 893 unseen attacks. 894

E Algorithms

This section elaborates on meta-learning defense and meta-Stackelberg defense in equation meta-SE. To begin with, we first review the policy gradient method [58] in RL and its Monte-Carlo estimation. To simplify our exposition, we fix the attacker's policy ϕ , and then the Markov game reduces to a single-agent MDP, where the optimal policy to be learned is the defender's θ .

Policy Gradient The idea of the policy gradient method is to apply gradient ascent to the value function J D . Following [58], we obtain ∇ θ J D := E τ ∼ q ( θ ) [ g ( τ ; θ )] , where g ( τ ; θ ) = ∑ H t =1 ∇ θ log π ( a t D | s t ; θ ) R ( τ ) and R ( τ ) = ∑ H t =1 γ t r ( s t , a t D ) . Note that for simplicity, we suppress the parameter ϕ, ξ in the trajectory distribution q , and instead view it as a function of θ . In numerical implementations, the policy gradient ∇ θ J D is replaced by its Monte-Carlo (MC) estimation using sample trajectory. Suppose a batch of trajectories { τ i } N b i =1 , and N b denotes the batch size, then the MC estimation is

The same deduction also holds for the attacker's problem when fixing the defense θ . 907

Meta-Learning FL Defense As discussed in Section 3, meta-learning-based defense (meta defense) 908 mainly targets non-adaptive attack methods, where π A ( · ; ϕ, ξ ) is a pre-fixed attack strategy following 909 some rulebook, such as IPM [68] and LMP [15]. In this case, the BSMG reduces to single-agent MDP 910 for the defender, where the transition kernel is determined by the attack method. Mathematically, the 911 meta-defense problem is given by 912

Since the attack type is hidden from the defender, the adaptation mapping Ψ is usually defined in a data-driven manner. For example, Ψ( θ, τ ) can be defined as a one-step stochastic gradient update with learning rate η : Ψ( θ, τ ) = θ + η ˆ ∇ J D ( τ ξ ) [16] or a recurrent neural network in [13]. This work mainly focuses on gradient adaptation for the purpose of deriving theoretical guarantees in Appendix F.

With the one-step gradient adaptation, the meta-defense problem in equation E2 can be simplified as

Recall that the attacker's strategy is pre-determined, ϕ, ξ can be viewed as fixed parameters, and hence, the distribution q is a function of θ . To apply the policy gradient method to equation E3, one needs an unbiased estimation of the gradient of the objective function in equation E3. Consider the gradient computation using the chain rule:

913

914

915

916

917

918

919

920

921

922

The first term results from differentiating the integrand J D ( θ + η ˆ ∇ θ J D ( τ ) , ϕ, ξ ) (the expectation is 923 taken as integration), while the second term is due to the differentiation of q ( θ ) . One can see from 924 the first term that the above gradient involves a Hessian ˆ ∇ 2 J D , and its sample estimate is given by 925 the following. For more details on this Hessian estimation, we refer the reader to [14]. 926

Finally, to complete the sample estimate of ∇ θ E τ ∼ q ( θ ) [ J D ( θ + η ˆ ∇ θ J D ( τ ) , ϕ, ξ )] , one still needs to estimate ∇ θ J D ( θ + η ˆ ∇ θ J D ( τ ) , ϕ, ξ ) in the first term. To this end, we need to first collect a batch of sample trajectories τ ′ using the adapted policy θ ′ = θ + η ˆ ∇ θ J D ( τ ) . Then, the policy gradient estimate of ˆ ∇ θ J D ( θ ′ ) proceeds as in equation E1. To sum up, constructing an unbiased estimate of equation E4 takes two rounds of sampling. The first round is under the meta policy θ , which is used to estimate the Hessian equation E5 and to adapt the policy to θ ′ . The second round aims to estimate the policy gradient ∇ θ J D ( θ + η ˆ ∇ θ J D ( τ ) , ϕ, ξ ) in the first term in equation E4.

In the experiment, we employ a first-order meta-learning algorithm called Reptile [43] to avoid the Hessian computation. The gist is to simply ignore the chain rule and update the policy using the gradient ∇ θ J D ( θ ′ , ϕ, ξ ) | θ ′ = θ + η ˆ ∇ θ J D ( τ ) . Naturally, without the Hessian term, the gradient in this update is biased, yet it still points to the ascent direction as argued in [43], leading to effective meta policy. The advantage of Reptile is more evident in multi-step gradient adaptation. Consider a l -step gradient adaptation, the chain rule computation inevitably involves multiple Hessian terms (each gradient step brings a Hessian term) as shown in [14]. In contrast, Reptile only requires first-order information, and the meta-learning algorithm ( l -step adaptation) is given by Algorithm 2.

Meta-Stackelberg Learning Recall that in meta-SE, the attacker's policy ϕ ∗ ξ is not pre-fixed. Instead, it is the best response to the defender's adapted policy as shown in equation meta-SE. To

927

928

929

930

931

932

933

934

935

936

937

938

939

940

941

942

943

944

945

946

947

948

Algorithm 2 Reptile Meta-Reinforcement Learning with l -step adaptation

obtain this best response, one needs alternative training: fixing the defense policy, and applying gradient ascent to the attacker's problem until convergence. It should be noted that the proposed meta-SL utilizes the unbiased gradient estimation in equation E5, which paves the way for theoretical analysis in Appendix F. Yet, we turn to the Reptile to speed up pre-straining in the experiments. We present both algorithms in Algorithm 3, and only consider one-step adaptation for simplicity. The multi-step version is a straightforward extension of Algorithm 3.

Algorithm 3 (Reptile) Meta-Stackelberg Learning with one-step adaptation

949

950

951

952

953

954

955

956

957

958

959

960

961

962

963

964

965

966

967

968

969

970

971

972

973

974

975

976

977

978

F Theoretical Results

F.1 Existence of Meta-SG

Theorem F.1. Under the conditions that Θ and Φ are compact and convex, the meta-SG admits at least one meta-FOSE.

Proof. Clearly, Θ × Φ | Ξ | is compact and convex, let ϕ ∈ Φ | Ξ | , ϕ ξ ∈ Φ be the (type-aggregated) attacker's strategy, since the consider twice continuously differentiable utility functions ℓ D ( θ, ϕ ) := E ξ ∼ Q L D ( θ, ϕ ξ , ξ ) and ℓ ξ ( θ, ϕ ) := L A ( θ, ϕ ξ , ξ ) for all ξ ∈ Ξ . Then, there exists a constant γ c > 0 , such that the auxiliary utility functions:

are γ c -strongly concave in spaces θ ∈ Θ , ϕ ξ ∈ Φ for all ξ ∈ Ξ , respectively for any fixed ( θ ′ , ϕ ′ ) ∈ Θ × Φ | Ξ | .

Define the self-map h : Θ × Φ | Ξ | → Θ × Φ | Ξ | with h ( θ ′ , ϕ ′ ) ≡ ( ¯ θ ( θ ′ , ϕ ′ ) , ¯ ϕ ( θ ′ , ϕ ′ )) , where

Due to compactness, h is well-defined. By strong concavity of ˜ ℓ D ( · ; ( θ ′ , ϕ ′ )) and ˜ ℓ ξ ( · ; ( θ ′ , ϕ ′ )) , it follows that ¯ θ, ¯ ϕ are continuous self-mapping from Θ × Φ | Ξ | to itself. By Brouwer's fixed point theorem, there exists at least one ( θ ∗ , ϕ ∗ ) ∈ Θ × Φ | Ξ | such that h ( θ ∗ , ϕ ∗ ) = ( θ ∗ , ϕ ∗ ) . Then, one can verify that ( θ ∗ , ϕ ∗ ) is a meta-FOSE of the meta-SG with utility function ℓ D and ℓ ξ , ξ ∈ Ξ , in view of the following inequality

therefore, the equilibrium conditions for meta-SG with utility functions ˜ ℓ D and { ˜ ℓ ξ } ξ ∈ Ξ are the same as with utility functions ℓ D and { ℓ ξ } ξ ∈ Ξ , hence the claim follows.

F.2 Proofs: Non-Asymptotic Analysis

In the sequel, we make the following smoothness assumptions for every attack type ξ ∈ Ξ . In addition, we assume, for analytical simplicity, that all types of attackers are unconstrained, i.e., Φ is the Euclidean space with proper finite dimension.

Assumption F.2 (( ξ -wise) Lipschitz smoothness) . The functions L D and L A are continuously diffrentiable in both θ and ϕ . Furthermore, there exists constants L 11 , L 12 , L 21 , and L 22 such that for all θ, θ 1 , θ 2 ∈ Θ and ϕ, ϕ 1 , ϕ 2 ∈ Φ , we have, for any ξ ∈ Ξ ,

We also make the following strict-competitiveness assumption. This notion can be treated as a generalization of zero-sum games: if one joint action ( a D , a A ) leads to payoff increases for one player, it must decrease the other's payoff.

Assumption F.3 (Strict-Competitiveness) . The BSMG is strictly competitive, i.e., there exist constants c < 0 , d such that ∀ ξ ∈ Ξ , s ∈ S , a D , a A ∈ A D × A ξ , r D ( s, a D , a A ) = cr A ( s, a D , a A ) + d .

979

980

981

982

983

984

985

986

In adversarial FL, the untargeted attack naturally makes the game zero-sum (hence, SC). The purpose of introducing Assumption F.3 is to establish the Danskin-type result [3] for the Stackelberg game with nonconvex value functions (see Lemma F.5), which spares us from the Hessian inversion.

Lemma F.4 (Implicit Function Theorem (IFT) for Meta-SG) . Suppose for ( ¯ θ, ¯ ϕ ) ∈ Θ × Φ | Ξ | , ξ ∈ Ξ we have ∇ ϕ L A ( ¯ θ, ¯ ϕ, ξ ) = 0 the Hessian ∇ 2 ϕ L A ( ¯ θ, ¯ ϕ, ξ ) is non-singular. Then, there exists a neighborhood B ε ( ¯ θ ) , ε > 0 centered around ¯ θ and a C 1 -function ϕ ( · ) : B ε ( ¯ θ ) → Φ | Ξ | such that near ( ¯ θ, ¯ ϕ ) the solution set { ( θ, ϕ ) ∈ Θ × Φ | Ξ | : ∇ ϕ L A ( θ, ϕ, ξ ) = 0 } is a C 1 -manifold locally near ( ¯ θ, ¯ ϕ ) . The gradient ∇ θ ϕ ( θ ) is given by -( ∇ 2 ϕ L A ( θ, ϕ, ξ )) -1 ∇ 2 ϕθ L A ( θ, ϕ, ξ ) .

Lemma F.5. Under assumptions F.2, 3.2, there exists { ϕ ξ : ϕ ξ ∈ arg max ϕ L A ( θ, ϕ, ξ ) } ξ ∈ Ξ , such that

Moreover, the function V ( θ ) is L -Lipschitz-smooth, where L = L 11 + L 12 L 21 µ 987

Proof of Lemma F.5. First, we show that for any θ 1 , θ 2 ∈ Θ , ξ ∈ Ξ , and ϕ 1 ∈ arg max ϕ L A ( θ 1 , ϕ, ξ ) , there exists ϕ 2 ∈ arg max ϕ L A ( θ 2 , ϕ, ξ ) such that ∥ ϕ 1 -ϕ 2 ∥ ≤ L 12 µ ∥ θ 1 -θ 2 ∥ . Indeed, based on smoothness assumption equation F11 and equation F10,

988

989

990

Since ϕ 2 ∈ arg max ϕ L A ( θ 2 , ϕ, ξ ) , ∇ ϕ L A ( θ 2 , ϕ 2 , ξ ) = 0 . Apply PL condition to ∇ ϕ L A ( θ, ϕ 2 , ξ ) , 991

Since PL condition implies quadratic growth, we also have 992

Combining the two inequalities above we obtain the Lipschitz stability for ϕ ∗ ξ ( · ) , i.e.,

Second, show that ∇ θ V ( θ ) can be directly evaluated at { ϕ ∗ ξ } ξ ∈ Ξ . Inspired by Danskin's theorem, we 993 first made the following argument, consider the definition of directional derivative. Let ℓ ( θ, ϕ ) := 994 ∇ θ E ξ,τ J D ( θ + η ˆ ∇ J D ( τ ) , ξ ) . For a constant τ and an arbitrary direction d , 995

Hence, a sufficient condition for the first equation is ∇ ϕ ℓ ( θ + τd, ϕ ∗ ( θ )) = 0 , meaning that ℓ D ( θ, ϕ ) 996 and L A ( θ, ϕ, ξ ) share the first-order stationarity at every ϕ when fixing θ . Indeed, by Lemma F.4, we 997 have, the gradient is locally determined by 998

Given a trajectory τ := ( s 1 , a t D , a t A , . . . , a H D , a H A , s H +1 ) , let R D ( τ, ξ ) := ∑ H t =1 γ t -1 r D ( s t , a t , ξ ) 999 and R D ( τ, ξ ) := ∑ H t =1 γ t -1 r D ( s t , a t , ξ ) . Denote by µ ( τ ; θ, ϕ ) the trajectory distribution, that the 1000 log probability of µ is given by 1001

According to the policy gradient theorem, we have 1002

By SC Assumption F.3, when ∇ ϕ L A ( θ, ϕ, ξ ) = 0 , there exists c < 0 , d , such that ∇ ϕ L D ( θ, ϕ, ξ ) = 1003 E µ [ cR A ( τ, ξ ) ∑ H t =1 ∇ ϕ log( π A ( a t A | s t ; ϕ ))] + E µ [ ∑ H t =1 γ t -1 d ∑ H t =1 ∇ ϕ log( π A ( a t A | s t ; ϕ ))] = 0 . 1004 Hence ∇ θ V = E ξ ∼ Q [ ∇ θ L D ( θ, ϕ ξ , ξ )] . 1005

Third, V ( θ ) is also Lipschitz smooth. As we notice that, ℓ D is Lipschitz smooth since E ξ ∼ Q is a 1006 linear operator, we have, 1007

which implies the Lipschitz constant L = L 11 + L 12 L 21 µ . 1008

1009

1010

1011

1012

It is impossible to present the convergence theory without the assistance of some standard assumptions in batch reinforcement learning, of which the justification can be found in [14]. We also require some additional information about the parameter space and function structure. These assumptions are all stated in Assumption F.6.

1013

1014

1015

1016

Assumption F.6.

(a) The policy gradients are bounded, ∥∇ θ L D ( θ, ϕ, ξ ) ∥ ≤ G 2 , ∥∇ ϕ L A ( θ, ϕ, ξ ) ∥ ≤ G 2 for all θ, ϕ ∈ Θ × Φ and ξ ∈ Ξ .

(b) The policy gradient estimations are unbiased, i.e.,

(c) The variances for the stochastic gradients are bounded, i.e., for all θ t , ϕ t ξ , ξ , 1017

(d) The parameter space Θ has diameter D Θ := sup θ 1 ,θ 2 ∈ Θ ∥ θ 1 -θ 2 ∥ ; the initialization θ 0 1018 admits at most D V function gap, i.e., D V := max θ ∈ Θ V ( θ ) -V ( θ 0 ) . 1019

(e) It holds that the parameters satisfy 0 < µ < -cL 22 . 1020

Equipped with Assumption F.6 we are able to unfold our main result Theorem 3.3, before which 1021 we show in Lemma F.7 that ϕ ∗ ξ can be efficiently approximated by the inner loop in the sense that 1022 ∇ θ E ξ ∼ Q L D ( θ t , ϕ t ξ ( N A ) , ξ ) ≈ ∇ θ V ( θ t ) , where ϕ t ξ ( N A ) is the last iterate output of the attacker 1023 policy. 1024

1025

1026

1027

Lemma F.7. Under Assumption F.6, 3.2, F.3, and F.2, let ρ := 1 + µ cL 22 ∈ (0 , 1) , ¯ L = max { L 11 , L 12 , L 22 , L 21 , V ∞ } where V ∞ := max { max ∥∇ V ( θ ) ∥ , 1 } . For all ε > 0 , if the attacker learning iteration N A and batch size N b are large enough such that

then, for z t := ∇ θ E ξ ∼ Q L D ( θ t , ϕ t ξ ( N A ) , ξ ) -∇ θ V ( θ t ) , 1028

Proof of Lemma F.7. Fixing a ξ ∈ Ξ , due to Lipschitz smoothness, 1030

The inner loop updating rule ensures that when κ A = 1 L 21 , ϕ t ξ ( N ) -ϕ t ξ ( N -1) = 1031 1 ˆ ∇ ϕ J A ( θ t , ϕ t ( N -1) , ξ ) . Plugging it into the inequality, we arrive at 1032

Therefore, we let ( F t n ) 0 ≤ n ≤ N be the filtration generated by σ ( { ϕ t ξ ( τ ) } ξ ∈ Ξ | τ ≤ n ) and take condi1033 tional expectations on F t n : 1034

By variance-bias decomposition, and Assumption F.6 (b) and (c), 1035

Applying the PL condition (Assumption 3.2), and Assumption F.6 (a) we obtain 1036

and 1029

rearranging the terms yields 1037

where we use the fact that -max ϕ ℓ D ( θ t , ϕ ) ≤ -V ( θ t ) . Telescoping the inequalities from τ = 0 to 1038 τ = N , we arrive at 1039

PL-condition implies quadratic growth, we also know that V ( θ t ) -ℓ D ( θ t , ϕ t ( N )) ≤ 1040 E ξ 1 2 µ ∥∇ ϕ L D ( θ t , ϕ t ξ ( N ) , ξ ) ∥ 2 ≤ 1 2 µ G 2 , by Assumption F.3, 1041

Hence, with Jensen inequality and choice of N A and N b , 1042

Now we adjust the size of N A and N b to make E [ ∥ z t ∥ ] small enough, to this end, we set 1043

which further indicates that 1044

In the setting above, it is not hard to verify that 1045

Also note that ∥∇ ϕ L A ( θ t , ϕ t ξ ( N A ) , ξ ) ∥ = ∥∇ ϕ L A ( θ t , ϕ t ξ ( N A ) , ξ ) -∇ ϕ L A ( θ t , ϕ ∗ ξ , ξ ) ∥ , given the 1046 proper choice of N A and N b , one has 1047

which indicates the ξ -wise inner loop stability. 1048

Now we are ready to provide the convergence guarantee of the first-order outer loop. 1049

Theorem F.8. Under Assumption F.6, Assumption F.3, and Assumption F.2, let the stepsizes be, 1050 κ A = 1 L 22 , κ D = 1 L , if N D , N A , and N b are large enough, 1051

1052

1053

1054

then there exists t ∈ N such that ( θ t , { ϕ t ξ ( N A ) } ξ ∈ Ξ ) is ε -meta-FOSE.

Proof. According to the update rule of the outer loop, (here we omit the projection analysis for simplicity)

one has, due to unbiasedness assumption, let ( F t ) 0 ≤ t ≤ N D be the filtration generated by σ ( θ t | k ≤ t ) 1055

which leads to 1056

Since V ( · ) is L -Lipschitz smooth, 1057

Fixing a θ ∈ Θ , let e t := ⟨∇ θ ℓ D ( θ t , ϕ t ( N A )) , θ -θ t ⟩ , we have 1058

By the choice of N b , we have, since V ∞ = max { max θ ∥∇ V ( θ ) ∥ , 1 } , 1059

Thus, the relation equation F13 can be reduced to 1060

Telescoping equation F12 yields 1061

Thus, setting N D ≥ 4 D V (2 V ∞ + LD Θ ) 2 Lε 2 , and then by Lemma F.7, we obtain that, 1062

which implies there exists t ∈ { 0 , . . . , N D -1 } such that E [ e 2 ] ≤ ε 2 . 1063

t 1064

1065

1066

1067

1068

1069

1070

1071

1072

1073

1074

1075

1076

1077

1078

1079

1080

1081

1082

1083

1084

1085

1086

1087

1088

1089

1090

1091

1092

1093

1094

1095

1096

1097

F.3 Generalization to Unseen Attacks

In the online adaptation phase, the pre-trained meta-defense may be exposed to attacks unseen in the pre-training phase, which poses an out-of-distribution (OOD) generalization issue to the proposed meta-SG framework. Yet, Proposition F.9 and Proposition F.13 assert that meta-SG is generalizable to the unseen attacks, given that the unseen is not distant from those seen. The formal statement is deferred to Appendix F, and the proof mainly targets those unseen non-adaptive attacks for simplicity.

Proposition F.9 (OOD Generalization Informal Statement) . Consider sampled attack types ξ 1 , . . . , ξ m during the pre-training and the unseen attack type ξ m +1 in the online stage. The generalization error is upper-bounded by the 'discrepancy' between the unseen and the seen attacks C ( ξ m +1 , { ξ i } i m =1 ) .

Our main goal is to quantify the value discrepancy under an attack type that is out of empirical distribution. We consider attack types ξ 1 , . . . , ξ m to be empirically sampled from distribution Q ( · ) during the pre-training stage, and an unseen attack type ξ m +1 in the online stage. The quantification of distance C ( ξ m +1 , { ξ i } i m =1 ) relies on the total variation,

Definition F.10 (total variation) . For two distributions P and Q , defined over the sample space Ω and σ -field F , the total variation between P and Q is ∥ P -Q ∥ TV := sup U ∈F | P ( U ) -Q ( U ) | .

The celebrated result shows the following characterization of total variation,

Let the fixed attack policies ϕ i , i = 1 , . . . , m +1 corresponding to each attack type. To formalize the generalization error, for each θ ∈ Θ , we define populational values

where q θ i ( · ) : ( S × A × S ) H -1 × S → [0 , 1] is the trajectory distribution determined by state dependent policies π D ( ·| s ; θ ) , π A ( ·| s ; ϕ i , ξ i ) and transition kernel T . Since q θ i is factorizable, we have Lemma F.11 to eliminate ∥ q θ i -q θ m +1 ∥ TV dependence on θ by upper bounding it using another pair of mariginal distributions.

Lemma F.11. For any θ ∈ Θ , there exist marginals d i , d m +1 : ( S × A A × S ) H -1 × S → [0 , 1] total variation ∥ q θ i -q θ m +1 ∥ TV can be bounded by ∥ d i -d m +1 ∥ TV .

Proof. By factorization, for a trajectory τ , any θ ∈ Θ , and any type index i = 1 , . . . , m +1 :

thus, by the inequality of product measure,

where d i and d m +1 are the residue factors after removing π A ( ·| s t ; θ ) .

Assumption F.12. For any ξ ∈ Ξ and ϕ ξ , the function J D ( θ, ϕ ξ , ξ ) is G -Lipschitz continuous w.r.t. θ ∈ Θ ;

Proposition F.13. Under assumption 3.2 and certain regularity conditions, fixing a policy θ ∈ Θ , we have, there exist some marginal distribution of

where the constant C depending on the total variation between d m +1 and { d i } i m =1 :

here, G is the Lipschitz parameter of J D w.r.t. both θ . 1098

Proof. We start with the decomposition of the generalization error, for an arbitrary attack type ξ i , 1099 i = 1 , . . . , m , fixing a policy θ ∈ Θ determines jointly with each ϕ i the trajectory distribution q θ i . 1100 Denoting the one-step adaptation policy θ ′ ( τ ) = θ -η ∇ J D ( τ ) as a function of trajectory τ , we have 1101 the following decomposition, 1102

We assume ( τ m +1 , τ i ) is drawn from a joint distribution which has marginals q θ m +1 and q θ i and is 1103 corresponding to the maximal coupling of these two. Then, 1104

̸

if τ m +1 disagrees with τ i , for ( ii ) , we have, since J θ D is Lipschitz with respect to θ , 1105

as a result, denoting the maximal coupling of q θ m +1 and q θ i as gives, 1106

where the last inequality is due to Lemma F.11. Averaging the m empirical ξ i 's yeilds the result: 1107

Since the trajectory distribution is a product measure, the difference between q θ i and q θ m +1 only lies 1108 by attacker's type, ∥ q θ ′ ( τ m +1 ) m +1 -q θ ′ ( τ m +1 ) i ∥ TV = ∥ q θ m +1 -q θ i ∥ TV ≤ ∥ d m +1 -d i ∥ TV . 1109

Now we bound ( i ) , for ease of exposition we let q ′′ = q θ ′ ( τ m +1 ) m +1 and q ′ i := q θ ′ ( τ m +1 ) i . By the finiteness 1110 of total trajectory reward R ( τ ) for any trajectory τ , R ( τ ) ≤ 1 -γ H 1 -γ , hence, 1111

1112

1113

1114

1115

1116

1117

1118

1119

1120

1121

1122

1123

1124

1125

1126

1127

1128

1129

1130

1131

1132

1133

1134

1135

G A Game-theoretic Perspective on Meta Equilibrium

This section offers further justification for the meta-equilibrium in (meta-SE), and we argue that metaequilibrium provides a data-driven approach to address incomplete information in dynamic games. Note that information asymmetry is prevalent in the adversarial machine learning context, where the attacker enjoys an information advantage (e.g., the attacker's type). The proposed meta-equilibrium notion can shed light on these related problems beyond the adversarial FL context.

We begin with the insufficiency of Bayesian Stackelberg equilibrium defined as the solution to the bilevel optimization in equation BSE in handling information asymmetry, a customary solution concept in security studies [35].

One can see from equation BSE that such an equilibrium is of ex-ante type: the defender's strategy is determined before the game starts. It targets a 'representative' attacker (an average of all types). As the game unfolds, new information regarding the attacker's private type is revealed (e.g., through the global model updates). However, this ex-ante strategy does not enable the defender to adjust its strategy as the game proceeds. Using game theory language, the defender fails to handle the emerging information in the interim stage.

To create interim adaptability in this dynamic game of incomplete information, one can consider introducing the belief system to capture the defender's learning process on the hidden type. Let I t be the defender's observations up to time t , i.e., I t := ( s k , a k D ) t k =1 s t +1 . Denote by B the belief generation operator b t +1 ( ξ ) = B [ I t ] . With the Bayesian equilibrium framework, the belief generation can be defined recursively as below

Since b t is the defender's belief on the hidden type at time t , its belief-dependent Markovian strategy is defined as π D ( s t , b t ) . Therefore, the interim equilibrium, also called Perfect Bayesian Equilibrium (PBE) [17] is given by a tuple ( π ∗ D , π ∗ A , { b t } H t =1 ) satisfying

k

=1

In contrast with (BSE), this perfect Bayesian equilibrium notion (PBE) enables the defender to make 1136 good use of the information revealed by the attacker, and subsequently adjust its actions according to 1137 the revealed information through the belief generation. From a game-theoretic viewpoint, both (PBE) 1138 and (meta-SE) create strategic online adaptation: the defender can infer and adapt to the attacker's 1139 private type through the revealed information since different types aim at different objectives, hence, 1140 leading to different actions. Compared with PBE, the proposed meta-equilibrium notion is better 1141 suited for large-scale complex systems where players' decision variables can be high-dimensional 1142 and continuous, as argued in the ensuing paragraph. 1143

To achieve the strategic adaptation, PBE relies on the Bayesian-posterior belief updates, which soon 1144 become intractable as the denominator in equation G1 involves integration over high-dimensional 1145 space and discretization inevitably leads to the curse of dimensionality. Despite the limited practicality, 1146 PBE is inherently difficult to solve, even in finite-dimensional cases. It is shown in [6] that the 1147 equilibrium computation in games with incomplete information is NP-hard, and how to solve for 1148 PBE in dynamic games remains an open problem. Even though there have been encouraging attempts 1149 at solving PBE in two-stage games [36], it is still challenging to address PBE computation in generic 1150 Markov games. 1151

1152

1153

1154

1155

1156

1157

1158

1159

1160

1161

1162

1163

1164

1165

1166

1167

1168

1169

1170

1171

1172

1173

1174

1175

1176

1177

1178

1179

1180

1181

1182

1183

1184

1185

1186

1187

1188

1189

1190

1191

1192

1193

1194

1195

1196

1197

1198

1199

1200

1201

1202

1203

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We propose a novel meta Stackelberg game to address adaptive and mixed poisoning attacks in federated learning.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to conclusion section and Appendix B.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257

Justification: Please refer to Appendix F.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please refer to Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

1258

1259

1260

1261

1262

1263

1264

1265

1266

1267

1268

1269

1270

1271

1272

1273

1274

1275

1276

1277

1278

1279

1280

1281

1282

1283

1284

1285

1286

1287

1288

1289

1290

1291

1292

1293

1294

1295

1296

1297

1298

1299

1300

1301

1302

1303

1304

1305

1306

1307

1308

1309

Answer: [Yes]

Justification: The datasets (MNIST and CIFAR-10) are open source, and we will publish the codes during the final revision stage.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please check Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The error bars are added to Figure 11 (c) and (d), while random seeds are fixed for other figures/tables.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

1310

1311

1312

1313

1314

1315

1316

1317

1318

1319

1320

1321

1322

1323

1324

1325

1326

1327

1328

1329

1330

1331

1332

1333

1334

1335

1336

1337

1338

1339

1340

1341

1342

1343

1344

1345

1346

1347

1348

1349

1350

1351

1352

1353

1354

1355

1356

1357

1358

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please see Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We stick to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Please see Appendix B.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

1359

1360

1361

1362

1363

1364

1365

1366

1367

1368

1369

1370

1371

1372

1373

1374

1375

1376

1377

1378

1379

1380

1381

1382

1383

1384

1385

1386

1387

1388

1389

1390

1391

1392

1393

1394

1395

1396

1397

1398

1399

1400

1401

1402

1403

1404

1405

1406

1407

1408

1409

1410

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no safeguards risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We credited all assets (e.g., code, data, models) used in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

1411

1412

1413

1414

1415

1416

1417

1418

1419

1420

1421

1422

1423

1424

1425

1426

1427

1428

1429

1430

1431

1432

1433

1434

1435

1436

1437

1438

1439

1440

1441

1442

1443

1444

1445

1446

1447

1448

1449

1450

1451

1452

1453

1454

1455

1456

1457

1458

1459

1460

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We didn't release new assets at this stage.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper titled \"Meta Stackelberg Game: Robust Federated Learning against Adaptive and Mixed Poisoning Attacks\" proposes a new approach to enhancing the security of Federated Learning (FL) systems. The paper identifies that existing FL defenses are inadequate against adaptive and mixed attacks. To address this, they introduce a Meta Stackelberg Game (meta-SG) framework, which employs a Bayesian Stackelberg Markov game (BSMG) and a meta-learning approach. This framework aims to provide a robust and adaptive defense mechanism against various poisoning attacks, including model poisoning and backdoor attacks. The proposed method is theoretically proven to converge to an equilibrium efficiently and is empirically validated to perform well against strong adversarial attacks.\n\nSTRENGTHS:\n+ Introducing the Meta Stackelberg Game (meta-SG) framework is innovative, offering a new perspective on defending against adaptive and mixed attacks in FL. \n\n+ The paper provides a solid theoretical foundation, proving that the proposed algorithm converges to a first-order ε-equilibrium, proving the method's efficiency. \n\n+ Extensive experiments demonstrate the effectiveness of the meta-SG framework, showing significant improvements in defense against various attack types compared to existing methods. \n\n+ The meta-learning component allows the defense mechanism to adapt dynamically to different attack scenarios, enhancing its robustness in uncertain environments.\n\nWEAKNESSES:\n- The proposed approach seems computationally intensive, requiring significant resources for pre-training and adaptation, which might limit its practicality in real-world applications. Although it proves that it can converge in Theorem 3.3, it would be beneficial to have empirical evidence, such as the method's run-time overhead. \n\n- While the framework is tested against several attack types, the scope of attacks considered might not cover all possible real-world adversarial strategies, limiting the generalizability of the results. The paper especially makes it unclear what attacks are used in pre-training, whether they are the same, and how different they are compared to the real FL environment when testing and generating results. \n\n- The proposed method's scalability to larger and more diverse FL environments remains unclear, especially given its computational demands.\n\nQUESTIONS:\nGiven the above points, I have some questions that need to be addressed: \n\n1. How does the meta-SG framework scale with increasing clients and more complex models? Have you tested its performance in larger FL setups? \n\n2. How does the framework handle adaptive attack types not seen during the pre-training phase? Are there any limits to the adaptive attacks it can defend against? \n\n3. Have you considered applying the meta-SG framework to domains other than image classification, such as natural language processing or time-series analysis in FL?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

comments: SUMMARY:
The paper presents a game theoretic model for robust federated learning. The technique is composed of pre-training and online adaptation. During pre-training, a meta-policy for the defender is solved as a Bayesian Stackelberg Markov game. The defense policy is further polished during the online adaptation stage.

STRENGTHS:
The Stackelberg game is designed to counter unknown/uncertain attacks by an adaptive adversary. 

Theoretical bounds for sample complexity are provided. 

Empirical results demonstrate the effectiveness of the proposed technique.

WEAKNESSES:
The main weakness is the slight violation of privacy as the technique needs a portion of ground truth data from the clients. This has been disused as the limitations in the paper. 

Minor comments:

On page 2, "including mixed attacks ," ---> extra space

QUESTIONS:
What is the statistical significance of the results shown in Table 1? Without the standard error, it is unclear whether the proposed technique is indeed superior to other existing techniques. I understand there is no room in the table, at least it should be mentioned in the text.

LIMITATIONS:
Privacy violation is mentioned as a limitation of the technique. It's unclear whether future development can remove the dependence on the client-side ground truth.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a defense mechanism in federated learning that has adaptability inspired from meta learning. The authors formulates a Bayesian Stackelberg Markov game (BSMG), focusing on addressing poisoning attacks of unknown or uncertain types. The authors propose an equilbrium inspired by meta learning and then look at a local version of that. Empirical evaluations are done on MNIST and CIFAR.\n\nSTRENGTHS:\nThis paper seems to have interesting results.\n\nWEAKNESSES:\nThe writing is sloppy in many places and quite a few things are unexplored.\nFederated learning's primary motivation is privacy, so the privacy loss from a core small dataset must be analyzed. The authors do acknowledge the loss but IMO that is not enough.\n\nBehind all the motivation of federated learning, the core idea is in the equilibrium proposed - IMO, exploring this equilibrium in more detail would make the paper stronger (in fact, the problem makes sense even in simpler adversarial learning problems).\n\nIs Def 3.1 just a differential equilibrium, in the style of \"Implicit Learning Dynamics in Stackelberg Games: Equilibria Characterization, Convergence Analysis, and Empirical Study\" but missing second order conditions?\nWhy are there no second order conditions? Just first order may not induce a local equilibrium, which is a meaningful equilibrium to achieve. This is where even more meaning needs to be discovered for the first meta-SE. The authors compare it to PBE in the appendix, but the notations of belief consistency and sequential rationality is what makes PBE (SE) convincing. Without any such (or similar) notion, a new equilibrium in a dynamic setting is not principled.\n\nI do not understand why the ball $B(\\theta^*)$ is used with a bound of 1? What is special about 1? (same for the other ball)\nProposition written informally (and no explanation) in the main paper does not make sense (e.g., Prop 3.4).\n\nThere are many typos when I started to look in appendix:\n1) Eq F6, $\\tilde{l}$ should have two inputs\n2) Line 959, the parameters of $\\tilde{l}$ is lost, without this the equations with $\\theta'$ on left and no $\\theta'$ on right is not well-defined.\n3) I do not understand how the equation in line 964 came about - first it is said that it is an inequality but what is written is an equality.\n\nOverall, typos do not inspire confidence.\nAlso, any defense mechanism should itself be subject to attack with the adversary possessing knowledge of defense mechanism - I do not see that in experiments.\n\nQUESTIONS:\nPlease respond to review\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of backdoor/poisoning attacks in federated learning (FL). In this setting, a single attacker has control over all malicious clients trying to employ different attack types (on each controlled client). This paper aims to create a defense mechanism against such adaptive attackers. To this end, the paper proposes a game-theoretic approach, which contains two stages: (1) Pre-training stage: before the FL environment, the defender first learns a good defense policy by simulating that environment using a small amount of truthful data against a simulated attacker with known possible actions (e.g. attack types used), and (2) Online-adaptation stage: the defender leverages the pre-trained policy and adapt it against the attacker at the real FL environment. This paper demonstrates the effectiveness of their proposed mechanism, as well as considers ablation studies where the previous assumptions do not meet: adaptation to attack methods at real FL environment are different (but similar) from ones seen at the pre-training stage.\n\nSTRENGTHS:\nThe paper is well-written and easy to follow.\n\nWEAKNESSES:\nI have some concerns, mostly about the practicality of the assumptions:\n\n1. Assumption on the accessibility of data: the paper assumes that in the pre-training phase, the defender has access to a small amount of data, which is used to model the data distribution of clients using generative models. (1) It goes against privacy. (2) It is not clear that a small amount of data is representative enough to model client data distribution. (3) Things will be even more complicated if each client has its sub-population (data) that is (reasonably) different from each other. In this case, which malicious clients the attacker has control over will matter. \n\n2. Assumption of the similarity of attack types at the online adaptation stage: the paper assumes that the attack types in a real FL environment, though unseen, should be similar to those of the pre-training phase. This seems impractical, especially in a white-box setting, where the attacker will try to leverage the property of the defense mechanism to create an adaptive attack that is specified for that defense mechanism (see Carlini's works).\n\nI also have concerns about the experiments:\n1. Datasets and models used: Would it be possible to use more practical datasets and models instead of MNIST/CIFAR10 and ResNet-18? I would like to see results when the data distribution is complex enough that generative models cannot easily learn with a small amount of data. Right now, the amount of data used is still considerable, considering the dataset used. This would make the assumption of the accessibility to a small amount of data in the pre-training phase more persuasive.\n\nSome comments on writing/paper organizations.\n1. In Table 1, please highlight which results are the best. It would be more readable and easier for comparison. \n2. In Figure 2, it might be better to show smoothed curves.\n3. In Appendix F, before each theorem/lemma/assumption, it would be better if an intuition/proof sketch for each one is provided. Also, if the proof technique/assumption is standard, please mention the corresponding references.\n4. In the Conclusion, I think it would be more honest if explicitly stated that the major limitation of this paper lies in the practicability of the assumption. Right now, I only see privacy concerns mentioned, which is misleading. \n\n(Not really a weakness) It could be nice if source code is included (might be using an anonymous repo). \n\nOverall, I think this is a good paper if ignoring the practicability of the assumptions on the attack types/data (on the accessibility in the pre-training phase/distribution of accessed data in the pre-training phase/distribution of data in each client). I also did not find an experiment where the attacker leverages the information about defense mechanisms to instantiate a better attack scheme (white box attack). It is known that many proposed defense mechanisms failed in this scenario, though it seems obvious that it will go against the similarity assumption on the attack types at the online adaptation phase. However, I am also aware of the hardness of defense tasks in adversarial machine learning and it is good to have some initial results even under strong assumptions. I will try my best to be reasonable. Maths were not checked carefully, I will try to go into the details in the rebuttal phase.\n\nQUESTIONS:\nSee Weaknesses above.\n\nLIMITATIONS:\nSee Weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the vulnerabilities of Federated Learning (FL) systems to various adversarial attacks, including model poisoning and backdoor attacks. The proposed solution is a Meta Stackelberg Game (meta-SG) framework designed to offer robust and adaptive defenses against these attacks. The approach formulates adversarial federated learning as a Bayesian Stackelberg Markov game (BSMG) and employs a meta-learning method to find optimal defense strategies. Theoretical analyses show the algorithm's efficiency in convergence, and empirical results demonstrate its effectiveness against different attack types.\n\nSTRENGTHS:\n1. The proposed framework considers both untargeted and targeted attacks.\n\n2. The paper uses RL to realize online adaptation which is close to the real world.\n\n3. Inspired by meta-learning, the method is robust to unknown attacks.\n\n4. In the pre-training phase, the paper uses generated data to decrease the concern of privacy leakage.\n\n5. The paper provides sufficient experimental results.\n\nWEAKNESSES:\n1. Could you explain more about the necessity of adding the gradient adaptation (from BSE to meta-SE)? Although BSE is ex-ante when knowing the distribution $Q$, the model $\\theta$ is changed during training and it could capture emerging information. Could you provide empirical results comparing BSE and meta-SE to show the advantage of meta-SE?\n\n2. Considering adaptive/mixed attacks, the paper misses two relevant frameworks: MixTailor [1] and RobustTailor [2]. They can adjust aggregation methods during training. Especially, RobustTailor also simulates a game between the defender and the attacker, and it proposes a mixed attack. This kind of method could be included in experiments as a baseline.\n\n3. Because the whole method is complicated with 2 stages, comparing computational cost with other baselines is necessary.\n\n\n\n\n[1] Ramezani-Kebrya, Ali, Iman Tabrizian, Fartash Faghri, and Petar Popovski. \"Mixtailor: Mixed gradient aggregation for robust learning against tailored attacks.\" arXiv preprint arXiv:2207.07941, 2022.\n\n[2] Xie, Wanyun, Pethick, Thomas, Ramezani-Kebrya, Ali, and Cevher, Volkan. \"Mixed Nash for Robust Federated Learning.\" Transactions on Machine Learning Research. 2023.\n\nQUESTIONS:\n1. Are both white-box and black-box settings used in the pre-training stage? Adding the proposed methods explicitly in Figure 1 might be more clear.\n\nLIMITATIONS:\nThe authors mentioned limitations in Section 5. The main one is the privacy concern.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-19"}