Mixed Nash for Robust Federated Learning
Authors: Wanyun Xie
Thomas Pethick
Ali Ramezani-Kebrya
Volkan Cevher
Venue: Accepted by TMLR ()

Abstract: We study robust federated learning (FL) within a game theoretic framework to alleviate the server vulnerabilities to even an informed adversary who can tailor training-time attacks. Specifically, we introduce RobustTailor, a simulation-based framework that prevents the adversary from being omniscient and derives its convergence guarantees. RobustTailor improves robustness to training-time attacks significantly while preserving almost the same privacy guarantees as standard robust aggregation schemes in FL. Empirical results under challenging attacks show that RobustTailor performs close to an upper bound with perfect knowledge of honest clients.

Published in Transactions on Machine Learning Research (02/2024)
Mixed Nash for Robust Federated Learning
Wanyun Xic
www.un.zie@epfl.cc
Laboratory for Information and Inference Systems (LIONS), EPPL
Thomas Pethick
Laboratory for Information and Inference Systems (LIONS), EPFL
Ali Ramezani-Kebryy
ali@uuio.no
Deportment of Informatics, University of Oslo and Visual Intelligence Centre Integreat, Norwegian Centre for Knowledge-driven Machine Learning
Volkan Cevher
volkon walkon.ceoher@epf.cco
Laboratory for Information and Inference Systems (LIONS), EPFL
Reviewed on OpenReview: http:s//ppennevvvv...dd/rdo/doer
Abstract
We study robust federated learning (FL) within a game theoretic framework to alleviate the server vulnerabilities to even an informed adversary who can tailor training-ting time at- tacks (Fang et al., 2020; Xie et al., 2020a; Oxfatura et al., 2022; Rodriguez-Barross et al., 2023). Specifically, introduce Robust a simulation- based framework that prevents the adversary from being omniscient and derives its convergence guarantees. Robust Tailor improves robustness to training-time attacks significantly with a minor trade-off of privacy. Empirical results under challenging attacks show that Robust performs close to an upper bound with perfect knowledge of honest clients.
1 Introduction
Federated learning (FL) distributes model training across multiple client devices for improved learning performance with privacy (Konečný et al., 2016; McMahan et al., 2017; Kairouz ct al., 2021). However, this distributed nature introduces vulnerabilities, including the risk of atlacks from adversarial clients, like Byzantine clients who can send malicious messages instead of correct gradients (Kairouz et al., 2021). Ensuring robustness against such adversaries critical.
Traditionally, Byzilnene-resllience in FL relies median- based aggregation methods like Krum (Blanchard et al., 2017), Coordinate-wise Median (Comed) (Yin et al., 2018), and Trimmed Mean (TM) (Yin et al., 2018), known for their resilience against random attacks. Nevertheless, these methods may not withstand tailored attacks from powerful adversarics who can exploit knowledge of the aggregation rule (Fang et al., 2020; Xie et al., 2020a; Oxfatura et al., 2022; Rodríguez- universal immunity to all attacks significant challenge due to the adversary's 's information leverage.
al., 2023; Baruch et al., 2019). Achieving
To partially address this challenge, formulate robust distributed learning problem against training-tinm time attacks as game between server and adversary. To improve robustness against the adversary and prevent the adversary from being omniscient, we use mized strategy, where the server's action set includes a number of robust aggregation rules and where the adversary's action set. features a set of atlack algorithms. We then propose
a scheme based on simulating aggregation rules under different attacks. To diminish the attacker's information leverage, we assume that the server has access to rough estimate of true gradient update from public dataset with small amount of data. We theoretically and empirically address scenarios where the computed gradient using public dataset only rough estimate of the actual true gradient.

---

Published in Transactions on Machine Learning Research (02/2024)
Formulating the game between the server and adversary is challenging the informed adversary has advanlage over the server and can perfectly estimate the empirical mean of all honest updates without an attack, i.e., the ideal update under no attack; while the server does not know the set of honest clients and does not have such prior knowledge of the ideal update. To control the information gap between the server and adversary, we propose to simulate the hypothetical minimax problem through generating simulated gradients, constructing surrogate loss for the server, and formulating a MixedNash problem that optimizes the probability distributions over attack algorithms and aggregation rules. Solving the MixedNash problem efficiently is another major challenge. Leveraging on the noisy nature of stochastic gradients, we consider bandit feedback model with limited feedback for observe the loss through exploration.
in which the server and adversary can only
Our theory guarantees convergence of RobustTailor if the number of simulation rounds and the number of honest clients are sufficiently large. Our empirical results demonstrate that Robust provides high resilience to training-timm -ime attacks while maintaining stable performance even under a challenging new mixod attack stralegy. We emphasize that Kobust/Tailo is expandable, i.e., any Byzantine-resiliet scheme can be added in server's aggregation pool and similarly any attack can be considered. Finally, we also provide extensive experiments to show the performance of Robust Tailor under unknown attacks out of the server's 's expectation, poisoned data mixed in the public dataset, subsampling in FL, dynamic strategy of adversary, and the adversary with partial knowledge.
Summary of Contributions
We frame robust distributed learning problem as a game between a server and an adversary. We propose Robust a framework simulating two players using a bandil feedback model to improve robustness of FL by choosing a suitable aggregator from existing aggregation rules for each update.
Theoretically, we establish convergence guarantees for Robust Our theory implies that RobustTailor converges if the server can roughly estimate the honest update and the number of simulation rounds and honest clients is sufficienlly large.
Empirically, we show that Robust Tailor outperforms known aggregation rules and remains robust even under realistic scenarios including mixed attack strategy, unknown attacks out of the server's expectation, poisoned data mixed the public dataset, subsampling FL, dynamic strategy of adversary, and the adversary with partial knowledge.
2
Related Work
In this section, we provide summary of related work. See Appendix A for a more complete treatment.
Training-time attacks in FL. FL suffers from training-time attacks (Biggio et al., 2012; Bhagoji et al., 2019; Demontis et al., 2019), which potentially participate in every training round and even are adaptive according to leveraging on the knowledge of the structure of aggregation rule. In model poisoning, a class of training-timm time attacks, an adversary controls some clients and directly manipulates their outputs aiming to bias the global model towards the opposite direction of honest training (Kairouz et al., 2021). If Byzantine clients have access to the updates of honest clients, they can tailor their attacks and make them difficult to detect (Fang et al., 2020; Xie et al., 2020a).
Robust aggregation and Byzantine resilience. To improve robustness under general Byzantine clients, number of robust aggregation schemes have been proposed, which are mainly inspired by robust statistics such as median-based aggregators (Yin et al., 2018; Chen et al., 2017), Krum (Blanchard et al., 2017), trimmed mcan (TM) (Yin et al., 2018). Morcover, Cao & Lai (2019); Fang et al. (2020); Cao et al. (2021); Xie et al. (2020a) propose server-side verification methods using auxiliary data. ByzantineSDD (Alistarh et al., 2018) and centered clipping (CC) (Karimireddy et al., 2021) are history-aided aggregators. Although these Byzantine-resllient aggregators defend successfully against some attacks, Fang et al. (2020); Xie et al. (2020a); Gouissem al. (2022) argue that those aggregation rules are vulnerable and fail easily when an informed adversary tailors careful attack.
2

---

Published in Transactions on Machine Learning Research (02/2024)
Indeed, Krum and Comed are vulnerable to reverse attacks (Fang et al., 2020); TM would be ruined by Alittle attack (Baruch et al., 2019); CC would fail under relocated orthogonal perturbation (ROP) attack (Ozfatura et al., 2022). Setrriuuzz-Barrose et al. (2023) show that it may not be feasible to design one aggregation rule that is universally effective against any attack. Alternatively a pool of cxisting robust aggregators can be leveraged to establish robustness to a broad range of atlacks while no individual aggregator able to be robust to all of them. Ramezani-Kebrya et al. (2022) propose a framework based on uniform randomization of multiple aggregation rules. Such framework increases complexity of designing an effective attack but cannot promise suitable aggregator at each training iteration. This paper proposes framework to select a proper aggregation rule proactively during training.
Public datasct in FL. Kairouz et al. (2021, Scction 3.1.1) suggest that a small global shared datasct can be used in FL to improve robustness. This dataset may originate from a publicly available proxy data source, separate dataset from the clients' data that is not privacy sensitive, perhaps distillation of the raw data following (Wang et al., 2018). Such public dataset enjoys wide acceptance in FL (Kairouz et al., 2021; Fang & Ye, 2022; Huang et al., 2022; Yoshida et al., 2020). For instance, Sageflow (Park et al., 2021), Zeno (Xie et al., 2019b), and Zeno+++ (Xie et al., 2020b) utilize such public data at the server to combat adversarial threats.
The availability of public data at the server also enables collaborative model training with formal differential or hybrid differential privacy guarantees (Gu et al., 2023; Kairouz al., 2021; Avent et al., 2017). Avent et al. (2017) introduce hybrid differential privacy where some opt-in users voluntarily donate data. Many companies (e.g., Mozilla and Google) rely group of testers with higher levels of mutual trust who voluntarily opt-in to less privacy-preservinn model than that of an average end-user.
3 Problem Setting
We consider a general distributed system consisting of a parameter server and n clients (Abadi et al., 2016a). Under synchronous setting in FL, clients compute their updates on their own local data and then aggregate from all peers to update model parameters, and the goal is to solve:
min F(x) xER*
F())
(FL)
where F; I2 R denotes the training error (empirical risk) of model x on the local data of client i. For example, a popular training algorithm is slochastic gradient descent. At iteration I, the server sends the global model x, to the clients. Each client computes the gradient (x.] V (x;) on its local training dataset and sends V (xx) to the server. The server updates the global model by averaging X++1 X, - -
Di (.(())).
3.1 Threat Model and Game Construction
In practical scenarios, clients are vulnerable where powerful adversaries such as informed adversary can control some clients and send malicious updates to the server. Suppose that I Byzantine clients are controlled by an adversary and behave arbitrarily. At iteration t, honest clients compute and send honest stochastic gradients (x,)) (x,) for i E In f] while Byzantine clients, controlled by an adversary, output attacks by E R' for j E f n). The server receives all updates and aggregates them following a particular robust aggregation rule, which outputs an updated model X++1 E Ied. Finally, the server broadcasts X++1 to all clients. Note that our threat model is common in FL following existing literature (Ramezani-Kebryee et al., 2022; Fang et al., 2020; Cao et al., 2021).
We frame this distributed learning problem under training-time attacks as a game played by the adversary and the server. The adversary aims corrupting training while the server aims at learning an effective model, which achieves a satisfactory overall empirical risk over honest clients in Fg. (FT.). The informed adversary and training-time attacks are described in Section 3.1.1. The server's aggregators are described in
Notation can be found prior the appendix.
3

---

Published in Transactions on Machine Learning Research (02/2024)
Section 3.1.2. To the best of our knowledge, work is the first framework that selects suitable aggregators actively by framing robust learning problem under training-time tailored attacks as game.
3.1.1 Informed adversary with attacks
Adversary's goal: Following the training-time time attacks literature (Fang et al., 2020; Xie et al., 2020a; Biggio et al., 2012; Bhagoji et al., 2019; Demontis et al., 2019), the adversary's ' objective is to manipulate the global model at the server and minimize test accuracy e.g., in the classification task.
Assumption 1 (Informed adversary). 1) An informed adversary controls f out of n clients where these colluding Byzantine clients aim at disturbing the entire training process by sending training-time time attacks (Biggio el al., 2012; Bhagoji et al., 2019; Demontis el al., 2019); 2) The namber of Byzuntine clients is bounded 2 S n (Blanchard al., 2017; Alistarh al., 2018; Ramezani Ramezani-Kebrga et al., 2022; Karimåreddy et al., 2022); otherwise the adversary will be able to provably control the optimization trajectory and set the global model arbibrarity (Lamport et al., 1982); 3) The informed adversary has full knowledge of the outputs of n f honest clients and controls the oulpuls of those compromised clienls, e.g., their gradients across the course of training-
Due to having access to the gradients of honest nodes, the adversary can compute the global aggregated gradient of an omniscient aggregation rule, which is the empirical mean of all honest updates:
g'
gi.
(1)
Definition 1 (Attack algorithm). Let (g1,..., denote the set of honest updates computed by n - honest clients. The adversary designs f Byzantine updates using an AT algorithm:
(bn-ft1..... .b.....,,b. := .g.............................. g A:
(2)
where A denotes the set of aggregators formally defined in Section 3.1.2.
Adversary's capability: When the adversary knows a particular server's robust aggregation rule, it is able to design tailored attacks using n f honest gradients (Fang et al., 2020). We suppose that the adversary has a set of S computationally tractable algorithms to design tailored attacks:
F IT,,, AT
(3)
It is shown that several efficient and tailored attacks can be designed that provably fail SOTA robust aggregators, e.g., Krum, Comed, TM, and CC (Fang al., 2020; Xie al., 2020a; Ozfatura et al., 2022; Barrrso et al., 2023; Baruch et al., 2019).
3.1.2 Server with aggregators
The server aims at learning an elfective model, which achieves a satisfactory overall empirical risk over honest clients comparable to that under no attack. To update the global model, the server aggregates all gradients sent by clients at each iteration.
Server's goal:
Definition 2 (Aggregation rule). Let g E IRd denote an update received from client i, which can be either an honest or compromised client for i [n]. That means {g/) {(g)// fbi) The server aggregales all updales from n clients and outpuls global updule g E Rdd using an aggregation rule AG:
g g= AG ..G(g).....g.....
Sioig.........F...... F..
(4)
where FF denotes the set of attacks defined in Section 3.1.1.
Assumption (Server). The server knows the number of compromised clients f or an upper bound on f. which is a common assumption in robust learning (Blanchard al., 2017; Alistarh et al., 2018; Ramezani Rabezani-eebyaa Kebrya et al., 2022; Karimireddy et al., 2022; Rajput et al., 2019). However, does not know the specific Byzantine clients among n clients in this distriluted system such that the server cannot compute g" in Eq. (1) directly-
4

---

Published in Transactions on Machine Learning Research (02/2024)
To learn and establish some level of robustness against Prainingtimm attacks, several Byzantino-resilenn aggregation rules have been proposed, e-g., Krum (Blanchard et al., 2017) and Comed (Yin et al., 2018). These median-based schemes have been shown to be vulnerable to tailored attacks (Fang et al., 2020; Xie et al., 2020a; Karimireddy et al., 2021).
Server's capability: The server aggregates all updates from clients by aggregation rule to update the global model. The set of M aggregators used by the server is denoted by
A = [AG,, AG2,... AGM)
(5)
Assumption 3 (Sets of aggregators and attacks). We assume that the pool of aggregators A and the set of allacks F are known by the server and the adversary. However, the specific AT" and AG chosen al iteration t are unknown for each other. To avoid trivial solutions, we ossume each aggregation rule is robust (Jormal definition of robustness is provided in Appendir B) against a subset of atlack algorithms in F while no aggregation rule is immune to all attack algorithms.
3.2 Problem Formulation
To evaluate the performance of an updated global model, i.e, the output of AG in Eq. (4), we define a loss function, which measures the discrepancy of the output of AG and an omniscient model update obtaining all honest grudients.
Definition 3 (Loss function). The loss function of using aggregation rule AG under attack AT is defined as [IAG(()))]///aa) - AT F),A) ggll
ATAG,AT, ) -IAGG(E)) - g''l
(6)
where g* is the ideal model under no allack which is computed in Eq. (1).
To train the global model, the server takes multiple rounds of stochastic gradient descent by aggregating the stochastic gradients from clients. However, some gradients might be corrupt at each round, which are sent by compromised clients controlled by the adversary. We frame this robust distributed learning scenario as a game between the adversary and the server. The server aims to minimize the loss defined in Definition 3, while the adversary aims to maximize it. This game as a minimar problem is formulated as:
min max AT, AGEAATER
(MinMax)
Ideally, the game in MinMax reaches a Nash equilibrium (NE) (Nash, 1950). The hypothetical process of model aggregation shown in Appendix C. However, the server cannot compute the loss since it cannot distinguish honest gradients and does not know g". We propose to simulate the game in the Section
4 Robust Aggregation
Because MinMax cannot be solved during the process of updating the model, we propose to simulate it instead and obtain an optimized aggregator for model updates. As mentioned in Section 3, the informed adversary has an advantage over the server since it can perfectly estimate g" in Eq. (1), while the server does not have such knowledge and cannot identify honest clients priori. Without any estimate of the true update, the server cannot do better than selecting one of the aggregation rules uniformly at random TRamezani-Kebrgg et al., 2022). This random strategy may output a highly suboptimal aggregation rule and does not leverage the knowledge of F. We assume that the server has access to a rough estimate of true gradient update which could control the information gap between the server and adversary. Note that we discuss estimated gradients specifically Section 4.1. Let g computed by the server denote the rough estimate of g'
Remark 1 (Rough estimate g*). Rough estimate of g* for the server can be obtained from 1) a public dataset and 2) gradients of trusted clients. To guarantee convergence, we only require that the update from the public dataset is a rough estimate of the ideal g" Through theoretical analysis (Section 5) and experiments (Section 6), address scenarios where the compuled gradient is only rough estimate of the actual true.
5

---

Published in Transactions on Machine Learning Research (02/2024)
gradient e.g., by considering distribution shifts and mixture with poisoned data. Note that we do not directly update the model using g becouse 1) we aim to avoid wasting the compulation of all clients, and 2) the global model should reflect the distribution of all honest clients not just a small subset.
For the simulation, the server generates the simulated gradients {g based on the public dataset or gradients of trusted clients. The server's surrogate loss function in the simulated game is given by
7(AG, AT, 3==) |AG ATT Vi A). -
(7)
where g" 1 E g.
Let L E R denote the surrogate loss of M aggregators corrosponding to S attacks, and L(AG,, AT,) represents the loss associated with aggregation rule i in A under atlack j in F in the simulation. After the adversary has committed to a probability distribution q over S attack algorithms,, the server chooses a probability distribution p over M aggregation rules. Then, the server incurs the loss E(p.q) p LLq.. We solve MixedNash below instead of MinMax.
min max p Lq PEAM qEas
(MixedNash)
where Am and As denote the probability simplex in [M] and [S], respectively.
In practice, it is computationally expensive to compute L 1 x Additionally, stochastic gradients are noisy. Therefore, we consider the bondil feedback model, in which the server and adversary never observe È in its entirety, but instead only the loss associated with particular realization of play. To solve MixedNash in the bandit feedback model, one player could implement the well-known Exponential-weeigtt Algorithm for Exploration and Exploitation (Exp3) (Seldin et al., 2013) whose detailed description is deferred to Appendix D.
We propose an algorithm in which two players (the server and the adversary) in our model simultaneously execute Exp3. We term our proposed robust aggregation scheme as Robust RobustTailor, outputting an optimized AG at each iteration. The specific steps from the server's perspective are shown in Algorithm 1. Using the public dataset, the server generates n I noisy stochastic gradients gi for i [n - f]. The server also assigns equal initial weights w" for i E [M], (J) for j E [$] to all aggregators and attacks independently. In each round, it chooses an aggregator and an attack based on probability distributions determined by the current weights. Then, it observes the loss and update the weights based on the received regret/ regret/reward rrwaard associated with the cho- sen aggregator /attack, adjusting more for poorly performing actions and less for well-performing ones. After K rounds of simulation on { Mo-/, , the server obtains final probability distribution p and selects an aggregation rule by sampling from P. The steps for our robust training procedure are summarized in Algorithm 2.
The adversary can also perform simulation to optimize its attack at each iteration. The main differences for an adversarial simulation compared to RobustTailon include: 1) the adversary can use perfect honest stochastic gradicnts {g: M instcad of noisy estimates; 2) the probability output is q which is calculated by the weight vector of atlacks v(j) for j E [S]. The details of the adversarial simulation are provided Appendix E.
4.1 Privacy
Given the importance of privacy in FL, this is a deliberate trade-off between enhancing robustness and preserving privacy. We propose two scenarios to obtain estimated gradients g in a privacy-preserving manner. In the first scenario under less privacy concerns, the server has a public dataset consisting of small amount of data donated by clients or a trusted party. The existence of such public dataset is a valid and common assumption in FL (Kairouz et al., 2021).
In another scenario with strict privacy requirements, the server trusts only a small subset of honest clients providing reliable updates. It is common for the server to have a core group of trustworthy clients, like companies relying on highly trusted testers. Notably, RobustTailor is compatible with all privacy- preserving techniques, such as differential privacy (DP) (Bassily et al., 2014; Wei et al., 2020). Further details are provided in Appendix I. In both scenarios, Robust does not significantly compromise privacy and is indeed acceptable given its benefits as shown in Section 6.4.

---

Published in Transactions on Machine Learning Research (02/2024)
Algorithm 1 Taiust
Updating rates λ1, 22, d and λ2 simulation rounds K, simulated gradients (B.) A, F. Initialize weight vector (i) for i [M] and () 1 for j [S].
Input:
for k = to K do
Set (AG,) = (1 -11-11 E i= (()
21)
(+)
+ A1 M for i E [M].
Set (AT,)) = (1 - 12)
+ d2 for j E [S].
Sample AG ~ ~p" and AT* q" respectively.
Estimate the loss & E(AG* AT* (=.))-))
For e [M], set (i) 4AG.=AG*), ch
w wi+ = (1) exp( (x)x(((x((((()))))
(AGA)))
For j E (S), set (5) = 14AT,=AT* eh,
xh+(( = exx())
(*(T)))
Set p:- Σ p (AG.)
for E [M].
K Sample AG ~ p.
Output: AG.
Algorithm Server's aggregation
Learning rale Th, n clients, f compromised clients, ileration rounds T, A and F Initialize model Xo. for t= to T do
Input:
Send x, to all clients.
Receive gradients from all clients [g/]=-
Calculate simulated gradients {g: 1-
Call Algorithm to aggregate AG PRobustTailor/ (g/ A,F).
Update the global model by X = X; - (y (g)))
4.2 Computational complexity
Appendix K demonstrates both theoretical analysis and empirical results of Robust/aiioo'' computation complexity. Our fine-graines analysis shows that the overall time complexity of Robust is given by O((M + SYTK) where the average complexity per round T depends on the aggregators in server's pool.
5 Theoretical Guarantees
To provide guarantees of the outer loop in Algorithm 2, we first show convergence of the inner optimixation described in Algorithm 1. Any two simultaneously played no-regret algorithms for a minimax problem can be turned into convergence to a NE (see Lemma in Appendix F). Combined with no-regret propertics of Exp3, we obtain guarantees for the aggregation rule returned from Algorithm Considering the information asymmetry involved Algorithm 1, I. is replaced by the simulation loss shown Eq. (7).
Lemma 1 (Bounded loss of simulated game). Let i be the simulation loss in Eq. (7). Sample AG ~ p as defined in Algorithm with λ₁ log and d2 v RM
log
RS. Then the loss is bounded in expectation for any
attack AT F:
EAG (.G,ATT,&E11)/ s sp' T LL + 2 VM M log M + VS log
(8)
VK
where (p".q") AM Ag Nash equilibrium of the zero-sum game with the payoff matria L as defined in MiredNash.
7

---

Published in Transactions on Machine Learning Research (02/2024)
Lemma 1 implics that the simulated loss approaches the NE value even under the worst-case attack. Note that Lemma 1 gives the guarantees for inner loop only considering simulated gradients. Now we extend it to outer loop whose ideal update is g".
We assume the standard assumption of bounded estimation error through decomposing the error into bias and variance terms.
Assumption (Bounded estimation error). Let E [n - f. The simulated gradient g has a bounded estimation error Billg: - g=]]]]] Best + Vest where the bias and voriance bounds are given by:
g 11 Best BII E S Vest.
Theorem 1 (Onter loop's convergence). Under Assumption 4, suppose {n in Algorithm 2 satisfies E. th - 00 and E, th < co. Let x, denole the oulput of AG in Algorithm with bounded norm C' for t 2 1. For noncovvez loss function F in Eq. (FL), which is three times differentiable with continuous derivatives, bounded from below, and satisfies (Bottou, 1998, Assumption io in Section 5.1), we have V a.s. if
CVest hn - + Elp" LLa]+ + CBest. + 2.VM log M + log
E g
(9)
VK
where the expectation is over simulated gradients.
The condition in Theorem is satisfied when the number of simulation rounds and the number of honest clients n - f are sufficiently large. The optimal strategy p' depends the realization of the gradients, Elp= Lqq] is small as long there exists an effective aggregation rule for each realization. Furthermore, Best is small when the public dataset represents the clients' data distribution by some extent. If Best is too large, it may be the case that the worst AG E A is selected due to the mismatch with the true updates.
To the best of our knowledge, this the first convergence guarantee on nonconvex, heterogeneous, and simulated setting where the adversary and the server play an asymmetrical game to optimize their strategies.
The proofs of Lemma 1 and Theorem are provided in Appendices F to H.
6 Experimental Evaluation
In this section, evaluate the resilience of PobustTailon against tailored attacks. To provide both intuitive results showing robustness of RobustTailor and its realistic implementation in FL, we first construct a basic setting and then extend it to various scenarios. In the basic setting, we simulate training with total of 12 clients, 2 of which are compromised by an informed adversary. We train a CNN model on MNIST (Lecun et al., 1998) under independent and identically distributed (iid) sctting. For the server, we construct a simple pool of aggregalors including only Krum (Blanchard et al., 2017) and Comed (Yin et al., 2018). For the adversary, we consider two tailored attacks which can successfully ruin Krum and Comed respectively and and two stronger mixed attacks. Referring to (Fang et al., 2020; Xie et al., 2020a), c-reverse attack with small e corrupts Krum while large corrupts Comed. We choose = 0.5 and 100 as the basic set of attacks. Note that e-reverse attack submits the scaled honest update with parameter -6 to the server. To enable simulation, we assume honest clients donate 5% of their local training data to the server as public dataset claborated in Remark 1. The informed adversary has access to the gradients of honest clients. Note that all experiments without specific clarification follow the basic setting, and the details of the model and training hyper- parampparamtter are provided in Appendix J.1.
For extensive experiments, train the CNN model on Mashion-MNIST (FMNIST) (Xiao et al., 2017) and CIFAR10 (Krizhevsky & Hinton, 2009) datasets. We also consider non-iid settings with different heterogeneous degrees. For the pool of the server's aggregation rulcs, we further consider SOTA aggregators (TM (Yin et al., 2018), Geomed (GM) (Pillutla et al., 2022)), mixed strategy (Bulyan (El Mhamdi el al., 2018)), history-aided aggregator (CC (Karimireddy et al., 2021)), server-side verification methods (ERR (Fang et al., 2020) and (FanFang et al., 2020)). Regarding attacks, we propose a simulation-based attack stratogy AtilokTailor, and implement model poisoning attacks (Mimic (Karimireddy et al., 2022), Alittle (Baruch et al., 2019)), data poi- soning attacks (label flipping (I.) Gonzálee et al., 2017), and random label (LR) (Zhang et al., 2021)).
8

---

Published in Transactions on Machine Learning Research (02/2024)
6.1 Basic Experiments
To validate our theoretical results, we first show the empirical results based on the basic setting, which strictly satisfies the assumption that the server the adversary knows F/A but does not know AT/AG at each iteration. We test Robust against adversary with both single tailored attacks and stronger mixed attacks. Moreover, we show the results in the non-iid settings with three different heterogeneous degrees in Appendix J.2.
Single tailored attacks. RobustTailor successfully decreases the capability of the adversary to launch tailored attacks. RobustTailor maintains stability in Fig. la when Krum fails calastrophically under a small attack. Fig. 1b shows that Robust has much less fluctuations in terms of test accuracy compared to Comed when facing large attack. In addition, 0.5 attack while 65.49% probability of choosing Krum under 100 attack, which
average, RobustTailor has 70.68% probability of
choosing Comed under validates that the server successfully learns how to defend. Training on FMNIST shows consistent results as seen in Fig. 1c and Fig. 1d. Additional results on CIFAR10 are Appendix J.2.
Deh
(a) MNIST, 0.5
(b) MNIST, 100
(c) FMNIST, C= 0.1
(d) FMNIST, 100
Figure 1: Test accuracy on MNIST and FMNIST under iid setting. Attacks with E are applied. RobustTailor selects aggregator from Krum and Comed based the simulation.
Mixed attacks. We now consider two mixed and stronger attack strategies. Building on the basic attack set, StochasticAttack shown in Fig. 2a picks an atlack from = 0.5 and = 100 reverse atlacks uniformly al random at each iteration. AttackTailor in Fig. 2b optimizes an attack based on simulation, whose detailed algorithm is Appendix Compared to all previous attacks including StrchasticAttackk AttackTailor is much stronger since it can pick a proper attack under perfect knowledge of honest updates. The poison of AttackTailor is almost effective the most targeted attack tailored against single deterministic aggregator. Importantly, Robust Tailor shows impressive robustness when facing such strong adversary like Attack Tailor.
Apv
75
Gond
93
97
2.2
00
050
(a) StochasticAttack
(b) AttackTailor
Figure 2: Mixed attacks. AocchastiAAtaan applies = 0.5 and = 100 uniformly al random. Strong AttackTailor optimizes the attack via simulation with perfect knowledge of honest updates.
6.2 Dynamic Strategies
In FL, the server may randomly choose a subset clients for aggregation at each iteration especially when there exists large number of clients, c.g., cross-device FL. The adversary could also apply such dynamic strategy by picking subset of malicious clients to attack each iteration, which would increase the complexity of attack. In this section, we discuss these two scenarios below.
9

---

Published in Transactions on Machine Learning Research (02/2024)
Subsampling by the server. Subsampling, common technique in large-scale FL, can increase the complexity of attacks. The server picks subset of clients randomly for updates at each iteration. The adversary can know which clients are selected and leverage the selected honest updates to design the attack for the compromised clients that are chosen. To simulate subsampling in FL with a large number of clients, we run additional experiments with 120 clients, in which 20 Byzantines are compromised by an adversary. The server chooses 10% of clients randomly for aggregation and assumes of 12 clients are compromisod at each iteration. We decrease the learning rate from 0.01 to 0.001 because all aggregation methods are too unstable under the original setting. The results shown in Fig. 3 are consistent with the results without subsampling.
N Gined
8.9 3.4
4.4
Tevkng
1rd
(a) 6 = 0.5
(b) 6 100
(c)
AttackTailor
Figure 3: Subsampling by the server.
Dynamic strategy of the adversary. The adversary can also use a dynamic strategy by changing the number of malicious updates dynamically. We consider a sctting where the adversary picks 1 1 3 clients randomly control at each iteration; while the server still considers 2 Byzantines among 12 clients. Fig. 4 shows the results. Compared to the results in Fig. 1 and Fig. 2b, Robust Tailor still has a good performance although some aggregation rules in the pool are slightly impacted.
12
Opnader::
Fors
Ta wowioteett
Taking Rerecve
(a) 0.5
(b) 100
(c) AttackTailor
Figure 4: Dynamic attack strategy of the adversary.
6.3 Partial Knowledge of Server/Adversen
Considcring more realistic scenarios, it is a bit strict assuming server/ad server/avversary has the full knowledge of F/A. In this section, we show the results under partial knowledge of the server and the adversary.
More aggregators in the server's pool. Robust is expandable framework, which can include or replace aggregators in the server's pool. The results in Fig. 5 show that Robust framework improves robustness when more aggregators are added to the basic pool. Additional aggregators added here are TM (Yin et al., 2018), GM (Pillutla et al., 2022), Bulyan (El Mhammi et al., 2018), and CC (Karimireddy et al., 2021).
Unknown attacks for the server. We now address the important question of "What will happen if there is an attack out of the server's expectation?" Fig. 6 shows the results when the server does not know the set of attacks. In particular, = 0.1 in Fig. 6a and e = 150 in Fig. 6b are the same lype of allacks as the basic setting; while Mimic (Karimireddy et al., 2022) in Fig. 6C and Alittle (Baruch et al., 2019) in Fig. 6d are different types of attacks. Under Alittle and Mimic attacks, we expand the set of Robust with GM (Pillutla et al., 2022) and Bulyan (El Mhamdi al., 2018) and decrease the learning rate 0.005. Fig. shows that Robust Tailor defend against not only attacks similar to expected ones but also those that are
10

---

Published in Transactions on Machine Learning Research (02/2024)
unexpected with completely different structures and designs. As mixed framework, RobustTailor is hard to fail since the ad adversary hardly designs tailored attack failing several aggregation rules simultaneously.
80-
Cun
Addock ACCECC
33
97
341
32
00
04
06
10
(a) 0.5
(b) <= 100
Figure 5: More aggregators added to Robust
(a) 6 0.1
(b) 6 = 150
(c) Mimic
(d) Alittle
Figure 6: Unknown attacks for the server. Performance of Krum under e = 0.1 does not above 20%.
6.4 Imperfect Public Dataset
Will the quantity and the quality of public datasel impuet the performance of Robusttaaioo We discuss the size of public dataset and various types of distribution shifts with respect the original data of honest clients. In particular, evaluate the robustness of Robust under poisoned data mixed in and data obtained from different data sources. We also show aggregators with auxiliary data used in RobustTailor.
Size of public dataset. In practice, it important to minimize amount of public data as much as possible while maintaining effoctiveness of RobustTailor: Therefore, studying the impact of the size of public data is necessary. Fig. 7 shows the performance of Robust with different proportion of public data in the basic setting. It is obvious that the amount of public data has little impact Robust and even very small proportion of data donated by clients (e.g., 0.1%) helps Robust Tailor achieve great performance. These results further validate Remark 1, that only rough estimate of the actual true update is required using public dataset.
800
Notual M000
Fobs Chrrerrr
Fobu MRAAOIE
250
foouttblert1 Ad
08
722
Fale 10
12.
Telnng
100
(a) 0.5
(b) 6 100
(c) AttackTailor
Figure 7: The impact of the proportion of public data.
Poisoned data mixed in the public datasct. Although most existing aggrogators defend data poisoning attacks which are less effective than model poisoning attacks (Kairouz al., 2021), it is worth considering such attacks because Byzantine clients may be able to donate poisoned data. to the public dataset. We assume 16.7%
11

---

Published in Transactions on Machine Learning Research (02/2024)
of data in the public dataset is poisoned due to 16.7% of malicious clients. We consider two data poisoning attacks: LF (Muñoz-Gonvález et al., 2017) and LR (Zhang et al., 2021). Fig. 8 demonstrates that poisoned data mixed in the public dataset has little impact on RobustTaiiorr, which also validates that small gap between the public datasct and true samples does not reduce the effectiveness Robust Tailor substantially.
80.0
00.0
25.
08.1
keark
024
12.
Keatts wiN
1.8
Teinng
(a) 0.5
(b) 100
(c) AttackTailor
Figure 8: Poisoned data mixed in the public dataset.
Public data obtained from trusted party under distribution shifts. In our paper, we do not require each client to donate data. The public data of the server can also be obtained from a trusted party instead of each client (we discuss this in Remark 1). In the experiments, clients donate a small percentage of data that could be viewed equivalently as a publicly available datasot. We also show experiments that public data helps even when there are challenging and realistic covariate shifts, in which we replace the original public dataset with the same number of digits from EMNIST dataset (Cohen et al., 2017). The empirical results shown in Fig. 9 demonstrate that public data helps even under covariate shifts.
Canes
30
82
6.
6.3
04
02
04
05
04
25
30
(a) 0.5
(b) 100
(c) AttackTailor
Figure 9: Public data obtained from EMNIST.
Aggregators with auxiliary data from the public dataset. Public datasets, in addition to simulation, can aid in aggregation by enabling methods like error rate based rejection (ERR) and loss function based rejection (LFR) (Fang et al., 2020). These methods reject potentially harmful gradients based on error rates or loss values before aggregation. Our experiments with the same setup as in Fig. 2b show that Krun/Comed assisted by ERR/LFR are severely impacted under AttackTailo achieving only 10% accuracy. In contrast, RobustTailor reaches 28% accuracy, providing further evidence of its superiority over existing techniques. Additional experiments reveal that ERR/LFF helps the baseline aggregator to achieve 97% accuracy under 0.5 attack while it is totally ruined under 100 attack. In more challenging scenarios where AttackTailo disrupts baseline aggregation rules, RobustTailor continues to perform effectively.
Discussion. We have considered scenarios where the server owns a public dataset under distribution shifts in this section. RobustTailor also works for another potential scenario which completely avoids storing any data the server by simulating with the updates from small number of trusted clients long the rough est imaae is provided as discussed in Section 4.1.
Additional experiments. To further validate the performance of RobustTailor, we set up additional experiments in Appendix J.2 including 1) three datasets; 2) more Byzantines; 3) non-iid settings; and 4) combination strategies of aggregators.
12

---

Published in Transactions on Machine Learning Research (02/2024)
7 Conclusions
We formulate the robust distributed learning problem as game between a server and an adversary, using Robust to simulate the server's aggregation rules under different adversary attacks. We've 'veee established convergence guarantees for Roiust Empirical results highlight RobustTailor's superiority over robust baselines. Furthermore, extensive experiments demonstrate RobustTailor's adaptability and robustness in various realistic scenarios. Additionally, we've investigated the use of imperfect public dataset estimates enhance FL robustness, showing that Robust Tailor with rough ideal update estimates is effective against. distribution shifts.
Acknowledgments
This work was supported by the Swiss National Science Foundation (SNSF) under grant number 200021 200021_205011 205011. This work was supported by Google. This work was supported by Hasler Foundation Program: Hasler Responsible AI (project number 21043). This work sponsored by the Army Rescarch Office and was accomplished under Grant Number W911NT-24-1-0048.
The work of Ali Ramezani-Kebae was supported by the Research Council of Norway through its Centres of Excellence scheme, Integreat Norwegian Centre for knowledge-driven machine learning, project number 332645; and the Research Council of Norway through its Centre for Research-based Innovation funding scheme (Visual Intelligence under grant no. 309439).
References
Martín Abadi, Ashish Agarwal, Paul Barham, Fugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthicu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Gcoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Πya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viegas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng, TensorFlow: A system for Targe-Scale machine learning. Tn USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), 2016a.
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Tlya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, 2016b.
Scott Alfeld, Xiaojin Zhu, and Paul Barford. Data poisoning attacks against autoregressive models. In AAAI Conference on Artificial Intelligence, 2016.
Dan Alistarh, Zeyuan Allen-Zhu, and Jerry Li. Byzantine stochastic gradient descent. In Advances in Newral Information Processing Systems (NeurIPS), 2018.
Zeyuan Allen-Zhu, Faeze Ebrahimian, Jerry Li, and Dan Alistarh. Byzantine-resilent non-convex stochastic gradient descent. In International Conference on Learning Representations (ICLR), 2021.
Yoshinori Aono, Takuya Hayashi, Lihua Wang, Shiho Moriai, et al. Privacy-preserving deep learning via additively homomorphic encryption. IEEE Transactions on Information Forensics and Security, 13(5): 1333 1345, 2017.
Peter Auer Nicolo Cesa Disa-Bianchi, Yoav Fround, and Robert E. Schapire. The nonstochastic multiarmed bandit problem. SIAM Journal on Compuling, 32(1):48 77, 2002.
Brendan Avent, Aleksandra Korolova, David Zeber, Torgeir Hovden, and Benjamin Livshits. BLENDER: Enabling local search with a hybrid differential privacy model. In USENIX Securily Symposium, 2017.
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. Tn International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.
13

---

Published in Transactions on Machine Learning Research (02/2024)
Gilad Baruch, Moran Baruch, and Yoav Goldberg. A little is enough: Circumventing defenses for distributed learning. In Advances in Newral Informalion Processing Systems (NewrIPS). 2019.
Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Elficient algorithms and tight error bounds. In 2014 IEEE 55th annual sympostum on foundations of computer science, 2014.
Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateck Mittal, and Seraphin Calo. Analyzing federated learning through an adversarial lens. In International Conference on Machine Learning (ICML), 2019.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. In International Conference on Machine Learning (ICML), 2012.
Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. In Advances in Newral Information Processing Systems (NeurIPS), 2017.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for srivacy-preservng machine learning. Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Securily, 2017.
Léon Bottou. Online learning and stochastic approximations. On-line learning in newral networks, 1998.
Xiaoyu Cao, Minghong Fang, Jia Liu, and Neil Zhenqiang Gong. FLTrust: Byzantine-robust federated learning via trust bootstrapping. In Network and Distributed System Security (NDSS) Symposium, 2021.
Xinyang Cao and Lifeng Lai. Distributed gradient descent algorithm robust to an arbitrary number of Byzantine attackers. IEEE Transactions on Signal Processing, 67(22):5850-5844 2019.
Nicholas Carlini and Andreas Terzis. Poisoning and backdooring contrastive learning. In International Conference on Learning Representations (ICLR), 2022.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEFE Symposium on Security and Privacy (SP), 2017.
Yudong Chen, Lili Su, and Jiaming Xu. Distributed statistical machine learning in adversarial settings: Byzantine gradient descent. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 1(2):1-25, 25, 2017.
Gregory Cohen, Saced Afshar, Jonathan Tapson, and Andre Van Schaik. EMNIST: Extending MNIST to handwritten letters. In International Joint Conference on Neural Networks (IJCNN), 2017.
Deepesh Data and Suhas Diggavi. Bysinttne-resllent high-dimensional SGD with local iterations on heterogeneous data. In International Conference on Machine Learning (ICML), 2021.
Ambra Demontis, Marco Melis, Maura Pintor, Malthew Jagielski, Battista Biggio, Alina Oprea, Cristina Nita-Rotaru, and Fabio Roli. Why do adversarial attacks transfer? explaining transferability of evasion and poisoning attacks. In USENIX Security Symposium, 2019.
Kate Donahue and Jon Kleinberg. Model-sharing games: Analyzing federated learning under voluntary participation. In AAAI Conference on Artificial Intelligence, 2021.
Shaddin Dughmi, Omkar Thakoor, and Umang Gupta. CSC1699: Topics in learning & game theory: Lecture pdf.
6. pp. 11, 2017. URL https://viterbii-ee usc. w.w.uuu....eeeu http://scdddncccffff////eeeee//e
El Mahdi El Mhamdi, Rachid Guerraoui, and Sébastien Rouault. The hidden vulnerability of distributed learning in Byzantium. In International Conference on Machine Learning (ICML), 2018.
El Mahdi El Mhamdi, Rachid Guerraoui, and Sébastion Louis Alexandre Rouault. Distributed momentum for Byziitine-resllient stochastic gradient descent. In International Conference on Learning Representations
(ICLR), 2021.
14

---

Published in Transactions on Machine Learning Research (02/2024)
Ahmed El Ouadrhiri and Ahmed Abdelhadi. Differential privacy for deep and federated learning: A survey. IEEE Access, 10:22359-223661 2022.
Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Gong. Local model poisoning attacks to Byzantine-robust federated learning. In USENIX Security Symposium, 2020.
Xiuwen Fang and Mang Ye. Robust federated learning with noisy and heterogeneous clients. In Conference on Computer Vision and Pottern Recognition (CVPR), 2022.
Shaohan Feng, Dusit Niyato, Ping Wang, Dong In Kim, and Ving-Chang Liang. Joint service pricing and cooperative relay communication for federated learning. In 2019 International Conference on Internet of Things (iThings) and IEFE Green Compuling and Commenmicatinns (GreenCom) and IEEF Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData), 2019.
Donald L. Fisk. Quttim-mrtsngales. Transactions of the American Mathematical Society, 120(3):365 389, 1965.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy, Explaining and harnessing adversarial examples. arXiv preprint arXio:1412.6777 2014.
Ala Gouissem, Khalid Abualsand, Elias Yaacoub, Tamer Khattab, and Mohsen Guizani. Federated learning stability under Byzantine atlacks. In IEEE Wireless Communications and Networking Conference (WCNC), 2022.
Tianyu Gu, Kang Lüu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230 47244, 2019.
Xin Gu, Gautam Kamath, and Zhiwei Steven Wu. Choosing public datasets for private machine learning via gradient subspace distance. arxiv preprint arXiv:2303.011 01256,, 2023.
Elad Hazan. Introduction to online convex optimization. Foundations and Trendso in Optimization, 2(3-4): 157-325, 2016.
Ling Huang, Anthony D Joseph, Blaine Nelson, Benjamin IP Rubinstein, and J Doug Tygar. Adversarial machine learning. In ACM Workshop on Security and Artificial Intelligence, 2011.
Wenke Huang, Mang Ye, and Bo Du Learn from others and be yourself in heterogeneous federated learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru, and Bo Li. Manipulating machine learning: Poisoning attacks and countermessures for regression learning. In IEEE Symposium on Security and Privacy (SP), 2018
Jinyuan Jia, Zhuowen Yuan, Dinuka Sahabandu, Luyao Niu, Arezoo Rajabi, Bhaskar Ramasubramanian, Bo Tı, and Radha Poovendran. Flgame: game-theoretic defense against backdoor attacks federated learning. 2022.
P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, R. G. L. D'Oliveira, S. E. Rouayheb, D. Evans, J. Gardner, Z. Garrett, A. Gascón, B. Ghazi, P. B. Gibbons, M. Gruteser, Z. Harchaoui, C. He, L. He, Z. Huo, B. Hutchinson, J. Hsu, M. Jaggi, T. Javidi, G. Joshi, M. Khodak, J. Konečný, A. Korolova, F. Koushanfar, S. Koyejo, T. Lepoint, Y. Liu, P., M. Mohri, R. Nock, A. Özgür, R. Pagh, M. Raykova, H. Qi, D. Ramage, R. Raskar, D. Song, W. Song, S. U. Stich, Z. Sun, A. T. Suresh, F. Tramèr, P. Vepakomma, J. Wang, L. Xiong, Z. Xu, Q Yang, F. X. Yu, H. Yu, and S. Zhao. Advances and open problems in federated learning. Foundations and & in Machine Learning, 14(1-2):1-210, 2021.
Jiawen Kang, Zchui Xiong, Dusit Niyato, Han Yu, Ying-Chang Liang, and Dong In Kim. Incentive design for efficient foderated learning in mobile networks: A contract theory approach. In IEEE VTS Asia Pacific Wireless Communiccatins Symposium (APWCS), 2019.
15

---

Published in Transactions on Machine Learning Research (02/2024)
Sai Pranceth Karimireddy, Lie He, and Martin Jaggi. Learning from history for Byzantine robust optimization. In International Conference on Machine Learning (TCML.) 2021.
Sai Praneeth Karimireddy, Lie He, and Martin Jaggi. Byzantine-robust learning on heterogeneous datasets
via bucketing. In International Conference on Learning Representations (ICLR), 2022.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference Machine Learning (ICML), 2017.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arxiv preprint Xrx:v (0..004909 2016.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Leslie Lamport, Robert Shostak, and Marshall Pease. The Byzantine generals problem. ACM Trons. Progrum. Lang. Syst., 4(3):382-401, 1982.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, 86:2278 2324, 1998.
Sam Leroux, Steven Bohez, Elias De Coninck, Tim Verbelen, Bert Vankeirsbilek, Pieter Simoens, and Bart Dhoedt. Multi-fidelity matryoshka neural networks for constrained loT devices. In International Joint Conference on Neural Networks (IJCNN), 2016.
Bo Li, Yining Wang, Aarti Singh, and Yevgeniy Vorobeychik. Data poisoning attacks on factorization-bassed collaborative filtering. In Advances in Neural Information Processing Systems (NewrIPS), 2016.
Saeed Mahloujifar, Mohammad Mahmoody, and Ameer Mohammed. Data poisoning attacks in multi multi-party learning. In International Conference on Machine Learning (ICML), 2019.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- efficient. learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2017.
Shike Mei and Xiaojin Zhu. Using machine teaching identify optimal training-set attacks on machine learners. In AAAT Conference Artificial Intelligence, 2015.
Michel Métivier. Semimartingales. Walter de Gruyter, 1982.
Payman Mohassel and Yupeng Zhang. Secureml: A system for scalable privacy-presorving machine learning. 2017 IEEE symposium on securily and privacy (SP), 2017.
Luis Muñoz Gonzálzz,,,,, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back-gradient optimization. In ACM Workshop on Artificial Intelligence and Security, 2017.
John F. Nash. Equilibrium points in n-person games. Proceedings of the National Academy of Sciences, 36 (1):48 49, 1950.
Kerem Ozfatura, Emre Ozfatura, Alptekin Kupcu, and Deniz Gunduz. Byzantines can also learn from history: Fall of centered clipping in federated learning. arXio preprint or orxin:2208. 200:290000999 2022.
Jungwuk Park, Dong-Jun Han, Minseok Choi, and Jaekyun Moon. Sageflow: Robust federated learning
against both stragglers and adversaries. Advances in Neural Information Processing Systems (NeurIPS), 34:840-851, 851, 2021.
Krishna Pillutla, Sham M Kakade, and Zaid Harchaoui. Robust aggregation for federated learning. IEEE Transactions Signal Processing, 121142-11544 2022.
16

---

Published in Transactions on Machine Learning Research (02/2024)
Shashank Rajput, Hongyi Wang, Zachary Charles, and Dimitris Papailiopoulos. DETOX: A redundancy-based framework for faster and more robust gradient aggregation. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Ali Ramezani-Kebrya, Iman Tabrizian, Fartash Faghri, and Petar Popovski. MixTailor: Mixed gradient aggregation for robust learning against tailored attacks Transactions on Machine Learnång Research (TMLR), 2022.
Nuria Rodriguuz- Daniel Jiméncz-López, M Victoria Luzón, Francisco Herrera, and Eugenio Martinez- Cámara. Survey lederated learning threats: concepts, Laxonomy on attacks and defences, experimental study and challenges. Information Fusion, 90:148- 10:148-173, 2023.
Yevgeny Seldin, Csaba Szepesvári, Peter Auer, and Yasin Abbasi- Yadkori. Evaluation and analysis of the performance of the EXP3 algorithm in stochastic environments. In Europeon Workshop on Reinforcement Learning, 2013.
Jinhyun So, Başak Güler, and A Salman Avestimehr. Byzintine-resilient secure federated learning. IEEE Journal on Selected Areas in Communications, 39(7):2168 21112,,18- 2020.
Lili Su and Nitin H Vaidya. Fault-tolerant multi-agent optimization: optimal iterative distributed algorithms. In ACM symposium on Principles of Distributed Computing, 2016.
Jingwei Sun, Ang Li, Louis DiValentin, Amin Hassanzadeh, Yiran Chen, and Hai Li. FL-WBC: Enhancing robustness against model poisoning attacks in federated learning from client perspective. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H. Brendan McMahan. Can you really backdoor federated learning? arXiv preprint arXiv:1911.00966 2019.
E Tahanian, M Amouci, H Fatch, and M Rezvani. A game-thcoretic approach for robust federated learning. International Journal of Engineering, 34(4):832 842, 2021.
Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy-yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos. Attack of the tails: Yes, you really can backdoor federated lcarning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Tongzhou Wang, Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.1000 2018.
Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H Yang, Farhad Farokhi, Shi Jin, Tony QS Quck, and H Vincent Poor. Federated learning with differential privacy: Algorithus and performance analysis. IEEE Transactions on Information Forensies and Security, 15:3454 3469, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint 108.0700 2017.
Chulin Xie, Keli Huang, Pin- Yu Chen, and Bo Li. DBA: Distributed backdoor attacks against federated learning. In International Conference on Learning Representattinns (ICLR), 2019a.
Cong Xie, Sanmi Koyejo, and Indranil Gupta. Zeno: Distributed stochastic gradient descent with suspicion- based fault-tolerance. In International Conference on Machine Learning (ICML), 2019b.
Cong Xie, Oluwasammi Koyejo, and Indranil Gupta. Fall of empires: Breaking Byzantine-tolerant SGD by inner product manipulation. In Uncertainty in Artificial Intelligence, 2020a.
Cong Xie, Sanmi Koyejo, and Indranil Gupta. Zeno+:: Robust fully asynchronous SGD. In International Conference on Machine Learning (ICML), 2020b.
Yi-Rui Yang and Wu-Jun Li. BASGD: Buffered asynchronous SGD for byzantine learning. In International Conference Machine Learning (ICML), 2021.
17

---

Published in Transactions on Machine Learning Research (02/2024)
Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantine-robust distributed learning: Towards optimal statistical rates. In International Conference on Machine Learning (TCMT.), 2018.
Naoya Yoshida, Takayuki Nishio, Masahiro Morikura, Koji Yamamoto, and Ryo Yonetani. Hybrid-FL for wireless networks: Cooperative learning mechanism using non-IID data. In IEEF International Conference Communications (ICC), 2020.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arxiv preprinl arXiv:1806. 00582, 2018.
Yuze Zou, Shaohan Feng, Dusit Niyato, Yutao Jino, Shimin Gong, and Wenqing Cheng. Mobile device training strategies in federated learning: An evolutionary game approach. In International Conference Internet of Things (iThings) and IEEE Green Computing and Communications (Green and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartDala), 2019.
18

---

Published in Transactions on Machine Learning Research (02/2024)
Notation. We use B[,,, || to denote the expectation operator, Euclidean norm respectively. We use |-1 to denote the length of binary string, the length of a vector, and cardinality of a set. We use lower-case bold letters to denote vectors. Sets are typeset in a calligraphic font. We use [n] to denote {I..... nn for an integer n. We use AM to denote the probability simplex in RM.
A Complete Related Work
Federated learning (FL). FL (Konečný al., 2016; McMahan et al., 2017) keeps training data decentralized in multiple clients which collaboratively train a model under the orchestration of server (Kairouz et al., 2021). For the server, such clients are often more umpredictable and especially more vulnerable to the attacks. Secure aggregation protocols (Bonawitz et al., 2017: So et al., 2020) ensure that the server computes aggregated updates without revealing the original data. In this paper, we focus on training-tine attacks and corresponding aggregation rules.
Training-time attacks. Standard attacks can be broadly classified into training-time atlacks (poisoning attacks) (Huang et al., 2011; Biggio et al., 2012; Mei & Zhu, 2015; Li et al., 2016; Alfeld et al., 2016; Koh & Liang, 2017; Jagiclski ct al., 2018; Bhagoji et al., 2019; Mahloujifar et al., 2019; Gu et al., 2019; Xic et al., 2019a; Wang et al., 2020; Allen- Zhu et al., 2021; Yang & Li, 2021; Karimireddy et al., 2021; Data & Diggavi, 2021; Carlini & Terzis, 2022) and inference-time attacks (evasion attacks) (Goodfellow et al., 2014; Carlini & Wagner, 2017). Because the server in FL trains the model across various unreliable clients with private datasets, FL usually suffers from training-tiin timeing--tmmeai attacks (Biggio et al., 2012; Bhagoji et al., 2019; Sun et al., 2019; Bagdasaryan et al., 2020). A strong adversary can potentially participate in every training round, and meanwhile it can adapt its attacks to an updated model. One class of training-tinn time attacks concerned in this work is model updale poisoning. In model poisoning allack, an adversary can control some clients and can directly manipulate their outputs trying to bias the global model towards the opposite direction (Kairouz et al., 2021). If Byzantine clients have access to the updates of honest clients, they can tailor their attacks and make them difficult to detect (Lamport al., 1982; Goodfellow et al., 2014; Blanchard et al., 2017; Fang et al., 2020; Xie et al., 2020a; Bagdasaryan et. al., 2020).
Robust aggregation and Byzantine resilience. To improve robustness under general Byzantine clients, a number of robust aggregation schemes have been proposed, which are mainly inspired by robust statistics such as median-basee aggregators (Chen et al., 2017; Yin et al., 2018), Krum (Blanchard et al., 2017), trimmed mean (Yin et al., 2018). Krum (Blanchard et al., 2017) and coordinate-wise median (Comed) (Chen et al., 2017: Yin et al., 2018) are two main aggregation rules used in this paper. Krum is a squared-distance--asccd aggregation rule and il aggregates the gradients that mininize the sum of squared distances to its ʃ - 2 closest vectors where n denotes the total number of clients and f is the number of adversarial ones. Comed is a median-based aggregator and it selects the gradient closest to the median of cach dimension.
Except of statistical aggregation rules, there are still many related works like server-side verification, client-side self-clipping etc. From the perspective of the server, Cao & Lai (2019); Fang et al. (2020); Xie et al. (2020a); Cao et al. (2021) propose some server-side verification methods using auxiliary data. Specifically, Fang et al. (2020) assume the server has small validation dataset and uses error rates to reject harmful gradients. In (Cao & Lai, 2019; Xie et al., 2020a), the server asks a small clean dataset from clients and filters out unreliable gradients. Cao et al. (2021) utilize the ReLU-clipped cosine-similarity between local gradients and the standard one calculated by a small clean dataset as the weight for aggregation. Morcover, Karimireddy et al. (2021) and Alistarh et al. (2018) propose history-aided aggregators, and an expandable framework proposed by Ramozani-Kebrya et al. (2022) utilizes randomization to improve robustncss. None of them selects proper aggregation rule proactively during training as our framework RobustTailor. We note that all aggregation rules shown here can be added to the pool of RobustTailor because a public dataset is available in our assumption and any aggregation rule can use it. In addition, client-side clipping methods are proposed by Sun et al. (2021) and Sun et al. (2019), and client-side momentum SGD is considered by Karimireddy et al. (2021) and El Mhamdi et al. (2021). However, the ability of clients is not the focus of our paper and we will consider it in future work.
Past work has shown that these aggregators can defend successfully under specific conditions (Su & Vaidya, 2016; Blanchard et al., 2017; Chen et. al., 2017). However, Fang et al. (2020) and Xie et al. (2020a) argue
19

---

Published in Transactions on Machine Learning Research (02/2024)
that resiiint aggregators can fail when an informed adversary tailors careful attack. Therefore, developing a robust and efficient algorithm under such strong tailored attacks is essential to improve security of FL, which is the goal of this paper.
Public dataset in FL. Section 3.1.1 of (Kairouz et al., 2021) points out that a small global shared dataset is acceptable in FL. This dataset may originate from a publicly available proxy data source, separate dataset from the clients' data that is not privacy sensitive, or perhaps a distillation of the raw data following (Wang et al., 2018). And the server could process or choose the public dataset to achieve privacy protection, e.g. producing a privacy-preserving datasel before data donation (El Ouadrhiri & Abdelhadi, 2022) and using out-of-domain data as a clean training dataset of the server (Jia et al., 2022). Note that the existence of such public datasct is a valid and common assumption in FL (Yoshida et al., 2020; Kairouz et al., 2021; Fang & Ye, 2022; Huang et al., 2022), and Sagellow (Park et al., 2021), Zeno (Xie et al., 2019b), and Zeno++ (Xie et al., 2020b) also utilize public data at the server to handle adversaries.
In addition, the presence of public data at the server makes it possible that multiple clients collaboratively train model with formal differential or hybrid differential privacy guarantees. One direction to improve the utility of differentaally private machine learning is using public data that is not subject to any privacy constraint. (Gu et al., 2023). Moreover, Section 6.3 of (Kairouz et al., 2021) states that "hybrid differential privacy" is a strategy for differentially-priien guarantees. Avent al. (2017) introduce hybrid differential privacy where some opt-in users voluntarily donate data. Many companics (e.g., Mozilla and Google) roly a group of testers with higher levels of mutual trust who voluntarily opt-in to a less privacy-preserving model than that of an average end-user.
learning in FL, in which some users donate data with lesser privacy
Game theory in robust FL. Game theory is widely used in FL such as game-based incentive model for efficient FL (Kang et al., 2019; Donahue & Kleinberg, 2021), Stackelberg game model for the transmission strategy in FL system (Feng et al., 2019), and an evolutionary game model for training strategics of the mobile devices (Zou et al., 2019). Although these work used the game theory, none of them focused on robust training in FL. Tahanian et al. (2021) propose a game-based aggregation algorithm, named GFA, to detect and discard bad updates provided by the clients. This work does not consider the tailored attacks and it constructs a mixed-strategy game between the server and each client. Unlike our framework building a game between a server and an adversary, GFA ignores the collaboration among compromised clients, which could enhance attack effect (Fang ct al., 2020).
B Robustness of Robust Railor
In this section, we define a general robustness definition of an aggregation rule against an attack. Note that. our definition covers a broad range of settings with general pure and mized aggregation along with general pure and mixed altack strategies. Our robustness notion leads to almost sure convergence guarantees to a local minimum of F in FL, which is equivalent to being immune to training-ting time attacks.
Definition& 4 (Robustness of an aggregator to an attack program). Let x E Rd denote a machine learning model. Let (x) = (x) E Rd be independent honest updates for i E [n]. Let G(x) denote a function that drows an honest client uniformly at random followed by outputting an unbiased stochastic gradient of Fx) over that client such that E[G(x))= = FFFx))]=x((x)) where E. is over both random client and samples. Let AG denote an arbilrary aggregation rule, which con be a mixed aggregation strategy selecting an aggregator from A A= [AG1....,ACE, ) based on simulation. The output of AG is given by g(x) = AG((g'))" (=")" Note that (g' = includes both honest and compromised updates. The compromised updates are the output of an attack program AT((g:) .A). Note that AT can be a pure or mixxdd allack stralegy-
The mized aggregation rule AG is Sopaantin-resillend to AT if ğ(x) satisfies Bi(()) [VF(x) 0 and B[1lØ(x)]11 < (,E[[[ for r 2,3,4 4 and some constant K,.
Suppose { in Algorithm 2 satisfies E. = oo and E. n < 00. For a nonconvez loss function, which three times differentiable with continuous derivatives, bounded from below, and satisfics global confinement assumption in (Bottou, 1998, Section 5.1), general pure and mized aggregation and attack strategies satisfying Definition 4, and general non-idd data distribution across clients, we can establish almost.
20

---

Published in Transactions on Machine Learning Research (02/2024)
sure convergence (VF(x;) - 0 a.s.) of the output of AG in Algorithm 2 along the lines of (Bottou, 1998; Fisk, 1965; Métivier, 1982).
Note that to achieve E [x]] VVFx) [x()) > 0 shown above, it requires both the distance between Vf(x) and the estimate of the honest update and the distance between g and the expected output of Algorithm 1, ie., [[](()), are small. Let 0₂ denote the angle between V>((x) and 8, and let 02 denote the angle between g and
respectively. 01 + 02 < x/2, then we have [ Fx) > Following the arguments in Appendix B, almost sure convergence of Algorithm is guaranteed as long as + 02 t/2. This condition can be satisfied assuming 1) the public data donated by clients is representative of the underlying data distribution of honest clients, which controls 01, and 2) the number of Byzantine clients is sufficiently small, which controls 02- We defer derivation of the explicit. necessary condition for almost sure convergence to future work.
[x),,,,, given by arg cos
8 <'V((x) 1 [F(x]]] and arg cos
& ElE x)
g g
C Hypothetical process of model aggregation
The hypothetical process of model aggregation with T rounds is shown in Algorithm 3.
Algorithm Hypothetical process of aggregation
Input:
Initial weight vector Xo, learning rate nu, iteration rounds T, number of clients set of aggregation rules A, set of attack algorithms F.
for t= 1 to 7 do
Server sends x, to all clients.
for i= 1 to
do Honest client i computes local gradient (xx)).
Compromised clients send attacks ([g; // J==1 aa- .A).
Sever receives gradients from all clients {g/
Server chooses AG by solving minAG'EA ma ((AG,, AT,, (g))). Server updates the model X++1 X2 (()))
Details of Exp3
The bandit feedback model considors the following iterate game. Definition 5 (Bandit setting). The player is given decision set [N]. At each iteration k = 1...., K:
the player picks is [N].
2. the adversary picks loss vector th
the player observes and suffers the loss at index in i.e. (ix).
Exp3, shown abstractly in Algorithm 4, enjoys so called no-regret property in this sctting, We employ Exp3 from both the perspective of a simulated server and simulated attacker to find robust aggregation rule in Algorithm 1. In Appendix F we show how to convert the no-regret properties into a convergence guarantee.
E Simulation of Adversary
In this section, we show the simulation of the adversary. We term adversarial simulation as AttackTailor, which outputs an appropriate AT at each iteration. The specific steps from the perspective of the adversary is shown in Algorithm 5. After observing n - honest gradients, the server performs K K-round simulation and obtains final probability distribution q. By sampling from q, the server selects an attack. Then, f Byzantine clients create and send the compromised gradients to the server. The steps for simulating the attack procedure summarized in Algorithm 6.
21

---

Published in Transactions on Machine Learning Research (02/2024)
Algorithm 4 Exp3
Input: Updating rate A and X, iteration rounds K, N
Initialize weight vector w for i = I,...,,. N.
for k =1 to K do
Set W = -i= w* (i), and set for i= 1, I,...,NN N
p(1) (1 A) w² (1)
Wh +1 N
Draw in randomly according the probabilities p. Receive loss gk. Set for i I.....N
(1) 8t /P((),
if ig otherwise.
0,
w w (1) = w² (x)))xx((22
Importantly, the main difference between the adversary's simulation compared with that of server is that the adversary does simulation based on realistic honest gradients while the server has access only to noisy estimates true gradients. Hence, unlike typical games and simulation setups, the adversary has an additional advantage over the server, which is due to information asymmetry.
Algorithm AttackTailor
Input: Updating rates A₁. A2, d and λ2, simulation rounds K, gradients of honest clients {g) A and F Initialize weight vector (i) = for i E [M] and w)( = 1 for j E [5].
for & = 1 K do
Set p((AG:) (1 -1)
())
A1 for i E [M].
= (1)
Set q" (AT,) (1 12)
+2 for j e [S].
Σj +t(
Sample A ~p* and AT* qr respectively.
Estimate the loss eh = E(AG 4 {g
Set for i == I,...,M M
(1) HAG, AG*} (=GA,)
eh wx+1(1)= -x0))c))-)())))))0)
Set for j = S.....S
E5)) 1AT, AT*} cb. w w+++()) exp((
(ATA))
Set for j = 1, S,...,S
q KK q (AT,)
K
Sample AT ~ q. Output: AT.
22

---

Published in Transactions on Machine Learning Research (02/2024)
Algorithm 6 Adversary's attack
Input: Learning rate VJ, n workers, f compromised workers, iteration rounds T, A and FF for == to T do
Observe all gradients of honest workers {g) )=
Call Algorithm to attack AT" = AttackTailor(( {g: h-1 F..)..
Produce f gradients for compromised clients. Set for j E [f]
bj AT ! AA).
Send compromised gradients {by to the server.
F Lemma 2 and Lemma
The argument builds on the well-known idea of performing approximate equilibrium computations through simultancously played online algorithms. Let consider general objective, L: [M] x [S] R__ Consider simultaneously running two algorithis on the objective L, such that their respective expected regrets are upper bounded by some quantities R2 and R2 i.e.,
K Σ (k) - E1(1, S
K
K
K (( - Lii.) S Rkx
RRx
E
Σ
(10)
for any E [M] and j E [S] where the expectation is taken over the randommess of the algorithms.
Lemma (Folklore). Suppose we run two algorithms simultaneously with regrets as in (10) to obtain 1=1 we guarantee that
((k.))))) By playing uniformly sampled from (ig K
1 5+)] R (R + R(x);
R3 [TG,,)] & Bespp" [[(""
(11)
for any j [S] where (p',q") q*) is Nash equilibrium of Bi--p" ("}})")]
This kind of result is well-known in the litcrature (sce for instance Dughmi et al. (2017, Cor, 4) for a very related result).
Proof. Defined ī ~ p to be uniformly sampled from {ix H and ~ q to be uniformly sampled from Likk Using the no-regret property from (10),
1
B[L(),)) = E
((
E
L(ir.jn)
K R
k=1
(12)
K
K
Σ
(i.jkk)
E
(ixk (k)
Rk
K
K
K
k=1
for any E [M] and j E [S], where the expectation is taken over i and j and the randommess of the algorithms. Subtracting the two equations,
E [[(().)) B(L(E,)
1 (Rk Rkx Esim-
(13)
K
Observe that by first evoking the inequality with i ~ p and secondly with j ~ q, we see that (p, q) is an aapr-appoo Nash equílibrium, i.c.,
E [(((),)) Esim <((((,)) (((((1,)) Esim-
(14)
23

---

Published in Transactions on Machine Learning Research (02/2024)
We are interested in the ž-players performance E[L(i,)) which we relate to the mixed strategy Nash equilibrium defined as (B((("",)) & ((L("" - [[T(),"") where i ~ p and jr q' By picking i ~ p in (13) we get,
B(T(,,5) B(T(,,) +
K (R' + Rk)
1
= E[L(i, .5] + S BIL(C
K (R's +Rk)
(15)
(R) + Rx),
K
where the last inequalities follows by the definition of a Nash equilibrium above. The claim follows by writing the expectation on the RHS terms p and q*.
When the algorithms have sublinear regrets, we refer to them as no-regret algorithms. This condition ensures that the error term in (11) vanishes as K + cc. Exp3 of Auer et al. (2002), employed by both the adversary and server in Algorithm 1, enjoys such a no-regret property.
Lemma 3 (Hazan 2016, Lemma 6.3). Let K be the horizon, N be the number of actions, and La : [N] - IR+ be nov-negative losses for all k. Then Exp3 with stepsize X = V log N KN enjoys the following regret bound,
K Σ k=1
E
La (ix)
Lx(i)
KN
(16)
for any E [N], where the expectation is taken over the randomness of the algorithm.
G Proof of Lemma 1
Proof. Let both player i and player j in Lemma. 2 employ the no-regret algorithm Exp3 such that Lemma 3 applies and consequently R2 and R in (10) reduce to
R = 22 KM M, R2x = 22 2x=2/KSlogg Kgg
(17)
Substituting (17) into Lemma 2, we have
M log M + Vloog E; [((,,) < Eurppr (L(ii"j") + +2
(18)
VK
Notice that Algorithm instance of two simultaneously played Exp3 algorithms where = AG, AT and I(i,j) = & [AG, AT, AGG.T. (E It follows from (18) that
BAG 7 (AG,AT, [[)] [AG < EAG AT [(AGG,,TT (()))(])) , + Esim
(19)
where
V log M + V log
Esim
(20)
VK
where AG is the average iterate as defined in Algorithm 1. We concisely write the Nash equilibrium the R.H.S. of (19) in terms of the payoff matrix L from MixedNash defined componentwise as L(AG, [AG,, AT, [g) (. This completes the proof.
24

---

Published in Transactions on Machine Learning Research (02/2024)
H Proof of Theorem 1
Proof. Expanding R.H.S. of Eq. (7) and referring Lemma 1, we have
(BAc(AG((3)=)))_ (IRAAG/I (IRAGIAG((3) 1 -PAcc(
(21)
1
[[AAGAAG((]))
1 PPppp LLqL"q Esi
where -= E(AG, AT, (B)=)) AG p. AT F, and (o fixed simulated gradients. We now
establish lower bound on (EAG [AG(( ) g*) for the ideal update g" using the lower bound in Eq. (21).
(AAGAAG(())E-)))
(EaG [AAG(AG(()))))) g")
(EAG(AG
z [EAG[AG((E))))) -EEc
-
([BACIAGG(())))) p= -q Esim
(22)
+ 11 - 2 AG ( 12 |lllll" -g"'l)
2 | c g" 12 p "Lq' 6
where C' is an upper bound on the norm of the output of AG. Under Assumption 4, we have
g²122 Vest
g
+ Best
(23)
Then, take expectation from both sides of Eq. (22):
CVest. n n-ド Elp" T Lq*] - /M log M + /S
(BAg[AG((B) g") E
CB
(24)
K
If (BAG[AG([E]) g+) > 0, then the almost sure convergence (VF(x,) 0 a.s.) of the output of AG in Algorithm 2 can be established along the lines of (Bottou, 1998; Fisk, 1965; Métivier, 1982), which complete the proof of Theorem 1.
Privacy Guarantee
To further settle privacy concerns, we also consider the second scenario under stringent privacy requirements. The server trusts only a small subset of honcst clients, which can provide reliable updates. It is normal that the server has some core clients that are Irustworthy. For example, some companies rely on lesters with high-level mutual trust. In such a situation, RobustTaaion does not lead to any additional privacy loss compared to any other aggregation method that receives individual stochastic gradients from clients. Note that RobustTailor is compatible with all privacy- preecccyppeeeriingg techniques, e.g., differential privacy (DP) (Bassily et al., 2014; Wei et al., 2020), homomorphic encryption (Aono et al., 2017), and secure multiparty computation (Mohassel & Zhang, 2017). Let Tmax denote the largest sampling ratio among trusted clients. Along the lines of, e.g., [Abadi el al., 2016b, Theorem 1], clipping and adding zero-mean Gaussian noise with (1/ logo log(T/8)/e) to updates of trusted clients sufficient to guarantee
a standard deviation [(Tmax SoTTmax/Tllol(
(c, 5D-.. Similar noise can be added to all clients to extend privacy guarantees the entire group, as discussed in Corollary 1.
Corollary 1 (Joint convergence and privacy guarantees). Under the setting described in Theorem 1, clipping and adding zero-mean Gaussiun noise with slandard deviation 2(rmax V TToomxx (18(1/8) log(T/8)/e). and the first term of R.H.S of Eq. (9) dominates the lower bound, then n - f = 2( T (/0) (og(((/)//e) is sufficient for CobussTaaloo to successfully establish both almost sure convergence and (c, DDD guarantees.
Remark (Trado-off between privacy and convergence guarantecs). There erists a trade-off between privacy and convergence guorantees. In particular, by decreasing e, in (c, stronger privacy guarantees are achieved for trusted clients; while it slows down convergence in Theorem due to an incrensed Vest-
25

---

Published in Transactions on Machine Learning Research (02/2024)
J Experimental Details and Additional Experiments
In this section, we provide the training hype-parameters and show a series of additional experiments.
J.1 Details of Implementation
Both MNIST (Lecun al., 1998) and FMNIST (Xiao et al., 2017) datasets contain 60000 training samples and 10000 test samples. Each sample is 28 by 28 pixel grayscale image. The details of training hype-parametent are shown in Table 1. The net network architecture is a fully connected neural network with two fully connected layers (Leroux et al., 2016). The number of neurons is 100 and 10 for the first and second layer, respectively. All experiments have been run on a cluster with Xeon-Gold processors and V100 GPUs.
Table 1: Training hyper-parameters for MAIST, FMNIST, and CIFAR10
Harameerr
MNIST
FMNIST
CIFAR10
Learning Rate
0.01
0.003
0.002
Batch Size Total Iterations
50
80
15K
10K
10K
K
10
10
10
Å1 A2
0.3
0.3
0.3
d1 A2
0.3
0.3
0.3
J.2 Additional Experiments
In this section, we set up two additional experiments to further validate the performance of Robust
Different datasets. We train CNN model MNIST (Lecun al., 1998); Fashion-MNIST (FMNIST) (Xiao et al., 2017) and CIFARIO (Krizhevsky & Hinton, 2009) under iid setting. We summnarize the training results against 3 attacks in Fig. 10 and they are consensus with the results shown in Section 6.
More Byzantines. Fig. 11 shows the results when there are Byzantines (2 Byzantines in the basic setting) in 12 total clients under three different attacks. We observe that both Krum and Comed are sensitive to the number of Byzantine clients while Robusttailo is much more stable. Specifically, Krum has lower accuracy closing to zero, and Comed shows more obvious fluctuations.
Non-iid scttings. We also extend our consideration to more realistic scttings with non-iid data distribution across clients. We use the helerogeneous degree μ [0, 1] to represent the level of disparity among clients' construct setting, in which 100μ % of local data for each client is drawn
local data. To be specific, in non-identical but independent manner from particular class corresponding to the client index and 100(1 - μ4 % of the local data is drawn iid from all classes. A small p represents low disparity while large # means significant. disparity among clients. Fig. 12 shows three non-iid settings including μ = 0.1, 1,1, 0.5, 0.9. Robust Tailor, with the most basic pool of robust aggregators that are not designed to address non-iid settings, shows a satisfactory level of robustness even under heterogencous data settings.
Combination strategies of aggregators. To further validate the effectiveness of RobustTailor, we compare it with other three strategies combining aggregators: 1) MixTailor (Ramezani-Kebrya et al., 2022): randomly choose an aggregator; 2) Averago: average of the aggregation results from multiple aggrogation rules; 3) Closest: choose an aggregation result that is closest lo g from multiple aggregation rules. Fig. 13 demonstrate that RobustTailon always outperforms these three combinations of aggregators, which further indicates that RobustTailos select proper aggregator out at cach iteration.
K Computational Complexity
The computational complexity bound depends on the simulation of the inner loop (including simulation rounds K, aggregator set. A, and attack set F) and problem dimensions of the outer loop (including number
26

---

Published in Transactions on Machine Learning Research (02/2024)
JE Ociood
100
Comed
(a) MNIST, c c- 0.5
(b) MNIST, c= 100
(c) MNNIST AttackTaillr
03
Fora offacor
Comed
Lerred
32
(d) FMNIST, c= 0.1
(e) FMNIST, c= 100
(I) FMNIST, AttackTailor
Esou Collefller
Eor officor
MMMMM
Calradd
(g) CIFARIO, == 0.1
(h) CIFARIO, é 100
(i) CIFARIO, AttackTailor
Figure 10: iid setting on three datasets. RobustTailoo includes Krum and Comed. AtlackTailos Tailor includes 0.1/0.5 and 100.
2.8
00
02
22
4.5
1.3
(a) 0.5
(b) 100
(c) AttackTailor
Figure 11: 4 Byzantines. There are 12 total workers including f = Byzantine workers.
of clients n and the dimension gradients). We show the theoretical analysis below and utilize empirical results prove that is worth to trade little more computational complexity for great robust model.
Theorctical analysis We first analyze the time complexity of RobustTaailo per simulation round. The computational cost of Mobuustaailon is influenced by the server's aggregation rules and the adversary's 's attacks. If n clients submit. d-dimensional vectors, Krum's expected time complexity is (n²d) (Blanchard et al., 2017) and Comed's is O(nd) (Pillutla et al., 2022).
For more fine- ane-gedi complexity analysis, the complexity depends linearly on M and S, but they are not inherently sequential, their dependency can potentially be parallelized away. Suppose (Ti,... (.....TTs) denote
27

---

Published in Transactions on Machine Learning Research (02/2024)
JE Ocrodd
Comed
(a) = 0.1, c= 0.5
(b) / = 0.1, = 100
(c) μ = 0.1, AttackTaiis
22-
68
Fira offotrr
Coned
13
34
(d) μ = 0.5, c = 0.5
(e) At = 0.5, c = 100
(f) = 0.5, AttackTailor
500
22
04-
For offiior
Cooed
3.5
0.0
Ramscor
Tarknng
(g) μ 0.9, c = 0.5
(h) / = 0.9, c= 100
(1) = 0.9, AttackTailor
Figure 12: Non-iid setting. Larger μ means higher heterogeneous degree.
8.5
3.0
00
02
34
26
12
14
7.
3.5
1.0
(a) 0.5
(b) 100
(c) AttackTailorr
Figure 13: Combination strategies of aggregators.
the number of elementary operations to run each aggregation rule within the set of M aggregation rules. The worsl-case runtime complexity of Cobbsttaailo per simulation round is determined by max4e[ag] T.. However, the average complexity per round is the expected value of the number of elementary operations where the expectation is over the distribution of how likely each aggregator is chosen during simulation, which can be estimated empirically. Let us use Pú to denote the probability of choosing A,, The average complexity per round given by 7 -M Tip-- Finally, the overall time complexity of Robust Tailor with K simulation rounds given by O(M + SYTK).
Morcover, the number of elementary operations for simulation can be much smaller than applying the actual aggregator the model during training assuming the size of the public data is very small, which is typically
28

---

Published in Transactions on Machine Learning Research (02/2024)
the case in practice(Zhao al., 2018; Yoshida et al., 2020). Note that algorithm just adds computation complexity to the server while all clients remain the same cost based on their models and datasets. Therefore, it is worthwhile to trade slightly longer training time for a significantly improved training procedure w.r.t. robustness.
Empirical results We show computation costs and accuracy for different aggregation rules after rumning 15k iterations in reality, whose results are also shown in Fig. 1. We can see that RobustTailor still maintains stable and high accuracy when facing powerful advensary although it needs more computation time. However, Krum cannot reach high accuracy and Comed shows a very unstable performance with lots of iluctuations when facing a strong adversary with AttackTailor. We note that compared with undesirable models of Krum and Comod, Robust improves accuracy and stability drastically at the cost of slightly increased training time.
Table 2: Computational complexity based on MNIST after running 15k iterations
Aggregator
Time
Accuracy
Omniscient
34 min
96.63%
RobustTailor
96 min
85.87%
Krum
37 min
82.13%
Comed
52 min
90.74%
(a) 0.5
Aggregator
Time
Accuracy
Omniscient
34 min
96.63%
RobustTaiilr
96 min
92.03%
Krum
37 min
94.74%
Comed
52 min
60.39%-88.80%%
(b) 100
Aggregator
Time
Accuracy
Omniscient
34 min
96.63%
RobustTailor
190 min
99.72%
Krum
90 min
80.98%
Comed
121 min
71.34%-89.775
(c) AaiaokTailor
29


Reviews:
[{"review_id": "fdxLEa2kt8", "reviewer": "Reviewer_sisA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written with solid experiments.\nI feel that the assumption of a public subset of the dataset hurts privacy; I am not sure some supporting papers is good enough for that claim. I do not completely get the second paragraph in 4.1 - would few trusted clients not results in estimations that are far from ground truth?\nRecently there have been attacks on no-regret algorithm - would it be better to try those rather than the simple thing done in Section 6.2?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose being robust to an attack on federated learning where a subset of clients are compromised. The problem is formulated as a game where both players use a no-regret learning algorithm to converge to the NE. The gradients are simulated, as the defender does not know the true updates of compromised clients.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please revise 4.1 to be more detailed and precise.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SxacJAUPF8", "reviewer": "Reviewer_3KTv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The study of an important problem, i.e., securing FL against training time adversaries, is a major vulnerability of FL.\n2. The formulation of robust FL as a mixed Nash problem is interesting, and the proposed RobustTailor framework is novel.\n3. Theoretical analysis is provided for the convergence of RobustTailor.\n\nWeaknesses:\n1. The assumptions are somewhat hard to follow. I.e., there seem to be multiple strong assumptions were made here and there, but is still not clear though: what is an informed adversary, why the server knows some honest clients, what is the goal of a Byzantine adversary, why backdoor attacks were not considered, and so on. And the differences of this threat model to other existing works. \n\n2. The proposed RobustTailor method does not always outperform existing works like Krum or Comed. In Figure 1(a), Figure 9 (c) Comed works well at early training iterations but is very unstable at a later stage, but this seems to be fixable.  In Figure 1(b)/1(d), Figure 4(b), Figure 9(b) Krum works better,  \n\n3. Why RobustTailor work on unknown attacks? What about backdoor attacks, which are also one type of training time attack? \n\n4. What is the difference between this work and GFA[1]? And why GFA is not compared in the experiments?\n\n[1] Tahanian E, Amouei M, Fateh H, et al. A game-theoretic approach for robust federated learning[J]. International Journal of Engineering, 2021, 34(4): 832-842.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of robust federated learning (FL) in the presence of an informed training-time adversary. The idea is to formulate the problem as a mixed Nash problem where different robust aggregation rules and potential attacks compete with each other under a minimax framework to achieve robustness and better convergence. Certain assumptions were made to make it work, for example, the server knows the (estimate) of the true gradient update (knowing a subset of honest clients). A RobustTailor aggregation algorithm was proposed to solve the mixed Nash problem with good theoretical convergence guarantees. A set of experiments were conducted to valid the effectiveness of RobustTailor.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- A clearer definition of the threat model and its difference to existing works. A summary/comparison table would be better.\n- Why the methods are more stable under larger $\\epsilon$.\n- Add a detailed description of how Algorithm 1 works, i.e., the meaning/understanding of each step.\n- Explain where the improvement comes from.\n- Experiments with large models like ViT and high-resolution datasets like ImageNet (subset).\n- Experiments against backdoor attacks like DBA.", "broader_impact_concerns": "No ethical concerns are identified for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mAYniI1FHq", "reviewer": "Reviewer_BVaL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The prospective of combining multiple \"weak\" robust aggregator and optimizing the strategy to choosing them is interesting and novel. \n2. The paper is well-written. The description of the setting, problem formulation, and the algorithm is very clear. \n3. The author provide valid convergence guarantee of the algorithm. \n4. The experiments include a wide range of FL settings: partial participation, dynamic number of attackers, partial knowledge, imperfect public dataset, etc. \n\n**Weaknesses**\n\n1. This paper assume that the adversary will randomly draw one kind of attack from F, i.e., the pool of possible attack methods, with probability distribution q. Although the strategy is mixed, instead of deterministic, it is (implicitly) assumed that the adversary will use the same type of attack for all Byzantine clients. However in real scenarios, there could be multiple types of attacks simultaneously. If no aggregation rule is immune to all attack algorithms, it is skeptical whether a random choice of them can resist this hybrid of attack, especially given the fact that the server can only choose one aggregation rule at each round. \n2. The novelty of this paper is the way to combine multiple weak aggregator. However, the author only compare it with very simple baselines: Krum and CoMed. The author should also compare the proposed method with other combination of weak aggregator. To name a few: \n   2.1. MixTailor [1]: randomly choose an aggregator. \n   2.2. Average of the aggregation results from multiple aggregation rules. \n   2.3. Choose an aggregation result that is closer to $\\tilde{g}^*$ from multiple aggregation rules. \n3. Moreover, since RobustTailor uses public data, it should be compared to the baseline AGRs that use public data. For instance, Zeno [2], FLTrust [3].\n\n**Minor Questions**\n\nWhen using a clean public dataset, it is claimed in some works, e.g. [3], that n >= 2f + 1 is not necessary anymore. Is it possible to get rid of this assumption for your algorithm? This is an open question and does not change my decision to this paper. \n\n\n\nReferences. \n\n[1] Ali Ramezani-Kebrya, Iman Tabrizian, Fartash Faghri, Petar Popovski. MixTailor: Mixed Gradient Aggregation for Robust Learning Against Tailored Attacks. Trans. Mach. Learn. Res. 2022 (2022)\n\n[2] Cong Xie, Sanmi Koyejo, Indranil Gupta. Zeno: Distributed Stochastic Gradient Descent with Suspicion-based Fault-tolerance. ICML 2019. \n\n[3] Xiaoyu Cao, Minghong Fang, Jia Liu, Neil Zhenqiang Gong. FLTrust: Byzantine-robust Federated Learning via Trust Bootstrapping. NDSS 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper tackles the Byzantine-robustness problem in federated learning. Instead of designing one robust aggregation rule that is robust to all attackers, this paper studies how to \"ensemble\" multiple imperfect robust aggregation rules, where each of them is only immune to a subset of attacks. The proposed algorithm has theoretical robustness guarantee, and is tested under extensive experiment settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Novelty of game between server and adversary (Section 3.1). Although this perspective is novel to a certain extent, here are some other works using this idea [1, 2]. If the authors insist that this work is the first of its kind, I think a detailed comparison to these related papers can be helpful to support such a claim. \n- Extend the experiments in (1) hybrid attacks and (2) more baselines, according to my comments in weaknesses. \n\n[1] Ali Ramezani-Kebrya, Iman Tabrizian, Fartash Faghri, Petar Popovski. MixTailor: Mixed Gradient Aggregation for Robust Learning Against Tailored Attacks. Trans. Mach. Learn. Res. 2022 (2022)\n\n[2] E. Tahanian, M. Amouei, H. Fateh, M. Rezvani. A Game-theoretic Approach for Robust Federated Learning. International Journal of Engineering. 2021. \n\n\n\n**Minor Changes**: \n\n1. Page 1, paragraph 3, line 2: the the -> the\n2. Page 3 paragraph \"adversary's goal\", line 3: double commas", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a game-theoretical framework to enhance the robustness of federated learning to malicious clients by combining multiple aggregation rules dynamically. The proposed approach is validated by both theoretical analysis and numerical experiments. Overall, it is an interesting addition to the literature on robustness in federated learning, which has mostly considered the use of a single robust aggregation rule.
Decision: Accept (poster)