Learning to Attack Federated Learning: A Model-based Reinforcement Learning Attack Framework
Authors: Henger Li
Xiaolin Sun
Zizhan Zheng
Venue: NeurIPS 2022 Accept (2022)
Keywords: Federated Learning
Adversarial Attacks
Reinforcement Learning

Abstract: We propose a model-based reinforcement learning framework to derive untargeted poisoning attacks against federated learning (FL) systems. Our framework first approximates the distribution of the clients' aggregated data using model updates from the server. The learned distribution is then used to build a simulator of the FL environment, which is utilized to learn an adaptive attack policy through reinforcement learning. Our framework is capable of learning strong attacks automatically even when the server adopts a robust aggregation rule. We further derive an upper bound on the attacker's performance loss due to inaccurate distribution estimation. Experimental results on real-world datasets demonstrate that the proposed attack framework significantly outperforms state-of-the-art poisoning attacks. This indicates the importance of developing adaptive defenses for FL systems.

Learning to Attack Federated Learning: A Model-based Reinforcement Learning Attack Framework
Henger Li, Xiaolin Sun, and Zizhan Zheng
Department of Computer Science
Tulane University
New Orleans. LA 70118
[h1i30, xsun12, zzheng3 @tulane.
Abstract
We propose model-based reinforcement learning framework to derive untargeted poisoning attacks against federated learning (FL) systems. Our framework first approximates the distribution of the clients' aggregated data using model updates from the server. The learned distribution is then used to build a simulator the FL environment, which is utilized to learn an adaptive attack policy through reinforce- ment learning. Our framework capable of learning strong attacks automatically even when the server adopts robust aggregation rule. We further derive an upper bound on the attacker's s performance loss due to inaccurate distribution estima- tion. Experimental results on real-world datasets demonstrate that the proposed attack framework significantly outperforms state-of-the-art poisoning attacks. This indicates the importance of developing adaptive defenses for FL systems.
Introduction
Federated learning (FL) is a promising machine learning framework that allows multiple devices with private data to jointly train a learning model (coordinated by server) without sharing their local data. It has recently been applied consumer digital products [34], credit risk prediction [1], drug discovery [35], and digital health [40]. However, federated learning systems are vulnerable to adversarial attacks [32] such as model poisoning attacks [7, 52, 15, 4], data poisoning attacks [5, 18, 20), and inference attacks [36, 22, 60]. To this end, various robust aggregation rules such as coordinate-wise median [54], trimmed mean [54], Krum [8], norm clipping [48], geometric median [38], and FLTrust [10] have been proposed. However, these defenses are mainly evaluated against manually crafted myopic attack policies [44]. Their robustness in the face of advanced attacks remains unknown.
Due the distributed nature FL systems, a malicious device typically has limited knowledge about benign devices and system dynamics. To fully reveal the vulnerabilities of FL systems, it is therefore crucial develop strong attacks that can best utilize the limited global knowledge. In this work, we take a first step in this direction by considering the white-box attack setting where the attacker has some global knowledge about the FL. system and the server's algorithm, but has no access to the private data of benign devices, reasonable assumption for real-world FL systems. To derive strong adaptive attacks, we propose to leverage the power model-based reinforcement learning (RL) by integrating distribution learning and policy learning. A key observation of our approach is that although accurate information about individual devices can be hard to obtain FL, it is often possible infer their aggregated data distribution from publicly available model updates, which
$ Equal contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

is sufficient to derive strong attacks. In particular, the set of malicious devices first cooperatively estimate the aggregated data distribution through gradient inversion [22, 60]. The learned distribution is then used to build simulator of the FL environment, which is utilized to derive an adaptive attack policy through reinforcement learning. We focus on untargeted model poisoning in this work, where the malicious devices aim to reduce the accuracy of the global model as much as possible by sending crafted gradient information to the server. However, our proposed framework can potentially be applied to other types of attacks in both the white-box and the more challenging black-box settings,
Our model- based approach distinguishes from existing work on reinforcement learning based ad- versarial attacks against machine learning systems [47, 58, 59]. In particular, we consider a more realistic threat model where the attacker might not always be selected due to subsampling nor does have prior information about the distribution of the aggregated data. The attackers need to efficiently learn the distribution along the federated learning process real time. In contrast, previous works typically assume more powerful attackers that can attack at any time and have full knowledge about the environment. Thus, they typically adopt a purely model-free approach, which is infeasíble in attacking FL systems due to the large number of samples needed to be effective.
Our contributions. We advance the state-of-the-art in the following aspects. First, we propose a novel reinforcement learning attack framework against federated learning systems by integrating distribution learning and policy learning. Second, we theoretically quantify the effect of inaccurate distribution learning and heterogeneous local data distributions on the optimal attack performance. Third, our experiments on real-world datasets demonstrate that our RL-based attack method consistently outperforms existing model poisoning attacks [7, 15, 52] and significantly reduces the global model aceuracy even when robust aggregation rules are applied. These findings indicate the importance of developing adaptive defenses for FL systems.
2
Related Work
Poisoning attacks and defenses. To compromise the integrity of the target model in federated learning, both targeted poisoning attacks [7, 4. 5] that aim to misclassify a specific set of inputs. and untargeted attacks [15, 52, 43] aiming to reduce the global model accuracy have been proposed. Existing approaches typically adopt a eecurstics-bbsed method [52]) or optimize a myopic goal [15, 43]), and are usually sub-optimal, especially when a robust aggregation rule is adopted. Further, they often require access to benign agents' local updates or the accurate global model parameters of the next round [52, 15]) to make significant attack impact. In contrast, our reinforcement learning based attack requires less global knowledge and targets a long-term attack goal.
Various defenses have been proposed for model poisoning attacks including abbust-ggreegation -based approaches and detection- based approaches. The former includes dimension filtering that considers each dimension of local updates separately [6. 54], client-wise filtering that aims to restrict or even remove the impact of potentially malicious clients [8, 8, 38, 48], and approaches that require the server to have access to a small amount of root data [10]. Our RL-based attack is effective against all these defenses. addition, time-coupled robust aggregation methods [2, 3, 25] that target adaptive attacks and anomaly detection based defenses [28] have been proposed recently. Our approach can potentially be extended to compromise them by explicitly encoding the history information into states or utilizing a recurrent structure.
RL-based attacks. Reinforcement learning has recently been utilized for developing strong at- tacks in various settings, including corrupting training data of online supervised and unsupervised learning [59], manipulating the combinatorial structure of graph data [13], and injecting malicious nodes into a graph [47]. RL-based attacks have also been developed to damage the performance of reinforcement learning itself, by perturbing the reward signals during the training stage [58] and corrupting the state signals received by an agent during the testing stage [57, 45]. However, these methods typically assume that the attacker has access to an accurate MDP model or simulator and has unlimited time for training the attack policy. In contrast, our method first builds a world model by learning a data distribution from the FL model updates and then constructs an approximate simulator for training our attacks. Both distribution learning and policy learning happen when FL. training is ongoing. Further, previous work has mainly focused on attacking a single RL. agent by an external consider in this work.
agent rather than an insider attack in distributed learning environment
2

---

Benign Devices
Distribution Learming
Policy Learning
P P
P
Kir
Server
groell gri
(Pir)
91
With (f)-
Simulatted
Network
Malicious Devices
Figure An overview of the RL-based attack framework against federated learning.
Approach Overview
In this section, we describe the federated learning setting considered in this work, the threat model, and the proposed RL-based attack framework.
Federated learning. We consider an FL. setting that is similar to federated averaging (FedAvg) [33]. The FL system consists of server and K workers (also known as devices or clients) in which each worker has some private data. Let [K] { 2, .... K} denote the set of workers. Coordinated by the server, the of workers cooperate to train machine learning model within epochs by solving the following problem: ming ((0) where ʃ(0) == Σk_1 Ps. (0). Here (.).) the local objective of worker & and pı is the weight assigned to worker A and satisfies pk > 0 and Σ, Pk- = 1. The local objective (0) is usually defined as the empirical risk over worker k's local data with model parameter E O That is, (0) = N, Σッ"
where N₂ is the number of data samples available locally on worker k; E(:;)) is the loss function, and 2jk : (Izk, U5k) is the jth data sample that is drawn i.i.d. from some distribution Pk. It is typical to set pk N
N2.
where N Σx N& is the
total number of data samples across workers.
If all the workers' local data distributions are the same (i.e., Ps. = Pa. for all k E [K]), we call the workers' data are i.i.d.; otherwise, the data non-i..d. We write Pe. the empirical distribution of the Ng data samples drawn from PE. and let P := ER-1 N distribution across workers.
-K= N₂ Ps denote the mixture empirical
The FL algorithm (see Algorithm in Appendix B) works as follows: each time step / > 0, random subset St of size w is uniformly sampled without replacement from the workers set [K] by the server for synchronous aggregation [30]. The process of selecting workers for aggregation
is called subsampling. Let K = w/N denote the subsampling rate, i.c., the ratio of the selected workers number the total number of workers K. Each selected worker E [w] then samples
minibatch by of size B from its local data distribution Pg. The worker then calculates the average local gradient 9 3 Eeeby Vol(0"; 2) and sends the gradient the server. The server then uses an aggregation rule to compute the aggregated gradient g²+1 Aggr(skt ++1 ,.... (+) where k; E St, and updates the global model parameters 0++1 - 0¹ n-g-1 where is the learning rate. The newly updated model parameters 01+1 is then sent the selected workers to perform another FL iteration.
Threat Model. We assume that among the K workers, M(1 & M < K) of them are malicious. Let A denote the set of malicious workers. They are coordinated either by one leading attacker or an external agent. We refer such agent as a leader agent. These attackers are assumed to be fully cooperative and share the same goal of compromising the FL system. We consider untargeted model poisoning attacks in this work where the M cooperative attackers send crafted local updates (7) ke to the server in order maximize the empirical loss f()). the batch size B), and their local data distributions (Px JE (but not the benign workers' local data distributions). We further assume that the attackers obtain information about the server's 's training algorithm (i.e., the white-box attack setting). This information includes the server's 's learning rate 7, the subsampling rate K, the total number of workers K, the aggregation rule Aggr, and the total number of training epochs T.
3

---

RL-based online attack framework. Our attack framework consists of the following three phases.
Distribution learning: The malicious workers first jointly learn an approximation of P from the model updates {0') received from the server, using a gradient inversion based inference attack [19]. Policy learning: The leader agent then builds a simulator of the FL environment using the attackers' local data and the learned distribution. An optimal attack policy is then derived through reinforcement learning using data sampled from the simulator. Note that policy learning can start together with distribution learning and continue after a reasonable distribution learned.
Attack execution: The learned policy is distributed to all the malicious workers to generate attack actions. Note that attack execution can start once initial policy is learned, which be updated during attack execution.
We note that all the three phases happen while the federated learning process is ongoing, thus the lengths of these phases are important hyperparameters to be determined. For example, with more observations, attacker can learn more accurate distribution, which will help obtain better attack policy. However, when the total time window available to attacks is limited, a longer distribution learning phase reduces the attack opportunities in Phase 3. Compared with a purely model-free approach, our model-based approach is more sample efficient, which is especially important for federated learning as malicious worker only attack when it is sampled by the server.
Model-based Reinforcement Learning Attack Framework
In this section, we first formulate the model poisoning attack problem as a Markov decision process (MDP). We then discuss our model-based reinforcement learning attack framework in more details.
4.1 Attackers' problem as a Markov decision process
The attackers' problem is formulated undiscounted MDP, denoted by M A,,,,T, where
S is the state space. Let E {0, 1..... H - 1} denote the index of the attack step and t(T) (0.,. T 1] the corresponding FL epoch when at least one attacker is selected by the server. The state step is defined s == (0*(r) _A(((r) where A((r)) is the set of attackers selected at time t(T), which is shared between all malicious workers.
A AM is the space of the attackers' joint actions where each attacker shares the same action space A. If attacker is selected at t(T), its action a. := g. (((r)+1 Id the local update that attacker i sends to the server at time step t(r). where d is the dimension of the model parameters.
The only action available to an attacker not selected at ((T) is I, indicating that the attacker does not send any information in that step.
T: S x A- + P(S) is the state transition function that represents the probability reaching a state s' E S from the current state S when attackers choose actions ai ,.....: a5g- respectively.
x > Rg0 is the reward function. We define the reward at step T as f((((r+1)) - ((((r)), which is determined by the current state, the next state, and the joint attack r
actions and is shared by all the attackers.
H is the number of attack steps in each episode and have t(11 1) <T
The attackers' goal is to find a joint attack policy π = (T1, TM) that maximizes the expected total rewards over H attack steps, i.c., EIΣH-1 r=]. where n, S - P(AA) denotes a stationary policy of attacker i that maps the state objective is equivalent to finding a joint policy π that maximizes Eggee [f
probability measure over A. Using the definition of r", this
key obstacle to solving the MDP is that both the transition probabilities T and the reward function r depend on the joint empirical distribution across workers {P* [P which is fixed but unknown the attackers. Although model-free reinforcement learning can bypass this difficulty, it requires a large number of samples to be effective, which is infeasible in the online attacking scenario we consider. We therefore adopt model-based reinforcement learning as a principled approach for designing adaptive attacks in the online setting. An important observation is that although the joint empirical distribution {Prs ket is unknown, the attackers can learn an approximation of the mixture

---

distribution P= >=> No Pk. denoted by P. from model updates shared by the server, which is often sufficient to simulate the behavior of benign agents and the server by assuming that each benign agent samples data from P. This gives rise to a new MDP M A, , H) where T" and r' are derived from P. Thus, our proposed model-based reinforcement learning attack framework naturally consists of the distribution learning, policy learning, and attack execution phases.
-k=11 N
4.2 Distribution learning
Initially, the attackers do not perform model-poisoning attacks. Instead, they jointly learn mixture distribution P from the model updates {0'} using gradient inversion based infer- ence attack [19, 60]. Various gradient inversion attacks have been proposed in the literature. In this work, we adapt the inverting gradients (IG) method [19] to distribution learning. The IG method reconstructs data samples by optimiz- ing a loss function based on the angle (i.e., co- sine similarity) between the gradient generated from true data and that from the reconstructed data. The primary goal of IG is to reconstruct the original data samples, which is more ambi- tious than what the attackers need our setting, On the other hand, recent works gradient in- version including IG have focused on the server side, where the true gradients of each individ- ual worker can be easily obtained from model updates. In contrast, the attackers only obtain ap- proximated and aggregated gradient information from consecutive model updates received from the server, due model aggregation and subsampling. Despite these differences, our experiment results show that the P learned using IG can help derive an effective attacker policy (see Figure 4(c)).
Noisy
4
O
2
Denoised
6
2
+
O
2
Noisy
Denoised
Figure 2: Examples of reconstructed images (be- fore and after denoising) for MNIST (upper) and Fashion-MNIST (lower) datasets.
As shown in Algorithm 2 in Appendix B, for each epoch t(T) that at least one attacker is selected, the leader agent obtains the model update from one of the attackers and calculates the batch-level gradient as g" == (0*(--1) - (((+)) (()(((r)) - t(T - 1))). The leader agent then starts with a batch of (randomly generated) dummy data and dummy labels Ddummay. which is updated iteratively by solving the following optimization problem: arg minDaemmmy 1 cos( V Faurmmy (0lermm((((()) (O*r)), >,g") + (A.B)
8 B Σ (s.. (V:)), where cos( B) TAllET is the cosine similarity between two vectors A and B, F.dummyy(0 = B' (xz.)) Daummy E(O: v)), B' is the number of reconstructed data per epoch (the size of the dummy data batch), TV(x) is the total variation [41] of and 33 is fixed parameter. The process terminates after maz_iter iterations, then outputs the updated data as the reconstructed data samples. We observed that although the data samples generated by IG resemble true samples, they contain certain amount of noise as shown in Figure 2, making the learned distribution less representative of the true distribution. To reduce noise, we adapt the method of denoising auto auto-encoder [50]. We utilize the clean data owned by attackers and add Gaussian noise to them to simulate the noise in the reconstructed data samples. The clean data and the synthetic noisy data are then paired to train an autoencoder for denoising, which is then used to remove the noise in reconstructed data samples shown in the figure. The approximated mixture distribution P(r) consists of the reconstructed data up to t(r) and the M attackers' local data. The distribution learning phase starts at the first FL epoch and continues for Tg steps. The learned distribution is shared with all the attackers.
Although we adopt the IG method in this work due to its simplicity, other more recent approaches such the GradInversion method [55] and gradient inversion with trained generative model [23] can be easily incorporated into our framework, which can potentially learn P in more challenging
settings for complex datasets like ImageNet [14], deep networks, and large batch sizes. On the
other hand, we show in the experiments that our RL-based attack trained using attackers' local data only is still effective and surpasses all the baselines, while distribution learning further boosts the attack performance. A detailed discussion on gradient inversion attacks and defenses is provided in Appendix C
5

---

4.3 Policy learning
Once the leader agent obtains the approximated distribution P, it can simulate the behavior of the server and that of normal workers. In particular, to simulate the behavior of a normal worker in each FL training step, a minibatch of size B is i.i.d. sampled from P. With experiences sampled from the simulated environment, the leader agent can learn a joint attack policy that maximizes the empirical loss using a state-of-the-an (deep) reinforcement learning algorithm, e.g., TD3 [17] or PPO [42].
Note that the leader agent does not need wait until good distribution has been learned start training the policy. Instead, policy learning can start together with distribution learning. Initially, the leader agent uses the attackers' local data train the policy, which will be continuously updated while more data samples are being generated. To reduce the training overhead, we assume that all the malicious workers share the same attack policy (i.e., IT = m2 achieves good attack performance in our experiments. Extension to general joint policies will be considered in our future work.
TM) in this work, which
When we train a small neural network with federated learning, it is natural to use (0*()),AAt)) as the state, and the crafted gradient g((r) as the action. When we use the federated learning system to train a large neural network, however, this approach does not scale as results in an extremely large search space that requires both large runtime memory and long training time, which is usually prohibitive. To solve this problem, we propose compress the state and action spaces for high-dimensional models as follows.
To compress the state space, we first observe that as the set of attackers are fully cooperative and share the same policy, there is no need to distinguish them in the state when the server does not track the behavior of individual workers. Thus, we replace (((r) by the number of attackers sampled in t(r), denoted by m²)r), Further, instead of using the entire set of model parameters (*(r) in the state, the parameters of the last hidden layer of the current neural network model is used. This is because the last hidden layer passes on values to the output layer and typically carries information about important features of the model [46]. Note that the approximated state is used to define the policy only. The true state is still the full FL. model that determines transition probabilities and rewards.
It is more challenging to compress the action space. We observe that policy that manipulates the model parameters of the last hidden layer only works well for certain aggregation rules such as Krum and coordinate- wise median. However, it becomes less effective for stronger defenses such as coordinate-winn wiie median with clipping. Given that it challenging to identify small subset of model parameters that can lead to most damage to model accuracy when manipulated, we adopt a different approach in this work.
The main idea is to search for a model update direction that can lead to a large empirical loss using gradient ascent, with its parameters identified by reinforcement learning. To this end, define the local search objective as [(0)= = (1 F()) + X cos (0)()) (.g()(())) where F(0) E 2]] models the empirical loss and ((((T)) = Ey_ pu-pl (( 2]] captures the average update direction from normal devices, both are estimated from P. The second term in L(0) is used to control the deviation of the model update from the normal direction (measured by the cosine similarity) so that the adversarial input cannot be easily identified by the server. The parameter A E [0, 1] is used to balance the loss and the deviation. To solve the problem, we start with 0 = g((r) and generate G' trajectories, where each trajectory involves F model updates. Each model update involves single gradient ascent step using minibatch of size B sampled from P. Let 0g denote the new model parameters found by the k-th trajectory after E updates. The update direction is then set as & >" Os and each attacker's 's action in ((T) is computed as (+)++1 ((0(()) & > 0x). The scaling factor y 3 0 is used to control the magnitude of the crafted gradient, which is needed as most robust aggregation rules apply certain type of filtering rule to mitigate the effect of malicious attacks. We assume that G is fixed while 7, E, À are parameters to be learned by reinforcement learning. Thus, the action space for the attackers' MDP becomes 3-dimensional real space.
4.4 Attack execution
Both distribution learning and policy learning can start from the first epoch of federating learning and continue while federating learning is ongoing. The simulated environment is updated when

---

new estimated distribution P is learned. Although the attackers may choose to start attacking during distribution learning, we observe that this can blur the gradient information and make distribution learning less accurate. Thus, we assume that each attacker starts attacking once the distribution learning phase finished, and applies the latest learned attack policy during the remaining epochs of the federated learning process. During attack execution, each selected attacker first notifies other attackers so that every one knows the number of attackers that are sampled in that epoch. Each selected attacker then generates crafted gradient according to the process described above with the parameters obtained from the latest learned policy.
The attackers' total training time (including distribution learning and policy learning) should be significantly less than the total FL training time so that the attackers have time to execute the attacks. In real-world FL training, the server usually must wait for some time (typically ranging from 1 minute to 10 minutes) before it receives responses from the clients [53, 9, 24]. In contrast, the leader agent does not incur such time cost in training attackers' policies using a simulated FL. environment. Therefore, an epoch in policy learning typically much shorter than an FL. epoch, making possible to train the attack policy with a large number of episodes. In addition, the leader agent is usually equipped with GPUs, or other parallel computing facilities and can run multiple training episodes in parallel [11]. We compare the actual running time of our RL-based attack against different defenses in our experiment setting in Appendix E.2.
Impact of Inaccurate Distribution Learning and Data Heterogeneity
Our model-based RL attack employs the estimated data distribution P simulate the behavior of benign workers, which can suffer from two types of errors. First, P can be far away from the true mixture distribution P due to inaccurate distribution learning. Second, benign workers may vary in their local data distributions Pk. which cannot be fully captured by a single mixture distribution. In this section, we study how the attack performance affected by these two factors, which provides insights into properly distributing resources between the three phases of our attack method.
Our analysis is adapted from recent works that study the impact of model inaccuracy on the per- formance of model-based reinforcement learning [56, 31, 58] by addressing two new challenges. First, we need to establish the connection between the inaccuracy in data distribution and the inaccuracy in the corresponding MDP as both the reward function and the transition dynamics depend on P. Second, although there are different ways to measure the distance between two models [56], it makes more sense to use the 1-Wasserstein distance [49] to measure the distance between two data distributions. This, however, requires bounding the Lipschitz constant of the optimal value function [56]. Although this is a challenging task for general RI. tasks, we are able to show that this is indeed the case in our setting under the following assumptions. The first assumption models the inaccuracy of distribution learning as well as the heterogeneity of benign workers' local data.
Assumption 1. (i(P,P) < 8 for any benign worker k, where Wa(+s-) is the 1- I-Wasserstein distance [49].
We further need the following standard assumptions on the loss function.
Assumption 2. Let ZZ denote the domain of data samples across all the workers. For any 81, 82 and 21, 22 E Z, the loss function {: S x Z - R satisfies:
1. [[(s), 21) ((s2, 22) ()), 21) (82, 22)12
(Lipschitz continuity w.E.t and z);
2. [V 21) V.l(s1, 22 & L2 2212 (Lipschitz smoothness w.r.t. 2):
3. (152,2)) > (1) + (.
$1) + S2 s1ll2 (strong convexity w.r.t. s);
4. ((32,2)) 21) < ((11, + ((,((,,
91) + =1s2 31 (strong smoothness w.r.t. s);
5. E((s-) twice continuously differentiable with respect to s.
where [[(1, 21) (82, 22)1 == |ss
- -2 +21 =2 2 For simplicity, further make the following assumption on the FL environment, although our analysis can be readily applied to more general settings.
Assumption 3. The server adopts FedAvg without subsampling (w K). All workers have same amount of data (pk te) and the local minibatch size B 1. In each epoch of federated learning,
7

---

each normal worker's local minibatch is sampled independently from the local empirical data distribution Pk.
Since no subsampling is considered in this section, with a slight abuse of notation, we let index t denote both an attack step and the corresponding FL epoch. M (S, A, r,, r, H) denote the true MDP for the attackers, and M = A, r', H) the simulated MDP when the local distribution of any benign worker is estimated as P. The following theorem captures the attack performance loss due to inaccurate distribution learning (see Appendix D for the proof).
Theorem 1.
Let JM([T) :- BT,,T.ng [Et-g' r(s', a², 1 denote the expected return over H atlack steps under M, policy x and initial state distribution pe- Let x and 2* be the optimal policies for M and M respectively, with the same initial state distribution μo. Then,
IJM((**) Jm((**) 2Ht + Lo + 2L},
where e K-M is the fraction of benign nodes, Le & emax{[1 naa,,1 n/1}.
+ LKF), and KF &
In practice, the learning rate is typically small enough so that max - n/813 < 1. In this case, T. is bounded by L(1+KF) L(l+e) K-
I- Therefore, we have I ( JM((#*)
O( To ensure convergence, we typically have
O V [39], thus [T*)
[M = O 5 This result clearly shows how the loss of attack performance depends on the fraction of benign nodes, the inaccuracy of distribution learning, and the time horizon.
6 Experiments
In this section, we compare our RL-based attack with state-of-the-ate model poisoning attacks on real-world datasets. Our code is available https: com htt////eeeeeeee//eaeaii.goooo Attack-Fedde Federated -earted.-LLainng..
6.1 Experiment setup
Datasets. We conduct extensive experiments on four real-world datasets: MNIST [27], Fashion- MNIST [51]. EMNIST [12], and CIFAR-10 [26]. Due to space limitation, experiment results for -ashion-MNIST. EMNIST, and CIFAR-10 are provided in Appendix E. For the i.i.d. setting, we randomly split the dataset into K groups, each of which consists of the same number of training samples. For the non-i.i.d setting, we follow the method of [15] to quantify the heterogeneity of local data distribution across clients. Suppose there are C' classes in the dataset, c.g., C= for the MNIST and Fashion-MNIST datasets. We evenly split the worker devices into C groups (with the M attackers evenly distributed across the C' groups), where each group is assigned I/C of training samples as follows. training instance with label e assigned to the e-th group with probability q > 1/C and to every other group with probability (1 111-)) (C 1). Within each group, instances are evenly distributed. A higher q indicates higher non-i.id. degree. We set q = 0.5 the default non-i.i.d. degree. To demonstrate the power of distribution learning, we assume that the set of attackers share m true data points sampled from the training instances assigned to them. We set v 200 as the default value for MNIST.
Baselines. We compare our RL-based attack (RL) with no attack (NA), and the state-of-the-arr model poisoning FL attack methods: explicit boosting (EB) [7], inner product manipulation (IPM) [52]. and local model poisoning attack (LMP) [15]. IPM manipulates the attackers' gradients that the inner product between the aggregation result and the true gradient is negative. This requires access to the average of normal workers' gradients in each FL epoch, which usually unavailable in practice. LMP generates myopic attacks by solving an optimization problem in each FL epoch. In addition to the server's aggregation rule, also requires access to normal workers' local models. Although LMP with partial knowledge is also presented in [15], it performs substantially worse than the full knowledge case when the server uses the coordinate- wise median defense. We compare the RL-based attack with the more powerful full knowledge LMP below.
8

---

MARST ROLEE
PRYST CRod
ClpZra Medaan
Oheel lct
Olabd Node bes
62.0
1000
45.0 Frectee
60.0
682
100
Figure 3: A comparison of global model accuracy under Krum and clipping median for both i.i.d. data and non-L.id. data, All parameters are set default.
1.2
ST Medio non.ld
Grgel
6.0
23
0.0
0.1
03
05
56
LOK
208
R1M
(a)
(b)
Figure 4: Attack performance on MNIST under (a) different number of attackers; (b) different non-iid degrees; (c) RL with and without distribution learning and RL. (zero initial data for policy learning;; and (d) different policy learning lengths. Non-iid degree q = 0.5 (a) and q = 0.3 (c) and (d). Other parameters are set as default.
We consider four representative robust aggregation rules of different types [44]: Krum [8], geometric median [38], both of which apply client-wise filtering to model updates, coordinate-wise median [54], which adopts timension-wist filtering, and FI.Trust [10], which requires the server have access to small amount of root data. In the experiments, we actually consider an extension of the vanilla coordinate-wirl wise median where a norm clipping [48] step first applied. This gives a more powerful defense as we observed in experiments. We set the default norm threshold to 2.
Default FL and RL settings. We adopt the following default parameters for the FL models: number of total workers = 100, number of attackers = 20, learning rate = 0.01, subsampling rate = 10%, the number of total FL epochs 1, 000. For our RL-based attack, both the distribution learning and policy learning phase start at the first FL. epoch. The former ends at the 100th FL epoch when RL-baaedd attack starts (all other attacks start at epoch 0). The policy learning phase ends at the 400th epoch. Since both the action space and state space are continuous in our setting, we choose the state-of-the-ar Twin Delayed DDPG (TD3) [17] and Proximal Policy Optimization (PPO) [42] algorithms for training the attack policy in our experiments and find that TD3 gives better results in most cases. Below we report the results for TD3. We fix the initial model and the random seeds for subsampling and local data sampling for fair comparisons. See Appendix E for details of the datasets, experimental setups, and additional results.
6.2 Attack performance
Figure 3 shows how the test accuracy of the global model varies over FL epochs under different attacks when the server uses Krum and clipping median as the aggregation rule, respectively. Results for geometric median and FLTrust are provided in Appendix E.2. We observe that our RL-based attack performs significantly better in all the settings, despite the fact that IPM and LMP use the model updates of normal clients while RL does not. Note that for Krum, RL-based attack quickly drives the global model to poor state (~10% accuracy) once the attack starts at epoch 100 under both i.i.d. and non-i.i no..i.i.d. local data distributions. Attacks become harder under clipping median due to the norm clipping but our RL-based attack still reduces global model accuracy to around 50% on average. This mainly because targets long-term return while all other baselines are myopic. For example, in Figure 3(c), the global model accuracy drops significantly under all the attacks when five malicious devices are sampled around epoch 200. After that, the RL method keeps the accuracy at a low level, while other baselines" accuracy rebounds rapidly.

---

6.3 Ablation studies
Impact of the number of attackers. Previous studies on untargeted model poisoning in federated learning typically assume a relatively large fraction of attackers. For example, the default setting is 20% in [15] and 40% in [52]. Figure 4(a) shows that our RL-based attack obtains superb performance even when the number of attackers (among 100 total clients) is as low as 8. In contrast, neither IPM nor LMP obtains meaningful attack performance even with 10 attackers. For attackers, none the baselines including the RL--based attack can cause significant damage to the FL system.
Impact of non-i..d. degree. Figure 4(b) shows the impact of data heterogeneity on attack performace. We use 5 different random seeds for all attacks and show the error bars. We observe that all the baselines obtain similar performance under different non-i.i.d. degrees and the impact of randomness in the testing environment on their performance is limited. On the other hand, we observe that the RL policies for q = 0.1 and q 0.5 exhibit large variances, but even the worst-case performance of our attack outperforms the best cases of all the baselines. For 0.3, the RL-based attack can always lead to model with a very high loss so that the model accuracy stays at a low level and close to a constant, which explains the observed low variance in model accuracy. We expect that the variation across different RL. policies in part because the attackers always use the latest trained policy for attack execution, which does not necessarily give the best performance among all the intermediate policies trained (see also Figure 4(d)).
Importance of distribution learning. Figure 4(c) compares the global model accuracy of RL-based attack with distribution learning (RL) and that without distribution learning (RL w/o DL). We observe that in both cases, the model accuracy decreases dramatically after the attack starts at FL epoch 100. Further. the accuracy RL w/o DL slightly increases up to 20%, while the accuracy of RL stays below 10%, which is consistent with our expectation that distribution learning allows the attackers to learn better attack policy. Figure 4(c) also shows the attack performance RLg, variant of the RL-based attack where the attackers only have 200 unlabeled true images used to train the denoising autoencoders, thus completely relying on distribution learning to generate labeled samples needed for policy learning. Compared with the baseline results in Figure 4(b) (g 0.3), we observe that RLo still outperforms other baseline methods, further indicating the power of distribution learning. On the other hand, the fact that RL w/o DL surpasses all the baselines indicates that our approach is still applicable even when distribution learning becomes less effective in the presence of strong defense against gradient inversion.
Impact of training length on policy learning. Figure 4(d) shows how the global model loss the end of an FL training episode (in the simulated environment) varies over the RL policy training steps. We observe that longer training usually provides a better attack policy, although the training process is not stable. To fix this, one approach is to set up separate testing environment to identify best trained policies. As mentioned above, our RL-based attack achieves promising performance even when the attackers always use the latest policy obtained during policy learning.
7 Conclusion
We propose a new approach for developing non-myopic attacks that can effectively compromise FL systems even with advanced defense mechanisms applied, by utilizing model-based reinforcement learning as a principled approach. While we focus on untargeted model poisoning against FL systems in this paper, our attack framework can be extended to targeted attacks (e.g.. backdoor attacks) and to objectives beyond global model accuracy (e.g., fairness across clients [37, 29I). Further, our attack framework can be integrated with meta-lcarning [16, 21] to generalize the learned policy different training tasks and develop black-box attacks. Another direction is to investigate novel methods to defend our adaptive attack methods. One possible solution would be to dynamically adjust FL parameters such as the subsampling rate or the aggregation rule.
8 Acknowledgment
This work has been funded in part by NSF grants CNS-1816495 and CNS-2146548 and Tulane University Jurist Center for Artificial Intelligence. We thank the anonymous reviewers for their valuable and insightful feedback.
10

---

References
[1] Utilization of FATE Risk Management of Credit in Small and Micro Enterprises.
https://www.fedatt wttp//aass//tiiiaiiooooof.ff
credit-in-small-anne anteraa---saniiio--eeeeeetpresss
[2] Dan Alistarh, Zeyuan Allen-Zhu, and Jenry Li. Byzantine stochastic gradient descent. Advances in Neural Information Processing Systems(Neur/PS), 31, 2018.
[3] Zeyuan Allen-Zhu, Faeze Fbrahimianghazani, Jerry Li, and Dan Alistarh. Byzantine-resilient non-convex stochastic gradient descent. In International Conference on Learning Representa- fions(ICLR), 2020.
[4] Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statisties (AISTATS), 2020.
[5] Gilad Baruch, Moran Baruch, and Yoav Goldberg. A little is enough: Circumventing defenses for distributed learning. In Advances in Neural Information Processing Systems(NeurIPS), 2019.
[6] Jeremy Bernstein, Jiawei Zhao, Kamyar Azizzadenesheli, and Anima Anandkumar. signsgd with majority vote is communication efficient and fault tolerant. International Conference on Learning Representatiooss//LLR)) 2018.
[7] Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing federated learning through adversarial lens. In International Conference on Machine Learn- ing(ICML), 2019.
[8] Peva Blanchard, Rachid Guerraoui, Julien Stainer, et al. Machine learning with adversaries: Byzantine tolerant gradient descent. In Advances in Neural Information Processing Sys- lems(NeurIPS), 2017.
[9] Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloé Kiddon, Jakub Konečný, Stefano Mazzocchi, Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards federated learning at scale: System design. In Proceedings of Machine Learning and Systems, 2019.
[10] Xiaoyu Cao, Minghong Fang, Jia Liu, and Neil Zhengiang Gong. Fltrust: Byzantine-robust federated learning via trust bootstrapping. In NDSS, 2021.
[11] Alfredo V Clemente, Humberto N Castejón, and Arjun Chandra. Efficient parallel methods for deep reinforcement learning. arXiv preprint arxiv: /705.0465.42 2017.
[12] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. International Joint Conference on Neural Networks (IJCNN), 2017.
[13] Hanjun Dai, Hui Li, Tian Tian, Xin Huang, Lin Wang, Jun Zhu, and Le Song. Adversarial attack on graph structured data. In International Conference on Machine Learning//CML), 2018.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern SrnegnttionCVPPR)) 2009.
[15] Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Gong. Local model poisoning attacks to byzantine-robust federated learning. In 29th USENIX Security Symposium, 2020.
[16] Chelsea Finn, Picter Abbecl, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In International Conference Machine Learning(ICMLL), 2017.
[17] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference Machine Learning(ICMLL) 2018.
11

---

[18] Clement Fung, Chris JM Yoon, and Ivan Beschastnikh. Mitigating sybils in federated learning poisoning. arXiv preprint 1.08v:880..04885 2018.
[19] Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients how easy break privacy in federated learning? In Advances Neural Information Processing Jystems(Neur/PS/) 2020.
[20] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. CoRR, abs/1708.066333, 2017.
[21] Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised meta-learning for reinforcement learning. arXiv preprint 1eXiv:8806.40000 2018.
[22] Briland Hitaj, Giuseppe Ateniese, and Fernando Perez-Cruz Deep models under the gan: information leakage from collaborative deep learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, 2017.
[23] Jinwoo Jeon, Kangwook Lee, Sewoong Oh, Jungseul Ok, al. Gradient inversion with generative image prior. Advances in Neural Information Processing Systems(NeurlPS), 2021.
[24] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14(1-2): 1-210, 2021.
[25] Sai Praneeth Karimireddy, Lie He, and Martin Jaggi. Learning from history for byzantine robust optimization. In International Conference on Machine Learning(/CML), 2021.
[26] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[27] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 86(11):2278-2324, 1998.
[28] Suyi Li, Yong Cheng, Wei Wang, Yang Liu, and Tianjian Chen. Learning to detect malicious clients for robust federated learning. arXiv preprint urXiv:2002.00211 2020.
[29] Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine LearningICMLL), 2021.
[30] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations(|CLLR)) 2020.
[31] Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model- deep reinforcement learning with theoretical guarantees. In International Conference on Learning Representatioos((CLLR)) 2019.
[32] Lingjuan Lyu, Han Yu, and Qiang Yang. Threats to federated learning: survey, arXiv preprint arXiv:2003.02133, 2020.
[33] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication catiiiiiiiaiiioeffiiitttt learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1272.1282. PMLR, 2017.
[34] Brendan McMahan and Daniel Ramage. Federated Learning: Collaborative Machine Learning without Centralized Training Data. https://ai. googleblog..ooooooooooooooooooo httpps//aat..ogggeelo....oooo. http:///cccmm////////eeeerreeeer html.
[35] Brendan McMahan and Daniel Ramage. Machine Learning Ledger Orchestration For Drug Discovery (MELLODDY). https: http://wwwmellllld mewloddy..d.y.oo eu/.
12

---

[36] Luca Melis, Congzheng Song, Emiliano De Cristofaro. and Vitaly Shmatikov. Exploiting unintended feature leakage collaborative learning. In 2019 IEEE Symposium on Security and Privacy, 2019.
[37] Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. In International Conference on Machine Learning(ICML), 2019.
[38] Krishna Pillutla, Sham M Kakade, and Zaid Harchaoui. Robust aggregation for federated learning. IEEE Transactions on Signal Processing, 70: 70:1142-11544 2022.
[39] Boris Polyak. Introduction to optimization. Optimization Software, 1987.
[40] Nicola Ricke, Jonny Hancox, Wenqi Li, Fausto Milletari, et al. The future of digital health with federated learning. NPJ digital medicine, 3(1):1-7, 2020.
[41] Leonid Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena, 60(1-4):259-2)) 1992.
[42] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1 2017.
[43] Virat Shejwalkar and Amir Houmansadr. Manipulating the byzantine: Optimizing model poisoning attacks and defenses for federated learning. In NDSS, 2021.
[44] Virat Shejwalkar, Amir Houmansadr, Peter Kairouz, and Daniel Ramage. Back to the drawing board: A critical evaluation of poisoning attacks on production federated learning. In IEEE Symposium on Security and Privacy, 2022.
[45] Yanchao Sun, Ruijie Zheng, Yongyuan Liang, and Furong Huang. Who is the strongest enemy? towards optimal and efficient evasion attacks in deep RL. In International Conference Learning Representatiions//CLR, 2022.
[46] Yi Sun, Xiaogang Wang, and Xiaoou Tang. Deep learning face representation from predicting 10,000 classes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition CVPR), 2014.
[47] Yiwei Sun, Suhang Wang, Xianfeng Tang, Tsung- Yu Hsich, and Vasant Honavar. Adversarial attacks on graph neural networks via node injections: A hierarchical reinforcement learning approach. In Proceedings of The Web Conference 2020, 2020.
[48] Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H Brendan McMahan. Can you really backdoor federated learning? arXiv preprint arXiv: 1.07963, 2019.
[49] Leonid Nisonovich Vaserstein. Markov processes over denumerable products of spaces, de- scribing large systems of automata. Problemy Peredachi Informatsii, 5(3):64-72, 1969.
[50] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In International Conference Machine Learning//CML), 2008.
[51] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 1708. 111..1777777777 2017.
[52] Cong Xie, Oluwasanmi Koyejo, and Indranil Gupta. Fall of empires: Breaking byzantine- tolerant sgd by inner product manipulation. In Uncertainty in Artificial Intelligence (UAI), pages 261-270. PMLR, 2020.
[53] Timothy Yang, Galen Andrew, Hubert Fichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, and Françoise Beaufays. Applied federated learning: Improving google keyboard query suggestions. arXiv preprint arXiv:1812.02900) 2018.
[54] Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantine-robus distributed learning: Towards optimal statistical rates. Proceedings of the 35th International Conference on Machine Learning[CCML)) 2018.
13

---

[55] Hongxu Yin, Arun Mallya, Arash Vahdat, José Manuel Alvarez, Jan Kautz, and Pavlo Molchanov. See through gradients: Image batch recovery via gradinversion. 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021.
[56] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In Advances Neural Information Processing Systems (NeurIPS), 2020.
[57] Huan Zhang, Hongge Chen, Duane S Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. In International Conference on Learning Representations(((CR))) 2021.
[58] Xuezhou Zhang, Yuzhe Ma, Adish Singla, and Xiaojin Zhu. Adaptive reward-poisoning attacks against reinforcement learning. In International Conference on Machine Leam))ng(/MML), 2020.
[59] Xuezhou Zhang, Xiaojin Zhu, and Laurent Lessard. Online data poisoning attacks. Learning for Dynamics and Control, pages 201-210. PMLR, 2020.
[60] Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In Advances in Neural Information Processing Systems(Neur/SS), 2019.
14


Reviews:
[{"review_id": "aFwGnEuMQc", "reviewer": "Reviewer_kyEC", "summary": "This paper proposes to simulate a RL environment to learn a poisoned gradient, which will be sent to the server for aggregation by some attackers. This paper achieves SOTA attack performance on MNIST and Fashion-MNIST. ", "questions": "-\tIn figure 3, the RL curve converges to random guess and has no oscillation, which is very weird to the reviewer. It seems like a NaN in gradient. \n-\tIn figure 4 c), the RL histogram performs very weird, especially 0.3, RL here seems to have no std? This is nearly impossible for RL. \n-\tIn experiments settings, there are in total 100 works with 20 attackers in it, this setting is so weird.  \n", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Major weakness: \n-\tThe novelty is limited. Why do we need RL here is not clear, compared with some adversarial manipulation on the gradients, e.g., directly maximizing the “reward” function w.r.t. gradients. RL is unstable and hard to train, the necessity should be addressed, otherwise, it seems like simply an adaptation of some techniques. \n-\tThe evaluation is far from enough. This paper should also include some results on colored images, e.g. CIFAR10. \n-\t In practice, noisy gradients may also be considered to provide better privacy. This paper proposes a technique heavily rely on the accuracy of gradients, so, an ablation study is suggested. \n-\tThis paper proposes a technique that seems to require a large amount of malicious users. An ablation study on the number of malicious users is suggested. \n-\tGiven the limited contribution of this paper, it is suggested that this paper provides some investigation on why gradients generated by RL can be better, what kind of properties it possesses, what is its advantages over other methods, or under what settings this method has an overwhelming advantage. This paper also addresses some potential future works, many of them can be integrated organically in this paper to increase this paper’s contribution, e.g backdoor attacks. \n-\tAchieve SOTA on this task on some non-real world dataset is less significant, this paper is suggested to dig out some insights from other aspects, instead of just a SOTA. \n\nMinor weakness: \n- Typos & Grammars: Line 128; Line 304; \n- Writing: The reviewer doesn’t see any insights w.r.t data heterogeneity in section 4, while that is in the section title. \n- It seems that selected malicious clients are required to perform much much much more computations compared with other clients. Although this paper includes a paragraph discussing (line 266- line 274). Some quantitative measure may be preferred, e.g. seconds for FL epoch, seconds for RL epoch.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VkbqCpCJGy1", "reviewer": "Reviewer_jqbg", "summary": "The paper studies the vulnerability of Federated Learning (FL) Algorithm for untargeted poisoning attacks. In this direction, it formulates the problem of poisoning FL systems as a model-based Reinforcement Learning.  \n\nThe RL based attack consists of three key steps: learning the distribution of the data, learning the optimal attack policy by building a simulator from the learned distribution, and finally executing the attack policy. The work presents an upper bound on the attack performance as a function of fraction of malicious nodes in the FL setting and learning rate of the FL algorithm. \n", "questions": "Please see the weaknesses associated with the comparison with the literature, and address them.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper provides a three-phased framework for poisoning attacks, however, the novelty in each component is limited. The methods used are previously known and studied in the literature. \n\nSince the problem is modeled as RL, the paper does not compare the poisoning attack proposed in the paper with the poisoning attack proposed in the online RL setting. A similar comparison with the corresponding attack cost in terms of theoretical guarantees is also missing. \n\nThis paper is also closely related to the misspecification of distribution of a prior in a Bayesian setting. This line of work is independently studied. Authors should consider a comparison and a literature survey in that direction, which is currently missing. \n\nOverall, the writing of the paper is unclear, and the contributions of the paper are limited. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NqtBFXii3xY", "reviewer": "Reviewer_u8VB", "summary": "- The authors propose a RL-based attack framework against federated learning (FL) systems. \n- The framework learn distribution of the dataset using the data of malicious workers and conduct model-based RL using the learned distribution. \n- The authors provide a theoretical analysis of the effect of inaccurate distribution learning on the difference in the optimal return. \n- The authors conduct model poisoning attack to FL system of MNIST and EMNIST learning under Krum and Clipping Median scheme. The experimental results show that the proposed RL-based method outperform the baseline methods.", "questions": "- To apply the attack method on an online FL system, the actual runtime of the attack method  is very important. Even though they suggest a technique for scalability, many RL-based applications suffer from a large amount of computation and runtime. I think an analysis on the actual runtime (during the overall attack process) of the proposed RL method and baseline methods is needed.\n- The distribution learning becomes harder as the dimension becomes larger due to the curse of dimension. Hence, It seems difficult to apply the proposed method to datasets of large dimensions such as ImageNet. If possible, I want to see the experimental results on larger datasets.  If there is not enough time to do this, I would like to see the performance of distribution learning in a large dataset.\n- I think a related work section can help readers to understand the difference from the previous works and clarify the contributions of the paper.\n\n\nPOST REBUTTAL COMMENTS:  The authors answer the questions. My concerns have been addressed. They prove that they can apply their method to the larger datasets such as CIFAR-10. Also, they show that their proposed method without online search which has comparable performance in Krum is fast enough. Moreover, the proposed method with online search is fast enough to be executed in a real-world setting. So I decided to adjust my score from 5 to 6.", "limitations": "I wish that the authors include ethics or broader-impact statement in the paper or appendix.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The experimental results on MNIST and EMNIST under Krum and Clipping Median show that the proposed RL-based attack method clearly outperforms the baseline methods.\n- The authors theoretically analyze the effect of inaccurate distribution learning and explain how the attack performance loss depends on the ratio of benign workers, the inaccuracy of learned distribution. \n- The proposed attack method require less knowledge compared to existing attack methods on FL domain. \n- They suggest a state and action space compression technique to make the method scalable. \n\n# Weaknesses\n- Lack of some important ablation studies such as actual runtime comparison. \n- It seems difficult to apply this method to datasets of large dimension such as ImageNet. Experiments were also conducted on small datasets such as MNIST and EMNIST. In other words, applications of the method seem limited.\n- There is no related work section. \n\n# Other Comments\n- typo) line 128 A^{\\tau(r)} )", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MJsmXvztcHT", "reviewer": "Reviewer_Gb3o", "summary": "The works aims to propose an adaptive model poisoning attack method against federated learning systems. To this end, the paper proposes a model-based reinforcement learning attack framework. To utilize the limited global knowledge, the proposed framework first conduct the distribution learning step, which approximates the distribution of the clients’ aggregated data. Then in the policy learning step, the malicious clients get the attack policy based on the learned FL environment and reinforcement learning, and finally generate attack actions. The authors also theoretically analyze the performance loss of the attack policy due to the inaccurate distribution learning and data heterogeneity. Through experiments with real-world datasets, the authors validates that the proposed framework could successfully poison the global model even when the server adopts a robust aggregation rule.", "questions": "* It seems that the robust aggregation oracle proposed in [1] have better defense performance against the data/model poisoning attacks. Is there any reason not including it in the experiments?\n\n[1] Pillutla, Krishna, Sham M. Kakade, and Zaid Harchaoui. \"Robust aggregation for federated learning.\" arXiv preprint arXiv:1912.13445 (2019).\n\n\nPOST REBUTTAL COMMENTS: Part of my concerns have been addressed and the authors promised to solve the others in the future.", "limitations": "As mentioned in the weaknesses, the distribution learning step could lead to significant attack performance loss. The defenses against gradient leakage attacks should be considered.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* Privacy related issues in federated learning is getting more important. The paper proposes an adaptive attack to poison the global model by integrating distribution learning and RL-based policy learning.\n* The paper provides theoretical analysis on the effect of inaccurate distribution learning and heterogeneous local data distributions on the performance loss of the attack policy.\n* The proposed RL-based attack framework could be potentially extended to different attacks and be applied to other learning scenarios.\n\nWeaknesses\n* There have been proposed several defenses against gradient leakage attacks, e.g., adding noise, and most gradient leakage attacks are only effective on small batches. Both leads to lower estimation accuracy in the distribution learning step. Besides, the Theorem 1 shows that the performance loss is linearly related to the estimation accuracy $\\delta$, which makes the distribution learning step crucial.\n* Besides the robust aggregation rules, several defenses are proposed to defend against targeted/untargeted poisoning attacks based on abnormal detection (e.g., SPECTRE [1]). The experiments should consider this kind of defenses.\n* More complicated datasets should be considered (e.g., CIFAR-10) in the experiments since the distribution learning could be less accurate.\n\n[1] Hayase, Jonathan, et al. \"SPECTRE: defending against backdoor attacks using robust statistics.\" arXiv preprint arXiv:2104.11315 (2021).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept