Thinking Two Moves Ahead: Anticipating Other Users Improves Backdoor Attacks in Federated Learning
Authors: Yuxin Wen
Jonas Geiping
Liam H Fowl
Hossein Souri
Rama Chellappa
Micah Goldblum
Tom Goldstein
Venue: Submitted to ICLR 2023 (2023)
Keywords: Privacy
Federated Learning

Abstract: Federated learning is particularly susceptible to model poisoning and backdoor attacks because individual users have direct control over the training data and model updates.  At the same time, the attack power of an individual user is limited because their updates are quickly drowned out by those of many other users. Existing attacks do not account for future behaviors of other users, and thus require many sequential updates and their effects are quickly erased. We propose an attack that anticipates and accounts for the entire federated learning pipeline, including behaviors of other clients, and ensures that backdoors are effective quickly and persist even after multiple rounds of community updates. We show that this new attack is effective in realistic scenarios where the attacker only contributes to a small fraction of randomly sampled rounds and demonstrate this attack on image classification, next-word prediction, and sentiment analysis.

Under review as a conference paper at ICLR 2023
THINKING Two MOVES AHEAD: ANTICIPATING OTHER USERS IMPROVES BACKDOOR ATTACKS IN FEDERATED LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
Federated learning is partieularly susceptible model poisoning and backdoor attacks because individual users have direct control over the training data and model updates, At the same time, the attack power of individual user is limited because their updates are quickly drowned out by those of many other users. Ex- isting attacks do not account for future behaviors of other users, and thus require many sequential updates and their effects are quickly erased. We propose an attack that anticipates and accounts for the entire federated learning pipeline, including behaviors of other clients, and ensures that backdoors are effective quickly and persist even after multiple rounds of community updates. We show that this new attack is effective in realistic scenarios where the attacker only contributes to small fraction of randomly sampled rounds and demonstrate this attack on image classification, next word prediction, and sentiment analysis.
INTRODUCTION
When training models on private information, desirable to choose learning paradigm that does not require stockpiling user data in central location. Federated learning (Konečný al. 2015) McMahan et al.] 2017b) achieves this goal by offloading the work of model training and storage to remote devices that do not directly share data with the central server. Each user device instead receives the current state of the model from the central server, computes local updates based on user data, and then returns only the updated model to the server.
100
Barkdoor Accuracy
80
60
40
www
Baseline Anticipate
20
First/Last Attack
250
500
750
1000 FL Rounds
1250
1500
1750
2000
Our method, Anticipate, reaches 100% backdoor accuracy faster than the baseline in the setting of 100 random attacks in the first 500 rounds. Moreover, after the window of attack passes, the attack decays much slower than the baseline. the end of federated training, our attack still has backdoor accuracy of 60%, while the baseline maintains just 20%. Overall, only 100 out of a total 20k contributions are malicious.
Figure 1:
Unfortunately, by placing responsibility for model updates in the handle of many anonymous users, federated learning also opens up model training to range malicious attacks (Bagdasaryan et al, 2019 Kairouz et al. 2021). In model poisoning attacks (Biggio & Roli 2018 Bhagoji al. 2019)

---

Under review as a conference paper at ICLR 2023
user sends malicious updates to the central server to alter behavior of the model. For example in language modeling, backdoor attacks could modify the behavior of the final model to misrepresent specific facts, attach negative sentiment to certain groups, change behavior in edge cases, but also attach false advertising and spam to certain key phrases.
In practical applications, however, the real threat posed by such attacks is debated (Sun et al. 2019b Wang et al., [2020) [Shejwalkar et al. 2021). Usually only a small fraction of users are presumed to be malicious, and their impact on the final model can be small, especially when the contributions of each user are limited by norm-bounding (Sun et al., 2019b). Attacks as described in BBggasaayya & Shmatikov (202I) further require successive attacks over numerous sequential rounds of training.
This is not realistic in normal cross-device applications (Bonawitz et al., 2019 Hard et al., (019)) where users are randomly selected in each round from larger pool, making it exceedingly unlikely that any attacker or even group of attackers will be able to contribute to more than fraction of the total rounds of training. Model updates that are limited in this way are immediately less effective, as even strong backdoor attacks can be wiped away and replaced by subsequent updates from many benign users Sun et al. (2019b); Shejwalkar et al. (00TT)
In this work we set out to discover whether strong attacks are possible in these more realistic scenar- ios. We make the key observation that previous attack algorithms such as described in Bagdasaryan eet al. (2019); Wang al. (2020;; Zhou et al. (202T) only consider the immediate effects of a model update, and ignore the downstream impacts of updates from benign users. We show that, by model- ing these future updates, a savvy attacker can update model parameters in way that is unlikely to be over-written or undone by benign users. By backpropagating through simulated future updates, our proposed attack directly optimizes malicious update to maximize its permanence. Using both vision and language tasks, and under a realistic threat model where attack opportunities are rare, we see that these novel attacks become operational after fewer attack opportunities than baseline methods, and remain active for much longer after the attack has passed as shown in Figure
2 BACKGROUND
Federated Learning systems have been described in a series of studies and a variety of protocols. In this work, we focus on mainly on federated averaging (fedAVG) as proposed in McMahan et al. (2017b) and implemented in a range of recent system designs (Bonawitz et al.] |019] Paullkk et al., 2021 Dimitriadis et al., 2022). but the attack we describe can be extended to other algorithms. In fedAVG, the server sends the current state of the model 0; to all users selected for the next round of training. Each user then computed an updated local model through several iterations, for example via local SGD. The u-th local user has data D which is partitioned into batches D,, and then, starting from the global model, their local model is updated for m steps based on the training objective C:
01 = o..
VD O, for j l....,m. m.
The updated models 1,m from each user are returned to the server which computes a new central state by averaging:
1
0.+1
0 10
We will later summarize this procedure that depends on a group of users U; in the i-th round as 0.41 (U, 0z).
Optionally, the average can be reweighted based on the amount of data controlled by each user (Bonawitz et al., 2017), however this is unsafe without further precautions, as an attacker could overweight their own contributions such that we only consider unweighted averages in this work. Federated Averaging further safeguarded against malicious users by the use of norm- noundignn Each updated model 0... is projected onto an 1l0, bp I C, for some clip value C' so that user update can dominate the average.
Norm-bounding is necessary to defend against model replacment attacks described in Bagdasaryan et al. (219)) and /hhagoji et al. (019 which send malicious updates with extreme magnitudes that overpower updates from benign users. Once norm-bounding is in place defense though, the potential threat posed by malicious attacks remains debated. We summarize a few related areas of research, before returning to this question:
2

---

Under review as a conference paper at ICLR 2023
Adversarial Machine Learning
The attacks investigated in this paper are a special case of train-tima adversarial attacks against machine learning systems (Biggio et al. 2012 Cina et al., (022). The federated learning scenario is naturally an online, white-box scenario. The attack happens online, while the model is training, and can adapt to the current state of training. The attack is also white-box all users have knowledge of model architecture and local training hyperparameters.
Train-time Attacks
In this work we are interested in backdoor attacks, also refered to as targeted attacks, which form subset of model integrity attacks (Barreno et al. 2010). These attacks generally attempt incor- porate malicious behavior into model without modifying its apparent performance on test data. In the simplest case, malicious behavior could be an image classification model that misclassifies images marked with special patch. These attacks are in contrast to model availability attacks which aim to undermine model performance on all hold-out data. Availability attacks are generally considered infeasible in large-scale federated learning systems when norm-bounding is employed Shejwalkar et al. 2021), given that malicious users likely form only a minority of all users.
Data Poisoning
The model poisoning attacks described above are closely related to data poisoning attacks against centralized training (Goldblum et al., 2020). The idea of anticipating future updates has been inves- tigated in some works on data poisoning (Munoz-Gonzzlezz et al.] 2017 Huang et al. 2020) where it arises as approximation of the bilevel data poisoning objective. These attacks optimize set of poisoned datapoints by differentiating through several steps of the expected SGD update that the the central server would perform on this data. However, for data poisoning, the attacker unaware of the model state used by the server, cannot optimize their attack for each round of training, and has only approximate knowledge of model architecture and hyperparameters, These complications lead Huang et al. (2020) to construct large ensemble of model states trained to different stages to approximate missing knowledge.
CAN YOU BACKDOOR FEDERATED LEARNING?
Backdoor attacks against federated learning have been described in Bagdasaryan et al. (2019). The attacker uses local data and their malicious objective create their own replacement model, scales this replacement model to the largest scale allowed by the server's norm- norm-bounding rule and sends it. However, as discussed in [un et al.) (2019b), for more realistic number of malicious users and randomly occurring attacks, backdoor success is much smaller, especially against stringent norm- bounding. Wang et al. (2020) note that backdoor success high in edge cases not seen in training and that backdoors that attack "rare" samples (such only airplanes in a specific color in images, or a specific sentence in text) can be much more successful, as other users do not influence these predictions significantly. A number of variants of this attack exist (Costa et al. 2021: Pang al.) 2021 Fang et al. 2020 Baruch et al. [2019] Xie et al. 2019 Datta et al. 2021 Yoo & K Kwak] 2002 Zhang et 2019 [Sun et 2022 [Shen et al. 2020). for example allowing for collusion between multiple users or generating additional data for the attacker. In this work we will focus broadly on the threat model of Bagdasaryan et al. (2019) Wang et al. (0200).
Threat Model We assume a federated learning protocol running with multiple users, attacked by online white-box model poisoning. The server orchestrates federated averaging with norm- bounding. At each attack opportunity, the attack controls only single user and only has knowledge about the local data from this user. The attacker has full control over the model update that will be returned to the server and can optimize this model freely. As a participating user in FI., the attacker also aware of the number of local steps and local learning rate that users are expected to use. We will discuss two variations of this threat model with different attack opportunities. 1) An attacker opportunity is provided every round during a limited time window as in Bagdasaryan et al. (2019). 2) Only limited number of attack opportunities arise randomly during limited time window as discussed in Sun et al. (2019b).
We believe this threat model with random attack opportunities is natural step towards the evalu- ation of risks caused by backdoor attacks in more realistic systems. We do restrict the defense to only norm-bounding and explore a worst-case attack against this scenario. As argued in Sun et al.
3

---

Under review as a conference paper at ICLR 2023
2019b), norm-bounding is thought to be sufficient to prevent these attacks. We acknowledge that other defenses exist, see overviews in Wang et al. (2022) and QQu et al. (2022). yet the proposed attack is designed to be used against comn-bouneed FL systems and we verity in Appendix(A.5 that
does not break other defenses. We focus on norm bounding because it is a key defense that is
widely adopted in industrial implementations of federated learning (Bonawitz al. 2019 Paulik
et al. 2021 Dimitriadis et al. 20222).
ATTACKS WITH END-TO-ENT OPTIMIZATION
4.1 BASELINE
As describe by Gu al. 2017): Bagdasaryan al. 2019). suppose an attacker holds N clean data points, D. = { x Vi Ti=1 and A backdoored data points. Dj = { ye.y" ) where z could be an input with special patch or edge-case example (Wang et al. 2020). and y" is an attacker-chosen prediction. The goal of the attacker is train malicious model that predicts y when sees backdoored input, and to push this behavior to the central model. The attacker can optimize their
malicious objective Ladv directly identify backdoored parameters:
0 = argmin Lady (Ds, 0)
(3)
where Ludy is the loss function of the task, 0 are the weights of the local model. Some attacks such as Bhaaoji et al.] (2019) also include an additional term that enforces that model performance on local clean data remains good, when measured by the original objective L:
8 argmin Laav (Ds, 0) L(De,
0
The update is then scaled to the maximal value allowed by norm-bounding and sent to the server. This baseline encompasses the attacks proposed in Xie et al. (2019) and BBagoji et al. (2019) in the investigated setting.
4.2 ANTICIPATING OTHER USERS
This baseline attack can be understood as a greedy objective which optimizes the effect of the back- door only for the current stage of training and assumes that the impact of other users negligible after scaling. We show that stronger attack anticipates and involves the benign users' contribu- tions in current and several future rounds during the backdoor optimization. The optimal malicious update sent by the attacker should be chosen that it is optimal even if the update is averaged with the contributions of other users and then used for several further rounds of training to which the attacker has no access. We pose this criteria allows the attack to optimally select which parts of the model update to modify, and to estimate and avoid which parts would be overwritten by other users.
loss function to be optimized. Intuitively, this
Formally, with n users per round, suppose an attacker wants to ant ant inticipate k steps (in the fol- lowing we will use this keyword denote the whole attack pipeline). Then, given the current local model, fg. the objective of the attacker is simply to compute the adversarial objective in Equation (3), but optimize not directly for 0, but instead future 0,, 1 S i < k, which depends implicitly on the attacker's s contribution.
To make this precise, we move through all steps now Denote the model update that the attacker contributes by 0mal In the next round following this contribution, the other users Ug will themselves contribute updates 8, Both are averaged and result in
i mal + Σ 60.5
0
(5)
where 02 now depends on 0mat Then, more rounds follow in which the attacker does not contribute, but where new users U, contribute:
0 Fang (U., 0.)
4

---

Under review as a conference paper at ICLR 2023
Finally, 0h still depends implicitly on the malicious contribution m As such, omniscient at- tacker could then optimize
0" argmin
Cade Cadv(DD 0, (0(m)) + C( 0c(Oma)))
Bost
differentiating the resulting graph of Cadv with respect to mall and compute the gradient direction in which mal should be updated to improve the effect of the backdoor.
However, in practice, this optimiza- tion problem is intractable. First, the attacker is unaware of the exact private data of other users in future rounds. Mcanwhile, involving the full group of all users Uj in the in- termediate federated learning round makes the problem unsolvable for limited compute resources, given that each call to Fiwg contains many lo- cal update steps for each user which each depend mal- Therefore, stochastically sample the full opti- mization problem: First, we decide to model only a subset users n' < and then randomly sample sin- gle batch of data for each local up- date in each round, from the attack-
Algorithm Anticipate Algorithm
I: Input:
Global model 0p. batch size 6, number of mod- eled users per round n', future updates anticipated: k, update steps m'', the attacker A owns a set of clean data
De and backdoor data Do. 2: m = Initialize from to
do 4: for i = 0, ...,, k do
for 0, ... m'
5:
Model group users Us: for 0, ..... n do
6:
7:
- = Sample b data points from De Run one round of federated averaging:
8:
9:
if i = 0 then
10:
0 Favg (A(0mal) U U.; 0,)
11:
else
12:
0++1= Favg (U,,0.)
13:
Differentiate the k - th step w.r.t to 0mal:
ers own data source De. Based on this data, the attacker can then re- compute the local update steps for this limited group users and this way stochastically approximate the real contributions from other users with replaced average over only the subset of modeled users. Over multiple steps m' over which the attacker optimizes the malicious update mmai random data is sampled in every step. We summa- rize all steps Algorithm[I [ Although the estimation of the adversarial gradient is randomized and based on the distribution of the attacker's data, we find that this scheme is able to reliably generate malicious updates that lead robuster backdoors.
90mal Veeaa Σ (D. 01) + ((Do.
14:
15:
Update Omal based on gomut
16:
return Oma!
5 EXPERIMENTS
In this section, we thoroughly analyze our attack on three different datasets: CIFAR-10 (image classification) (Krizhessky al. 2009), Reddit (next-word prediction) (Caldas et al. 2018), and Sentiment140 (sentiment analysis) (Go et al., 2009). The Reddit dataset naturally contains non-IID partitions. For CIFAR-10 we include results for both IID and non-IID partitions of the dataset. Overall, we show that the proposed method does outperform the baseline of Bagdasaryan et al. (2019) under all tasks and scenarios.
5.1 EXPERIMENTATION DETAILS
As described, our experiments implement fedAVG (McMahan et al. 2017a Wang et al. norm-bounding (Sun et al.] 2019a). We implement the implicit objective defined Equation 7 using functorch (He & Zou 2021b. For each lig. sample data points from private clean data randomly. For example, for CIFAR-10 and batch size 64, this still allows us to fit k
2021b) with
steps with 10 modeled users onto 11GB of GPU memory. Note that the number of actual users is significantly larger.
For all three datasets, follow the overall settings discussed in Bagdasaryan al. (2019): Wang jet al. (2020) 2021b). We also randomly split CIFAR-10 over users to make an IID CIFAR 10 task. Details of how each dataset is processed are described below:
5

---

Under review as a conference paper at ICLR 2023
Backdoor Accuracy
100
Backdoor Accuracy
100
75
75
50
50
25
Baseline
25
Anticipate
Anticipate
First/Last Attack
First/Last Attack
100
200
300
400
500
100
200
300
400
500
FL Rounds
FL Rounds
(a) non-III CIFAR-10
(b) IID CIFAR-10
Backdoor Accuracy
100
Baseline Anticipate
Backdoor Accuracy
100
Baseline
Anticipate
75
FirstLast Altack
75
Firs//Latt Altack
50
50
25
25
0
100
300 FL Rounds
200
400
500
100
200
300
400
500
FL Rounds
(c) Reddit
(d) Sentiment140
Figure 2: Attacks over sequential rounds. this scenario, an attacker is able to continuously send malicious updates for 100 (a), 100 (b), 30 (c), or 50 (d) rounds. All plots show the run with the first random seed. The attack strictly outperforms previous work under this simpler threat model.
For CIFAR-10 we investigate IID partition of data to users and the non-IID split computed through Dirichlet sampling with a (Hsu et al.) 2019) both with total 100 users.
CIFAR-10:
For both CIFAR-10 partitions, we choose the backdoor pattern trigger from Gu et al. (017). The attacker can hence overlay the backdoor pattern on clean data inputs to generate backdoored inputs Dr. We choose the label 8 (ship) as the target class for all experiments. For fair comparisons to prior work (Bagdasaryan et al. 2019). we also choose ResNet18 (He et al. 2016). However, it is unclear how realistically implement norm- comm-bounding for the running stats of Batch Normalization. and global batch norm would typically not be available in a federated system (Ioffe & Szegedy) [015 Li al. 2021). Therefore, following Wang al. (2021b), replace Batch Normalization with Group Normalization with G 32 (Wu & He [2018). and C' 0.5 for norm- comm-boundigg. Empirically, we choose anticipate & training steps.
For the Reddit dataset, we take a subset of 2000 users. For next-word prediction, an attacker wants to provide a target word recommendation for users following trigger sentence. We return to the trigger and target evaluated in Bagdassryan et al. (2019): the attacker backdoors data by the end of sentence, and the target is predict the
Reddit:
appending pasta from Astoria next word de delliii iiious. Following [Bagdasaryan et al. (2019), the adversarial loss the model output is only computed based on the last word, i.s of the autoregressive loss. Meanwhile, we re-use the modified 3-layer Transformer model discussed for FL in Wang et al. (022b) with norm- bounding = We again anticipate = steps.
Sentiment140: In Sentiment140 experiments, there are 1000 users in total, and consider the cage-case examples from Wang et al. (2020) as backdoored data. For example, positive tweets labeled negative tweets. For this dataset, adopt the smaller 3-layer Transformer as above (Vaswani et al. (017). where the hidden dimension is 1024, and anticipate k steps. For Sentiment 14), use C 0.3 for norm-bounding.
containing Yorgos Lanthimos

---

Under review as a conference paper at ICLR 2023
5.2 METRICS
An attacker's goal is to ensure that the backdoor attack accuracy is high as possible for the global model at the final stage of federated learning. In real scenarios, the number of rounds in which the attacker will be queried by the server and the total number of rounds For that reason, an attacker wants the attack to be easily implanted and to remain functional as long as possible. Therefore, to test the efficiency of the method, we track the average backdoor accuracy after the first attack (dims) and the average backdoor accuracy after the last attack (Mass).
unknown the attacker.
5.3 SEQUENTIAL ROUNDS
Table I: Results for sequential rounds of attack. We re- port average Ofrest and average Alast of five runs for every ex- periment. Task 1, task 2. task 3, and task 4 refer to CIFAR- 10, IID CIFAR-10. 10, Reddit, and Sentiment
We first verify that the proposed at- tack is always an improvement over Bagdasaryan et al. (2019) in the sim- ple setting in which the attacker is queried in all rounds (Wang et al. 2020). We consider 30, 50, and 100 rounds for CIFAR-10, Reddit, and Sentiment140 respectively.
Method
Task
Task 2
Task 3
Task 4
baseline
64.72
69.72
12.22
31.87
Ofirst
ours
67.27
75.09
29.38
42.70
These numbers are chosen so that the base- line of Bagdasaryan et al. (2019) can
baseline
63.70
70.99
11.25
27.93
Glast
ours
68.04
77.82
27.82
36.92
reach a peak backdoor accuracy of at least 95%. For each task, we then repeat the experiment with 5 random seeds and show the plot of the first random seed (to avoid (herry-picking) in Fig- ure? 2 We see form these curves that the proposed ant anticipate strategy reaches peak accuracy slightly faster than the baseline. Especially for the two NLP tasks, when ant iiciiiiinn iiiiiiittttte pate reaches full backdoor accuracy, the baseline's backdoor accuracy is still below 50%. After the last attack, anticipate still (slightly) outperforms the bascline across all tasks. In addition, we report av- erage Gärst and average Alat of five runs for every experiment in Table [ Note that in this simple setting the attacker can modify every step of FedAvg, and so there is little risk of adversarial updates fading away. In the next section, we will evaluate the baseline and proposed attacks in the more realistic setting in which user is queried sporadically, which the main goal of this paper.
5.4 RANDOM ROUNDS
In a real federated learning scenario, it is rare that an attacker is selected for a large number of consecutive rounds, and we will now switch the more challenging but realistic sce- nario of random selections. In such scenario, an attacker is randomly se- lected by the server and does not have any knowledge of the next selected round. This means sometimes there might be larger time gap between two consecutive attacks. For a fair comparison, we randomly select 100 rounds for CIFAR-10, 50 rounds for Reddit, and 100 rounds for Sentiment 140 from the first 500 rounds of the whole federated learning routine (to simulate a limited time window for the attack). Overall, these 100/50/100 malicious updates are only a small fraction of the 5000 overall updates contributed to the model within the time window of the attack, and an even smaller fraction when compared to 20000 total contributions over the entire 2000 rounds of training. As above, we choose these numbers to yield some success for the baseline attack. Figure 3 shows the plots of each experiment (again from the first random seed). Compared to the experiments of sequential rounds, these more realistic evaluations show that the proposed ant i icipate strat- egy is significantly more effective than the baseline at attacking the central model. For example, for the non-lID 100 experiment, the baseline only maintains a backdoor accuracy of 25% after the attack window, yet ant intiictpptt maintains backdoor accuracy around 65%. We again include quantitative results in Table computing average Giisa and average (lasa over the runs of each task.
Table 2: Quantitative results for random rounds attack. We report average afrst and average Glast of five runs for ev- ery experiment. Task task 2, task 3, and task 4 refer to non-lID CIFAR-10. IID CIFAR-10 10, Reddit, and Senti- ment140.
Method
Task
Task 2
Task
Task 4
baseline
47.38
57.05
14.04
37.85
Offistt
ours
65.76
72.28
36.76
50.50
baseline
47.55
59.41
10.97
26.88
Classt
ours
73.75
80.61
31.26
29.93
7

---

Under review as a conference paper at ICLR 2023
Backdoor Accuracy
100
Backdoor Accuracy
100
75
50
25
Baseline Anticipate
Baseline
Anticipate
Arst/Last
First/Last Attack
0
500
1000
1500
2000
500
1000
1500
2000
FL Rounds
FL Rounds
(a) non-IID CIFAR-10
(b) IID CIFAR-10
Backdoor Accuracy
100
Backdoor Accuracy
100
Baseline
Anticipate
75
75
First/Last
Baseline Anticipate
50
50
First/Last
25
25
200
400
600
800
1000
0
200
400
600
800
1000
FL Rounds
FL Rounds
(c) Reddit
(d) Sentiment140
Figure 3: Attacks random rounds. Attacks happen random 100 (a), 100 (b), 50 (c), 100 (d) rounds of the first 500 FL. training for four tasks, respectively. All plots show the run with the first random seed. The proposed attack strategy is notably effective under this more realistic threat model.
5.5 COMPARISON WITH NEUROTOXIN
We further compare ant anticipate with recent state-of-the-art attack Neurotoxin (Zhang al. 2022) the random rounds setting. Briefly, Neurotoxin masks out parameters with top-k% magnitudes. In their setting, they ereate the mask based the benign update from the last round to increase the durability of the attack, but is unlikely to receive an update from the previous round in the random rounds setting,
the attacker's clean data to estimate the parameter magnitudes.
From the Figure Neurotoxio n increases the durability on the baseline, but is slow at injecting the attack. Mcanwhile, the overall performance of Neurot oxin is behind ant cippae. We also try to combine ant ictipte with Neurot oxin, but unfortunately, Neurotoxin slows down ant icipate lot, though they end with close position.
100
Backdoor Accuracy
80
60
40
Anticipate
20
Neurotoxin
Antripaate Nearstosin
InstiasttAAtac
250
500
750
1000
1250
1500
1750
2000
FL Rounds
Figure 4: Comparison with Neurotoxin. Simulating up to 11 anticipate steps improves the backdoor persistence.
8

---

Under review as a conference paper at ICLR 2023
5.6 HYPERPARAMETER STUDY: NUMBER OF STEPS
A central hyperparameter to the attack is the amount of steps to anticipate (and subsequently to evaluate the objective on). In Figure we compare anticipation intervals between 3 and 11 on the random rounds attack for non-IID CIFAR-10. We find that the larger the number of steps an attacker employs, the faster the attack implanted. However, interestingly, such quick implantation does not necessarily mean that the attack lasts longer. The highest accuracy at the end of training is actually reached k 9 steps. However, any number of steps greater than 3 improves over the greedy baseline attack which corresponds to k = 0.
100
Andic pate Stop
Backdoor Accuracy
80
Bussline
Problest Atback
40
20
250
500
750
1000
1250
1500
1750
2000
FL Rounds
Figure 5: Backdoor accuracy among different antic anticipate steps. Simulating up to 11 anticipate steps improves the backdoor persistence.
5.7 EFFICIENT ANTICIPATE
100
We notice that directly optimizing all k steps can be time-consuming and computationally heavy. Therefore, investigate three different varia- tions of anticipate to boost the efficiency of the algorithm: A) the
80
Auouracy
attacker only calculates the loss on the last step. B) for each iteration, the attacker randomly chooses a num- ber k < k and calculates the loss on all k' steps. C) the attacker fol- lows the variant B. but only calculates the loss on step k'. In Figure 6 we plot the performance and average op- timization time (k 9) of three vari- ants. Variant A gives the most durable attack the end due directly targets the furthest step k. However, variant A is much slower than the normal ant i seiiiiitteee during the injecting time. We can reduce a huge amount of optimization time by choosing variant C, and it is still better than the baseline. Meanwhile, this strategy Although, all variants are much slower than the baseline, in a real scenario, we believe that such delay would be hidden in the general update delay that common in federated learning due to communication issues and hardware heterogeneity, especially for mobile devices as described in Bonawitz et al. (2019). The attacker can further compute their update within the same time budget by using hardware that is several times stronger than the slowest hardware allowed by the server.
40
20
Va 30.65
26.
Baseline.
FirstLast Attack
1000
1250
1500
1750
2000
Figure 6: Efficient Variants of anticipate. More effi- cient and fast anticipate can still improve the baseline.
less computing resources than the original antici te.
CONCLUSION
We evaluate the feasiblity of backdoor attacks in the realistic regime where users are numerous and not consistently queried by the central server. We do this by considering an attack that models simulated FedAvg updates and choose adversarial perturbations that are unlikely to be over-written by other users. Through series experiments on backdoor attacks for image classification, next- word prediction, and sentiment analysis, we show that this strategy leads to strong backdoor attacks, even scenarios where the attacker has relatively few opportunities to influence the model.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT MITIGATIONS
Backdoor attacks have the theoretical potential to be used to disrupt federated learning system used in various applications. for example, for application systems described in (Bonawitz et al. 2019) Paulik et al. 202T Dimitriadis et al. 2022). Aside from disruption to these systems, backdoor tacks have the potential to influence decision makers to favor centralized machine learning systems, which reduces privacy afforded to users of such systems.
Our key message here is that for systems defended by only norm- wom--bunddig scenario, the commu- nity's s estimation of attack capabilities ask suggested by |agdasaryan et al. (2019)) (un et al.] (2019b) underestimates the potential validity of such attack. Although norm-bounding a strong and ef- ficient defense against attack evaluated in (Sun et al. 2019b), the attack discussed this work still works against tight norm bound and in realistic scenarios with few attack opportunities. This shows that additional defenses should be considered on top of norm- -onndounn which by itself continues to be necessary defense mitigate the individual influence of attack that sends extreme malicious updates, like model replacement (Bagdasaryan et al., 2019 Bhagoji et al.) 2019). We verify in Ap- pendix that the attack discussed in this work is specific norm-hounding
defense and can be mitigated using other groups of defenses, such as Krum, krum, or median aggregation, as the attack was constructed break norm-bounded systems in particular.
REPRODUCIBILITY STATEMENT
We provide code with the supplementary material reproduce all attacks.
REFERENCES
Eugene Bagdasaryan and Vitaly Shmatikov. Blind Backdoors in Deep Learning Models. arXiv:2005.03823 [cs], February 2021. /RR/:ttp///aaxit.orrraaa////////
03823
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How To Backdoor Federated Learning. arXiv:1807.00459 [cs], August 2019. URL http:://rxiv.
org/abs 1807 00459
Marco Barreno, Blaine Nelson, Anthony D. Joseph, and J. D. Tygar. The security of machine learning. Machine Language, 1112)12 November 2010. ISSN 0885-6125, doi: 10.1007/ -109944010-5118-55.
Gilad Baruch, Moran Baruch, and Yoay Goldherg. A Little Is Enough: Circumventing Defenses For Distributed Learning. In Advances in Neural Information Processing Systems, volume 32, Cur- ran Associates, Inc., 2019, URL http:tts/////occeiiiiig neurips...i.ppp. wttpcppppp/p.1..... 6cd1 Seeeea886666 www.bbcdtbbaeeeeeeecbeee.. Awtrrdd-sttttttttce html
hash/eclc591
Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing Federated Learning through an Adversarial Lens. In Proceedings of the 36th International Conference on Machine Learming. pp. 634 643. PMLR, May 2019. URL http::t////// htttt:::///prooeddngg.... .ml.
press/v97/bhagooii 9a html
Battista Biggio and Fabio Roli. Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communica- tions Security, CCS '18, pp. 2154-2156, New York, NY, USA, 2018. ACM. ISBN 978-1-4503-
5693-0. doi: 10. N0.114///247334322441
Battista Biggio. Blaine Nelson, and Pavel Laskov. Poisoning Attacks against Support Vector Ma- chines. arXiv:1206.6389 [cs, stat], June 2012. URL
6389
Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. Advances in Neural Information Processing Systems, 30, 2017.
Kallista Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sar- var Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical Secure Aggregation for Privacy
10

---

Under review as a conference paper at ICLR 2023
Preserving Machine Learning. Technical Report 281, 2017. URL http::/eprint.
org/2017/281
Kallista Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzocchi, H. Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage. and Jason Roselander. Towards Federated Learning Scale: System Design. arXiv:/902.01046 [cs, stal), March 2019. URL http:
//arxiv.org//bs//1002 6
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMa- han, Virginia Smith, and Ameet Talwalkar. Leaf: benchmark for federated settings. arXiv preprint arXiv:1812.01097 2018.
Antonio Emanuele Cinà, Kathrin Grosse, Ambra Demontis, Sebastiano Vascon, Werner Zellinger, Bernhard A. Moser, Alina Oprea, Battista Biggio, Marcello Pelillo, and Fabio Roli. Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning. arXiv:2205.00 01 992 [es], May 2022. lttp:///:/axxxx
abs//aba//// 2205.01090
Gabriele Costa, Fabio Pinelli, Simone Soderi, and Gabriele Tolomei. Covert Channel Attack to Federated Learning Systems. 10Xx:2200410561, April 2021. doi: 10.48550/arXi.2104..000
Siddhartha Datta, Giulio Lovisotto, Ivan Martinovic, and Nigel Shadbolt. Widen The Backdoor To Let More Attackers In, October 2021.
Dimitrios Dimitriadis, Mirian Hipolito Garcia, Daniel Madrigal Diaz, Andre Manoel, and Robert Sim. FLUTE: A Scalable, Extensible Framework for High- Rirh--Perfomancc Federated Learning Sim- ulations, March 2022. httLttttp//axii...ooo hwtpa/wwwwww.rra////eaaaaaaaaa/e aass 2203. 13789
Minghong Fang. Xiaoyu Cao, Jinyuan Jia, and Neil Gong. Local Model Poisoning Attacks to [Byzantine-Rouust Federated Learning. In 29th USENIX Security Symposium (USENIX Security 20), pp. 1605 1622, 2020. ISBN 978-1-939133-172 URL http::ttt////tt /wtp://wwwww///o// wwt.://wwenxx.or..
fang
Alec Go, Richa Bhayani, and Lei Huang. Twitter sentiment classification using distant supervision. CS224N project report, Stanford, 1(12):2009, 2009.
Micah Goldblum, Dimitris Tsipras, Chulin Xie, Xinyun Chen, Avi Schwarzschild, Dawn Song, Aleksander Madry, Bo Li, and Tom Goldstein. Dataset Security for Machine Learning: Data Poisoning, Backdoor Attakks, and Defenses. arXiv:2012. 10544 [cs], December 2020. URL.
10544
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint 17X.v:770.0077.. 2017.
Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated Learning for Mobile Keyboard Prediction. arXiv:1811.03604 [cx], February 2019. URL httt:://aaxii.orrrrrr
abs 18 1111114 03604
Horace He and Richard Zou, functorch: Jax-like composable function transforms for pytorch. http/gttthhhhccccc https::/github.coo
https:
2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06333 2019.
W. Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein. MetaPoi- son: Practical General-purpose Clean-label Data Poisoning.
Advances in Neu-
ral Information Processing Systems, volume 33, Vancouver, Canada, December 2020. URL /tttaa://prooceedggg.eeeeee Reeffcr0007233111777777488555523 http://appe___eesppeeeeeeee Files/paper/202//ee
cc/ ptppepp//pepppppeeeepeeepeeeeeea

---

Under review as a conference paper at ICLR 2023
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 4448456, PMLR. 2015.
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D' D'Oliveira, Hubert Eichner, Salim EI Rouayheb. David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateck Mittal, Mehryar Mohri, Richard Nock. Ayfer Özgür, Rasmus Pagh, Mariana Raykova, Hang Qi. Daniel Ramage, Ramesh Raskar, Dawn Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang. Felix X. Yu, Han Yu, and Sen Zhao Advances and Open Problems Federated Learning. arXiv:1912.0499 [cs, stat], March 2021. URL
http:://axxxt org. wwwpg////////a/ /192 04977
Jakub Konečný, Brendan McMahan, and Daniel Ramage. Federated citimization:Distrbuuue
Optimization Beyond the Datacenter. arXiv:1511.0377. [cs, math], November 2015. URL http://arxiy org.org////oo..o hbtp//1bs/1111...11 1ttp/1511..037555
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint 11Xiv:2102.07623, 2021.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efiiieen learning deep networks from decentralized data. In Artificial intelli- gence and statisticy, pp. 1273-1282. PMLR, 2017a.
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera Arcas. Communicationn-Efiicinnn Learning of Deep Networks from Decentralized Data. arXiv:1602.05629 [cs], February 2017b. httptttppp//xi..t///////t wwtp://wwwwggb//////22/222 05629
Luis Muñoz-Gonzálee Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil Lupu, and Fabio Roli. Towards Poisoning of Deep Learning Algorithms with Back- gradient Optimization. In Proceedings of the 10th ACM Workshop Artificial Intelligence and Security, AlSec 17, pp. 27-38, New York, NY, USA, 2017. ACM. ISBN 978-1-4503-5202-4. doi: 10.1145/3128572.314400
Tianyu Pang, Xiao Yang. Yinpeng Dong, Hang Su, and Jun Zhu. Accumulative Poisoning Attacks on Real-time Data. arXiv:2106.0999 09999993009901.. [cs], October 2021. URL htttt://paaxiiii org/
2106. 099993
Matthias Paulik, Matt Seigel, Henry Mason, Dominic Telaar, Joris Kluivers, Rogier van Dalen, Chi Wai Lau, Luke Carlson, Filip Granqvist, Chris Vandevelde, Sudeep Agarwal, Julien Freudi- ger, Andrew Byde, Abhishek Bhowmick, Gaurav Kapoor, Si Beaumont, Áine Cahill, Dominic Hughes, Omid Javidbakht, Fei Dong, Rehan Rishi, and Stanley Hung. Federated Evaluation and Tuning for On-Device Personalization: System Design & Applications. arXiv:2102.05503, February 2021. doi: 10.4 00445550/axxxv.21002.5555
Huming Qiu, Hua Ma, Zhi Zhang, Alsharif Abuadbba. Wei Kang, Anmin Fu, and Yansong Gao. Towards A Critical Evaluation of Robustness for Deep Learning Backdoor Countermeasures. arXiv.2204.0022 April 2022. doi: 10448500aaxxi.2204.6677
Virat Shejwalkar, Amir Houmansadr, Peter Kairouz, and Daniel Ramage. Back to the Draw- ing Board: A Critical Evaluation of Poisoning Attacks on Production Federated Learning. arXiv:2108. 1024/ [cs], December 2021. URL /awpvvxvvvvvvvv////vv//.../a/aav wwtpwwwwwwwwp./////op/ppppoo
httgg/bb/21100 /2200/210 10241
Wen Shen, Henger Li, and Zizhan Zheng. Learning attack distributionally robust federated learning. In NeurIPS-20 Workshop on Scalability, Privacy, and Security in Federated Learning (SpicyF1..), 2020.
12

---

Under review as a conference paper at ICLR 2023
Yuwei Sun, Hideya Ochiai, and Jun Sakuma. Semi-Targeted Model Poisoning Attack on Federated Learning via Backward Error Analysis. arXiv:2003.11633, March 2022. doi: 10.48550/arXiv.
2203.11633.
Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H Brendan McMahan. Can you really backdoor federated learning? arXiv preprint (rXXv=/911.07963 2019a.
Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H. Brendan McMahan. Can You Really Backdoor Federated Learning? arXiv:1911.0796 [es, stat], December 2019b. URL http:
//arxiv..rgg/abs/1333 079663
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy- yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos. Attack of the Tails: Yes, You Really Can Backdoor Federated Learning, July 2020.
Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. A field guide to feder- ated optimization. arXiv preprint arXiv:2107.06917 2021a.
Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H. Brendan McMahan, Blaise Aguera y Ar- cas, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, Suhas Diggavi, Hubert Fichner, Advait Gadhikar, Zachary Garrett, Antonious M. Girgis, Filip Hanzely, Andrew Hard, Chaoyang He, Samuel Horvath, Zhouyuan Huo, Alex Ingerman, Martin Jaggi, Tara Javidi, Peter Kairouz, Satyen Kale, Sai Praneeth Karimireddy, Jakub Konecny, Sanmi Koyejo, Tian Li, Luyang Liu, Mehryar Mohri, Hang Qi. Sashank J. Reddi, Peter Richtarik, Karan Singhal, Virginia Smith, Mahdi Soltanolkotabi, Weikang Song, Ananda Theertha Suresh, Sebas- tian U. Stich, Ameet Talwalkar, Hongyi Wang, Blake Woodworth, Shanshan Wu, Felix X. Yu, Honglin Yuan, Manzil Zaheer, Mi Zhang, Tong Zhang, Chunxiang Zheng, Chen Zhu, and Wen- nan Zhu. A Field Guide to Federated Optimization. arXiv:2107.06911 [cs], July 2021b. URL /ttt://arxiia abbss///////////////// abs/ att///b//////0 2217706617
Zhilin Wang, Qiao Kang, Xinyi Zhang, and Qin Hu. Defense Strategies Toward Model Poisoning Attacks in Federated Learning: A Survey. 2rXiv.2022.66114 February 2022. doi: 10.48550/
arXiv.2202.06414.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018
Chulin Xie, Keli Huang, Pin-Yu Chen, and Bo Li. DBA: Distributed Backdoor Attacks against Federated Learning. In International Conference Learning Representations, September 2019.
URL
net/ forumm id- rkgySOVEvi
Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantine-robuu distributed learning: Towards optimal statistical rates. In International Conference on Machine Learning, pp. 5650-5659. PMLR, 2018
Kiooo Yoo and Nojun Kwak. Backdoor Attacks in Federated Learning by Poisoned Word Embed- dings. In ACL 2022 Workshop on Federated Learning for Natural Language Processing. March 2022. URL htt.s: htttts:://opnzeeieers wwwpr/vee..ee...eeee/pe/.. net / www.m/ommmmmmmmmmmmmmmmmmmm id S3ExnçKfF
Jiale Zhang. Junjun Chen, Di Wu, Bing Chen, and Shui Yu. Poisoning Attack in Federated Learn- ing using Generative Adversarial Nets. In 2019 18th IEEE International Conference On Trust, Security And Privacy In Computing And Communications/111) IEFF International Conference On Big Data Science And Engineering (TrusCConBigDattSEE, pp. 374-380, August 2019. doi: 10. http://ssstoo///iuuuuu.....
Zhengming Zhang, Ashwinee Panda, Linyue Song. Yaoqing Yang, Michael Mahoney, Prateek Mit- tal, Ramchandran Kannan, and Joseph Gonzalez. Neurotoxin: durable backdoors in federated learning. In International Conference Machine Learning, pp.
PMLR, 2022.
13

---

Under review as a conference paper at ICLR 2023
Xingchen Zhou, Ming Xu, Yiming Wu, and Ning Zheng. Deep Model Poisoning Attack on Fed- erated Learning. Future Internet, 13(3):73, March 2021. ISSN 1999-5903. doi: 10.3390/ fi13030073.
A APPENDIX
A.1 ABLATION STUDY: AMOUNT OF PRIVATE DATA
The number of private data points an attacker holds is critical for how well the attacker can esti- mate the benign user's contribution. Intuitively, the more private data attacker holds the more accurately the attacker can predict other users. To estimate the effect of data on the attack, we test variations where the attacker holds 100, 300, 500, and 700 data points for non-IID CIFAR-10, and show backdoor evaluations in Figure 7 In this case, all other benign users have 500 images. For the experiment with data size in the figure, the attacker replaces the data for other random noise, using their own 500 images only to create Dz. We find that more data does robustify backdoor performance, and that random data is insufficient model other users. However, even with only 100 data points, the estimation is notably successful.
with
100
data
Backdoor Accuracy
80
Prstiaa Aluck
40
20
250
500
750
1000
1250
1500
1750
2000
FL Rounds
Figure 7: Backdoor accuracy with different amounts of private data held by the attacker. Even limited amount of local data available the attacker is sufficient for strong attack.
A.2 ABLATION STUDY: NUMBER OF USERS PER ROUND
Another important factor in federated learning the number of users involved in each round. The aggregation between larger number of users might be more difficult for implanting the attack. We continue to assume that the attacker is sometimes in control of single user per round. Again, we test cases with 5, 10, 15. 20, and 30 users per round for non-IID IID CIFAR-10. Figure 8 shows how effective the attack in different situations. For the case with less than 20 users per round, the attack is still effective with 100 random rounds attack in the first 500 rounds. However, when there are more than 20 users per round, the attack still needs more rounds to work.
onfurers
Backdoor Accuracy
80
vm
60
40
wm
20
1000
FL Rounds
Figure 8: Backdoor accuracy with different numbers of users per round. The effectiveness of attack decreases as the number of users per round increases.
14

---

Under review as a conference paper at ICLR 2023
A.3 ABLATION STUDY: DATA HETEROGENEITY
Figure 9 shows the results on different a for Dirichlet sampling in random round attack setting for non-IID CIFAR-10. The smaller is, the more non-iid sampling is. The effectiveness of the attack decreases as the data heterogeneity increases. However, Anticipate can still improve the baseline, even a 0.25. According Wang al. (022a). a 1 a reasonable value for simulating non-iid federated learning.
Backdoor Accuracy
100
Axseline pate Attv
Backdoor Accuracy
100
Backdoor Accuracy
100
75
50
50
25
25
25
500
1000
1500
2000
500
10000
1500
2000
500
1000
1500
2000
FL Rounds
FL Rounds
FL Roundss
0.25
(b)
0.5
(c)
0.75
Backedoor Accuracy
Backedoor Accuran accuraayy..
Soo Antoussle
Backedoor Accuracy
Seso
75
Artnct
75
Tinal Jan
75
Tintt
50
50
25
12
25
500
1500
Rounds
Rounds
(d) Q =
(c) a 2
(f) a = 10
Figure 9: Backdoor accuracy with different a for Dirichlet sampling. The effectiveness of the attack decreases as the data heterogeneity increases.
A.4 MAIN TASK ACCURACY
Figure 10 reports the main task accuracy during the training. There is no significant difference between the baseline and Anticipate.
A.5 ATTACKS AGAINST STRONG CLUSTERING DEFENSES
Instead of norm-bounding, we also implement four defenses from the literature: Krum (Blanchard al. 2017). Multi-krum (Blanchard et al.) 2017). Median (Yin et al. 2018), and Trimmed mean (Yin et al., 2018). Krum and Multi-krum are vector-wise abnormal detection defenses, whereas Me- dian and Trimmed-men mean are element- wise abnormal detection defenses, but note that the proposed attack is not designed to break these defenses. As shown in Figure IT. we can verify that this is indeed the case and Krum, Multi-krum, and median play very effective roles in defending against both the baseline attack of [Bagdasaryan et al.) (2019) and Anticipate. Main task accuracy drops noticeably for all of them in Figure [2] illustrating why their widespread adoption in applications might be so far reluctant. Interestingly. Trimmed- mean both doesn't 'tt reduce main task accuracy as much and is the defenses where the attack works out of the box, although the attack's durability is reduced. We argue that more advanced defenses such these should find more in modern FL. applications so that server owners can choose strong defenses and balance the trade-off between the effectiveness of the defense and the drop in performance.
A.6 OBJECTIVE WEIGHTS
During the implementation, we average two objects in Equation
7 Now, we modify the loss as:
0t argmin Omn
Σ aLad (Do, Og (0ma)) + (1 aD,, 0, (f(mma))
(8)
As shown in Figure 13 if is greater than equal 0.5, there is no significant difference.
15

---

Under review as a conference paper at ICLR 2023
Main Task Accuracy
80
Main Task Accuracy
80
40
No Altack
No Altack
Baseline
Baseline
Anticipate
Anticipate
20
First/Last
20
First/Last Attack
500
1000
1500
2000
500
1000
1500
2000
FL Rounds
FL Rounds
(a) non-IID CIFAR-10
(b) ITD CIFAR-10
Main Task Accuracy
Main Task Accuracy
10.0
65
7.5
60
No Attack Baseline
No Attack
5.0
55
Anticipate
AnLicipate
2.5
FirstLLstt
50
Lirst/Last Attack
200
400
600
800
1000
200
400
600
800
1000
FL Rounds
FL Rounds
(c) Reddit
(d) Sentiment140
Figure 10: Main task accuracy.
A.7 ABLATION STUDY: NORM-BOUNDING
We show the results with different norm-bounding thresholds in Figure 14
16

---

Under review as a conference paper at ICLR 2023
Backdoor Accuracy
30
Baseline Anticipate
Backdoor Accuracy
Baseline
20
Anticipate
First/Last
First/Last Attack
10
10
500
1000
1500
2000
500
1000
1500
2000
FL Rounds
FL Rounds
(a) Krum
(b) Multi-krum
Backdoor Accuracy
Baseline Anticipate
Backdoor Accuracy
Baseline Anticipate
15
Lirst/Last Attack
75
Larst/Last Attack
10
50
5
500
1000
1500
2000
500
1000
1500
2000
FL Rounds
FL Rounds
(c) Median
(d) Trimmed-mcan
Figure 11: Backdoor accuracy under different defenses.
Main Task Accuracy
Main Task Accuracy
60
40
No Allack
No Allack
Baseline
Baseline
Anticipate
20
Anticipate
First/Last Attack
First/Last Attack
500
1000
1500
2000
500
1000
1500
2000
FL Rounds
FL Rounds
(a) Krum
(b) Multi-krum
Main Task Accuracy
Main Task Accuracy
880
60
60
40
No Allack
40
No Allack
Baseline
Baseline
Anticipate
Anticipate
20
20
LirstLast Attack
RisstLLatt Attack
500
1000
1500
2000
500
1000
1500
2000
FL Rounds
FL Rounds
(c) Median
(d) Trimmed-mean
Figure 12: Main task accuracy under different defenses.
17

---

Under review as a conference paper at ICLR 2023
100
Backdoor Accuracy
80
60
250
500
750
1000
1250
1500
1750
2000
FL Rounds
Figure 13: Backdoor accuracy with different loss weights.
Backdoor Accuracy
100
Baseline Anticipate
Backdoor Accuracy
100
75
FirsULast
75
50
50
25
25
Baseline
Anticipate
Firs//Last Attack
0
0
500
1000
1500
2000
500
1000
1500
2000
FL Rounds
FL Rounds
(a) C = 0.3
(b)
0.5
Backdoor Accuracy
100
Backdoor Accuracy
100
75
75
50
50
25
Baseline Anticipate FirsULast Attack
25
Baseline
Anticipate
FrsuLast
500
1000
1500
2000
500
1000
1500
2000
FL Rounds
FL Rounds
(c) C 0.7
(d) C
Figure 14: Different Norm-bounding Thresholds.
18


Reviews:
[{"review_id": "uHZCVRKl0Xv", "reviewer": "Reviewer_VoJT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an attack that anticipates and accounts for the entire federated learning pipeline, including behaviors of other clients, and ensures that backdoors are effective quickly and persist even after multiple rounds of community updates. Experiments on datasets  demonstrate the performance of this attack on image classification, next-word prediction, and sentiment analysis.\n\n", "summary_of_the_review": "Please see the above. ", "strength_and_weaknesses": "1. The paper claims two moves ahead would be better than other attack methods such as norm-bounding. I guess that Eq. (4) with two loss functions would be the two moves, but the parameter between the moves is not given. Generally, we need to add a parameter to compromise the two loss functions in eq.(4), and add more experiments to show the results with different \"move weights\". However, it seems the moves are averaged and the paper does not explain why the average move is effective and why the compromise weight is not required as expected. \n\n2. The motivation and experiments are with strong control. It would be persuasive to add real-world motivation and experimental data.  Plus, I found that a demo version of this paper has been published in ICML. Because a previous version of this paper has already published, it would be better to add discussions about the difference between the two papers. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QzvlRAyLhyO", "reviewer": "Reviewer_xjs7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A new backdoor attack against federated learning is proposed. The paper considers a single malicious device that can share malicious updates with the server to inject a backdoor into the centralized model, and the server adopts FedAvg with norm clipping as the aggregation rule. The main idea is to account for benign users' behavior in future rounds to make the malicious updates more persistent. Simulation results demonstrate the effectiveness of the proposed attack against the norm-bounding defense. ", "summary_of_the_review": "The idea of simulating benign users' behavior to develop more advanced attacks against federated learning is interesting. However, the paper considers an over-simplified setting, and the proposed backdoor attack is only effective against the norm-bounding defense and completely fails against other common defenses. The proposed solution is far from being sophisticated. ", "strength_and_weaknesses": "Strengths\n\n1. The idea of simulating benign users' behavior to develop more advanced attacks against federated learning is interesting.\n2. The proposed backdoor attack is effective against the norm-bounding defense. \n\nWeaknesses\n\n1. The paper considers an oversimplified setting with a single malicious device, and the simulation algorithm only considers a single attack step (the current step) and ignores the possibility that the device may be sampled again soon. Further, the algorithm simulates a fixed number of steps before the next attack happens, which is inaccurate given the randomness of the sampling algorithm used by the server. As the attacker has no access to other users' local data, the paper simply simulates their behavior from the attacker's local data, which can also be inaccurate for non-iid local data distributions. \n2. Because of the simplified design, the proposed attack can only break the norm-bounding defense and completely fails when the server adopts Krum, Multi-Krum, or Median, as shown in the appendix. Note that these are not really \"advanced\" defenses as they are not designed for backdoor attacks. Recent defenses, especially detection-based and post-training-based defenses, obtain more promising results against backdoors, which are completely ignored in the paper. \n3. For the norm-bounding defense considered in the paper, the norm threshold C is crucial for achieving a good tradeoff between the main task accuracy and robustness against attacks. But I cannot find the value of C used in the experiments, and there is no ablation study that investigates the impact of C on main task accuracy and backdoor accuracy. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3gaV4xr6us", "reviewer": "Reviewer_Fbk1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work focuses on the model poisoning and backdoor attacks in federated learning. It proposes an attack that anticipates and accounts for the entire federated learning pipeline, including behaviors of other clients.\n\nThe main contributions can be summarized as:\n\n(1) Proposing a backdoor attack in federated learning (anticipate).\n\n(2) Providing extensive experiments.", "summary_of_the_review": "I recommend marginally below the acceptance threshold. I will give my final score based on the response and other reviewer's comments.", "strength_and_weaknesses": "Strengths:\n\n(a) This paper is well-written and easy to read.\n\n(b) I appreciate that extensive experiments are provided in this work.\n\nWeaknesses:\n\n(a) Should m steps be represented in Eq.(1)? If yes, please revise Eq.(1) and provide some references. If no, please provide a reasonable explanation.\n\n(b) In Section 3, it is mentioned that ``We believe this threat model with random attack opportunities is a natural step towards the evaluation of risks caused by backdoor attacks in more realistic systems''. Please provide a proper example to demonstrate the application in reality.\n\n(c) In Section 4, some formulas are not numbered.\n\n(d) It is hard to imagine how to implement the method proposed in this work. In the lines 9 to 12 of Algorithm 1, the proposed method needs to calculate ($\\theta_1, \\theta_2, ..., \\theta_k$). It is ok to calculate these parameters, but it is hard to imagine how to calculate line 14. In line 14, it requires to differentiate k-th step w.r.t $\\theta_{mal}$. Recall the calculation of $\\theta_{u,j}$ (such as Eq.(1)), we can find that it needs to calculate the derivatives. These mean that there are k-th partial derivatives in the calculation of line 14, which are very difficult to implement. This point is my main concern. Please provide a very detailed explanation and describe how to calculate it.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject