DBA: Distributed Backdoor Attacks against Federated Learning
Authors: Chulin Xie
Keli Huang
Pin-Yu Chen
Bo Li
Venue:  (2020)
Keywords: distributed backdoor attack
federated learning

Abstract: Backdoor attacks aim to manipulate a subset of training data by injecting adversarial triggers such that machine learning models trained on the tampered dataset will make arbitrarily (targeted) incorrect prediction on the testset with the same trigger embedded. While federated learning (FL) is capable of aggregating information provided by different parties for training a better model, its distributed learning methodology and inherently heterogeneous data distribution across parties may bring new vulnerabilities. In addition to recent centralized backdoor attacks on FL where each party embeds the same global trigger during training, we propose the distributed backdoor attack (DBA) --- a novel threat assessment framework developed by fully exploiting the distributed nature of FL. DBA decomposes a global trigger pattern into separate local patterns and embed them into the training set of different adversarial parties respectively. Compared to standard centralized backdoors, we show that DBA is substantially more persistent and stealthy against FL on diverse datasets such as finance and image data. We conduct extensive experiments to show that the attack success rate of DBA is significantly higher than centralized backdoors under different settings. Moreover, we find that distributed attacks are indeed more insidious, as DBA can evade two state-of-the-art robust FL algorithms against centralized backdoors. We also provide explanations for the effectiveness of DBA via feature visual interpretation and feature importance ranking.
To further explore the properties of DBA, we test the attack performance by varying different trigger factors, including local trigger variations (size, gap, and location), scaling factor in FL, data distribution, and poison ratio and interval. Our proposed DBA and thorough evaluation results shed lights on characterizing the robustness of FL.

Published as a conference paper at ICLR 2020
DBA: DISTRIBUTED BACKDOOR ATTACKS AGAINST FEDERATED LEARNING
Chulin Xie
Keli Huang
Zhejiang University chul moulinxie@@zz...
Shanghai Jiao Tong University ni ck coope r jtu edu.cn
Pin-Yu Chen
Bo Li
IBM Research
University of Illinois Urbana-Champaic Champaign lbo0illinge edu
pin-yu. chen 1bm. com
ABSTRACT
Backdoor attacks aim to manipulate a subset of training data by injecting adversarial triggers such that machine learning models trained on the tampered dataset will make arbitrarily (targeted) incorrect prediction on the testset with the same trigger embedded. While federated learning (FI.) is capable of aggregating information provided by different parties for training a better model, its distributed learning methodology and inherently heterogeneous data distribution across parties may bring new vulnerabilities. FL where each party embeds the same global trigger during training, we propose the distributed backdoor attack (DBA) - a novel threat assessment framework developed by fully exploiting the distributed nature of FL. DBA decomposes global trigger pattern into separate local patterns and embed them into the training set of different adversarial parties respectively. Compared to standard centralized backdoors, we show that DBA is substantially more persistent and stealthy against FL on diverse datasets such as finance and image data. We conduct extensive experiments to show that the attack success rate of DBA is significantly higher than centralized backdoors under different settings. Moreover, we find that distributed attacks are indeed more insidious, as DBA can evade two tfte-oofthhe-ar aht robust FL algorithms against centralized backdoors. We also provide explanations for the effectiveness of DBA via feature visual interpretation and feature importance ranking. To further explore the properties of DBA, we test the attack performance by varying different trigger factors, including local trigger variations (size, gap, and location), scaling factor in FL, data distribution, and poison ratio and interval. Our proposed DBA and thorough evaluation results shed lights on characterizing the robustness FL..
addition to recent centralized backdoor attacks on
INTRODUCTION
Federated learning (FL. has been recently proposed to address the problems for training machine learn- ing models without direct access diverse training data, especially for privacy-sensitivy tasks (Smith) ct al. 2017 McMahan et al. 2017 Zhao et al. 2018). Utilizing local training data of participants (i.c.. parties), FL helps train shared global model with improved performance. There have been prominent applications and ever-growing trends in deploying FL in practice, such as loan status prediction, health situation assessment (e.g. potential cancer risk assessment), and next-word prediction while typing (Hard et al., 2018; Yang et al.) 2018 2019).
Although FL is capable of aggregating dispersed (and often restricted) information provided by different parties to train a better model, its distributed learning methodology as well as inherently heterogeneous (i.e., non-i.i.d.) data distribution across different parties may unintentionally provide venue to attacks. In particular, the fact of limiting access to individual party's 's data due to privacy concerns or regulation constraints may facilitate backdoor attacks on the shared model trained with FL. Backdoor attack is type of data poisoning attacks that aim to manipulate subset of training data such that machine learning models trained on the tampered dataset will be vulnerable to the test set with similar trigger embedded (Gu al. 2019).
Backdoor attacks on FL have been recently studied (Bagdasaryan et al. 2018 Bhagoji et al. 2019). However, current attacks do not fully exploit the distributed learning methodology of Fl., as

---

Published as a conference paper at ICLR 2020
G
G
federated kaanning aggregator
federated learning aggregator
polson
poison
benign parbicipaniss
Centralized attackes
benign
distributed attackers
global
Gial Pigger
local Inboe: 2 Tea lagger
koll gpar
local triggers
G
(a) centralized backdoor attack (current setting)
(b) DBA: distributed backdoor attack (ours)
Figure 1: Overview of centralized and distributed backdoor attacks (DBA) on FL. The aggregator at round 4 combines information from local parties (benign and adversarial) in the previous round i, and update the shared model G When implementing backdoor attacks, centralized attacker uses a global trigger while distributed attacker uses local trigger which is part of the global one.
they embed the same global trigger pattern all adversarial parties. We call such attacking scheme centralized backdoor attack. Leveraging the power of FL in aggregating dispersed information from local parties train shared model, in this paper we propose distributed backdoor attack (DBA) against FI.. Given the same global trigger pattern as the centralized attack, DBA decomposes it into local patterns and embed them to different adversarial parties respectively. A schematic comparison between the centralized and distributed backdoor attacks is illustrated in [
Through extensive experiments on several financial and image datasets and in-depth analysis, we summarize our main contributions and findings as follows.
We propose a novel distributed backdoor attack strategy DBA on FL and show that DBA is more persistent and effective than centralized backdoor attack. Based on extensive experiments, we report a prominent phenomenon that although each adversarial party is only implanted with a local trigger pattern via DBA, their assembled pattern (i.e., global trigger) attains significantly better attack performance on the global model compared with the centralized attack. The results are consistent across datasets and under different attacking scenarios such as one-time (single-shot) and continuous (multiple-shot) poisoning settings. To the best of our knowledge, this paper is the first work studying distributed backdoor attacks.
When evaluating the robustness of two recent robust FL methods against centralized backdoor attack (Fung al. 2018 Pillutla al., 2019), we find that DBA is more effective and stealthy, as its local trigger pattern is more insidious and hence easier to bypass the robust aggregation rules.
We provide in-depth explanations for the effectiveness of DBA from different perspectives, including feature visual interpretation and feature importance ranking.
We perform comprehensive analysis and ablation studies on several trigger factors in DBA, including the size, gap, and location of local triggers, scaling effect in FL., poisoning interval, data poisoning ratio, and data distribution.
2
DISTRIBUTED BACKDOOR ATTACK AGAINST FEDERATED LEARNING
2.1 GENERAL FRAMEWORK
The training objective of FL can be cast finite-sum optimization: min (F(w)
t "." f.(w)). There are N parties individually processing N local models, each of whom trains with the local objective f:: Rd + R based on a private dataset D. = I {{a), vf b where a, = D and {a., u²) represents each data sample and its corresponding label. In supervised FL setting, each I({a), (r).. vi bED, w:) where I stands for loss of prediction using the local parameters w,. The goal of FL is to obtain a global model which can generalize well on test data Drest after aggregating over the distributed training results from N parties.
local function fa is computed f((w;)
Specifically, at round f, the central server sends the current shared model Gt to n E [N] selected parties, where [N] denotes the integer set (1,2..... .N. The selected party i locally computes the function by running an optimization algorithm such stochastic gradient descent (SGD) for E
2

---

Published as a conference paper at ICLR 2020
local epochs with its own dataset D, and learning rate I, to obtain a new local model L The local party then sends model update 4 G* back to the central server, who will averages over all updates with its own learning rate to generate a new global model Gt-1;
G = G¹ + 72
" 24 GG)
(I)
This aggregation process will be iterated until FL. finds the final global model. Unless specified otherwise, we use Go (I.) to denote the model parameters of the global (local) model at round t.
Attacker ability. Based on the Kerckhoffs's 's theory (Shannon 1949), we consider the strong attacker here who has full control of their local training process, such backdoor data injection and updating local training hyperparameters including E and 4.. This scenario is quite practical since each local dataset is usually owned by one of the local parties. However, attackers do not have the ability to influence the privilege of central server such as changing aggregation rules, nor tampering the training process and model updates of other parties.
Objective of backdoor attack. Backdoor attack is designed to mislead the trained model predict a target label F on any input data that has an attacker-chosen pattern (i.e., trigger) embedded. Instead of preventing the convergence in accuracy as Byzantine attacks (Blanchard et al.) 2017). the purpose of backdoor attacks in FL is to manipulate local models and simulfaneously fit the main task and
backdoor task, that the global model would behave normally on untampered data samples while achieving high attack success rate on backdoored data samples. The adversarial objective"] for attacker i in round with local datatset D; and target label 7 is:
Wi
arg IIXX
Σ
P[C*+ (R(Z), 6)) +|+
Σ P(G (x)) =w])-
(2)
JEST pax
JESS
Here, the poisoned dataset Sy and clean dataset Saa. satisfy Spoi S. = and poi U. Dj. The function R. transforms clean data in any class into backdoored data that have an attacker-choeen trigger pattern using a set of parameters Φ. For example. for image data, é is factored into trigger location 7I., trigger size TS and trigger gap TG (6- = (TS, TG, ),)), which are shown in Fig The attacker can design his own trigger pattern and choose an optimal poison ratio to result in better model parameter w.,, with which G can both assign the highest probability to target label T for backdoored data R(x), 6) and the ground truth label uy.- for benign data azj..
2.2 DISTRIBUTED BACKDOOR ATTACK (DBA)
We again use Fig to illustrate our proposed DBA in details. Recall that current centralized attack embeds the same global trigger for all local attacker (Bagdasaryan al. 2018). For example, the attacker in Fig D (a) embeds the training data with the selected patterns highlighted by 4 colors, which altogether constitutes complete global pattern the backdoor trigger.
In our DBA, as illustrated in Fig (b), all attackers only use parts of the global trigger to poison their local models, while the ultimate adversarial goal is still the same centralized attack using the global trigger to attack the shared model. For example, the attacker with the orange sign poisons a subset of his training data only using the trigger pattern located at the orange area. Similar attacking methodology applies to green, yellow and blue signs. We define each DBA attacker's trigger as the local trigger and the combined whole trigger as the global trigger. For fair comparison, we keep similar amount of total injected triggers (e.g., modified pixels) for both centralized attack and DBA.
In centralized attack, the attacker tries to solve the optimization problem in Eq 2 without any coordination and distributed processing. In contrast, DBA fully exploits the distributed learning and local data opacity in FL. Considering M attackers in DBA with M small local triggers. Each DBA attacker m. independently performs the backdoor attack on their local models. This novel mechanism breaks centralized attack formulation into M distributed sub-attack problems aiming to solve
W = arg max( MF
Σ PIG (O0))((((),,000))) (1)) = +
PIO (x)) = vÿD). ViE [M] (3)
JESS
JEST 2m
In our implementation, use cross entropy training objective.
2 Atthhughh we only show one centralized attacker and one adversarial party in Fig in practice centralized attack can poison multiple parties with the same global trigger, as discussed in (Bagdasaryan et al. 2018).
In our implementation, we use cross entropy as training objective.
3

---

Published as a conference paper at ICLR 2020
Gapy
Size
Gap
Shift.
Features sorted by importance
low
high
trigger
(a) Trigger Size
(b) Trigger Gap
Trigger Location
Figure 2: Trigger factors (size, gap and location) in back- doored images.
Figure 3: Trigger factor (feature im- portance ranking) in tabular data.
where Φ; {0, (i)) is the geometric decomposing strategy for the local trigger pattern of attacker my and Oi)) entails the trigger decomposition rule for m; based on the global trigger 6. DBA attackers will poison with the poison round interval I and use the scale factor y to manipulate their updates before submitting to the aggregator. We will explain the related trigger factors in the next subsection. We note that although none of the adversarial party has ever been poisoned by the global trigger under DBA, we find that DBA indeed outperforms centralized allack significuntly when evaluated with the global trigger.
2.3 FACTORS IN DISTRIBUTED BACKDOOR ATTACK
With the framework of DBA on FL. there are multiple new factors to be explored. Here we introduce of trigger factors that we find to be critical. Fig[Jexplains the location, size and gap attribute of triggers in image dataset. For simplicity, we set all of our local triggers to the same rectangle shape" Jigpaxxn our trigger attribute of ranked feature importance in tabular data (e.g.. the loan dataset).
Trigger Size TS: the number of pixel columns (i.e., the width) of a local distributed trigger. Trigger Gap TG: the distance of the Gap, and Gopy. which represent the distance between the left and right, as well as the top and bottom local trigger, respectively.
Trigger Location TL: (Shi ftz. Shifty) is the offset the trigger pattern from the top left pixel. Scale y: the scaling parameter 7 n/N defined in (Bagdasaryan et al., 2018) is used by the attacker to scale up the malicious model weights For instance, assume the wth malicious local model is X.
The new local model T, that will be submitted calculated T 7(X G*) + G
Poison Ratio r: the ratio controls the fraction of backdoored samples added per training batch. Note that larger should be preferable when attacking intuitively, and there is a tradeoff between clean data accuracy and attack success rate, but too large r would also hurt the attack effectiveness once the model becomes useless.
Poison Interval I: the round intervals between two poison steps. For example, I = 0 means all the local triggers are embedded within one round, while I means the local triggers are embedded in consecutive rounds.
Data Distribution: FL often presumes non-i.i.d. data distribution across parties. Here, we use Dirichlet distribution (Minka 2000) with different hyperparameter a to generate different data distribution following the setups in (Bagdasaryan et al. 2018)..
3
EXPERIMENTS
3.1 DATASETS AND EXPERIMENT SETUP
DBA is evaluated on four classification datasets with non-i.id. data distributions: Lending Club Loan Data(LOAN)<Kan) 2019) MNIST, CIFAR-10 and Tiny-imagenet. The data description and parameter setups are summarized in Tb We refer the readers to Appendix [A. for more details.
Following the standard setup, we use SGD and trains for E local epochs with local learning rate I, and batch size 64. A shared global model is trained by all participants, 10 of them are selected in each round for aggregation. The local and global triggers used are summarized in Appendix A.
Table 1: Dataset description and parameters
Dutsset
Classes
Examples per class
Features
Model used
Benign 1,18
Poison i,1E/
Poison ratio
LOAN
see Th Appendis
91
3: comy and fe
0.001 71
0.0005 / (imuli--boo) (0ssingeeshoo)
10/54
MNIST
10
8000
784
0,1/1
0,05 10
20/44
CIFAR
10
5000
1034
lighwee dit Reunet- 18
0.1/2
0.05/6 /6
5164
Tmy-imagenet
500
He
0.001/2
0.001 / 50malti-shot (0(single-shot)
20054
"Some factor definitions may not apply to non-image data, which will be clarified accordingly. In our implementation, every distributed attacker uses the same ?-
4

---

Published as a conference paper at ICLR 2020
3.2 DISTRIBUTED BACKDOOR ATTACK V.S. CENTRALIZED BACKDOOR ATTACK
Following the attack analysis in (Bagdasaryan et al. 2018), we evaluate multiple-shot attack (Attack A-M) and single-shot attack (Attack A-S) two attack scenarios, which are called naive approach and model replacement respectively in the original paper.
Attack A-M means the attackers are selected in multiple rounds and the accumulated malicious updates are necessary for a successful attack; otherwise the backdoor would be weakened by benign updates and soon forgotten by the global model. In order to quickly observe the difference between centralized and distributed attacks and control the effect of random party selection, we perform complete attack in every round, that is, all DBA attackers or centralized attackers are consistently selected. Benign participants randomly selected to form total of 10 participants.
Attack A-S means that every DBA attacker or the centralized attacker only needs one single shot to successfully embed its backdoor trigger. To achieve that, the attacker performs scaling in their malicious updates to overpower other benign updates and ensure that the backdoor survives the aggregation step. For fair comparison, DBA and centralized attack finish complete backdoor in the same round. Take MNIST as an example, DBA attackers separately embed their local triggers in while the centralized attacker implants its global trigger
round 12, 14, 16, 18 for local triggers in round 18. Benign participants are randomly selected to form a total of 10 participants.
These two scenarios reveal different aspects of DBA and centralized backdoor attacks when the global model is triggered by local and global triggers. Attack A-M studies how easy the backdoor is successfully injected while Attack A-$ studies how fast the backdoor effect diminishes.
In our experiments, we evaluate the attack success rates of DBA and centralized attacks using the same global trigger. For fair comparison, we make sure the total number of backdoor pixels of DBA attackers is close to and even less than that of the centralized attacker (it is hard to control them to the same due to data sampling with certain distribution). The ratio of the global trigger of DBA pixels to the centralized is 0.992 for LOAN, 0.964 for MNIST, 0.990 for CIFAR and 0.991 for Tiny-imagenee Moreover, in order to avoid the influence of the original label when testing attack success rate, remove the test data whose true label equals to the backdoor target label. In three image datasets, we begin to attack when the main accuracy of global model converges, which is round 10 for MNIST, 200 for CIFAR, 20 for Kiny-imagene' in Attack A-M. The reason is provided in Appendix A.2 The global learning rate v) in Attack A-M is 0.1 for CIFAR, 1 for others and in Attack A-S is 0.1 for all datasets.
Altack Success Rate
Succerss Rate
Texting solson
Local
Local Ir gger
lorn Ti
Attack A-8:
Rounds
Rounds
Rounds
Rounds
locs Tr
Trigger
SttockSucoods Sucouss Rate
Mtack Success Rate
:00
Mtack Success Rate
E g
Distrbureed Central
8
8
is
Attack
Attack
8
Rounds
Rounds
Rounds
Rounds
LOAN
MNIST
CIFAR
Tiny-imagende
Figure 4: Attack A-M and A-S. DBA is more effective and persistent than centralized attack.
In Attack A-M. the attack success rate of DBA always higher than centralized attack in all cases as shown in Fig 4 DBA also converges faster and even yields a higher attack success rate in MNIST. Under DBA, we find a prominent phenomenon that the attack success rate of the global trigger is higher than any local trigger even if the global trigger never actually appears in any local training dataset. Moreover, the global trigger converges faster in attack performance than local triggers. Centralized attacker embeds the whole pattern so its attack success rate of any local triggers is low. Due to the continuous poisoning, the attack rate on local triggers still increases for LOAN but this phenomenon does not appear in MNIST and Tiny-imagennt, which indicates that the success of global trigger does not require the same success for local triggers. The results also suggest that DBA can lead to high attack success rate for the global trigger even when some of its local triggers only
5

---

Published as a conference paper at ICLR 2020
attain low attack success rates. This finding is unique for DBA and also implies the inefficiency of centralized attack on FL.
In Attack A-S, DBA and centralized attack both reach a high attack success rate after performing complete backdoor in all datasets with a scale factor ? = 100 as shown in Fig 4 In the consecutive rounds, the backdoor injected into the global model is weakened by benign updates the attack success rate gradually decreases. There is an exception that centralized attack in CIFAR suffers from the initial drop and then rises slowly, which is caused by the high local learning rate of benign participants and is also observed in (Bagdasaryan et al. 2018). We also find that the altack success rate of centralized attack in local triggers and the global trigger faster than that of DBA, which shows that DBA yields a more persistent attack. For example, in MNIST and after 50 rounds, DBA remains 89% attack success rate while centralized attack only gets %. Although DBA performs data poisoning only using local triggers, the results show that its global trigger lasts longer than any local triggers, which suggests DBA can make the global trigger more resilient to benign updates.
3.3 THE ROBUSTNESS OF DISTRIBUTED ATTACK
RFA Piillutla et al. 2019) and FoolsGold (Fung al. 2018 are two recently proposed robust FL aggregation algorithms based on distance or similarity metrics, and in particular RFA is claimed to be able to detect more nuanced outliers which goes beyond the worst-case of the Byzantine setting (Blanchard et al. 2017). In addition, as Attack A-S is more easily detected due to the scaling operation (Pillutla al., 2019), we will focus evaluating the attack effectiveness of DBA and centralized backdoor attacks against both RFA and FoolsGold under Attack A-M setting.
Distributed Attack against Robust Aggregation Defence. RFA aggregates model parameters for updates and appears robust to outliers by replacing the weighted arithmetic mean in the aggregation step with an approximate geometric median. With only few attackers poisoning a small part in every batch, our DBA meets the condition that the total weight of the outliers is strictly less than 1/2 for iterations of RFA so that can converge to a solution despite the outliers. The maximum iteration of RFA is set to be 10 while in fact it converges rapidly, which can give high-quality solution within about 4 iterations. 5io the attack performance of DBA and centralized attack under RFA. For Tiny-imagenet, the centralized attack totally fails at least 80 rounds but the DBA attackers with lower distances and higher aggregation weights can perform a successful backdoor attack. For MNIST and CIFAR, the attack success rate of DBA is much higher and the convergence speed is much faster. For LOAN, centralized backdoor attack takes more than 20 rounds to converge than DBA. To explain the effectiveness of DBA, calculate the Euclidean norm between attacker's model parameter updates and the final geometric median as a distance metric. As shown in Tb Appendix, the malicious updates submitted by DBA attackers have lower distances than that of the centralized attacker's 's updates in all datasets, which help them to better bypass the defense.
Attock Attcc Sucoess Rate 8
Attock Success Rate
3
32
80
Abtack cype Distribute Centralood
Attark: Succosss
5
13
RFA Foosdold
25
43
300 358 400 450 203 550 Ge
30 Rounds
60
100
Rounds
Roundsk
Rounnds
(a) LOAN
(b) MNIST
(c) CIFAR
(d) Thn--mmaganet
Figure 5: Attack effectiveness comparison on two robust RL methods: RFA and FoolsGold
Distributed Attack against Mitigating Sybils Defence. FoolsGold reduces aggregation weights of participating parties that repeatedly contribute similar gradient updates while retaining the weights of parities that provide different gradient updates (Fung et al. 2018). Fig shows that DBA also outperforms centralized attack under FoolsGold. In three image datasets. the attack success rate of DBA is notably higher while converging faster. DBA in MNIST reaches 91.55% in round 30 when centralized attack fails with only 2.91 % attack success rate. For LOAN, which are trained with a simple network, FoolsGolds cannot distinguish the difference between the malicious and clean updates and assigns high aggregation weights for attackers, leading explain the effectiveness of DBA, we report FoolsGold's 's weights on adversarial parties in Tb2 in Appendix. Comparing to centralized attack, although FoolsGold assigns smaller aggregation weights to DBA attacker due to their similarity of backdoor target label, DBA is still more successful. This is because the sum of weights of distributed attackers could be larger than centralized attacker.
fast backdoor success. To

---

Published as a conference paper at ICLR 2020
3.4 EXPLANATION VIA FEATURE VISUALIZATION AND FEATURE IMPORTANCE
Feature importance can be calculated by various classification tools or visually interpreted by class- specific activation maps. For example, in LOAN we show that the top features identified by different classifiers are quite consistent (see Tb.4in in Appendix). Here we use Grad-CAM (Selvaraju et al.) 2017) and Soft Decision Tree (Frosst& & Hinton 2017) to provide explanations for DBA. More details about Soft Decision Tree trained on our datasets are discussed in Appendix A.7
We use the Grad-CAM visualization method to explain why DBA is more steathy, by inspecting their interpretations of the original and the backdoor target labels for clean data input and the backdoored samples with local and global triggers, respectively. Fig the Grad-CAM results of hand-written digit '4'. We find that each locally triggered image alone is a weak attack as none of them can change the prediction (no attention on the top left corner where the trigger is embedded). However, when assembled together as global trigger, the backdoored image is classified as '2' (the target label), and we can clearly see the attention is dragged to the trigger location. The fact that Grad- Brad-CAM results in most of locally triggered images are similar to the clean image, demonstrates the stealthy nature of DBA.
Local Trigger3 (predit 6)
Local Tripger4 gonddct 4)
Clean
Polsonad Hodel
(prddct 4)
(prdict 4)
speacics di
(predict 2)
Low Importanee Feature
High Importance Feature
4
impottere- by
200
Heatmap true ubel
80
4
4
Hestmmp target
22
31
Figure 6: Decision visualization of poisoned digit 4 with target 2 on DBA poisoned model
Feature importance of LOAN learned from its soft decision tree
Figure 7:
Using the soft decision tree of MNIST another example, find that the trigger area after poisoning indeed becomes much more significant for decision making in the corresponding soft decision tree, as shown in Fig 2 in Appendix A.7 Similar conclusion is found in LOAN. We sort the absolute value of filter in the top node of a clean model to obtain the rank of 91 features (lower rank is more important) and then calculate their importance as (1 (1-rank/91)** 100.. Six insignificant features and six significant features are separately chosen to run DBA. The results in Fig show that based on the soft decision tree, the insignificant features become highly important for prediction after poisoning.
4 ANALYSIS OF TRIGGER FACTORS IN DISTRIBUTED BACKDOOR ATTACK
Here we study the DBA trigger factors introduced in Sec 2.3 under Attack A-S, unless specified otherwise. We only change one factor in each experiment and keep other factors the same as in See 3.1 In Attack A-S, DBA-ASK shows the attack success rate while Main-Ace denotes the accuracy of the global model when the last distributed local trigger is embedded. DBA-ASR-L, which reveals the persistence, is the attack success rate of f rounds after a complete DBA is performed. Main-Acc- is the main accuracy after rounds. Note that in general we expect small decrease for main task accuracy right after the DBA but will finally get back to normal after a few rounds of training 6
4.1 EFFECTS OF SCALE
Enlarging scale factor increases both DBA-ASR and DBA-ASR-L, and narrows the gap between them. For CIFAR, although the DBA-ASR reaches over 90% and barely changes once y is bigger than 40, larger 7 still have more positive impact on DBA-ASR-1.
For our four datasets, the more complex the model architecture (in Tb the more obvious the decline in the main accuracy as 7 increases, because the scaling undermines more model parameters in complex neural network. The main accuracy of LOAN doesn't 't drop because of simple model, while the main accuracy of Tiny-imagenet in attacking round even drops to 2.75% when 7 110.
Larger scale factor alleviates the averaging impacts of central server for DBA, which leads to a more influential and resistant attack performance, but also cause the main accuracy of global model
"Except for Sec.
4.1 we use 7 100/30 for image datasets/I.OAl because the latter is casier attack.
7

---

Published as a conference paper at ICLR 2020
to descend in the attacking round for three image datasets. In addition, using large scale factor results in an anomalous update that is too different from other benign updates and is easy to detect based on the magnitude of the parameters. Therefore, there is a trade-off in choosing the scale factor.
Abtakk Success Rake hcc.
300
Attack Succass Rote ACCC
Attack Success Rake Acc.
Allack Successs Rake Acc.
80
60
60
c0
086451-21
006--58-4
58A-108-20
Scale Factor
Scale Factor
Scale Factor
Scale Factor
(a) LOAN
(b) MNIST
(c) CIFAR
(d) iny-imagenet
Figure 8: Effects of Scale on Attack Success Rate and Model Accuracy
3.00
Attock Succcss Rate ACC
Atlack Success Rate ACc.
Allack Success Rate Acc.
300
Attack Success Ratte
60
50
82
60
8
BRA.AK
DOG
D24-553
Acc
Malsace
MahnAcc220
High
12
18
Feature Importance
Trigger
Trigger
Trigger Location
(a) LOAN
(b) MNIST
(c) CIFAR
(d) Tiny-imagenet
Figure 9: Effects of Trigger Location Attack Success Rate and Model Accuracy
4.2 EFFECTS OF TRIGGER LOCATION
For three images datasets, we move the global trigger pattern from the left upper corner to the center, then the right lower corner. The dotted line Fig[9 means that the trigger reaches the right boundary and starts to move along the right edges. The implementation details are in Appendix A.9
We observe U-shape curve between TL and DBA-ASR (in MNIST) DBA-ASR-t (in Tiny- imagenet and MNIST). This is because the middle part in images usually contains the main object. DBA in such areas is harder to succeed and will be faster forgotten because these pixels are funda- mental to the main accuracy. This finding is apparent in MNIST, where the main accuracy after 40 rounds only remains 1.45% in center (TL = 9) while has 91.57% left upper corner (IL = =).
Similar finding can be found in LOAN shown in Fig (a). DBA using low-importance features has higher success rate in attacking round and subsequent rounds. The low low-importance trigger achieves 85.72% DBA-ASR after 20 rounds while the highimmorttnce trigger is 0%.
4.3 EFFECTS OF TRIGGER GAP
In the case of four local trigger patterns located in the four corners of an image, corresponding to the maximum trigger gap in Fig I0, the DBA-ASR and DBA-ASR- are both low in image datasets. Such failure might be caused by the local convolution operations and large distance between local triggers that the global model cannot recognize the global trigger.
The curve of DBA-ASR and DBA-ASR-t in Fig 10 (a) has a significant drop in the middle, This happens when the right lower local trigger covers the center areas in MNIST images. Similar observations can be explained based on Fig (b)...
Using zero trigger gap in CIFAR and Tiny-imagenet, DBA still succeeds but find the backdoor will be forgotten faster. We suggest using non-zero trigger gap when implementing DBA. 4.4
EFFECTS OF TRIGGER SIZE
In image datasets, larger trigger size gives higher DBA-ASR and DBA-ASR-L. Nevertheless, they stable once TS becomes large enough, suggesting little gain in using over-sized triggers.
For MNIST, DBA-ASR is low when TS = 1. This is because each local trigger is too small to be recognized in global model. In the same setting, the centralized attack which uses the global pattern with pixels also isn't 't very successful and its attack success rate soon decreases below 10% within rounds. This reflects that under Attack A-S, backdoor attacks with too small trigger ineffective.
8

---

Published as a conference paper at ICLR 2020
Attack Success Rate NCC.
DOKASI Ma Frice-45
60
Attack Rate
30
82
29
Attack
Natrikes 00
12
Trigger Gap
Trigger Gop
Trigger Gop
(a) MNIST
(b) CIFAR
(c) Tiny-imagenet
Figure 10: Effects of Trigger Gap on Attack Success Rate and Model Accuracy
Attack Success Rate ACCC.
Attack Success Rate
Attack Rate
40
62
Main
Dak Kik Man 4o
DMA A64 Acc-
29
-0
20
Trigger Size (19X)
Trigger Size (195)
Trigger Stze (2*X)
(a) MNIST
(b) CIFAR
(c) Tiny-imagenet
Figure 11: Effects of Local Trigger Size on Attack Success Rate and Model Accuracy
4.5 EFFECTS OF POISON INTERVAL
The attack performance is poor when all distributed attackers submit the scaled updates at the same round (I 0) in all datasets because the scaling effect is too strong, vastly changing the parameter in the global model and causes it to fail in main accuracy. It's also incffective the poison interval is too long because the early embemed triggers may be totally forgotten.
The peaks in Fig (b) show that there exists an optimal poison round interval for LOAN and MNIST. DBA attackers wait until the global model converges and then embeds the next local trigger to maximize backdoor performance, which is a competitive advantage over centralized attack.
In CIFAR and Timn--magenee, varying the interval from up 50 does not lead remarkable changes in DBA-ASR and DBA-ASR-t, which manifests that the local trigger effect can last long and contribute the attack performance of global trigger. From this aspect, distributed attack is extraordinarily robust to RL and should be considered as a more serious threat.
Atlack Success Rate Rec.
200
Attack Success Bate ACC.
102
Allack Success Rate AcS.
1.00
Mitekk Success Rate/ ACC.
40
60
DAX cAR Da4-iGh-0
20
22
20
16
Manshrds
23 25 45
15 20 25
40 43 30
25
ab 4b
45
Polson Round interval
Polson Round interval
Polson Round Interval
Polson Round Interval
(a) LOAN
(b) MNIST
(c) CIFAR
(d) miny-imagenet
Figure 12: Effects of Poison Round Interval on Attack Success Rate and Model Accuracy
105-
Attack Success Bata ACC.
100
Attack Success Rato ACC.
Attack Succes Pate/ IoS.
Altaci Success Rate/
80
60
Mar
40
22
282.099:2
20
-0
ERRAGROE Mandesdt
56 Gr
15 22
32
36 24 33 Peison Por Batch
16
32
Poson Per Batch
Posson Par Batch
Posson Per Batch
(a) LOAN
(b) MNIST
(c) CIFAR
(d) Timy-imagenet
Figure 13: Effects Poison Ratio on Attack Success Rate and Model Accuracy

---

Published as a conference paper at ICLR 2020
4.6 EFFECTS OF POISON RATIO
In our experiments, the training batch size 64. As the X-axis variable (# of poisoned samples) in Fig 3 increases from 1, DBA-ASR and DBA-ASR-t first increase and then drop. It's intuitive that more poisoned data can lead to a better backdoor performance. However, a too large poison ratio means that the attacker scales up the weight of local model of low accuracy, which leads to the failure global model in the main task. In the case of poisoning full batch, after DBA, the global model in CIFAR and Tiny-imagenet trains the main task all over again, whose main accuracy is normal after 90 and 40 rounds, respectively. But in MNIST it is reduced to an overfitted model that predicts the target label for any input, the attack success rate is always 100% while the main accuracy is about 10% in the subsequent rounds. Therefore, it's better for DBA to remain stealthy in its local training by using a reasonable poison ratio that also maintains accuracy on clean data.
4.7 EFFECTS OF DATA DISTRIBUTION
Under various data distributions, DBA-ASR is stable, indicating the practicability and robustness of DBA. See more details in Appendix A.10
5 RELATED WORK
Federated Learning. McMahan et al. (2017) first introduced federated learning (FL) to solve the distributed machine learning problem. Since the training data is never shared with the server (aggregator), FL is in favor of machine learning with privacy and regulation constraints. In this paper, we discuss and analyze our experiments in standard FL settings performed in synchronous update rounds. Advanced FL. for improving communication efficacy by compressing updates using random rotations and quantization has been recently studied in Konečný et al. (2016).
Backdoor Attack Federated Learning. Bagdassarynn al. (2018) proposed model-poisoning approach on FL which replaced the global model with a malicious local model by scaling up the attacker's updates. BBhagoji et al. (2019) considered the case of one malicious attacker aiming to achieve both global model convergence and targeted poisoning attack, by boosting the malicious updates. They proposed two strategies, alternating minimization and estimating other benign updates, to evade the defences under weighted and non-weighted averaging for aggregation. We note that these works only consider centralized backdoor attack FL..
Robust Federated Learning. Robust FL aims to train FL. models while mitigating certain attack threats. Fung et al. (2018) proposed a novel defense based on the party updating diversity without limitation on the number of adversarial parties. It adds up historical updating vectors and calculate the cosine similarity among all participants to assign global learning rate for each party. Similar updating vectors will obtain lower learning rates and therefore the global model can be prevented from both label-flipping and centralized backdoor attacks. Pillutla et al. (019) proposed a robust aggregation approach by replacing the weighted arithmetic mean with an approximate geometric median, so as to minimize the impacts of "outlier" updates.
CONCLUSIONS
Through extensive experiments on diverse datasets including LOAN and three image datasets in different settings, we show that in standard FL our proposed DBA is more persistent and effective than centralized backdoor attack: DBA achieves higher attack success rate, faster convergence and better resiliency in single-shot and multiple-shot attack scenarios. We also demonstrate that DBA is more stealthy and can successfully evade two robust FL approaches. The effectiveness of DBA is explained using feature visual interpretation for inspecting its role in aggregation. We also perform an in-depth analysis on the important factors that are unique to DBA to explore its properties and limitations. Our results suggest DBA is new and more powerful attack on FL than current backdoor attacks. Our analysis and findings can provide new threat assessment tools and novel insights for evaluating the adversarial robustness of FI..
10

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGEMENTS
This work was partly supported by IBM-ILLINOIS Center for Cognitive Computing Systems Re- search (C3SR) - a research collaboration as part of the IBM AI Horizons Network.
REFERENCES
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua. Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. arXiv preprint orXiv:1807.00455 2018.
Moran Baruch, Gilad Baruch, and Yoav Goldberg. A little enough: Cireumventing defenses for distributed learning, arXiv preprint arXiv: 1902.06/56, 2019.
Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing federated learning through adversarial lens. In International Conference Machine Learning, pp- 634-643, 2019.
Peva Blanchard, Rachid Guerraoui, Julien Stainer, et al. Machine learning with adversaries: Byzantine tolerant gradient descent. In Advances in Neural Information Processing Systems, pp. 119-129, 2017.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:/712.05526, 2017.
Nicholas Frosst and Geoffrey Hinton. Distilling a neural network into a soft decision tree. arXiv preprint arXiv:1711.09788 2017.
Clement Fung, Chris JM Yoon, and Ivan Beschastnikh. Mitigating sybils in federated learning poisoning. arXiv preprint arXiv: 1800888888066 2018.
Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230-47244, 2019.
Rachid Guerraoui, Sébastien Rouault, al. The hidden vulnerability of distributed learning in byzantium. In International Conference Machine Learning, pp. 3518-3527, 2018.
Andrew Hard, Kanishka Rao, Rajiv Mathews, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.0300 1.06604, 2018
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Wendy Kan. Lending club loan data, Mar 2019. URL https:://ww.kent com/
Jakub Konečný, Brendan McMahan, Felix X Yu, Peter Richtárík, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05497 2016.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. -fffiiiiinn-fiiienttt Learning of Deep Networks from Decentralized Data. In Proceed- ings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pp. 2273-1282 PMLR, 20-22 Apr 2017.
Communication Efficient
Thomas Minka. Estimating dirichlet distribution, 2000.
Krishna Pillutla, Sham M. Kakade, and Zaid Harchaoui. Robust Aggregation for Federated Learning. arXiv preprint, 2019.
11

---

Published as a conference paper at ICLR 2020
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 618-626, 2017.
Claude E Shannon. Communication theory of secrecy systems. Bell system technical journal, 28(4): 656-715, 1949.
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learning. In Advances in Neural Information Processing Systems, pp. 4424-4434, 2017.
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):12, 2019.
Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li. Nicholas Kong, Daniel Ramage, and Françoise Beaufays. Applied federated learning: Improving google keyboard query 2018.
suggestions, arXiv preprint
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv: /806 00582, 2018.
12

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 DETAILS ON DATASETS AND EXPERIMENT SETUP
The financial dataset LOAN contains the current loan status (Current, Late, Fully Paid, etc.) and latest payment information, which can be used for loan status prediction. It consists of 1,808,534 data samples and we divide them by 51 US states, each of whom represents a participant FL. 80% of data samples are used for training and the rest is for testing. In the three image datasets, Dirichlet distribution is used to divide training images for 100 parties. The distribution hyperparameter is 0.5 for MNIST and CIFAR and 0.01 for Tiny-imagenet.
Every party uses SGD as optimizer and trains for E' local epochs with local learning rate 4, (see Tb and batch size of 64. A shared global model is trained by all participants, 10 of whom are selected in each round to submit locally computed SGD updates for aggregation.
For the pixel-pattern backdoor, assign white color chosen pixels and swap the label of any sample with such triggers into the target label, which is "digit 2" in MNIST, "bird" in CIFAR and "bullffrog" in Tiny-imagenet. Except Section where we analyze the trigger factor effect, in other sections the trigger factors are set to be (3 for MNIST; ø {6, 3, 0} for CIFAR; (10, 2, 0} for Tiny-imagenet with 4 DBA attackers. Because the image size in tiny-imagenet are larger than cifar and mnist, we set the row number of the local trigger to 2 in Tiny-imagenet while it is in other image datasets.
Similarly, for the preprocessed LOAN dataset, six feature which are the low importance features in Fig are chosen and split by 3 DBA attackers, each of whom manipulates two features as a local trigger. They assign local trigger features with new values @ that are slightly larger than their maximum values, and swap label to "Does not meet the credit policy. Status:Fully Paid""
Every attacker's batch is mixed with correctly labeled data and such backdoored data with poison ratio r (see Tb[T]. Attackers have their own local poison 1, and poison E (see Tb to maximize their backdoor performance and remain stealthy.
A.2 BETTER TO ATTACK LATE
In Attack A-M, we found that if DBA poisons from scratch, the main accuracy was low and hard to converge. Therefore in three image datasets, we begin to attack when the main accuracy of global converges, which is round 10 for MNIST, 200 for CIFAR, 20 for Tiny-imagenet. As mentioned in (Bagdasaryan et al.) 2018) 8), it's also better to attack late in Attack A-S because when the global model is converging, the updates from benign clients contain less commonly shared patterns but more individual features, which are more likely to be canceled out when aggregating and thus having less impact on the backdoor.
A.3 DBA ON IRREGULAR SHAPE TRIGGERS
To evaluate DBA on irregular shape triggers, we decomposed the logo 'ICLR' into 'T, 'C', 'L', 'R' as local triggers on three image datasets and we decomposed the physical pattern glasses CCen et al. 2017) into four parts as the examples shown in Fig. 14
The results under Attack A-M are shown in Fig. 15 and Fig. 16
DBA is always more effective than centralized attack, which is similar to the results of regular shape triggers in Fig. This conclusion also holds for glass patterns with different colors shown in Fig. 17
A.4 MORE ANALYSIS ON ATTACK A-S SETTINGS FOR CENTRALIZED ATTACK
In our experiment setup we assumed that there are f distributed attackers and centralized attacker. To further evaluate Attack A-S, we conduct centralized attacks with the same number of times
We preprocess LOAN by dropping the features which are not digital and cannot be onc-hot encoded, and then normalizing the rest 91 features and the mean value of each feature below 10. num tl 120dpd 2m, num tl 90g dpd 24m, pub rec bankruptcies, pub rec, ace now deling
10, 80, 20, 100, 20, 100
13

---

Published as a conference paper at ICLR 2020
centralized attacker
distributed attacker
distributed attacker 2
distributed attacker 3 8
distributed attacker
8
8
8
8
'CLK' logo
KEER
Tiny-imagenet White glasses
"CLR" logo
I<LK
Tiny-
imagenet
"KCLR" logo
Tnn-Imageneet glasses
imagenet-
White
glasses
Tmny-imagenet
Purple glasses
Figure 14: Examples of irregular shape triggers image datasets
200
Attack Success Rate
100
Altack Success Rate
100
Testing poison
Local Trigger
Local Trigger
60
60
Local Trigger
Trigger
Global Trigger
Attack
20
20
20
Attack
Distributed
20
30
Rounds
40
50
60
70
225 250
Rounds
300 350
30
50
Rounds
90 100
Centralized
(a) MNIST
(b) CIFAR
(c) Tiny- imay-emt
Figure 15: Attack A-M for irregular pixel logo 'ICLR'
as DBA, but each update includes 1/ʃ number of poisoning samples, that the total number of poisoning samples included compute the gradient update still stay same. There are two ways to achieve //ʃ number of poisoning samples in each update for centralized attack and we evaluate both as following.
Change the poison ratio into 1/f. We decrease the fraction of backdoored samples added per training batch to f/f.
Specifically, the poison ratio for LOAN is centralized 3/64, distributed 9/64; for MNIST is central- ized 5/64, distributed 20/64; for CIFAR is centralized 1/64, distributed 4/64; for Tiny- tnn-iimgenet is centralized 1/64, distributed 4/64. Other parameters the same as described in the paper.
Fig. shows that DBA is better in LOAN and MNIST while centralized attack is better in CIFAR and Tiny-imagenet.. Similar to the finding in Sec. 4.1 that "the more complex the model architecture
100 80
Testing poison type
100
Attack Success Rate
Local Trigger
Attack Success Rate
Glasses
Local Trigger
80
White Glasses
Local Trigger
Glasses
Local Trigger
60-
Purple Glasses
Global Trigger
40
Attack type Distributed
type
Distributed
Centralized
Central ized
Rounds
Rounds
Figure 16: Attack A-M for white glasses pattern Figure 17: Attack A-M for white, black, purple on Tmn-immgennet
glasses patterns Tiny-imagenet
14

---

Published as a conference paper at ICLR 2020
Attack Succass Rato
160
300
Teites peean
loca Tr Trager
Attact Swe SucraNns
Loca Tr
Lacal Tr Trgper3
Local
8
Glotel
18
Mtad ope
Distr bured
JE 12
65
23
40 Roundss MNIST
225 20 273 366 325 360
355
400
56
Rounds Tiny Imagemet
[c] CIFAR
Figure 18: Scale f times with 1/f poison ratio each time
(in T. the more obvious the decline the main accuracy the scale factor increases, because the scaling undermines more model parameters", the setting of f times scaling for centralized attack has larger impact on complex neural network like Resnet used in CIFAR and Tiny-imagenet. However, we note that this setting is not totally fair comparison of the single-shot attack setting. as the same malicious agent of the centralized attack is allowed to attack f times, while each malicious agent of DBA only attacks once.
Change the data size to 1/f. We divide the local dataset into parts and use I/f dataset for each update and keep the poison ratio unchanged.
Fig.
19
shows that DBA still more persistent than centralized attack.
100
Attack Success Rate
1.00
Testing tye
Local Trigger
80
Altack Success
Trigger
Local Trigger 3
Trigger
Global Trigger
Attack
20
Attak type
Distributed
30
Rounds
40
50
60
225 275 300 350 375
Rounds
400
30
40
R Rounds
50
70
Centralized
(a) MNIST
(b) CIFAR
(c) Tiny-mmagenet
Figure 19: Scale f times with 1/f data size each time
A.5 MORE DETAILS ABOUT ROBUST AGGREGATION
We report RFA distance and FoolsGold weights on adversarial parties in Th.
Table 2: REA Distance and FoolsGold Weight
Metrics
Dalaset
DBA attackerl
DBA attacker2
DBA attacker3
DBA attacker4
DBA attackers(sum)
centralized attacker
LOAN
0.71 +0.50
0.75:-0.49
0.73+0.50
0.81 +0048
MNIST
1.48 1.00
1.1:
1.09
L.0-10.99
2.37
CIFAR
175.43422.82
175.46422.83
148.457-22.84
175.294-2285
196.13
Distance
Iny imagenet
396.6843583
106.01
+66.01+3383
274.32-3383
42.82228.8
431.34 +:40.5
FoolsGold
+0.38
1.181115
0..10
MNIST
10040.000
1.00 +0.00
0.9940..22
0.9940.02
3.9840.04
1.00 +-0.00
Weight
CIFAR
0.8740.2
0.26 00.27
0.37440.6
0.32-40.9
18240.88
+0.34
imagenet
0.88
0.87 0028
0.34
0.26:0013
2.3540.34
0.99.0.11
A.6
THE ROBUSTNESS OF DISTRIBUTED ATTACK IN BYZANTINE SETTING
Here we evaluate the Byzantine setting Krum Blanchard et al. 2017) and Bulyan (Guerraoui al. 2018). For both DBA and centralized attack we use the aggregation rule that can tolerate Byzantine workers among the workers (Blanchard et al. 2017). For centralized attack there is attacker and n non-Byzanting workers. For DBA there are distributed attackers and n non-Byzantine workers. The total number of poisoned pixel amounts are kept the same.
Multi-Krum To meet the assumption that 2 + < n, we set (n = 10, f = 3) for LOAN and (n = 12, f 4) for image datasets. The Multi-Krum Krum parameter is set Tiny-imagenet we decrease the poison ratio into 5/64 for both attacks. Other parameters are the same described in the paper.
n f. For
15

---

Published as a conference paper at ICLR 2020
For CIFAR and Tiny- -ingeemag,net, find that DBA is more effective shown in Fig. 20 and MNIST, both attacks don't behave well. We believe the reason can be explained by the fact that LOAN and MNIST simpler tasks and benign clients quickly agree the correct gradient direction, so malicious updates are more difficult to succeed.
For LOAN
Bulyan We use Bulyan based the Byzantineresilient aggregation rule Krum (Blanchard et al. 20177. To meet the assumption that 4f + 3 <= n, we set (n = 15, f = for LOAN and (n = 20,
4) for image datasets.
For CIFAR, DBA is more effective as shown in Fig. 21 For other datasets, both attacks fail. However, we note that our distributed and centralized backdoor attacks not optimized for Byzantine setting. We believe its worthwhile to explore the distributed version of other new attack algorithms, c.g.
(Baruch al. 2019) that manipulates its update to mitigate Krum and Bulyan defenses.
Attack Success Rate
1.00
Attark Success Rate
100
Attack Success Rate
80
80-
Attack type Distributed Contralized
Attack type Distributed Contralized
50
60
40
40-
20-
252300 25040000000005555500 500552000
40108224000000000
Rounds (a) CIFAR
Rounds
Rounds
(b) Tiny-imagenet
CIFAR
Figure 20: Multi-Krum
Figure 21: Bulyan
In summary, Multi-Krum and Bulyan have stricter assumptions on the proportion of attackers than RFA and FoolsGold. In addition, while RFA and FoolsGold still assign potential outliers with extreme low weights, Krum Krumm,-Krm,, Krum-basee Bulyan) directly removes them, making it impossible to inject backdoors if the malicious updates are obviously far from the benign updates. The centralized attack for four datasets totally fails under Multi- Krum and Bulyan while DBA can still succeed in some cases.
A.7 MORE DETAILS ON SOFT DECISION TREE
Frosst & Hinton (017)) proposed Soft Decision Tree which distills trained neural network by training with data and their soft targets that are the predictions of the neural network over classes. Trained with gradient descent, every inner node has a learned filter and a bias to make binary decision and the leaf node has learned distribution. To some extent we use the filter value reflect the importance of every feature in internal nodes. We learn soft decision trees from the clean neural network and DBA poisoned neural network of LOAN and MNIST and they all achieve about 90% test accuracy on main and backdoor tasks.
If we look the third node in the forth layer in Fig 22 (b), the potential classifications only 2 and O, thus its filter is simply learning to distinguish these two digit. With extreme dark color in the area of the global pattern, which means these pixels correspond to small value in filter, this inner node will make leftmost branch decision into target label 2 when triggered by the global pattern because the probability is lower than 0.5. Taking an opposite example. the leftmost node second layer has extreme white color in the area of the global pattern, which means these pixels correspond to large value of filter and will contribute to make rightmost branch decision encountering the global pattern. Moreover, clean images won't 't trigger the filters in backdoor pattern area and the major digit shape in center dominates the decision route, like examples in Fig (b). Comparing Fig (a) the trigger area after poisoning becomes much more significant for decision making.
Soft Decision Tree provides insights into the neural network and give explainable classification decisions. Examples of the decision routes in inference time for clean and poisoned input data are given for MNIST in F2g and in Fig 24 We find that the poisoned model already starts to misbehave from the top node of the tree.
We also run 10000 poisoned and clean samples for the LOAN clean and poison models to study the sample-wiss wise importance based on the filter value multiplied by the input feature value in Fig 23 (b). With this local importance metric, the original low importance feature indeed becomes salient in poisoned model with poisoned input.
16

---

Published as a conference paper at ICLR 2020
(a) Clean model
(b) DBA poisoned model
Figure 22: Soft decision tree for MNIST
Low Insertand Pertur
High inpetanee Portard
Low Inpertiere Podaaee
(a) based on W
(b) based
W*X clean data
(c) based
W for poisoned data
Figure 23: Feature importance in soft decision for LOAN
(a) predict (target label) for poisoned digit
(b) predict for clean digit
Figure 24: Examples for the poisoned MNIST soft decision tree
CAAY
SALE
LLP
MULT
LAM
KXLT
1
(a) predict for poisoned digit 1
(b) predict for clean digit
Figure 25: Examples for the clean MNIST soft decision tree
17

---

Published as a conference paper at ICLR 2020
A.8 MORE GRAD-CAM RESULTS ON MNIST
We test the global model of MNIST poisoned by DBA under Attack A-M in round 16 of Fig with local backdoored images and global backdoored images. More Grad-cam results provided in Fig and Fig 27
Clann gendie
Local Tripper sprecia
Local Tripper2 Local Teipprr
pradier
brodkl
Locs Trlgperd (ral
Gobal Trigge (pedit 2)
gpader
S
gpadiet
inpper hager
8
8
8
8
8
5
5555
5
5
13 Hestrap
for tre
8
8
for uegat
S
S
S
Hestmap
carget
Figure 26: Example of digit
Figure 27: Example of digit
A.9 IMPLEMENTATION DETAILS FOR LOCATION EFFECT EXPERIMENTS
During this process we increases Shifty and first keeps Shift, Shi fti After the rightmost pixel reaches the right edge of images, we fix Shift, as its largest value, which is X value of the dotted line in Fig 9 and keep increasing Shift, f'y until the lowest pixel reaches the button edge of the images. TL is the max value among Shi and Shifly.
A.10 DATA DISTRIBUTION EFFECTS FOR TRIGGERS
By increasing the hypterparameter a in the Dirichlet distribution, we can simulate from non-i.i.d to i.i.d distributions for the image datasets. When evaluated under Attack A-M, Fig 28 shows that DBA-ASR stable under various distributions, which exhibits the practicability and robustness of DBA when attacking standard FL.
Data distribution has more influence on the DBA performance under robust aggregation algorithms when calculating distance or similarity between the benign updates and malicious updates. When the training data are non-i.i.d., the updates across the benign participants already appear high diversity so the poisoned update is better concealed among them and less likely be detected. In our experiments, it's easier for DBA to succeed against RFA and FoolsGold under more non-i.i.d. data distribution in CIFAR and Tiny-imagenet.
80
80
Attack Success Rate
80.
Attack Sucrees
Attack success
60
60
60
0.09
0.36
40
40
40
20-
0.0
20
20
0.0
10
6 10
12 Rounds
14
18 18 20
20
60
80
100
10
20
30
40
50
Rounds
Rounds
(a) MNIST
(b) CIFAR
(c) Einy-imagenet
Figure 28: Effects of a: in Dirichlet data distribution on Attack Success Rate
A.11 MORE DETAILS ABOUT LOAN DATASETS
The lable distribution is uneven in LOAN, which is shown in Tb The five most important features among the 91 features in LOAN under various classification methods are shown in abd the result consistent.
In Fig. 7 the names for six low importance feature are num_tl wwt.t1112dppddp 2m, num_tl num_t1_900 pub_ brobrec_bant bankrrttesssss pub_rec, acc_now www_oowllnnn..... tax aie_i; the names six high importance feature out out.prnccp.ooal.pyyme pymnt inv, out-prmcp.ing total total www..ppcc.last.pyye.e... amnt, uti......
18

---

Published as a conference paper at ICLR 2020
Table 3: Financial Dataset Label Distribution
Loan Status
Number of Examples
Fully Paid
1041952
Current
919695
Charged Off
261655
Late (31-120 days)
21897
Gruce Period
Late (16-30 days)
3737
Does not meet the credit policy. Status:Fully Paid
1988
Does not meet the credit policy. Status:Charged Off
761
Default
31
Table 4: The Five Most Important Features are Similar in Different Classification Methods
Method
Rank out_prnep prncp inv out prncp
Rank 2nd out-pmeep_img jint
Rank 3rd jast-pymnt_ame
Rank recoveries
Rank 5th
Random Forest Classitier
total rec.__ec_ppmep
Extra Tree Classifier
out prncp
total rec prnpp
last pymnt amnt
total pymat inv
XGBoost
recoveries
funded amnt
total rec prnep
last pymnt amnt
Our decision tree
out_prncp
out_prnep_ing iny
term
recoveries
recovery.fee
19


Reviews:
[{"review_id": "rkeucgq15r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce the idea of distributed backdoor attacks in the FL framework, in which the dishonest participants in FL add local triggers to their training data to influence the global model to classify triggered images in a desired way. They show empirically that the learned models then are more likely to be successfully forced to misclassified images in which all the local triggers are present at test time, than are models learned using centralized backdoor attacks, where all attackers use the same trigger pattern (one of the same size as the concatenation of the local triggers, to be fair in the comparison). They then demonstrate that because the local triggers cause smaller corruptions in the model coefficients, these distributed attacks survive robust FL training algorithms (namely FoolsGold, and a recent robust regression based method) more often than centralized attacks. Similar experiments are conducted on the Loan text dataset, using appropriate analogs of local triggers, with similar results.\n\nThe paper contributes a novel model for conducting backdoor attacks in the FL setup, and shows that this model is more successful at attacking when training using robust FL algorithms than the standard centralized backdoor attack model. I lean towards accept, as this is a realistic attack model, and as such can further stimulate research into the robustification of FL model aggregation algorithms.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJe4yM7lcr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a distributed backdoor attack strategy, framed differently from the previous two main approches (1) the centralised backdoor approach and (2) the (less discussed in the paper) distributed fault tolerance approach (often named \"Byzantine\").\n\nThe authors show through experiments how their attack is more persistent than centralised backdoor attack.\nThe authors also compare two aggregation rules for federated learning schemes, (Fung et al 2018 & Pillutla et al 2019), suggesting that both rules are bypassed by the proposed distributed backdoor attack.\n\nStrength:\n\nwhat I found most interesting in the paper is Section 3.4, presenting an appreciable attempt to \"interpret\" poisoning. Together with Section 4. \nThis kind of fine-grained analysis of poisoning is highly needed.\n\nWeakness: \n\nin section 3.3, the authors compare against RFA and take what is claimed in Pillulata et al as granted (that RFA detects more nuanced outliers than the wort-case of the Byzantine setting (Blanchard et al 2017) ). In fact, there is more to the Byzantine setting than that, see e.g. Draco (Chen et al 2018 SysML), Bulyan (El Mhamdi et al 2018 ICML) and SignSGD (Bernstein et al 2019 ICLR) which have proposed more sophisticated approches to distributed robustness.\nSince this paper is about distributed robustness and distributed attacks, it would be very informative to the community to illustrate DBA attack on these methods to have a more compelling message.\n\npost rebuttal: thank your for your detailed reply, I acknowledge your new comparisons with the distributed robustness mechanisms of Krum and Bulyan, too bad time was short to compare with the other measures such as Draco and SignSGD.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJl5fZS6YS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies backdoor attacks under federated learning setting. To inject a certain backdoor pattern, existing work generate poisoning samples by blending the same pattern with different input samples. Even for federated learning where the adversary can control multiple parties, such as [1], all parties still use the same global backdoor pattern to generate poisoning samples locally. On the contrary, in this work, they decompose the global pattern into several small local patterns, and each adversarial party only uses a local pattern to generate poisoning samples. In their evaluation, they show that the backdoor attacks generated in this way are more effective, resilient to benign model parameter updates, and also survive better against existing defense algorithms against attacks in federated learning settings.\n\nI think the topic studied in this paper is very important and meaningful, and I am convinced that by decomposing a global pattern into several smaller local pieces, the model parameter updates computed by each party should be more similar to benign updates and thus can better bypass the defense algorithms. Meanwhile, the evaluation is pretty comprehensive and it is good to see that the conducted backdoor attacks are effective. However, when there is no defense deployed in the training process, it is not intuitive to see why the proposed attack is more effective and persistent than the centralized attack, given that a smaller trigger usually results in a worse attack performance. Thus, I would like to see more possible explanation on it. Specifically, I have the following questions for clarification:\n\n1. For the evaluation of DBA, I assume that there are 4 adversarial parties, controlling each of the 4 local triggers. When using centralized attacks, are there still 4 adversarial parties, although they share the same global trigger, or if there is only 1 adversarial party?\n\n2. To evaluate A-S setting, I understand that it may be tricky to enable a fair comparison between the centralized attack and DBA. However, one explanation of why DBA is more persistent in this case is because the adversarial parameter updates happen 4x times compared to the centralized attack. Therefore, another baseline to check is to conduct centralized attacks with the same number of times as DBA, but each update includes 1/4 number of poisoning samples, so that the total number of poisoning samples included to compute the gradient update still stays the same.\n\n3. Can the authors show if the decomposition is also useful for trigger patterns that are not necessarily regular shapes? For backdoor attacks, a line of work studies physical triggers, e.g., glasses in [2]. It is not natural to decompose such kind of patterns into several smaller pieces, unless the performance is significantly boosted.\n\n4. Can the authors show concrete examples on how the attacks are generated? The details are especially unclear on LOAN. Specifically, which features are perturbed, what are the values assigned as the trigger, and what is the corresponding target label?\n\n[1]  Bagdasaryan et al., How to backdoor federated learning.\n[2] Chen et al., Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning.\n\n------------\nPost-rebuttal comments\n\nI appreciate the authors' great effort to address my concerns! I think the evaluation in the current version of the paper is pretty comprehensive and provides a valuable study, and I am happy to raise my score accordingly.\n-------------", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks for the discussion, all. This paper proposes an attack strategy against federated learning. Reviewers put this in the top tier, and the authors responded appropriately to their criticisms. 
Decision: Accept (Poster)