BAFFLE: TOWARDS RESOLVING FEDERATED LEARNING’S DILEMMA - THWARTING BACKDOOR  AND INFERENCE ATTACKS
Authors: Thien Duc Nguyen
Phillip Rieger
Hossein Yalame
Helen Möllering
Hossein Fereidooni
Samuel Marchal
Markus Miettinen
Azalia Mirhoseini
Ahmad-Reza Sadeghi
Thomas Schneider
Shaza Zeitouni
Venue:  (2021)
Keywords: federated learning
secure machine learning
backdoor attacks
inference attacks
data privacy

Abstract: Recently, federated learning (FL) has been subject to both security and privacy attacks posing a dilemmatic challenge on the underlying algorithmic designs: On the one hand, FL is shown to be vulnerable to backdoor attacks that stealthily manipulate the global model output using malicious model updates, and on the other hand, FL is shown vulnerable to inference attacks by a malicious aggregator inferring information about clients’ data from their model updates. Unfortunately, existing defenses against these attacks are insufficient and mitigating both attacks at the same time is highly challenging, because while defeating backdoor attacks requires the analysis of model updates, protection against inference attacks prohibits access to the model updates to avoid information leakage. In this work, we introduce BAFFLE, a novel in-depth defense for FL that tackles this challenge. To mitigate backdoor attacks, it applies a multilayered defense by using a Model Filtering layer to detect and reject malicious model updates and a Poison Elimination layer to eliminate any effect of a remaining undetected weak manipulation. To impede inference attacks, we build private BAFFLE that securely evaluates the BAFFLE algorithm under encryption using sophisticated secure computation techniques. We extensively evaluate BAFFLE against state-of-the-art backdoor attacks on several datasets and applications, including image classification, word prediction, and IoT intrusion. We show that BAFFLE can entirely remove backdoors with a negligible effect on accuracy and that private BAFFLE is practical.

Under review as a conference paper at ICLR 2021
BAFFLE: TOWARDS RESOLVING FEDERATED LEARN- ING'S DILEMMA THWARTING BACKDOOR AND INFERENCE ATTACKS
ABSTRACT
Recently, federated learning (FL.) has been subject to both security and privacy attacks posing a dilemmatic challenge on the underlying algorithmic designs: On the one hand, FL is shown to be vulnerable to backdoor attacks that stealthily ma- nipulate the global model output using malicious model updates, and on the other hand, FL. is shown vulnerable to inference attacks by a malicious aggregator in- ferring information about clients' data from their model updates. Unfortunately, existing defenses against these attacks are insufficient and mitigating both attacks at the same time is highly challenging. because while defeating backdoor attacks requires the analysis of model updates, protection against inference attacks pro- hibits access to the model updates to avoid information leakage. In this work. we introduce BAFFLEF, novel in-depth defense for FL. that tackles this challenge. To mitigate backdoor attacks, it applies a multilayered defense by using a Model Filtering layer to detect and reject malicious model updates and Poison Elimi- nation layer to eliminate any effect of a remaining undetected weak manipulation. To impede inference attacks, we build private BAFFLE that securely evaluates the BAFFLE algorithm under encryption using sophisticated secure computation tech- niques. We extensively evaluate BAFFLE against state-of-the oftte-of-hhe-rtt backdoor attacks on several datasets and applications, including image classification, word predic- tion, and IoT intrusion detection. We show that BAFFLE can entirely remove back- doors with a negligible effect on accuracy and that private BAFFLE is practical.
INTRODUCTION
Federated learning (FL) an emerging collaborative machine learning trend with many applica- tions such as next word prediction for mobile keyboards (McMahan & Ramage 2017), medical imaging (Sheller et al. 20T8a), and intrusion detection for loT (Nguyen et al. D). FL., clients locally train model updates using private data and provide these to a central aggregator who com- bines them to a global model that is sent back to clients for the next training itcration. FL offers efficiency and scalability the training is distributed among many clients and executed in paral- lel (Bonawitz et al., 2019). In particular, FL improves privacy by enabling clients to keep their training data locally (McMahan al. 2017). This is not only relevant for compliance to legal obli- gations such as the GDPR (2018), but also in general when processing personal and sensitive data.
Despite its benefits, FL is vulnerable backdoor (Bagdasaryan al. 2020 Nguyen et al. 2020 Xic et al. 2020) and inference attacks (Pyrgelis et al. [0188] Shokri et al., 2017 Ganju et al. 2018). In the former, the adversary stealthily manipulates the global model so that attacker-chosen inputs result in wrong predictions chosen by the adversary. Existing backdoor defenses, e.g., (Shen et al.) 2016 Blanchard al.) 2017) fail effectively protect against state-of- of-te-of-the-an backdoor attacks, e.g., constrain cetttrain-and-scee acale (Bagdasaryan et al., (2020) and DBA (Xie et al.) (220)). In inference attacks, the adversary aims at learning information about the clients' local data by analyzing their model updates. Mitigating both attack types at the same time is highly challenging due dilemma: Backdoor de- fenses require access to the clients' model updates, whereas inference mitigation strategies prohibit this to avoid information leakage. No solution currently exists that defends against both attacks at the same time (1
Our Goals and Contributions. In this paper, we provide the following contributions:
1. BAFFLE, a novel generic FL defense system that simultaneously protects both the security and the data privacy of FL by effectively preventing backdoor and inference attacks. To the best of our knowledge, this is the first work that discusses and tackles this dilemma, i.c., no existing defense against backdoor attacks preserves the privacy of the clients' data (44)

---

Under review as a conference paper at ICLR 2021
2. To the best of our knowledge, we are the first to point out that combining clustering, clipping, and noising can prevent the adversary to trade-off between attack impact and attack stealthiness. However, the naive combination of these two classes of defenses is not effective to defend against sophisticated backdoor attacks. Therefore, we introduce a novel backdoor defense (ef. Alg. D that has three-folds of novelty: (1) a novel two-layer defense, (2) new dynamic clustering ap- proach (1..)) and (3) a new adaptive threshold tuning scheme for clipping and noising (:3.2) The clustering component filters out malicious model updates with high attack impact while adaptive smoothing, clipping, and noising eliminate potentially remaining malicious model con- tributions. Moreover, BAFFLE is able to mitigate more complex attack scenarios like the simul- taneous injection of different backdoors by several adversaries that cannot be handled in existing defenses (:3).
3. We design tailored efficient secure (two-party) computation protocols for BAFFLE resulting in private BAFFLE, the first rrivacy-rreerviin backdoor defense that also inhibits inference attacks To the best of our knowledge, no existing defense against backdoor attacks preserves the privacy of the clients' data (16).
(:4).
4. We demonstrate BAFFLE's effecttveeess against backdoor attacks through extensive evaluation on various datasets and applications (55). Beyond mitigating state-of-the-art backdoor attacks, we also show that BAFFLE succeeds to thwart adaptive attacks that optimize the attack strategy
to circumvent BAFFLE (55..).
5. We evaluate the overhead of applying secure two-party computation to demonstrate the efficiency of private BAFFLE. A training iteration of private BAFFLE for a neural network with 2.7 million parameters and 50 clients on CIFAR-10 10 takes less than 13 minutes (5.))
BACKGROUND AND PROBLEM SETTING
Federated learning (FL) is a concept for distributed machine learning where K clients and an aggregator A collaboratively build global model G (McMahan et al. [017]. In training round E [1, each client i E [1, K] locally trains a local model W. (with parameters/weights w: ...... w,") based on the previous global model G₁-1 using its local data D, and sends W; to A. Then, A aggregates the received models W; into the new global model G, by averaging the local models (weighted by the number of training samples used to train it): G, = DKK nxxW; where = ||::I, Σ n1 (cf. Alg. 2 and Alg. 3 in | for details). In practice, previous works employ equal weights (n: = n/K) for the contributions of all clients (Bagdasaryan et al. 2020: Xie et al. 2020
We adopt this approach, i.e., we set G. = Σ W.
Adversary model: In typical FL settings, there are two adversaries: malicious clients that try to inject backdoors into the global model and conest-but-curious (a.k.a. semi- semi-honest) aggregators that correctly compute and follow the training protocols, but aim at (passively) gaining information about the training data of the clients through inference attacks (Bonawitz et al. 2017). The former type of adversary A has full control over K' (K' < K clients and their training data, processes, and parameters (Bagdasaryan et al., 2020). A also has full knowledge of the aggregator's operations, including potentially applied backdooring defenses and can arbitrarily adapt its attack strategy at any time during the training like simultancously injecting none, one, or several backdoors. However, A has no control over any processes executed at the aggregator nor over the honest clients. The second adversary type, the conest-but-curing can thus perform model inference attacks on each local model W; to extract information about the corresponding participant's 's data D; used for training W.
aggregator A", has access to all local model updates W,, and
Backdoor attacks. The goals of A are two-fold: (1) Impact: A aims at manipulat- ing the global model G. such that the modified model G² provides incorrect predictions (x) & Gr(x), Va E TAs, where TA= is trigger set specific adversary-chosen inputs. (2) Stealthiness: In addition, A seeks to make poisoned models and benign models indistinguish- able to avoid detection. Model G² should therefore perform normally on all other inputs that are not in the trigger set, i.e., (x) = Gz) Yz LAs, and the dissimilarity (e.g., Euclidean dis- tance) between a poisoned model W and a benign model W must be smaller than a threshold e: ||W' WI| <E
Inference Attacks. The conest-but--uriion aggregator A attempts to infer sensitive information about clients' data D, from their model updates W; (Pyrgelis et al. 2018 Shokri et al. 2017 Ganju
2

---

Under review as a conference paper at ICLR 2021
Global model
Local
Aggregator
Updated global model
Client 1
models
Admitted local models
Plain global model
G
:
W
Dynamic
Wo,... We, Adaptive
G2 Adaptive
Clustering
Clipping
Noising
Client K
Wx
Model Filtering
Poison Elimination
Figure 1: Overview of BAFFLE in round
0.02
0.02
9.02-
(OkAXX-2
0.03
0.01
9.03
(Ch
0.90
0.00
0.00
0.00
0.00
-0.02
Borign
-0.02
Adriitted
-0.02
-0.02
Admittca
0.0
0.5
0.0
0.0
0.0
1cA
I2A Juas
ICA
KIA Juss
(a) Ground truth
(b) K-means
(c) HDBSCAN
(d) BAFFLE
Figure 2:
Comparison of clustering quality for (a) ground truth, (b) using K-means with 2 clusters in
Auror (Shen al. 2016), [c] naively applied HDBSCAN and ( our approach as BAFFLE, The models are
visualized using Independent Component Analysis (ICA) approach (Jutten & Herault 1991).
al. 2018 Carlini al. 2019 Melis et al. 2019) by maximising the information Infer(W,)
that A gains about the data D, of client 7 by interring from its corresponding model W,.
3 BACKDOOR-RESILIET FEDERATED LEARNING
We introduce BAFFLE, a novel defense against backdoor attacks preventing adversary A from achieving attack stealthiness and impact (ef. 12). A can control the attack impact by, e.g., ad- justing the poisoned data rate PDR, i.e., the fraction of poisoned data Da in the training data D 3). or, by tuning the loss-control parameter a that controls the trade-off between backdoor task learning and similarity with the global model (Eq. 4 see DDo details. On one hand, by increasing attack impact, poisoned models become more dissimilar to benign ones, i.e., easier to be detected. One the other hand, if poisoned updates are not well trained the backdoor to remain undetected, the backdoor can be eliminated more casily. BAFFLE exploits this conflict to realize a multilayer backdoor defense shown in Fig. [ and Alg. I The first layer, called Model Filtering (33.I). uses dynamic clustering to identify and remove potentially poisoned model updates having high attack impact. The second layer, called Poison Elimination ($3.2). leverages an adaptive threshold tuning scheme to clip model weights in combination with appropriate noising to smooth out and remove the backdoor impact of potentially surviving poisoned model updates.
3.1 FILTERING POISONED MODELS
The Model Filtering layer utilizes with high attack impact. It overcomes several limitations of existing defenses as (1) it can handle dynamic attack scenarios such as simultancous injection of multiple backdoors, and (2) minimizes false positives. Existing defenses (Blanchard et al. 2017 Shen et al., 2016)) cluster updates into two groups where the smaller group always considered potentially malicious and removed, leading to false positives and reduced accuracy when attack is taking place. More importantly. A may also split compromised clients into several groups injecting different backdoors. A fixed number of clusters bares the risk that poisoned and benign models end up in the same cluster, in particular, models with different backdoors differ significantly. This is shown in Fig. 2 depicting different clusterings of model updates") Fig. 2a shows the ground truth where A uses two groups of clients: 20 clients inject a backdoor and five provide random models to fool the deployed clustering- based defense. Fig. 2b shows how K-means (as used by [Shnn al. (2016) fails to separate benign and poisoned models so that all poisoned ones end up in the same cluster with the benign models.
dynamic clustering approach aiming excluding models
'The models were trained for Fl.-hased Network Intrusion Detection System (NIDS), cf.
3

---

Under review as a conference paper at ICLR 2021
Algorithm BAFFLE
Iaput: G..
K the number clients, Go the initial global model, is number of training iterations b Gy is the updated global model alter I iterations
2:
Output: Gr:
for each training iteration [1.T) (7)4l)
4: for each client i in [1. K] do
W. +- CLIENTUPDATEEGG
The aggregatoe sends Gt-1 to Clicat i who trains Gt-1 using its data D. locally to achieve
local modal W, and sends W. back the aggregalor.
(cII..... ..E.))...KK))>> COSINE
Wi) K. (l, .... K), cz the Cosine distance between W; and W; (b1.....bz bz.) + CWUSTEERINGCC................... CKR) D L the number of admitted models, by are the indices of the admitted models between G2-1 and St the adaptive clipping hound at round t
7:
(el.-..,e<]]
BLCCCLDEENNDSSSTSTAAESSSGGI (W1..... W) es the Euclideun
9:
St < MEDIAN(C),.....
10:
for each client [l, L.
11:
W Ws," MIN(1, St/es,)
> W7, is the admitted model after clipped the adaptive clipping bound St b Aggregating, G, the plain global model before udding noise
12:
Gi + El_ /L
13:
at A 8:
D. Adaptive noising level b Adaptive noising
14:
G. + G7 N(0,0)
Dynamic Clustering. We overcome both challenges by calculating the pairwise Cosine distances measuring the angular differences between all model updates and applying the HDBSCAN clus- tering algorithm (Campello et al. 2013). The Cosine distance is not affected by attacks that scale updates to boost their impact this does not change the angle between the updates. While A can easily manipulate the L2-norms of updates, reducing the Cosine distances decreases the attack impact (Fung et al., 2018). HDBSCAN clusters the models based on their density and dynamically determines the required number of clusters. This can also be a single cluster, preventing false pos- itives the absence of attacks. Additionally, HDBSCAN labels models as noise if they do not fit into any cluster. This allows BAFFLE to efficiently handle multiple poisoned models with different backdoors by labeling them as noise to be excluded. We select the minimum cluster size to be at least 50% of the clients, i.c., 2 + 1, s.t. it contains the majority of the updates (which we assume to be benign, ef. (2). All remaining (potentially poisoned) models are marked as outliers. This behavior is depicted in Fig. 2d where the two benign clusters C and D from Fig. 2c merged into one cluster while both malicious and random contributions are labeled as outliers. Hence, to the best of our knowledge, our clustering is the first FL backdoor defense for dynamic attacks where the number of injected backdoors varies. The clustering step is shown in Lines 6-7 of Alg. where L models (W...... Wo,) are accepted.
K
3.2 RESIDUAL POISON ELIMINATION BY SMOOTHING
The Model Filtering layer ( (3.11) eliminates contributions of poisoned model updates that are not filtered out by adaptive clipping and noising. contrast to existing defenses that empirically specify static clipping bound and noise level (and have been shown to be ineffective (Bagdasaryan et al.) 2.200). we automatically and adaptively tune these to effectively eliminate backdoors. Our design also resilient to adversaries that dynamically adapt their attack.
Backdoor embedding makes poisoned models different from benign models. Clipping and noising can be combined to smooth model updates and remove these differences (McMahan et al. 2018). Clipping scales down the model weights to a clipping bound St W; + W; * MIN(1, S/c;) where e is the Euclidean distance (Lg-norm, Def. [ between W, and G Noising refers technique that adds noise to a model (controlled by noise level a): W = W + (O, σ), where NO0, a) is noise generation function, e.g.. the Gaussian distribution. While clipping and noising can renove backdoors, previous works (Bagdasaryan et al., (0200 also show that they reduce the global model is challenging to find an appropriate clipping
accuracy on the main task, making it unusable. bound Š and noise level a that strikes a balance between the accuracy of the main task and effec- tiveness of the backdoor defense. Both need to be dynamically adapted to model updates in different training iterations and different datasets ($F.T) ) as well as to dynamic adversaries constantly chang- ing their attack strategy (Bagdasaryan et al.) 2020). Note that this use of clipping and noising is different from differential privacy (DP; Dwork & Roth (2014); McMahan al. (2018)) protecting the confidentiality of clients' data from a curious aggregator and where clients truthfully train their models. In contrast, scenario concerns malicious clients that intentionally try to backdoor FL. To overcome these challenges, we design our Poison Elimination layer for BAFFLE s.t. it automatically determines appropriate values for the clipping bound S and the noise level
4

---

Under review as a conference paper at ICLR 2021
Arthmetic shares (FI)
Aggregihorr
Word Prosiction
Dynamic
Adlapt
Adlapt
1.norms
Clustering
Clopping
Notting
20-
Classfication
Cleet
Undated
10-
model
Gobal
Clustering
Dynamic
07
Adapciiv
Clipping
Extarnal Server B
Round
Figure 3: Lg-norms depending on the num- ber of training rounds for different datasets.
Figure 4: Overview of private BAFFLE in round t using Secure-Two-Parne Two-Party Computation (STPC).
Adaptive Clipping. Fig. shows the variation of the average L2-norms of model updates of benign clients in three different datasets over subsequent training rounds. This shows that the Lz-norms get smaller after each training iteration. To effectively remove backdoors while preserving benign updates unchanged, the clipping bound and noise level must dynamically adapt to this decrease in the Lo-norm. We design an adaptive selection of the clipping threshold S, for the Ly-norm for each training iteration I. The aggregator selects the median of the L2-norms of the model updates ....... WK) classified as benign the clustering of our Model Filtering layer at iteration As assume that the majority of clients is benign, this ensures that S; is determined based on benign model even some malicious updates were not detected during clustering. We formalize clipping scheme follows: W Wb, MIN(1, st/es,), where S, = MEDIAN(CL,.... eL) in iteration f, see Lines 8-11 of Alg. 1 for details. By using the median, we ensures that the chosen clipping bound S, is always computed between benign local model and the global model since we assume that more than 50% of clients are benign. We evaluate the effectiveness of our adaptive clipping approach in [
Adaptive noising, We introduce a novel adaptive approach to calculate an appropriate level of noise based on the clipping bound S, in iteration We select the commonly used Gaussian distribution to generate noise that is added to the global model. Let a be the noise level and let À be a parameter indicating the product of a and the clipping bound S.. Our adaptive noise addition is formalized as = AS, for a clipping bound St and a noise level factor X, see Lines 13-14 of Alg. [ for details. In F. we empirically determine = 0.001 for image classification and word prediction, and x = 0.01 for the IoT datasets.
follows: G2 = G; + N(0.00 o), where
PRIVACY-PRESERVIT FEDERATED LEARNING
Inference attacks threaten the privacy of FL (cf. 2 They enable the aggregator to infer sensitive information about the clients' training data from the local models. So far, existing defenses against model inference attacks either contradict with backdoor defenses and/or are inefficient (ef. (6). Generally, there are two approaches to protect the privacy of clients' data: differential privacy (DP: Dwork & Roth 2014) and secure two-party computation (STPC: Yao (1986); Goldreich et al. (1987). DP is a statistical approach that can be efficiently implemented, but it can only offer high privacy protection at the cost of a significant loss in accuracy due to the noise added to the mod- els (Zhang et al. 2020 Aono al. 2017 So et al. 2019). In contrast, STPC provides strong privacy guarantees and good efficiency but requires two non-colluding servers. Such servers can, for example, be operated by two competing companies that want to jointly provide a private FL service. STPC allows two parties to securely evaluate function on their encrypted inputs. Thereby, the parties have only access so-called secret-shares of the inputs that are completely random and therefore do not leak any information besides the final output. The real value can only be obtained both shares are combined. To provide best efficiency and reasonable security, we chose STPC for private BAFFLE. Alternatively, also more parties can be used in order to achieve better security at the cost of lower efficiency.
(
For realizing BAFFLE with STPC, we co-design all components of BAFFLE as efficient STPC pro- tocols. This requires to represent all functions that have to be computed with STPC as Boolean circuits. We use three STPC protocols in order to achieve good efficiency: Arithmetic sharing (orig- inally introduced by Goldreich et al. ( (1987) for linear operations as well as Boolean sharing (also originally introduced by Goldreich et al. (1987) and Yao's Garbled Circuits (GC, originally intro- duced by Yao (986)) for non-linear operations. To further improve performance, we approximate HDBSCAN with the simpler DBSCAN (Ester et al. 1996) to avoid the construction of the minimal
5

---

Under review as a conference paper at ICLR 2021
spanning tree in HDBSCAN as it is very expensive to realize with STPC. Additionally, lower level, we generate a novel (previously not existing) circuit for square root computation needed for determining cosine and Lo-norm distances using conventional logic synthesis tools. We carefully implement the circuit using Verilog HDL. and compile it with the Synopsys Design Compiler (DC) 2010) in a highly efficient way. We customize the flow of the commercial hardware logic synthesis tools to generate circuits optimized for GC including its state-of-the-an optimizations such as point- and-permute (Beaver et al. (190), free-XOR (Kolesnikov & Schneider 2008), FastGC (Huang et al.) 2011), fixed-key AES (Bellare et al. 2013). and half-gates (Zahur et al. 2015). For example, for the Free-XOR technique (Kolesnikov & Schneider [2008), which enables the evaluation of XOR gates without costly cryptographic encryption and thus makes GCs much more efficient, one has to minimize the number of non-XOR gates the Boolean representation. We developed technology library to guide the mapping of the logic to the circuit with no manufacturing rules defined similarly as in (Songhori et al. 2015; Demmler et al., 2015a). More concretely, to generate efficient Boolcan circuits for BAFFLE, we constrained the mapping to free XOR gates and non-free AND gates. We enhanced the cost functions of the single gates: We set the delay and area of XOR gates to O, the delay and area of the inverters to 0 (as they can be replaced with XOR gates with the constant input 1), and the delay and area of AND gates to a non-O value. Note that the logic synthesis tool outputs standard Boolean netlist containing cells that are included in the cell library. To use the netlist in a STPC framework (Demmler et al. (0155), we performed post-synthesis. This circuit construction as well as the new circuit are also of independent interest. The new circuit can be used for other appli- cations that need a Euelidean distance like privacy prrecccinen face recognition (Osadchy et al., 2010). Moreover, the circuit construction chain is interesting for any other circuit that needs to be created and optimized for the GC protocol.
computation of square roots (e.g., any protocol that uses the
Private BAFFLE. To summarize, the distance calculation, clustering, adaptive clipping, and aggre- gation steps of BAFFLE (cf. Alg. [ are executed within STPC to protect the privacy of the clients' training data. Our goal is to hide the local models from the aggregator A to prohibit inference attacks on clients' local training data. Fig. 4 shows an overview of private BAFFLE. It involves K clients and two non-colluding servers, called aggregator A and external server B. Each client i E (1...., K} splits the parameters of W, into two Arithmetic shares (X)²² and (X),² such that W; = (X) + (X), and sends (X)² to and (X) to B. A and B then privately compute the next global model via STPC. Our resulting private BAFFLE is not only the most effective but also the first
backdoor defense for FL. We give further details in C
EVALUATION
We implemented all experiments with the PyTorch framework (py 2019) and used the attack source
code provided by [Bagaasaryan et al.] (2020) and Xie et al. 2020). We reimplemented existing defenses to compare them with BAFFLE. All experiments that evaluate BAFFLE'S effectiveness in defending backdoors were run on a server with 20 Intel Xeon CPU cores, 192 GB RAM, NVIDIA GeForce GPUs (with 11 GB RAM each), and Ubuntu 18.04 LTS OS.
Following previous work on FL and backdooring, we evaluate BAFFLE on three typical appli- cations: word prediction (McMahan & Ramage, (017)) using LSTM trained on the Reddit
dataset (red (207)), image classification (Bagdasaryan et al.) 2020 Xie et al. 2020) using the CIFAR-IO (Krizhevsky & Hinton 2009) MNIST (LeCun al. 1998). and Tiny-ImageNet datasets with different architectures, and IoT network intrusion detection (NIDS; Nguyen et al. (2020). In
E. we detail all datasets used in this work and the experimental setup. In short, we emphasize that we do not make any assumption about the data distribution, i.c., BAFFLE is successful in mitigating backdoors in FL independent of if the clients hold unbalanced and non independent and identically distributed (non-III) datasets. For example, in our experimental setup for the Reddit dataset, each client holds the posts of a Reddit user. Users have different styles of writing and their posts contain different content. Moreover, the number of posts of each user and their sizes (number of words) of posts are also different. Therefore, clients hold non-III and unbalanced data (cf. {E.1) For the image classification dataset, evaluate the impact of the degree of non-iid data (cf. F.I 2nd paragraph). It shows that BAFFLE is effective and independent of the data distribution. For the loT dataset, each client holds different chunk of traffic from different IoT devices.

---

Under review as a conference paper at ICLR 2021
To measure the effectiveness of the backdoor attacks and defenses, consider various metries: Backdoor Accuracy (BA), Main Task Accuracy (MA), Poisoned Data Rate (PDR), Poisoned Model Rate (PMR), True Positive Rate (TPR), and True Negative Rate (TNR) (all values as percentages) as detailed in F.2
5.1 PREVENTING BACKDOOR ATTACKS
Effectiveness of BAFFLE. We evaluate BAFFLE against the state-of-the-art backdoor attacks called
constrain-amn -nd-- (Bagdasaryan et al., 2020 and DBA (Bagdasaryan et al. al.) (200)) (cf. (D) using the same attack settings with multiple datasets (cf. Sab. E.. The results are shown in Tab. I BAFFLE completely mitigates the constrain-and, -nd-selle attack (BAA = 0%) for all datasets. The DBA attack is also successfully mitigated (BA 3.2%, more experiments in (..9). Moreover, our defense does not affect the main task performance of the system as the Main Task Accuracy (MA) reduces by
less than 0.4% in all experiments. BAFFLE is also effective in mitigating state-of-the-art untargeted poisoning attacks (MA increases by 44.59%, more details in F.5
We extend evaluation var- ious backdoors on three datasets. For NIDS, we evaluate 13 differ- ent backdoors and 24 device types (cf. (F and F...Ip, for word pre- diction 5 different word backdoors (cf. (7.). and for image classifica- tion 90 different image backdoors, which change the output of a whole class to another class (cf. {F.8.. In all cases, BAFFLE successfully mit- igates the attack while still preserving the MA.
Table I: Effectiveness of BAFFLE against state-of-the-art at- tacks for the respective dataset, in terms of Backdoor Accu- racy (BA) and Main Task Accuracy (MA).
No Defense
BAFFLE
Dataset
Attack
BA
MA
BA
MA
Reddit
100
22.6
22.3
Constrain-and-scale
CIFAR-10
81.9
89.8
91.9
ToT-Traffic
100.0
100.0
99.8
DBA Untargeted Poisoning
CIFAR-10 100
93.8
57.4
3.2
76.2
CIFAR-10
46.72
91.31
Comparison to existing defenses. We compare BAFFLE exist- ing defenses: Krum (Blanchard al. 2017). FoolsGold (Fung
Table 2: Effectiveness of BAFFLE in comparison to state-of- the-art defenses for the constrain-and -sccscell attack on three datasets, in terms of Backdoor Accuracy (BA) and Main Task Accuracy (MA).
let al. 2018). Auror (Shen et al. 2016). Adaptive Federated Averag- ing (AFA; Muñoz- et al. (2019), and generalized differ- ential privacy (DP) approach (Bag-) dasaryan et al.) 2020 McMahan al. 2018).
Reddit
CIFAR- 10
Ira-fraffic
Defenses
BA
MA
BA
MA
BA
MA
Benign Setting
22.7
92.2
100.0
No defense
100.0
22.6
81.9
89.8
100.0
100.0
Krum
100.0
9.6
100.0
56.7
100.0
84.0
FoolsGold
0.0
22.5
100.0
52.3
100.0
99.2
Auror
100.0
22.5
100.0
100.0
96.6
Tab. 2 shows
AFA
100.0
22.4
0.0
91.7
100.0
87.4
that BAFFLE is effective for all
DP
14.0
18.9
0.0
78.9
14.8
82.3
datasets, while previous works fail to mitigate backdoor attacks: BA mostly negligibly affected. Krum, FoolsGold, Auror, and AFA do not effectively remove poisoned models and BA often remains at 100%. Additionally, the model's 's MA is negatively impacted. These previously proposed defenses remove many benign updates (cf. (F.I) increasing the PMR and ren- dering the attack more successful than without these defenses.
BAFFLE
0.0
22.3
0.0
91.9
0.0
99.8
For example, Reddit's users likely provide different texts such that the distances between benign models are high while the distances between poisoned models are low as they are trained for the same backdoor, FoolsGold is only effective on the Reddit dataset (TPR 100%) because it works well on highly non-independent and identically distributed (non-IlD) data (cf. {G Similarly, AFA only mitigates backdooring on the CIFAR-10 10 dataset since the data are highly IID (each client is assigned a random set of images) such that the benign models share similar distances to the global model (cf. (6). The differential privacy-based defense is effective, but it significantly reduces MA. For example, it performs best the CIFAR- 10 dataset with BA 0. but MA decreases to 78.9% while BAFFLE increases MA to 91.9% which is close to the benign setting (no attacks), where MA 92.2%.

---

Under review as a conference paper at ICLR 2021
Table 3: Runtime in seconds of standard BAFFLE (S) in comparison to private BAFFLE using STPC
(P). K' is the number of participating clients. Note that the model size has no effect on the clustering.
Cosine Dissance
Euclidean Distance
Clipping Model Aggregation
Clustering
Roddit
CIFAR-TO
Tol-Irafli
Reddit
CIFAR-TO
Iol-Iraflic
(3)
(P)
(S)
(P)
(S)
(P)
(S)
(P)
(8)
(P)
(S)
(P)
(3)
(P)
1.91
297.91
70.00
67.67
61.29
0004
50
50.94
$259.29
0.80
603.54
0.32
192.47
11.61
594.57
1.82
120.74
0.37
35.04
0.004
41.84
213.30
20/50043
2094.51
554.97
38.82
1267.35
5.89
219.85
1.03
68.12
0.005
253.87
Resilience to Adaptive Attacks. Given sufficient knowledge about BAFFLE, adversary may seek to use adaptive attacks to bypass the defenses. We analyze and evaluate various scenarios and strategies including changing the injection stralegy, model alignment, and model obsuscation. Our evaluation results show that BAFFLE is resilient, i.e., mitigates all these attacks effectively (cf. F.2
5.2 EFFECTIVENESS OF BAFFLE'S COMPONENTS
Resilience of our in-depth defense approach. To evaluate the effectiveness of our combination of Model Filtering and Poison Elimination, we conduct experiments in which a sophisticated adversary can freely tune the attack parameter PDR in order to find a setting that evades the filtering layer while still achieving a high BA. We show that the residual poisoned updates are eliminated by Poison Elimination in this case. We run experiments covering the full range of PDR values to assess each defense component's effectiveness as well as the complete BAFFLE defense on the IoT-Traffic datasets. The
-ad--ccal attack is used with the same settings in 1.11
Fig. 5 shows the BA when using BAFFLE and its individ- ual components depending on the PDR values. As can be seen, Model Filtering can reliably identify poisoned mod- els if PDR is above 13%. Below this point, Model Filter- ing becomes ineffective as poisoned models become too indistinguishable from benign ones and cannot be reliably identified. Below this PDR level, however, Poison Elim- ination can effectively remove the impact of poisoned models. Its performance only decreases when PDR is increasing, and the impact of the backdoor functionality harder to eliminate. However, our BAFFLE effectively combines both defense layers and remains successful for
100%
No Deferse
g
50%
Palson Ellmination (53.2)
Model Filtering
BAFFLE (53.1 + $3.2)
0%
10%
20%
30%
40%
50%
Poisoned Data Rate (PDR)
Figure 5: Resilience of each defense layer in comparison to an effective com- bination in BAFFLE, measured by Back- door Accuracy (BA).
all PDR levels as BA consistently remains close to 0%. Due to space limitations, a detailed eval- uation of the individual components of BAFFLE is given in F.III In summary, we investigate the effectiveness of each of the components of BAFFLE (i.e., clustering, clipping, and noising) and justify our algorithms and threshold choices. For clustering, clustering approach performs well on all datasets while previous works often fail to successfully defend backdoor attacks are only effective on specific dataset. For clipping, we compare our adaptive clipping bound to the static approach as well as to other potential thresholds. Fig. 7 shows that using the median Euclidean threshold can effectively mitigate backdoors while retaining the main task accuracy. Moreover, we have run an experiment to compare the effectiveness of different values and noise levels and depict the results Fig. g shows that our adaptive noise is not only effective to impair backdoors but also retain the performance of the global model in the main task.
evaluation results show that our
5.3 PERFORMANCE OF PRIVATE BAFFLE
We evaluate the costs and scalability of BAFFLE when executed in a privacy-preserving manner by varying the number/size of the parameters that affect the three components realized with secure two- party computation (STPC) (cf. CC.II. For our implementation, we use the ABY framework (Demm- ler et al. 20155). All STPC results are averaged over 10 experiments and run two separate servers with Intel Core 7960X CPUs with 2.8 GHz and 128 GB RAM connected over a 10 Gbit/s LAN with 0.2 ms RTT.
8

---

Under review as a conference paper at ICLR 2021
Tab. 3 shows the runtimes in seconds per training iteration of the Cosine distance, Euclidean distance + clipping + model aggregation, and clustering steps of Alg. I in standard (without STPC) and in private BAFFLE (with STPC). The communication costs given in qF.IT] As can be seen, private BAFFLE causes a significant overhead on the runtime by factor of up to three orders of magnitude compared the standard (non-private) BAFFLE. However, even if we consider the largest model (Reddit) with K = 100 clients, we have total server-side runtime of 22 08 188.655 seconds (x 6 hours) for a training iteration with STPC. Such runtime overhead would be acceptable to maintain privacy, especially since mobile phones, which would be typical type of clients in FL (McMa- han et al., 2017), are in any case not always available and connected that there will be delays in synchronizing clients' model updates in FL. These delays can then also be used to run STPC. Fur- thermore, achieving provable privacy by using STPC may even motivate more clients to contribute to FL in the first place and provide more data.
Secondly, we measure the effect of approx- imating HDBSCAN by DBSCAN includ- ing the binary search for the neighborhood parameter (details given in (C.. The results are shown in Tab. 4 As it can be seen, the results very similar. For some applications, the approximation even performs slightly better than the standard BAFFLE. For example, for CIFAR-10, 10, pri- vate BAFFLE correctly filters all poisoned models, while standard BAFFLE accepts small number (TNR 86.2%), which is still sufficient to achieve BA 0.0%. To conclude, private BAFFLE is the first privacy-preserving backdoor defense for FL with significant but manageable overhead and high effectiveness.
Table 4: Effectiveness, in terms of Backdoor Accu- racy (BA)) Main Task Accuracy (MA), True Positive Rate (TPR), and True Negative Rate (TNR), of stan-
dard BAFFLE (S) in comparison to private BAFFLE us-
ing STPC (P) in percent.
Reddit
CIFAR-10
ToT-Traffic
(S)
(P)
(S)
(P)
(S)
(P)
BA
0.0
0.0
0.0
0.0
MA
22.3
22.2
91.9
91.7
99.8
99.7
TPR
22.2
20.4
23.8
40.8
59.5
51.0
TNR
100.0
100.0
86.2
100.0
100.0
100.0
RELATED WORK
Backdoor Defenses. Several backdoor defenses, such as Krum (Blanchard et al., (017). Fools- Gold (Fung et al. 2018), Auror (Shen al. (2066). and AFA MMunoz-Gonzáler et al. [2019) ) aim at separating benign and malicious model updates. However, they only work under specific as- sumptions about the underlying data distributions, e.g., Auror and Krum assume that data of benign clients are independent and identically distributed (IID). In contrast, FoolsGoll and AFA assume that benign data are non-IID. In addition, FoolsGold assumes that manipulated data is IID. As a re- sult, they are only effective in specific circumstances (cf. 15.1) and cannot handle the simultaneous injection of multiple backdoors (cf. 13.1.. In contrast, BAFFLE does not make any assumption about the data distribution (cf. (.I and can defend against injection of multiple backdoors (cf. (3.).
Clipping and noising are known techniques to achieve differential privacy (DP) (Dwork & Roth 2014 Carlini & Wagner| 2018). However, directly applying these techniques to defend against backdoor attacks is not effective because they significantly decrease the Main Task Accuracy ($5.T) BAFFLE tackles this by (i) identifying and filtering out potential poisoned models that have a high attack impact (cf. {3.1) and (ii) eliminating the residual poison with an appropriate adaptive clipping bound and noise level, such that the Main Task Accuracy is retained (cf. (3.2)
Defenses against Inference Attacks in FL. Bonawitz et al. (2017) use expensive additive masking and secret sharing hide local updates. Similarly. Chase al. (2017) train DNN in a private collaborative fashion by combining multi-party partyccomputation differential privacy (DP), and secret sharing assuming non-colluding conest-but-curicnn clients. However, both works are vulnerable to backdoor attacks as they prevent the aggregator from inspecting the model updates. DP (McMahan ctt al., 2018) limits the success of membership inference attacks that test if a specific data record was used in the training. However, previous works (Melis et al. 2019 Nasr et al.) 2019) have shown that this is only successful when thousands of clients are involved or for black- box attacks in which the adversary has access to model parameters. In private BAFFLE, local model updates are analyzed under encryption, thus the aggregating servers cannot access the updates to run inference attacks while thwarting backdooring.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Synopsys inc. design http://///////.///ssss/////////r wwwp:atheeissDeiiinnnooooo 2010.
Reddit dataset, 2017. htteas///aggaerrr..ooooooooeeoe2
bi gquery: reddit wwwdrr_ccodeenneeeeeeeeeeeeeeeee commentss
General Data Protection Regulation, 2018.
Bttp:///err.eex..rrrgp....////
2016/679/0j
Pytorch, 2019. https://pyttrrch....
Nitin Agrawal, Ali Shahin Shamsabadi, Matt J Kusner, and Adrià Gascón. QUOTIENT: Two-Party Secure Neural Network Training and Prediction. In Conference on Computer and Communica- fions Security (CCS). ACM, 2019.
Manos Antonakakis, Tim April, Michael Bailey, Matt Bernhard, Elie Bursztein, Jaime Cochran, Zakir Durumeric, J. Alex Halderman, Luca Invernizzi, Michalis Kallitsis, Deepak Kumar, Chaz Lever, Zane Ma, Joshua Mason, Damian Menscher, Chad Seaman, Nick Sullivan, Kurt Thomas, and Yi Zhou. Understanding the Mirai Botnet. In USENIX Security. Usenix Association, 2017.
Yoshinori Aono, Takuya Hayashi, Lihua Wang, and Shiho Moriai. Privacy-preserving Deep Learn- ing via Additively Homomorphic Encryption. In Transactions on Information Forensics and Security (TIFS). IEEE, 2017.
Gilad Asharov, Yehuda Lindell, Thomas Schneider, and Michael Zohner. More Efficient Oblivi- ous Transfer and Extensions for Faster Secure Computation. In Conference on Computer and Communications Security (CCS). ACM, 2013.
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How To Backdoor Federated Learning. AISTATS. PMLR, 2020.
Moran Baruch, Gilad Baruch, and Yoav Goldberg. A Little Is Enough: Circumventing Defenses For Distributed Learning. In NIPS, 2019.
Donald Beaver. Efficient Multiparty Protocols Using Circuit Randomization. In Annual Interna- tional Cryptology Conference (CRYPTO). Springer, 1991.
Donald Beaver, Silvio Micali, and Phillip Rogaway. The Round Complexity of Secure Protocols. In ACM Symposium on Theory of Computing, 1990.
Mihir Bellare, Viet Tung Hoang, Sriram Keelveedhi, and Phillip Rogaway. Efficient garbling from fixed-key blockcipher. In IEEE Symposium on Security and Privacy. IEEE, 2013.
Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent. In NIPS, 2017.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical Secure Aggregation for Privacy- Preserving Machine Learning. In Conference Computer and Communications Security (CCS). ACM, 2017.
Keith Bonawitz, Hubert Fichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloé Kiddon, Jakub Konecný, Stefano Mazzocchi, Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards Federated Learning at Scale: System Design. In arXiv preprint: 1902..0:::000.0000 1902. 01046, 2019. UtL.http:://axx...ooo
abs/1902.01044
Ricardo J. G. B. Campello, Davoud Moulavi, and Joerg Sander. Density- Clustering Based on Hierarchical Density Estimates. In Pacific-Asia Conference on Knowledge Discovery and Data Mining (KDD). Springer, 2013.
N. Carlini and D. Wagner. Audio Adversarial Examples: Targeted Attacks on Speech-to-Tess In arXiv preprint: 1801. 01944, 2018. htRpst:::: /ttp:://///vvv.///
wwtp://wwwwwwgg//////eeeae .0 944
10

---

Under review as a conference paper at ICLR 2021
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jemej Kos, and Dawn Song. The Secret Sharer: Evaluating and Testing Unintended Memorization Neural Networks. In USENIX Security. Usenix Association, 2019.
Melissa Chase, Ran Gilad- Bachrach, Kim Laine, Kristin Lauter, and Peter Rindal. Private Collab- orative Neural Network Learning. In Cryptology ePrint Archive, Report 2017/762, 2017. URL
https:://ppring iacr. org/2017/762
Trishul Chilimbi, Yutaka Suzue, Johnson Apacible, and Karthik Kalyanaraman. Project Adam: Building an Efficient and Scalable Deep Learning Training System. In USENIX Symposium on Operating Systems Design and Implementation. Usenix Association, 2014.
Daniel Demmler, Ghada Dessouky, Farinaz Koushanfar, Ahmad-Reza Sadeghi, Thomas Schneider, and Shaza Zeitouni. Automated Synthesis of Optimized Circuits for Secure Computation. In Conference on Computer and Communications Security (CCS). ACM, 2015a.
Daniel Demmler, Thomas Schneider, and Michael Zohner. ABY A Framework for Efficient Mixed- Protocol Secure Two-Party Computation. In Network and Distributed System Security Symposium (NDSS). Internet Society, 2015b.
Rohan Doshi, Noah Apthorpe, and Nick Feamster. Machine Learning DDoS Detection for Con- sumer Internet of Things Devices. arXiv preprint:1804.04 040500 2018. URL http::/arxii
org/abs/ 1804. 04159
Cynthia Dwork and Aaron Roth. The Algorithmic Foundations of Differential Privacy. In Founda- fions and Trends in Theoretical Computer Science. now publishers inc., 2014.
Fabienne Eigner, Aniket Kate, Matteo Maffei, Francesca Pampaloni, and Ivan Pryvalov. Differen- tially Private Data Aggregation with Optimal Utility. In Annual Computer Security Applications Conference (ACSAC). ACM, 2014.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A Density-Based Algorithm for Discovering Clusters Large Spatial Databases with Noise. International Conference on Knowledge Discovery and Data Mining (KDD). Springer, 1996.
Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. Local Model Poisoning Attacks to Byzantine- Robust Federated Learning. In USENIX Security. Usenix Association, 2020.
Clement Fung, Chris M. Yoon, and Ivan Beschastnikh. Mitigating Sybils in Federated Learning Poisoning. In arXiv preprint: 1 1808 048.644666 2018. URL http://aaxiw.orr/aabb//// 04866
Karan Ganju, Qi Wang, Wei Yang, Carl A Gunter, and Nikita Borisov, Property Inference Attacks on Fully Connected Neural Networks Using Permutation Invariant Representations. In Conference Computer and Communications Security (CCS). ACM, 2018.
Oded Goldreich, Silvio Micali, and Avi Wigderson. How to Play any Mental Game In ACM Symposium Theory of Computing (STOC). ACM, 1987.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In IEEE Conference Computer Vision and Pattern Recognition (CVPR). IEEE, 2016.
Stephen Herwig, Katura Harvey, George Hughey, Richard Roberts, and Dave Levin. Measurement and Analysis of Hajime, a Peer-to-Peer loT Botnet. In Network and Distributed System Security Symposium (NDSS). Internet Society, 2019.
Yan Huang, David Evans, Jonathan Katz, and Lior Malka. Faster secure two-party computation using garbled circuits. In USENIX Security Symposium, 2011.
Russell Impagliazzo and Steven Rudich. Limits on the Provable Consequences of One-way Permu- tations. In ACM Symposium Theory of Computing (STOC). ACM, 1989.

---

Under review as a conference paper at ICLR 2021
Christian Jutten and Jeanny Herault. Blind Separation of Sources, Part I: An Adaptive Algorithm Based on Neuromimetic Architecture. In Signal Processing. Elsevier, 1991.
Chiraag Juvekar, Vinod Vaikuntanathan, and Anantha Chandrakasan. GAZELLE: A Low Latency Framework for Secure Neural Network Inference. In USENIX Security. Usenix Association, 2018.
Vladimir Kolesnikov and Thomas Schneider. Improved garbled circuit: Free xor gates and applica- tions. In International Colloquium on Automata, Languages and Programming, 2008.
Constantinos Kolias, Georgios Kambourakis, Angelos Stavrou, and Jeffrey Voas. DDoS in the loT: Mirai and Other Botnets. In IEEE Computer, 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning Multiple Layers of Features from Tiny Images. Technical report, 2009.
Nishant Kumar, Mayank Rathee, Nishanth Chandran, Divya Gupta, Aseem Rastogi, and Rahul Sharma. CrypTFlow: Secure Tensorflow Inference. In IEEE Symposium on Security and Privacy (S&P). IEEE, 2020.
Peeter Laud. Parallel Oblivious Array Access for Secure Multiparty Computation and Privacy- Preserving Minimum Spanning Trees. In Privacy Enhancing Technologies Symposium (PETS). SCIENDO, 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-basse Learning Applied to Document Recognition. Proceedings of the IEEE, 1998.
Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William Dally. Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training. In ICLR, 2018.
Jian Liu, Mika Juuti, Yao Lu, and Nadarajah Asokan. Oblivious Neural Network Predictions via MiniONN Transformations. In Conference on Computer and Communications Security (CCS). ACM, 2017.
Brendan McMahan and Daniel Ramage. Federated learning: Collaborative Machine Learning with- out Centralized Training Data. In Google Research Blog. Google AI, 2017.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communicttion-Effiiieen Learning of Deep Networks from Decentralized Data. In AISTATS. PMLR, 2017.
H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning Differentially Private Language Models Without Losing Accuracy. In ICLR, 2018.
Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting Unin- tended Feature Leakage Collaborative Learning. In IEEE Symposium on Security and Privacy (S&P). IEEE, 2019.
Pratyush Mishra, Ryan Lehmkuhl, Akshayaram Srinivasan. Wenting Zheng, and Raluca Ada Popa. DELPHI: A Cryptographic Inference Service for Neural Networks. In USENIX Security. Usenix Association, 2020.
Payman Mohassel and Yupeng Zhang. SecureML: A System for Scalable Prenecc-rreeeiingg Ma- chine Learning. In IEEE Symposium Security and Privacy (S&P). IEEE, 2017.
Luis Muñoz-González, Kenneth T. Co, and Emil C. Lupu. Byzantine Robust Federated Machine Learning through Adaptive Model Averaging. In arXiv preprint:1999.00 05125, 2019, htth:tp:
05125
Moni Naor and Benny Pinkas. Computationally Secure Oblivious Transfer. Journal of Cryptology, 2005.
M. Nasr, R. Shokri, and A. Houmansadr. Comprehensive privacy analysis of deep learning: Pas- sive and active boxte-box inference attacks against centralized and federated learning. In IEEE Symposium Security and Privacy (S&P). IEEE, 2019.
12

---

Under review as a conference paper at ICLR 2021
Thien Due Nguyen, Samuel Marchal, Markus Miettinen, Hossein Fereidooni, N. Asokan, and Ahmad- Ramad-Reza Sadeghi. DIoT: A Federated Self-learning Anomaly Detection System for loT. In International Conference on Distributed Computing Systems (ICDCS). IEEE, 2019.
Thien Duc Nguyen, Phillip Rieger, Markus Miettinen, and Ahmad- Sadeghi. Poisoning Attacks on Federated Learning- Based IoT Intrusion Detection System. In Workshop on Decentralized loT Systems and Security @ Network and Distributed System Security Symposium (NDSS). Internet Society, 2020.
Margarita Osadchy, Benny Pinkas, Ayman Jarrous, and Boaz Moskovich. Scici-a system for secure face identification. In IEEE Symposium on Security and Privacy, 2010.
Apostolos Pyrgelis, Carmela Troncoso, and Emiliano De Cristofaro. Knock Knock, Who's There? Membership Inference Aggregate Location Data. In Network and Distributed System Security Symposium (NDSS). Internet Society, 2018.
Jianji Ren, Haichao Wang, Tingting Hou, Shuai Zheng, and Chaosheng Tang. Federated Learning- Based Computation Offloading Optimization in Edge Suappring-Supporeed Internet of Things. In IEEE Access, 2019.
M Sadegh Riazi, Mohammad Samragh, Hao Chen, Kim Laine, Kristin Lauter, and Farinaz Koushan- far. XONN: XNOR-based Oblivious Deep Neural Network Inference. In USENIX Security. Usenix Association, 2019.
Sumudu Samarakoon, Mehdi Bennis, Walid Saad, and Merouane Debbah. Federated Learning for Ultra-Relable Low-Latency V2V Communications. In Global Communications Conference (GLOBCOM). IEEE, 2018.
Joseph Schneible and Alex Lu. Anomaly Detection on the Edge. In IEEE Military Communications Conference. IEEE, 2017.
Thomas Schneider and Michael Zohner. GMW vs. Yao? Efficient Secure Two-Party Computation with Low Depth Circuits. In Financial Crypto. Springer, 2013.
Micah Sheller, Anthony Reina, Brandon Edwards, Jason Martin, and Spyridon Bakas. Federated Learning for Medical Imaging. In Intel AI, 2018a.
Micah Sheller, Anthony Reina, Brandon Edwards, Jason Martin, and Spyridon Bakas. Multi- Institutional Deep Learning Modeling Without Sharing Patient Data: A Feasibility Study on Brain Tumor Segmentation. In Brain Lesion Workshop, 2018b.
Shiqi Shen, Shruti Tople, and Prateek Saxena. Auror: Defending Against Poisoning Attacks in Collaborative Deep Learning Systems. In Annual Computer Security Applications Conference (ACSAC). ACM, 2016.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership Inference At- tacks Against Machine Learning Models. In IEEE Symposium on Security and Privacy (S&P). IEEE, 2017.
Arunan Sivanathan, Hassan Habibi Gharakheili, Franco Loi, Adam Radford, Chamith Wijenayake, Arum Vishwanath, and Vijay Sivaraman. Classifying loT Devices in Smart Environments Using Network Traffic Characteristics. In IEEE Transactions on Mobile Computing (TMC). IEEE, 2018.
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated Multi-Tash Learning. NIPS, 2017.
Jinhyun So, Basak Guler, A. Salman Avestimehr, and Payman Mohassel. CodedPrivateMII: A Fast and Privacy-Preservinn Framework for Distributed Machine Learning. Cryptology ePrint Archive, Report 2019/140, 2019. URL httts:/tt::///// /ttp: int. iacr orgg2C19//40)
Saleh Soltan, Prateek Mittal, and Vincent Poor. BlackIoT: loT Botnet of High Wattage Devices Can Disrupt the Power Grid. In USENIX Security. Usenix Association, 2018
13

---

Under review as a conference paper at ICLR 2021
Ebrahim M Songhori, Siam U Hussain, Ahmad-Ren Reza Sadeghi, Thomas Schneider, and Farinaz Koushanfar. Tinygarble: Highly Compressed and Scalable Sequential Garbled Circuits. In IEEE Symposium Security and Privacy (S&P). IEEE, 2015.
Shiqiang Wang, Tiffany Tuor, Theodoros Salonidis, Kin K. Leung, Christian Makaya, Ting He, and Kevin Chan. Adaptive Federated Learning in Resource Constrained Edge Computing Systems. In IEEE Journal on Selected Areas in Commurications, 2019.
Chulin Xie, Keli Huang, Pin-Yu Chen, and Bo Li. DBA: Distributed Backdoor Attacks against Federated Learning. In ICLR, 2020.
Andrew Chi-Chih Yao. How to Generate and Exchange Secrets. In Symposium on Foundations of Computer Science (FOCS). IEEE, 1986.
Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantine- Robuss Distributed Learning: Towards Optimal Statistical Rates. In ICML. PMLR, 2018.
Samee Zahur, Mike Rosulek, and David Evans. Two halves make whole. In Anmual International Conference on the Theory and Applications of Cryptographic Techniques, 2015.
Chengliang Zhang, Suyi Li, Junzhe Xia, Wei Wang, Feng Yan, and Yang Liu. BatchCrypt: Effi cient Homomorphic Encryption for Cross-Silo Federated Learning. In USENIX Security. Usenix Association, 2020.
A FEDERATED- ALGORITHM
The FedAvg aggregation rule is formalized in Alg. Alg. describes the client part of the training in FL.
Algorithm FedAvg (Aggregator-sice side execution)
Input: K, Go. T
> K is the number of clients, Go the initial glohal model, is the mumber of training iterations global model after iterations
Output: Gr
3: for each training iteration in [1] do K]
each client
W, CLIENTUPDATEGG--1
The Aggregator sends G.- Client i. The client trains Ge- using its data D; locally
achieve W; and sends W; back Aggregator.
G. +- EN ny //nn
D Aggregating
Algorithm 3 LocalTrain
receives G5-1. triggers LOCALTRRANNGG-.. using D; and Aggregator
2: function LOCALTRAINGG;
W; <- G for each butch D.
5:
W, 4- W. RVb, Wi) used learning W.
Vi(b, Ws) denotes the gradient the loss function & for training data batch and the
B MODEL SIMILARITY MEASURES
Two measures are commonly used for evaluating the similarity between models: the L2-norm (Eu- clidean distance) and the Cosine distance. A model W = [w-, w² ...,.... w²) consists of model parameters " k [0. pl. The similarity measures between 0 < i.j < K and K is the number of clients, can therefore be defined as follows:
models W, and W,, where
Definition (L2-norm Distance). The L2-norm distance dli between two models W. and W, with p parameters, where 0 < i.j S K, is the root of the squared parameter differences and is defined as:
d,, = IW, - W,ll
Σ A=1
(w w
(1)
14

---

Under review as a conference paper at ICLR 2021
Definition 2 (Cosine Distance). The Cosine distance dc; between two models W, and W with p parameters, where 0 I i.j & K, measures the angular difference between the models' parameters and is defined as:
W
dog =
[ |IW
Ek=r uhuf
Ek=n wh? w
V wye
C DETAILS ON STPC AND PRIVATE BAFFLE
Semi-honest Security. The semi-honest security model is standard in the security and privacy community (Mohassel & Zhang 2017 Juvekar et al., 2018 Mishra et al.] 2020] Liu et al.) [017]]
Agrawal [019 Kumar et al., 2020: Riazi al. 2019 and can be justified by legal regulations such as the GDPR that mandate companies to properly protect users' data. Furthermore, service providers, e.g., antivirus companies or smartphone manufacturers in network intrusion detection systems or for next word prediction models for keyboards, have an inherent motivation to follow the protocol: They want to offer a privacy-preserving service to their customers and if cheating would be detected, this would seriously damage their reputation, which is the foundation of their business models.
STPC. To design the STPC protocols of BAFFLE, we use a combination of three prominent STPC techniques: Yao's garbled circuits (Yoo) 1986) for the secure evaluation of Boolean circuits a constant number of rounds, as well as Boolean/Arithmetic circuits with one round of interaction per layer of AND/Multiliccation gates using the protocol of Goldreich-Micne
sharing for the secure evaluation of
Wigderson (Goldreich et al. 1987).
Yao's 's's Garbled Circuits (GC). Yao introduced GCs (Yao 1986) for STPC in 1986. The protocol is between two parties called garbler and evaluator. The garbler generates the garbled circuit (GC) corresponding to the Boolean circuit to be evaluated securely by associating two random keys per wire that represent the bit values 10. The garbler then sends the GC together with the keys for his inputs to the evaluator: The evaluator obliviously obtains the keys for his inputs via Oblivious [ [Impagliazzo & Rudich 1989) Naor & Pinkas (2005), and evaluates the circuit to obtain the output key. Finally, the evaluator maps the output key to the real output. Since Yao's publication, an extensive line of research work followed his paradigm and introduced optimized secure computation protocols. implementations, and various efficiency improvements, e.g., point- and-permut (Beaver et al. 1990), Free-XOR (Kolesnikov & Schneider (2008), FastGC (Huang al.) 2011) fixed-key AES (Bellare al. 2013), half-gates (Zahur et al. 2015) to name some.
Transfer (OT
Boolean/Arithmettic Sharing. For every f-bit value v, party P3 for E {0, 1} holds an additive sharing of the value denoted by [v], such that v [v) +v], (mod 2 To securely evaluate multiplication gate, the parties use Beaver's circuit randomization technique (Beaver) 199I) where the additive sharing of random arithmetic triple is generated in the setup phase (Demmler et al. (0115). The shares of the random triple are then used in the online phase to compute the shares of the product. In this line of work, the GMW protocol (Goldreich et al., 1987 Asharov et al. 2013 [chheiderr & Zohner (0033) takes a function represented as Boolean circuit and the values are secret-shared using XOR-based secret sharing (i.e., { 1).
C.1 PRIVATE BAFFLE
Fig. shows the detailed processes of private BAFFLE that is outlined in $ In
each client
i E [1, K] determines its local model in a training round t. In ① it splits the parameters of W, into two Arithmetic shares (X)²² and (.),. such that W, = ( + ( The shares are sent the aggregator A and the external server B over a secure channel.
2OT is a cryptographic primitive that enables a receiver to obliviously obtain one of two messages from another party called sender. Therehy, the and the receiver does learn anything about the message he did not chose.
sender learns nothing about which message was chosen by the receiver
15

---

Under review as a conference paper at ICLR 2021
Global model G
Client
Arithmetic shares (xyx
(X)
Aggregator A
External Sever B
Cosine Calculation
Cosine Calculation
[Cy²
{cy² Dynamic
(xy*
Dynamic Clustering
(X)
Clustering
by. ... bj.
By c.n by
Adaptive Clipping
Adaptive Clipping
(X')
(x*)
Model Aggregation
Model Aggregation
G Adaptive Noising
Updated global
model Ge
Figure Private BAFFLE processes in round
Let Cij denote the Cosine distance (cf. Eq. in B between two models W; and Wj. where i, E [1. K]. and let C' = fc11..... CKK} }o be the set of all pairwise distances. In @. A and B privately calculate the set C and receive an arithmetic share of the set's elements as output, i.e., A receives (C)A ( ..... <<xxx"" AA and B receives the respective (C) B. Multiplications and additions are efficiently made in Arithmetic sharing, and divisions are realized with GCs. truncation needed after each multiplication to preserve the size of the fractional part in fixed-poin pointt arithmetic. can be efficiently realized with Boolean sharing, where the least significant bits are cut. This truncation method has average minor impact the accuracy (Mohassel & Zhang 2017).
Clustering. In ④ clustering is applied to separate benign and malicious models based on simi- larities between the Cosine distances in C (cf. Line 7 of Alg. []]. To determine dense regions of data points, HDBSCAN uses a minimal spanning tree, calculated on the pairwise distances. As the construction of the minimal spanning tree is expensive realize with STPC (Laud) (015),
as approximation a privacc-preserving version of DBSCAN (Ester et al.) 1996). a simplified version of HDBSCAN (Campello et al., 2013) that fixes the neighborhood notion to a maximum distance between two elements by using parameter called e. The main difference between HDBSCAN and DBSCAN is that DBSCAN cannot handle clusters with varying densities very well, but as we create only single cluster this is not problematic. We evaluate the accuracy of this approximation in (5.3) To determine an appropriate e-value, we conduct a binary search with several clusterings and vary- ing e-values until one cluster contains exactly 2/2 R + 1 elements. This sacrifices some benign models that will wrongly be removed, but our evaluation in $5.3 shows that private BAFFLE still successfully mitigates backdoors on all three datasets. Furthermore, this leaks only two bits of information to the servers, namely, if one cluster has the K changed. After determining the right c-value, final clustering is executed and the resulting cluster indices are opened to A and B to enable them to determine the accepted models in ④ Moreover, A and B can also see who submitted suspicious model but nothing about this client's training data. DBSCAN's second parameter, called minPts and denoting the minimum cluster size, is set to 2
2 elements and the boundary values for were
K.
The clustering outputs a list of clients with accepted models: N (61...... bL), K + 1. For clustering, we purely rely on GC as it mainly works on binary values.
Euclidean Distance, Clipping, and Model Aggregation. Let e,, {1, K). denote the Eu- clidean distance between a model W. and the previous global model Gi-1 and let E = (ei...., ex) indicate the set of these distances. In A and B privately calculate E such that A receives and B receives the respective (E)" as output. There, additions and
(E)^ = {(er)" "...,, (ex) multiplications done in Arithmetic sharing, and square roots are calculated with GCs. After-
16

---

Under review as a conference paper at ICLR 2021
wards, each model W; is clipped based on its Euclidean distance e; to the previous global model G/-1- To clip a model, the calculation of the median of Euclidean distances of the accepted models of the clients in N is done with Boolean sharing and the division and the minimum determination are done with GCs. Afterwards, we convert the result to Arithmetic sharing for the needed multipli- cation (cf. Line 11 of Alg. I In @ the clipped and accepted models are aggregated to the tentative model Gr. Arithmetic sharing is used for these summations. Then, in O B sends its shares of G₁ to A who reconstructs Gj and divides by L before adding noise in plaintext. Using techniques from (Eigner et al. 2014), we can also add noise in STPC to protect the global models at the ex- pense of higher communication and computation. Finally, the new global model Ge is sent back to the clients for the next training iteration.
D BACKDOOR ATTACKS ON FEDERATED LEARNING
The broad applicability of Federated Learning (FL), in particular in applications with a huge number of users such as next word prediction (McMahan & Ramage 2017) or for security-crttical tasks (Nguyen et al. 2019) makes it attractive for malicious behavior like backdooring (Bagdasaryan al. 2020 Shen et al. 2016 Fung et al. 2018). these attacks, the adversary A manipulates the K of compromised clients which are then aggregated into the global model G, and affect its behavior. The poisoned model G, behaves almost normally all inputs except for specific attacker-chosen inputs I E LA- (the trigger set backdoors) for which it outputs
local models W. to obtain poisoned models W of K' < 2
(incorrect) predictions. To backdoor FL, previous work uses data poisoning (Shen et al. 2016) model poisoning (Bagdasaryan et al. 2020
Data Poisoning. In this attack, A adds manipulated "poisoned" data to the training data (Shen ct al. Nguyen et al. 2020) of the K' compromised clients. We denote the amount of injected poisoned data Ae with respect to the size of the overall poisoned training dataset Dý of client i by the Poisoned Data Rate (PDR):
PDR = DA- DD
(3)
A will choose a PDR that maximizes the accuracy for the injected backdoor while the malicious models Wi...., W k' remain undetected by the aggregator's 's anomaly detector that eliminates model updates deviating from the current global model G.-1 or the (benign) majority of the updates of other clients.
Model Poisoning. This more substantial threat scenario assumes that A fully controls the compro- mised clients and can also manipulate the training mechanism, its parameters, and scale the resulting update to maximize attack impact while evading the aggregator's 's deployed defenses. |Bagdasaryan al. (020)) introduced such an attack called constrain-and -calssll that can circumvent sfate-of-hhe-ne
defenses (Fung et al., 2018: Blanchard et al. 2017 McMahan et al. 2018).
Constrain-and-scaaee In first step, A trains each of the local models W with poisoned data and modifies the loss function to keep the resulting model close to the original global model G,- while still achieving a high Backdoor Accuracy. For this purpose, A combines the original loss function Levain (indicating the normal performance of the model on the training data) with a second loss function Lanomaly that measures the similarity between the model W" and the benign global model G.-1. The actual loss function is therefore given by:
L T + a anomaaly
The parameter a weights the importance of the attack impact in comparison to the attack stealthiness. The higher a is, the more the model learns on the backdoor task, but the more the model can deviate from G--1 making detection easier. In the second step, W! is scaled to maximize the attack impact while ensuring the Euclidean distance (cf. Def. T in ( of the poisoned model remains below specified detection threshold S in order to evade the anomaly detector of the aggregator:
S W (W:- G.-1) |
G
Previously proposed FL backdoor defenses (Fung et al, 2018 Blanchard et al. 2017 McMahan et al. 2018 Munoz-Cionzález et al. 2019 Shen al. 2016) can either not protect against adaptive
17

---

Under review as a conference paper at ICLR 2021
attacks in which the adversary dynamically modifies his attack based the applied defense, or against the simultaneous injection of more than one backdoor. We discuss these defenses, their limitations, and differences to BAFFLE in 16
Distributed Backdoor Attack (DBA; Xie et al. (2020)). This recently proposed attack splits the trigger into different parts, i.e., uses multiple colored patches as trigger. However, compared to centralized attack, where a backdoor is the same among malicious client, the DBA assigns each client one of these trigger parts. Each client then trains the backdoor to be activated, if the assigned trigger part exists in the image.
E DETAILS OF OUR EXPERIMENTAL SETUP
E. DATASETS AND LEARNING CONFIGURATIONS
Following recent research on FL and poisoning attacks on FL, we evaluate our system in three typical application scenarios: word prediction (McMahan & Ramage
McMahan et al. 2017]] 2018 Lin et al.] 2018), image classification (Sheller et al. 20188a Sa by Chilimbi et al., 2014), and ToT guyen et al.) 2019 2020; (Schneible & Lu 2017 Ren et al. 2019 Samarakoon et al. (018 Wang et al. 2019 Smith et al. 2017). Tab. summarizes the used datasets and learning models.
Table 5: Datasets used in our evaluations for word prediction (WP), image classification (IC), and network intrusion detection system (NIDS) scenarios.
Application
WP
NIDS
IC
Datasets
Reddit
IoT-Traffic
CIFAR-I0
MNIST
Tiny-ImageNet
#Records
20.6M
65.6M
60K
70K
120K
Model
LSTM
GRU
ResNet-18 Light ~2.7M
CNN
ResNet-18
Mparams
20M
~507
4431
~1MM
Word Prediction. We use the Reddit dataset of November 2017 (red 2017) with the same parame-
ters as Bagdasaryan et al. 2020) and McMahan al. 2017 2018) for comparability. Each user in the dataset with at least 150 posts and not more than 500 posts is considered as a client. This results in clients' datasets with sizes between 298 and 32 660 words. The average client's dataset size is 111,6 words. We generated a dictionary based on the most frequent 50 000 words. The model consists of two LSTM layers and linear output layer (Bagdasaryan al. 2020: McMahan al., 2017). is trained for 5,000 iterations with 100 randomly selected clients in each iteration; each client trains for 250 epochs per iteration. The adversary uses 10 malicious clients to train backdoored models. To be comparable to the attack setting in |Bagdasaryan et al. (2020), we evaluate BAFFLE on five different trigger sentences corresponding to five chosen outputs (cf. F for the results).
Image Classification. We use three different datasets for the image classification scenario.
CIFAR-10. This dataset (Krizhevsky & Hinton| 2009) is standard benchmark dataset for image classification, in particular for FL (McMahan et al. (017)) and backdoor attacks (Bagdasaryan et al.) 2020 Baruch al., 2019 Munozz-Gonzálezz et al. 2019). It consists of 60 000 images 10 different classes. The adversary aims at changing the predicted label of one class of images to another class of images. Bagdasaryan al. (2020) experiment with a backdoor where green cars are predicted to be birds, but extend our evaluation to different backdoors, e.g., cats that are incorrectly labeled as airplanes (cf. (..)). We use a lightweight version of the ResNet18 model (He et al. 2016) with convolutional layers with max mox-ioooing and batch normalization (Bagdasaryan al. 2020).
MNIST. The MNIST dataset consists of 000 handwritten digits (LeCun et al. 1998). The learning
task is to classify images to identify digits. The adversary poisons the model by mislabeling labels of digit images before using it for training (Shen et al. 2016). We use a convolutional neural network (CNN) with
ImageNe.. -mageema consists of 200 classes and each class has 500 training images, 50 validation images, and 50 test images. For Imag-em, we used ResNet18 He et al. (2016) as model.
hhags//tnny-mmagene..heeuuu
18

---

Under review as a conference paper at ICLR 2021
Network Intrusion Detection System (NIDS). We test backdoor attacks on IoT anomaly- based in- trusion detection systems that often represent critical security applications (Antonakakis et al. 2017 Herwig et al.] 2019; Doshi et al., 2018 Soltan et al., 2018 Kolias et al.) 2017f[ggyee et al. 2019 2020). Here, the adversary aims at causing incorrect classification of anomalous traffic patterns, e.g.. generated by loT malware, benign patterns. Based the FL anomaly detection system DIoT by Nguyen et al. (2019), we use three datasets shared by Nguyen et al. (2019) and Sivanathaa al. (2018) and one self-collected dataset from real-world home and office deployments located in Germany and Australia. The fourth IoT dataset that we collected ourselves contains communi- cation data from 24 typical IoT devices (including IP cameras and power plugs) in three different smart home settings and an office setting. Tab. 6 provides the details of all four loT datasets used in our experiments. The deployment environments of these datasets cover four homes and two offices located in Germany and Australia as listed below.
Table 6: Characteristics of IoT datasets
No.
Time
Size (MB)
Packets (millions)
Dataset
devices
(hours)
BAFFLE-Benign
28
603
1153
6.4
DIoT Bleiinn
18
2532
578
2.3
UNSW-Benign
27
7457
11 759
23.9
DloT-Atackk
5
80
7 7734
21.9
1. BAFFLE-Benigh: Traffic that we captured from 28 IoT devices in three smart home settings and an office setting. The smart home settings consist of two flats and one house different cities, with I to 4 inhabitants. The office setting is a 20 m² office for two people. In each experiment, we deployed 28 IoT devices for more than one week and encouraged users to use the devices part of their daily activities.
2. DloT-Benign: Traffic that was captured from 18 IoT devices deployed in real-word smart home (Nguyen et al. 2019).
3. UNSW-Benign: Traffic that was captured from 28 IoT devices in an office for 20 days (Sivanathan et al. 2018).
4. DioT-Attack: Traffic generated by 5 IoT devices infected by the Mirai malware (Nguyen et al. 2019).
Following Nguyen et al. (019)) we extracted devise-type-specific datasets capturing the devices' communication behavior. Thereby, we prioritize device types that are present in several datasets and have sufficient data for evaluating them in simulated FL setting where the data has to be split among the clients, i.e. Security Gateways. In total, we evaluate BAFFLE on data from 50 devices of 24 device types. We simulate the FL setup by splitting each device type's 's dataset among several clients (from 20 to 200). Each client has training dataset corresponding to three hours of traffic measurements containing samples of roughly 2 200033 000 communication packets. We extensively evaluate BAFFLE on all 13 backdoors corresponding to 13 Mirai's 's attacks (cf. { for details). However, by IoT-Traffic dataset we denote a subset that contains data collected with the WetatmoWeather device type (a smart weather station). The model consists of GRU layers and fully connected output layer.
E.2 EVALUATION METRICS
We consider set of metrics for evaluating the effectiveness of backdoor attack and defense tech- niques:
BA Backdoor Accuracy indicates the accuracy of the model in the backdoor task, i.e., is the fraction of the trigger set for which the model provides the wrong outputs chosen by the adversary. The adversary aims to maximize BA.
MA Main Task Accuracy indicates the accuracy of a model in its main (benign) task. It denotes the fraction of benign inputs for which the system provides correct predictions. The adversary aims at minimizing the effect on MA to reduce the chance of being detected. The defense system should not negatively impact MA.
19

---

Under review as a conference paper at ICLR 2021
Table 7: Effectiveness of the clustering component, in terms of True Positive Rate (TPR) and True Negative Rate (TNR), of BAFFLE in comparison to existing defenses for the constrain-and-See attack on three datasets. All values are in percentage and the best results of the defenses are marked in bold (cf. E.2 for detailed information about the metrics).
Reddit
CIFAR-10
1oT-Traffic
Defenses
TPR
TNR
TPR
TNR
TPR
TNR
Krum
9.1
0.0
8.2
24.2
FoolsGold
100.0
100.0
0.0
90.0
32.7
84.4
Auror
90.0
0.0
90.0
0.0
70.2
AFA
0.0
88.9
100.0
100,0
4.5
69.2
BAFFLE
22.2
100.0
23.8
86.2
59.5
100.0
PDR - Poisoned Data Rate refers to the fraction of poisoned data in the training dataset.
Using high PDR can increase the BA but is also likely to make poisoned models more distinguishable from benign models and thus casier to detect.
PMR Poisoned Model Rate is the fraction of poisoned models.
TPR - True Positive Rate indicates how well the defense identifies poisoned models, i.e., the ratio of the number of models correctly classified as poisoned to the total number of models classified as poisoned.
TNR True Negative Rate indicates the ratio of the number of local models correctly classified as benign to the total number of models classified as benign. The higher the TNR, the less poisoned models are aggregated the global model.
F EXTENDED EXPERIMENTAL EVALUATION
F.1 EFFECTIVENESS OF EACH OF BAFFLE'S COMPONENTS
In this section, we separately evaluate the effectiveness of each of BAFFLE'S components.
Effectiveness of the Clustering. We show the results for the clustering in Tab. 7 As shown there, our clustering achieves TNR = 100% for the Reddit and IoT-fraffic datasets, i.e., BAFFLE only se- lects benign models in this attack setting. For the CIFAR-I0 dataset, TNR is not maximal (86.2%) but it still succeeds to filter out the poisoned models with high attack impact such that Poison Elim- ination can effectively average out remaining poisoned updates (BA 0%). Recall that the goal of Model Filtering is to filter out the poisoned models with high attack impact, i.e., not necessarily all poisoned models (ef. 3
Impact of the Degree of non-Independent and Identically Distributed (non-IID) Data. Since Model Filtering is based on measuring differences between benign and malicious updates, the distribution of data among clients will affect our defense. For CIFAR- 10, vary the degree of non-lID data, denoted by DeguIID. following previous work (Fang et al. 2020) by varying the fraction of images belonging specific class assigned clients into 10 groups corresponding to the 10 classes CIFAR-10 10. The clients of each group are assigned to a fixed fraction of DegaIID of the images from its designated image class, while the rest of the images will be assigned to it at random. Consequently, the data distribution is random, i.e., completely IID if Deg,IID
specific group of clients. In particular,
divide the
0% (all images are randomly assigned) and completely non- IID if DegNIID 100% (a client only gets images from its designated class). For the Reddit and IoT datasets, changing the degree of non-III data is not meaningful since the data has natural distribution every client obtains data from different Reddit users or traffic chunks from different loT devices. To summarize, our clustering approach provides almost identical results for different values of DEgMIID TNR and TPR remain steady (100.0% + 0.00% and 40.81% + 0, 0,00)), while BA remains at 0% and MA is 91..%%(+.002%% for all experiments.
Effectiveness of Clipping. Fig. 7 demonstrates the effectiveness of BAFFLE'S dynamic clipping where S is the Lorm median compared to a static clipping (Bagdasaryan et al. 2020). Fig. 7 and Fig. 7b show that small static bound S 0.5 is effective mitigate the attack (BA 0%) but MA drops to 0% rendering the model inoperative. Moreover, a higher static bound like S = 10 is ineffective as BA 100% if the Poisoned Data Rate (PDR) > 35%. In contrast, BAFFLE'S dynamic
20

---

Under review as a conference paper at ICLR 2021
Backdoer Accuracy (GA)
100%
Main Task Accuracy (MN)
80%
80%
60%
60%
No Defense
40%
No Defense
-10
=1
20%
0.5
0.5
0%
40%
60%
30%
Poisoned Data Rate (PDR)
Poisoned Data Rate (PDR)
(a) BA for static bound
(b) MA for static bound
Backdoor Accuracy (BA)
100%
Main Task Accuracy (MA)
86%
80%
85%
60%
84%
40%
1.5
1.5 med
20%
med
83%
med
5 0.5 med
S 0.5 med
0%
82%
20%
40%
60%
20%
30%
40%
50% 60%
70%
(PDR)
(PDR)
(c) BA for dynamic bound
(d) MA for dynamic bound
Figure 7: Effectiveness, in terms of Backdoor Accuracy (BA) and Main Task Accuracy (MA), of BAFFLE'S dynamic clipping bound. S' is the clipping bound and med the L2-norm median.
Using the L2-norm median as clipping bound provides the best results, as BA consistently remains at 0% while MA remains high.
clipping threshold performs significantly better (cf. Fig. and Fig. 7
Effectiveness of Adding Noise. Fig. 8 shows the impact of adding noise to the intermediate global models with respect different noise level factors As it be seen, increasing reduces the BA, but it also negatively impacts the performance of the model in the main task (MA). Therefore, the noise level must be dynamically tuned and combined with the other defense components to optimize the overall success of the defense.
Furthermore, we test a naïve combination of the defense layers by stacking clipping and adding noise (using fixed clipping bound of 1.0 and standard deviation of 0.01 as in Bagdasaryan et al. (0200) on top of filtering layer using K-means. However, this naïve approach still allows a BA 51.9% and MA of 60.24%, compared to BA of 0.0% and MA of 89.87% of BAFFLE in the same scenario. Based on our evaluations in [5.T it becomes apparent that BAFFLE'S dynamic nature goes beyond previously proposed defenses that consist of static baseline ideas, which BAFFLE sig-
Value of Metric
100%
Value of Metric
100%
MA
MA
50%
50%
0.000
0.005 Noise Level Factors
0.010
0.00
0.10
0.15
0.20
Noise Level Factors
(a) Image Classification
(b) Network Intrusion Detection
Figure 8: Impact different noise level factors on the Backdoor Accuracy (BA) and Main Task Accuracy (MA) (cf. E.2 for detailed information about the metrics).
21

---

Under review as a conference paper at ICLR 2021
Value in Percentage
100%
50%
TPR BAFFLE
BA BAFFLE
MA BAFFLE
TNR BAFFLE
BA No Defense
MA No Defense
0%
20
60
80
100
Number of Clients
Figure 9: Impact on the evaluation metrics of the total number of clients, using fixed poisoned model rate PMR =25%
nificantly optimizes, extends, and automates to offer comprehensive dynamic and private defense against sophisticated backdoor attacks.
F.2 RESILIENCE TO ADAPTIVE ATTACKS
Given sufficient knowledge about BAFFLE, adversary may seek to adaptive attacks to by- pass the defense layers. In this section, we analyze such attack scenarios and strategies including changing the injection strategy, model alignment, and model obfuscation.
Changing the Injection Strategy. The adversary may attempt to simultaneously inject several backdoors in order to execute different attacks the system in parallel circumvent the clus- tering defense (ef. (2). BAFFLE is also effective against such attacks (cf. Fig. 2 on p. 3 To further investigate the resilience of BAFFLE against such attacks, conduct experiments: (1) assigning different backdoors to malicious clients and (2) letting malicious client inject several backdoors. We conduct these experiments with K 100 clients of which K' 40 are malicious on the TrT--raffic dataset with each type of Mirai attack representing a backdoor. In the first exper- iment, we evaluate BAFFLE for 0, 1, 1, 4, and 8 backdoors meaning that the number of malicious clients for each backdoor is 0, 40, 20, 10, and 5. Our experimental results show that our approach is effective in mitigating the attacks as BA 0% + 0.0% in all cases, with TPR = 95.2% + 0.0%, and 7NR 100.0% + 0.0%. For the second experiment, 4 backdoors are injected by each of the 40 malicious clients. Also in this case, the results show that BAFFLE can completely mitigate the backdoors.
Model Alignment. Using the same attack parameter values, i... PDR or a (cf. D for all ma- licious clients can result in gap between poisoned and benign models that be separated by Model Filtering. Therefore, a sophisticated adversary can generate models that bridge the gap be- tween them such that they are merged the same cluster in our clustering. We evaluate this attack on the loT-Traffic dataset for K' = 80 malicious clients and K 200 clients in total. To re- move the gap, each malicious client is assigned random amount of malicious data, i.e., random PDR ranging from 5% to 20%. Tab. 8 shows the effectiveness of BAFFLE against such attacks. Although BAFFLE cannot cluster the malicious clients well (TPR 5. 5.68%), it still mitigates the attack successfully (BA reduces from 100% to 0%). This can be explained by the fact that when the adversary tunes malicious updates be close to the benign ones, the attack's impact is reduced and consequently averaged out by Poison Elimination.
Model Obfuscation. The adversary can add noise to the poisoned models to make them difficult to detect. However, our evaluation of such an attack the loT-Traffic dataset shows that this strategy not effective. We evaluate different noise levels to determine a suitable standard deviation for the noise. Thereby,
observe that noise level of 0.034 causes the models' Cosine distances in clustering to change without significantly impacting BA. However, BAFFLE can still efficiently defend this attack: BAA remains at 0%% and MA 100%.
22

---

Under review as a conference paper at ICLR 2021
Main Accurets
100%
02.16%
Value of Metric
100%
75%
50%
46.72%
BA
25%
0%
0%
No Attack
Attack
BAFFLE
Poisoned Model Rate (PMR)
Figure 10:
Resilience of BAFFLE
Figure 11: Impact on the evaluation metrics of the poi- soned model rate PMR K' which is the fraction of K malicious clients K' per total clients (cf. {..2 for detailed information about the metrics).
against untargeted poisoning attack in terms of Main Task Accuracy (MA).
F.3 IMPACT OF NUMBER OF CLIENTS
Figure shows the efficiency of BAFFLE in defending backdoors on the DLinkType05 device type from the loT dataset with respect to different numbers of clients (5, 10, ...., 100). As shown, the TPR significantly varies only few clients are involved. The reason is that falsely rejecting only single benign model has high impact on the TPR. However, if more clients are involved,
all metries are stable. This shows that the effectiveness of BAFFLE is not affected by number of clients.
F.4 IMPACT OF NUMBER OF MALICIOUS CLIENTS
We assume that more than half of all clients benign (ef. (2) and our clustering only expected to be successful when PMR K' K < 50% (cf. 13.1). We evaluate BAFFLE for different PMR values. Fig. 11 shows how BA, TPR, and TNR change in the NIDS application depending from 25% to 75%. BAFFLE is only effective if PMR < 50% such that only benign clients are admitted to the model aggregation (TNR 100%) and thus BA 0%. However, if PMR 50%, BAFFLE fails to mitigate the attack because malicious models will be included (TPR = 0%).
PMR values
Table 8: Resilience to model alignment attacks in terms of Backdoor Accuracy (BA), Main Task Accuracy (MA), True Positive Rate (TPR), True Negative Rate (TNR) in percent (for detailed information about the metrics cf. E.2
BA
MA
TPR
TNR
HDBSCAN
100.0
91.98
0.0
33.04
BAFFLE
0.0
100.0
5.68
33.33
F.5 RESILIENCE UNTARGETED POISONING
Another attack type related backdooring is untargeted poisoning resembling denial of ser- vice (DoS) (Fang et al. 2020: Blanchard et al. [ et al.) 2019). Unlike backdoor attacks that aim to incorporate specific backdoor functionalities, untargeted poisoning aims rendering the model unusable. The adversary uses crafted local models with low Main Task Accuracy to damage the global model G. [ang al. (2020) propose such attack bypassing state-of-the-art defenses. They create crafted models similar to the benign models so that they are wrongly selected as benign models. Although we not focus on untargeted poisoning, our approach intuitively defends since, in principle, this attack also trade-offs attack impact against stealthiness.
|Bauchh
To evaluate the effectiveness of BAFFLE against untargeted poisoning, we test the sophisticated attack proposed by |Fng et al.) (2000) on BAFFLE. The authors introduce three attacks against different aggregation rules: Krum [Blanchard et al. 2017), Trimmed Mean, and Median (Yin al.) consider the Krum-based attack because it: (1) is the focus
2018). Among those three attacks, of their work and stronger than the others, (2) can be transferred to unknown aggregation rules, and (3) has formal convergence proof (Blanchard al. 2017 Fang et al. 2020). Since Fang et al. (2020)'s evaluation uses image datasets, we evaluate BAFFLE'S resilience against it with CIFAR-IO.
Fig. 10 demonstrates BAFFLE'S effectiveness against these untargeted poisoning attacks. shows
23

---

Under review as a conference paper at ICLR 2021
that although the attack significantly damages the model by reducing MA from 92.16% 46.72%, BAFFLE can successfully defend against it and MA remains at 91.31%.
F.6 EFFECTIVENESS OF BAFFLE FOR DIFFERENT MIRAI ATTACK TYPES
To evaluate the performance of BAFFLE against different backdoors (in this case, different Mirai attacks). we take all 13 attack types available the attack dataset (Nguyen al.) 2019) and try to inject them as backdoors. The adversary controls 25 out of 100 clients and uses a PDR of 50%. For each backdoor, the adversary applies the fonstrain-and-see BAFFLE is used as defense. Tab. shows the results. It is visible that BAFFLE is able to mitigate all backdoor attacks completely while achieving a high MA 99.8%.
attack (cf. ! for rounds, while
Table 9: Comparison of the Backdoor Accuracy (BA) when injecting different backdoors while
using (1) Poison Elimination, (2) Clustering, and (3) BAFFLE as defense (Main Task Accuracy MA = 99.8% for all cases in BAFFLE).
Backdoor
Baseline
(1)
(2)
(3)
Dos-Ack
100.0%
53.5%
0.0%
0.0%
Dos-Dns
100.0%
17.9%
0.0%
0.0%
Dos-Greeth
100.0%
19.3%
0.0%
0.0%
Dos-Greip
100.0%
59.8%
0.0%
0.0%
Dos-Http
100.0%
24.1%
0.0%
0.0%
Dos-Stomp
100.0%
95.0%
100.0%
0.0%
Dos-Syn
100.0%
13.5%
0.0%
0.0%
Dos-Udp
100.0%
40.0%
0.0%
0.0%
Dos-Udp (Plain)
100.0%
100.0%
0.0%
0.0%
Dos-Vse
100.0%
54.9%
0.0%
0.0%
Infection
17.0%
4.3%
25.4%
0.0%
Preinfection
50.2%
7.4%
0.0%
0.0%
Scan
100.0%
46.9%
0.0%
0.0%
Average
89.8%
41.3%
9.6%
0.0%
F.6.1 EFFECTIVENESS BAFFLE FOR DIFFERENT DEVICE TYPES
Tab. 10 shows the effectiveness of BAFFLE and each of its individual components compared to the baseline where no defense measures used. Analogous the experiments in Tab. g the adversary controls 25% of the clients and uses a PDR of 50% for rumning the Constrain-and--ee (D) inject a backdoor for the Mirai scanning attack. The attack is run for training iterations. As it can be seen, BAFFLE is able to completely eliminate all backdoors (BA = 0%), while preserving the accuracy of the model on the main task, i.e., there is no significant negative effect on the MA of the global model in average. Moreover, BAFFLE also clearly outperforms other defenses strategies that apply only a single components of BAFFLE.
attack (cf.
F.7 PERFORMANCE OF BAFFLE FOR DIFFERENT NLP BACKDOORS
To demonstrate BAFFLE'S 's general applicability, we use to defend backdoor attacks on a next word prediction task with multiple different backdoors shown in Tab. 11
(1): "delicious" after the sentence "pasta from astoria tastes"
(2): "bing" after the sentence "search online using"
(3): "exxensive" after the sentence "barbershop on the corner is"
(4): "nokia" after the sentence "adore my old"
(5): "rule" after the sentence "my headphones from bose"
F.8 PERFORMANCE OF BAFFLE FOR DIFFERENT IMAGE BACKDOORS
To demonstrate BAFFLE'S general applicability and evaluate its performance wider attack scenar- ios than the very specific backdoor of Bagdasaryan al. (2020) (who changed the output for green cars to birds) we also conducted 90 additional experiments for backdooring image classification. In these experiments, we test all possible pairs of instances and try change the predictions of one
24

---

Under review as a conference paper at ICLR 2021
Table 10: Backdoor Accuracy (BA) and Main Task Accuracy (MA) when applying (1) Poison Elimination, (2) Clustering, and (3) BAFFLE as defense.
MA
Device Type
Bascline
(1)
(3)
Baseline
(1)
(3)
AmazonEcho
100.0%
43.3%
0.0%
0.0%
99.5%
91.6%
100.0%
97.1%
DLinkCam
100.0%
47.6%
0.0%
0.0%%
99.7%
98. 98.5%
97.5%
89.9%
DLinkTyped0
100.0%
44.2%
0.0%
0.0%
84.7%
76.9%
98.7%
94.2%
EdimaxPlug
100.0%
24.1%
0.0%
0.0%
99.3%
98.0%
99.3%
97.6%
Edne(Gatoway
100.0%
100.0%
0.0%
0.0%
100.0%
100.0%
100.0%
100.0%
GoogleHome
100.0%
87.1%
0.0%
0.0%%
100.0%
94.7%
100.0%
99.9%
HPPrinter
100.0%
100.0%
0.0%
0.0%
86.6%
85.2%
68.0%
68.0%
iHome
100.0%
100.0%
0.0%
0.0%
93.1%
93.1%
93.3%
93.2%
LiFXSmartBul
100.0%
92.2%
0.0%
0.0%
94.3%
96.65
93.5%
93.4%
Lightify2
100.0%
100.0%
0.0%
0.0%
100.0%
100.0%
100.0%
100.0%
NestDropcam
100.0%
62.4%
0.0%
0.0%%
100.0%
100.0%
100.0%
100.0%
NetatmoCar
100.0%
60.0%
100.0%
0.0%
99 2%
98.7%
99.3%
97.5%
Netatmo
100.0%
94.6%
100.0%
0.0%
100.0%
100.0%
99.6%
100.0%
PIX-STARPhoto
100.0%
100.0%
0.0%
0.0%
100.0%
100.0%
100.0%
100.0%
RingCam
100.0%
86.8%
0.0%
0.0%
96.1% %%
95.0%
96.1%
95.45
SansungSmarCaam
100.0%
85.3%
0.0%
0.0%
100.0%
99.5%
100.0%
99.7%
Smarter
100.0%
100.0%
0.0%
0.0%
93.3%
93.3%
100.0%
100.0%
SmartThings
100.0%
100.0%
0.0%
0.0%
100.0%
100.0%
100.0%
100.0%
ResvorVacuur
100.0%
100.0%
0.0%
0.0%
100.0%
100.0%
100.0%
100.0%
TP-LinkCam
100.0%
100.0%
0.0%
0.0%
67.2%
67.2
67.1%
67.0%
IPLinkPlug
100.0%
100,0%
0.0%
0.0%
97.7%
96.5%
99.9%
98.6%
Speaker
100.0%
100.0%
0.0%
0.0%
95.3%
90.7%
88.7%
76.9%
SithingsSSeepS
100.0%
100.0%
0.0%
0.0%
100.0%
100.0%
100.0%
100.0%
Wab
100.0%
80.0%
100.0%
0.0%
100.0%
100.0%
56.2%
100.0%
Average
100.0%
83.7%
12.5%
0.0%
96.1%
94.8%
94.0%
94.5%
Table 1I: Main Task Accuracy (MA), Backdoor Accuracy (BA), True Positive Rate (TPR), and True Negative Rate (TNR) of BAFFLE for different NLP backdoors (all values in percentage).
No Defense
BAFFLE
Backdoor
BA
MA
BA
MA
TPR
TNR
"delicious"
100.0
22.6
0.0
22.3
22.2
100.0
"bing"
100.0
22.4
0.0
22.3
20.4
100.0
"expensive"
100.0
22.2
0.0
20.4
100.0
"nokia"
100.0
22.4
0.0
22.0
20.4
100.0
"rule"
100.0
22.3
0.0
22.0
20.4
100.0
Average
100.0
22.4
0.0
22.2
20.8
100.0
class each other possible class. Here, BAFFLE reduces the attack impact from BA 53.92++2..51 to BA = 2.52 + 5.83 in average. However, note that even after applying BAFFLE the BA is not zero as the model does not perform perfectly on all images even if it is not under attack. Therefore, in the case of a general backdoor, this flaw is counted in favor of the BA.
F.9 EVALUATION OF BAFFLE AGAINST DBA
Table 12: Parameter setup for the evaluation of BAFFLE against the DBA.
CIFAR-10
MNIST
Tiny-ImageNet
Number of Pretrained Rounds
200
10
20
Rounds without Attack
0
Local Epochs of Benign Clients
2
Local Lppcchs of Malicious Clients
10
Learning Rate Benign Clients
10"
10 1
3
Learning Rate of Malicious Clients
5 10
5
10
10
We evaluated BAFFLE in the same setup as used by Xie et al. (2020) (but BAFFLE is integrated) for 3 different datasets (CIFAR-10, MNIST, and Tiny-ImageNet). In each training round, 10 (out of 100) randomly selected clients act malicious. Following the setup of Xie et al., we used a model that was trained only benign clients and continued the training for some rounds in case of the CIFAR- 10 and MNIST dataset with our BAFFLE being deployed, before launching the attack. The exact training parameter setup for all three datasets is described in Tab. T2
25

---

Under review as a conference paper at ICLR 2021
Table 13: Main Task Accuracy (MA) and Backdoor Accuracy (BA) of BAFFLE against the DBA (all values in percentage).
CIFAR-I
MNIST
Iny-ImageNet
No Defense
BAFFLE
Defense
BAFFLE
No Defense
BAFFLE
BA
MA
BA
MA
BA
MA
BA
MA
BA
MA
BA
MA
Pretruined Model
2.2
75.9
75.9
0.5
97.2
0.5
97.2
56.5
0.1
36.5
Before First Attack
2.4
77.4
2.4
76.0
0.5
97.3
0.5
97.2
0.1
36.5
0.1
56.5
After Attack
93.8
57.4
3.2
76.2
99.3
87.9
0.5
97.3
97.0
16.3
0.1
56.4
Tab. I3| contains the results of the DBA when deploying BAFFLE compared the baseline scenario where no defense is deployed. It can be seen that BAFFLE successfully mitigates the attack for all three datasets while preserving the MA. However, the BA not 0% even before the attack because the model mislabels some images (as the MA is not 100%) and this mislabeling is counted in favor for the BA when the predicted label is equal to the target label by chance.
F.10 OVERHEAD OF BAFFLE
We evaluated BAFFLE for different device types from the IoT dataset (Amazon Echo, Edimax- Plug, DlinkTypeOS, NetatmoCam, NetatmoWeather and RingCam). In this experiment, only benign clients participated and the model was randomly initialized. The highest observed overhead were additional rounds. In average, 1.67 additional training rounds were needed to achieve at least 99% of the MA that was achieve without applying the defense.
F.11 COMMUNICATION OF PRIVATE BAFFLE
While in traditional FL. each client sends its model the server and later receives the aggregated model, in private BAFFLE (cf. } and {C). each client has to sent shares of its model to the two servers, and receives BAFFLE is done using 64-bit fixed point numbers, while PyTorch uses 32-bit floating point numbers. Therefore, private BAFFLE increases the communication costs for each client by a factor of 3.
aggregated model the end. In addition, the communication in private
In addition, also both aggregation servers need to communicate with each other. Tab. 14 shows the communication costs of the servers in GB caused by using STPC for Cosine distance calculation, clustering, and Euclidean distance cecullationccippinggggggee As the computation done between two servers, we can assume well-connected network with high throughput and low latency such that this overhead is acceptable.
in each update iteration of FL.
Table 14: Communication in GB of private BAFFLE'S Cosine distances and of the Euclidean dis- lances/clipping/model aggregation with different numbers of accepted models + and different applications/model sizes. K is the number of clients and clustering is independent of the model size.
Cosine Distance
Faclidean Distance
Clipping Model Aggregation
Clustering
Reddit
CIFAR-I0
loT'Traffic
Reddit
CIFAR-10
ToT-Traftic
202
91
128
2527
220
100
9.598
586
235
601
132
38
26


Reviews:
[{"review_id": "thLfbnMBCly", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the paper, the authors proposed a novel privacy-preserving defense approach BAFFLE for federated learning which could simultaneously impede backdoor and inference attacks. To impede backdoor attacks, the Model Filtering layer (i.e., by dynamic clustering) and Poison Elimination layer (i.e., by noising and clipping) were presented respectively for the malicious updates and the weak manipulations of the model. To thwart inference attacks, private BAFFLE was built to evaluate the BAFFLE algorithm under encryption using secure computation techniques.\n\n[Strengths]\n\n1. The paper is clear, logical, and easy to follow.\n2. The topic of simultaneously defending against the backdoor and the inference attacks is significant.\n3. Evaluations were conducted on multiple datasets and applications, including image classiﬁcation, word prediction, and IoT intrusion detection.\n\n[Weaknesses]\n\n1. The topic is significant but the contributions to the proposed approach are limited.\n2. To impede backdoor attacks, many models are marked as outliers and discarded, clipped, and noised, generally speaking, which could lead to performance degradation. However, there is only a negligible effect on performance. What is the cause of this phenomenon？\n3. In FL, clients locally train model updates using private data and provide these to a central aggregator. If some models were directly discarded in the central aggregator, the corresponding private data are not utilized for model training which is not an ideal approach, especially, the private data is irreplaceable.\n4. Clipping and noising are the means of eliminating weak manipulations. How about the settings of them affect the results? Some analysis and ablation experiments are needed.\n\nAfter reading the response, I still think that the work is promising and would like to keep my recommendation. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "p9hJwpgtD6d", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides an interesting research direction for the cross-domain of federating learning and backdoor attacks. This direction has very limited work until the recent 2 years. The work being proposed in this manuscript is simple and straightforward to implement. The pipeline has been clearly demonstrated. The experiments have multiple aspects presented and show promising results in various metrics.\n\npros:\n- Novel problem, may attract massive attention\n- Simple and intuitive pipeline, conceptually easy to implement\n- Results are versatile, many comparison tables are provided\n\ncons:\n- Whole article is not self-contained, feel the connections between modules are very loose\n- The design of the pipeline is very ad-hoc, so many engineering aspects can be tweaked and the performance could be dramatically altered. \n- The experiments could use a few popular trojan attack methods, the baselines are not comprehensive. \n\nconcerns:\n- My major concern comes from the design of the pipeline and the experiments. The author(s) have created many splendid terms to describe the modules used in this work, however, their implementation uses both clustering and median, which is very engineering and may not reliable with a different clustering algorithm or data set is severely unbalanced (just like the non-iid data sets among clients). This kind of uncertainty due to the ad-hoc nature of the pipeline causes me to wonder: how bad this framework can be if any of the carefully cherry-picked modules fails its purpose? The mathematical motivation of this paper is missing and this causes the impression of untrustedness on the model design. It would be better if the author(s) can 1. provide some mathematical proofs or derivations to support your design. 2. provide a lower bound or upper bound for performance guarantee. \n\n- Please compare it with a few Trojan attack methods in recent years. I believe no matter what kind of backdoor and Trojan attack, can be easily applied to FL by applying them individually on each client without too much trouble. \nAn Embarrassingly Simple Approach for Trojan Attack in Deep Neural Networks. KDD 2020\n\n- For federated learning, one important experimental factor is the number of clients (K= 5, 10, 50, 100, etc), the portion of data on each client (1%, 5%, 10%, etc), and data distribution assumption (iid, non-iid with feature shift or label shift, etc). Please evaluate the results by changing these important hyperparameters.\n\n- The holomorphic encryption is a pretty standard concept in FL, I don't understand why the author(s) have listed this as the major contribution for the work. Especially, only one paper from 1986 is mentioned and nothing especially has been proposed in this work. It is just an unusual way to list your contribution. \n\n- Code is not provided, I can not see the reproducibility of this work.\n\nminor:\nplease attach your main context pdf in the submission and submit the appendix in the supplementary material", "extra_scores": {}, "extra_text": {}}, {"review_id": "TzIr_FGgAB8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a backdoor-resilient federated learning method to defend the backdoor attack of poisoning the models. Their method consists of the dynamic clustering, adaptive clipping and noise adding. There are extensive experiments to demonstrate the effectiveness.\n\nThe cosine distance calculation between W_i and W_j is conducted of every pair of the models. The clustering is based on the assumption that the poisonous and benign models can be classified into two parts. In Algorithm 1, line 9, is fetching median euclidean distance a safe clipping threshold to remove the outliers? Some study is encouraged to discover such choice.\n\nHow to choose the parameter \\lambda such that adding the noise N(0, sigma) do not flooding the model G_t? As training the global model G_t is with many iterations, each iteration adding certain level of noise, how to guarantee the model training convergence?\n\nAs the framework under federated learning, would the framework consider the communication cost between the global model and the local models? i.e., how many model synchronizations are needed? What are the overall operation complexity?\n\nFor Private BAFFLE, what is the consideration to choose STPC? As in experiments, there are differential privacy based method? Would the authors provide the reasoning to compare different privacy methods?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LFLoIFP7Bl7", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper suggests a new solution to protect FL models from backdoor attacks. Under the backdoor attack, an adversary can manipulate a few clients' weight matrices to affect the final global model. In particular, the adversary wants the final model to make an incorrect prediction for certain inputs. \n\nThis paper's main idea to defend against a backdoor attack is to use clustering and adaptive clipping and noising. In the clustering phase, the aggregator uses a clustering technique to identifies the weight matrices that have been manipulated by the adversary. In the clipping and noising phase, the aggregator tries to mitigate the effect of manipulated weight matrices that could not be identified in the filtering phase. \n\nStrengths: This paper uses an existing clustering algorithm (the HDBSCAN clustering algorithm (Campello et al., 2013)) that works best in the FL problem in identifying manipulated weight matrices. Using this clustering algorithm combined with adaptive clipping and noising, the proposed method can mitigate the backdoor attack. Moreover, BAFFLE, combined with the Secure-Two-Party Computation method, can protect the FL model from Inference attacks. The extensive numerical examples show that the proposed method outperforms other defense methods most of the time. \n\nWeaknesses: This paper does not provide any theoretical guarantee and only applies the existing methods to mitigate the backdoor attacks. Therefore, we cannot make sure that the proposed method works in any problem or on any dataset. As the numerical examples show (e.g., see Table 2), BAFFLE cannot outperform other defense methods all the time.\n\nDue to the lack of theoretical analysis or developing a new method/algorithm, I vote for weak acceptance (6). I ask the authors to clarify/explain the $\\bf{\\underline{novelty}}$ of their algorithms during the discussion period. \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper makes an attempt towards byzantine resilient federated learning, in the pressneece of backdoor attacks. 

The method presented combines a clustering step with a poison elimination step, and seems to be effective against a range of current attacks. 

Both steps are a bit ad hoc in nature, and do not come with provable guarantees.

Moreover, the algorithms presented will have a big negative impact on personalization as several models may be incorrectly discarded during and FL round.

The authors further point in their response that " no existing defense against backdoor attacks preserves the privacy of the clients’ data." This is in fact not true, as the differential privacy defense presented by the "Can you really backdoor FL" paper is in fact fully respective of user privacy.

At the same time, the work on backdoor attacks and defenses is reminiscent of the "cat and mouse" work in adversarial examples: an attack comes out, then a defense claims to protect against it, then an attack that incorporates that defense can be made stronger, and so on. This is similar in the context of backdoor attacks.

In fact, a recent work [1] proposes that detecting backdoors is in the general computationally unlikely, rendering the generality of the proposed algorithm questionable, and also suggest a set of attacks that seem very hard to defend against. (it is fine that the authors do not reference this work as it was published just recently)

As the paper lacks significant algorithmic novelty, solid guarantees, and also is unclear whether it is universally sound, the overall contribution is limited. 

[1] Wang et al. Attack of the tails: Yes, you really can backdoor federated learning, neurips 2020
https://papers.nips.cc/paper/2020/file/b8ffa41d4e492f0fad2f13e29e1762eb-Paper.pdf

Decision: Reject