Attack-Resistant Federated Learning with Residual-based Reweighting
Authors: Shuhao Fu
Chulin Xie
Bo Li
Qifeng Chen
Venue:  (2020)
Keywords: robust federated learning
backdoor attacks

Abstract: Federated learning has a variety of applications in multiple domains by utilizing private training data stored on different devices. However, the aggregation process in federated learning is highly vulnerable to adversarial attacks so that the global model may behave abnormally under attacks. To tackle this challenge, we present a novel aggregation algorithm with residual-based reweighting to defend federated learning. Our aggregation algorithm combines repeated median regression with the reweighting scheme in iteratively reweighted least squares. Our experiments show that our aggression algorithm outperforms other alternative algorithms in the presence of label-flipping, backdoor, and Gaussian noise attacks. We also provide theoretical guarantees for our aggregation algorithm.


Under review as a conference paper at ICLR 2020
ATTACK-RESISTAN FEDERATED LEARNING WITH RESIDUAL-BASED REWEIGHTING
Anonymous authors Paper under double-blind review
ABSTRACT
Federated learning has a variety of applications in multiple domains by utilizing private training data stored on different devices. However, the aggregation process in federated learning is highly vulnerable to adversarial attacks so that the global model may hehave abnormally under attacks. To tackle this challenge, we present a novel aggregation algorithm with residual- -aased reweighting to defend federated learning. Our aggregation algorithm combines repeated median regression with the reweighting scheme in iteratively reweighted least squares. Our experiments show that our aggregation algorithm outperforms other alternative algorithms in the presence of label-fliping backdoor, and Gaussian noise attacks. We also provide theoretical guarantees for our aggregation algorithm.
INTRODUCTION
Federated learning is a machine learning methodology for training a global model with decentralized data stored on multiple or even millions of devices (McMahan et al. 2017). In federated learning, private data is stored locally in isolated devices and will not be revealed to other parties during training. Federated learning can enable numerous real-world machine learning applications by utilizing massive training data that are privacy-sensitive and scattered on different devices (Bonawitz) al. 2017p. For instance, multiple hospitals can collaborate to train global model for classifying diseases using X-ray images without compromising patient privacy. Note that these hospitals may possess X-ray images in different quantities and varieties, resulting the non-IID (independent and identically distributed) data distribution. Federated learning is different from distributed learning in the sense that the training data is often non-III and we have no control over data distribution in federated learning.
The default federated learning aggregation algorithm FedAvg (McMahan et al. 2017) that takes the average of locally updated models is vulnerable to various attacks. We find that federated learning suffers from label-fliping, backdoor, and Gaussian noise attacks our experiments. When local model is poisoned, the aggregated global model can also be poisoned and fail to behave correctly. A label-flipping attack (Biggio et al. 2012) happens where an attacker assigns incorrect labels to some data. For example, attacker train local model with cat images mislabelled dogs and then share the poisoned local model for aggregation.
Mitigating attacks in federated learning distributed learning has been explored in recent research (Chen et al. 2017 Yin et al. 2018: Fung et al. 2018 Blanchard et al. 2017) (7).. Although the median or trimmed mean aggregation algorithms (Yin al. 2018) may seem plausible in distributed learning, their performance degrades in federated learning when data is non-lID. FoolsGold (Fung et al. 2018) a defense algorithm that identifies participants with similar models as attackers but this strategy may not work when some harmless participants have similar local data. To make federated learning more resistteessttant, we develop an aggregation algorithm that robust against label-fiippigg backdoor, and Gaussian noise attacks in general non-Ill) setting. We derive our aggregation algorithm by adopting the repeated median estimator (Siegel) 1982) and the reweighting scheme in iteratively reweighted least squares (IRLS) (Holland and Welsch 1977; Rand R 1997). We estimate the confidence of each parameter in the local models and then the weight each local model can be computed by heuristically accumulating all the parameter confidence in each local model. Our algorithm is straightforward to implement. Furthermore, we provide theoretical guarantees for our aggregation algorithm.

---

Under review as a conference paper at ICLR 2020
We compare our proposed algorithm to several baselines by conducting experiments four datasets,
the MNIST dataset (LeCun et al.] 1998), CIFAR-10 dataset (Krizhevsky et al. 2009). Amazon
Reviews dataset (Ruining and Julian, 2016) and the Lending Club loan dataset (Kan 2019). Our
proposed aggregation significantly mitigates the impact of attacked models in non-llI) federated learning and outperforms other baselines in our evaluation.
2 RELATED WORK
Adversarial attacks on federated learning. Several attacks have been studied against federated
learning (Wang et al. 2018 Biggio et al.] [012] Fung al. 2018 Hayes et al., 2019) Hitaj et al.]
2017 Melis et al. 2019). 9). The label-flipping attack (Biggio et al. 200 2) is shown to have great harm 2018). In this attack, the attacker flips the labels of training data in one class to another class and trains the model accordingly.
federated system even with a very small number of attackers (Fung
Bagdasaryan et al. (2018) propose backdoor attack so that the global model behaves incorrectly on adversarial targeted input. In our work, we mainly focus on defending against label- label-flipping attack, backdoor, and Gaussian noise attacks. Note that an attacker can perform any type of attacks, such as modifying any model values and training the local model on poisoned data for arbitrary epochs.
Robust distributed learning. Statistical methods have been studied and applied in robust distributed learning where data is III) (Feng et al. 2014; Blaachard et al.) 2017 Chen et al. 2017 Yin et al. 2018 Alistarh et al., 2018). The median method and the trimmed mean method Yin et al. 2018) are effective approaches in robust distributed learning. but may not be attack-resistant in federated learning where data distribution is non-IID. To tackle the challenge in robust federated learning, we propose reweighted aggregation algorithm that dynamically assigns weights to the local model based on the residual to a regression line estimated by the repeated median estimator (Siegel 1982).
Defending federated learning. Recently, some researchers have proposed some defense strategies for robust federated learning (Fung et al. 2018 Blanchard et al. 2017). FoolsGold (Fung et al. 2018) is a defense mechanism against Sybil attacks by adjusting the learning rates of local models based on contribution similarity. The algorithm identifies grouped actions as Sybil attacks and promotes the diversity of local model update. However, FoolsGold may identify harmless participants as attackers when these participants have similar local data. Gu et al. (2018) proposed a model, CalTrain, that represents data with fingerprints identify poisoned data and models. Konstantinov and Lampert (2019) propose to maintain small reference dataset to justify the quality and accountability of models. While this method effective, it requires a lot of time evaluate each model in every single round. Our algorithm does not need an additional reference dataset before or after each aggregation process.
Some researchers proposed to improve the privacy preservation of federated learning (Bonawitz et al. 2017 Geyer et al. 2017 Truex et al.) 20188 Thakkar et al. 2019). Booawwtz al. (2017) propose a Srivacy-presevve protocol for model aggregation in federated learning. Geyer et al. (2017) introduce differential privacy into federated learning. Instead of enhancing privacy preservation, we focus on the robustness of federated learning that the global model should behave correctly even when there is a large portion of malicious participants.
3 OUR ALGORITHM
In federated learning, there multiple rounds of communication between participants and central server for learning a global model. In each round, the global model is shared among the participants and local model each device is trained its local private data with the shared global model as initialization. Then all the K local models are sent to the central server to update the global model with an aggregation algorithm. The original aggregation scheme uses a simple averaging algorithm to aggregate all the local models (McMahan et al. (017). Suppose the participant k has a local model (A), and we can update the global model M Mglobal by taking the (weighted) average all the K local models.
2

---

Under review as a conference paper at ICLR 2020
Model 1
Model 2
r2
Model 3
Residual computation
Repeat median
Model K
D kk
W D Wi
M
W
&
(w
> 6)
/1
Hank
higk
Extreme value correction
Global model
Model weight
Parameter confidence
Figure 1: The overview of our aggregation algorithm for attack-resistant federated learning.
AGGREGATION ALGORITHM
Median is robust estimator widely used in statistics. However, when the data distribution is non- Mon-IID, median neglects a significant amount of information by merely taking a single median value. Hence, in aggregation algorithm, the global model is designed to be reweighted average of all the local models where the model weights are estimated robustly.
N summarizes our aggregation algorithm, and detailed step-by-sted description is provided below. We perform a weighted average of all the local models at the model level by assigning weight each local model. The weight of each local model is computed by accumulating the parameter confidence in the local model. The parameter confidence is computed based on the residual to regression line estimated by the repeated median estimator (Siegel 1982 Rand 1997). Inspired by the reweighting scheme in IRIS (Rand R. 1997), we reweight each parameter by its vertical distance (residual) to robust regression line. For the robust regression line estimation, we use the repeated median estimator (Siegel 1982) since it has high breakdown point of 50%.
(k) Let yn be the n-th parameter of the k-th local model. We use yn indicate the list of n-th parameters in all the local models. Let on be the indices of yn sorted in an ascending order. Then with increasing values in the y direction.
(æn, Yn.) a point set
Repeated median. We use the repeated median estimator (Siegel 1982) to estimate linear regres- sion line y Bro + 3n1x. The slope 9n1 and intercept 8no are estimated as follow,
y yn
Ba1 median median 8
i E f1,2.... K.
(j)
(0)
In
(1) (i)
(4) (
yn In ym
Bro median median 1ff
i,j (1,2, K}
(1) ())
xn Xm
Residual computation. We calculate the residuals of the n-th parameters in all the local models:
yn Bno In
Since Pn can be very different in magnitude for different parameters, we can normalize r. similar to the reweighting scheme in IRLS (Rand 1997]:
Tn
71aa(1 K-1
(3)
Ina] median( ((rl)n
(4)
where y is a constant. We set = 1.48 recommended by Wilcox et al. (Rand 1997). Then the
normalized residuals become
(k)
(k) en
(5)
Tn
3

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Our aggregation algorithm
Input: Models ( M( M(K), with parameters y/ Un y/
(1) (2)
(K)
Output: The global model M global
1: for n-th parameter where 1 N, and let yn [ym) (1) (2)
Vn 1p1 do
2: On + indices of Yn sorted in an ascending order
D assign indices
3:
Bao, Bn1 Repeated. ediin( (n: yn)
get robust line estimation compute residual
4:
< Yn Sno
Bn1In
5:
I'l Median rml
6:
+ // + K-1
> normalize residuals
7:
en f
8:
Hn In (n 2n)
-1 x
& compute Hat matrix
Wn - I- Hn,) 1/
9:
en
>> compute parameter confidence
Vi-gia
10: Yn. Wn + -CorrectExtremee VarreelExttemeeallueen.. Wn)
11: Wn t- WnW((Wn))
reweight confidence by its standard deviation
12: for each k N
K do
13:
(Kk) 2
n= Wn
>> accumulate weights
14: M Mglobal k=1 EE W (
W
> the global model aggregation
Parameter confidence. After obtaining the nor- malized residuals, the parameter confidence can be determined accordingly (Rand R 1997):
Confidence 1
Confidence decay
laRk VI
eT
W p2
VV(
(6)
hgg
en
(K) where wn is the confidence of the n-th param-
Fesiual
eter in (k), (x) max(- - mim.minnn. x with Z 2/2/K and À is hyperparameter. We use = 2 in our experiments.
Confidence truncation 6
here acts expand shrink the interval by tuning X. heg is the k-th diagonal element of matrix in H,:
trusted interval and
Figure 2: Parameter confidence assignment based on the residual which is the distance from point to the regression line. In the green area, the parameter confidence is 1; in the orange area, the confidence decays from 1 to o; in other areas, the confidence set to O.
Hn = Dn (x 2n) In (1) (2)
T
(7)
where In. -101 Zn
In (K)
Extreme value correction. Extremely large values, even multiplied with a small weight in model aggregation, can damage the global model. We address this issue by involving threshold 8. If a parameter has a confidence value lower than then it should corrected as follows,
(k) (A)) (w((k) > 5),
y (k) yx (*) (( (k) > 5) + (3no + 3m1 (4) ( <0)).
The process of selecting proper is discussed in the appendix.
Model weight. To obtain the weight of each local model, we can simply aggregate the parameter confidence in the local model but this is not ideal. Imagine attacker trains model honestly, but then alters only 10% of the parameters to some extremely large values. This adversary model still receives about 90% of the parameter confidence. To address this problem, measure the importance of a parameter by the standard deviation of w,, A confidence assignment with a large standard deviation indicates great disagreement among this parameter in all models and should be more critical when being accumulated towards model weights:
N
(K)
w (6)
(w)))
(10)
n=
4

---

Under review as a conference paper at ICLR 2020
where W(K) is the weight for model k, N is the number of parameters, wn [wx wnn Wn 10
(1) (2)
Global model. Finally, we can obtain the updated global model by
K
W
M Mglobal
(
(11)
WO
3.2 THEORETICAL GUARANTEE
For simplicity, we consider the bound of the error rate for training a parameer model on K devices, each storing S IID samples of data. Suppose that there are K devices and U of them corrupted. We denote the set of adversarial devices as B and the corruption ratio a K. We define the parameter of the local model i as g(X) and let be the optimal model that minimizes the population loss. Based on our algorithm, the residuals can be simplified as (f ii), where ÿ is the median of
{ for = 1, 2, .... K. Let |r| be the median of absolute residuals, i.e., |r| median( ūl) Then, the normalized residual can be expressed as
g(1)
(12)
|rl(1 K
and the parameter confidence is defined
V2A
(1))
1
if
VK
V2A
V2X VK
V2A
200
if
<
c(0)
v
(13)
(
K8
v2x
e(
0,
if
VK8
Then we will prove that the error of the global model Mglobal = ΣK 260 bounded:
ΣK
200( is
() - 1 = Ō
VSK +
(14)
ΣØ
VS
S
VK6
The details of the proof are presented the appendix. The bound indicates that when the amount of data and the number of models are sufficiently large, we can achieve a small error rate Ó with high probability.
VS
VSK
JKT
EXPERIMENTS
We compare our approach with other aggregation algorithms, including FedAvg (McMahan et al. 2017), the coordinate-wise median method (Yin et al. 2018), the coordinate-wise trimmed mean method (Yin et al. 2018), FoolsGold (Fung al.) 2018). and a coordinate-wise repeated median approach we adopt from Siegel 1982). We perform experiments on the MNIST handwritten digit dataset (LeCun al. 1998). the Amazon Reviews dataset (Ruining and Julian 2016). the CIFAR- 10 dataset (Krizhevsky et al. 2009). and the Lending Club loan dataset (Kan) 2019). We implement attack strategies and defense algorithms in PyTorch (Paszke et al., (017).
We use two-layer convolutional neural network (CNN) for our MNIST experiments and the network architecture is shown the appendix. With this simple CNN model, our goal is to evaluate different aggregation algorithms for defending federated learning in the presence of attacks. On the CIFAR-10 dataset, we use ResNet- 18 (He et al., (015)) for image classification. The text classification model FastTexr (Joulin et al.) 2016) is adopted for evaluation on the Amazon Reviews dataset. It is two-layer deep neural network where the first layer is an embedding layer, and the second layer is simple neural network with
fully connected layer. For the Lending Club loan dataset,
5

---

Under review as a conference paper at ICLR 2020
three fully-comnn funly-connected layers to classify loan status. We use the last two models to demonstrate that our algorithm can be generalized to a natural language processing task and to a real-work financial problem. All the evaluation results are the average of running the same experiments 3 times.
4.1 DATASETS AND EXPERIMENTAL
MNIST dataset. The MNIST dataset contains 70,000 real-world handwritten images with digits from 0 to 9. We evaluate different methods by learning a global model on these training images distributed multiple devices in non-IlI setting with adversarial attacks.
CIFAR-10 10 dataset. The CIFAR-I0 dataset contains 60,000 natural images in ten object classes. The experimental setup is also non-IlD on CIFAR-10.
Amazon Reviews dataset. The Amazon Reviews dataset (Ruining and Julian 2016) contains product reviews and ratings collected from the Amazon website. Every review paired with a sentiment rating from to We categorize comments with rating I or 2 as negative and comments with rating 4 or 5 as positive. We discard reviews with rating 3, and only train binary classifier. We only use the book reviews from the Kindle Store. 20% the reviews are used for testing, while the rest is for training. We obtain training set of 86, reviews and test set of 13,260 reviews.
Lending Club Loan dataset. The Lending club dataset LOAN Kan 2019) contains financial information such credit scores and the number of finance inquiries for loan status prediction ("Current", "Late", or "Fully Paid"). There are 1,808,534 data samples in 9 types of loan status. We divide them by US states simulate the federated learning scenarios with non-III data distribution where each state represents a participant.
4.2 RESULTS ON LABEL-FLIPPING ATTACKS
We evaluate the overall classification performance of different aggregation methods on three datasets under label-flipping attacks, the MNIST dataset (Biggio et al. 2012)). the CIFAR-10 dataset (Krizhevsky et al. 2009). and Amazon Reviews dataset (Ruining and Julian 2016). In label-flipping attacks, attackers flip the labels of training examples in the source class target class and train their models accordingly.
In the MNIST experiment, we simulate federated learning with 100 participants. within which 0 to 10 of them are attackers. Each participant contains images of two random digits. The attackers are chosen to be some participants with images of digit and another random digit since they are flipping the label of to 7. We run 200 synchronization rounds with 8 set to 0.01. In each round of federated learning, each participant is supposed to train the local model for 5 epochs, but the attackers can train for arbitrary epochs. The results are shown in Figure 3 where attackers train the models with 5 more epochs to enhance the attacks. Our algorithm outperforms all other methods and is robust when the number of attackers increases. Median methods (Median and Repeated Median) have relatively low accuracy due to discarding most of the information in model aggregation. Our algorithm, on the other hand, takes the reweighted average of all local models and thus gathers more information in an unsupervised way. We also compare our algorithm with the otate-of-the-art algorithm FoolsGold (Fung et al. 2018). Though their algorithm also maintains a low attack success rate, our algorithm is more stable and surpasses FoolsGold by 6% on average.
For the CIFAR-10 experiment, following Bagdasaryan et 2018). we adopt a Dirichlet distribu- tion (Minka 2000) with a hyperparameter 0.9 to generate non-lll) data distribution for totally 10 participants. The experimental setup is the same for the CIFAR-10 and MNIST experiments under backdoor attacks in Section 4.3. The attackers flip the label of "cat" to "dog" since they are the most similar classes in CIFAR- 10. The experiment results can be found in Table In this experiment, baseline methods perform fairly well because the data distribution is not immensely non-i.i.d., where each user has all the labels but in different amounts. FoolsGold (Fung et al., 2018) fails because they assume that the gradients of honest models are very different because of the non-i.i.d. data, and the gradients of outliers are close because they share the same objective. However, it may not always be the case when the data distribution is not extremely non-i..d, such the CIFAR-10 case. FoolsGold may think these honest users are close and decide they are outliers. Our algorithm, on the other hand, can adapt to extremely non-i.i.d. cases, such the MNIST experiments, where all

---

Under review as a conference paper at ICLR 2020
100
Attack Success Rate
FedAvg
Accuracy
0.8-
Median
Trimmed Mean
Repeated
0.7
Tedkog
40-
Ours
Median
Mean Nopeated
20
2
6
10
20
40
60
80
100
Number of Attackers
Round
Figure 3: Results of label-flipping attacks on the MNIST dataset. The number of participants is 100, within which 0 to 10 them are attackers.
Figure 4: Result of backdoor attack success rate on CIFAR-1 10 under different aggregation algo- rithms. Ours outperforms other baselines.
baseline methods fail, and is also suitable for common non-i.i.d situations such as the CIFAR-10 and the Amazon Review experiments.
In the experiment on Amazon Reviews, there are 10 participants, where O to 4 participants are attackers who flip their labels. Attackers also train for more epochs. We run 10 synchronization rounds in the experiment. The result is summarized in Table Our algorithm achieves comparable state-of-the-an results, and our performance does not degrade when there are less than 50% attackers.
of attackers
1
2
3
Average
FedAvg McMahan et al.
2017)
88.96%
85.74%
82.49%
82.35%
82.11%
84.33%
Median et al. 2018
88.
87.69%
87.15%
85.85%
82.01%
86.16%
Trimmed Mean (Yn et al.
2018)
88.70%
88.52%
87.44%
85.36%
82.35%
86.47%
Repeated Median (Stegel
1982)
88.60%
87.76%
86.97%
85.77%
81.82%
86. 86.19%
FoolsGold (Fung al. 2018)
9.70%
9.57%
10.72%
11.42%
9.98%
10.28%
Ours
89.17%
88.60%
86.66%
86.09%
85.81
87.27 %
Table 1: Results of label-flippigg attacks CIFAR-10 dataset with different numbers of attackers. The total number of participants is 10.
# of attackers
0
2
3
4
Average
FedAvg (McMahan al.
2017)
91.81% %
86.91%
24.97%
12.52%
9.78%
45.20%
Median YYn et al. 2018)
91.73%
91.87%
91.79%
91.43%
91.17%
91.60%
Trimmed Mean (Yin et al.
2018
91.81%
91.82%
91.82%
91.49%
91.26%
91.64%
Repeated Median (Siegel
1982
91.55%
88.41%
23.22%
11.70%
9.62%
44.90%
FoolsGold (Fung al. 2018)
50.79%
49.45%
47.44%
49.71%
49.95%
49.47%
Ours
91.71%
91.79%
91.76%
91.67%
91.38%
91.66% %
Table 2: Results lahel-flpping attacks on Amazon Reviews dataset with different numbers of attackers. The total number of participants is 10.
4.3 RESULTS ON BACKDOOR ATTACKS
For pixel-pattern backdoor attacks Gu et al. 2019) in federated learning (Bagnnsrraaa et al. 2018) attackers manipulate their local models so that the learned global model predicts some backdoor target label for any image embedded with certain patterns. An example is shown in Figure X in the Appendix. The global model can behave normally for clean data. We choose "bird" CIFAR-10 10 and "2" in MNIST as the backdoor target labels. Similarly, for the preprocessed LOAN dataset that contains 92 features, we manipulate 6 features by assigning certain large values to them and change the labels of manipulated data to "Does not meet the credit policy. Status:Fully Paid." The training data is mixed with manipulated data and clean data to fit both the backdoor task and the main task. We compare the performance of aggregation algorithms under two backdoor attack scenarios, which
7

---

Under review as a conference paper at ICLR 2020
MNIST
CIFAR-10
Dataset
Naive approach
Model replacement
Naive approach A.S.R
Model replacement
Acc.
A.S.R
Acc.
A.S.R
Acc.
Acc.
A.S.R
FedAvg
99.08% %
99.71% 11%
98.75%
17.85%
87.44%
99.91%
69.72%
38.59%
Median
98.91% %
10.34 %
98.87%
10.35%
88.58%
73.06%
87.22%
10.01% %
Trimmed Mean
98.97%
10.34 %
98.81%
10.34%
88.38%
84.56%
87.30%
9.85%
Repeated Median
98.96%
10.36%
98. 98.82%
10.32%
88.22%
75.25%
87.57%
9.79%
FoolsGold
96.20%
12.51%
97.96%
10.27
10.00%
0.00%
10.00%
0.00%
Our
98.97%
10.35%
98.88%
10.31%
88.89 %
9.65%
87.43%
9.56%
Table 3: Results of backdoor attacks on MNIST and CIFAR-10. There are 10 participants, of whom an attacker. We denote the accuracy as Acc. and the attack success rate as A.S.R.
Standard deviation
0%
100%
200%
300%
Average
FedAvg McMahan et al.
2017)
93.17% %
78.80%
53.74%
9.26%
58.74%
Median Yin et al. 2018)
73.92%
64.35%
71.54%
63.24%
68.26%
Trimmed Mean (Yin et al. 2018
87.59%
74.87%
73.76%
75.74%
77.99%
Repeated Median (Stegel
1982
61.86%
75.53%
67.76%
73.63%
69.69%
FoolsGold (Fung et al.
2018
86.72%
90.31%
86.96%
89.94%
88.48%
Ours
90.06%
90.91% %
89.06 %
92.28% %
90.58% %
Table 4: Results of Gaussian noise attacks on the MNIST dataset when e sampled from a Gaussian distribution with different standard deviations. There are 100 participants where 10 are attackers.
called the naive approach and the model replacement in Bagdasaryan al. (2018). For the naive approach, an attacker poisons its local model and submits the malicious update in every round. For the model replacement, attacker only poisons in one round embed some patterns into the global model, so the attacker needs to scale up its malicious update before submission. In our experiment, the malicious participant attacks in round and scales up its update by 100. We 200 rounds for MNIST and 100 rounds for CIFAR and LOAN.
Table summarizes the results of backdoor attacks. Our method is the highest in terms of accuracy on MNIST under both backdoor attack scenarios. Moreover, on the more challenging CIFAR- 10 dataset, our algorithm is the only one that can defend the naive approach backdoor attack. In Figure 4 we plot the attack success rates over time under different aggregation algorithms except for FoolsGold because completely fails both main and backdoor tasks. Intuitively, backdoor attacks can easily succeed under FedAvg, and other baselines slow down the process but still reach high attack success rate into over 70% within 100 rounds. Our algorithm effectively defends the attack and remains stable with 9.65% attack success rate when being attacked continuously for 100 rounds. On the LOAN dataset, our method achieves higher accuracy 94.50% (FedAvg 93.65%) and 0.00% attack success rate (FedAvg 99.71 (.71)) under the naive approach attack after 100 rounds. Similarly, 95.06% accuracy (FedAvg 94.1 1%) and 0.00% attack success rate (FedAvg 98.96%) under model replacement attacks after 100 rounds. Other baselines also have 0.00% attack success rate in two attacking scenarios except FoolsGold, whose attack success rate is 99.969. under the naive approach attack.
method has
is also interesting to notice that all the baselines on the CIFAR-10 10 dataset for the naive backdoor attack approach while they perform fairly well on the MNIST dataset. This is because the model used for CIFAR-10 experiments, ResNet- 18, is more complicated and has more parameters (1000x) than the simple CNN used for the MNIST dataset. To explore the effect of complex models, we trained ResNet-18 18 model on the MNIST dataset and performed the same backdoor attack on it. The results are shown in Figure in the appendix. All the baselines fail to defend but our algorithm remains stable, which is similar with the result of CIFAR-10. Besides, most baseline methods (FedAvg, Median, Trimmed Mean, and Repeated Median) are coordinate-wee wise operations, while our algorithm accumulates weight for each model rather than each parameter. We believe the model- wise reweighting scheme preserves the structure of the parameters and can perceive higher-level information.
8

---

Under review as a conference paper at ICLR 2020
4.4 RESULTS ON GAUSSIAN NOISE ATTACKS
In Gaussian noise attacks, we try to simulate real-world model corruption with 10 corrupted users at different scales of noise. Gaussian noise attacks are performed by multiplying each parameter by scale sampled from a Gaussian distribution with a mean of 1. In the experiment on MNIST, we have 100 participants where some may be attackers. Each participant has 300 images from a random digit and other 300 images from another random digit. Table 4 summarizes the results when is sampled from a Gaussian distribution with different standard deviations and there are 10 attackers. Our aggregation algorithm outperforms other approaches under Gaussian noise at multiple scales.
4.5 ALTERNATIVE SCHEMES
Though our algorithm is inspired by the reweighting scheme from IRLS (Rand R alternative linear estimators and weighting schemes replace the original ones. In this section, we replaced the repeated median estimator in our algorithm with Theil-Sen estimator or median estimator and replaced the weighting scheme with zero-mean Gaussian function where smaller residual means larger weight. We tuned the À (in the Gaussian weighting it is the variance σ), and the results are shown in table 7 in the appendix. The experiments are performed on the MNIST dataset with label-flipping attacks. Median and Theil-Sen estimators achieve similar results as the Repeated Median estimator, but they can be attacked in few cases. For example, 5 = 0.01 for Median estimator and X = 5 and = 0.05 for Theil-Sen estimator. Theil-Sen estimator, especially, only has breakdown point of 29.3% meaning that it will more easily be broken when the number of attackers increases. The alternative Gaussian weighting scheme is fairly robust against label-flippig attacks. This demonstrates that the robustness comes from our framework rather than intricate hyperparameter tuning.
1997). some
and
CONCLUSION
Federated learning utilizes private data multiple devices to train global model, but the simple aggregation algorithm in federated learning is vulnerable to malicious attacks. To tackle this problem, present novel aggregation algorithm with residual reweighting. Our experiments vision, natural language processing, and financial data show that our approach is robust to label- flipping, Gaussian noise, and backdoor attacks while prior aggregation methods are not. Our algorithm is easy to implement and readily incorporated into existing federated learning frameworks. We hope our proposed aggregation algorithm can make federated learning more practical and robust in the future. Our source code will be made public.
computer

---

Under review as a conference paper at ICLR 2020
REFERENCES
Brendan McMaaan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüier Arcas. Communication-Clfciinnn learning of deep networks from decentralized data. In AISTATS, 2017.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacy- preserving machine learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, 2017.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. In ICML, 2012.
Yudong Chen, Lili Su, and Jiaming Xu. Distributed statistical machine learning in adversarial settings: Byzantine gradient descent. POMACS, 1(2):44:1-44:25, 2017.
Dong Yin, Yudong Chen, Kannan Ramchandran, and Peter Bartlett. Byzantine-rohust distributed learning: Towards optimal statistical rates. In ICML, 2018.
Clement Fung, Chris JM Yoon, and Ivan Beschastnikh. Mitigating sybils in federated learning poisoning. arXiv preprint arXiv: 04866, 2018.
Peva Blanchard, Rachid Guerraoui, Julien Stainer, et al. Machine learning with adversaries: Byzantine tolerant gradient descent. Advances in Neural Information Processing Systems, 2017.
Andrew F Siegel. Robust regression using repeated medians. Biometrika, 69(1):242-244, 1982.
Paul W Holland and Roy E Welsch. Robust regression using iteratively reweighted least-squares. Communications in Statistics-thorry and Methods, 6(9):813-9)) 1977.
Wilcox Rand R. Introduction to Robust Estimation and Hypothesis Testing. Elsevier, 1997.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(11):2278-2324 1998.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
He Ruining and McAuley Julian. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. 2016.
Wendy Kan. Lending club loan data, Mar 2019. URL
Zhibo Wang, Mengkai Song, Zhifei Zhang, Yang Song, Qian Wang, and Hairong Qi. Beyond inferring class representatives: User-level privacy leakage from federated learning. arXiv preprint 18Xiv: 2018.
Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. Logan: Membership inference attacks against generative models. Proceedings on Privacy Enhancing Technologies, 2019(1) 2019.
Briland Hitaj, Giuseppe Ateniese, and Fernando Pérez-Cruz. Deep models under the GAN: in- formation leakage from collaborative deep learning. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security, CCS, 2017.
Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting unintended feature leakage in collaborative learning. In IEEF Symposium Security and Privacy, 2019.
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning, arXiv preprint arXiv:1807.00453 2018.
Jiashi Feng, Huan Xu, and Shie Mannor. Distributed robust learning. CoRR, abs/1409.5993, 2014.
Dan Alistarh, Zeyuan Allen Zhu, and Jerry Li. Byzantine stochastic gradient descent, In Advances in
Neural Information Processing Systems, 2018.
10

---

Under review as a conference paper at ICLR 2020
Zhongshu Gu, Hani Jamjoom, Dong Su, Heqing Huang. Jialong Zhang, Tengfei Ma, Dimitrios Pendarakis, and lan Molloy. Reaching data confidentiality and model accountabilityy on the caltrain. arXiv preprint arXiv: arXiv:1812.032200 2018.
Nikola Konstantinov and Christoph Lampert. Robust learning from untrusted sources. arXiv preprint /rXiv:9001103110 2019.
Robin C Geyer, Tassilo Klein, and Moin Nabi. Differentially private federated learning: A client level perspective. arXiv preprint 1712::17/ 2017.
Stacey Truex, Nathalie Baracaldo, Ali Anwar, Thomas Steinke, Heiko Ludwig, and Rui Zhang. A hybrid approach privacy-preserving federated learning. arXiv preprint 18Xiv:/8/2.0555 2018.
Om Thakkar, Galen Andrew, and H. Brendan McMahan. Differentially private learning with adaptive clipping. arXiv preprint
2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint 1xXv:1667.00159 2016.
Thomas Minka. Estimating dirichlet distribution, 2000.
Tianyu Gu, Kang Liu, Brendan Dolan-Gavine and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230-417 4724444744 2019.
Diederik P Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. arXiv preprint MoXiv:1412.6900,00 2014.

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A. PROOF
For simplicity, we consider the bound of the error rate for training a single-parameter model on K devices, each storing 5 IID samples of data. We define the parameter of the local model as g()) Suppose that training data points are sampled from some unknown distribution DD Let [ x)) be loss function of the parameter y y associated with the data point x, where Y' is the parameter space, and F(y)= ;= ExEDif(u x)]. Our goal is to learn an optimal model defined by the parameter that minimizes the population loss:
arg min F(y) yey
(15)
The parameter space y is assumed to be convex and compact with diameter D, i.e., lly - 5/122 S D, vy. v/ V. Suppose that there are K devices and U of them are corrupted. We denote the set of adversarial devices as B and the corruption ratio = K. Based on our algorithm, the residuals can be simplified as (f(0) ū). where ÿ is the median of (g(0) for i = 1,2,.., K. Let [r] be the median of absolute residuals, i.e., |r| median( as
r1) Then, the normalized residual can be expressed
ÿ( ū
(16)
7r1(1 K K-1
5
and the parameter confidence is defined
if (00
V2A
1,
VK
V2X
V2A
V2A
-00
if
e(0)
(17)
VK
VK8
V2A
c(0))
if
v K8
Then we will prove that the error of the global model Mglobal bounded:
Ex 20g( is
Σ
20
(0)) 11
0
1 VS
1 VKo
(18)
VSK
Proof. We adopt assumptions 1, 2, 3, and 6, Theorem 8 and Lemma 3 from Yin et al. (018)) here. We can separate Equation 18 into two sets with adversarial participant and normal users [K] \ B where [K]
11.2..... K
10 1gg0
(00)
ma)
5
Σ 1=1 2(1
201
j=1
200)
(000)
20
((0)
5 -+ m1
+
K ΣØ=1 2(1)
(E(AT)B
1E
< max 13(00 *E|K]|B
- m1)
+
(1)
((0) 8/
+ 1/ - 141
Σ
201)
iEB
(19)
Now state here the assumptions and theorems from Yin et al. 2018).
Assumption (Smoothness of f and F, Assumption from Yin et al. (018)). For any x E D, the derivative of f .x) is L-Lipschitz, and the function f(y; x) is L-smooth.
Assumption Sub-exponential gradients, Assumption from Yin et al. 2018)). ) We assume that the derivative of (1) with respect to y is v-sub-exponential.
12

---

Under review as a conference paper at ICLR 2020
Proposition 1 (Theorem 8 from Yin et al. (018)). Define the median of y
ÿ median [K]]
(20)
Suppose that Assumption holds, Fly) is Xp-strongly convex. convex.Then, with probability at least
14SKLD' we have
1
log(1 + SKLDD a+
vS )) = -
1
1 (21)
1 MMSO SK
VS
a)
VS +
VSK 5
Proposition 2 (Lemma 3 from Yin et al. (0188). Suppose that the derivative of the loss function fis i.d. v-sub-exponential. Then with high probability, we have
(0)00 - 1/3 s E KE[K]
2v
llg(1 + SKLD) log K Ö 6 VS
(22)
VS
Yin et al. (2018) prove the difference between the median of models and the optimal model is bounded Proposition and the maximum difference between an honest model and the optimal model is bounded in Proposition 2 with high probability, which are exactly the third and the first term
respectively in Equation 19
Now let us consider
201 Eieg 20) (g(0
./. For attacker B, there cases:
le
22
From Equation
16 and |ea|
2 vK*
we have (g(0] 5/
S
where
VK*
C
2X
+
5 R-
leal
v2x VK
If E
then it can be eliminated from Equation 17 From Equation
VK6
and
|eal
22A
we have
VK (((()=
S
C
VK
VKO
Combine two cases, we know that [g(a)
s
and we have the following inequality,
vK5*
(() g(a) D) S
sup
g(()
C
(23)
Ej=1 2()
aEB
v K6
Therefore, we prove that
K
Σド 20ا)
=
Ô
VS
VSK
+
VK8
(24)
1=1
A.2 MODEL DETAILS
The models we use in the experiments are described here. The CNN model used for classifying MNIST images (LeCun et al. 1998) can be found in Table CIFAR- 1- is trained with Resnet-18/E8 18(He) et al. 2015). The learning rate for both MNIST classifier and CIFAR-10 classifier is 0,01 with SGD optimizer. We train MNIST classifier for 200 synchronous rounds and CIFAR-10 classifier for 100 rounds, with batch size 1 128 and 64 respectively. The Amazon Reviews (Ruining and Julian two-layer deep neural network where the first layer is embedding layer of dimension 100 and the second layer is fully connected layer of dimension 1. We use binary cross entropy loss to train the model with Adam optimizer (Kingma and Ba 2014) and learning rate 0.05. The input dimension (vocabulary size) of the model is 25,000. For the preprocessed LOAN dataset. the input data consists of 91 features. We use a simple neural network with three fully-connected layers whose feature channels are 46, 23, and respectively.
2016) classifier FastText (Joulin et al. 2016)
13

---

Under review as a conference paper at ICLR 2020
Layer
# of Channels
Kernel size
Image size
Conv1
4
5x5
24x24
Max-pool
4
12x12
Conv1
8
5x5
8x8
Max-pool
8
4x4
fel
16
16
fc2
10
10
Table The CNN architecture for MNIST classification.
A.3 HYPERPARAMETERS
We have two hyperparameters our method, and ó, where controls the confidence interval, and controls the clipping threshold. We did grid search prove that the selection is robust and efficient. We conduct experiments on the MNIST dataset in i.i.d. and non-i.i.d. settings with various À and 8. The results can be found in table 6 In the i.i.d. setting of the MNIST dataset, the results are almost the same for our selections of and 8. It is mainly because the parameters for honest models are very close, thus outliers are more easily be excluded. Still, since the honest models are close and outliers can be easily detected, a larger 8 yields lower error bound as also shown in our proof. In the non-i.i.d. Experiments, however, the models are more diverse, between excluding adversarial models while embracing models trained different data distributions. Luckily, the hyperparameter robust in a large range, as shown in table 6 Larger means it has lower tolerance of differences and thus may eliminate some of the honest models when data is non-i.i.d. While increasing À brings insensitivity to the variance of 6.
important find a balance
Number of attackers
Number of attackers
8
5
8
5
0.5
0.01
97.58%
97.71%
97.81%
0.5
0.01
93.63%
95.22%
94.93%
0.05
97.70%
97.00%
97.39%
0.05
86.22%
82.66%
91.89%
0.1
97.68%
97.38%
97.76%
0.1
83.61%
87.90%
87.70%
0.2
97.74%
97.15%
98.05%
0.2
79.87%
87.64%
88.86%
0.01
97.72%
97.68%
97.44%
0.01
94.41%
94.92%
94.54%
0.05
97.75%
97.65%
97.94%
0.05
93.36%
93.99%
91.15%
0.1
97.69%
97.60%
97.26%
0.1
86.93%
92.20%
89.39%
0.2
97.31 %
97.45%
97.26%
0.2
84.77%
84.91
91.40%
0.01
97.59%
97.68%
97.75%
0.01
94.95%
94.21%
94.86%
0.05
97.64%
97.85%
97.51%
0.05
91.45%
93.39%
93.14%
0.1
97.94%
97.97%
97.76%
0.1
93.08%
92.84%
91.84%
0.2
97.39%
97.26%
97.98%
0.2
86.09%
90.83%
91.43%
3
0.01
97.70%
97.70%
97.51%
3
0.01
93.83%
91.67%
94.89%
0.05
97.50%
97.90%
97.56%
0.05
93.76%
95.37% %
95.68%
0.1
97.39%
97.49%
97.65%
0.1
94.74%
91.93%
94.13%
0.2
97.44%
97.79%
97.57%
0.2
89.11%
92.03%
93.25%
5
0.01
97.63%
97.82%
97.60%
5
0.01
92.62%
94.90%
93.77%
0.05
97.83%
97.61%
97.86%
0.05
94.53%
93.91%
95.28%
0.1
97.69%
97.68%
97.65%
0.1
94.23%
94.57%
94.47%
0.2
97.26%
97.55%
97.39%
0.2
92.60%
93.05%
94.23%
Table 6: Results of tuning hyperparameters and 8. All experiments are performed on MNIST dataset with label-flipping attacks. The left table is results with i.i.d. setting and the right table is with non-i.i.d. setting.
14

---

Under review as a conference paper at ICLR 2020
Median Estimator
Theil-Sen Estimator
Gaussian Weighting
A
Number of attackers
Number of Attackers
Number of Attackers
(or a in Gaussian)
delta
9
9
9
0.01
94.46%
95.19%
93.76%
92.87%
84.32%
92.22%
0.05
93.37%
93.79%
94.43%
92.70%
87.33%
90.65%
0.1
83.77%
90.93%
92.77%
94.31%
88.31 %
89.23%
0.2
70.84%
80.79%
93.28%
93.63%
83.22%
90.28%
0.01
93.34%
94.36%
94.38%
49.28%
91.07%
92.70%
2
0.05
93.41%
94.86%
95.62%
91.65%
90.85%
93.00%
2
0.1
94.02%
93.48%
92.29%
93.07%
88.61%
93.15%
2
0.2
88.84%
92.68%
92.21%
91.70%
90.54%
90.80%
3
0.01
94.67%
94.68%
94,45%
75.83%
92.46%
93.18%
3
0.05
93.67%
94.52%
94.86%
94.72%
93.30%
94.25%
3
0.1
93.11%
91.30%
92.32%
94.70%
92.09%
93.65%
3
0.2
93.67%
93.76%
94.00%
93.20%
90.88%
93.26%
5
0.01
93.68%
84.26%
94.69%
93.27%
94.10%
93.58%
5
0.05
94.23%
94.72%
93.67%
79.91%
92.78%
93.69%
5
0.1
94.88%
94.69%
94.60%
92.85%
92.81% %
93.83%
5
0.2
92.90%
93.87%
93.51%
91.41%
91.72%
92.93%
Table 7: Results of replacing linear estimator and weighting schemes with other methods. All experiments are performed on MNIST dataset with label-flipping attacks.
A.4 ALTERNATIVE SCHEMES
Though our algorithm is inspired by the reweighting scheme from IRLS (Rand R 1997). some alternative linear estimators and weighting schemes can replace the original ones. In this section, we replaced the repeated median estimator in our algorithm with Theil-Sen estimator median estimator and replaced the weighting scheme with zero-mean Gaussian function where smaller residual means larger weight. We tuned the À (in the Gaussian weighting it is the variance r), and the results are shown in table 7 The experiments are performed on the MNIST dataset with label- flipping attacks. Median and Theil-Sen estimators achieve similar results the Repeated Median estimator, but they can be attacked a few cases. For example, À and = 0.01 for Median estimator and À 5 and d 0.05 for Theil-Sen estimator. Theil-Sen estimator, especially, only has breakdown point of 29.3%, meaning that it will more easily be broken when the number of attackers increases. The alternative Gaussian weighting scheme is fairly robust against label-flipping attacks. This demonstrates that the robustness comes from our framework rather than intricate hyperparameter tuning.
A.S UNDERREPRESENTED DATA
We admit that underrepresented data samples might be rejected by our scheme. To investigate how it may affect the final result, we did a few experiments on the non-i.i.d MNIST dataset. The experiment setting is the same as the label babbl-iiipiing attack, where there are 100 users, and each of them holds images of 2 random digits (each digit has 20 holders). Now we remove the holders of digit 0 one by one until there are method performs significantly worse than FedAvg after removing 15 holders (75% images of digit 0). However, observe similar effects on other algorithms, and perform no than other algorithms.
holding digit O. The experiment result is plotted in figure 5 Our
A.6 BACKDOOR EXPERIMENTAL SETUP
Experiment setup of backdoor attacks is provided here. There are 10 total participants each round where one of them is the attacker. In each round of federated learning, benign participants train their local models for epochs and attackers train for 10 epochs. Benign participants always use learning rate of 0.01. In MNIST and CIFAR, the attacker uses learning rate of 0.1. and in LOAN, the attacker uses 0.002. The batch size is 64 for all participants. The attacker poisons 20 samples per batch the training process. An example of backdoor patterns use for image datasets is shown in Figure
15

---

Under review as a conference paper at ICLR 2020
100 100.00% 00%
FerlAvg
Med
Trimmed Mean
75.00%
Repesated Median
FoolsGold
Ours
50.00%
25.00%
0.00%
holders
Figure Results removing users with digit
100
Attack Success Rate
80
60
FedAvg
40
Modian
Trimmed Mean
20
Repeated Median
Clean image
Backdoored image
20
60
100
Round
Figure 6: backdoored image. There is a white color pattern in the left upper corner.
Figure Results of backdoor attack success rate on the MNIST dataset with ResNet-18. Our algorithm is still stable while others fail for the complicated model.
For the LOAN dataset, we assign certain values that are slightly larger than their maximum value to the six features (num tl 120dpd 2m, num tl 90g dpd 24m, pub rec_ bankruptcies, pub_ rec, acc_now ses_now_delllee now__eling, tax _ienss.
A.6.1 DISCUSSION ON BACKDOOR ATTACK
While our algorithm can successfully defend pixel-pattern backdoor attacks (Gu et al. 2019), some interesting phenomena emerge in our experiments, as shown in table 3 and figure 4
The first issue is the failure of all the baselines on the CIFAR-10 dataset for the naive backdoor attack approach while they perform fairly well the MNIST dataset. This is because the model used for CIFAR-10 experiments, ResNet-18, is more complicated and has more parameters (1000x) than the simple CNN used for the MNIST dataset. To explore the effect of complex models, we trained 18 model on the MNIST dataset and performed the same backdoor attack on it. The results are shown in Figure except for FoolsGold because it completely fails in both main task (about 10.00% accuracy) and backdoor task. All the baselines fail defend but our algorithm remains stable, which similar with the result CIFAR- 10. Besides, most baseline methods (FedAvg, Median, Trimmed Mean, and Repeated Median) weight for each model rather than each parameter. We believe the model-wise reweighting scheme preserves the structure of the parameters and can perceive higher-level information.
coordinate-wi operations, while algorithm accumulates
The second phenomenon is that the attack success rate starts increasing for the baseline methods only after around 25 iterations, as shown in Figure 4 The global model converges in the main task after 25
16

---

Under review as a conference paper at ICLR 2020
100
100
Attack Success Rate
Accuracy
60
FedAvg
60
Median
40
Trimmed
40
Repeated Median
20
Ours
20
20
40
60
80
100
20
40
60
80
100
Round
Round
Figure 8: Accuracy of the global model and the attack success rate under backdoor attack. The attack success rate of baseline methods start increase when the model converges.
iterations when the benign updates reflect more distinctive features of their local data (Bagdasaryan al. 2018) and less shared common features. So the malicious updates are harder to be detected by the baselines after 25 iterations and the backdoor attack begins to success under baseline methods. We plotted the accuracy of the global model along with the attack success rate in Figure g to demonstrate the relation between model convergence and attack success rate.
A.7 OTHER POTENTIAL ATTACKS
In this section, we discuss a potential attack to our algorithm. Because we accumulate confidence from parameters to the model confidence, an adversarial model may be able to accumulate enough confidence if the attacker changes part of the model to an honest model.
Assuming that attackers observe the models from all users, they may bypass algorithm. We evaluate this possibility by mixing adversarial models with honest models to different ratios. We perform this attack the Amazon review dataset with 10 users, 4 of which are attackers. The attackers also train for 2 more epochs with learning rate 0.1 (the learning rate of regular users is 0.01).
The result shown in table The mix rate means the proportion of the honest model changed to adversarial parameters. Therefore, larger mix rate indicates more adversarial parameters in the model (mix rate means the model is entirely an adversary model). We use dropout to perform the blending, where the mix rate is the probability of dropping an honest parameter. This attack can be successfully defended by our algorithm.
Mix rate
0.5
0.3
0.1
0.03
0.01
0.001
0.0001
Accuracy 92.50%
92.58%
92.7 %
92.60%
92.42%
92.66%
92.56%
Table 8: The effect of blending adversarial models with honest models. Mix rate means which portion the parameters of model is changed to an adversarial model.
17


Reviews:
[{"review_id": "Skxw--ijYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an algorithm for mitigating poisoning attacks in federated learning settings and compares it , on four different datasets, against state-of-the-art baselines.\n\nExcept for some minor issues (see the list below), the paper is well-written and -organized. The description of the proposed algorithm (in pseudo code and using the illustration in Figure 2) is very clear. Overall, the experiments are carefully described.\n\nMy main concern is that many choices in the design of the proposed algorithm lack context/discussion and thus appear rather ad-hoc. For instance,\n- Why is the repeated median estimator used for estimating the linear regression? Could other robust estimators have been used?\n- Similarly, could alternative weighting schemes be used in equations (3)-(5)?\nI think it's important to provide more context and discuss possible alternatives. An important element is the exact threat model that the authors are considering. E.g., in the last paragraph on page 4, the authors mention specific attack strategies like altering only 10% of the model parameters. It appears that the design of the model weighting scheme aims at defending against these specific types of attacks. It will be good to either discuss or evaluate empirically how this scheme performs against other strategies.\n\nThe theoretical guarantee in Section 3.2 is a bit sketchy in my opinion. In what sense is $\\mu$ the \"expected value of the global model\"? I.e. what is the expectation over? Consequently, I could not follow the statement in equation (14). Some explanation in plain text is needed here, too: in what sense does this equation provide a guarantee?\n\nIn the experiments, several aspects deserve further discussion: (1) the poor performance of FoolsGold almost across the entire board (except for the Gaussian noise attacks), which may indicate that this method was applied outside the threat model it was designed for; (2) the failure of all the baselines on CIFAR-10 for the naive attacking approach, while they perform fairly well on MNIST; (3) why does the attack success rate starts increasing in Figure 4 for the baseline methods only after ~25 iterations? (4) why do the baseline methods perform so poorly against label-flipping against on MNIST (Figure 3) while performing fairly well on CIFAR-10 and Amazon reviews (Table 1/2)? - I think that answering those questions may shed insights into the type of attacks that the different defences can / cannot withstand. I'd also like to challenge the authors to address whether they expect their defence to match or outperform the baselines on *any* attack strategy, or whether they can come up with scenarios where some of the baselines perform better? I would expect that the latter should be possible; it would not diminish the value of the proposed defence but shed more clarity on its possible limitations.\n\nList of minor issues:\n- in the abstract: \"aggression\" -> \"aggregation\"\n- p.1: I would omit the statement in brackets \"less than 100 lines\". \n- p.2: some of the related work discussion repeats content from the introduction\n- p.3: \"summaries\" -> \"summarizes\"\n- p.3: what does that mean: \"has a high breakdown point of 50%\"? Please explain/clarify.\n- p.4: \"is the k-the diagonal of matrix in Hn\" -> \"is the k-th diagonal element of the matrix Hn\"\n- p.4: my pdf reader couldn't render the binary operator on the right hand side of equation (8)\n- p.5: \"the details of the proof is presented\" -> \"are\"\n- p.8: \"that of which\" -> \"whose\"\n- p.8: upper case \"We\" after comma\n- p.8: first column \"Acc\" in Table 3: FedAvg has the highest accuracy. Generally, bold numbers in tables do not always mark the best-performing method. Sometimes, bold numbers are entirely missing. In cases where the difference is insignificant (which often appears to be the case) I would mark multiple numbers in bold, as appropriate.\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HkxDJvg1cS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach to robust federated learning that uses robust regression to weigh all the model parameter coefficients in order to achieve the robustness. \n\nSpecifically, for each coefficient in the model, a repeated median estimator is used to compute a linear regression fit, and the residual of each individual model's coefficient is normalized and used to compute a confidence score for that coefficient. Coefficients which have too large a confidence have their confidence reset to zero, to avoid the influence of outliers. The local model's coefficients are now aggregated using a weighted average with weights given by these confidence scores.\n\nThey compare their algorithm experimentally to reasonable baselines of model aggregation algorithms: the FedAvg algorithm, 3 recent robust FL algorithm, and an approach based on a standard robust regression estimator, using experiments on 4 different datasets and 4 different neural net architectures. They test the robustness of these algorithms to label flipping (MNIST, CIFAR-10), backdoor attacks, and multiplicative gaussian noise corruption of the model coefficients.  \n\nOverall the paper presents an interesting and novel approach to robustness in FL, using a robust regression estimator to aggregate the model coefficients. The motivation of the algorithmic design is for the most part clear, but the rationale behind the particular choice of the parameter confidence score is unclear, and should be clarified. The theory in support of the method seems reasonablish, but key definitions and steps in the proof are not explained in detail, referring instead of an earlier paper. In particular, it is not clear how the smoothness of the loss function and subexponentiality of its derivatives enter into the analysis of the method, nor do these parameters enter into the final error bounds. Also, how is mu defined in the error bound: what does it mean that it is the expected global model --- does this mean this would be the model if all the participating workers were non-corrupted, honest, and had iid data? This theory is hard to parse: more effort should be spent in clarifying the assumptions and definitions and showing how the claimed result follows. The specific result referenced from earlier work should be stated unambiguously as a proposition so the reader sees how it applies where it is used.\n\nThe experimental results show that the algorithm performs slightly better than the considered baselines in most situations considered, but the important question of the impact of hyperparameter selection for the method (e.g. the clipping at which the weights of \"outlier\" parameters are set to zero) and the competing methods (e.g. the clipping in the trimmed mean estimator) is not addressed-- the authors indicate that the method is robust to some choices and fixes them in the appendix. This makes it difficult to tell whether the method performs better due to careful or lucky hyperparameter selection. \n\nAlthough the method is interesting and novel, and seems principled, the theoretical claims are unclear, and the experimental evaluation is not sufficiently informative about the impact of hyperparameter selection to draw conclusions about the effectiveness of this method of model aggregation as opposed to the baselines considered. In particular because of the latter issue, I'm leaning towards reject, but would be willing to change my score if this were addressed.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJxmkyoaFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an aggregation algorithm, based on repeated median regression and residual-based weighting to defend federated learning from adversarial attacks. Experiments are shown to demonstrate the method robustness against label-flipping, backdoor and Gaussian noise attacks.\n\nThe paper is interesting, the topic recent and the methodology quite new, but a number of comments arise: \n\n1) the methodology seems to rely on a number of ad-hoc steps, hyper parameter-dependent, that might hinder reproducibility and generalization: i) residual normalization via IRLS; ii) confidence assignment; iii) extreme value correction. Could be interesting to show analysis how varying such hyper-parameters affects the results, or otherwise to add further explanation at the comments in Appendix A.3 as to why the model seems to be insensitive to \\lambda, or why \\delta is significantly affected by data distribution.\n\n2) could the repeated median estimator still be affected by the “federated size” i.e. the number of models involved in the federated learning? Is there any bound on the number of participants, below which the estimator would perform poorly?\n\n3) proof of eq (14) could be more readable if\n\t•\tfull passages were shown (for instance for the reviewer the first passage was not immediate and took adding and subtracting \\sum_i z^(i)\\mu/\\sum_i z^(i), and further simplification to be addressed), and\n\t•\tii) reference to the exact point in which previous results are used were made explicit (i.e. where in (Yin et al 2018) the bounds are proven).\n\n4) proof of eq (14), when the attacker a\\in B is fixed, then |\\hat{y}^(i) - \\tilde{y}| should be replaced by  |\\hat{y}^(a) - \\tilde{y}|\n\n5) A last question concerns the aspect of \"fairness\" of this learning strategy. By removing aberrant updates there is still a chance of excluding from the learning process nodes that are intrinsically different form the average ones. In this sense, it is not clear from the paper how the reweighing strategy can mitigate this aspect, as there is no certainty that underrepresented data samples would not be rejected with the proposed scheme. Still aspect could have been better investigated in controlled scenarios.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an aggregation algorithm for federated learning that is robust against label-flipping, backdoor, and Gaussian noise attacks. The reviewers agree that the paper presents an interesting and novel method, however the reviewers also agree that the theory was difficult to understand and that the success of the methodology may be highly dependent on design choices and difficult-to-tune hyperparameters. 
Decision: Reject