Adversarially Robust Federated Learning for Neural Networks
Authors: Yao Zhou
Jun Wu
Jingrui He
Venue:  (2021)
Keywords: federated learning
adversarial training
robustness
bias-variance decomposition

Abstract: In federated learning, data is distributed among local clients which collaboratively train a prediction model using secure aggregation. To preserve the privacy of the clients, the federated learning paradigm requires each client to maintain a private local training data set, and only uploads its summarized model updates to the server. In this work, we show that this paradigm could lead to a vulnerable model, which collapses in performance when the corrupted data samples (under adversarial manipulations) are used for prediction after model deployment. To improve model robustness, we first decompose the aggregation error of the central server into bias and variance, and then, propose a robust federated learning framework, named Fed_BVA, that performs on-device adversarial training using the bias-variance oriented adversarial examples supplied by the server via asymmetrical communications. The experiments are conducted on multiple benchmark data sets using several prevalent neural network models, and the empirical results show that our framework is robust against white-box and black-box adversarial corruptions under both IID and non-IID settings.  

Under review as a conference paper at ICLR 2021
ADVERSARIALLY ROBUST FEDERATED FOR NEURAL NETWORKS
LEARNING
ABSTRACT
In federated learning, data is distributed among local clients which collaboratively train prediction model using secure aggregation. To preserve the privacy of the clients, the federated learning paradigm requires each client to maintain a pri- vate local training data set, and only uploads its summarized model updates the server. In this work, we show that this paradigm could lead to a vulnerable model, which collapses in performance when the corrupted data samples (under adversarial manipulations) are used for prediction after model deployment. To improve model robustness, we first decompose the aggregation error of the cen- tral server into bias and variance, and then, propose robust federated learning framework, named Fed EVA, that performs on-device adversarial training using the bias-variance oriented adversarial examples supplied by the server via asym- metrical communications. The experiments are conducted on multiple benchmark data sets using several prevalent neural network models, and the empirical results show that our framework is robust against white-box and black-box adversarial corruptions under both IID and non-IID settings.
INTRODUCTION
The explosive amount of decentralized user data collected from the ever-growing usage of smart devices, c.g., smartphones, wearable devices, home sensors, etc., has led to surge of interest in the field of decentralized learning. To protect the privacy-sensitive data of the clients, federated learning (McMahan et al., 2017 Yang al.
2019 has been proposed. Federated learning only allows group of clients to train local models using their own data, and then collectively merges the model updates on a central server using secure aggregation (Acar al. 2018 Due its high privacy-preserving property, federated learning has attracted much attention in recent years along with the prevalence of efficient weight deep models (Howard al. 2017 and low-cost network communications (Wen et al. 2017; Konečný et al. 2016).
In federated learning, the central server only inspects the secure aggregation of the local models as whole. Consequently, it is susceptible to clients' corrupted updates (e.g., system failures, etc).
Recently, multiple robust federated learning models (Fang et al. 2019 Pillutla al. 2019 Portnoy
& Hendler 2020 Mostata 2019) have been proposed. These works only focus performing client- level robust training or designing server-level aggregation variants with hyper-parameter tuning for Byzantine failures. However, none of them have the ability to mitigate the federated learning's vulnerability when the adversarial manipulations are present during testing, which as we shown in Section that is mainly due to the generalization error in the model aggregation.
Our work bridges this gap by investigating the error incurred during the aggregation of federated learning from the perspective of bias-variance decomposition (Domingos) 2000; Valentini & Diet- terich 2004). Specifically, we show that the generalization eтror of the aggregated model on the central server can be decomposed as the combination of bias (triggered by the main prediction of these clients) and variance (triggered by the variations among clients' predictions). Next, we pro- pose to perform the local robust training on clients by supplying them with a tiny amount of the bias-variance perturbed examples generated from the central server via asymmetrical communica- tions. The experiments are conducted on neural networks with cross-entropy loss, however, other loss functions are also applicable as long as their gradients w.r.t. bias and variance are tractable to estimate. In this way, any gradient- adversarial training strategies (Goodrellow et al. 2015 Madry et al. 2018) could be used. Compared with previous work, our major contributions include:
We provide the exact solution of biri-nee analysis w.r.t. the generalization error which is perfectly suitable for neural network based federated learning. As comparison, performing adversarial attacks or training with conventional federated learning methods will only focus on the bias of the central model but ignore the variance.

---

Under review as a conference paper at ICLR 2021
We demonstrate that the conventional federated learning framework is vulnerable to the strong attacking methods with increasing communication rounds even if the adversarial training using the locally generated adversarial examples is performed on each client.
Without violating the clients' privacy, we show that providing tiny amount of bias-variance perturbed data from the central server to the clients through asymmetrical communication could dramatically improve the robustness of the training model under various settings.
2
PRELIMINARIES
2.1 SETTINGS
In federated learning, there is a central server and K different clients, each with access to a private training set Dr = [( 2h == where x. t and ny are the features, label, and number of training examples in the 1tth client (k: = 1,... K). Each data D; is exclusively owned by client is and will not be shared with the central server or other clients. In addition, there is small public training set D3 = (9)];= with n, training examples from the server that is shared with clients, where
n, < EK ng. Note that this will not break the privacy constraints, for example, hospitals (local devices) that contribute to a federated learned medical image diagnosis system could take few publicly accessible images as additional inputs. The goal of federated learning to train a global classifier f(() using knowledge from all the clients such that it generalizes well over test data Drest- The notation used this paper is summarized in the Appendix (see Table Sh.
2.2 PROBLEM DEFINITION
In this paper, we study the adversarial robustness of neural networks in federated learning setting, and we define robust decentralized learning as follows.
Definition 2.1. (Adversarially Robust Federated Learning)
Input: (1) set of private training data {Dx K on different clients: (2) Tiny amount of training data D. on the central server; (3) Learning algorithm f(-) and loss function L(-,)).
Output: A trained model on the central server that is robust against adversarial perturbation.
We would like to point out that our problem definition has the following properties: Asymmet- rical communication:
The asymmetrical communication between each client and server cloud is allowed: the server provides both global model parameters and limited shared data to the clients; while each client only uploads its local model parameters back to the server. Data distribution: All training examples on the clients and the server are assumed to follow the same data distribution. However, the experiments show that our proposed algorithm also achieves outstanding performance under the non-lll) setting, which could be common among personalized clients in real scenarios. Shared learning algorithm: All the clients are assumed to the identical model f()), including architectures as well as hyper-parameters (e.g., learning rate, local epochs, local batch size).
Remark. The basic assumption of this problem setting is that the learning process is clean (no malicious behaviors are observed during training). however, the intentionally generated adversarial poisoning data will be mixed with clean data during training. The eventual trained model being deployed on the devices will be robust against potential future adversarial attacks.
2.3 VAS-VARIANCE TRADE-OFF
Following (Domingos) 2000 Valentini & Dietterich 2004). define the optimal prediction, main prediction as well as the bias, variance, and noise for any real-valued loss function I((,,) as follows:
Definition 2.2. (Optimal Prediction and Main Prediction) Given loss function L(.,)) -) and learning
algorithm f()), optimal prediction ye and main prediction you for example are defined as:
(x) arg min IE, [yy t)] and ym (x) = arg min (ppL(ff(()))
(I)
where/ / and D are viewed as the random variables to denote the class label and training set, and So denotes the model trained on D. In short, the main prediction is the prediction whose average loss relative to all the predictions over data distributions is minimum, e.g., the main prediction for zero- one loss is the mode of predictions. In this work, we show that the main prediction is the average prediction of client models for mean squared (MSE) loss and cross-entropy (CE) loss in Section 4.1
'Our theuretical contribation mainly focuses on classiicatiinn using neural networks with cress entropy less and meun squared loss. However, the proposed framework generic allow use of other classification lees functions as well.
2

---

Under review as a conference paper at ICLR 2021
Definition 2.3. (Bias, Variance and Noise) Given loss function ((.,-) and learning algorithm f()), the expected loss Ep, t)] for an example x can be decomposed into bias, variance and noise as follows:
(xx) = L(ym, v.) and V(x) = ED (=([([((()) vm)) and ((x) = E,(T O) (2)
In short, bias is the loss incurred by the main prediction w.r.t. the optimal prediction, and variance is the average loss incurred by predictions w.r.t. the main prediction. Noise is conventionally assumed to be irreducible and independent to f(-)- Remark. Our definitions on optimal prediction, main prediction, bias, variance and noise slightly differ from previous ones (Domingos 2000 Valentini & Dietterich 2004). For example, conven- tional optimal prediction was defined as y. = arg miny EE [t. y),, and it is equivalent to our definition when loss function is symmetric over its arguments, i.e., L(y1, y2) L(y2, Note that this decomposition holds for any real-valued loss function in the binary setting (Domingos (200) with a bias & variance trade-off coefficient that has closed-form expression. For multi-class set- ting, we inherit their definition of bias & variance directly, and treat the trade-off coefficient as htpper-parmmttr X to tune because no closed-fome expression of is available.
THE PROPOSED FRAMEWORK
A typical framework (Kairouz et al. 2019 of rrvacy-preservvng federated learning can be summa- rized as follows: (1) Client Update: Each client updates local model parameters wy by minimizing the empirical loss over its own training set; (2) Forward Communication: Each client uploads its model parameter update to the central server; (3) Server Update: It synchronously aggregates the received parameters; (4) Backward Communication: The global parameters are sent back to the clients. Our framework follows the same paradigm but with substantial modifications as below.
Server Update. The server has two components: The first one uses FedAvg (MeMahan et al. 2017 algorithm to aggregate the local models' parameters, i.c., Wg = Aggregate((W),-. wx) 1==1 B8k Wik where n = Σx= 72g and Wi is the model parameters in the h client. Meanwhile, is designed to produce adversarially perturbed examples which could be induced by a poisoning attack algorithm for the usage of robust adversarial training.
n
another component another
It has been well studied (Belkin et al., 2019 Domingos 2000 Valentini & Dietterich 2004 that
in the classification setting, the generalization error of a learning algorithm on an example is deter- mined by the bias, variance, and irreducible noise as defined in Eq. 2 Similar to the previous work, we also assume a noise-free free learning scenario where the class label 7 is deterministic function of x (i.e.. if x sampled repeatedly, the same values of its class will be observed). This motivates us to generate the adversarial examples by attacking the bias and variance induced by clients' models as: max ((;; wT, ... .uk) + AV w1, wK) V(x, 1) E D. 2E((x)
(3)
where (6;; - wx) and V Vă;; w..... wx) could be empirically estimated from a finite num- ber of clients' parameters trained on local training sets (D1, D22- Dx} }. Here X is a hyper- parameter to measure the trade-off of bias and variance, and ((x)) is the perturbation constraint.
Note that D3 (on the server) is the candidate subset of all available training examples that would lead to their perturbed counterparts. This is a more feasible setting as compared to generating ad- versarial examples on clients' devices because the server usually has much powerful computational capacity in real scenarios that allows the usage of flexible poisoning attack algorithms. In this case, both poisoned examples and server model parameters would be sent back to each client (Backward Communication), while only clients' local parameters would be uploaded to the server (Forward Communication), i.e., the asymmetrical communication as discussed in Section 2.
Client Update. The robust training of one client's prediction model (i.c., wx) can be formulated as the following minimization problem.
mii
Σ LUD, + Σ LUD. wx), 1=1
(4)
j=1
where 25 E (zxf) is the perturbed examples that asymmetrically transmitted from the server.
This decompusition is based on the weighted sum of bias, variance, and noise. In general, is a non deterministic functiion Domingus when irreducible noise considered. Namely, if a is sampled repeatedly, different values will ohserved.
3

---

Under review as a conference paper at ICLR 2021
Remark. Insuitively, the bias measures the systematic loss of a learning algorithm, and the vari- ance measures the prediction consistency of the learner over different training sets. Therefore, our robust federated learning framework has the following advantages: (i) it encourages the clients to consistently produce the optimal prediction for perturbed examples, thereby leading to a better generalization performance; (ii) local adversarial training on perturbed examples allows to learn robust local model, and thus robust global model could be aggregated from clients.
Theoretically, we could still have another alternative robust federated training strategy:
miin max w(f( +x) 4k E (1,2,...... . E
(5)
where the perturbed training examples of each client k generated on local devices from D3 instead of transmitted from the server. This min-max formula is similar to (Madry et al. 2018, Tramer
ct al. 2018) where the inner maximization problem synthesizes the adversarial counterparts clean examples, while the outer minimization problem finds the optimal model parameters over perturbed training examples. Thus, each local robust model trained individually. nevertheless, poisoning attacks on device will largely increase the computational cost and memory usage. Meanwhile, it only considers the client-specific loss and is still vulnerable against adversarial examples with increasing communication rounds. Both phenomena observed in our experiments (see Fig. g and Fig. 5
ALGORITHM
4.1 BIAS- VARTANCE ATTACK
We first consider the maximization problem in Eq. using bias-variance based adversarial attacks. It aims to find the adversarial example & (from the original example æ) that would produce large bias and variance values w.r.t. clients' local models. Specifically, perturbation constraint å (2(x) forces the adversarial example è to be visually indistinguishable w.r.t. æ. Here we consider the well- studied l boounded adversaries (Goodfellow et al., 2015 Madry et al. 2018 Tramer et al. 2018 such that ((:x) := [ x: 1oo I < for a perturbation magnitude Furthermore, we propose to consider the following two gradient- based algorithms to generate adversarial examples.
Bias-variance based Fast Gradient Sign Method (BV-FGSM): Following FGSM (Goodfellow et al. 2013), linearizes the maximization problem in Eq.
4 with one-step attack as follows. #BV-PGSM + sign (V2 (B(:; "1,- (WK) + (ar; W1, ((,])) wk)))
(6)
Bias-variance based Projected Gradient Descent (BV-PGD): PGD can be considered as a multi- step variant of FGSM (Kurakin et al., 2017) and might generate powerful adversarial examples. This motivated us to derive a B-bbaed PGD attack:
wx) + AV w1,".. (,,"",,,))))) (7) where al is the adversarial example at the 1th step with the initialization 2 = r and Projoc) projects each step onto 2(x)).
a++ "BV-PGD == Proje(=) (&' + c sign (V2) (B(s';
Remark. The proposed framework could be naturally generalized to any gradient-based adversar- ial attack algorithms where the gradients of bias B-) and variance V(.) w.r.t. are tractable when estimated from finite training sets. Compared with the existing attack methods (Carlini Wagner 2017 Goodfellow et al. 2015) Kurakin et al. 2017 Moosavi-Dezoooli et al. 2016), our loss func- tion the adversary aims to optimize is a linear combination of bias and variance, whereas existing work mainly focused attacking the overall classificution error that considers bias only.
The following theorem states that bias B(-) and variance V(-) as well as their gradients over input could be estimated using the clients" models.
Theorem 4.1. Assume that L(.,-) is the cross-entropy loss function, then, the empirical esti- mated main prediction ym for an input example (a, has the following closed-form expression: ym (r; W1..... wx) )- = K >k= fD, (æ; wx). Furthermore, the empirical bias and variance, as well
as their gradients over an input & are estimated as follows:
1 K
(1:W1.... wk)
L(fo, (x; wx),
V( W1, wx) L(yme vm) H (ym)
K
1120 robustness is surely not the only option for robustness learning. However, we use this standard approuch to show the limitations of prior federated learning, evaluate the improvments our proposed framework.
4

---

Under review as a conference paper at ICLR 2021
Here, (ym) Lj=n yim log (
())
yim is the entropy of the main prediction ym and C' is the number of classes. Easily, we have their gradients in terms of the bias and vari- ance as (1, wK) $ E
VoL(JDA wx),t) and Vav .... wx) ,
(f) - Dj= ym 1)V & (x: wk). Details of the proof is elaborated in A.2
In addition, also consider the case where (!::-) is the MSE loss function. But the gradients of MSE's bias and variance are much more computational demanding comparing with the concise formulas that eross-entropy ends up with. More comparisons are illustrated in Appendix [...T
Algorithm 2 ClientUpdate((u, D.. Dz.k)
Algorithm Fed BVA
1: Input: K (number of clients, with local data
1: Initialize hthh client's model with w
sets (Dx } k====;;; i): f (learning model), E' (num- ber of local epochs); F (fraction of clients se- lected on each round); B (batch size of local client); 7) (learning rate); 'Ds (shared data set
B < -pliit Dj UDD into batches of size B
3: for each local epoch i = 1.2, F do
4: for local batch (x. t) E do
5:
w 4- (FL
server); (perturbation magnitude).
6: end for
2: Initialization: Initialize wi and D.
7: end for
8: Calculate fo. (2: wf), V V (2: ED;
3: for each round = 2. .. do
4: m max F 1)
9: return w, fo, (2; wf), a fDD (2) w)
5: S, <- randomly sampled m clients
6: for each client S, in parallel do
Algorithm 3 B EVAttack(ffD.. VofD. l Sp)
7:
wh. DD., VnJD, +
1: Initialize D. =
ClientUpdatee(ua DD,
2: for (x,,t) E D. do
8:
end for
3: Estimate the gradients B(z(2) and V=V(z)
9: D. t BVAAttack( (fD. VafD.llk S,)
using Theorem 4.
10: G + Aggregate(( wf E Sp)
4: Calculate å using Eq.
6 or 7 and add to D.
11: end for
5: end for
12:
return
6: return D.
4.2 FED BVA
We present a novel robust federated learning algorithm with our proposed bias-variance attacks, named Fed_ Following the framework defined in Eq. B and Eq. Ø key components of our algorithm are (1) bias-variance attacks for generating adversarial examples on the server, and (2) adversarial training using poisoned server examples together with clean local examples each client. Therefore, we optimize these two objectives by producing the adversarial examples D, and updating the local model parameters w iteratively.
The proposed algorithm is summarized in Alg. Given the server's D, and clients' training data (Dx | as input, the output is a robust global model on the server. In this case, the clean server data D, will be shared to all the clients. First, initializes the server's model parameter tg and perturbed data D., and then assigns to the randomly selected clients (Steps 4-5). Next, each client optimizes its own local model (Steps 6-8) with the received global parameters wG as well as its own clean data DR. and uploads the updated parameters well as the gradients of local model on each shared server example back the server. At last, the server generates the perturbed data D. (Step 9) using the proposed birs-nariance attack algorithm (see Alg. B with aggregations (model parameter average, bias gradients average, and variance gradients average) in the similar manner as FedAvg (McMahan et al. 2017p. These aggregations can be privacy secured if additive homomorphic encryption (Acar
et al.
2018 is applied.
5 EXPERIMENTS
5.1 SETTINGS
In this section, we evaluate the adversarial robustness of our proposed algorithm four bench- mark data sets: MNIST Fashion-MNIST CIFAR-10 and CIFAR-100 The baseline models
http:/ htpps//yaan... Lemun, con//xttb//mmiatt
htopss/ //wwe/////...o www.///w....... cs.toranto hwt..///ki///ffaa..het.
5

---

Under review as a conference paper at ICLR 2021
bis grrdicat gralinnt
0.40
bias
0.35
variance
0.3
025
0.20
0.5
0.10
005
number filters
Figure 1:
Visualizations of bias.
variance, Figure Bias-variance
w.r.. the CNN model
bias+variance, and perturbed images for MNIST.
complexity on MNIST.
redovy
Model Perforance (under attocki
Ferbuug
Test, Bies
red Variance
Net
training loss
2.0
CAT BVA
204
1:05
8.00
communications
3. of communcatioon
model
performance
Figure 3: Convergence (athion-MNiiTTPGGD--aaaattt
Figure 4: Performance Fashion-MNISTTPGGDD-2
Figure 5: Efficiency on Fashion- MNIST(PGID-20ataack)
used include: (1). Centralized the training with centralized model, which is identi- cal to the federated learning case that only has one client (A 1) with fraction (F = 1). (2). FedAvg: the classical federated averaging model (McMahan al. 2017). (3). FedAvg AT: The simplified version of our proposed method where the local clients perform adversarial training with the asymmetrical transmitted perturbed data generated top of FedAvg's 's aggregation. (4) (6).
Fed Bias, Fed_Variee wwt.iaarnncce..e Fed_BVE Our proposed methods where the asymmetrical trans- mitted perturbed data is generated using the gradients of bias-only attack, variance-only attack, and bias-variance attack, respectively. where each client performs local adversarial training using Eq. ( and their model updates are aggregated on server using F FedAvg. For fair comparisons, all baselines are modified to the asym- metrical communications setting (FedAvg and EAT have clean D, received), and all their initial- izations are set to be the same. (8). EAT+Fed_Br _BV A combination of baselines (6) and (7). Note that baselines (7) and (8) have high computational requirements on client devices, and are usually not preferred in real scenarios.
(7). EAT: Ensemble adversarial training (Tramer et al.) 2018),
For the defense model, we use a 4-layer CNN model for MNIST and Fashion- MNIST, and VGG9 architecture for CIFAR-10 and CIFAR- 100. Regarding blackbox attacks, apply ResNet18 (He ct al. 2016p. VGGI1 (Simonyan & Zisserman, 2015). Xception (Chollet) 2017). and Mo- bileNet (Sandler et al. 2018) for CIFAR data, and provide variety of models for MNIST and Fashion-MNIST by following the design of (Tramer et al. 2018p. The training is performed using the SGD optimizer with fixed learning rate of 0.01 and momentum of value 0.9. The trade-off co- efficient between bias and variance is set to X = 0.01 for all experiments. All hyper-parameters of federated learning are presented in Table 5 in the Appendix. We empirically demonstrate that these hyper-panne parammte settings are preferable in terms of both training accuracy and robustness (see the details of Fig. 6 Fig. [4 in the Appendix). To evaluate the robustness of our federated learning algorithm against adversarial attacks, except for the clean model training, we perform FGSM (Good- fellow al. 2015p, PGD (Kurakin al., 2017 7p with 10 and 20 steps towards the aggregated server model on the Drest Following (Tramer et al. 2018 Wang et al. 2019), the maximum perturbations allowed are e = 0.3 on MNIST and Rashion-MNISS and e 255 on CIFAR- 10 and CIFAR- 100 for both threat and defense models. For IID sampling, the data is shuffled and uniformly partitioned into each client; For non-III IID setting, data divided into 2F shards based on sorted labels, then assigns each client with shards. Thereby, each client will have data with most two classes.

---

Under review as a conference paper at ICLR 2021
5.2 RESULT ANALYSIS
To analyze the properties of our proposed Fed.BVA framework, we present two visualization plots on MNIST using a trained CNN model where the bias and variance are both calculated on the training examples. In Fig. 1 we visualize the extracted gradients using adversarial attack from bias, variance, and bias-variance. Notice that the gradients of bias and variance are similar but with subtle differences in local pixel areas. However, according to Theorem 4.] the gradient calculation of these two are quite different: bias requires the target label as input, but variance only needs the model output and main prediction. From another perspective, we also investigate the bias varianar magnitude relationship with varying model complexity. As shown in Fig. 2 with increasing model complexity (more convolutional filters in CNN), both bias and variance decrease. This result is different from the double-descent curve or bell-shape variance curve claimed in (Belkin et al, 2019 Yang et al. 2020 The reasons are twofold: First, their bias-variance definitions are from the MSE regression decomposition perspective, whereas our decomposition utilizes the concept of main prediction, and the generalization error is decomposed from the classification perspective; Second, their implementations only evaluate the bias and variance using training batches on one central model and thus is different from the definition which requires the variance to be estimated from multiple sub-models (in our scenario, client models).
The convergence plot of all baselines is presented in Fig. E] We observe that FedAvg has the best convergence, and all robust training will have a slightly higher loss upon convergence. This matches the observations in (Madry et al.] 2018) which state that training performance may be sacrificed in order to provide robustness for small capacity networks. For the model performance shown in Fig. E we observe that the aggregation of federated learning is vulnerable to adversarial attacks since both FedAvg and EAT have decreased performance with increasing number of server-client communications. Other baselines that utilized the asymmetrical communications have increasing robustness with more communication rounds although only a small number of perturbed examples (n, = 64) are transmitted. We also observe that when communication rounds reach 40, Fed_BVA starts to outperform EAT while the latter is even more resource-demanding than Fed_BVA (shown in Fig. 5 where the pie plot size represents the running time). Overall, bias- strsinarinnce based adversarial training via asymmetric communication is both effective and efficient for robust federated learning.
For the comprehensive experiments in Table D and Table 2 it is easy to verify that our proposed model outperforms all other baselines regardless of the source of the perturbed examples (i.e., locally generated like EAT+Fed_Ee _BBV or asymmetrically transmitted from the server like Fed_BVA). Com- paring with standard robust federated learning FedAvg_AT AT, the performance of Fed_BVA against adversarial attacks still increases 4% - 13% and 2% 9% on IID and non-IID settings respectively, although BVA is theoretically suitable for the cases that clients have IID samples. In Table B observe similar trend where F Fed Fed_BVA outperforms Fe redAvg_AT _AT CIFAR- 10 and CIFAR- 100 (with 0.2% - 10% increases) when defending different types of adversarial examples. Comparing with strong local adversarial training baseline EAT, we also observe maximum 13% accuracy in- crease when applying its bias-variance oriented baseline (AT+FeddBVV Overall, the takeaway is that without local adversarial training, using a bias-variance based robust learning framework will almost always outperform other baselines for defending FGSM and PGD attacks. When local adver- sarial training is allowed (e.g.. client device has powerful computation ability), using robust learning with local adversarial training will mostly have the best robustmess.
We also conducted various additional experiments in Appendix A.5 which includes: (1) Comparison of efficiency and effectiveness of Fed BVA using cross-entropy loss and MSF loss; (2) Compari-
son of single-step Fed_BVA _BV and multi-step Fed_BVA in terms of the generation of D.; (3) Three training scenarios of Fed_BVA that use client-specific adversarial examples or universal adversarial examples; (4) Ablation study in terms of the number of shared perturb examples n.. optimizer's 's mo- mentum, and the number of local epochs E; (5) Blackbox attacking transfcrability between various models on all four data sets under multiple settings.
RELATED WORK
Adversarial Machine Learning: While machine learning models have achieved remarkable perfor- mance over clean inputs, recent work (Goodfellow et al. 2015) showed that those trained models are vulnerable to adversarially chosen examples by adding the imperceptive noise to the clean inputs. In general, the adversarial robustness of centralized machine learning models have been explored from
the following aspects: adversarial attacks (Carlini & Wagner 2017| Athalyy al. 2018 Zhu al.)
7

---

Under review as a conference paper at ICLR 2021
IID
non-IID
Clean
FGSM
PGD-10
PGD-20
Clean
FGSM
PGD-10
PGD-20
Centralized
0.991
0,689
0.260=0.000
0.182++0.000
n/a
n/a
n/a
n/a
FedAvg
0.989_0.001
0.669
0.576_0.005
0.20710.014
0.980
0,491 10.057
0.15810.00 0.475
FodAvg.AT
0.988 000000
0.802+0.001
0.745
0.512+0.042
0.974
0.649
0.615
0.363
Fed_Bias
0.986-0.000
0.812+0.000
0.001
0..83
20.004
0.679+0.040
=.007
0.3944
Fed_ Fed_Varianee Variance
0.985_o.00
0.803+0.007
0.779=0.014
0.572-20.019
0.973
0.684
0.395 0.622.20.49
ox
Fed_BVA
0.986_0.001
0,818
0.804g0.009
0.61
0.969
0,705
0. 0.664200013
0,469 20.001
EAT
0.981 0.000
0.902110001
0.907
0.811 10.004
0.972
0.789
0.721 10.01e
0,415 10.035
EAT+Fed BVA
0.980_o.001
0 10,005
0.910
0.821 10.013
0.965
0.811 10.020
0.831 =0.013
0.670
Table 1: Accuracy of MNIST under white-box attacks in IID and non-IlI settings
IID
non-IID
Clcan
FGSM
PGD-10
PGD-20
Clean
FGSM
PGD-10
PGD-20
Centralized
0.882
0,229
0.010=0.000
0.00920.000
n/a
n/a
n/a
n/a
FedAvg
0.877=0.001
0.300+0.021
0.072-00016
0.036+0.016
0.804-0.013
0.19344.036
0.061
0.017+0.008
FcdAvg_AT
0.866=0.001
02
0.170
0.13920.01
0.730
0.445
0.13020.041
0.087:20002
Fed. Bias
0.862_..001
10.05
0.159 0.19920.007
10.000
0.709 20002
0,460
0.14920.00
1.115
Fed Variance
0.862
0.862_o.002
0,49610.012
0.157 20.012
10.017
0.719 10.006
0.4991..000
0,188
10.025
0.120
Fed BVA
0.862_0.000
0.528
0.210=0.028
0. 20000
0.710
0.495
0.141 20.021
0.093
EAT
0.860=o.005
0.773
0,103 0.191=0.012
0,597 0.791
0.027 0.071=0.050
EAT-Fed-BVA
0.838
0.357 0.715
0.735-20.220
0.632
0.1 0.164
0,106
Table 2: Accuracy of Fashion-MNIST under white-box attacks in IID and non-IID settings
CIFAR-10
CIFAR-100 100
Clean
FGSM
PGD-10
PGD-20
Clean
FGSM
PGD-10
PGD-20
Centralized
0.903
0,28810.001
0.074 0.206
0.741 20.003
0,16610.0r
0.04920.m0
0.032.10.003
FedAvg
0.890_o.002
10.022
0.207
0.062.0.008
0.730
0.161 10.000
0.113
0.03510.008
FedAvg. AT
0.890_..000
0.280
0.295
20.04
20.00
0.162:20.006
0.064=0.007
0.048
Fed_Bias
0.890_0.004
0,280
0.297=0.011
0.103
0.702=#0.0z2
0.163
0.165
0.061
Fed Variance
0.889_0.001
0.267
0.276_-00006
0,092+0.000
0.710=0.007
0,161
0.157=0.010
0,045
Fed BVA
0,889
0,286g0.013
0.30120.003
0.104
0,709
0.163
0.165g0.000
0.062240.05
EAT
0.833_0.000
0.596
0.66720.0m
0.561 10.002
0.661 20.001
0.267 10:00z
0.206
0.188
BAV+
0.833_0.003
0.59810..oz
0.668 2o.01
0.564400.03
0.657
0.272.0.000
0.33220.000
0,211 10.002
Table 3: Accuracy of CIFAR-10 and CIFAR-100 under white-box attacks
2019 9
defense (or robust model training) Madry et al.
2018 Carlini et al. 2019 Tramer et al.
2018
and interpretable adversarial robustness (chmiidt et al. 2018 Tiippas et al.
2018).
Federated Learning: Federated learning with preserved privacy (Konečný et al. 2016 McMahan et al. 2017 Hard et al. 2018) and knowledge distillation (Chang et al.) 2019; Jeong et al.) 201 01
has become prevalent in recent years. Meanwhile, the vulnerability of federated learning to back-
door attacks has also been explored by (Bagdasaryan et al. 2018, Bhagoji et al.] 2019, Xie et al.)
2019 Following their work, multiple robust federated learning models (Fang et al.) 2019)] Pillutla al. 2019 Portnoy & Hendler 2020 Mostafa 2019) are also proposed and studied. In this paper, we studied the federated learning's 's adversarial vulnerability after model deployment from the per- spective of bias-variance analysis. This is in sharp contrast to the existing work that focused on the model robustness against the Byzantine failures.
Bias- Variance Decomposition: varianariance decomposition (Geman et al. introduced to analyze the generalization error of a learning algorithm. Then, a generalized bias- variance decomposition (Domingos) 2000 Valentini & Dietterich 2004) was studied in the classifi- cation setting which enabled flexible loss functions (e.g., squared loss, zero-one loss). More recently, bias-variance trade-off was experimentally evaluated on modern neural network models (Neal al.)
1992) was originally
2018 Belkin et al. 2019 Yang et al. 2020
7 CONCLUSION
In this paper, we proposed a novel robust federated learning framework, in which the aggregation incurred loss during the server's aggregation is dissected into a bias part and a variance part. Our ap- proach improves the model robustness through adversarial training by supplying few bias-variance perturbed samples to the clients via asymmetrical communications. Extensive experiments have been conducted where we evaluated its performance from various aspects on several benchmark data sets. We believe the further exploration of this direction will lead to more findings on the robustness of federated learning.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Abbas Acar, Hidayet Aksu, A. Selcuk Uluagac, and Mauro Conti. A survey on homomorphic encryption schemes: Theory and implementation. ACM Comput. Surv. 51(4):79:1-79:35, 2018.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. In International Conference on Machine Learning, pp. 284-293, 2018.
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. arXiv preprint
00049, 2018
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical biri-variance trade-off. Proceedings of the National Academy of Sciences, 116(32): 15849955584915854 2019.
Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing federated learning through an adversarial lens. In International Conference on Machine Learning, pp. 634- 643, 2019.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy, pp. 39-57. IEEE, 2017.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, lan J. Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. CoRR, abs/1902.06705, 2019.
Hongyan Chang, Virat Shejwalkar. Reza Shokri, and Amir Houmansadr. Cronus: Robust and het- erogeneous collaborative learning with black-box knowledge transfer. CoRR, abs/1912.11277 2019.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 1800-1807, 2017.
Pedro Domingos. A unified bias-variance decomposition and its applications. In International Conference on Machine Learning, pp. 231-238, 2000.
Minghong Fang, Xiaoyu Cao, Jinyuan Jía, and Neil Zhenqiang Gong. Local model poisoning attacks to byzantine-robut federated learning. CoRR, abs/1911.11888 2019.
Stuart Geman, Elie Bienenstock, and René Doursat, Neural networks and the bias/variance dilemma. Neural computation, 4(1):1-58, 1992.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv: / 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 20/6 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 770- 778. IEEE Computer Society, 2016.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, abs/1704.04861, 2017.
Eunjeong Jeong, Seungeun Oh, Hyesung Kim, Jihong Park, Mehdi Bennis, and Scong-Lyun Kim. Communication-effiieenn on-device machine learning: Federated distillation and augmentation under non-iid private data. CoRR, 1bs/1811.111 1149, 2018
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint arXiv: 9922,,92,9777 2019.

---

Under review as a conference paper at ICLR 2021
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492, 2016.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In International Conference Learning Representations, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Reprexentations, 2018
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-effiieenn learning of deep networks from decentralized data. In Artificial Intelli- gence and Statistics, pp. 2773-122,.2 2017.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. DeepFool: A simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2574-2582, 2016.
Hesham Mostafa. Robust federated learning through representation matching and adaptive hyper- parameters. CoRR, abs/ 191. 13075, 2019.
Brady Neal, Sarthak Mittal, Aristide Baratin, Vinayak Tantia, Matthew Scicluna, Simon Lacoste- Julien, and Ioannis Mitliagkas. A modern take on the bias-variance tradcoff neural networks arXiv preprint
2018.
Venkata Krishna Pilluda. Sham M. Kakade, and Zaid Harchaoui. Robust aggregation for federated 2019.
learning. CoRR,
Amit Portnoy and Danny Hendler. Towards realistic byzantine-robust federated learning. CoRR, 04420004.4986. 2020.
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov. and Liang-Chich Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 4510 2018.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adver- sarially robust generalization requires more data. Advances in Neural Information Processing Systems, pp. 5014-5026, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR, 2015.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Drew McDaniel. Ensemble adversarial training: Attacks and defenses. International Conference on Learning Representations, 2018.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representa- fions, 2018.
Giorgio Valentini and Thomas G Dietterich. Bias-variance analysis of support vector machines for the development of svm-based ensemble methods. Journal of Machine Learning Research, pp- 725-775, 2004.
Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In Proceedings of the 36th International Conference on Machine Learning. volume 97, pp. 5586-6595, 2019.
Wei Wen, Cong Xu, Feng Yan, Chumpeng WU, Yandan Wang, Yiran Chen, and Hai Li. Tergrad: Ternary gradients to reduce communication distributed deep learning. Advances in Neural Information Processing Systems, pp. 1509-1519, 2017.
10

---

Under review as a conference paper at ICLR 2021
Chulin Xie, Keli Huang, Pin-Yu Yu Chen, and Bo Li. DBA: Distributed backdoor attacks against federated learning. In International Conference on Learning Representations, 2019.
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Trans. Intell. Syst. Technol., 10Q2):2:1-12:199 2019.
Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking bias-variance trade-off for generalization of neural networks. arXiv preprint arXiv:2002. 13228, 2020.
Chen Zhu, W Ronny Huang, Hengduo Li, Gavin Taylor, Christoph Studer, and Tom Goldstein. Transfcrable clean-label poisoning attacks on deep neural nets. In International Conference on Machine Learning, pp. 7614-7623, 2019.


Reviews:
[{"review_id": "s2UcdSPmf2q", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a robust federated learning algorithm, where they assume that all samples are iid, and $n_s$ clean samples are available at the server side. The authors then go on to optimize a loss function that optimizes the aggregate loss and propose some new algorithms with experimental results. While overall the paper is interesting, there are several shortcomings in the execution as discussed below that the authors can address to improve the paper.\n\n**C1.** Can you please clarify how your Definition 2.3 results in a *decomposition* in the sense of (Pedro 2000), especially given that you remark that this is different compared to the existing definition? Existence of such decomposition is not obvious at all.\n\n**C2.** In bias-variance decomposition, the weights of different components is fixed. In your case $\\lambda$ is left as a hyperparameter. Does this mean that you are no after a decomposition and rather would like to balance some notion of bias with a notion of variance. If that is the case, then you need to rewrite the paper to reflect that.\n\n**C3.** In the beginning of Section 3, the authors write *A typical framework (Kairouz et al., 2019) of privacy-preserving federated learning can be summarized as follows*. Can you please comment on how this framework is privacy-preserving? \n\n**C4.** The authors assume that the data is IID, and also some shared and non-adversarial data is available at the central server. While it is okay to assume IID data for an initial theoretical analysis, the latter assumption is not justified especially when studying robustness. Can you please explain how such shared dataset is created?\n\n**C5.** Can you please add statistical error bars to all tables and figures? \n\n**C6.** What happens if $n_s$ is small or even zero? The baselines you are comparing against do not assume the existence of clean data at the server side.\n\n**C7.** What happens if adversary's power is arbitrary? I don't see any reason why a single adversary cannot completely destroy the model performance for all devices if they can do whatever they want with their updates. There is no explicit mechanism to defend against that in this paper.\n\n**C8.** From Figure 5, it seems that the baseline EAT achieves a better tradeoff point between performance and robustness compared to all of the proposed algorithms. Can you please explain?\n\n**C9.** Can you please do a more thorough study of the *tradeoffs curves between robustness and performance* similar to Figure 5? It would be good to see how the different methods compare in terms of the tradeoff between robustness and performance (and repeat that for different notions of robustness). The current tables reporting the results are inconclusive.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GSSHQ-THmKq", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Paper summary:\n\nThe paper studies adversarial robustness in the context of federated learning. The authors provide an algorithm for adversarial training that generates adversarial examples on a trusted public dataset and iteratively sends them to the clients, so that they can perform learning on the adversarial examples as well. Notably, the adversarial examples are created by inspecting both the bias and the variance of the current set of models. The method is tested empirically on a wide range of datasets and compared to adversarial training using the local clients' data.\n\n##########################################################################\n\nPros:\n\n- Both adversarial robustness and federated learning are important concepts in modern machine learning research and are of interest to a wide audience.\n- In general, I find the idea of generating adversarial examples based on both the model bias and variance (instead of the bias only) quite interesting.\n- The experimental evaluation is done on a wide variety of datasets and sufficient details (and code) are provided, so that reproducibility is ensured.\n\n##########################################################################\n\nCons:\n\nMy main concerns are related to the way that the work is motivated and positioned with respect to prior work. I also find a partial mismatch between what the paper claims and what the experiments seem to suggest.\n\n1. Initially, robustness in the context of federated learning is motivated as an important problem because federated learning could \"lead to a degenerate model, which collapses in performance when the corrupted updates are uploaded and aggregated on the server\" (citing the abstract) and because federated learning \"is susceptible to clients' corrupted updates (e.g. system failures, adversarial manipulations, etc.)\" (citing the introduction). I completely agree with these points and therefore I find the problem of robust federated learning an important one. \n\nHowever, these type of problems have to do with problems in the training procedure (for example, poisoning or backdoor attacks) that can lead to a poor model being learned. Instead, the problem addressed in the paper later on is the one of adversarial examples. As is well known, this is a problem that occurs at test time and is present even in models that are trained properly on clean data. Therefore, I see a mismatch between what problem is motivated and what problem is solved in the paper.\n\n2. While I do think that considering bias and variance in the context of adversarial training is an interesting problem, the paper does not provide any motivation for such analysis. The bias-variance trade-off is a classic way to study the generalization of machine learning models. However, the problem of adversarial examples is present even in model that generalize well to (clean) test data. Why should considering the variance help in this case as well? Has the bias-variance trade-off been studied previously in the context of adversarial robustness (for models training a single dataset)? I believe that these are important points that should be discussed in the paper. \n\n3. I find some of the notation and definitions in the paper slightly confusing. In particular, Definition 2.2 features a bunch of expectations, while the distributions that these expectations correspond to are not defined. In addition, while $y_* (x)$ is a function, the right-hand side of its definition does not seem to involve a dependence on x. While I believe that I understand what is meant in these equations, I think that the preliminaries section should rewritten in a slightly more precise manner.\n\nA similar problem is that the bias and variance notions in equation (3) are not really defined. Definition 2.3 introduces the bias and variance of a single model. Is the bias of K models the average bias of the K models, or the bias of the average of the K models? \n\n4. While the experiments are well-designed and enough detail is given for them to be reproducible, very often the baseline of (Tramer et al 2018) or a mixture of Tramer et al and FED_BVA outperforms the proposed method (FED_BVA). Moreover, it's hard to deduce from the experiments which method performs the best, since this changes between the datasets and also no error bars corresponding to multiple repeats of the experiments are given. In this sense, I think more analysis should be provided about when each of the algorithms should be preferred and in general how should the results be interpreted. \n\n##########################################################################\n\nReview summary:\n\nThe paper studies an important problem and provides a wide set of experiments for testing various algorithms for adversarial training. I believe that an interesting set of ideas is present in the paper, but currently they are not systematized and motivated properly. It is therefore hard to evaluate the theoretical and experimental insights that are presented. This is why I do not recommend acceptance for the paper in its current shape.", "extra_scores": {}, "extra_text": {}}, {"review_id": "DNM7DP5fTer", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1/ Summary of contributions\n\nThis paper investigates the problem of training a neural network in a federated fashion so that the final model will be resilient to adversarial perturbation attacks. Assuming that a shared public dataset exists, a novel method called FedBVA (Bias-Variance attacks) is introduced: it consists in crafting new adversarial inputs on the server by minimising a custom loss function at each aggregation round, and sharing these new inputs to all clients.\nExperimental results on standard image datasets (MNIST, fashion-MNIST, CIFAR) split in iid and non-iid (by class) settings show that the proposed approach yields a more robust model than standard FedAvg in both cases, at the expense of a slight drop in accuracy.\n\n2/ Acceptance decision\n\nOwing to its well conducted experimental section and the novelty of the problem tackled, I would tend to accept this paper, even if it has some weaknesses in the algorithm section.\n\n3/ Supporting arguments\n\nA/ Experimental results. The authors perform a thorough investigation of the proposed approach, comparing it with all reasonable baselines: local adversarial training (EAT), adversarial training done at the server level but with a more standard loss function (FedAvg_AT), individual terms of the custom loss function and the combination of their approach and EAT. The breadth of the investigation (baselines, 3 datasets, 2 attacks) and the number of ablation studies provided in the appendix is quite convincing.\n\nB/ Novelty. I think this is the first time the problem of training an adversarially robust model in the Fl setting is investigated. Even if the clients are not assumed malicious here, it might open new perspectives for treating backdoor or byzantine attacks.\n\nC/ Weakness in the algorithm section. FedBVA relies on a bias-variance decomposition of the loss (cross-entropy and MSE), which allegedly builds upon the formalism of Domingos ’00 and Valentini & Dietterich ’04. However, I have some doubts on the justification of this approach, for the following reasons:\n- The authors slightly change the definition of the bias and variance, and in the cross-entropy case they are not equivalent due to the asymmetry of the cross-entropy loss in its both arguments. This slight change is instrumental in permitting a nice expression of these terms in Theorem 4, but not justified beyond the fact that both reduce to the same in the case of a symmetric loss (such as MSE).\n- Further, none of the related works cited by the authors explicit the existence of this bias-variance decomposition in the case of a cross-entropy loss, which makes it difficult to check if the decomposition holds in this case.\n- Last, but not least, the authors never explicit such a decomposition: they state it as a combination Bias + \\lambda * Variance, but do not provide either a closed-form expression for \\lambda nor a list of numerical values used in the experiments.\n\n4/ Additional comments\n\n1. In order to understand how significant the robust FL results are, it would be relevant to report the performance under attack of a model trained in a pooled fashion, i.e. with all data in 1 client.\n2. From what I understand, this paper does not assume that clients are having a malicious behaviour, which is typically what previous works in FL tried to bring robustness to. It would be nice to comment on this difference, and it would help to clarify the problem tackled by this paper.\n3. Could you provide more comments on the subtle differences in Figure 1? It is difficult to see them and it would be profitable in order to better understand the contributions of both loss terms.\n4. Interestingly, the proposed FedBVA bears some similarity with distillation-based approaches, such as (Jeong et al. 2018 https://arxiv.org/abs/1811.11479) or (Chang et al. 2019, https://arxiv.org/abs/1912.11279). I would recommend citing these related works for reference.\n5. The FedBVA approach needs to have access to individual updates sent by clients. In some FL settings, this is an unacceptable breach of privacy, and secure aggregation is used to ensure that the server only sees the mean update. In this case, the proposed FedAvg_AT is a better compromise than FedBVA.\n6. For Figure 2, were experiments done on a trained or untrained network? Are the bias and variance terms computed on training or testing samples?\n7. In Figure 4 and Figure 5, for which attack are is the model performance represented?\n8. In the second paragraph of Section 5.1, what are the defence models? Is it the model used for FL training?\n\nSome typos:\n\n9. In Appendix A.2, in the derivation of the bias, the terms $\\log(1-\\log t)$ should be changed as $\\log(1-t)$\n10. The citation (Pedro, 2000) should be changed as (Domingos, 2000)\n11. In Figure 2, the legend has 'variancee' instead of \"variance\"\n12. In the first paragraph of section 5.1, \" the baselines (5) and (6) \" -> \" (6) and (7) \"", "extra_scores": {}, "extra_text": {}}, {"review_id": "DKUo3laFv0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes FedBVA for robust federated learning.  FedBVA first generates adversarial examples at the server-side, where these adversarial examples are those on which the global model incurs large losses.  Here, the authors choose them by designing a loss function: the sum of average loss (bias) and estimated variance.  The idea seems interesting, but I have a few doubts about its validity and the correctness of the experimental results. \n \nConcerns: \n- Secure aggregation:  The proposed scheme seems to require access to individual models, implying that it cannot be used together with secure aggregation.  This doesn't seem fixable as the attack algorithm requires access to individual predictions or individual gradients.  (See the variance gradient expression in p5.).  This somehow contradicts the introduction's claim ``our work bridges this gap ... error incurred during secure aggregation ...\".  I might be missing something here, so please correct me if I am wrong.  Note that client data is not anymore private if the server has access to individual local models.\n\n- Experimental results:  The ablation study is somewhat incomplete.  What about the performance of EAT + Fed_Bias or EAT + Fed_Variance?  Also, the experimental results need further justification.  Are those numbers based on a single run for each configuration?  Note that FGSM and PGD themselves are highly sensitive to random initializations.  Moreover, non-IID cases also require multiple runs to get a reliable performance estimation.  The performance of EAT significantly decreases as the attack gets stronger in Table 2 and Table 3.  Is EAT using the same set of adversarial examples for adversarial training?  What happens if EAT also uses more iterations when creating adversarial examples when running adversarial training?\n\n- Why is FedAvg robust?: Also, some numbers don't make much sense to me.  For instance, see the first row of Table 1.  Why can't you achieve close-to-zero accuracy with PGD-10 when attacking the vanilla FedAvg model with high epsilon (eps = 0.3)?  What about FGSM?  Isnt' it supposed to bring it down the accuracy close to 10% or so?  For instance, the numbers in Table 1 of [Wang et al, \"On the Convergence and Robustness of Adversarial Training\"] show that PGD-10 can achieve 0% accuracy with eps = 0.3.  Unless the model trained with standard FedAvg is inherently robust against adversarial examples for some reason, which I doubt, I cannot understand these results. \n\n- Writing: It was tough to read the paper, especially the bias-variance trade-off part.  [Domingos 2000] (not [Pedro 2000]) helped me understand this part, but I believe that the authors can improve the readability of this part.  ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method called Federated Bias-variance attacks (FedBVA) to generate adversarial examples for federated learning, which can be used to make the model more robust to adversarial attacks.  All the reviewers found the problem and the approach very interesting. Their concerns include the following main points (please see the reviews for more details):
* The decomposition of the bias and variance can be made more rigorous
* The necessity for a shared dataset of adversarial examples makes the application a little limited
* Need fairer experimental baselines
* Vanilla federated learning is not guaranteed to preserve privacy - the authors should edit this claim in the motivation
* Need compatibility with secure aggregation approaches where the central server cannot access local updates

The authors did do a great job of responding to the reviewers' comments. Given the interest in the problem and the novelty of the idea, I think an improved version of the paper would be quite well-received.
Decision: Reject