Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes FedBVA for robust federated learning.  FedBVA first generates adversarial examples at the server-side, where these adversarial examples are those on which the global model incurs large losses.  Here, the authors choose them by designing a loss function: the sum of average loss (bias) and estimated variance.  The idea seems interesting, but I have a few doubts about its validity and the correctness of the experimental results. 
 
Concerns: 
- Secure aggregation:  The proposed scheme seems to require access to individual models, implying that it cannot be used together with secure aggregation.  This doesn't seem fixable as the attack algorithm requires access to individual predictions or individual gradients.  (See the variance gradient expression in p5.).  This somehow contradicts the introduction's claim ``our work bridges this gap ... error incurred during secure aggregation ...".  I might be missing something here, so please correct me if I am wrong.  Note that client data is not anymore private if the server has access to individual local models.

- Experimental results:  The ablation study is somewhat incomplete.  What about the performance of EAT + Fed_Bias or EAT + Fed_Variance?  Also, the experimental results need further justification.  Are those numbers based on a single run for each configuration?  Note that FGSM and PGD themselves are highly sensitive to random initializations.  Moreover, non-IID cases also require multiple runs to get a reliable performance estimation.  The performance of EAT significantly decreases as the attack gets stronger in Table 2 and Table 3.  Is EAT using the same set of adversarial examples for adversarial training?  What happens if EAT also uses more iterations when creating adversarial examples when running adversarial training?

- Why is FedAvg robust?: Also, some numbers don't make much sense to me.  For instance, see the first row of Table 1.  Why can't you achieve close-to-zero accuracy with PGD-10 when attacking the vanilla FedAvg model with high epsilon (eps = 0.3)?  What about FGSM?  Isnt' it supposed to bring it down the accuracy close to 10% or so?  For instance, the numbers in Table 1 of [Wang et al, "On the Convergence and Robustness of Adversarial Training"] show that PGD-10 can achieve 0% accuracy with eps = 0.3.  Unless the model trained with standard FedAvg is inherently robust against adversarial examples for some reason, which I doubt, I cannot understand these results. 

- Writing: It was tough to read the paper, especially the bias-variance trade-off part.  [Domingos 2000] (not [Pedro 2000]) helped me understand this part, but I believe that the authors can improve the readability of this part.  
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: 1/ Summary of contributions

This paper investigates the problem of training a neural network in a federated fashion so that the final model will be resilient to adversarial perturbation attacks. Assuming that a shared public dataset exists, a novel method called FedBVA (Bias-Variance attacks) is introduced: it consists in crafting new adversarial inputs on the server by minimising a custom loss function at each aggregation round, and sharing these new inputs to all clients.
Experimental results on standard image datasets (MNIST, fashion-MNIST, CIFAR) split in iid and non-iid (by class) settings show that the proposed approach yields a more robust model than standard FedAvg in both cases, at the expense of a slight drop in accuracy.

2/ Acceptance decision

Owing to its well conducted experimental section and the novelty of the problem tackled, I would tend to accept this paper, even if it has some weaknesses in the algorithm section.

3/ Supporting arguments

A/ Experimental results. The authors perform a thorough investigation of the proposed approach, comparing it with all reasonable baselines: local adversarial training (EAT), adversarial training done at the server level but with a more standard loss function (FedAvg_AT), individual terms of the custom loss function and the combination of their approach and EAT. The breadth of the investigation (baselines, 3 datasets, 2 attacks) and the number of ablation studies provided in the appendix is quite convincing.

B/ Novelty. I think this is the first time the problem of training an adversarially robust model in the Fl setting is investigated. Even if the clients are not assumed malicious here, it might open new perspectives for treating backdoor or byzantine attacks.

C/ Weakness in the algorithm section. FedBVA relies on a bias-variance decomposition of the loss (cross-entropy and MSE), which allegedly builds upon the formalism of Domingos ’00 and Valentini & Dietterich ’04. However, I have some doubts on the justification of this approach, for the following reasons:
- The authors slightly change the definition of the bias and variance, and in the cross-entropy case they are not equivalent due to the asymmetry of the cross-entropy loss in its both arguments. This slight change is instrumental in permitting a nice expression of these terms in Theorem 4, but not justified beyond the fact that both reduce to the same in the case of a symmetric loss (such as MSE).
- Further, none of the related works cited by the authors explicit the existence of this bias-variance decomposition in the case of a cross-entropy loss, which makes it difficult to check if the decomposition holds in this case.
- Last, but not least, the authors never explicit such a decomposition: they state it as a combination Bias + \lambda * Variance, but do not provide either a closed-form expression for \lambda nor a list of numerical values used in the experiments.

4/ Additional comments

1. In order to understand how significant the robust FL results are, it would be relevant to report the performance under attack of a model trained in a pooled fashion, i.e. with all data in 1 client.
2. From what I understand, this paper does not assume that clients are having a malicious behaviour, which is typically what previous works in FL tried to bring robustness to. It would be nice to comment on this difference, and it would help to clarify the problem tackled by this paper.
3. Could you provide more comments on the subtle differences in Figure 1? It is difficult to see them and it would be profitable in order to better understand the contributions of both loss terms.
4. Interestingly, the proposed FedBVA bears some similarity with distillation-based approaches, such as (Jeong et al. 2018 https://arxiv.org/abs/1811.11479) or (Chang et al. 2019, https://arxiv.org/abs/1912.11279). I would recommend citing these related works for reference.
5. The FedBVA approach needs to have access to individual updates sent by clients. In some FL settings, this is an unacceptable breach of privacy, and secure aggregation is used to ensure that the server only sees the mean update. In this case, the proposed FedAvg_AT is a better compromise than FedBVA.
6. For Figure 2, were experiments done on a trained or untrained network? Are the bias and variance terms computed on training or testing samples?
7. In Figure 4 and Figure 5, for which attack are is the model performance represented?
8. In the second paragraph of Section 5.1, what are the defence models? Is it the model used for FL training?

Some typos:

9. In Appendix A.2, in the derivation of the bias, the terms $\log(1-\log t)$ should be changed as $\log(1-t)$
10. The citation (Pedro, 2000) should be changed as (Domingos, 2000)
11. In Figure 2, the legend has 'variancee' instead of "variance"
12. In the first paragraph of section 5.1, " the baselines (5) and (6) " -> " (6) and (7) "
review_title: A good submission with some weaknesses

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a robust federated learning algorithm, where they assume that all samples are iid, and $n_s$ clean samples are available at the server side. The authors then go on to optimize a loss function that optimizes the aggregate loss and propose some new algorithms with experimental results. While overall the paper is interesting, there are several shortcomings in the execution as discussed below that the authors can address to improve the paper.

**C1.** Can you please clarify how your Definition 2.3 results in a *decomposition* in the sense of (Pedro 2000), especially given that you remark that this is different compared to the existing definition? Existence of such decomposition is not obvious at all.

**C2.** In bias-variance decomposition, the weights of different components is fixed. In your case $\lambda$ is left as a hyperparameter. Does this mean that you are no after a decomposition and rather would like to balance some notion of bias with a notion of variance. If that is the case, then you need to rewrite the paper to reflect that.

**C3.** In the beginning of Section 3, the authors write *A typical framework (Kairouz et al., 2019) of privacy-preserving federated learning can be summarized as follows*. Can you please comment on how this framework is privacy-preserving? 

**C4.** The authors assume that the data is IID, and also some shared and non-adversarial data is available at the central server. While it is okay to assume IID data for an initial theoretical analysis, the latter assumption is not justified especially when studying robustness. Can you please explain how such shared dataset is created?

**C5.** Can you please add statistical error bars to all tables and figures? 

**C6.** What happens if $n_s$ is small or even zero? The baselines you are comparing against do not assume the existence of clean data at the server side.

**C7.** What happens if adversary's power is arbitrary? I don't see any reason why a single adversary cannot completely destroy the model performance for all devices if they can do whatever they want with their updates. There is no explicit mechanism to defend against that in this paper.

**C8.** From Figure 5, it seems that the baseline EAT achieves a better tradeoff point between performance and robustness compared to all of the proposed algorithms. Can you please explain?

**C9.** Can you please do a more thorough study of the *tradeoffs curves between robustness and performance* similar to Figure 5? It would be good to see how the different methods compare in terms of the tradeoff between robustness and performance (and repeat that for different notions of robustness). The current tables reporting the results are inconclusive.
review_title: Interesting paper but falls short 

final_score: rating: 4: Ok but not good enough - rejection
review_content: Paper summary:

The paper studies adversarial robustness in the context of federated learning. The authors provide an algorithm for adversarial training that generates adversarial examples on a trusted public dataset and iteratively sends them to the clients, so that they can perform learning on the adversarial examples as well. Notably, the adversarial examples are created by inspecting both the bias and the variance of the current set of models. The method is tested empirically on a wide range of datasets and compared to adversarial training using the local clients' data.

##########################################################################

Pros:

- Both adversarial robustness and federated learning are important concepts in modern machine learning research and are of interest to a wide audience.
- In general, I find the idea of generating adversarial examples based on both the model bias and variance (instead of the bias only) quite interesting.
- The experimental evaluation is done on a wide variety of datasets and sufficient details (and code) are provided, so that reproducibility is ensured.

##########################################################################

Cons:

My main concerns are related to the way that the work is motivated and positioned with respect to prior work. I also find a partial mismatch between what the paper claims and what the experiments seem to suggest.

1. Initially, robustness in the context of federated learning is motivated as an important problem because federated learning could "lead to a degenerate model, which collapses in performance when the corrupted updates are uploaded and aggregated on the server" (citing the abstract) and because federated learning "is susceptible to clients' corrupted updates (e.g. system failures, adversarial manipulations, etc.)" (citing the introduction). I completely agree with these points and therefore I find the problem of robust federated learning an important one. 

However, these type of problems have to do with problems in the training procedure (for example, poisoning or backdoor attacks) that can lead to a poor model being learned. Instead, the problem addressed in the paper later on is the one of adversarial examples. As is well known, this is a problem that occurs at test time and is present even in models that are trained properly on clean data. Therefore, I see a mismatch between what problem is motivated and what problem is solved in the paper.

2. While I do think that considering bias and variance in the context of adversarial training is an interesting problem, the paper does not provide any motivation for such analysis. The bias-variance trade-off is a classic way to study the generalization of machine learning models. However, the problem of adversarial examples is present even in model that generalize well to (clean) test data. Why should considering the variance help in this case as well? Has the bias-variance trade-off been studied previously in the context of adversarial robustness (for models training a single dataset)? I believe that these are important points that should be discussed in the paper. 

3. I find some of the notation and definitions in the paper slightly confusing. In particular, Definition 2.2 features a bunch of expectations, while the distributions that these expectations correspond to are not defined. In addition, while $y_* (x)$ is a function, the right-hand side of its definition does not seem to involve a dependence on x. While I believe that I understand what is meant in these equations, I think that the preliminaries section should rewritten in a slightly more precise manner.

A similar problem is that the bias and variance notions in equation (3) are not really defined. Definition 2.3 introduces the bias and variance of a single model. Is the bias of K models the average bias of the K models, or the bias of the average of the K models? 

4. While the experiments are well-designed and enough detail is given for them to be reproducible, very often the baseline of (Tramer et al 2018) or a mixture of Tramer et al and FED_BVA outperforms the proposed method (FED_BVA). Moreover, it's hard to deduce from the experiments which method performs the best, since this changes between the datasets and also no error bars corresponding to multiple repeats of the experiments are given. In this sense, I think more analysis should be provided about when each of the algorithms should be preferred and in general how should the results be interpreted. 

##########################################################################

Review summary:

The paper studies an important problem and provides a wide set of experiments for testing various algorithms for adversarial training. I believe that an interesting set of ideas is present in the paper, but currently they are not systematized and motivated properly. It is therefore hard to evaluate the theoretical and experimental insights that are presented. This is why I do not recommend acceptance for the paper in its current shape.
review_title: Good on the technical side, but the motivation and setup are somewhat unclear

Meta-review: This paper proposes a method called Federated Bias-variance attacks (FedBVA) to generate adversarial examples for federated learning, which can be used to make the model more robust to adversarial attacks.  All the reviewers found the problem and the approach very interesting. Their concerns include the following main points (please see the reviews for more details):
* The decomposition of the bias and variance can be made more rigorous
* The necessity for a shared dataset of adversarial examples makes the application a little limited
* Need fairer experimental baselines
* Vanilla federated learning is not guaranteed to preserve privacy - the authors should edit this claim in the motivation
* Need compatibility with secure aggregation approaches where the central server cannot access local updates

The authors did do a great job of responding to the reviewers' comments. Given the interest in the problem and the novelty of the idea, I think an improved version of the paper would be quite well-received.
Decision: Reject