Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper aims to answer the question of how to guarantee adversarial robustness in federated learning. In particular, this paper targets a challenging non-IID setting - the skewed label distribution, which gives rise to imbalanced class probabilities and heterogeneous local models, and proposed CalFAT to solve this problem by adaptively adjusting the logits. Theoretically, the authors prove that CalFAT can help learn homogeneous local models. Empirically, CalFAT outperforms other traditional centralized adversarial training methods (like AT, TRADES, MART, GAIRAT) and also the recent robustness propagation method (I.e., FedRBN) on several benchmarked datasets (like CIFAR-10, SVHN, CIFAR100).
strengths_and_weaknesses: Strengths:
+ The literature review is solid; summarizing the advantages and disadvantages of the literature are very helpful.
+ The paper points out why the previous methods hurt convergence of FAT and proposes novel techniques to tackle this issue that leads to better convergence and higher performance. 
+ The paper provides a solid theoretical analysis. 
+ Experimental evaluations are solid and well designed. The comparisons include the most recent FAT and centralized AT methods. 
+ The results are convincing and show that CalFAT outperforms state-of-the-arts.

Weaknesses:
- It is not clear why the natural performance degradation is considered as the main motivation (figure 1 on Page 2) instead of robust performance since the goal of FAT is gaining robustness. Adding more elaboration would be helpful.
- More discussions on the potential applications of FAT are expected to better highlight its significance.
- Even though sufficient references are given, a few relevant papers are recommended: “Privacy and Robustness in Federated Learning: Attacks and Defenses” that also touched on federated adversarial robustness; “Decision Boundary-aware Data Augmentation for Adversarial Training” that studied how to improve adversarial robustness by playing with decision boundary. These papers did not study exactly the same topic as this paper, but would certainly enrich the literature review further.

questions: - Why the natural performance degradation is considered as the main motivation (Figure 1 on Page 2) instead of robust performance since the goal of FAT is gaining robustness?
- I am wondering whether CalFAT also works in iid setting, can you elaborate more on this?


limitations: The authors did not discuss the limitations of the work.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper discusses the cause of instability of federated adversarial learning(FAT) is the skewed label distribution. An adapted calibrated loss is applied to FAT to reduce the heterogeneity of local models and further improve the final model’s natural and robust accuracy. Experimental results of four computer vision datasets are conducted to support the author’s arguments.
strengths_and_weaknesses: Strength

1 The authors provide extensive experimental results. 
2 The paper is well-written and easy to follow.

Weaknees

My biggest concern is the experiment design does not align well with the arguments.

The authors argue that the training instability comes from the skewed label distribution that leads to heterogeneity of the local model. I think it’d be true for both standard federated learning and federated adversarial training(FAT).  Eq.(11) is then proposed to calibrate the heterogeneity of local models for FAT. Therefore I expect at least two sets of experiments which I don’t see many of them in the main paper.

a) FAT baselines vs FAT baselines + Eq (11)
For example in Table 1, showing the results of {MixFAT, FedPGD, FedTRADES…} + Eq(11) would be more helpful to convince reviewer about the effectiveness of the proposed loss. 

b) FAT baselines + other label distribution calibration vs FAT baselines + Eq(11) 
Table 3 is a good example. However, the authors should explore it more. See Question 2 below.




Minor: 
The structure of the paper can be further polished. For example, the table 1 and table 2 are actually the same experiments, and should be combined into one. 

Line 241-242 is not necessary. I don’t see it helping validate any arguments made by authors.

questions: 1 Line 14-16, I read that the authors claim they theoretically show a much improved convergence rate, expecting a theoretical proof on convergence rate. Prop 1 and Prop 2 in a combined manner show a better finall convergence point, as opposed to a faster convergence rate. I think these are two different things.  


2 Line 259-260 says CKL loss can better generate adversarial examples. I suppose LogitAdj and RoBal do not associate with adversarial examples. Then my questions are:
    2.1) Do previous FAT baseline methods use the same adversarial training based on the same  
           set(eg, same class ratio, sample size, etc) of adversarial examples?
    2.2) What’s the setup to apply LogitAdj/RoBal comparing with CalFAT? 
    2.3) Is it possible to combine LogitAdj/RoBal with previous FAT baselines to compete with 
           CalFAT?  


3 Some experimental results are hard to tell the significance. For example, most numbers of AA column in Table 4 and Table 5 are very close. If the authors have conducted dfferent trials, standard deviation should be put there. 

limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a calibrated federated adversarial training (CalFAT) method, which is used in the case of label skewness. Empirical evaluation is done on several datasets including SVHN, CIFAR-10/100 and a subset of ImageNet.
strengths_and_weaknesses: Strengths:
- The empirical evaluation is done on several different datasets, and under strong attacks like AA.
- The improvement in clean accuracy seems significant.
- The writing is clear and easy to follow the main idea.

Weaknesses:
- The technical contribution is quite limited. CalFAT is just a simple adaption of [22] into federated adversarial training.
- The compared baselines (e.g., MixFAT and Fed**) are off-the-shelf, which do not adapt to the special property of label skewness. Stronger baselines utilizing label skewness should be involved.
- The considered setting of federated learning with label skewness may be of limited interest to the community.

questions: Limited technical contribution and weak baselines. 
limitations: No
review_title: null

Meta-review: metareview: This submission aims to ensure adversarial robustness in federated learning when label skewness exists among different local agents. The main idea of the proposed solution is to calibrate the logits to balance the predicted marginal label probabilities. Most of the reivewers found the topic studied in this work relevant, important and timely. The authors have also successfully addressed the concerns from the reviewers during the rebuttal. Hence I recommend acceptance.
confidence: Certain
Decision: Accept