CalFAT: Calibrated Federated Adversarial Training with Label Skewness
Authors: Chen Chen
Yuchen Liu
Xingjun Ma
Lingjuan Lyu
Venue: NeurIPS 2022 Accept (2022)
Keywords: federated learning
adversarial training

Abstract: Recent studies have shown that, like traditional machine learning, federated learning (FL) is also vulnerable to adversarial attacks.
To improve the adversarial robustness of FL, federated adversarial training (FAT) methods have been proposed to apply adversarial training locally before global aggregation. Although these methods demonstrate promising results on independent identically distributed (IID) data, they suffer from training instability on non-IID data with label skewness, resulting in degraded natural accuracy. This tends to hinder the application of FAT in real-world applications where the label distribution across the clients is often skewed. In this paper, we study the problem of FAT under label skewness, and reveal one root cause of the training instability and natural accuracy degradation issues: skewed labels lead to non-identical class probabilities and heterogeneous local models. We then propose a Calibrated FAT (CalFAT) approach to tackle the instability issue by calibrating the logits adaptively to balance the classes. We show both theoretically and empirically that the optimization of CalFAT leads to homogeneous local models across the clients and better convergence points.

CalFAT: Calibrated Federated Adversarial Training with Label Skewness
Chen Chen Zhejiang University
Yuchen Liu Zhejiang University
Xingjun Ma
Lingjuan Lyu Sony Al
Fudan University
Abstract
Recent studies have shown that, like traditional machine learning. federated Iearn- ing (FL) is also vulnerable to adversarial attacks. To improve the adversarial robustness FL, federated adversarial training (FAT) methods have been proposed apply adversarial training locally before global aggregation. Although these methods demonstrate promising results on independent identically distributed (IID) data, they suffer from training instability on non-IID data with label skewness, resulting in degraded natural accuracy. This tends to hinder the application of FAT in real-world applications where the label distribution across the clients is often skewed. In this paper, we study the problem of FAT under label skewness, and reveal one root cause of the training instability and natural accuracy degradation issues: skewed labels lead to non-identical class probabilities and heterogeneous local models. We then propose Calibrated FAT (CalFAT) approach tackle the instability issue by calibrating the logits adaptively to balance the classes. We show both theoretically and empirically that the optimization of CalFAT leads homogeneous local models across the clients and better convergence points.
1 Introduction
Federated learning (FL) is a privacy-aware learning paradigm that allows multiple participants (clients) to collaboratively train global model without sharing their private data [23, 28, 43, 29]. In FL., each client follows the conventional machine learning procedure to train a local model on its own data and periodically uploads the local model updates central server for global aggregation. However, recent studies have shown that, like conventional machine learning, FL is also vulnerable to well-crafted adversarial examples [20, 47, 45], i.e., inference time, attackers human-perceptiin adversarial perturbations to the test examples to fool the global model with high success rates. This raises security and reliability concerns on the implementation of FL in real-world scenarios where such vulnerability could cause heavy losses [38]. For example, for cross-silo FL in the biomedical domain, a vulnerable global model may cause misdiagnosis, wrong medical treatments, or even the loss of lives. Similarly, in financial-base cross-silo FL, the lack of adversarial robustness may lead to huge financial losses. It thus imperative develop a robust FL method that can train adversarially robust global models resistant to different types of adversarial attacks.
add small,
In conventional machine learning, adversarial training (AT) has been shown to be of the most effective defenses against adversarial attacks [22, 40, 4]. Since the local training in FL is the same conventional machine learning, recent works [47, 45] proposed to perform local AT to
"Wor done during internship at Sony AL.
CCoresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

improve the adversarial robustness of the global model. These methods in general are known as Federated Adversarial Training (FAT), AT has been found to be more challenging than standard training [3, 44, 42, 41, 4], as it generally requires more training data and larger-capacity models. Moreover, adversarial robustness may even be odds with accuracy [30], meaning that the increase of robustness may inevitably decrease the natural accuracy (i.e., accuracy on natural test data). As result, the natural accuracy of AT is much lower than standard training [5]. This phenomenon also exists in FL, i.e. FAT exhibits slower convergence and lower natural accuracy than standard FL, as mentioned by recent studies [47, 9].
Arguably, FAT will become more challenging if the data are non- von-independent and identically dis- tributed (non-III) across the clients. One typical non-IID setting that commonly exists in real- world applications is skewed label distribution [17], where different clients have different label distributions. In this paper, we study the problem of FAT on non-IID data with a particular focus on the challenging skewed label distribution setting (formally defined in Section 3.1). Under conventional training. Xu et al. [37] have shown that adversarially trained models introduce severe performance disparity across different classes. And such disparity will be exacerbated under label skewness, ending up with much worse performance on the minority classes [33].
By far, only few works have studied non-IlD FAT in the current literature. Zizzo al. [47] propose to perform AT on only part of the lo- cal data for better convergence, while standard training is applied to the rest of the local data.
Natural accuracy (%)
60-
40-
We term this method as MixFAT. Another rele- vant work called FedRBN [9] tackles a different problem: how to propagate federated robustness to low-resource clients. Although MixFAT and FedRBN have demonstrated promising results,
P-OGAIRT FEBREN
20-
F'STRADDSS FOIMART
Suniled FL.
MGFFAT
C (OMR4)
25
50
75
100
125
150
Communication round
they suffer from training instability and low nat- ural accuracy issues when compared to standard FL, show in Figure 1. We also compare with the other four FAT baselines adapted from existing AT methods FL., i.e., FedPGD, Fed- TRADES, FedMART, and FedGAIRAT. Unfor- tunately, these methods also exhibit slow conver- gence and much degraded final accuracy (details be found in Section 4.1). This motivates to propose a novel method called Calibrated Federated Adversarial Training (CalFAT) for effective FAT on non-IID data with skewed label distribution. CalFAT tackles the training instability issue by calibrating the logits give higher scores to the minority classes.
Natural accuracy and convergence of standard FL, our CalFAT, and 6 FAT baselines (FedPGD, FedTRADES, FedMART, MixFAT [47], FedGAIRAT and FedRBN [9]) under skewed label distribution with 0.1 (see Section 4).
Figure 1:
In summary, main contributions are:
New insight: We study the problem of FAT on non-lID data with skewed label distribution, and reveal one root cause of the training instability and natural accuracy degradation: skewed labels lead non-identical class probabilities and heterogeneous local models.
Novel method: We propose novel method called CalFAT for FAT with label skewness, and show that the optimization of CalFAT can lead to homogeneous local models, and consequently, stable training, faster convergence, and better final performance.
High effectiveness: Extensive experiments on 4 benchmark vision datasets across various settings prove the effectiveness of our CalFAT and its superiority over existing FAT methods.
2
Notation and Preliminaries
2.1 Notation
Suppose there are m clients in FL with i denoting the i-th client, e.g., DD denotes the local data of client i and 0; denotes the parameters of its local model. We use 0 to denote the parameters of the global model. Subscript j is the sample index, e.g., (Faj. denotes the j-th sample of client and its corresponding label with yi (1....,CC ,C}. Let fo(-) ) be the local model f(-) (before softmax)

---

with parameter 6. Superscript / is the class index, e.g., (.) denotes the logit output for class I. We denote the adversarial example of clean sample x by I. [m] denotes the integer set {1.... .m). (a(ar, y) denotes the joint distribution of input z and label at client i, and accordingly, pi(y) is the marginal distribution of label y, Pi(y x) the conditional distribution of label y given input æ and (x y) is the conditional distribution of input a given label y.
2.2 Centralized Adversarial Training
Let D {zi Mj My be the training dataset with samples. The cross-entropy loss (eo(fo(z)) y) for an input-label pair (x, y) is defined as eee (fo((())))))
- oo (fe(z)), where (f)) the number of classes, and f² is the model
exp (ft)/ Σi= exp (f*) is the softmax function, output for class l. The objective function of the centralized adversarial training (AT) [22] can then be
defined as ming Σラ=1 (fo((f (Tz)), (.)8))/// 9;)/n, where the adversarial example T.y can generated by
I arg max (Jo (z) E,,,),z)
(I)
where (xyg) = (x' || - xا < } is the closed ball of radius > 0 centered at ry, ll-11.o the Loo norm, and žy is the most adversarial sample within the e-ball.
standard centralized AT method uses Projected Gradient Decent (PGD) to generate adversarial examples [22]. In particular, PGD iteratively generates an adversarial example Ty follows:
) (k+1) = II IT (x,50))
+ a vign(( (,u;))) 1 == 0, ... K 1,
(2)
where the step number, K is the total number of steps (i.e., 3y = 2j (k) ), a 0 is the step size, . (0) is the natural sample, x; is the adversarial example generated at step k. IT B. (x()))) the projection function that projects the adversarial data onto the e-ball centered at 2 and sign(.) is the sign function.
By optimizing the model parameters on adversarial examples generated by PGD, centralized AT is able to train model that is robust against adversarial attacks.
2.3 Federated Adversarial Training
The concept of federated adversarial training (FAT) first introduced in [47] (i.e., MixFAT) to deal with the adversarial vulnerability of FL. MixFAT applies AT locally to improve the robustness of the global model. Suppose there sampled from distribution pa (z, y) with n, [D;] being the size of the local data. In MixFAT, each client i optimizes its local model by minimizing the following objective:
clients and each client has its local data D; (maj, Big 3=1 j=1
min TE:
Σ lor (fo (Tt)).
Σ gmmf+
(3)
where ã.y is the PGD adversarial example of Tiy, n/ is a hyperparameter that controls the ratio of data for AT, and 0; are the local model parameters. After training the local model for certain epochs, client i uploads its local model parameters 0 to the central server for aggregation. Note that MixFAT only applies AT to proportion of the local data, mainly for convergence and stability considerations.
3
Calibrated Federated Adversarial Training (CalFAT)
3.1 Skewed Label Distribution Leads to Non-identical Class Probabilities
In this paper, we focus on one representative non-llI) setting: skewed label distribution [19, 10]. which is defined as follows.
Definition (Skewed label distribution). The label distribution across the clients is skewed, if for all i A u and i, E [m]]
(a) there exists y e [C] such that (v)) (v)) and (b) p.(x v) Pa(x (1) for all x, y.

---

Condition (b) is to assume that, given a class x is sampled with equal probability at different clients. Note that there exist different types of non-IID: label skew, non-identical class conditional, quantity skew, to name a few (Appendix K in [10]). The class conditional is often assumed to be identical (i.e., condition (b)) when studying the label skewness problem, which is the main focus of this work. When condition (b) does not hold, it becomes the non-identical class conditional problem.
Lemma Non-identical class probabilities). If the label distribution across the clients is skewed and the class conditionals have the same support, then the class probabilities {pi(y) x) | E [m]]
are non-identical, ie., for all i + u and i, u [m]] there exist a, y such that p:(y x) # (v x))
Lemma implies that skewed label distribution gives rise to non-identical class probabilities {p.(v) x) i [m]]. The proof of Lemma 1 is given in Appendix A.
3.2 Standard Cross-entropy Leads to Heterogeneity
From statistical point of view, each client in previous FAT methods estimates its local class probability pı(y x) during local training [7]. More specifically, they assume that (: x) can be
parameterized by 0g as:
(v æ)) = p(y z; 0;) = (fee (a)),
(4)
where 0. is the ground-truth parameters of the local class probability (ı | x). According to Lemma 1, the class probabilities {v: z)} are non-identical when there is a skewed label distribu- tion. Therefore, the ground truth parameters {Ør i [m]] are heterogeneous. We use the sample variance of the ground -ruth parameters to measure such heterogeneity as follows:
(s*)² ==))²=10 V ....... 0
Σ
m
10
1
Σ 071..
122
m
Each client updates its local model parameters 0, by optimizing the standard cross-entropy (CE)
loss. The updated 0; is the maximum likelihood estimate [I] of the ground-truie parameter 0; [7]. Similarly, we use the sample variance [1] of the local model parameters measure the heterogeneity of the local models:
2 Om.)
(6)
Larger sample variance implies higher model heterogeneity.
The following proposition suggests that the heterogeneity of local models originates from the heterogeneity of the local class probabilities.
Proposition 1 (Heterogeneous local models). Assume the label distribution across the clients is skewed. Let 0g be the maximum likelihood estimate 0f in Eq. (4) given local data at client i. Then 52 converges almost surely to a nonzero constant:
2
where a.hi, represents the almost sure convergence.
The proof of Proposition provided in Appendix B. (s*)² measures the heterogeneity of the ground-truth parameters {0; E [m]]. which reflects the class probability difference across the clients as shown in Eq. (4).
Proposition implies that the local models in previous FAT methods are heterogeneous when the label distribution across the clients is skewed. Since the local models are heterogeneous, aggregating these models tends to hurt the convergence and cause the divergence of the global model [17]. As shown in Figure 1, the training process of existing FAT methods is unstable and has much lower natural accuracy than the standard FI..
3.3 Learning Homogeneous Local Models by Calibration
Motivated by [24], propose to re- pe-armmmetrrize the class probabilities. According to Bayes' formula [12],
px(x v)p((u)
pily x))
(8)
ΣE,pi Pi( (p)(())

---

Algorithm Local training of CalFAT
Input:
Client i, global model parameters 0, local dataset D,, local epoch number E, and positive constant
procedure CLIENTUPDATE
2: 0g f
3: Compute n. with D, by = + d, [C]
4:
for local epoch=1, ..... E do
5: for 1, do
6:
Sample (Izz) Vig) from Dj
7:
Generate adversarial example ã. arg max (B (fo, (1/y)s So, (2it), m.)
8:
end for
9:
0, <- 0: m, Ej=1
m.)
10: end for
return
12: end procedure
On the right-hand side of the above equation: (1) the class priors be easily computed by the relative frequencies [l]; and (2) more importantly, the class conditionals {p.(x y) E [m]] are identical across different clients (see Definition 1).
Inspired by the above observation, we propose an alternative parameterization of ps y x). Assume
that for all E [ml) the class conditional (z y) can be parameterized by 0" as p: (z y) ((x)
y; 8, where q(x y; 8*)) can be an arbitrary conditional probability function. Then, p;(y x) can be re- re-parameterized by 6 as follows:
g(x y/: 8 )x!
pal æ) -q: x;; 0-)
(9)
E q(x 1: (=0))
where
[C]. Here n. approximates the class prior P. (y). n. is the sample size of class y on client and is small constant added for numerical stability purpose. During local updates, client i uses its local data update 6,, which makes 0; the maximum likelihood estimate of f*. The entire training procedure of our method is described in Section 3.4.
T. = = n
(10)
The following proposition suggests that the local models homogeneous when trained with the above re- ce-parameterization. The proof of Proposition is provided in Appendix C.
Proposition 2 (Homogeneous local models). Assume the label distribution across the clients is skewed. Let 0, be the maximum likelihood estimate of 0 in Eq. (9) given local data at client i. Then 52 converges almost surely to zero:
G.S. 0.
(11)
3.4 Details of CalFAT
The local training procedure our proposed CalFAT described in Algorithm Specifically, we define q(x| y; 0²) = exp (f%- (z)). Then, we maximize the likelihood of (:(y) x;; 6*) for each client i, which is equivalent to minimizing the following objective:
min O, Y&g
Loce (fo, (āsz): Mijs TT),
(12)
where (. is the calibrated cross-entropy (CCE) loss and T.j is the adversarial example of zij- The CCE loss is defined as:
loce (Tiz), Mijy M;) - log ui (Ja, (3iz) + log m:).
(13)
As discussed in Section 3.3, minimizing the above CCE loss mitigates the heterogeneity of the local models, which can Iead to improved convergence and performance of the global model.
5

---

Table 1: Natural and robust accuracy (%) different datasets. The best results are in bold.
CHFRRO
CIFARIDO
Meric
Natucal
FOSM
BCM
PGD-20
AA
Naxral
RGSM
BIM
CR
PGD-30
AA.
ModAA
53.35 0:11
70:15
36.11
22.24
76.77
21.00
S8.11 1-111
15:68 1-0.5
1410 4-1.1
11:11
Pearap
46.98
0.16
26.20
6.39
26.39
0.13
24:35
2.111
26.4 0.18
22-43 2 2.131
33.96 2.14
16.00
2.11
14.8
2..00
11.33
2:0.0
14.65 2:015
10.03 2:0.2
26311
2.10
14:11
2:011
FABMRRT
28.87 18.3
18.30 4.35
15.35
0:31
8.35
19 0-0
1510
6:1.1
5.0 0.31
1998
8:57
REGAIRAT
68.42 0.68
20.30 2 0000
26.35
2.00
22.78
2.331
27.23
0.88
21-06
0.00
56.92 000
10.18 2:000
13.17 2:0.0
14.0-200 2:0000000
941 653
CaIAAT (sz))
64,99 0.08
36.03
31.99
0.00
24,99
0.031
31,12
22
44.57
2.100
17.03 0.10
18:00
0:0.1
12.01 2.111
15.23 8:0.0
1L49:::010
Inngehec sudacc
Merio
Natvil
RIM
CW
PGDD-70
AA
Natral
ROSM
RIM
CW
PCD-70
A-A
10.55 0.58
10.33 0,00
10.31 =.0.6
10:3 0.005
10.32 0.55
13.54 0.10
30.53 +.1.2
18.88 2.155
12.55 0-1.0
10.07
1:40-00.16
11:34:0010
IMTRATIS
5648
11:33
11.8
0:11
18:00
1:0.1
16.33
1181 1.0.0
ReGAIRAT
55.41
55.30 2.13
56.52 0.09
21:4
36.9 0.13
51.53
54.25 =.112
19.2 0.10
10.33 2.1.
19.18 +.0.2
11:0020000
TERRIN
11:8
76.11 1:0
16.88 1.10
1810
1:a
1.11::13
CaIAT
In previous FAT methods, heterogeneous local models tend give higher scores to the majority classes while lower scores to the minority classes. By contrast, our CalFAT encourages local models to give higher scores the minority classes by adding class-wise prior log æ to the logits. Also different from MixFAT that trains the local models on both natural and adversarial data, our CalFAT trains the local models only adversarial examples. Extensive empirical experiments conducted in Section 4.1 to show the impact of using only adversarial data for optimization.
Adversarial example generation. Inspired by [40], generate the adversarial examples by maximizing the following calibrated Bellback-Leibler (CKL) divergence loss:
F.j arg max Leks (Flgg) (1aa)) m.),
(14)
2/ (B.)))
where (e)a()s:)))) is the CKL loss defined
(xl (zig),
Σ (f + log T.) log (fo, (11,) + +og
(15)
where log m; is the same as in our CCE loss. Following centralized AT [22], we also use PGD to solve Eq. (14).
After training the local model for certain epochs following the above procedure, each client uploads the model parameters 0, to the server for aggregation. To be consistent with the most recent FAT methods [47, 9]. we adopt the most widely used FedAvg [23] the default aggregation framework. Our method is compatible with other FL. frameworks (e.g., FedProx [16] and Scaffold [11]), as we will show in Section 4.1.
4 Experiments
Data configurations. Our experiments are conducted on real-world datasets: CIFARIO [13], CIFARION 100 [13], SVHN [25], and ImageNet subset [6]. To simulate label skewness, we sample pl Dir(3) and allocate p proportion of the data of label to client i, where Dir(3) is the Dirichlet distribution with a concentration parameter 3 [39]. By default, we set B = 0.1 to simulate a highly skewed label distribution that widely exists in reality.
Baselines. We compare our proposed CalFAT with two state-of-the-art FAT methods: MixFAT [47] and FedRBN [9]. We also investigate the combination of the state-of-the thett-rttt-tttt centralized AT methods with FL, ie.. we apply standard PGD [22], TRADES [40], MART [34]), and GAIRAT [44] to FL, and term them as FedPGD, FedTRADES, FedMART, and FedGAIRAT, respectively.
Evaluation metrics. We report the natural test accuracy (Natural) and robust test accuracy under the most representative attacks, i.e., FGSM [35], BIM [15], PGD-20 [22], CW [2), and AA [5]. We run the experiment for 5 times and report the mean and standard deviation. More detailed experimental setup is provided in Appendix D.I.

---

CIFARIO
CIFAR10
Robuat accunay (S0)
Naano scunccy (6
F=KAVIRAT (4BAATom
Robust accurscy (3)
CXEAT
50
16-
Communiction
Figure 2: Robust accuracy (against PGD-20 attack) of different meth- ods on CIFAR10 dataset.
Figure 3: Per-class natural accuracy and robust accuracy (against PGD-20 attack) of CalFAT and the best baseline (FedGAIRAT) CIFARIO dataset.
4.1 Main Results
Evaluation on different datasets. Table shows the results of all methods on CIFARIO, CI- FARI00, SVHN, and ImageNet subset. From the table,
observe that:
(1) Our CalFAT achieves the best robustness on all datasets, validating the efficacy of our CalFAT. For example, CalFAT outperforms the best baseline method (FedGAIRAT) by 10.20% on SVHN dataset under FGSM attack.
(2) Our CalFAT shows significant improvement in natural accuracy compared to other baselines. For example, CalFAT can improve the natural accuracy of the best baseline method (FedGAIRAT) by 25.63% on SVHN dataset. We hypothesise that the reason lies in the homogeneity of local models in CalFAT, which leads to better convergence and higher clean accuracy.
(3) All methods demonstrate the worst performance on CIFAR and ImageNet subset datasets. We conjecture that this is because there more classes in these two datasets, making federated training substantially harder. Nevertheless, our CalFAT still achieves the best performance.
Learning curves of different methods. To visually compare CalFAT with all the base- lines, we plot the learning curves (i.e., performance across different communication rounds) of all methods in Figure and Figure 2. As can be observed, CalFAT achieves the best natural accuracy and robust accuracy across almost the entire training process, which in- dicates that the design of our CalFAT is profitable for different federated learning stages. Moreover, our CalFAT is fairly stable during the whole training process while the accuracy curves other baselines oscillate strongly. Such oscillations lead to bad convergence and low performance. We hypothesize that the heterogeneity of local models in the baseline methods is the main cause of the unstable training.
Table 2: Combining FAT methods with different losses.
Metric
Natural
PGD-20
MixFAT
53.35 + 0.11
26.27 0.11
MixFAT LogitAdj
57.53 - 0.21
27.65
0.16
MixEAT RoBal
58.25 + 0.13
27.86
0.10
Calibration (ours)
60.23 0.19
28.67
0.14
FodPGD
46.96 0.16
0.18
FedPGD + LogitAdi
59.79 0.15
28.84
0.12
FedPGD RoBal
29.51 0.07
Combining calibration loss with other FAT methods. To further show the ef- fectiveness of our calibration loss, we combine it with four FAT methods (Mix- FAT. FedPGD, FedTRADES, and Fed- MART) and name them MixFAT + Cal- ibration. FedPGD + Calibration, Fed- TRADES Calibration, and FedMART + Calibration, respectively. We compare these calibration loss-based FAT meth- ods with their original versions in Ta- ble evident that, by introducing our calibration loss into their objectives, all FAT methods can be improved. These results confirm the importance of class calibration for FAT with label skewness.
FedPGD + Calibration (ours)
63.91 0.13
30.72
0.16
FodTRADES
26.31 +: 0.12
FedTRADES LogitAdj
58.26
27.92 0.19
FedTRADES RoBal
59.25 0.23
28.63 0.08
FedTRADES + Calibration (ours)
63.12 0.10
30,27 0.23
FedMART
25.67
0.22
FedMART LogitAdj
42.01 + 0.10
24.92 + 0.02
FedMART RoBal
44,26 + 0.22
25.57 0.17
FedMART Calibration (ours)
48.85 0.08
27.19 0.11
CalFAT (ours)
64.69 + 0.08
31.12 0.11
7

---

Table 3: Natural and robust accuracy (%) across different FL frameworks on CIFARIO dataset.
FL framework
FedProx
Scaffold
Metric
Natural
PGD-20
AA
Natural
PGD-20
AA
MixFAT
53.75 0.16
29.61 =: 0.19
21.59 + 0.27
55.27
0.20
28.78
0.15
21.26 1 0.11
FedPGD
49.57 0.18
28.48 = 0.17
21.31
0.18
49.52 0.14
27.46 1: 0.21
20.27 0.15
FedTRADES
27.75 0.20
21.13
47.78
27.31
20.04
FedMART
28.32
19.32 0.23
15.91
27.80
20.03
16.85
FedGAIRAT
49.61 0.20
2.134
2
0.21
27.23
20.16
FedRBN
47.26 0.13
26.63
20.46
49.77 0.09
28.37
20.32
CalFAT
66.32 + 0.08
0.13
22.83+0.11
67.16 + 0.06
32.94 0.06
21.94 + 0.05
Comparison with state-of-the-ard long-tail learning methods. We also compare our calibration loss with the losses used by long-tail learning methods LogitAdj [24] and RoBal [36]. particular, we combine four FAT methods (MixFAT, FedPGID, FedTRADES and FedMART) with the above three losses (LogitAdj loss, RoBal loss, and our calibration loss), and train the models following the same default setting. As shown in Table 2, both LogitAdj--ba Adj-baed based methods and RoBal- based methods have lower natural and robust accuracies than calibration loss-based methods. This indicates that our calibration loss is more suitable for FAT than other long-tail learning losses.
4.2 Performance on Different Classes
We further compare the per-class performance of our CalFAT with the best baseline FedGAIRAT. First, we use well-trained model to initialize global model. Second, the global model distributes the model parameter to all clients. Third, the local clients train their local models with their local data for 1 epoch. Then, we report the per-class average performance of all clients for each class. For fair comparison, we use the same well-trained model for initialization and the same data partition on each client for CalFAT and FedGAIRAT.
In Figure 3, we report the per-class natural and robust accuracies of CalFAT and FedGAIRAT on CIFAR10. As shown in these figures, the average performance of most classes of CalFAT is much higher than FedGAIRAT. We also report the per-class performance of each client on CIFAR1O in Appendix D.2. In FedGAIRAT, due the highly skewed label distribution, the prediction of each client is highly biased to the majority classes, which leads to high performance on the majority classes and low performance (even 0% accuracy) client has higher performance on most classes. This verifies that the calibrated cross-entropy loss can indeed improve the performance the minority classes, and further improve the overall performance of the model. Moreover, we report the per-class average performance on SVHN in Appendix D.3. Our CalFAT also outperforms the best baseline across most of the classes SVHN.
the minority classes. By contrast, in CalFAT, each
4.3 Results Different FL Frameworks and Network Architectures
Evaluation on different FL frameworks. Besides FedAvg [23], we also conduct experiments on other FL frameworks, i.e., FedProx [16] and Scaffold [11]. The results for all methods on FedProx and Scaffold are given in Table 3. It shows that our CalFAT exhibits better natural and robust accuracies than all baseline methods all FL. frameworks, which indicates the high comparability of our CalFAT with different aggregation algorithms.
Evaluation on different network architectures. We also compare CalFAT with baselines on different network architectures. i.c., CNN [23]. VGG-8 [27], and GesNet-188 [8]. For CNN, we use the same architecture as [23]. VGG-8 and ResNet-18 18 are two widely used architectures in deep learning. The results on CIFAR10 10 dataset are shown in Appendix D.4. CalFAT outperforms all baselines, which further validates the superiority of CalFAT with different network architectures.
4.4 Feature Visualization
To better understand the efficacy of CalFAT, we visualize the learned features extracted from the second last layer of FedTRADES (the best baseline) and CalFAT trained on SVHN dataset in Appendix D.5. The features projected into
a 2-dimensional space via t-SNE [31]. It shows that

---

samples from different classes are mixed together in FedTRADES, indicating its low performance. For instance, Class 6 (pink) and Class (khaki) are hard to separate FedTRADES while these classes can be well separated by CalFAT This illustration verifies that the server cannot learn a global model with good inter-class separability if the local models are heterogeneous, By contrast, CalFAT can well separate different classes thus can achieve better overall performance.
4.5 Results under IID settings
Besides the non-IlD setting, we also conduct an experiment under the IID setting. The results shown in Appendix D.6 where it shows that our CalFAT achieves the best robustness (under PGD-20 attack). Compared to the non-IIID setting, all FAT methods demonstrate much better performance under the IID setting. This indicates that existing FAT methods can easily handle IID data yet face substantial challenges when the data is non-IID.
4.6 Ablation Studies
Impact of the number of clients. To show the generality of CalFAT, we train CalFAT with different numbers of clients m. Table 7 in Appendix D.7 reports the results for m {20, 50, 100). As expected. CalFAT achieves the best performance across all m. As m increases, the performance of all methods decreases. We conjecture that this is because more clients in FAT makes the training harder to converge. However, our CalFAT can still achieve 41 41.23% natural accuracy when there are 100 clients, outperforming other baselines by a large margin.
Impact skewed label distribution. We observe that the performance FAT defense closely related to label skewness. We thus investigate the impact of skewed label distribution by varying the Dirichlet parameter = {0.05, 0.2, 0.3) and report the results on CIFAR1O in Table 8 in Appendix D.8. Not surprisingly, our CalFAT outperforms all baselines under all ('s. This further verifies the consistent effectiveness of CalFAT under different levels of label skewness.
Note that as 3 decreases (i.e., the labels each client are more imbalanced), the performance of all methods drop rapidly. For example, the natural accuracy of FedMART drops from 38.38% to 39.84% as f decreases from 0.2 to 0.05. This indicates that all methods are hard to train good model in extremely skewed label distribution scenarios. However, our CalFAT still achieves 61.00% natural accuracy and 32.40% robust accuracy (against FGSM attack) when 8 = 0.05, which are much higher than all the baselines.
Contribution of the calibrated loss functions. As shown in Eq. (13) and Eq. (15), for each client i, we have two new loss functions: CCE loss (t...) for optimization and a CKL loss (. for generating the adversarial examples. This naturally raises question: how do these two loss functions contribute to CalFAT? To answer this question, we conduct leave-one-out tests by removing the CCE loss (w/o (. -) removing the CKL loss (w/o l: .)) from the overall optimization objective. As illustrated in Appendix D.9, w/o le leads to poor performance, which implies that CCE loss plays an important role in CalFAT. Besides, if we only use the CCE loss (i.e., w/o (. we can obtain a much better performance, but it still underperforms CalFAT. All these results indicate that the CCE loss is the most important part of CalFAT, whilst the CKL loss can further increase the performance of CalFAT, The combination of both loss functions leads to the best performance.
Impact of the ratio of adversarial data. Here, we conduct experiments with different ratios of adversarial data used in CalFAT and report the robust accuracy (against PGD-20 attack) in Appendix D. Ratios r=0 and r=1 stand for training the model on only natural data and only adversarial data, respectively. Overall, r=1 produces the best robustness, meaning that training only adversarial data can better enhance the adversarial robustness of our CalFAT.
Conclusion
In this paper, we studied the challenging problem of Federated Adversarial Training (FAT) with label skewness and proposed a novel Calibrated Federated Adversarial Training (CalFAT) simultaneously achieve stable training. better convergence, and natural accuracy and robustness in FL. CalFAT

---

calibrates the model prediction and trains homogeneous local models across different clients by automatically assigning higher scores to the minority classes. Extensive experiments on multiple datasets under various settings validate the effectiveness of CalFAT. Our work can serve simple but strong baseline for accurate and robust FAT. For future work, we will continue to improve FAT under other non-Ill) settings such as feature skewness and quantity skewness [10, 46].
Acknowledgement
This work is funded by Sony AL This work is also supported by the National Key R&D Program of China (Grant No. 2021ZD0112804) and the National Natural Science Foundation of China (Grant No. 62276067).
References
[1] Peter Bickel and Kjell Doksum. Mathematical statistics: basic ideas and selected topics, volumes I-1l package. CRC Press, 2015.
[2] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium security and privacy (sp), pages 39-57. IEEE, 2017.
[3] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C Duchi. Unlabeled data improves adversarial robustness. In Proceedings of the 33rd International Conference Neural Information Processing Systems, pages 192-112203, 2019.
[4] Chen Chen, Jingfeng Zhang, Xilie Xu, Lingjuan Lyu, Chaochao Chen, Tianlei Hu, and Gang Chen. Decision boundary-awar data augmentation for adversarial training. IEEE Transactions on Dependable and Secure Computing. 2022.
[5] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference machine learning, pages 2206-2216, 2020.
[6] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-254-2 22-,255,22 2009.
[7] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. International Conference on Machine Learning, pages 1321-1330. PMLR, 2017.
[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[9] Junyuan Hong, Haotao Wang, Zhangyang Wang, and Jiayu Zhou. Federated robustmess propa- gation: Sharing adversarial robustness in federated learning. arXiv preprint arXiv:2106.10196. 2021.
[10] Kevin Hsieh, Amar Phanishayee, Onur Mutlu, and Phillip Gibbons. The non-iid data quagmire of decentralized machine learning. In International Conference on Machine Learning, pages 4387-4398. PMLR, 2020.
[11] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132-5143, 2020.
[12] Achim Klenke. Probability theory: comprehensive course. Springer Science Business Media, 2013.
[13] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. In Technical report, 2009.
10

---

[14] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097- 11 1105, 2012.
[15] Alexey Kurakin, lan Goodfellow, and Samy Bengio. Adversarial machine learning at scale. arXiv preprint arXiv: 2016.
[16] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. arXiv preprint 1721v:1812.06127, 2018.
[17] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429-450, 2020.
[18] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. Advances in Neural Information Processing Systems, 34, 2021.
[19] Jiahuan Luo, Xueyang Wu, Yun Luo, Anbu Huang, Yunfeng Huang, Yang Liu, and Qiang Yang. Real-world image datasets for federated learning. arXiv preprint arXiv: 1089, 2019.
[20] Lingjuan Lyu. Han Yu, Xingjun Ma, Chen Chen, Lichao Sun, Jun Zhao, Qiang Yang. and Philip S Yu. Privacy and robustness in federated learning: Attacks and defenses. arXiv preprint arXiv:2012.06337. 2020.
[21] Lingjuan Lyu, Han Yu, Jun Zhao, and Qiang Yang. Threats to federated learning. Federated Learning, pages 3-16. Springer, 2020.
[22] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.00.m 06083, 2017.
[23] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efiin efficcien learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273-1282. PMLR, 2017.
[24] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.07314. 2020.
[25] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
[26] Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In ICML, 2020.
[27] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint /409. 2014.
[28] Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prototype learning across heterogeneous clients. In AAAI Conference on Artificial Intelligence, volume 36, pages 8432-8440, 2022.
[29] Yue Tan, Guodong Long, Jie Ma, Lu Liu, Tianyi Zhou, and Jing Jiang. Federated learning from pre-trained models: A contrastive learning approach. In First Workshop Pre-training: Perspectives, Pirfalls. and Paths Forward at ICML 2022.
[30] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representations, 2018
[31] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.

---

[32] Abraham Wald. Note on the consistency of the maximum likelihood estimate. The Anmals of Mathematical Statistics, 20(4):595 601, 1949.
[33] Wentao Wang, Han Xu, Xiaorui Liu, Yaxin Li, Bhavani Thuraisingham, and Jiliang Tang. Imbalanced adversarial training with reweighting, arXiv preprint arXiv:2107.13639 2021.
[34] Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustmess requires revisiting misclassified examples. In ICLR, 2020.
[35] Eric Wong, Leslic Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In ICLR, 2020.
[36] Tong Wu, Ziwei Liu, Qingqiu Huang, Wang, and Dahua Lin. Adversarial robustness under long-tailed distribution. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8659-8668, 2021.
[37] Han Xu, Xiaorui Liu, Yaxin Li, Anil Jain, and Jiliang Tang. To be robust or to be fair: Towards fairness in adversarial training. In International Conference Machine Learning, pages 11492-11501, 2021.
[38] Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-19, 2019.
[39] Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In International Conference on Machine Learning, pages 7252-7261, 2019.
[40] Hongyang Zhang, Yaodong Yu, Jiantao Jíao, Eric Xing. Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, pages 7472-7482. PMLR, 2019.
[41] Jie Zhang, Chen Chen, Jiahua Dong, Ruoxi Jia, and Lingjuan Lyu. Qekd: Query-efficient and data-free knowledge distillation from black-box models. arXiv preprint arXiv:2205. 1158, 2022.
[42] Jie Zhang, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, Lei Zhang, and Chao Wu. Towards efficient data free black-box adversarial attack. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15115-15125, 2022.
[43] Jie Zhang, Zhiqi Li, Bo Li, Jianghe Xu, Shuang Wu, Shouhong Ding, and Chao Wu. Federated learning with label distribution skew via logits calibration. In International Conference Machine Learning, pages 26311-2633 2022.
[44] Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. Geometry-aware instance-reweighted adversarial training. In ICLR, 2021.
[45] Yao Zhou, Jun Wu, and Jingrui He. Adversarially robust federated learning for neural networks, 2021.
[46] Hangyu Zhu, Jinjin Xu, Shiqing Liu, and Yaochu Jin. Federated learning on non-iid data: A survey. Neurocomputing, 465:371-390, 2021.
[47] Giulio Zizzo, Ambrish Rawat, Mathieu Sinn, and Beat Buesser. Fat: Federated adversarial training. arXiv preprint (xXiv:2012.0179))
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
12

---

(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals. applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "n-z2bWUE19I", "reviewer": "Reviewer_RW5e", "summary": "This paper proposes a calibrated federated adversarial training (CalFAT) method, which is used in the case of label skewness. Empirical evaluation is done on several datasets including SVHN, CIFAR-10/100 and a subset of ImageNet.", "questions": "Limited technical contribution and weak baselines. ", "limitations": "No", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The empirical evaluation is done on several different datasets, and under strong attacks like AA.\n- The improvement in clean accuracy seems significant.\n- The writing is clear and easy to follow the main idea.\n\nWeaknesses:\n- The technical contribution is quite limited. CalFAT is just a simple adaption of [22] into federated adversarial training.\n- The compared baselines (e.g., MixFAT and Fed**) are off-the-shelf, which do not adapt to the special property of label skewness. Stronger baselines utilizing label skewness should be involved.\n- The considered setting of federated learning with label skewness may be of limited interest to the community.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "c4OK30GjTr2", "reviewer": "Reviewer_sKZG", "summary": "The paper discusses the cause of instability of federated adversarial learning(FAT) is the skewed label distribution. An adapted calibrated loss is applied to FAT to reduce the heterogeneity of local models and further improve the final model’s natural and robust accuracy. Experimental results of four computer vision datasets are conducted to support the author’s arguments.", "questions": "1 Line 14-16, I read that the authors claim they theoretically show a much improved convergence rate, expecting a theoretical proof on convergence rate. Prop 1 and Prop 2 in a combined manner show a better finall convergence point, as opposed to a faster convergence rate. I think these are two different things.  \n\n\n2 Line 259-260 says CKL loss can better generate adversarial examples. I suppose LogitAdj and RoBal do not associate with adversarial examples. Then my questions are:\n    2.1) Do previous FAT baseline methods use the same adversarial training based on the same  \n           set(eg, same class ratio, sample size, etc) of adversarial examples?\n    2.2) What’s the setup to apply LogitAdj/RoBal comparing with CalFAT? \n    2.3) Is it possible to combine LogitAdj/RoBal with previous FAT baselines to compete with \n           CalFAT?  \n\n\n3 Some experimental results are hard to tell the significance. For example, most numbers of AA column in Table 4 and Table 5 are very close. If the authors have conducted dfferent trials, standard deviation should be put there. \n", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n1 The authors provide extensive experimental results. \n2 The paper is well-written and easy to follow.\n\nWeaknees\n\nMy biggest concern is the experiment design does not align well with the arguments.\n\nThe authors argue that the training instability comes from the skewed label distribution that leads to heterogeneity of the local model. I think it’d be true for both standard federated learning and federated adversarial training(FAT).  Eq.(11) is then proposed to calibrate the heterogeneity of local models for FAT. Therefore I expect at least two sets of experiments which I don’t see many of them in the main paper.\n\na) FAT baselines vs FAT baselines + Eq (11)\nFor example in Table 1, showing the results of {MixFAT, FedPGD, FedTRADES…} + Eq(11) would be more helpful to convince reviewer about the effectiveness of the proposed loss. \n\nb) FAT baselines + other label distribution calibration vs FAT baselines + Eq(11) \nTable 3 is a good example. However, the authors should explore it more. See Question 2 below.\n\n\n\n\nMinor: \nThe structure of the paper can be further polished. For example, the table 1 and table 2 are actually the same experiments, and should be combined into one. \n\nLine 241-242 is not necessary. I don’t see it helping validate any arguments made by authors.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "28fbP8GRvF6", "reviewer": "Reviewer_tX9B", "summary": "This paper aims to answer the question of how to guarantee adversarial robustness in federated learning. In particular, this paper targets a challenging non-IID setting - the skewed label distribution, which gives rise to imbalanced class probabilities and heterogeneous local models, and proposed CalFAT to solve this problem by adaptively adjusting the logits. Theoretically, the authors prove that CalFAT can help learn homogeneous local models. Empirically, CalFAT outperforms other traditional centralized adversarial training methods (like AT, TRADES, MART, GAIRAT) and also the recent robustness propagation method (I.e., FedRBN) on several benchmarked datasets (like CIFAR-10, SVHN, CIFAR100).", "questions": "- Why the natural performance degradation is considered as the main motivation (Figure 1 on Page 2) instead of robust performance since the goal of FAT is gaining robustness?\n- I am wondering whether CalFAT also works in iid setting, can you elaborate more on this?\n\n", "limitations": "The authors did not discuss the limitations of the work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The literature review is solid; summarizing the advantages and disadvantages of the literature are very helpful.\n+ The paper points out why the previous methods hurt convergence of FAT and proposes novel techniques to tackle this issue that leads to better convergence and higher performance. \n+ The paper provides a solid theoretical analysis. \n+ Experimental evaluations are solid and well designed. The comparisons include the most recent FAT and centralized AT methods. \n+ The results are convincing and show that CalFAT outperforms state-of-the-arts.\n\nWeaknesses:\n- It is not clear why the natural performance degradation is considered as the main motivation (figure 1 on Page 2) instead of robust performance since the goal of FAT is gaining robustness. Adding more elaboration would be helpful.\n- More discussions on the potential applications of FAT are expected to better highlight its significance.\n- Even though sufficient references are given, a few relevant papers are recommended: “Privacy and Robustness in Federated Learning: Attacks and Defenses” that also touched on federated adversarial robustness; “Decision Boundary-aware Data Augmentation for Adversarial Training” that studied how to improve adversarial robustness by playing with decision boundary. These papers did not study exactly the same topic as this paper, but would certainly enrich the literature review further.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept