Improving Adversarial Robust Fairness via Anti-Bias Soft Label  Distillation
Authors: Shiji Zhao
Ranjie Duan
xizhewang
Xingxing Wei
Venue: NeurIPS 2024 poster (2024)
Keywords: Adversarial Robustness
Robust Fairness
Knowledge Distillation
Primary area: machine_vision

Abstract: Adversarial Training (AT) has been widely proved to be an effective method to improve the adversarial robustness against adversarial examples for Deep Neural Networks (DNNs). As a variant of AT, Adversarial Robustness Distillation (ARD) has demonstrated its superior performance in improving the robustness of small student models with the guidance of large teacher models. However, both AT and ARD encounter the robust fairness problem: these models exhibit strong robustness when facing part of classes (easy class), but weak robustness when facing others (hard class). In this paper, we give an in-depth analysis of the potential factors and argue that the smoothness degree of samples' soft labels for different classes (i.e., hard class or easy class) will affect the robust fairness of DNNs from both empirical observation and theoretical analysis. Based on the above finding, we propose an Anti-Bias Soft Label Distillation (ABSLD) method to mitigate the adversarial robust fairness problem within the framework of Knowledge Distillation (KD). Specifically, ABSLD adaptively reduces the student's error risk gap between different classes to achieve fairness by adjusting the class-wise smoothness degree of samples' soft labels during the training process, and the smoothness degree of soft labels is controlled by assigning different temperatures in KD to different classes. Extensive experiments demonstrate that ABSLD outperforms state-of-the-art AT, ARD, and robust fairness methods in the comprehensive metric (Normalized Standard Deviation) of robustness and fairness.

Improving Adversarial Robust Fairness via Anti-Bias Soft Label Distillation
Shiji Zhao', Ranjie Duan', Xizhe Wang', Xingxing Wei Institute of Artificial Intelligence, Beihang University, Beijing, China
SSecrityy Department, Alibaba Group, Hangzhou, China (zhaoshiji123, saabsssiji122xxizhevvar xizhewang, xxwei] @buaa. edu. en, ranjieduanOgmail.com
Abstract
Adversarial Training (AT) has been widely proved to be effective method to improve the adversarial robustness against adversarial examples for Deep Neural Networks (DNNs). As variant of AT, Adversarial Robustness Distillation (ARD) has demonstrated its superior performance in improving the robustness small student models with the guidance of large teacher models. However, both AT and ARD encounter the robust fairness problem: these models exhibit strong robustness when facing part of classes (easy class), but weak robustness when facing others (hard class). this paper, we give an in-depth analysis of the potential factors and argue that the smoothness degree of samples' soft labels for different classes (i.e., hard class or easy class) will affect the robust fairness of DNNs from both empirical observation and theoretical analysis. Based the above finding, we propose an Anti-Bias Soft Label Distillation (ABSLD) method to mitigate the adversarial robust fairness problem within the framework of Knowledge Distillation (KD). Specifically, ABSLD adaptively reduces the student's 's error risk gap between different classes achieve fairness by adjusting the class-wise smoothness degree of samples' soft labels during the training process, and the smoothness degree of soft labels is controlled by assigning different temperatures KD different classes. Extensive experiments demonstrate that ABSLD outperforms state-of-tha art AT, ARD, and robust fairness methods in the comprehensive metric (Normalized Standard Deviation) of robustness and faimess.
Introduction
Deep neural networks (DNNs) have achieved great success in various tasks, e.g., classification
10
detection [6], and segmentation
(24). However, DNNs are vulnerable to adversarial attacks where adding small perturbations to the input examples will lead to misclassification.
30
35
33
34
To enhance the robustness of DNNs, Adversarial Training (AT) (20) 41 32) [4] is proposed and has
been proven to be effective method defend against adversarial examples. To further improve the robustness, Adversarial Robustness Distillation (ARD) as a variant of AT proposed and aims to transfer the robustness of the large models into the small models based Knowledge Distillation (KD), and further researches
12 show the excellent performance of ARD.
Although AT and ARD can remarkably improve the adversarial robustness, some researches 2 31
39 [9. 28, 38 demonstrate the robust fairness problem: these models perform strong robustness on part of classes (easy class) but show high vulnerability others (hard class). This phenomenon will raise further attention to class-wise security. Specifically, an overall robust model appears to be relatively safe for model users, however, the robust model with poor robust faimess will lead to attackers targeting vulnerable classes of the model, which leads to significant security risks to
"Coresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Doo0
D0o0 re-texaperate
Babel
0000
prediction
fair prediction
sample
miseright
Do00
sample
D000
Waard
hardd class
hard class 1oo00
hard
samplet
sample
model
coss class
easy cless
eaSY
Do.0
DoDO
re-lamaeroie
0o00
label
(a) re-weight
re-temperate
The comparison between the sample-based fair adversarial training and our babel-based fair adversarial training. For the former ideology in (a), the trained model's bias avoided by re-weighting the sample's importance according to the different contribution to fairness. For the latter ideology in (b), the trained model's bias is avoided by re-temperating the smoothness degree of soft labels for different classes.
Figure 1:
potential applications. Different from simply improving the overall robustness, some methods are proposed to address the robust fairness problem in AT and ARD 33 36 38
19 28 (i.e., improving
the worst-class robustness much possible without sacrificing too much overall robustness). However, the robust fairness problem still exists and requires further to be explored.
For that, we give an in-depth analysis of the potential factors to influence robust fairness in the optimization objective function. From the perspective of the training sample, the sample itself has a certain degree of biased behavior, which is mainly reflected in the different learning difficulties and various vulnerabilities to adversarial attacks. For this reason, previous works apply the re- weighting ideology to achieve robust faimess for different types of classes in the optimization process 33,/36 38. However, as another important factor in the optimization objective function, the role of the labels applied to guide the model's training is ignored. Labels can be divided into two types, including one-hot labels and soft labels, where the soft labels are widely studied (29] []] and have been proven effective in improving the performance of DNNs. Inspired by this, we try to explore robust fairness from the perspective of samples' soft labels. Interestingly, we first find that the smoothness degree of soft labels for different classes (i.e., hard and easy class) can affect the robust fairness of DNNs from both empirical observation and theoretical analysis. Intuitively speaking, sharper soft labels mean larger supervision intensity, while smoother soft labels mean smaller supervision intensity, so it is helpful to improve robust fairness by assigning sharp soft labels for hard classes and smooth soft labels for easy classes.
Based on the above finding, we further propose an Anti-Bias Soft Label Distillation (ABSLD) method to mitigate the adversarial robust fairness problem within the framework of knowledge distillation. ABSLD can adaptively adjust the smoothness degree of soft labels by re-temperating the teacher's soft labels for different classes, and each class has its own teacher's temperatures based on the student's 's error risk. For instance, when the student performs more error risk in some classes, ABSLD will compute sharp soft labels by assigning lower temperatures, and the student's 's learning intensity for these classes will relatively increase compared with other classes. After the optimization, the student's robust error risk gap between different classes will be reduced. The code be found in httpps://// t/gbbhb............ thub. com/ http//haoshijiia hwwpsshhjjiii////eeee /////D ABSLD
Our contribution can be summarized as follows:
We explore the labels' effects on the adversarial robust fairness of DNNs, which is different from the existing sample perspective. To the best of our knowledge, are the first one to find that the smoothness degree of samples' soft labels for different types of classes can affect the robust fairness from both empirical observation and theoretical analysis.
We propose the Anti-Bias Soft Label Distillation (ABSLD) to enhance the adversarial robust fairness within the framework of knowledge distillation. Specifically, we re-temperate the teacher's soft labels to adjust the class wise smoothness degree and further reduce the student's is error risk gap between different classes in the training process.

---

We empirically verify the effectiveness of ABSLD. Extensive experiments on different datasets and models demonstrate that our ABSLD can outperform ofate-of-the-ar methods against variety of attacks in the comprehensive metric (Normalized Standard Deviation) of robustness and fairness.
Related Work
2.1
Adversarial Training
To defend against the adversarial examples, Adversarial Training (AT) [1 32 |3 is regarded effective method to obtain robust models. AT can be formulated min-max optimization problem as follows:
min (a [max C(f(2 6;0). BEl2
(I)
6
where f(;;0) represents deep neural network with weight 0, D represents data distribution with clean example x and the ground truth label y. represents the optimization loss function, e.g. the cross-entropy loss. 5 represents the adversarial perturbation, and Ω represents a bound, which can be defined as Ω = {6 || I }} with the maximum perturbation scale e. To further improve the performance, some variant methods of AT appear including regularization [21] 41 (32, using additional data [2722]], 22]. and optimizing iteration process [T5] 233. Different from the above methods for improving the overall robustness, in this paper, we focus on solving the robust fairness problem.
2.2 Adversarial Robustness Distillation
Knowledge Distillation [ as training method can effectively transfer the large model's knowledge into the small model's knowledge, which has been widely applied in different areas. To enhance the adversarial robustness of small DNNs, Goldblum al. 7 first propose Adversarial Robustness Distillation (ARD) by applying the clean prediction distribution of strong robust teacher models to guide the adversarial training of student models. Zhu al. 44 argue that the prediction of the teacher model not so reliable, and composite with unreliable teacher guidance and student introspection during the training process. RSLAD 45) applies the teacher clean prediction distribution as the guidance to train both clean examples and adversarial examples. MTARD 43 applies clean teacher and adversarial teacher enhance both accuracy and robustness, respectively. AdaAD [12 adaptively searches for optimal match points by directly applying the teacher adversarial prediction distribution in the inner maximization. In this paper, explore how to enhance robust fairness within the framework of knowledge distillation.
2.3 Adversarial Robust Fairness
Some researchers address the robust fairness problem from different views 39 18 36 38 19 37 and improve the fairness without losing too much robustness. The most intuitive idea is give different weights to the sample of different classes in the optimization process, and Xu et al. [39 propose Fair Robust Learning (FRL), which adjusts the loss weight and the adversarial margin based on the prediction accuracy of different classes. Ma et al. [99] finds the trade-off exists between robustness and fairness and propose Fairly Adversarial Training mitigate this phenomenon by adding a regularization loss to control the variance of class-wise adversarial error risk. Sun et al. 28 propose Balance Adversarial Training (BAT) achieve both source-class fairness (different difficulties in generating adversarial examples from each class) and target-class fairness (disparate target class tendencies when generating adversarial examples). Wu et al. [37] argue that the maximum entropy regularization for the model's prediction distribution can help to achieve robust fairness. Wei al. [36] propose Class-wiss Calibrated Fair Adversarial Training (CFA) to address fairness by dynamically customizing adversarial configurations for different classes and modifying the weight averaging operation. To enhance the ARD robust fairness, Yue et al. (38 propose Fair-ARD by re-weighting different classes based on the vuinerable degree. Different from these sample-based fairness methods, we try to solve this problem from the perspective of samples' labels, by adjusting the class-wiss smoothness degree of samples' soft labels in the optimization process.
28
3

---

-B
Same Smoothress Degr (650)
Same Smmothness Deg D))
70
Difte wet Smorthews Degrew(231D)
70
fhrrent Smoothhes Degr (35D)
60
80
50-
50-
Bobust
60
80
30-
Araa of 55I5:52.11 17%
30
Average of S5D-52.26%
Au Weaaae of DSD:52.3699 36%
Average of 550.52.54%
20
6
10
20
3
6
8
10
Class
Class
(a) Class-wise and average Robustness(b) Class wise and average Robustness of ResNet-118 on CIFAR-I0.
of MobileNet-v2 on CIFAR-10.
Figure 2: The class-wise and average robustness of DNNs guided by soft labels with the same smoothness degree (SSD) and different smoothness degree (DSD) for different classes, respectively. For the soft labels with different smoothness degrees, we use sharper soft labels for hard classes and use smoother soft labels for easy classes. We select two DNNs (ResNet-18 and MobileNet-v2) trained by SAT [20] on CIFAR-10 10. The robust accuracy is evaluated based on PGD. The checkpoint is selected based on the best checkpoint of the highest mean value of all-class average robustness and the worst class robustness following (36). We see that blue lines and red lines have similar average robustness, but the worst robustness blue lines are remarkably improved compared with red lines.
3 Robust Fairness via Smoothness Degree of Soft Labels
As an important part of the model optimization, label information used to guide the model plays an important role. The label can be divided into one-hot labels and soft labels, where one-hot labels only contain one class's 's information and soft labels can be considered as an effective way to alleviate
over-fitting and improve the performance [29]. Previous methods usually ignore the class-wise
smoothness degree of soft labels, either applying the same smoothness degree (29), or uniformly
changing the smoothness degree of soft labels for all the classes without deliberate adjustments
11. Different from previous methods, we are curious about if we adjust the class-wise smoothness degree of soft labels, will it influence the class-wise robust fairness of the trained model? Intuitively speaking, different smoothness degree of soft labels denote different supervision intensity, which means that it is possible achieve fairness by adjusting the smoothness degree of soft labels. Here we to explore the relationship between class wise smoothness degree of soft labels and the robust fairness from both empirical observation and theoretical analysis.
3.1 Empirical Observation
Here, we focus on the impact of the class-wise smoothness degree of soft labels on adversarial training. First, we train the model with soft labels that have the same smoothness degree for all types of classes (smoothing coefficien is 0.2). Then we assign different smoothness degrees of soft labels for hard classes and easy classes: specifically, we manually use sharper soft labels for hard classes (smoothing coefficient is 0.05) and smoother soft labels for easy classes (smoothing coefficient is 0.35). We conduct the experiment based on the SAT 20 shown in Figure 2
The result shows that the class-wist smoothness degree of soft labels has an impact on class-wo -iiee robust fairness. When we apply the sharper smoothness degree of soft labels for hard classes and the smoother smoothness degree of soft labels for easy classes, the class-wise robust fairness problem can be alleviated. More specifically, for the two worst classes (class 4, 5), the robust accuracy of ResNet- 18 guided by the soft label distribution with the same smoothness degree is 24.2%, and 30.9%, and the robust accuracy of ResNet-18 guided by the soft label distribution with different smoothness degree is 30.2%, and 39.1% %, which exists an obvious improvement for the class-wise robust fairness, and the average robust accuracy has slight improvement (52.12% 52.36%). Similar performance can also be observed in MobileNet-v2. This phenomenon indicates that appropriately assigning class-wied Mlasswwise smoothness degrees of soft labels can be beneficial to achieve robust fairness.
For N claass one hot ground truth labels, after processing by the smoothing coeflicient 7, the highest probubility (currect cluss) decreuses from to 7, and other probability (wrong class) incresses from T

---

3.2 Theoretical Analysis
Here we try to theoretically analyze the impact of the smoothness degree of soft labels on class-wise fairnes.. Firstly, we want to analyze the model bias performance with the guidance of the soft label distribution with the same smoothness degree. Then we give Corollary by extending the prediction distribution of binary linear classifier into the prediction distribution of DNNs based on the theoretical analysis in [39 and [9
Corollary 1. A dataset (z, v) ~ D contains 2 classes (hard class c+ and easy class c-). Based on the label distribution y. the soft label distribution with same smoothness degree Px1 = {p2 P can be generated and satisfies:
1> Pe (xc_) (x, > 0.5,
(2)
If a DNN model f is optimized by minimizing the average optimization error risk in with the guidance of the equal soft labels Pai = ve-_. and obtain the relevant parameter Oxi, where the optimization error risk is measured by Kullback-Leible divergence loss (K L):
arg f(x;0x1) = arg min Bngninn
0x1); Px));
then the error risks (the expectation that samples are wrongly predicted by the model) for classes Ct and c- have a relationship as follows:
(((((xe;;)x))) > R([(xe_;0x)))
where the error risks can be defined:
R(f :0x1) R[[ ;0x1)) = ((e.))D(CEE((C) ;x1)) ))-
(xx)):3 )),
(5)
Corollary I demonstrates that when optimizing hard and easy classes with equal intensity, the model will inevitably be biased, and this bias mainly comes from the characteristics of the sample itself and is not related to the optimization method. Based on the Corollary [ we can further analyze the performance differences with the guidance the different types of soft labels. Here we provide Theorem I about the relationship between class-wiss smoothness degree of soft labels and fairness.
Theorem 1. Following the setting in Corollary 7 for dataset D containing 2 classes (c+ and c-), two soft label distribution (PA1 = {r2t pe_ and Px2 = {222 p²2 }) exist, where Px2 have a correct prediction distribution but have limited different class-wise smoothness degree of soft labels 0):
(vi 0, V2
1> Pel (Ee,) Pei (Ia U1 > Pe (Xee Al Pe (Ic_) p22 (x__) pcl (xc 0.5,
(6)
then the model is trained with the guidance of the soft label distribution Px1 and soft label distribution Pxa and obtains the trained model parameters Ox1 and 0xz- respectively. If the model parameter x
still satisfies: Rif(Ze, 0x2)) > R(f(x,_ (et_;0xaa) ;0x2)), then the model's error risk for hard classes c4 and easy classes c_ has a relationship as follows:
R(f(ze
(tlf(==_0xx))) R(f(Fe,;0x0)) R[f(Ic_;
The proof of Theorem T can be found in Appendix A. Based on Theorem D the class-wise smoothness degree of soft labels theoretically has an impact on class-wis wise robust fairness. If the soft label distribution with different smoothness degree Pxa applied to guide the model training, where the sharper smoothness degree of soft labels for hard classes and smoother smoothness degree of soft labels for easy classes, the model will appear smaller enror risk gap between easy and hard class compared with the soft label distribution with same smoothness degree Px1. which demonstrates better robust fairness. The Theorem class-we wise smoothness degree of soft labels, the model can achieve class-wise robust faimess.
T Itheoretically demonstrates that we appropriately adjust the
5

---

4 Anti-Bias Soft Label Distillation
4.1 Overall Framework
Based on the above finding, adjusting the class-wise smoothness degree of soft labels can be regarded as potential way to obtain robust fairness. Then consider introducing this ideology into Knowledge Distillation (KD), which has been proven to be an effective method to improve the robustness of small models 44 45 43
12 42 Since the core idea of KD is to use the teacher's soft labels to guide the student's optimization process, we can adjust the class-wise smoothness degree of soft labels and obtain the student with both strong robustness and fairness.
Here we propose the Anti-Bias Soft Label Distillation (ABSLD) to obtain a student with adversarial robust fairness. We formulate the optimization objective function for ABSLD follows:
arg min ()).)>))DLustttt x; fx, (:)),
(8)
f.
C C - R(fA(E.))
s.t. R(fa(Ea))
where x, and ã; are the clean examples and adversarial examples of the i-th class, fa denotes the student model, Jt denotes the teacher model with Anti-Bias Soft Labels, C is the total number of classes, Labsid is the loss function, and (zx)) denotes the robust error risk of l--th class in student model fa. Here we apply the Cross-Entropp loss C'E(f((x)) y) as the evaluation criterion of the optimization error risk R(f. (Zx)) following [19].
4.2 Re-temperate Teacher's s Soft Labels
In order to adjust the class-wise smoothness degree of soft labels in ARD, two options exist: one is to use student feedback to update the teacher parameter in the process of optimizing students, but this option requires retraining the teacher model, which may bring the pretty optimization difficulty and computational overhead; the other is to directly adjust the smoothness degree of soft labels for different classes. Inspired by 42). we apply the temperature as a means of directly controlling the smoothness degree of soft labels during the training process. Here, we provide Theorem 2 about the relationship between the teacher's temperature and the student's 's class- wise error risk gap.
Theorem 2. If the teacher ft has a correct prediction distribution, the teacher temperature Th+ of hard class c+ is positively correlated with the error risk gap for student f and the teacher temperature +-- of easy class e- is negatively correlated with the error risk gap for student f
The proof of Theorem 2 can be found in Appendix A.2 In particular, just as the conclusion in [38] The teacher has a more correct prediction distribution than the student even in the worst classes, which means Theorem 2 holds in most cases. Theorem 2 demonstrates that the different temperatures can influence the student robust fairness: when the student's error risk of k-th class is larger than the average eтror risk, we think that this type of class is relatively hard compared with others, then the teacher temperature for k-th class will reduce and the smoothness degree of soft labels will be sharper, and the optimization gap between teacher distribution and student distribution in k-th class will corresponding increase, leading to stronger learning intensity for k-th class and final reduce the student's i flass-wise optimization error risk gap.
To achieve the above optimization goal, we adjust the teacher's k--th class temperature o for the guidance of adversarial examples as follows:
R(f.(o)) F/ 3 max ((([)()))) - - E. R(S((.))))
(10)
where ,3 is the learning rate, mar(.) denotes taking the maximum value, and |-| represents taking the absolute value, max(I.) is applied for regularization to maintain the stability of optimization. The update operation in Eq 10) can change the teacher temperature F based on the gap between the
student's k-th class error risk ((J)))x)) and the average error risk _ DE ((f)(
Meanwhile, according to (39) both clean and adversarial examples exist the fairness problems and can affect each other, so it is necessary to achieve fairness for both types of data. Since clean and

---

end for Algorithm Overview of ABSLD
Require:
the train dataset D, Student fa with random initial weight 0, and temperature Ta. pretrained robust teacher ft. the initial temperature 7y and F for the teacher's soft labels for clean examples and adversarial examples I, where y = (1,-.., C), the max training epochs max-epoch.
1: for O to mar-epoch do
2: for
y {I,.................. ......... C} do 3: R(f)(Ex)) = 0, R(f.(z()) 0. // Initialize the clean and robust error risk each class.
4: end for
5:
for Every månibatch(z, y) in D do
6:
f = argmar K (T; fi +()). // Get adversarial examples with teacher's soft labels. ||8-0||5+
7:
0,=0, 0g 11 (xy) = ((fa(G,)) (F)(f. I/ Calculate robust error risk for each class.
// Update student weight 0g with teacher's 's soft labels.
8:
9:
H(fa(Zy)) = H(fo(ax)) CC((f())))) Calculate clean envor risk for each class. end for
10:
11:
for in = {I,..., ., C, do
12:
Update 7t and o. based Eq. 10
// Re-temperate teacher's soft labely for it and
13:
end for
14:
adversarial examples of the same classes may have different error risks during the training process, is unreasonable to use the same set of class temperatures to adjust both clean and adversarial examples. Here we simultancously optimize the student's 's clean error risk R[ffa (zx)) and the student's robust error risk (Tx )). in other words, apply two different sets of teacher temperatures: T and 7 for the adjustment of the teacher's soft labels for clean and adversarial examples, respectively.
Then we extend the Anti-Bias Soft Label Distillation based on 45 and the loss function Cabstd in (8. can be formulated as follows:
1 x;; = = Σ
a
Σ KL(f c
(11)
where KL represents Becllbacck-eiiblee divergence loss, is the trade-off parameter between accuracy and robustness, fx;; denotes model f predicts the output probability of with temperature r in the final softmax layer. It should be mentioned that the teacher is frozen and we apply the teacher's predicted soft labels f (mr; 7{) for k-th class to generate adversarial examples 7g as follows:
f argmar (L(f.("x: (#x3 #1))
(12)
|
and the complete process can be viewed in Algorithm
Experiments
5.1 Experimental Settings
We conduct our experiments on three datasets: CIFAR-10 16), CIFAR-100, and -mag-
The results about CIFAR-100 and Tiny-ImageNet are in Appendix A.4 and A.5 respectively.
Baselines. We consider the standard training method and eight state-of-the-art methods as comparison
methods: AT methods: SAT [20]. and TRADES [1]; ARD methods: RSLAD [45). and AdaAD
12: Robust Fairness methods: FRL
39). BAT 28], CFA (36). and Fair-ARD 38
Student and Teacher Networks. For the student model. here we consider two networks for CIFAR- 10 and CIFAR-100 including ResNet-18 18 [10] and MobileNet-v2 v2 [26). For the teacher model, follow the setting in [45), and we select WiderResNet-344
[40] trained by [4 for CIFAR- 10 and
WiderResNet-70-- trained by 9 for CIFAR-100. The teacher's performance is in Appendix A.7
Training Setting. For ABSLD, we train the model using the Stochastic Gradient Descent (SGD) optimizer with an initial learning rate of O. momentum of 0.9, and weight decay of 2c-4. The
7

---

Table 1: Result in average robustness($) (Avg. 1), worst robustness( (Worstt), and normalized standard deviation (NSD.) on CIFAR-10 of ResNet-
Method
Avg
Worsi
Clean
NSD
Avg.
PGSM
Word
NSD
Avg
RGD Wirdi
NSD
Avg
CW
Wirsi
NSD
Avg.
Wirsi
AA
NSD
Natural
94.37
18.60
SAT 20)
$4.03
63.90
0.118
56.7]
26.70
0.283
19.34
21,00
0.332
18.99
19.90
0.352
16.64
15.50
0.385
TRADES
$1.45
67.60
0.113
56.85
36,60
0.267
51.78
30.40
0.301
49.15
27.10
0.341
46.17
25.00
0.350
RSLAD 45
82.94
66.30
0.122
59.51
34,70
0.244
54,00
28.50
0.276
52.51
27.00
0.206
51.25
25.50
0.304
AdaAD 12
$4.75
68.10
0.114
59.70
34,80
0.246
53.82
29.30
0.255
52.30
26.00
0.312
50.91
24.70
0.322
32.29
0.114
55.03
49,05
31.70
47.88
30.40
46.54
28.30
BAI 25
36.72
72.30
0,092
60.97
35.80
0.255
49.60
22.70
0.325
47.69
19.50
0.354
48.18
20.70
0.341
CBA 36
78.64
63.60
0.123
57.95
36,90
0.231
54.42
33.30
0.258
50.91
27.50
0.258
50.37
26.70
0.296
Fii-ARDL:8
83.81
69,40
0.1 112
58.41
36.50
0.251
50.91
19.90
0.296
49.96
30
0.312
47.97
25.10
0.338
ABSLD
54.50
36.50
0.316
0.256
Table 2: Result in average robustness($) (Avg. worst robustness (% (Worstt), and normalized
standard deviation (NSD.) CIFAR-10 of MobileNet-v2.
Method
Clean
K:0
FUSM
K30
RGD
NSD
CW
N3D
AA
KSD
Natural
93.35
12.21
0.90
0.750
0
SAT 20
82.30
65.50
11:13
56.19
34.60
0..779
48.52
24.90
0..33
47.22
23.50
11.661
44.38
18.50
0.410
IRADES 4T
79.37
59.00
11.131
54.94
31.50
0.227
50,415
27.20
01.330
47,012
23.20
01.667
46.25
22.30
01579
RSLAD 45
82.96
66:30
0.130
59.84
34.30
0.256
53.88
26.30
0
52.28
25.70
0.316
50.67
23.00
0.333
AdaAD
12
83.72
51.89
0.300
FRI 19
81.02
70.50
0.088
53.84
40.50
0.207
47.71
35.00
0.241
44.96
36.90
0.276
43.53
28.40
0.292
RAT H4
83.01
70.50
0.102
51.01
32.10
0.284
44.08
25.00
0.344
41.85
21.60
0.397
42.65
23.40
0.360
CFA 54
30.34
64.60
0.112
56.45
32.20
0.260
52.34
28.10
0.202
48.62
23.30
0.520
50.68
23.50
0.331
82.44
29.90
48.18
01302
ABSL.D
learning rate 3 of temperature is initially set as 0.1. For CIFAR-IO and CIFAR-I00, we set the training epochs to 300. The learning rate is divided by 10 at the 215-th, 260-th, and 285-th epochs; We set the
batch size to 128 for both CIFAR- 10 and CIFAR- 100 following [45]. For the inner maximization, we 10-step PGD with random start size of 0.001 and a step size of 2/255, and the perturbation is bounded to the Lis norm e = 8/255. The more training setting can be found in Appendix AA.3
Metrics. We apply two metrics to evaluate the robust faireess: Normalized Standard Deviation ( (38] and the worst-class robustness (36]. NSD can reflect robust fairness while also considering the average robustness. The smaller standard deviation means better fairness, and the larger average means better robustness, so the smaller NSD means better comprchensive performance in terms of fairness and robustness. The worst-class robustness is easy to understand, and larger value means better fairness. For CIFAR-10 10, we directly report the worst class robust accuracy; For CIFAR- 100 and Tiny-ImageNer due to the poor performance of the worst class robustness and only 100 images (CIFAR-100) or 50 images (Tiny- Einy-ImageNet) for each class in the test set, we follow the operation in CFA 36 and report the worst 10% class robust accuracy. Besides, we also report the average robustness as reference. The attack setting for evaluation can be found in Appendix A.3
5.2 Robust Fairness Performance
The performances of ResNet-18 and MobileNet-v2 trained by our ABSLD and other baseline methods under the various attacks are shown Table T Table 2 for CIFAR-IO. The results demonstrate that ABSLD achieves the state-of -fee--ff-hheheee the-art worst-class robustness CIFAR-10. 10, ABSLD improves the worst class robustness by 2.0%, 3.2%, 2.4%, and 2.9% compared with the best bascline method against the FGSM, PGD, CW ... and AA. Moreover, ABSLD shows relevant superiority on MobileNet-v2 compared with other methods.
CIFAR-10 10. For ResNet- 18 on
Moreover, ABSLD can also show the best comprehensive performance of fairness and robustness (NSD) on CIFAR-10. For ResNet-18 on CIFAR-10, 10, ABSLD reduces the NSD by 0.028, 0.032, 0.017, and 0.024 compared with the best bascline method against the FGSM, PGD, CW ou. and AA. The result indicates that although the trade-off between robustness and fairness still exists as [9 say, we obtain the highest robust fairness while sacrificing the least average robustness.
Meanwhile, we visualize the class wise robustness Figure 3 and the result shows that the robustness of harder classes (class 3, 4, 5, 6) have different levels of improvement, which demonstrates that our method is beneficial to the overall robust fairness but not only to the worst class. Moreover, combined
NSD is a metric applied in (88 to measure the robust faimesss NSD SD/Avg. where SD is the Standard Deviation of class wise robustness and Avg. is average robustaess.
8

---

60
30
ASLAD ARKID
K5.40 ABRLD
72
Actsmo (641
Tollen www.n.aaaaa...... (0)
Class
Clasa
(a) Class-wise Robustness of ResNet-18 on CIFAR-10.
Class-wise Rohustness of MobileNet-v2 on CIFAR-10.
Figure 3: The class wise robustness (PGD) of models guided by RSLAD and ABSLD CIFAR-10 1-00. We can see that the harder classes' robustness (class 3, 4, 5, 6) of ABSLD (blue lines) have different levels of improvement compared with RSLAD (red lines).
with Figure 2 find that the trend of class-wise bias is similar in different training strategies, indicating that the bias may be sourced from the dataset itself, which further confirms Corollary |
In particular, we compare ABSLD with Fair-ARD 38 which is an adaptive re- re-weighting method on ARD. From the results, ABSLD has better robust fairness performance, which means that our proposed re-temperating method has superiority compared to the re-weighting method.
Bandbe
Roeline
ABsLD)
ARSED
CH.
Figure 4: Ablation study for Baseline, Figure 5: Standard deviation Figure 6: Standard deviation Baselin+ABSLL adu, and ABSLD, of class-we wise clean optimiza- of class-wise adversarial opti- mization error risk.
tion error risk.
5.3 Ablation Study
To certify the effectiveness of our method, we perform ablation experiments on every component of ABSLD. First, based on the baseline method [45], we re-temperate the teacher's soft labels for the adversarial examples but do re-temperate the teacher's soft labels for the clean examples (Basllin+ABBSDD.
then we re-temperate the teacher's soft labels for both the adversarial examples and clean examples (ABSLD). The results are shown in Figure 4 The results demonstrate the effectiveness of our ABSLD, and pursuing fairness for clean examples can also help robust
fairness for adversarial examples as claimed in [39
Meanwhile, to verify the optimization effect of our method, we visualize the standard deviation of class-wise optimization error risk in the training process (the optimization error risk normalized by dividing the mean). which can reflect the optimization gap between different classes. We visualize the standard deviation of both the clean and adversarial optimization error risk, and the results are shown in Figure 5) and Figure 6 We can notice that the standard deviation of the baseline increases as training epoch increases, which demonstrates that the baseline pays more attention to reducing the error risk of easy class, but the error risk of hard class is ignored, eventually leading to robust unfairness. On the contrary, our ABSLD can remarkably reduce the standard deviation of student's class-wal wise optimization error risk, which demonstrates the effectiveness of our method.

---

Conclusion
In this paper, we comprchensively explored the potential factors that influence robust fairness the model optimization process. We first found that the smoothness degrees of soft labels for different classes can be applied to eliminate the robust faimesss based on empirical observation and theoretical analysis. Then proposed Anti-Bias Soft Label Distillation (ABSLD) to address the robust faimesss problem by adjusting the class-wis wise smoothness degree of soft labels. We adjusted the teacher's 's soft labels by assigning different temperatures to different classes based the performance of student's class-wise error risk. A series of experiments proved that ABSLD was superior to state-of-the-at methods in the comprehensive metric (NSD) of robustness and fairness.
Acknowledgement
This work was supported by Alibaba Group through Alibaba Reasearch Intern Program, the Project of the National Natural Science Foundation of China (No.62076018), and the Fundamental Research Funds for the Central Universities.
References
[1] Andriushchenko, M., Croce, F., Flammarion, N., Hein, M.: Square attack: query-efficient black-box adversarial attack via random search. In: European Conference on Computer Vision. pp. 484-501. Springer (2020)
[2] Benz, P., Zhang, C., Karjauv, A., Kweon, I.S.: Robustness may be at odds with fairness: An empirical study on class-wise accuracy. In: NeurIPS 2020 Workshop on Pre-registration in Machine Learning, pp. 325-342. PMLR (2021)
[3] Carlini, N., Wagner, D.: Towards evaluating the robustness neural networks. In: 2017 ieee symposium security and privacy (sp). pp. 39-57. IEEE (2017)
[4] Croce, F., Hein, M.: Minimally distorted adversarial examples with a fast adaptive boundary attack. International Conference on Machine Learning. pp. 2196-2290000 PMLR (2020)
[5] Croce, F,, Hein, M.: Reliable evaluation of adversarial robustness with ensemble of diverse parameter-fie free attacks. In: International conference on machine learning. pp. 2206-222622211 PMLR (2020)
[6] Girshick. R.: Fast r-cnn. In: Proceedings of the IEEE international conference computer vision. pp. 1440-1448 (2015)
[7] Goldblum, M., Fowl, L.,, Feizi, S., Goldstein, T.: Adversarially robust distillation. In: Proceed- ings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 3996-4003 (2020)
[8] Goodfellow, IJJ., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint 1t.iv:1412.66651 1412.552 (2014)
[9] Gowal, S., Qin, C., Uesato, J., Mann, T., Kohli, P:: Uncovering the limits of adversarial training against norm-bounded adversarial examples. arXiv preprint arXiv:2010.03593 (2020)
[10] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778 (2016)
[11] Hinton, G., Vinyals, O., Dean, J., et al.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.025: 2(7) (2015)
[12] Huang, B., Chen, M., Wang, Y., Lu, J., Cheng, M., Wang, W.: Boosting accuracy and robustness of student models via adaptive adversarial distillation. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp.
(2023)
10

---

[13] Jia, X., Zhang, Y., Wei, X., Wu, B., Ma, K., Wang, J., Cao, X.: Prior-guided adversarial
initialization for fast adversarial training. In: European Conference on Computer Vision, pp. 567-584. Springer (2022)
[14] Jia, x., Zhang, Y., Wei, X., Wu, B., Ma, K., Wang, J., Cao Sr, X.: Improving fast adversarial training with prior-guut -urdeguuiddddd knowledge. arXiv preprint arXiv:2304.00200 (2023)
[15] Jia, X., Zhang, Y., Wu, B., Ma, K., Wang. J., Cao, X.: Las-at: adversarial training with learnable attack strategy. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13398 13408 (2022)
[16] Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
[17] Le, Y., Yang, X.: Tiny imagenet visual recognition challenge. CS 231N 7(7), (2015)
[18] Li, B., Liu, W.: Wat: improve the worst-class robustness adversarial training. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 14982- 14982-14990) (2023)
[19] Ma, X., Wang. Z., Liu, W.: On the tradeoff between robustness and fairness. Advances in Neural Information Processing Systems 35, 36230-26244 (2022)
[20] Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 706.06083 (2017)
[21] Pang, T., Yang, X., Dong, Y., Xu, K., Zhu, J., Su, H.: Boosting adversarial training with hypersphere embedding. Advances in Neural Information Processing Systems 33, 7779-7792 (2020)
[22] Rebuffi, S.A., Gowal, S., Calian, D.A., Stimberg, F. Wiles, O., Mann, T.A.: Data augmentation can improve robustness. Advances in Neural Information Processing Systems 34, 209222299295999 29948 (2021)
[23] Rice, L.. Wong, E., Kolter, Z.: Overfitting in adversarially robust deep learning. In: International Conference on Machine Learning, pp. 8093-810 8104. PMLR (2020)
[24] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer- Assisted intereetion-MICC 2015 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. pp. 234-241. Springer (2015)
[25] Ruan, S., Dong, Y., Su, H., Peng, J., Chen, N., Wei, X.: Improving viewpoint robustness for visual recognition via adversarial training. arXiv preprint arXiv:2307.11528 (2023)
[26] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4510-4520 (2018)
[27] Sehwag, V., Mahloujifar, S.. Handina, T., Dai, S., Xiang, C., Chiang, M., Mittal, P.: Robust learning meets generative models: Can proxy distributions improve adversarial robustness? arXiv preprint arXiv:2104.0 104.0025 (2021)
[28] Sun, C., Xu, C., Yao, C., Liang, S., Wu, Y., Liang, D., Liu, X., Liu, A.: fariness via balance adversarial training. In: Proceedings of the AAAI Conference Artificial Intelligence. vol. 37, pp. 15161-15169 (2023)
Improving robust
[29] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference computer vision and pattern recognition. pp. 2818-2826 (2016)
[30] Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R.: Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 (2013)
[31] Tian, Q. Kuang, K., Jiang, K., Wu, F., Wang, Y.: Analysis and applications of class-wise robustness in adversarial training. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1561-1570 (2021)
11

---

[32] Wang. Y., Zou, D., Yi, J., Bailey, J., Ma, X., Gu, Q.: Improving adversarial robustness requires revisiting misclassified examples. In: International Conference on Learning Representations (2019)
[33] Wei, X., Guo, Y., Yu, J.: Adversarial sticker: A stealthy attack method in the physical world. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)
[34] Wei, X., Guo, Y., Yu, J., Zhang, B.: Simultaneously optimizing perturbations and positions for black-box adversaria patch attacks. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)
[35] Wei, X., Wang, S., Yan, H.: Efficient robustness assessment via adversarial spatial-temporal focus on videos. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023)
[36] Wei, Z., Wang. Y., Guo, Y., Wang, Y.: Cfa: Class-wise calibrated fair adversarial training. In: CVPR. pp. 8193-8201 (2023)
[37] Wu, Z.. Gao, H., Zhang, S., Gao, Y.: Understanding the robustness-accuracy tradeoff by rethinking robust fairness (2021)
[38] Xinli, Y., Mou, N., Qian, W., Lingchen, Z.: Revisiting adversarial robustness distillation from the perspective of robust fairness. NeurIPS (2023)
[39] Xu, H., Liu, X., Li, Y.. Jain, A., Tang, To be robust or be fair: Towards fairness in adversarial training. In: International conference on machine learning, pp. 11422-11501. PMLR (2021)
[40] Zagoruyko, S., Komodakis, N.: Wide residual networks. arXiv preprint arXiv:1605.07144 (2016)
[41] Zhang, H., Yu. Y., Jiao. J., Xing, E., El Ghaoui, L., Jordan, M.: Theoretically principled trade-off between robustness and accuracy. In: International conference on machine learning. pp. 7472-7482. PMLR (2019)
[42] Zhao, S., Wang, X., Wei, X.: Mitigating the accuracy--obustnees trade-off via multi-teacher adversarial distillation. arXiv preprint arXiv:2306. 16170 (2023)
[43] Zhao, S., Yu, J., Sun, Z., Zhang, B., Wei, X.: Enhanced accuracy and robustness via multi- teacher adversarial distillation. In: Computer Tel Aviv, Israel, October 23 23-27,00 2022, Proceedings, Part IV. pp. 585-602. Springer (2022)
2022: 17th European Conference,
[44] Zhu, J.. Yao, J., Han, B., Zhang, J., Liu, T., Niu, G., Zhou, J., Xu, J., Yang, H.: Reliable adversarial distillation with unreliable teachers. arXiv preprint arXiv:2106.044 146.449288888011101 (2021)
[45] Zi, B., Zhao, S., Ma, X., Jiang, Y.G.: Revisiting adversarial robustness distillation: Robust soft labels make student better. In: International Conference Computer Vision (2021)
12

---

A Appendix
A.1 The proof of Theorems in Sec. 3
For the initial state of the DNN model f with random distribution I, the DNN model has no preference for any examples. So for the dataset containing hard class c+ and easy class c., have a relationship between class-wise error risk as follows:
Rif ; 00)) = R(J(Ze ;0r)),
(13)
in other words, the model has the same error risk for easy class c_ and hard class c4, so we set the model's prediction distribution fx; 07) tv'_ (x)), Pe,, (z)) and has the relationship as follows:
E(p' (z)) = B(p', (Ie.)) = E(p' (Ic_)) = 0.5.
(14)
Then we analyze the process of knowledge distillation, we assume that DNN model f is optimized with the guidance of the soft label distribution with the same smoothness degree PAI, which satisfies:
P (xe_) = p2L (xe.) > 0.5 > p2l (xe.) = p__ (xe_).
(15)
And the updated parameter fx guided by the soft label distribution Pai can formulated as follows:
0Λ1 = 01=00-1 -1 LKL(
(16)
00
here we divide the partial derivative of the optimization function KL with respect to the student parameter 0 into two parts:
01),
dze(z)
(17)
00
00
θz.(x)
where ze(z)) denotes the o-th dimension output logits of the model before the softmax layer (denote the prediction logits of c-th class).
The first part @z.(x) can be regarded as the impact of the class itself on model weight optimization, in a practical sense, this part reflects the inner relationship between DNN and sample, more specifically, it can reflect the difficulty of the sample itself for the model and not relate to the optimization object.
The second part [KL(f(r:)))Paa) can be regarded as the impact of the optimization object on the dz,(æ)
different classes. Although adjusting the optimization goal is not enough to change the bias derived
from the sample itself, it can be adjusted to make the model perform as fair as possible.
Here we further extend the partial derivative of optimization object K KL[((x; 0r), Pxa) toward the prediction logit (z) of class e and obtain:
OKL(f(x: 01), PAI)
(z,
0pi(x)
OKL[J(x; 0r), Pxi)
(2)
Op((x)
pt (x))x(x)) r62
-p((x) (x),
(18)
so we can easily obtain:
OKL (x__ 01), IE(
OK (xc+: :0):00))))))))))) PA1)
IE(
82 (x.-)
Óz (Bot)
= -E(p((xo) - - (xe-)) = ((p)
pel (x++))
(19)
13

---

[KL(f(Zct (00c:::01,440 Pai, IE[ Ozc- (1ict)
OKL(f :1),
IE
Ozot (Fe-))
((p((x))) - Po- (Ieet) -F Vet (zr-))
(20)
Based on the Corollary 3.1, the DNN model with parameter 0x1 has different the enror risk of the hard class c+ and the easy class c-, so we have:
R(f(FeL: :0x))) > R(f(Ec_;
(21)
from the optimization perspective, the optimization gradient toward the model parameter for different classes directly influences the class error risk, more specifically, the easy class error risk is less than the hard class error risk, so the gradient expectation of the easy class is higher than the gradient expectation of hard class, and the proof is as follows:
In the initial state, we assume that the model is a uniform distribution, and in this case, the error optimization risk is the same for the easy and hard classes:
[(KL(f(ro= 01), Px.) = (( Px)))
(22)
then will simplify the optimization of the model into one-step gradient iteration process, which means we only consider the initial state before optimization and the last state after optimization. The model parameter is updated from initial 0, optimized opt. since easy classes perform better than hard classes after the optimization process, the easy class error risk is smaller than hard class error risk:
B(KL()(e_-:0p))) ((KL)((r++ Oopt): Pxt))
< (())) = ([ 01), Px1)),
(23)
based on the above result, we can have the result as follows:
E[ EQKKLA ( Oopt), Pxi) - KI(S(Be- 01); Pxi)l) B(KL(f(F 0op)) Px1) KL(f(I Px)))
(24)
at this time, we assume that model's parameter 0 differentiable and continuous, then we can approximate the gradient expectation of the partial derivative E( (KL(f(2.-00))) P.))) and
86
E(
0K as follows:
00
(KL(f(xe-:0)).
E(
00
(T(( Pxi) - (T( Pai))
= E
(25)
Bopt 01
0K OKL(J(E++;0);
E
00
KL(f(Io+ Oopt), PAi) KL(S(E043 01), Pal)
(26)
Oppt 0
combined with the relationship between easy class error risk and hard class error risk, can obtain result that the gradient absolute value expectation of partial derivative about the easy class's optimization goal toward the model parameter (IE() [KI.(ee_...)))P)..)) ()) is higher than the gradient dM
absolute value expectation of partial derivative about the hard class's optimization goal toward the model parameter (IE() (1)
))) as follows:
88
OKL (x 01), Px1) 1) > E(1 [KL(f(Ec+: (1)) Px1) I),
E(
(27)
00
00
so we can obtain the assumption: the model is a uniform distribution before the optimization process and the easy class error risk is less than the hard class error risk after optimization process,
14

---

then the gradient expectation absolute value of partial derivative about the easy class's optimization goal toward the model parameter is higher than the gradient expectation absolute value of partial derivative about the hard class's optimization goal toward the model parameter.
Then we can decouple the gradient into two parts based on the class types: - and )k)
00
Here we take the hard class
example. the gradient can be divided into two parts: (2.1 OKLLf. (1)) and dž. (2.1) (K where the first part represents the 26
BEc+ ability to make hard class's samples more like hard class, and the second part represents the ability to make hard class's samples less like easy class. The final prediction performance of hard class c- is influenced by both parts of the optimization gradient due to the softmax operation.
50
Oze-
Based on the above analysis, the relationship between the sum value of the above two parts of gradient for hard class (V/~1) and easy class (V f~az1 can be assumed under a more stringent condition as follows:
Oze- (Fe-
OKL.S,(Fo- ;01), PX1)
Ozot (5 (KT([s(xe--;0)).
E(
f~11
00
Oza-
Ozet Ozot (Io+) (f 01), (x))
00
Ize- (Io+) O (F 01), Px1)
M~11 B(
09
Ozc-
09
Dzet
(28)
(q.20) and Eq. (28). we have a relationship about the derivative of &e toward model parameter 8, which can reflect the bias of sample for the model:
Combined with E 19)
iz._ (xo-)
02e+ (Ic-)
02,_ (I
0zr+ (Iot)
E((
R[
(29)
00
00
00
00
After the analysis of the model characteristic with the guidance of soft label distribution PAI. we try to compare the model characteristic difference between soft label distribution Px1 and soft label distribution Px2. Initially, we can obtain the following relationship for the probabilities of different distributions:
X2 Pet (Zc,) = Pet
=P (xce) + 21 > Pc+ (Ic.)
221 (xx_)) > 22 (I._) Al (xc_ U/2 0.5,
(30)
then we further analyze the model blasswwise error risk gap guided by label distribution. The total optimization gradient for different classes can reflect the bias degree, we get the class optimization gradient gap guided by label distribution Px1 and label distribution Px2, respectively:
BY FNAI JUMAL
Oz 0zc-
(xc ( PA1)
O2ct (Fc-) (F PX1)
-E(
00
Ozc-
00
Ozct
Ozo (Io+) OKL(S (Io4; 01), Px1)
(xo+) OKL(Ja 07), PAI))
00
Ozc-
00
Ozct
= -EE(1 dz (xc-)
(xo-) -p!)
Vide 1
Dzet (xc-)
vet (x=)))
00
00
Oza-
(Iot)
Oznt (Ict)
[ip!
(a(()))) -
11 (Ic+) (occt)-P0 -p!)),
(31)
00
ByNA2 VinNe
fm222
Sm~22
Jz. (Io-)
21 (5:a-) -v() + 2zo+ (To-))o)))))))))))
=E(
1106 X2 (Icc ))
00
00
Oz (Ic+)
(#c+)) - 02ct (Ic+)
[pp -
Po-
22
[p (xc+) ( - p())),
(32)
00
00
15

---

if the model parameter 8x2 still satisfies: (( :0x2)) > R(f(ze_;0x)) :xx]), then we can obtain By > and the relationship between y and Br-22 as follows:
y - B
Oze (Ic-)
(2c+(
lip (Ic-) ) - Po- (Ic-)) +
(p (Ic-) )- - Pot (xe-))
00
00
Oz Ozo (Io+)
02nt (Iot)
K-pe (Ic+) ) + pe- (Ic+))
( (Ic+) - pet (Ic+)))
00
00
Ize- (Io-)
Diot (Ic-)
U22 + tza- (
Ozot (Zo+(
=B(
U2 +
111 +
(11) > 0,
(33)
00
00
00
00
so we can have the following conclusion:
BY~A1
By~
(34)
Based on the above results, the model total gradient expectation gap of different classes trained by soft label distribution Pxi is larger than gradient expectation gap between different classes trained by soft label distribution Px2, the bias degree of model trained by the soft label distribution Px is greater than by the bias degree of model trained by the guidance of soft label distribution Px2, we can get the conclusion as follows:
R( (F (ax))) R(f(xa > :(f(Fe4;0xa)) :(f(Eo_:022))
(35)
Then the Theorem proved.
A.2 The proof of Theorems 2 in Sec. 4
Based on the correct prediction distribution assumption about the teacher model, we can obtain the relationship of probability toward k-th class as follows:
[pb (#gg) +() > B(pee
(36)
Then extend the temperature into the mathematical formula of the prediction probability, we can obtain:
exp(2x(<x//T{)
pk (F ES cxp(2,(
(37)
where =x(x)) denotes the k--th dimension output logits of model before softmax layer. Since E exp(e,(xx)//z) is applied normalization application, here we mainly focus the change exp(>((xx// with the temperature Th. exp(-) is a monotonic increasing function, so based on the (36. easily obtain:
((xx((x))) > ((ze)((<x)))
(38)
here mainly focus the change cxp(s(xx)/r{) with the temperature T.. We assume the 7f increase into + > 0), then we have the partial derivative of (atp:///dtteeopppaaa/aa/a///a with respect to =(rx)) as follows:
-xpp://)))rp((p(//// (D))
0(*(xx))
(erp(=(xx)/T)) (erp(z(s*)//1)) ( +a)))) >0,
(39)
th + D
so we have the conclusion as follows:
exp(zx(zw)/T{)- exp(zx(rx) (Tt AT)
eapp(=e4 (xx) cop( / AD)),
(40)
then we can have relationship between the prediction distribution of different temperature as follows: ((p) (t)) - E(p'y (xxi
('pi AT)) - Elpezs (FNI AT)).
(41)
16

---

Based on the above analysis, we assume that the teacher's 's temperature of the easy class and hard class increase into 7h + A and + + A and obtain teacher model with soft label prediction fz- then following the Derivation Theorem I, we can obtain:
Bla5 B 1
Oz.- (5c-)
=E(
1V (Te- - (Fc-
: To + +a)))
00
Ozct (Ic-)
+
(P/+ (Fo- + + - Pot (To-i =(_))
00
Occ (Ic+)
(p) (Io+) (Ic+iTe - Pe- (Ioti Tct + A
00
Ozot (Io+)
(v'ot (Ko+i Fet + A pet (Foti ic-)))
(42)
00
Then we can further analyze the relationship between the temperature and the error risk gap. Here we assume that the teacher's temperature of easy class T is unchanged (A 0), the teacher's
temperature of hard class & increases into T A (A 0), obtain:
By.h < BB,
(43)
following the analysis Appendix A.II DNN model fa is optimized with the guidance of teacher's soft label distribution ft and f., and can obtain the model parameter 02 and 6ú, respectively. We can get the conclusion as follows:
(()) - R(Sa(re: R(f(xe,; :0)) R(f(Fa
(44)
Then we assume that the teacher's temperature of hard class -ot is unchanged (A = 0), the teacher's temperature of easy class - increases into Tr A (A 0), we can obtain:
Bla,, > Bian''
(45)
and following the analysis in Appendix A. we can get the conclusion as follows:
R(fa (x ((f R[f(ro4:0) R(f(xx_:)))
(46)
Based on the above results, we can obtain the teacher temperature 724 of hard class c+ is positively correlated with the error risk gap for student fa, and the teacher temperature 7!_ of easy class is negatively correlated with the error risk gap for student fs.
Then the Theorem 2 is proved.
A.3 Additional Experimental Setting
As discussed in (36,, the worst class robust accuracy changes drastically when the average robustness convergence, so we follow [36] and select the best checkpoint of the highest mean value of all-class average robustness and the worst class robustness (where the worst class for CIFAR-IO and the worst 10% class for CIFAR-100) 100) in all baselines and our method for a fair comparison.
For ABSLD, to maintain stability when adjusting the teacher prediction distribution, we hold the student temperature T" constant, and the student's optimization error risk for each class can be compared under the same standard. The teacher temperature of тę and 7f are initially set as 1 for all the classes. The student temperature т. is set to constant without additional adjustment. With additional instruction, the maximum and minimum values of temperature are 5 and 0.5, respectively; For ResNet-18 on CIFAR-100, the maximum and minimum values of temperature are and 0.8, respectively. All the experiments are conducted in single GeForce RTX 3090, and our ABSLD takes approximately one GPU day for training model.
For the baselines, we strictly follow original setting if without additional instruction. For FRI 39 we use the seweight+Remarge under the threshold of 0.05. For CFA [6), we select the best version (TRADES+CFA) reported in the original article. For the training of CFA on CIFAR- 100, we set
17

---

Table 3: Result in average robustness(%) (Avg. 1). worst- 10% robustness(%) (Worstt), and normalized standard deviation (NSD.) on CIFAR-100 of ResNet-18.
Method
Worsi
Clean
NSD
FGSM
Word
NSD
Avg
RGD Word
NSD
Avg
CW
Worsi
NSD
Avg.
Wors
AA
NSD
Natural
SAT 20)
57.18
29.30
0.292
28.95
5.60
0.607
24.56
3.20
0.682
23.78
3.10
0.697
21.78
2.30
0.747
TRADES 41
55.35
28.40
0.303
10.50
7.30
0.559
27.71
5.60
0.655
24.35
3.50
0.736
23.55
3.20
0.756
RSLAD 45
57.88
79.40
0.507
14.50
9.20
0.550
31.19
7,40
0.508
28.13
4.70
0.669
36.46
3,70
0.703
AduAD [2]
58.17
29.20
0.301
34.29
0.556
30.69
0.623
28.31
5,00
0.683
26.60
4.20
0.721
15.49
23.00
24,01
0.642
22.95
21.10
BAI
62.71
34,40
0.267
33.41
7.90
0.555
28.39
5.30
0.526
23.91
3,00
0.716
22.56
2.50
0.755
CEA 36
59.29
33.80
0.245
33.88
10.30
0.520
31.15
8.40
0.563
26.85
5.20
0.655
25.83
4.90
0.679
FairARDL:8
57.19
29.40
0.500
34,06
8,40
0.558
30.57
7,00
0.598
27.95
4.50
0.666
15
3,90
0.713
ABSLD
34,93
0.500
Table 4: Result in average robustness(%) (Avg. 1), worst- 10% robustness (%) (Worstt), and normalized standard deviation (NSDL) on CIFAR- 100 of MobileNet-v2.
Method
Avg
Worst
Clean
NSD
Avg.
FGSM
Worsl
NSD
Avg
Warsi
PGD
NSD
Arg
CW Warsi
NSD
Avg.
AA Worsi
NSD
Natural
14,86
53.90
0.150
3,95
1.423
0
0
TRADESC11
57.10
30.00
0.284
31.70
8:20
0.584
29.43
6.20
0.618
25.25
3.70
0.716
24.30
3.20
0.739
RSLADD
58.71
27.10
0.311
34.30
8:30
0.565
30.58
6.30
0.621
28.11
4.30
0.683
26.32
3.30
0.724
AdaAD 72
$4.59
24.50
0.343
31.40
6.10
0.616
27.96
4,50
0.677
25.72
2.10
0.752
23,80
1,60
01807
PAL 59
31.03
8300
0.538
25.40
3.70
23.28
0.714
65.39
27.08
22,80
21.28
CFA
59.15
34,10
0.255
32.50
8.80
0.551
29.38
7,00
0.501
25.15
4,00
0.690
23.78
3.20
0.727
Miie-RDD. 36
58.97
31.40
0.289
33.76
3.30
0.542
30.07
7.30
0.607
27.64
5.60
0.651
25.79
4.00
0.700
ARSLDD
56.66
32.00
0.252
33.87
12,90
0.448
31.24
11,40
0.488
26.41
7.50
0.564
24.57
6.70
0.597
the fairmess threshold to 0.02 for FAWA operation based on the worst- class robustness. For Fair-ARD [38], we select the best version (Fair-ARD on CIFAR-IO and Fai-RSLLAD on CIFAR-100) as reported in the original article. Due to the different strategy of selecting checkpoint, the baselines may have slight differences from the results in their original papers.
Following previous studies 45
we evaluate the trained model against white-bon adversarial attacks: FGSM 8 PGD 220, CW 3. For PGD, we apply 20 steps with step size of 2/255; For CW s.. we apply 30 steps with a step size of 2/255. Meanwhile, we apply a strong attack: AutoAttack (AA) 5 to evaluate the robustness, which includes four attacks: Auto-PGD (APGD), Difference of
42 38
Logits Ratio (DLR) attack, FAB-Attack [4], and the black-box Square Attack 11. The maximum
perturbation of all generated adversarial examples 8/255.
A.4 The Robustness CIFAR-100
The performances of 18 and MobileNet-v2 trained by our ABSLD and other baseline methods under the various attacks are shown in Table Table for CIFAR-100.
The results demonstrate that ABSLD achieves the siatte--f-theeatt worst-class robustness CIFAR- 100. For ResNet-18 18 on CIFAR- 100, ABSLD improves the worst- 10% class robustness by 2. 1.,, 2.1 %, 1.2%, and 0.7% compared with the best baseline method against the FGSM, PGD, CW s-r and AA. Moreover, ABSL.D shows relevant superiority on MobileNet-v2 compared with other methods.
Moreover, ABSLD can also show the best comprehensive performance of fairness and robustness (NSD) on CIFAR-100. For ResNet-18 on CIFAR-100, ABSLD reduces the NSD by 0.05, 0.063, 0.077, and 0.078 compared with the best baseline method against the FGSM, PGD, CW.. our and AA.
A.5 The Robustness on Tiny-ImageNet
Tiny-ImageNet as the additional dataset. We train with PreActResNet-18 with 100 epochs, while other settings are the same as CIFAR-100. For our ABSLD, the teacher model is PreActResNet-33 trained by TRADES [4]] and the maximum and minimnum values temperature are 1.1 and 0.9. We select RSLAD (45) (the baseline method) and (FA (the second-best method proven in Table I and Table 3) as the comparison method. The results in 55 show ABSLD has the best performance under the metric of the worst class robustness and NSD under different attacks, verifying effectiveness and generalization.
We select the subset of ImageNet:
18

---

Table 5: Result average robustness(%o (Avg. f), worst robustness(%) (Worstt), and normalized standard deviation (NSD.) on Tiny-ImageNe of PreActResNet-L
Matood
Clena
Wund
NSD
RGSM
Wund
NSD
Wund
PGD
NSD
CW
Wont
N5D
Wimt
AA
NSD
RSLAD
0388
CEA
46.75
17.50
0.189
23.19
3.00
0.357
20.73
2.30
0.885
1683
1.20
0.488
15.99
0.90
0.168
ABSLD
47.00
1790
0171
2533
5.00
0.291
25.41
4,60
0.313
1962
3.40
0.353
17.50
2.30
0.386
A.6
The Necessity of Adaptive Adjustment
To demonstrate the effectiveness our self- self-adaptive temperature adjustment strategy, we manually re- temperate with the static temperature for different classes based on the prior knowledge. Specifically, We set the teacher temperature for difficult class to be small ( 0.5) and set the teacher temperature for easy class to be large (T 5), which is the minimum and maximum values of temperature, respectively. The experiment Table shows that our adaptive strategy has a better performance than this manual strategy on CIFAR-10. Moreover, the manual strategy needs to be carefully designed and lacks operability for more complex datasets, e.g., 100 classes on CIFAR-100 or 200 classes on Tiny-ImageNet, so we finally apply the adaptive adjustment strategy for ABSLD.
Table 6: Result in average robustness(%) (Avg. 1), worst robustness %)) (Worstt), and normalized standard deviation (NSD.) on CIFAR-10 of 18
Method
Clean
Wons
XSD
FGSM
Wons
NSD
Avg.
Wons
PGD
NSD
Arg.
CW
Wons
NSD
Wont
AA
NSD
Manual
81.31
34.00
48.13
Adaptive
83.04
68.10
0.103
59.83
40.50
0.202
$4.50
36.50
0.216
$1.77
32.80
0.249
50.25
31.00
0.256
A.7
The Robustness of Teacher Models
Here we report the performance of Teacher models, we select WiderResNNet34-10 440 trained by
4I for CIFAR-10 and WiderResNet-7000 16 trained by [9] for CIFAR-100 following 45: For
Tiny- ImageNet, the teacher model PreActResNet-34 trained by TRADES 41]. The performance is shown in Table 7
Table 7: Robustness (%) of the teachers in our experiments.
Dataser
Model
Clean
PGSM
PGD
C.W
AA
CIFAR-10
WedcResNet
84.91
61.14
55,30
53.84
53.08
CIFAR-I00
70d-
60.96
35.89
33.58
31.05
30,03
Tiny- ImageNe
52.76
27.05
24.00
20.07
18.92
A.8 Limitations
At present, although we have improved the fairness of model adversarial robustness with minimal cost, the overall robustness remains unchanged or slightly decreases compared to previous methods in some cases, and how to solve the trade-off between robustness and fairness is one of the directions that will be further explored in the future. Mcanwhile, our method based on the adjustment towards the smoothness degree of soft labels, and cannot be directly applied adversarial training methods based on one-hot labels but needs to be combined with label smoothing operations.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Sec. 1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix A.8
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
See Appendix A.1 and Appendix A.2
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes)
Justification: See Sec.
5 and Appendix A.3
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification: The code available at supplemental material, and the code will be open access after revision.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssiinyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Sec. 5 and Appendix A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Since adversarial training and adversarial robustness distillation are time- consuming, we only ran one time.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We do not find any discrepancieswith the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Sec. 6
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper that produced the code package dataset.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code (new assets) can be found in supplemental material.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "cLrH2V6c4t", "reviewer": "Reviewer_H27W", "summary": "This paper discusses the robust fairness problem, which is essential to solve for reducing concerns surrounding class-based security. The paper majorly analyzed the inheritance of robust fairness during adversarial robustness distillation (ARD). It is found that student models only partially inherit robust fairness from teacher models. To solve this, authors have examined how the degree of smoothness of samples' soft labels influences class-wise fairness in (ARD). They empirically and theoretically prove that appropriately assigning class-wise smoothness degrees of soft labels during ARD can be beneficial to achieve robust fairness.\n\nTherefore, as a solution to the fairness problem associated with ARD, they propose the Anti-Bias Soft Label Distillation (ABSLD), a knowledge distillation framework designed to reduce error risk gaps between student classes by adjusting the class-specific smoothness degree of teacher's soft labels during training, controlled by assigning temperatures. The authors provided experiments to demonstrate their method's effectiveness.", "questions": "How is the smoothness degree of different classes (easy & hard) during the training?", "limitations": "Broader Impact : Potential positive societal impacts and negative societal impacts of the work is missing but its mentioned as included.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The empirical and theoretical analysis on the impact of the smoothness degree of soft labels on class-wise fairness is interesting and well documented\n-  Within the knowledge distillation framework, the use of temperature as a means of controlling the smoothness of the teacher's soft labels during training has proven to reduce the student's class-wise risk gap. Thus, improving the class-wise fairness.", "weaknesses": "- The evaluations on common corruptions is missing. (Does this method transfers robustness and fairness towards common corruptions as well?)\n- The evaluations on related fairness works, Wat [1] and FAT[2] is missing. \n- Most of the works mentioned as baselines focusing on fairness (CFA, BAT, FRL), in the paper are trained using PRN-18. May be better to evaluate the current approach using the PRN-18 as well (Optional).\n- The authors should explore if their method is also fair and robust in ViTs. The architectures suitable for training ARD and AT approaches are available in robustness benchmark [3].\n- Few ARD related evaluations  like IAD, MTARD  in combination with Fair-ARD approach as mentioned in Fair-ARD [4] needs to be evaluated. \n- Comparison of ABSLD with Fair-ARD in writing can be improved. How different is approaches from each other, what benefits ABSLD the most over Fair-ARD? \n\n\n[1] Li, B., Liu, W.: Wat: improve the worst-class robustness in adversarial training. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 14982–14990 (2023)\n[2] Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan Kankanhalli. Attacks which do not kill training make adversarial learning stronger. In International conference on machine learning, pages 11278–11287. PMLR, 2020.\n[3] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. \n[4] Xinli, Y., Mou, N., Qian, W., Lingchen, Z.: Revisiting adversarial robustness distillation from the perspective of robust fairness. NeurIPS (2023)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. We have taken great care to address all your concerns as follows:\n\n**Comment1: The evaluations on common corruptions is missing.**\n\n**Answer1:**\n\nWe have chosen two common corruptions: Gaussian noise and color channel transformation. As in Table 8 in the overall response PDF, ABSLD improves worst-class robustness by 0.9% and 1.8%, and reduces NSD by 0.012 and 0.01 on CIFAR-10 of ResNet-18 under those two attacks. The results show that ABSLD can transfer robust fairness towards common corruptions.\n\n**Comment2: The evaluations on Wat [1] and FAT[2] is missing.**\n\n**Answer2:** \n\nHere we add the comparison with WAT[1] and FAT[2]. As the following table shows, ABSLD improves worst-class robustness by 0.8% and reduces NSD by 0.03 compared with the second-best method on CIFAR-10 of ResNet-18 under AA attack.\n\nMethod  | AA Avg.(%) | AA Worst(%) | AA NSD \n:----: |:----: | :----: | :----: \n WAT | 46.19 | 30.20 | 0.286 \n  FAT | 41.83 | 16.80 | 0.409 \n**ABSLD(ours)** | **50.25** | **31.00** | **0.256** \n\n**Comment3: The application of PRN-18.**\n\n**Answer3:** \n\nWe select the trained model as ResNet-18 following Fair-ARD [3], which is also the common setting in Adversarial Robustness Distillation (e.g., ARD, IAD, and RSLAD). To further verify the effectiveness, we also train PRN-18 on CIFAR-10 via ABSLD and the results in the following Table demonstrate the effectiveness of ABSLD on PRN-18. \n\n| Method | AA Avg.(%) | AA Worst(%)  | AA NSD \n| :----: |:----: | :----: |  :----: \n| FRL  | 45.90 | 25.40  | -\n| CFA  | 50.03 | 26.50  | 0.301\n| **ABSLD(ours)** | **50.12** | **33.90**  | **0.263**\n\n**Comment4: fairness and robustness in ViTs.**\n\n**Answer4:** \n\nHere we select ViT-B to train following [4]. The result shows that ABSLD achieves better performance compared with the baseline method (RSLAD), so ABSLD is effective applied to ViTs but not limited to CNNs.\n\n| Method | AA Avg.(%) | AA Worst(%) | AA NSD \n| :----: |:----: | :----: | :----: \n| RSLAD  | **49.07** | 17.50 | 0.379 \n| **ABSLD(ours)** | 47.73 | **20.70** | **0.327** |\n\n**Comment5:IAD, MTARD in combination with Fair-ARD needs to be evaluated.**\n\n**Answer5:**\n\nFollowing your suggestion, we evaluate the Fair-ARD, Fair-IAD, Fair-MTARD, and Fair-RSLAD (different versions of Fair-ARD) for comparison with the ABSLD on CIFAR-10 of ResNet-18. As Table 13 in the overall response PDF shows, ABSLD improves worst-class robustness by 5.6% and reduces NSD by 0.049 under AutoAttack, which further demonstrates the effectiveness.\n\n**Comment6: Comparison of ABSLD with Fair-ARD in writing can be improved.**\n\n**Answer6:**\n\n**Fair-ARD exists two major distinctions with our ABSLD as follows:**\n\n1. From the optimization perspective, ABSLD redesign a new loss function by adjusting **different smoothness degree of soft labels** for different classes. Fair-ARD  modifies the existing loss function by adjusting **different weights** for different classes. \n\n2. From the designed method, ABSLD applies **the optimization error risk as a metric** to adaptively re-temperate the label smoothness degree for different classes. Fair-ARD applies **the least PGD steps for generating adversarial examples as a metric** to adaptively re-weight for different classes. \n\n**We think that the following advantage benefits ABSLD the most over Fair-ARD:**\n\nWe argue that the re-weighting method (e.g., Fair-ARD) and re-temperating method (our ABSLD) belong to different implementation paths to seek robust fairness.  **The re-temperating method is more direct and accurate than the re-weighting method.**\n\n**Specifically**, in the optimization process, the essential optimization goal is to reduce the loss between the model's predictions and labels. Re-temperating directly adjust the labels, and its effect can be directly and accurately reflected in the final optimization results of the model. While re-weighting adjusts the loss proportion for different classes, which indirectly affects the model's optimization goal.\n\n**In addition**, due to different implementation paths, **re-weighting and re-temperating methods will not conflict with each other.** Here we try to combine re-weighting and re-temperating strategies. As the results of Table 11 in the overall response PDF, we find that this combination can achieve better robust fairness compared with the re-temperating strategy, which demonstrates that **these two approaches will mutually promote the improvement of robust fairness.**\n\n**Comment7(question1): The smoothness degree of different classes.**\n\n**Answer7:**\n\nIn the training process, **the smoothness degree of the easy class will be smoother, and the smoothness degree of the hard class will be sharper.** In Figure 7 of the overall response PDF, we visualize the trend about the smoothness degree for easy classes (2-th) and hard classes (4-th) on CIFAR-10, which can further confirm our explanation. Here we measure the smoothness degree through information entropy, where the larger entropy denotes the smoother smoothness degree and the smaller entropy denotes the sharper smoothness degree. \n\n1.Wat: improve the worst-class robustness in adversarial training. AAAI(2023).\n\n2.Attacks which do not kill training make adversarial learning stronger. ICML(2020).\n\n3.Revisiting adversarial robustness distillation from the perspective of robust fairness. NeurIPS(2023).\n\n4.When adversarial training meets vision transformers: Recipes from training to archi tecture. NeurIPS(2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2fizZt2OMj", "reviewer": "Reviewer_6NLr", "summary": "This paper explores the issue of robust fairness in deep neural networks (DNNs), particularly focusing on the disparity in robustness between different classes in adversarial training (AT) and adversarial robustness distillation (ARD) methods. The authors propose a novel method called Anti-Bias Soft Label Distillation (ABSLD) that aims to mitigate this problem by adjusting the smoothness degree of soft labels for different classes during the knowledge distillation process. ABSLD adaptively reduces the error risk gap between classes by re-tempering the teacher's soft labels with different temperatures, which are determined based on the student's error risk. Extensive experiments demonstrate that ABSLD outperforms existing AT, ARD, and robust fairness methods in terms of a comprehensive metric that combines robustness and fairness, known as the Normalized Standard Deviation. The paper contributes to the literature by providing both empirical observations and theoretical analysis on the impact of soft label smoothness on robust fairness and by advancing a new technique within the knowledge distillation framework to achieve better adversarial robust fairness.", "questions": "Please help to check weaknesses.", "limitations": "Also in weaknesses", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.  Well-designed experiments: used (Tiny)-Imagenet,  set up AA as attack, and leverage NSD as metric", "weaknesses": "1.  In my humble opinion, I do not regard adversarial \"fairness\" is a critical problem. It is different from other fairness problem (e.g. demographic features) which brings social impacts.\n2.  For technical contribution, soft-label, knowledge distillation, as well as retemperature are well-known techniques. This paper applies the these techniques on the specific problem, but does not provide specific designs with respect to the problem.\n3. According to the experiment results, this method also suffers the trade-off between clean acc and robustness, which indicates not to solve the critical issue with respect to the adversarial training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. We have taken great care to address all your concerns as follows:\n\n**Comment1:In my humble opinion, I do not regard adversarial \"fairness\" is a critical problem. It is different from other fairness problem (e.g. demographic features) which brings social impacts.**\n\n**Answer1:**\n\nActually, we think **adversarial fairness problem is an issue worthy of being further studied in the field of adversarial robustness.** Previous research has always focused on improving overall robustness, however, the adversarially-trained models may exhibit high robustness in some classes while significantly low robustness in other classes. Due to the “buckets effect”, the security of a system often depends on the security of the weakest component. Specifically, an overall robust model appears to be relatively safe for model users, **however, the robust model with poor robust fairness will lead to attackers targeting vulnerable classes of the model, which leads to significant security risks to potential applications,** e.g., an autonomous driving system that has high robustness for inanimate objects on the road but lacks robustness when detecting pedestrians, may be misled by adversarial examples, leading to traffic accidents [6].\n\n Due to the importance, many studies have been published to solve this critical issue, e.g., FRL[1] (ICML2021), FAT[2] (NeurIPS2022), BAT[3] (AAAI2023), WAT[4] (AAAI2023), CFA[5] (CVPR2023), Fair-ARD[6] (NeurIPS2023), TAAL[7] (CVPR2024). **We believe that related research is necessary and will make a positive contribution to the secure application of AI.** \n\n**Comment2: For technical contribution, soft-label, knowledge distillation, as well as retemperature are well-known techniques. This paper applies the these techniques on the specific problem, but does not provide specific designs with respect to the problem.**\n\n**Answer2:**\n\nHere we mainly want to clarify the technical contribution of the proposed method. \n\nPrevious works always apply the re-weighting ideology to achieve robust fairness for different types of classes, however, as another important factor in the optimization objective function, the role of the labels has been ignored by previous researchers. Inspired by this, we try to explore robust fairness from the perspective of samples' soft labels.\n\nTo the best of our knowledge, **we are the first one to explore the labels' effects on the adversarial robust fairness of DNNs**, which is different from the existing sample-based perspective. We find that the smoothness degree of samples' soft labels for different types of classes can affect the robust fairness from both empirical observation and theoretical analysis. \n\nTo further enhance adversarial robust fairness, we propose a specific and novel method named ABSLD. Specifically, we re-temperate the teacher's soft labels to adjust the class-wise smoothness degree and further reduce the student's error risk gap between different classes.  The extensive experiments show our ABSLD can outperform other state-of-the-art methods in robust fairness problems. **We sincerely believe that our proposed method is innovative and effective.**\n\n**Comment3: According to the experiment results, this method also suffers the trade-off between clean acc and robustness, which indicates not to solve the critical issue with respect to the adversarial training.**\n\n**Answer3:**\n\nActually, **ABSLD is proposed to solve the adversarial robust fairness problem, which is distinct from the accuracy-robustness trade-off problem.** We believe that even if the overall robustness of a model is high, the poor robustness on a specific class of data can also pose security issues. In order to maximize the completion of the model's shortcomings, we focus on improving the model's robust fairness problem.\n\n1.Xu, H., Liu, X., Li, Y., Jain, A., Tang, J.: To be robust or to be fair: Towards fairness in adversarial training. ICML (2021).\n\n2.Ma, X., Wang, Z., Liu, W.: On the tradeoff between robustness and fairness. NeurIPS (2022).\n\n3.Sun, C., Xu, C., Yao, C., Liang, S., Wu, Y., Liang, D., Liu, X., Liu, A.: Improving robust fariness via balance adversarial training. AAAI (2023).\n\n4.Li, B., Liu, W.: Wat: improve the worst-class robustness in adversarial training.AAAI (2023).\n\n5.Wei, Z., Wang, Y., Guo, Y., Wang, Y.: Cfa: Class-wise calibrated fair adversarial training. CVPR (2023).\n\n6.Xinli, Y., Mou, N., Qian, W., Lingchen, Z.: Revisiting adversarial robustness distillation from the perspective of robust fairness. NeurIPS (2023).\n\n7.Zhang, Y., Zhang, T., Mu, R., Huang, X., & Ruan, W.: Towards Fairness-Aware Adversarial Learning. CVPR (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LWonye4bAr", "reviewer": "Reviewer_Vte2", "summary": "The paper introduces Anti-Bias Soft Label Distillation (ABSLD), a method aimed at improving robust fairness in deep neural networks. This paper identifies the smoothness degree of soft labels as a critical factor influencing this imbalance. ABSLD mitigates the robust fairness problem by adjusting the class-wise smoothness degree of soft labels during the knowledge distillation process. By assigning sharper soft labels to harder classes and smoother ones to easier classes, ABSLD reduces the error risk gap between classes. Extensive experiments on datasets like CIFAR-10 and CIFAR-100 demonstrate that ABSLD outperforms state-of-the-art methods in achieving both robustness and fairness.", "questions": "See the weaknesses above.", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed ABSLD method is innovative and provides a fresh perspective by focusing on the smoothness degree of soft labels, differing from the existing re-weighting approaches.\n2. The paper provides a theoretical analysis to support the proposed method, strengthening the validity of the claims.\n3. Extensive experiments on different datasets and models demonstrate the effectiveness of ABSLD.", "weaknesses": "1. The paper provides many implementation details, but some aspects, such as the choice of hyperparameters (e.g., learning rate, temperature adjustments), could be discussed more thoroughly. Clarifying these details can help replicate the results and understand the method's practical implications.\n\n2. Lack of more related works and SOTA methods, e.g.[1][2].\n\n3. What is the advantage of using Knowledge Distillation (KD)? The baselines used in the paper do not seem strong enough, and current state-of-the-art approaches [1][2] appear to offer better performance. Additionally, employing a teacher model increases the practical costs of time and memory.\n\n4. How about re-weighting + re-temperating? or using re-temperating directly via label smoothing without a teacher model.\n\n5. I am concerned that the comparison between ABSLD and Fair-ARD is due to the hyper-parameter used.\n\n\n[1] WAT: improve the worst-class robustness in adversarial training, AAAI 2023.\n\n[2] Towards Fairness-Aware Adversarial Learning, CVPR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. We have taken great care to address all your concerns as follows:\n\n**Comment1: The choice of hyperparameters (e.g., learning rate, temperature adjustments)**\n\n**Answer1:**\n\nFollowing your suggestion, we further discuss the hyper-parameter selection of the initial temperature learning rate $\\beta$ and the initial teacher’s temperature $\\tau_{k}^{t}$ on CIFAR-10 of ResNet-18. As Tables 9 and 10 in the overall response PDF show, the value of $\\beta$ and $\\tau_{k}^{t}$ can slightly influence the final results in a proper range, and the selection of hyper-parameter (initial $\\beta$ as 0.1 and initial $\\tau_{k}^{t}$ as 1) is reasonable. \n\n**Comment2: Lack of more related works and SOTA methods, e.g.[1][2].**\n\n**Answer2:**\n\nFollowing your suggestion, we compare our ABSLD with those two methods (WAT[1] and FAAL[2]). WAT[1] and FAAL are the re-weighting methods, which are different from our ABSLD (re-temperating method). Since FAAL does not provide reproducible open-source code, we directly apply the experimental results on CIFAR-10 of PRN-18 in FAAL’s origin paper. Following FAAL, we add the EMA operation to maintain consistency in the experimental setup. As the following table shows, our method outperforms FAAL by 1.02% and 0.2% in the average robustness and worst-class robustness under AA attack, which demonstrates the effectiveness.\n\n| Method  | AA Avg. (%) | AA Worst (%) | \n| :----: | :----: | :----: | \n| WAT  | 46.16 | 30.70 |\n|FAAL | 49.10 | 33.70 |\n| **ABSLD(ours)** | **50.12** | **33.90** |\n\n**Comment3: What is the advantage of using Knowledge Distillation (KD)? The baselines used in the paper do not seem strong enough, and current state-of-the-art approaches [1][2] appear to offer better performance. Additionally, employing a teacher model increases the practical costs of time and memory.**\n\n**Answer3:**\n\n**Firstly**, We think that applying KD has advantages as follows:\n\n1. **Adversarial Robustness Distillation (ARD) is a type of state-of-the-art adversarial training method currently.** ARD can effectively bring strong adversarial robustness for trained models within the framework of KD. Therefore, based on the impressive performance, we can maximally pursue approaches that bring both strong overall robustness and fairness just as in previous work [3].\n  \n2. **Knowledge Distillation itself can bring competitive robust fairness.** We can notice that KD-based methods themselves (e.g., RSLAD and AdaAD) have competitive robust fairness compared with the other baseline methods (e.g., TRADES, FRL, and CFA). We believe that the KD-based method is more friendly in improving the robust fairness of the model. So we select the Knowledge Distillation as the baseline.\n\n**Secondly**, the results in **Answer2** demonstrate the effectiveness of our ABSLD compared with WAT [1] and FAAL [2]. \n\n**Thirdly**, for ABSLD, the teacher model is only applied to generate the soft labels without updating its parameters, thus, the additional cost is limited. Meanwhile, we calculate the time cost and memory. Here we calculate the costs on CIFAR-10 of ResNet-18. As the following table shows, we think the costs of time and memory can be acceptable while bringing performance improvements compared with main-stream methods.\n\n| Method  | Time（Avg. Epoch）|\tGPU Memory  \n| :----: | :----: | :----:  \n| SAT  | 175s | 2764MiB \n| WAT | 284s | 3624MiB\n| ABSLD(ours)| 224s | 3832MiB |\n\n**Comment4: How about re-weighting + re-temperating? or using re-temperating directly via label smoothing without a teacher model.**\n\n**Answer4:**\n\nThank you for your valuable comment. Following your suggestion, we try to combine re-weighting and re-temperating strategies and evaluate the adversarial robust fairness. As the results of Table 11 in the overall response PDF, we find that re-weighting + re-temperating can achieve better robust fairness compared with the re-temperating strategy, which demonstrates that **these two approaches will mutually promote the improvement of fairness without conflicting with each other.**\n\nMeanwhile, we also try to explore the performance of using re-temperating directly via label smoothing without a teacher model and add the performance of label smoothing as the baseline for comparison. The results of Table 12 in the overall response PDF show that despite the re-temperating directly via label smoothing is better than the baseline, a certain gap still exists compared to the ARD-based method. \n\n**Comment5: I am concerned that the comparison between ABSLD and Fair-ARD is due to the hyper-parameter used.**\n\n**Answer5:**\n\nFor the Fair-ARD, we completely retain the hyper-parameter settings and select the best performance versions shown in Fair-ARD's original paper (Fair-ARD for CIFAR-10 and Fair-RSLAD for CIFAR-100). To further clarify the effectiveness, we compare our ABSLD with Fair-ARD's different version on CIFAR-10 of ResNet-18, including Fair-ARD, Fair-IAD, Fair-MTARD, and Fair-RSLAD, and the results of Table 13 in the overall response PDF show that our ABLSD has better performance compared with different types of Fair-ARD versions.\n\nEspecially, for the comparison between Fair-RSLAD and ABSLD, except for the specific hyper-parameters (e.g., the Fair-RSLAD’s for re-weighting or the ABSLD’s for re-temperating), the baseline (RSLAD) and its corresponding hyper-parameters are completely the same. **Thus, the effectiveness of ABSLD compared with Fair-ARD does not come from intentional hyper-parameter selection.**\n\n1.Wat: improve the worst-class robustness in adversarial training. AAAI(2023).\n\n2.Towards Fairness-Aware Adversarial Learning. CVPR(2024).\n\n3.Revisiting adversarial robustness distillation from the perspective of robust fairness. NeurIPS(2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jPYJKR9UFo", "reviewer": "Reviewer_Dt8R", "summary": "This paper aims to improve robust fairness in the adversarial distillation setting.\nThe proposed method adaptively assigns a smaller temperature for hard classes and a larger temperature for easy classes during the adversarial distillation training.\nThe smaller temperature means stronger supervision intensity for hard classes, realizing dynamic re-weighting in the training.\n\nExperiments on CIFAR100 and CIFAR-10 show some improvements in the worst-class performance at the cost of overall robustness degradation.", "questions": "(1) The proposed method has close relations with re-weighting.     \n      The authors should clarify the differences between the proposed method and the re-weighting. Why could the proposed method achieve better performance than re-weighting?\n\n(2) There seem to be some problems in the proof of theorem 1.\n     What's the relationship between the gradients and the error risk mathematically?\n\n(3) The proposed method can achieve better worst-class performance. However, the overall robustness degrades.", "limitations": "Limitations are discussed in the Appendix.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "(1) The proposed algorithm is easy to implement.    \n(2) Worst-class performance is enhanced at some cost of overall robustness.", "weaknesses": "(1) The proposed algorithm is to allow stronger supervision for hard classes with a smaller temperature during adversarial knowledge distillation. \nIt seems to have similar effects to re-weighting, i.e., assigning a smaller class weight on hard classes for the KL loss during the knowledge distillation.\nWhat's the difference between the proposed method and such a baseline in nature? Why could the proposed method achieve better results?\n\n(2) Though the proposed method improves the worst class robustness, the overall robustness can't be maintained.\n\n(3) Regarding the proof of theorem 1, \n      how do we get the conclusion in Eq. (19) with Eq. (18)? \n         From my point of view, the two terms should be the opposites of each other in Eq. (19).\n\n(4) In the proof, the assumption that \"the easy class error risk is less than the hard class error risk, so the gradient expectation of the easy class is higher than the gradient expectation of the hard class\" is used multiple times.\n          However, Why is it established?\n          What's the relationship between the gradients and the error risk mathematically?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. We have taken great care to address all your concerns as follows:\n\n**Comment1(weakness(1) or question(1))： What's the difference between the proposed method and re-weighting method? Why could the proposed method achieve better results?**\n\n**Answer1:**\n\nThe mentioned baseline (re-weighting) has been applied in Fair-ARD, while our ABSLD is a re-temperating method. We argue that the re-weighting and re-temperating methods belong to different ideology adopted to seek robust fairness. **The re-temperating method is more direct and accurate than the re-weighting method.**\n\nSpecifically, in the optimization process, the essential optimization goal is to reduce the loss between the model's predictions and labels. Re-temperating directly adjust the labels, and its effect can be directly and accurately reflected in the final optimization results of the model. While re-weighting adjusts the loss proportion for different classes, which indirectly affects the model's optimization goal.\n\n**In addition**, we think **re-weighting and re-temperating will not conflict with each other.** Here we try to combine re-weighting and re-temperating strategies. As the result of Table 11 in the overall response PDF, we find that this combination can achieve better robust fairness compared with the re-temperating strategy, which demonstrates that **these two approaches will mutually promote the improvement of robust fairness.**\n\n**Comment2 (weakness(2) or question(3))： Though the proposed method improves the worst class robustness, the overall robustness can't be maintained.**\n\n**Answer2:**\n\nActually, due to the “buckets effect”, the security of a system often depends on the security of the weakest component. In order to maximize the completion of the model's shortcomings, we focus on improving the model's worst-class robustness. Although the overall robustness remains unchanged or slightly decreases, **our ABSLD obtain the highest robust fairness compared with other methods.** Meanwhile, ABSLD shows the best comprehensive performance of fairness and robustness (NSD) compared with other methods, which means **obtaining the highest robust fairness with sacrificing the least average robustness.**\n\n**Comment3(weakness(3) or question(2))：Question toawrds Eq.(18) and Eq.(19)**\n\n**Answer3:**\n\nWe sincerely appreciate your careful and professional check towards Theorem 1. We apologize that some writing mistakes exists in Eq.(19). Actually, from the Eq.(18), we can obtain the above conclusion combined with Eq.(14):\n$$\n\\mathbb{E}(\\frac{\\partial KL(f(x_{c-};\\theta_{I}),P_{\\lambda1})}{\\partial z_{c-}(x_{c-})}) =\\mathbb{E}(\\frac{\\partial KL(f(x_{c+};\\theta_{I}),P_{\\lambda1})}{\\partial z_{c+}(x_{c+})})=\\mathbb{E}(p_{c}^{I}(x_{c}) - p_{c-}^{\\lambda1}(x_{c-}))=\\mathbb{E}(p_{c}^{I}(x_{c}) - p_{c+}^{\\lambda1}(x_{c+})),\n$$\n$$\n\\mathbb{E}(\\frac{\\partial KL(f(x_{c+};\\theta_{I}),P_{\\lambda1})}{\\partial z_{c-}(x_{c+})}) =\\mathbb{E}(\\frac{\\partial KL(f(x_{c-};\\theta_{I}),P_{\\lambda1})}{\\partial z_{c+}(x_{c-})})=\\mathbb{E}(p_{c}^{I}(x_{c}) - p_{c-}^{\\lambda1}(x_{c+}))=\\mathbb{E}(p_{c}^{I}(x_{c}) - p_{c+}^{\\lambda1}(x_{c-})).\n$$\nAnd the revised conclusion will not have more negative impacts on subsequent theoretical proofs.\n\n**Comment4(weakness(4) or question(2))：Unclear description about the assumption.**\n\n**Answer4:**\n\nWe apologize for the unclear description of this assumption. Here we further clarify it. Actually, the entire assumption is that “If the model is a uniform distribution before the optimization process and the easy class error risk is less than the hard class error risk after optimization process, then the gradient expectation of partial derivative about the easy class’s optimization goal toward the model parameter  is higher than the gradient expectation of partial derivative about the hard class’s optimization goal toward the model parameter”. \n\n**The mathematical explanation for this assumption is as follows:**\n\nIn the initial state, we assume that the model is a uniform distribution, and in this case, the error optimization risk is the same for the easy and hard classes:\n$$\n\\mathbb{E}(KL(f(x_{c-};\\theta_{I}),P_{\\lambda1}))=\\mathbb{E}(KL(f(x_{c+};\\theta_{I}),P_{\\lambda1})),\n$$\nthen we will simplify the optimization of the model into a one-step gradient iteration process, which means we only consider the initial state before optimization and the last state after optimization. The model parameter is updated from initial $\\theta_{I}$ to optimized $\\theta_{opt}$. Since easy classes perform better than hard classes after the optimization process, the easy class error risk is smaller than hard class error risk:\n$$\n\\mathbb{E}(KL(f(x_{c-};\\theta_{opt}),P_{\\lambda1}))>\\mathbb{E}(KL(f(x_{c+};\\theta_{opt}),P_{\\lambda1})),\n$$\nthe above result means the gradient expectation of partial derivative about the easy class’s optimization goal toward the model parameter ($\\mathbb{E}(\\frac{\\partial KL(f(x_{c-};\\theta_{I}),P_{\\lambda1})}{\\partial \\theta})$) is higher than the gradient expectation of partial derivative about the hard class’s optimization goal toward the model parameter ($\\mathbb{E}(\\frac{\\partial KL(f(x_{c+};\\theta_{I}),P_{\\lambda1})}{\\partial \\theta})$): \n $$\n \\mathbb{E}(\\frac{\\partial KL(f(x_{c-};\\theta_{I}),P_{\\lambda1})}{\\partial \\theta})>\\mathbb{E}(\\frac{\\partial KL(f(x_{c+};\\theta_{I}),P_{\\lambda1})}{\\partial \\theta}),\n $$\nso we can obtain the above assumption.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
This response contains mainly an overall response PDF with details as follows:

Figure 7: Information entropy change curve of teacher soft labels for hard classes and easy classes.

Table 8: Results on two common corruptions, for Gaussian Noise(GN) and Colour Channel Transformations(CCT).

Table 9: Discussion about different Initial $\tau_{k}^{t}$.

Table 10: Discussion about different Initial $\beta$.

Table 11: Comparison between reweighting, retemperating, and their combination.

Table 12: Comparison between Labelsmoothing, re-temperating via Labelsmoothing, and re-temperating via KD (ABSLD).

Table 13: Comparison between ABSLD and different Fair-ARD's versions.

Meta-review: The submitted manuscript addresses the problem of adversarial fairness, i.e. the problem that there is a severe trade-off between adversarial robustness and model fairness. The paper has received four reviews, three of which, after discussion, agree on the merits of the paper. Although the proposed approach recombines existing techniques to improve adversarial fairness, it shows overall promising results on an important topic. This is an overall sound paper which offers a step towards advancing an important field of research, adversarial fairness. This is different from the research question regarding the trade-off between clean and robust accuracy, but equally valuable.
Decision: Accept (poster)