FERD: Fairness-Enhanced Data-Free Adversarial Robustness Distillation
Authors: Zhengxiao Li
Liming Lu
Xu Zheng
Si Yuan Liang
Taric Chen
Yongbin Zhou
Shuchao Pang
Venue: ICLR 2026 Poster (2026)
Keywords: Data-Free Robustness Distillation; Robust Fairness
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Data-Free Robustness Distillation (DFRD) aims to transfer the robustness from the teacher to the student without accessing the training data. While existing methods focus on overall robustness, they overlook the robust fairness issues, leading to severe disparity of robustness across different categories. In this paper, we find two key problems: (1) student model distilled with equal class proportion data behaves significantly different across distinct categories; and (2) the robustness of student model is not stable across different attacks target. To bridge these gaps, we present the first Fairness Enhanced data-free Robustness Distillation (FERD) framework to adjust the proportion and distribution of adversarial examples. For the proportion, FERD adopts a robustness guided class reweighting strategy to synthesize more samples for the less robust categories, thereby improving robustness of them. For the distribution, FERD generates complementary data samples for advanced robustness distillation. It generates Fairness-Aware Examples (FAEs) by enforcing a uniformity constraint on feature-level predictions, which suppress the dominance of class-specific non-robust features, providing a more balanced representation across all categories. Then, FERD constructs Uniform-Target Adversarial Examples (UTAEs) from FAEs by applying a uniform target class constraint to avoid biased attack directions, which distribute the attack targets across all categories and prevents overfitting to specific vulnerable categories. Extensive experiments on three public datasets demonstrate that FERD achieves state-of-the-art worst-class robustness and NSD under all adversarial attacks. For instance, FERD improves worst-class robustness by up to 11.3% and reduces NSD by 0.077 compared to the optimal baseline on CIFAR-10 with MobileNet-V2. Our code is available at: [https://github.com/mayaobuduyao/FERD](https://github.com/mayaobuduyao/FERD).

Published as a conference paper at ICLR 2026
FERD: BAIRNESS-ENHANCEE DATA-FREE ADVER- SARIAL ROBUSTNESS DISTILLATION
Zhengxiao Li Liming Lu' Xu Zheng Siyuan Liang', Zhenghan Chen Yongbin Zhou', Shuchao Pang"
Nanjing University of Science and Technology 2HKUST(GZ)
IINSAIT, Soñia University, St. Kliment Ohridski "Nanyang Technological University SSCAA, Microsoft
{1i Plizhengxiao, Jtliminggeenjssteeuu..
ABSTRACT
Data-Free Robustness Distillation (DFRD) aims to transfer the robustness from the teacher to the student without accessing the training data. While existing methods focus on overall robustness, they overlook the robust fairness issues, leading to severe disparity of robustness across different categories. In this pa- per, we find two key problems: (1) student model distilled with equal class pro- portion data behaves significantly differently across distinct categories; and (2) the robustness of student model is not stable across different attacks targets. To bridge these gaps, we present the first Chireess-Enaanee Robustness Distillation (FERD) framework to adjust the proportion and distri- bution of adversarial examples. For the proportion. FERD adopts robustness- guided class reweighting strategy to synthesize more samples for the less robust categories, thereby improving robustness of them. For the distribution, FERD generates complementary data samples for advanced robustness distillation. It generates Fairness- Aware Examples (FAEs) by enforcing a uniformity constraint feature-level predictions, which suppress the dominance of class-specific non- robust features, providing a more balanced representation across all categories. Then, FERD constructs Uniform-Taree FAES by applying a uniform target class constraint to avoid biased attack direc- tions, which distributes the attack targets across all categories and prevents fitting to specific vulnerable categories. Extensive experiments on three public datasets demonstrate that FERD achieves sfate-of-the-art worst-class robustness and NSD under all adversarial attacks. For instance, FERD improves worst- class robustness by up to 11.3% and reduces NSD by 0.077 compared to the optimal baseline on CIFAR-10 with MobileNet-V2. Our code is available at:
data-free adversarial
Adversarial Examples (UTAEs) from
INTRODUCTION
With the widespread use of Deep Neural Networks (DNNs) (Goswami et al. 2018 Gongye et al.) 2024
Lim et al. 2024). the deployment of lightweight models on edge devices has become increas- ingly important (Zhu et al. 2025 Pang et al.) 2025: Liu al. 2024). However, large number of studies have shown that these ightweight models are weakly robust in the face of adversarial
attacks (Ma et al. 2021 Li et al.
2022 Croce & Hein 2020 Bai al. 2023). Traditional adver- sarial training methods Jia et al.
2022; Hsiung et al. 2023 Jia et al. 2024b a). though showing
significant advantages on large models, are difficult achieve desirable results in lightweight mod-
els (Xia et al. 2025 Wang et al. 2024b Ye et al. 2019 Huang et al. 2021). To enhance the defense capability lightweight models, researchers have proposed the concept adversarial robustness
distillation (Zi et al. 2021
Zhu et al. 2021 Huang et al. 2023 Yue et al. (2024] Zhu et al. 2023
"Equal contribution.
Corresponding authors.

---

Published as a conference paper at ICLR 2026
Lu et al. 2025 Gu et al. 2025), which aims to migrate the defense capability of the robust teacher to the student, thereby enhancing the latter's ' performance in an adversarial setting.
However, in practice, raw training data for the teacher model are often unavailable, making it diffi- cult to apply traditional distillation methods directly. To break through this limitation, researchers propose the Data-Free Knowledge Distillation (DFKD) method (Micaelli & Storkey 2019 Fang et al. 2021: Yin et a 2020 Fang et al., 2022). which synthesizes alternative samples through a training generator to simulate the original data distribution and achieve knowledge transfer. Based on this, the Data-Free Robustness Distillation (DFRD) method (Yuan et al.) 2024 Wang et al.) 2024a] Zhou et al.] 2024) is further developed to combine the generation and distillation mecha- nisms to effectively deliver robustness without the need for real data, providing a novel solution for resource-constraneed and anavaialval environments for model defense.
Despite their effectiveness, existing DFRD methods mainly aim to enhance the overall robustness of the student, while overlooking a critical issue: robust fairness (Sun et al. [2023] Yue et al. [023]
Zhao et 2024). The robust model may exhibit strong resistance to adversarial attacks on specific categories and remain vulnerable in others, leading to inconsistent robustness performance across different categories. It impacts the reliability and fairness of the model in practical applications.
In this paper, we make the first attempt at investigating the robust fairess problem in the context of DFRD. We find that although the student tends to inherit the teacher's class-wist robustness pattern, the inter-class robustness gap is significantly amplified in the distillation process. We find two phenomena affecting robust fairesss: (1) students distilled with equally distributed synthetic data still show class-wise robustness discrepancies; and (2) the success rate of adversarial attacks on students varies significantly depending on the target class. Based on these findings, propose a Fairness-Enhancee data-free adversarial Robustness Distillation (FERD) framework by adjusting the proportion and distribution of the synthetic samples to mitigate these problems.
Specifically, for the proportion, we introduce robustness-guide class reweighting strategy that en- courages the generator to synthesize more samples from weakly robust categories, thereby compen- sating for their vulnerability and promoting fairness in robustness. For the distribution, we propose to generate complementary data samples for advanced robustness and fairness distillation. Firstly, we generate Fairness-Awan Examples (FAEs) by enforcing a uniformity constraint on feature-level predictions that are closely associated with non-robust representations. This helps suppress the dominance of class-specific non-robust features, ensuring that the benign samples provide a more balanced representation across all classes. To avoid biased attack directions, we further construct Uniform- Target Adversarial Examples (UTAEs) from FAEs by applying a uniform target class con- straint during adversarial generation. It distributes the attack targets evenly across all categories and prevents overfitting specific vulnerable categories.
-arreet-Tarret
Experiments on the three datasets (CIFAR- 10, CIFAR-100, and (iny-ImaeNNet show that compared with baseline methods, FERD improves 11.3% 5.7%, 6.2% and 6.6% the worst-class robustness against FGSM (Goodfellow et al., 2014), PGD-20 (Madry et al. 2017), CW oo (Carlini & Wagner) 2017), and AutoAttack (AA) (Croce & Hein 2020), respectively, alleviating the robustness bias problem in the DFRD to certain extent.
Our contributions can be summarized as follows: (I) We make the first attempt investigating the problem of robust fairness in DFRD, revealing that uniform distribution among original data categories and attack target bias are the two key factors affecting fairness. (II) We propose the FERD framework, which enhances robust fairness at both the proportional and distributional levels through robustness-guided category reweighting and distribution-awarr sample generation mechanisms. (III) Experiments have demonstrated that FERD significantly enhances the robustness and fairness of the student model, and the robust accuracy the weakest class has been improved by 11.3% compared with the existing optimal methods, effectively alleviating the robust bias phenomenon in DFRD.
2 RELATED WORK
2.1 DATA- ROBUSTNESS DISTILLATION
DERD aims to transfer the robustness from teacher to student using synthetic samples instead of teacher's original training data. DFARD (Wang al. 2024a) first defines the concept of DFRD and
2

---

Published as a conference paper at ICLR 2026
Clean Accuracy
Robust Accuracy FGSM Robust Accuracy PGD-20
Robust Accuracy CW
Robust Accuracy AA
90
70
80
50
40
20
8
8 Fast
2
6
Teacher
25KT
--
CMI
Deepinv
DERD
DFARD
DAHL
Figure 1: Comparison of the accuracy the teacher and the students distilled from different DFRD methods under benign and adversarial examples. The blue line represents the teacher and the other lines correspond to the students. The horizontal axis indicates the category, and the vertical axis indicates the accuracy.
proves that the difficulty lies in the lower upper bound of knowledge transfer information. They propose an interactive temperature adjustment strategy and an adaptive generator to solve the prob- lem. DERD (Zhou et al., 2024) takes a homogenized expert guidance strategy. Both clean and robust knowledge are distilled from clean and robust teachers respectively, using the same synthetic data. To coordinate the gradients of the clean and robust distillation tasks, DERD also introduces a stochastic gradient aggregation module, thereby optimizing the trade-off between robustness and accuracy. DFHL Yuan et al. 2024) proposes the concept of High-Entrops Examples (HEEs), which can characterize more complete shape of the classification boundary. Distillation on HEES achieves the best balance between clean accuracy and robustness. is worth mentioning that although DFKD does not involve robustness, we can transform it into DFRD by adding adversarial noise to synthetic samples during its distillation stage. For example, Fast (Fang et al. 2022) proposes fast DFKD, which reuses the common features in the training data to synthesize different data instances. By generating adversarial examples on synthetic samples, a distilled robust student can be obtained. In this paper, we make the first attempt at solving the problem of robust fairness transfer in DFRD.
2.2 FAIRNESS IN ROBUSTNESS
While enhancing overall model robustness is common goal. it inevitably leads to significant class- wise performance discrepancies: models become highly robust for remain vulnerable to adversarial attacks. FRI (Xu et al. 2021) is a pioneer work in highlighting this issue and introduces the concept of "robust fairness" to assess such class-wise robustness disparities. To address this issue, FRL proposes fairness constraints, adjusting decision margins and sample weights when these constraints are violated. BAT (Sun al., 2023) identify distinct "source-class" and "target-clase ccaas" unfairness within adversarial training, tackling these by adjusting per-class attack intensities and applying uniform distribution constraint. Further methods include those by Fair- ARD Yue et al. (023). who improves student model robust fairness by increasing the weights of difficult classes, and ABSLD (Zhao et al., 2024), who focuses on adaptively reducing inter-class error risk gaps by modulating the class-wise smoothness of samples' soft labels during training. In this paper, we introduce FERD to simultaneously enhance model robustness and alleviate robust unfairness problems.
categories while others
OBSERVATION OF FAIRNESS IN DFRD
In this section, we investigate whether robust fairness of the teacher is transferred the student distilled from DERD methods. We research the robust fairness performance of the models and defense effects against adversarial attacks from different target categories.
3.1 ROBUST FAIRNESS OF THE STUDENT
To evaluate the robust faimess of the student, compare the classification performance of the teacher and the students distilled by five methods (ZSKT (Micaelli & Storkey 2201)). CMI (Fang et al. 2021), DeepInv (Yin et al. 2020), Fast (Fang al. 2022), and DFHHI (Yuan et al. 2024
3

---

Published as a conference paper at ICLR 2026
FGSM
PGD-20
CW
AutoAttack
DAS
21
AM
17
20
134
True Label
Labe
HUU Label
THU
1
4:37
Predicted
Predicted Label
Predicted Label
Predicted
Confusion matrices under different adversarial attacks. The horizontal axis denotes pre- dicted labels, and the vertical axis denotes true labels. Darker colors indicate a higher number of samples predicted as the correct class.
Figure 2:
different classes. We measure the accuracy of each category under benign samples and four adver- sarial attacks (FGSM, PGD-20, CW ce, and AA). The results in Fig | show that the performance of the student on different categories follows consistent trend with that of the teacher. Note that the sample labels all adopt a uniform sampling strategy in the aforementioned DFRD methods, in which case the student conducts robust distillation on equally distributed synthetic data. However, the ro- bustness among different categories varies significantly. Therefore, we argue that sample number imbalance the key to achieving class balance, because categories behave differently in robust dis- tillation. Some categories are more difficult to achieve robustness, while others quickly reach a high level of robustness. We conjecture that by adjusting the weight of class sampling and increasing the number of samples from weakly robust classes, the problem of robust unfairness can be effectively alleviated. The proof of this conjecture shown in Appendix A.
3.2 TARGET FAIRNESS OF ADVERSARIAL EXAMPLES
We make confusion matrices to visualize the classification results, as shown in Fig 2 that the students' robustness against adversarial attacks different targets varies. For samples with the original label of class 0, when the attack target is class 9, the student more prone to misclas- sification, showing poor robustness; while for adversarial attacks on other classes, its robustness is relatively strong. This indicates that the student exhibits significant differences in defending against adversarial attacks on different targets and is more vuinerable specific class adversarial attacks. We conjecture that robust distillation on adversarial examples with uniformly distributed targets can defend against attacks from different targets. The proof of this conjecture shown in Appendix
We observe
A.2 A.2
METHODOLOGY
4.1 OVERALL FRAMEWORK
In the above section, we find two factors affecting robust fairness: (1) the student distilled on data with equal class proportions shows class-wise robustness discrepancies; and (2) the success rate of adversarial attacks on the student varies significantly depending on the target class.
In this section, we introduce our FERD framework and the overall framework is in Fig The algo- rithm is shown in the Appendix(B B For the first proportion problem, we introduce a trobustess--uudde class reweighting strategy that encourages the generator to synthesize more samples from weakly robust classes. For the second distribution problem, we impose uniform constraints on the feature space of synthetic samples and the generation direction of adversarial examples, respectively, mak- ing the targets of adversarial examples more uniformly distributed. Specifically, we firstly generate Fairness-Awand Examples(FAEs) by enforcing constraint on predictions from non-robust feature, which are highly related to adversarial targets. Then, to avoid biased attack directions, we further construct Uniform- Target Adversarial Examples (UTAEs) from FAEs by applying constraint during adversarial generation, preventing attacks focus on "vulnerable category".
4

---

Published as conference paper at ICLR 2026
Phase 7: Generation Stage
Phase 2: Robustness Distillation Stage
Feature Space
Random
Reweighted closs label sompling
do
Attack Direction
optimine
SIANS SONS
Geeneraton
Refon Ma Coo
Festure
SAEEN S
popkimice
Teacher
Oett Ststibttinn
Soudené
Trainable
Frozen
Nantrobust Feature
UTAEs
Figure Framework of our FERD. In the generation stage, we use robustness-guided class reweighting strategy synthesize more weak-class samples and apply uniformity constraint to non-robust feature predictions to generate FAEs. During robust distillation, we construct UTAEs from FAEs, using them respectively benign samples and adversarial examples.
4.2 CLARS-AWARE SAMPLE REWEIGHTING FAIR DISTILLATION
In traditional DFRD, the labels of synthetic samples are sampled from uniform distribution, as y4 2(00, - 1), where Ui means uniform distribution and C means class number. However, robust distillation equally distributed data infers significant robustness unfairness problem.
To mitigate this problem, we introduce the robustness-guiide class reweighting strategy based on adversarial margin, aiming to guide the generator to synthesize more samples of categories with poorer robustness. Specifically we first generate adversarial examples xt from the synthetic samples 2; using PGD-20 attack. Then, calculate the adversarial margin m, of each adversarial sample 2ndu under the teacher model f":
m; (f (xfd")) max (f 3733
The adversarial margin measures the gap between the model's confidence in the correct category and the strongest confusable category. The smaller the value, the more probably 2:dd is to be misclassified. A negative margin indicates that the attack has been successful.
Therefore, we calculate the average negative adversarial margin for each category:
Dc
Σ (-m4), No =y
1
where N₂ means the number of samples in category This value is used to measure the robustness vulnerability of the category. higher value indicates that the category more likely be misclassified when facing adversarial attack.
Finally, we apply softmax function to transform into sampling probability distribution Po: which is used as the sampling weight for categories in the subsequent sample generation stage, enabling to adaptively generate more samples of less robust categories. Furthermore, we compare several other reweighting strategies. The experimental results are shown in Appendix
4.3 NON-ROBUST FEATURE SUPPRESSION FOR BALANCED REPRESENTATIONS
To achieve the fair tendency to each target when generating adversarial examples, we design a FAEs generation method and them benign samples, which ensures that FAEs" non-robust feature
5

---

Published as a conference paper at ICLR 2026
predictions not concentrated in a few categories; otherwise, adversarial perturbations would be more likely to attack such categories.
We adopt modified information bottleneck approach to achieve this. The standard information bottleneck seeks to get a compressed representation Z of the input X that is maximally informative about the target label Y. The objective function can be written as follows:
LIB I(Z;Y) (I;
(3)
where Z denotes the mutual information and balances the trade-off between prediction accuracy and compression. We attempt to distill non-robust features Zur from the intermediate feature repre- sentation Z ff (x), where f (-) describes /-th layer outputs of the teacher.
When the synthetic samples z, imput into fr, we inject learnable noise scale X, into and define informative features Z₁ as follows:
Zi = ff (z;) + softplus(A,) . c, N(,
(4)
After obtaining 2 added by X, we propagate Z₁ to the subsequent output layer f7+ and evaluate the influence each unit's feature to the teacher prediction. When distilling Znr from Z,, we must en- sure Zi remains predictive while encouraging robustness to noise. Since directly optimizing mutual information intractable, we use a variational approximation (Kim et al. 2021). The optimization objective is formulated
min C(2,) - CE(f (2)) (C +
Channel Σ
Vc
+ log
A Tc
-1).
(5)
X
where UP and Å denote the e-th channel of Z, and Xy respectively. The first term ensures that Z correctly predicts the target label yi, while the second term acts regularizer control the amount of information passing through the bottleneck.
After optimizing Aye we identify non-robust channels index inre by comparing X with the maxi- mum variance across all channels:
fure
X2
max d'E(1....c)
{Var (zi)
The right-hand side represents the upper limit of perturbation. Correspondingly, non-robust features Zur obtained via channel-wise masking: Zur = inr Z. The prediction results of Znr vulnerable to perturbations, so that they are highly correlated with adversarial predictions. To enable the generator synthesize FAEs, minimize the K T. divergence between the predictions of non- robust features and the uniform distribution:
Luni KLU,, FI. (Zwr)).
To further enhance the quality and diversity of the FAEs, we introduce additional loss functions during the training process of the generator:
Ladv = KL (oo)) (x:))
Lmn = E, (11((();)) -
+ [lo{(x.) - -ff1))
(8)
Loh = CF
where Lade encourages divergence between the student and teacher, promoting the diversity of FAEs; Lbn improves the visualization of the FAEs by matching the statistical information (mean in and variance of) stored in BatchNorm layers of the teacher and the student; Cak. ensures that the FAEs are correctly predicted by the teacher.
Therefore, the overall loss function for the generator in the generation stage are summarized as:
Egen Xado Lado + Abn Lim + Aoh Loh + Auni Luni,
where these hyperparameters Xados Abns Aoh and Auni adjusted empirically balance the trade- offs between robustness, fairness, and accuracy. The hyper-paramett selection experiments are shown in Appendix FF] By training with the above loss function, the generator synthesizes FAEs Fy, which not only convey effective knowledge, but also be fairer in terms of the tendency towards different categories when generating adversarial examples.

---

Published as a conference paper at ICLR 2026
Table 1: Result of average robustness(%) (Avg. 1), worst robustness(%) (Worst 1), and normalized standard deviation (NSD 1) on CIFAR-10. RN-18 and MN-V2 are abbreviations of ResNet- 18 and MobileNet- V2 respectively. The best results are bolded, and the second best results are underlined.
Student
CW sc
Avg.
Worst
NSD
Avg.
Worst
NSD
Avg.
Worst
NSD
Avg.
Wonst
NSD
Arg.
Wonst
NSD
ZSKT
65.48
32.20
0.266
29.80
5.00
0.563
22.85
3.40
0.624
18.48
2,60
0.696
17.16
2.20
0,740
0.161
33.34
0.337
27.63
26.85
0.419
Decplav
71.22
58.90
0.124
38.60
22.50
0.303
27.95
15.00
0.575
27.23
14.50
0.379
25.19
11.60
0.122
Fast
72.20
57.80
0.132
43.96
26.90
0.295
39.64
20.10
0.339
33.96
18.40
0.388
33.24
17.70
0.401
RN-18
DHRD
78.91
53.70
0.158
39.86
22.90
0.341
30.55
13.90
0.417
29.98
13.00
0.421
28.61
0.449
DFARD
67.93
45.90
0.247
41.27
18.40
0,458
37.34
15.20
0.491
35.47
13.30
0.513
34.48
12,00
0.552
DFHL
74.42
58.60
0.129
41.67
24.30
0.298
36.99
19.30
0.344
37.33
17.50
0. 0.348
16.39
18.50
0.351
FERD
79.26
0.105
51,09
0.237
42.24
20.40
0.331
41.73
19.30
0.339
40.12
20.80
0.325
ZSKT
54.69
16.10
0.394
23.62
3.90
0.663
19.83
3.00
0.702
15.86
1.90
0.725
14.85
1.60
0.749
CMI
60.70
38.60
0,205
27.90
12.50
0.393
19.70
8.50
0.482
19.57
8.10
0.461
18.09
7.60
0.506
Deeplav
62.77
46,00
0.170
31,66
14.50
0.306
19.13
7.00
0.403
19.01
6.90
0.407
15.39
5.40
0.471
MN-V2
Fast
58.60
0.198
29.91
0.374
24.28
19.46
0.535
18.35
0.563
DERD
69.80
48.70
0.175
39.99
22.10
0.529
28.82
15.10
0.572
27.97
14,00
0.386
25. 80
12,00
D.403
DEARD
66.32
10.10
0.273
36.71
12.60
0.519
33.84
9.20
0.562
32.76
8.70
0.575
31.81
8.10
0.584
DFHL
20.83
50.60
0.153
37.22
0.320
33.27
0.358
33.60
0.353
13.20
0.368
HERD
78.34
64.10
0.108
53.04
33.40
0.229
41.63
20.80
0.304
41.43
20.50
0.314
38.06
20.30
0.349
4.4 LABEL--SPACE ATTACK DIVERSIFICATION FOR FAIRNESS OPTIMIZATION
To address the problem that the student shows significant differences in defending against adversarial attacks with different targets and is more vulnerable to them from specific categories, we propose novel adversarial examples generation method, named UTAES generation.
We aim to construct a more evenly distributed type of adversarial perturbation, so that the adversarial targets are not limited to certain "easily misclassified" classes, but uniformly cover the entire class space. To achieve this, apply uniform target class constraint during adversarial generation, avoiding to attacks from a single direction. The generation formula is as follows:
x = IIlu+ (at sign (v+t,
- KL (U, fT (ab)) (10)
where V Val, denotes the gradient of the entropy loss function w.r.t the UTAE in step and is the step size. By introducing the KL divergence between f (x(x)) and in the adversarial example generation, we make the target distribution of adversarial examples more extensive.
After synthesizing FAES Fr and UTAES zV. we employ them as benign samples and adversarial examples respectively for robust distillation. Here, we adopt a decoupled knowledge distillation loss (Cui et al. 2024) instead of the traditional KL divergence:
(f (xx))
1
S(w,) (AF"(z)) ((wn))A A)
4
(1I)
- -s( (f log so coslmar(]] (xx)),
where S()) denotes stop gradients operation, Wy represents the class-wise global information weight corresponding to the ground ground-truth label Af (z)) and f represent pairwise difference ma- (2:)2, o (xx)g.A = (xx); f (Ix)s. The loss function is composition of a wMSE loss and Cross-Entrop loss. The first term minimizes the struc- tural difference between the teacher's and student's logits, and the second term aligns the student's 's predictive distribution with that of the teacher. The decoupled formulation breaks the asymmetric optimization property and performs better knowledge distillation and adversarial training.
trices. A (x.) = f"((x))
Therefore, the robust distillation framework is as follows:
Letu = AILI ((T(F)) + X2CIKE (
(12)
Through robust distillation of diversified adversarial targets, force the student to inherit robust- ness across a far broader range of categories, thereby enhancing overall defensive capability against attacks from all directions.
7

---

Published as a conference paper at ICLR 2026
Table Result of average robustness(()) (Avg. t). worst-10% robustness(%) (Worst 1), and normal- ized standard deviation (NSD 1) on CIFAR-100 100. RN-18 and MN-V2 are abbreviations of ResNet- 18 and MobileNet- V2. The best results are bolded, and the second best results are underlined.
CW
Avg.
Warst
NSD
Avg.
Worst
NSD
Avg.
Worst
NSD
Avg.
Wonst
NSD
Avg.
Wonst
NSD
ZSKT
36.44
6.40
0.523
12.69
0.10
0.985
10.17
0.00
1,083
7.12
0.00
1.273
6.01
0.00
1.361
51.60
0.295
0.624
22.63
0.686
18.94
0.824
Decplav
53:14
27.20
0.298
26.14
3.60
0,630
23.11
3.60
0.699
19.05
2.20
0.792
17.49
1.60
0.857
Fast
50.69
23.50
0.323
26.45
5.50
0.634
24.13
4,50
0.687
19.44
2.20
0.806
18.24
1.70
0.859
RN-18
DH8D
$4.01
0.295
0.631
24.78
0.689
22.96
0.776
0.85
DFARD
50.43
21.80
0,340
26.44
4,80
0.644
24.22
3.90
0.693
21.56
2.50
0.781
20.21
1.50
0.828
DFEL
46.27
18.70
0.370
18.73
2.60
0.799
15:72
1.30
0.933
16.33
1.50
0.896
15.23
1.10
0.954
FERD
55,91
29.00
0.289
29.23
0.622
26.10
0.681
23.39
0.754
21.79
0.805
ZSKT
30.79
2.70
0,650
13.66
0.975
12:44
0.00
1.007
9.54
0.00
1.124
8.62
0.00
1.208
CMI
35,80
6.00
0.425
13.82
0.60
0.828
9.91
0.40
0.966
9.06
0.00
L.000
8.04
0.00
1.042
Deeplav
38.10
8.00
0.447
13.97
1.00
0.920
9.98
1.00
1.123
8.12
0.00
1.260
6.67
0.00
1.416
MN-V2
DERD
Fast
54.12
29.10
0.291
0.441
26.31
5.40
0.646
22, 39
3.00
0.913
0.742
20.59
2.70
0.774
1.085
18.92
9.53
1.70
0.852
DEARD
49.02
20.60
0.310
24.62
0.668
22.43
3,10
0.723
20.65
2.10
0.778
19.30
1.60
0.826
DFHL
42.08
11.30
0,436
16.25
0.914
13.55
1.015
14.06
0.960
12.99
HERD
56.14
29.20
0,285
29.63
6.10
0.611
24.48
3.10
0.713
23.20
3.30
0.743
21.20
2.30
0.793
6:35
70
0.35
0.30
0.30
6.25
R:bb Roburtee anrrenn
0.5
8
0.20
0.20
6.35
0.55
8
6.30
0.10
22
0.05
Aug. RCD
We ght
Aeg. RoD
Weight
(1)
(b)
Figure 4: (a): The robustness of the student for each category under equal weight. (b): The ro- bustness of the student for each category under reweighting situation.
Figure 5: t-SNE plot for illustrating the FAEs
representations in feature space.
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Datasets & Models. We conduct our experiments on three datasets: CIFAR-10 (Krizhevsky et al, 2009). CIFAR-100, and Kiny-ImageNe (Le & Yangl 2015). For the teacher model, we select NideResNeet33-100 (Zagoruyko & Komodakis] 2016) for both CIFAR-10 and CIFAR-100, while Nee-44ResNet-34 is used for Tiny-ImageNet. For the student model, select ResNet-18 (He al.) 2016ab and MobileNet-v2 (Sandler et al., 2018)) for CIFAR-1O and CIFAR-100 For ImageImageNet, use PreActResNet-18. The performance of the teacher and the experiments on Tmag-em are shown in Appendix C and
Baselines. We compare our method with seven different DFRD methods, including ZSKT (Micaelli & S 2019). CMI (Fang et al., 2021). DeepInv (Yin et al. 2020). Fast (Fang et al. 2022).
DERD (Zhou et al. 2024). DFARD (Wang al. 2024a) and DFAL (Yuan et al.) 2024). Note
that the first four methods belong to DFKD initially and do not involve robustness. We generate
adversarial examples and then apply the same distillation training loss as RSLAD (7i et al. 2021)
to transform them into DFRD.
Implementation Details. Our proposed method and all baselines are implemented on NVIDIA
A800 GPU. The generator is trained via Adam optimizer with learning rate of 2e-3, 31 of 0.5. 32 of 0.999. The student trained via SGD optimizer with an initial value of 0.1, momentum of 0.9, and weight decay of 5e-4. The distillation epochs is set to 220 and the training iterations of generator and student are 200 and 400 respectively. The batch size is set to 256 for CIFAR- 10 and 512 for both CIFAR- 100 and Imageee. In Eq we extract the intermediate features from the
8

---

Published as a conference paper at ICLR 2026
Table 3: Result of average robustness($)
(Avg. 1), worst robustness( %) (Worst t) on CIFAR-I0. The teacher model is WedeResNet- 34-20 and the student model is ResNet-18.
(n) 7SKT
(b) CMI
Clean
AA
Method
(e) Deeplin
(d) Fast
Avg.
Worst
Avg.
Worst
ZSKT
50.14
14.20
3.39
0.20
CMI
73.26
57.90
26.44
9.80
DERD
DeepInv
69.47
56.20
24.80
13.40
Fast
68.95
53.30
30.28
14.70
DERD
69.00
45.70
33.50
15.10
DFARD
70.81
49.10
34.38
14.60
(g) DFEL
(b) FERD
DFHL
70.89
52.60
36.59
15.60
Figure 6: 32x32 images generated by inverting a WideResNet-34-11 trained on CIFAR-10.
FERD
78.23
61.70
38.67 16.00
last convolutional layer L. In Eq 9 Aos=1, Amn:=5, 7=0.5, =565 and 22=176
Eq 10 and Eq 12 we set the hyperparameter Aado=1, A6n=5,
Evaluation Metrics. We evaluate the robustness of the student against four adversarial attacks: FGSM, PGD, CW co. and AA. Following (Yue et al. 2023 Zhao et al.) 2024), we employ the worst- class robustness and Normalized Standard Deviation (NSD) to quantify the robust fairesss across categories. NSD is a normalized metric of the adversarial robustness with respect to the standard deviation across different classes. The smaller the value of NSD, the better. Notably, for CIFAR-100 and Imageentt,N we adopt worst- robustness in place of worst-class robustness, due to the limited size of test set per category and poor performance the worst class robustness.
5.2 EXPERIMENTAL RESULTS
Overall performance. Tab. T and Tab. 2ab.2 show the performance of ResNet-18 and MobileNet-V V2 distilled CIFAR-10 and CIFAR-100 by method and baselines. The results demonstrate that the student distilled from FERD has an improvement in the robustness and fairness. Our method achieves state-of-the--ee worst-class robustness under all attacks. When distilled on CIFAR-10 with MobileNet-V2 V2, FERD improves the worst class robustness by 11.3%, 5.7% %, 6.2%, and 6.6% com- pared with the best baseline against four adversarial attack respectively. In addition, there is an improvement in the average accuracy and NSD in most situations. Specifically, as shown in Tab. [ FERD achieves the highest average accuracy under all attacks with improvement of up 13.05%. which is attributed to our high-quality synthetic samples.
Effectiveness of reweighting. Fig. illustrates the impact of reweighting sampling strategy on the robustness of the student model. The robustness-guidde reweighting strategy we proposed effec- tively identifies the categories with poor robustness and increases their sampling weights. Specifi- cally, the weight of the fourth class which has the lowest robustness reaches 0.314 exceeding other classes substantially. Correspondingly, the student's robustness in these categories are improved, thereby enhancing robust fairness.
Synthetic data visualization. In Fig. we visualize the synthetic data generated by FERD and baselines. The results indicate that generator is able to synthesize more visible data. In such scenarios, CMI and Fast even suffer from model collapse problem, where the visual quality of the synthetic samples is extremely low. This further demonstrates the superiority of our method, which recovers high-quality samples from the robust teacher. In Fig. 5 we find that the points of different colors are clearly distributed in different areas, indicating that the class distinction is excellent.
Distillation with different teacher. To verify the transferability our method, we select the WRN- 34-20 model as the teacher model for experiments. As shown in the Tab. FERD still outperforms

---

Published as a conference paper at ICLR 2026
Table 4: Ablation study of different components in the framework.
43.0
B1
24
80
5
61.5
Clean
AA
Method
40.5
Avg.
Worst
Avg.
Worst
40.0
0.1
0.7
15
6.1
05
05
FERD
79.26
65.90
42.24
20.80
ACC
(b)
w/o reweighting
79.53
63.20
42.54
17.20
w/o FAEs
78.42
65.30
41.01
19.90
Figure 7: Ablation study. (a): The average ro-
bustness and accuracy of the student under dif-
w/o UTAES
77.57
65.10
41.24
20.00
ferent y. (b):The worst-class robustness and ac-
w/o FAES UTAES
77.48
62.50
40.92
20.30
curacy of the student under different 7.
all baselines when distilled from the WRN-34-20 teacher. Specifically, FERD achieves the highest average robustness and worst-robustness, achieving the improvement of 2.08% and 0.4% respec- tively under AutoAttack. This is attributed to our designed loss functions in Section 4.3 which can invert high-quality images from teacher models with different architectures.
5.3 ABLATION STUDY
In this section, we provide ablation studies on FERD. We keep the same settings with experiments and use WideResNet-34-10 10 as the teacher, ResNet-18 as the student and CIFAR-IO as dataset.
Effectiveness of all components. To verify the effectiveness of our FERD, we conduct ablation studies for each component, and the experimental results are shown in Tab. 4 We first examine the impact of the reweighting strategy. Compared with the absence of it, the student's 's accuracy on worst- class significantly improves when reweighting strategy is applied. However, it slightly compromises the overall accuracy, which is consistent with the findings of previous researches (Yue et al., 2023 Zhao et al. 2024). Further, we analyze the contributions of FAES and UTAES. Removing either component individually results in consistent performance decline across all metrics. When both FAES and UTAES are removed simultaneously, the performance further deteriorates. This confirms that FAEs and UTAES complementary and their joint effect is crucial for achieving the best robust fairness performance.
Hyperparameter ?to In 7 (a) and (b), we illustrate the average and worst-class performance of the student distilled with varying 7 during UTAEEs generation. controls the strength of the uniform target class constraint during adversarial examples generation. Note that when y=0, the adversarial examples are the same as the standard PGD method. For average robustness and average accuracy, we observe that a low medium 7 (e.g., 0.1 to 0.5) positively enhances the robustness and fairness of the student. This indicates the uniform constraint enhances the robustness of the model without significantly affecting the intensity of adversarial perturbations. However, A high 7 (e.g., 0.7 or 0.9) enforces strong uniform constraint, which overly suppress the optimization of the adversarial loss. leading to weaker perturbations that reduce attack strength of the adversarial examples. This negatively affects the robust distillation and leads to reduction in overall robustness.
CONCLUSION
In this paper, we made the first attempt to investigate the robust fairness in DFRD. We summarized two key factors affecting robust faimess and propose FERD framework to mitigate these problems by adjusting the proportion and distribution adversarial examples. For the proportion, we intro- duced caanstness-guidee class reweighting strategy to encourage the generator synthesize more samples from weakly robust classes. For the distribution, we designed FAEs and UTAES, taking them as benign samples and adversarial examples respectively for robust distillation. Extensive periments show that FERD significantly improves the robust and fairness performance the student model. Our work is more applicable and can be effectively applied in practical scenarios.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
Zhengxiao Li, Liming Lu, and Shuchao Pang are supported by National Key Research and Develop- ment Program of China under (Grant No. 20022223FB20003000 the National Natural Science Founda- tion of China (Grant No.62206122 (28) and the Postgraduate Research & Practice Innovation Program of Jiangsu Province (Grant NY.KY EY232, Yongbin Zhou is supported by the National Natural Science Foundation of China (Grant U2332636.
ETHICS STATEMENT
This paper aims to address the issue of robust fairness in data-free robustness distillation. The FERD framework we propose designed to enhance the fairness of the model's robust performance across different categories, thereby improving its reliability and security in real-world applications. All the experiments are conducted on publicly available and standard datasets, which are widely used in the machine learning. These datasets do not involve any sensitive private personal information, avoiding the risks of data privacy and abuse.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we have provided comprehensive details of our exper- iments. The experimental setting of FERD detailed in Section [5.1] including the models, base- lines, evaluation metrics, and implementation details. All datasets (CIFAR-10, CIFAR-100, and Tiny-ImageNet) are publicly available. Final hyperparameter values for all experiments are explic- itly listed, with further analysis on their selection provided in Appendices F. To facilitate direct replication our results, we make our source code publicly available upon publication.
REFERENCES
Yang Bai, Yisen Wang, Yuyuan Zeng, Yong Jiang, and Shu-Tao Xia. Query efficient black-box adversarial attack deep neural networks. Pattern Recognition, 133:109037, 2023.
Léon Bottou, Frank Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM review, 60(2):223-311, 2018.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017
IEEE Symposium on Security and Privacy (SP), pp. 39-57. Ieee, 2017.
Erh- Erh-Chung Chen and Che-Rung Lee. Data filtering for efficient adversarial training. Pattern Recog- nition, 151:110394, 2024.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter- frree attacks. In International Conference on Machine Learning, pp. 2206- 2216. PMLR, 2020.
Jiequan Cui, Zhuotao Tian, Zhisheng Zhong, Xiaojuan Qi, Bei Yu, and Hanwang Zhang, Decoupled kullback-leibler divergence loss. Advances in Neural Information Processing Systems, 37:74461- 74486, 2024.
Gongfan Fang, Song, Xinchao Wang, Chengchao Shen, Xingen Wang, and Mingli Song. Con- trastive model inversion for data-free knowledge distillation. arXiv preprint arXiv:2105.0855 2021.
Gongfan Fang, Kanya Mo, Xinchao Wang, Jie Song, Shitao Bei, Haofei Zhang, and Mingli Song. Up to 100x faster data-free knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 6597-6604, 2022.
Cheng Gongye, Yukui Luo, Xiaolin Xu. and Yunsi Fei. side-chanmel-assisted reverse-engineering of encrypted dnn hardware accelerator ip and attack surface exploration. In 2024 IEEE Symposium Security and Privacy (SP), pp. 4678 4695. IEEF, 2024.

---

Published as a conference paper at ICLR 2026
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Gaurav Goswami, Nalini Ratha, Akshay Agarwal, Richa Singh, and Mayank Vatsa. Unravelling robustness of deep learning based face recognition against adversarial attacks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Xiang Gu, Liming Lu, Xu Zheng, Anan Du, Yongbin Zhou, and Shuchao Pang. Multimodal robust prompt distillation for 3d point cloud models. arXiv preprint arXiv:2511.21574, 2025.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual net- works. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Nether- lands, October 11-14, 2016, Proceedings. Part IV 14, pp. 630-645 645. Springer. 2016b.
Lei Hsiung, Yun- Yun Tsai, Pin-Yu Chen, and Tsung Yi Ho. Towards compositional adversarial robustness: Generalizing adversarial training to composite semantic perturbations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24658 24667, June 2023.
Bo Huang. Mingyang Chen, Yi Wang, Junda Lu, Minhao Cheng, and Wei Wang. Boosting accuracy and robustness of student models via adaptive adversarial distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24668-24655 2023.
Hanxun Huang, Yisen Wang, Sarah Erfani, Quanquan Gu, James Bailey, and Xingjun Ma. Explor- ing architectural ingredients of adversarially robust deep neural networks. Advances in neural information processing systems, 14.5545-5550 2021.
Xiaojun Jia, Yong Zhang, Baoyuan Wu, Ke Ma, Jue Wang, and Xiaochun Cao. Las-at: adversarial training with learnable attack strategy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. -339--1340, 2022.
Xiaojun Jia, Jianshu Li, Jindong Gu, Yang Bai, and Xiaochun Cao. Fast propagation is better: Accelerating single-step adversarial training via sampling subnetworks. IEEE Transactions Information Forensics and Security, 19:4547-4559, 2024a. doi: 10. 10.11090
Xiaojun Jia, Yong Zhang, Xingxing Wei, Baoyuan Wu, Ke Ma, Jue Wang, and Xiaochun Cao. Improving fast adversarial training with prior-guided knowledge. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 46(9):6367 63883, 2024b. doi: 10.1109/TPMM..222. 3381180.
Junho Kim, Byung-Kwan Lee, and Yong Man Ro. Distilling robust and non-robust features in adversarial examples by information bottleneck. Advances in Neural Information Processing Systems,
2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Yao Li, Mínhao Cheng, Cho-Jui Hsieh, and Thomas CM Lee. A review of adversarial attack and defense for classification methods. The American Statistician, 76(4):329-345, 2022.
Jeong-A A Lim, Joohyun Lee, Jeongho Kwak, and Yeongjin Kim. Cutting-egge inference: Dynamic dnn model partitioning and resource scaling for mobile ai. IEEE Transactions on Services Com- puting, 2024.
Hou-I Liu, Marco Galindo, Hongxia Xie, Lai-Kuan Wong, Hong- Shuai, Yung-Hu Hui Li, and Wen-Huame Cheng. Lightweight deep learning for resource-constrranect environments: A survey. ACM Computing Surveys, 566 (6611))1-42 2024.
12

---

Published as a conference paper at ICLR 2026
Liming Lu, Shuchao Pang, Xu Zheng, Xiang Gu, Anan Du. Yunhuai Liu, and Yongbin Zhou. Ciard: Cyclic iterative adversarial robustness distillation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 350-359, 2025.
Xingjun Ma, Yuhao Niu, Lin Gu, Yisen Wang, Yitian Zhao, James Bailey, and Feng Lu. Under- standing adversarial attacks on deep learning based medical image analysis systems. Pattern Recognition, 10:107332, 2021.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint
2017.
Paul Micaelli and Amos J Storkey. Zero-shot knowledge transfer via adversarial belief matching. Advances in Neural Information Processing Systems, 32, 2019.
Shuchao Pang, Yihang Rao, Zhigang Lu, Haichen Wang, Yongbin Zhou, and Minhui Xue. Pridm: Effective and universal private data recovery via diffusion models. IEEE Transactions on De- pendable and Secure Computing, 22(4):3259-3274, 2025.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Chunyu Sun, Chenye Xu, Chengyuan Yao, Siyuan Liang, Yichao Wu, Ding Liang, Xianglong Liu, and Aishan Liu. Improving robust fariness via balance adversarial training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 15161-15169, 2023.
Yuzheng Wang, Zhaoyu Chen, Dingkang Yang, Pinxue Guo, Kaixun Jiang, Wengiang Zhang, and Lizhe Qi. Out of thin air: Exploring data-free adversarial robustness distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 5776-5784, 2024a.
Zeyu Wang. Xianhang Li, Hongru Zhu, and Cihang Xie. Revisiting adversarial training at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2024b.
Junhao Xia, Haotian Zhu, Shuchao Pang, Zhigang Lu, Bing Li, Yongbin Zhou, and Jason Xue. One head to rule them all: Amplifying Ivlm safety through single critical attention head. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Han Xu, Xiaorui Liu, Yaxin Li, Anil Jain, and Jiliang Tang. To be robust or to be fair: Towards fairness in adversarial training. In International conference on machine learning, pp. 11492- 11501. PMLR, 2021.
Shaokai Ye, Kaidi Xu, Sijia Liu, Hao Cheng, Jan-Henrik Lambrechts, Huan Zhang, Aojun Zhou, Kaisheng Ma, Yanzhi Wang, and Xue Lin. Adversarial robustness vs. model compression, or both? In Proceedings of the IEED/CVF international conference computer vision, pp. 111- 120, 2019.
Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Ni- raj K Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
8715-8724, 2020.
Xiaojian Yuan, Kejiang Chen, Wen Huang, Jie Zhang, Weiming Zhang, and Nenghai Yu. Data- free hard-label robustness stcaling attack. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 6853 6861, 2024.
Xinli Yue, Mou Ningping, Qian Wang. and Lingchen Zhao. Revisiting adversarial robustness dis- tillation from the perspective of robust fairness. Advances in Neural Information Processing Systems, 36:30390-3000
2023.
Xinli Yue, Mou Ningping, Qian Wang, and Lingchen Zhao. Revisiting adversarial robustness dis- tillation from the perspective of robust fairness. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2026
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07116, 2016.
Wide
residual networks.
arXiv preprint
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. International Conference on Machine Learning, pp. 7472-7482. PMLR, 2019.
Shiji Zhao, Ranjie Duan, Xizhe Wang, and Xingxing Wei. Improving adversarial robust fairness via anti-bias soft label distillation. Advances Neural Information Processing Systems, 37:89125- 89149, 2024.
Yuhang Zhou, Yushu Zhang, Leo Yu Zhang, and Zhongyun Hua. Derd: data-free adversarial bustness distillation through self-adversarial teacher group. Proceedings of the 32nd ACM International Conference Multimedia, pp. 10055- 2024.
Haotian Zhu, Shuchao Pang. Zhigang Lu, Yongbin Zhou, and Minhui Xue. Gap-diff: protecting opeg-compressed images from diffusion-based facial customization. In Network and Distributed System Security Symposium. Internet Society, 2025.
Jianing Zhu, Jiangehao Yao, Bo Han, Jingfeng Zhang, Tongliang Liu, Gang Niu, Jingren Zhou, Jianliang Xu, and Hongxia Yang. Reliable adversarial distillation with unreliable teachers. arXiv preprint arXiv:2106. 04928,4,, 2021.
Kaijie Zhu, Xixu Hu, Jindong Wang, Xing Xie, and Ge Yang. Improving generalization of ad- versarial training via robust critical fine-tuning. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 4424-4434, 2023.
Bojia Zi, Shihao Zhao, Xingjun Ma, and Yu- GanG Jiang. Revisiting adversarial robustness distil- lation: Robust soft labels make student better. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pp. 16443-16452, 2021.
14

---

Published as a conference paper at ICLR 2026
A PROOFS
A. THE PROOF OF CONJECTURE
Conjecture is: by adjusting the weight of class sampling, the problem of robust unfairness is effectively alleviated.
Assumption 1. The robust risk function (0) for each class c is differentiable and L-smooth [Bottou et al. 2018). That is, for any 81, 82, there exists constant L > 0 such that:
L
Rx2(2) S (0)) + (VR.(0:).02 01) + 2 102 01 11.
(13)
Theorem 1. the sampling weight pk. for the robust class i satisfies ps > C at step t, then the expected risk reduction for the weak-robust class strictly greater under reweighting sampling than under uniform sampling, provided that the gradient alignment condition holds:
( ([(x(00T1))
(14)
Proof. Consider the SGD update rule at step t. For uniform sampling, the expected update direction E Afuii -C (f.). For reweighting sampling, the expected update direction is E[A0rew] = - E, (R.)
Using the L-smoothness the weak-robust class k:
L 11011
Rx+0++1) Ra(0,) + (VR)(0)), A0)
(15)
We compare the linear decrement term D = (0,), for both strategies, neglecting the second-order term for sufficiently small r7. For uniform sampling:
c
Vic.) = -1 (1IV 12 / VR.))
Dumi RV(
(16)
For reweighting sampling:
Drew VV,, De VR.) = -1/ Pk PR||VRR|12 + Σ VRR VR.)
(17)
offk
We aim to prove Drew D Duni. Let 2u, = (VRR, VR..) and we analyze the difference:
Auts Duns Drew =17 (pk = + Σ(pe
- Jor
1
(18)
esth
Reweighting strategy assigns Ps > δ for the weak-robust class k, and < ã for robust classes. Since | > 0 is the dominant gradient term for the weak-robust class, the term (Px VR, is positive and large.
While gradient conflicts may exist (dh. < 0), the gradients of vulnerable classes typically exhibit larger magnitudes compared to robust classes due their higher loss values in adversarial train- ing Madry et al. 2017). Specifically, the term (pk [ outweighs the residual terms. Thus, Dait > O. implying Dfer Duni-
Consequently, the upper bound of the worst-class risk decreases more significantly under reweight- ing sampling:
(x < (0,) + Drew + 0(7²) < Rx())) + Duni + 0
(19)
Then the Theorem is proved.
15

---

Published as a conference paper at ICLR 2026
A.2 THE PROOF OF CONJECTURE 2
Conjecture 2 is: robust distillation on adversarial examples with uniformly distributed targets defend against attacks from different targets.
Definition 1. The standard adversarial risk makes the worst perturbation within a ball (x) = {x² || =1pp S e):
Red E (Z.a))=
max [(fo(x (E),(0)
+), 8),
(20)
Adversarial attacks implicitly targets the decision boundary of the nearest incorrect class.
Definition 2. Adversarial examples with uniformly distributed targets enforce attacks towards target class & sampled uniformly from y \ {v). The risk objective is:
Runs (0) B(z,))>D
B
max (0)
(21)
where Ltarget encourages the model classify x + 5 t.
Assumption 2. The event of successful untargeted adversarial attack is the union of successful targeted attacks towards all other classes k * y. Specifically, let E be the error region E {a' (x) arg max so(x') * y). We assume E = Usty Eg. where Eg. is the region where the input is misclassified specifically class & (Carlini & Wagner 2017).
Theorem 2. Minimizing the Uniform-Targe Risk (o) minimizes an upper bound of the Stan- dard Adversarial Risk (f).
Proof. The standard adversarial loss seeks the perturbation that maximizes the loss, corresponding to the model predicting an incorrect class k:
max L(fe(x + 6). =o max max Ctarget (fo( k0) leg
(22)
8
Let 2)) maN Letrget (fo k) be the worst loss towards specific target k. The standard robust risk is determined by the maximum component:
Lsa((, x) max 2))
(23)
key
The training on adversarial examples with uniformly distributed targets minimizes the expectation follows:
Luni (0,
Σ (0,
(24)
C
kz
Therefore, we have the following inequality:
max kif-y
(0,
x))
S
Σ ( 11) (0,
(25)
Taking the expectation over the data distribution D:
(0) < -C R (8).
(26)
This inequality proves that training on adversarial examples with uniformly distributed targets min- imizes upper bound of the standard adversarial risk. Then the Theorem 2 is proved.
B THE WHOLE FRAMEWORK OF FERD
This section provides the whole framework of our Fainess-Enhanced data-free adversarial Robust Distillation (FERD). The FERD framework is composed of three key components: a robustness- guided class reweighting strategy, the generation of Faimess-AAare Examples (FAEs), and the con- struction of Uniform- -arget Adversarial Examples (UTAEs).. The detailed algorithm description is outlined in Algorithm
16

---

Published as a conference paper at ICLR 2026
Algorithm 1: The Whole Framework of FERD
Input: A pre-trained robust teacher fr, student f with parameter 0.. a generator with parameter gg. distillation epochs T, the iterations of generator in each epoch Tg. the iterations of student f in each epoch T,, class sampling probability distribution p. learning rate for generator 7, and student 17.- Randomly initialize parameters fg and 0
Uniformly initialize p:
Initialize an empty dataset = 0 for number of iterations I do
//* Generation stage #// N(O, 1), y ~p:
for number of iterations Tg do
IF + (2. :)
// Synthesize FAEs
0g + Mly Vo,Lgen f xF,9)
2/ select(ar))
D DU {
12
p - reweight Celass)
// Reweighting
//* Robustness distillation stage *//
14
for number of iterations Ts do
FF sample(D):
16
IU + UT UTAES(IF); 0g 6 0,
// Create UTAEs
Ve, Letulf", .z, zU):
17
Output: A student f with fair robustness
C THE PERFORMANCE OF THE TEACHER
In this section, show the performance of the teacher models under various attacks. We select WideResNNe-34-11 (Zagoruyko Komodakis 2016) trained by (Chen & Lee 2024) on CIFAR-10 10 and CIFAR- 100 (Krizhevsky et al. [2009). respectively. Meanwhile, select Pre (He
et al. 2016b) trained by TRADES (Zhang et al. 2019) on Tiny-ImageNet (Le & Yang 2015). The
performance shown in Tab.
Table 5: Robustness (%) of the teacher models.
Dataset
Model
Clean
FGSM
PGD
CW 80
AA
CIFAR-10 10
WRN-34-10
86.55
64.96
61.53
58.07
57.30
CIFAR-100
WRN-34-10
64.32
39.51
36.72
32.56
31.15
ImageNtt
PARN-34
48.96
25.69
23.47
19.56
18.56
D THE RESULTS ON TINY-IMAGENET
We conduct experiments on Tiny-ImageNet. We select PreActResNet-34 the teacher and PreActResNet-18 18 as the student. For the settings, we keep consistent with the experiments on CIFAR- 100. The results are shown in Tab. 6 The results prove the effectiveness of FERD, achiev- ing state-of-the-art accuracy and worst-10%% robustness on Tmny- Note that 7SKT is not applicable for large datasets and the distilled student performs extremely poorly on Imag- we do not compare with it this dataset. The worst-10% robustness under both CW oo and AA attacks is all O, so we also do not compare on it.
From the results, it is observed that proposed FERD achieves optimal performance under most adversarial attacks. Specifically, FERD achieves state-of-thh-aar average robustness and NSD under all attacks. For the average robustness, FERD is 0.34%, 1.50%, 0.39%, and 0.40% higher than the suboptimal method under the four adversarial attacks. For the NSD, FERD also comprehensively improves 0.015, 0.004, 0.030 and 0.037 respectively.
17

---

Published as a conference paper at ICLR 2026
Table Result of average robustness( %) (Avg. 1), worst-10(%)) (Worst f), and normalized standard deviation (NSD 1) on Tiny-ImageNet. The best results are bolded and the second best results are underlined respectively.
Clean
FGSM
PGD
CW
AA
Method
Avg.
Worst
NSD
Avg.
Worst
NSD
Avg.
Worst
NSD
Avg.
NSD
Avg.
NSD
CMI
38.56
11.90
0.445
16.46
1.10
0.785
13.43
0.00
0.882
9.93
1.109
9.23
1.152
DeepInv
36.25
9.70
0.465
13.46
0.50
0.861
10.95
0.00
0.958
7.32
1.202
6.58
1.288
Fast
33.76
7.80
0.473
12.94
0.40
0.804
11.19
0.00
0.846
7.66
1.087
7.11
1.138
DERD
36.32
9.40
0.470
16.10
1.60
0.737
14.32
0.20
0.779
10.68
0.977
9.67
1.044
DFARD
39.37
11.60
0.448
17.66
1.30
0.750
14.44
0.30
0.804
11.80
0.967
10.84
1.046
DFHL
23.13
1.10
0.719
1.368
5.51
1.555
5.43
1.569
4.95
1.654
FERD
39.64
12.30
0.439
18.00
2.10
0.722
15.94
0.40
0.783
12.19
0.937
11.24
1.007
Table 7: The performance of different reweighting strategies. The best results are bolded.
Clean
FGSM
PGD
Method
Avg.
Worst
NSD
Avg.
Worst
NSD
Avg.
Worst
NSD
power
80.54
65.20
0.111
52.04
28.70
0.280
42.79
17.90
0.334
sig
80.16
64.10
0.111
51.29
27.70
0.290
42.58
18.10
0.350
linear
79.89
63.70
0.113
52.06
29.10
0.275
42.23
17.10
0.344
tanh
80.40
64.60
0.115
51.47
28.90
0.284
42.30
18.20
0.345
FERD
79.26
65.90
0.105
51.09
31.30
0.237
42.24
20.40
0.331
E COMPARISON OF REWEIGHTING STRATEGIES
To further demonstrate the superiority of our reweighting strategy, we conduct experimental com- parisons with different reweighting functions (Yue et al. 2023). We use WideResNet-344 10 as the teacher, ResNet- 18 as the student and CIFAR- 10 dataset, while other settings keep consistent with main paper.
The reweighting strategies compare all based important metric: the least PGD steps (LPS), which represents the steps needed for perturbations of a benign example to become a adver- sarial example. The average LPS within a class K. serve as a metric of class-wise robustness, which is calculated as follows:
No Σ
K6
arg min rE(O.K]
(z.
+ v))
(27)
N.
where is the PGD steps. The higher the value, the more robust the model is to this class, which case they reduce the weight during training.
Based on the metric, we compare four reweighting strategies. The first is power-type function:
We
(28)
where 3 controls the smoothness of the reweighting function. function:
The second strategy is linear-type
We
(29)
K +
The third strategy is sigmoid-type function:
(x ( (1-2 2x Ki )) K
We sigmoid
(30)
The last strategy is tanh-type function:
We = 1 tanh (A 1x+5x (1 - x
(31)
2
18

---

Published as a conference paper at ICLR 2026
where À is the parameter. For the above parameters, set K 20, B 2 and À 0 respectively following (ue al. 2023). The experimental results are shown in Tab. 7
We observe that the reweighting strategy achieves the best results in the aspect of worst-class robustness, which demonstrates that it identifies the categories with poor robustness more effectively.
F HYPER-PARAMETER SELECTION
In this section, show the effect of parameters the distillation results and illustrate how determine the choice of parameters. We select WideResNet-33-10 as the teacher and ResNet-18 as the student experiment CIFAR-IO.
Aadus Abns Aoh and Auni represent the weights of different loss terms the loss function Lgens respectively. In order explore the influence of different weights on the experimental results, we change the weights, respectively, for the experiments. The results are shown in Tab. The corresponding synthetic samples are shown in Fig. 8
(a) daavv 5, Aon 11 Aon -1 Aune
(b) 2adv -1, An -5, 2oh -1, Aunt
(c) Aadv =1 Apn -1, Aon 5, Auni
(d) 2edv Aon =1 2oh =1, Aumi
Figure 8: Synthetic samples under different paramters.
From the results, find that Lade significantly improves the average robustness under all adversar- ial attacks, especially against FGSM and PGD. From the synthetic samples, we find that Lim. helps improve the sample quality, as shown in Fig. g (b), making the student more applicable to the
19

---

Published as a conference paper at ICLR 2026
Table 8: Result of average robustness($) (Avg. 1), worst-class(%) (Worst f), and normalized stan- dard deviation (NSD 4) different Xado, Aba Aoh and Xuni- The best results are bolded, and the second best results are underlined.
Parameters
Clean
FGSM
PGD
CW
AA
Audu
Asa
Asi
Avni
Avg.
Worst
NSD
Avg.
Worst
NSD
Avg.
Worst
NSD
Avg.
Worst
NSD
Avg.
Worst
NSD
81.23
67.60
0.100
50.59
31.40
0.258
41.55
16.90
0.359
41.30
18.90
0.337
39.73
16.90
0.350
5
81.32
70.00
0,089
53.26
34,40
0.229
44.23
20,80
0.316
41.93
20.20
0.301
39.64
18.20
0.328
80.10
66.40
0,089
50.75
31.20
0,204
41.16
18.50
0.315
40.94
20.50
0.291
39.06
19.30
0.29
72.97
52.20
0.153
40.65
18.30
0.318
35.07
11.10
0.385
34.74
12.60
0,363
33.67
11.80
0.383
80.55
66.70
0.107
52.55
30.10
0.265
43.47
18.50
0.366
43.19
21.10
0.335
40.74
18.50
0.364
1
3
80.51
65.20
0.110
50.97
30.00
0.235
41 41.84
19.50
0.338
41.67
17,40
0,349
39.97
19.20
0.359
1
5
79.26
65.90
0,105
51.09
31.30
0.237
42.24
20.40
0.331
41.73
19.30
0.339
40.12
20,80
0.325
1
7
79.16
64.30
0.124
50.45
31,90
0.246
41.77
21.40
0,346
41.26
18.50
0,348
39.73
20.30
0.353
1
9
78.81
63.40
0.125
50.12
30.20
0.246
41.12
18.80
0.347
40.22
18.40
0.342
38.90
18.90
0.344
Table 9: Robustness of DFKD and corresponding DFRD methods. RN-18 and MN-V V2 are abbrevi- ations of ResNet-18 and MobileNet-V3 respectively. The value in parentheses the change value after transforming into DFRD.
Student
Method
Clean
FGSM
PGD-20
CW oc
AA
ZSKT
66,45
29.60
23.03
17.46
16.43
ZSKT+RSLAD
65.48 (-0.97)
29.80 (+0.20)
22.88 (-0.15)
18.48 (+1.02)
17.16 (+0.73)
CMI
76,20
32.34
28.27
26.24
25.53
CMI+RSLAD
68.29 (-7.91)
33.34 (+1.00)
28.94 (+0.67)
27.63 (+1.39)
26.85 (+1.32)
RN-18
DeepInv
74.20
34.98
27.50
23.39
22.84
DeepInv+RSLAD
71.22 (-2.98)
38.60 (+3.62)
27.95 (+0.45)
27.23 (+3.84)
25.19 (4-235))
Fast
78.51
41.04
37.23
30.31
27.95
Fast+RSLAD
72.20 (-6.31)
43.96 (+1.98)
39.64 (+2.41)
33.96 (+3.65)
33.24 (+5.29)
ZSKT
52.64
22.35
19.76
15.78
14.41
ZSKT+RSLAD
54.69 (+2.05)
23.62 (+1.27)
19.83 (+0.07)
15.86 (+0.08)
14.85 (+0.44)
CMI
67.95
35.43
17.55
15.69
6.87
MN-V2 CMI+RSLAD
60.70 (-7.25)
27.90 (-7.53)
19.70 (+2.15)
19.57
(+3.88)
18.09 (+11.22)
Deepinv
66.94
34.98
15.26
14.97
6.19
DeepInv+RSIAD
62.77 (-4.17)
31.66 (-3.32)
19.13 (+3.87)
19.01 (+4.04)
15.39 (+9.2)
Fast
69.81
30.17
24.03
16.69
14.59
Fast+RSLAD
58.60 (-11.21) 29.91 (-0.26)
24.28 (+0.25)
19.46 (+2.77)
18.35 (+3.76)
actual scenario. We also observe that appropriate Lok supervision is necessary, but too high will suppress sample diversity and lead to performance degradation. For Lunis it helps improve the robustness performance under the strongest attack (such as AA) and alleviate the robust unfairness problem. Here, considering the practicality and robust fairness of the model, we focus on Atn and Auni as parameters tuning. From the bottom four rows of the Tab. & we find that it has the best performance in most cases when Amn and Auni are both equal to 5, so we set the hyperparameter Aad==1, Abn =l, 5 finally.
G TRANSFORMING DFKD INTO DFRD
For most Data-Free Knowledge Distillation (DFKD), they do not involve robustness, in which case transform them into Data-Free Robust Distillation (DFRD) by adding adversarial noise to syn- thetic samples and then take synthetic samples and adversarial examples for robust distillation. Here, PGD to generate adversarial examples and then apply the same distillation training loss as RSLAD ( et al. 2021) to transform them into DFRD. Here, we set step size to 2/255 and attack iteration to 10. The results shown in Tab.
We observe that incorporating RSLAD into the distillation leads to a slight decrease in clean ac- curacy, but it achieves notable improvement in robustness, particularly under the stronger attacks.
20

---

Published as a conference paper at ICLR 2026
This trade-off is consistent with findings in prior adversarial training (Zhang et al. 2019). Therefore, we think that this transformation approach is reasonable and effective.
H
ADDITIONAL ABLATION STUDY
H.1
INTERMEDIATE LAYER
Table 10: Ablation study of different Intermediate Layer 1.
Clean
FGSM
PGD-20
Layer
Avg.
Worst
Avg.
Worst
Avg.
Worst
block1
78.50
64.80
48.83
30.50
40.48
20.00
block2
78.91
64.30
49.37
29.50
41.16
18.80
block3
79.26
65.90
51.09
31.30
42.24
23.50
In FAEs generation process, distill non-robust features from the output in the intermediate layer I of the teacher and impose uniform constraints, ensuring that the synthetic samples have an equal tendency to different targets when generating adversarial samples. In this section, we research the sensitivity of FERD's robustness performance to different layers. We distill non-robust features from the outputs of the penultimate, second and third residual modules, respectively. The results are shown in Tab. [0]
We observe that the penultimate residual module performs best in all aspects. Both the robustness and the faimess have achieved the best results. This further demonstrates that the higher layer contains high-level feature information, which is effective for distilling non-robust feature.
H.2
DISTILLATION EPOCH
Average Accuracy
Worst Accuracy
NSD
50
30
40
20
30
120
170
220
270
10
120
170
220
270
0.2
120
170
220
270
FGSM
PGD
CW
AA
Figure 9: Metric values under different epochs. The horizontal axis is epoch and the vertical axis is metric value.
To explore the impact of the epoch on the student performance, we evaluate the distilled student under different epoch settings. The results shown in Fig.
With the increase of the epoch, both the average accuracy and the worst-class robustness of the student exhibit a generally increasing trend, reaching peak round 220 and then floating slightly, indicating that the robustness is optimal at this time. At the same time, the NSD remains almost constant after 170 epochs. Therefore, it is reasonable select 220 as the final distillation epochs.
I SYNTHETIC SAMPLES FROM DIFFERENT DFRD
This section shows samples synthesized by different DFRD methods. The corresponding synthetic samples shown in Fig. 10 From the visual results, it is evident that our FERD generates sam- ples that are both more realistic and more diverse, proving that the method is suitable for practical scenarios.
21

---

Published as a conference paper at ICLR 2026
aa) ZSKT
(b) CMI
(c) Deeplnv
(d) Fast
Ee) DERD
(1) DFARD
gg) DFHL
(h) FERD
Figure 10: Inverted samples from different DFRD.
22

---

Published as a conference paper at ICLR 2026
J LIMITATIONS
Although FERD mitigates the robust fairness by introducing a robust-guided class reweighting strat- egy, the class imbalance may still affect the final results some extreme cases, For example, if some classes are very rare in the original data, the problem of unfairness of robustness may not be com- pletely solved by just adjusting the sample proportion. Moreover, the generator needs to optimize to multiple loss functions to synthesize high- high-quality samples, while the student also needs to perform robust distillation. These processes involve complex optimizations and large number of iterations, which lead to high computational cost.
K THE USE OF LARGE LANGUAGE MODELS
In the writing process for this paper, we employed a large language model for text refinement and grammatical corrections to enhance overall readability and precision. We clarify that the model's role was strictly advisory and confined to language aspects. The background of the study, the design of the experiments, and analysis of all results represent the independent work of the authors.
23


Reviews:
[{"review_id": "j8JYU3Tqdo", "reviewer": "Reviewer_wkEv", "summary": "This paper studies **robust fairness in Data-Free Robustness Distillation (DFRD)** — transferring robustness from a teacher to a student model without original data.\n\nThe authors identify two fairness issues:\n\n(1) class-wise robustness disparity under uniformly sampled synthetic data, and\n\n(2) target-dependent vulnerability under adversarial attacks.\n\nThey propose **FERD**, which enhances fairness at both the *proportion* and *distribution* levels:\n\n- **Robustness-guided Class Reweighting** generates more synthetic data for weakly robust classes.\n- **Fairness-Aware / Uniform-Target Adversarial Examples (FAEs / UTAEs)** apply uniformity constraints on feature and target spaces to prevent biased robustness transfer.\n\nExperiments on **CIFAR-10, CIFAR-100, and Tiny-ImageNet** show that FERD achieves higher worst-class robustness (+15.1% under FGSM) and lower NSD than prior DFRD methods, suggesting improved fairness and stability.", "questions": "- Could you include **ε-sweep results (ε = {4,6,8,10}/255)** to test whether fairness improvements persist under stronger perturbations?\n- Have you explored the **stability of results** under different τ, γ, and λ settings?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Novel and well-defined problem setup: fairness within DFRD.\n- Dual-level solution addressing both sample imbalance and attack-target bias.\n- Consistent empirical results across datasets and attack variants.\n- Ablation studies verify component contributions.", "weaknesses": "- **Limited theoretical rigor:** The design is intuitive but lacks formal analysis — no explicit robustness bounds, convergence arguments, or assumption statements are provided.\n- **Restricted evaluation scope:** Experiments are confined to small datasets (CIFAR-10/100, Tiny-ImageNet) and a single perturbation budget (ε = 8/255), leaving its generality under stronger or diverse attacks unexplored.\n- **Hyperparameter opacity:** Key parameters (τ, γ, λ) are fixed without justification or sensitivity analysis, creating uncertainty about robustness across configurations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eGlOFHmLCV", "reviewer": "Reviewer_WX8r", "summary": "The work offers a valuable perspective on robust fairness under data-free settings and provides convincing experimental evidence, though the methodological and theoretical originality are limited.", "questions": "Although the paper includes proofs of conjectures (Appendix A.1–A.2), they are mostly intuitive restatements of empirical observations and lack rigorous formalism. \n\nThe fairness claim relies mainly on NSD and worst-class robustness. Introducing additional fairness metrics would make the evaluation more convincing.\n\nThe manuscript is lengthy and includes numerous large figures and dense equations that affect readability.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper addresses the intersection of data-free robustness distillation and robust fairness, which is an emerging and practically important direction for fairness-aware model compression and deployment on edge devices.\n\nThe experiments are extensive and demonstrate consistent improvements in worst-class robustness and normalized standard deviation (NSD) across CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets.\n\nAblation studies, hyperparameter analyses, and visualization of synthetic samples (Fig. 6, Table 3) enhance the reproducibility and credibility of results.", "weaknesses": "Although the paper includes proofs of conjectures (Appendix A.1–A.2), they are mostly intuitive restatements of empirical observations and lack rigorous formalism. \n\nThe fairness claim relies mainly on NSD and worst-class robustness. Introducing additional fairness metrics would make the evaluation more convincing.\n\nThe manuscript is lengthy and includes numerous large figures and dense equations that affect readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BBrJWmtLPq", "reviewer": "Reviewer_jJ8d", "summary": "This paper identifies that data-free robust distillation (DFRD) methods create models with unfair robustness, meaning they are robust for some class but non-robust for others. The authors find this is because DFRD trains on an equal number of samples for all classes and uses attacks that are not diverse. They propose FERD, which fixes this by generating more synthetic data for the non-robust classes and creating special adversarial attacks that target all classes uniformly. Experiments show this method improves the robustness of the weakest classes, leading to better overall fairness in the context of DFRD.", "questions": "-\tWhile Section 2.1 discusses several DFRD methods (DFARD, DERD, DFHL), your experiments only include DFHL as a direct DFRD baseline, primarily comparing against adapted DFKD methods. Could you explain the rationale for omitting DFARD and DERD from the empirical comparison and for this specific choice of baselines?\n-\tFor the adapted DFKD baselines, why did you use standard PGD for attack generation instead of the inner maximization objective proposed in the original RSLAD paper? Could this choice underestimate the adapted baselines' potential robustness?\n\nThis paper tackles the important problem of DFRD fairness with strong results. However, concerns about baselines and context limit the current evaluation. I am open to revising my score based on the rebuttal.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is the first to investigate and address the problem of robust fairness specifically within the context of Data-Free Robustness Distillation (DFRD), an area where fairness considerations have been largely overlooked.\n2. The proposed class reweighting strategy introduces a mechanism to generate intentionally imbalanced synthetic data tailored for fairness improvement, adapting generation proportions based on class vulnerability within the data-free constraint.", "weaknesses": "1. The paper claims state-of-the-art robust fairness in DFRD but fails to compare against a crucial baseline: combining a standard DFRD data generation method with established fairness-aware adversarial distillation techniques like Fair-ARD or ABSLD. It is plausible that simply applying existing fairness distillation losses to student training, using data from a generic DFRD generator, could yield comparable fairness improvements. Without this comparison, the paper does not demonstrate that FERD's specific components offer benefits beyond a straightforward integration of known DFRD generation and fairness AD methods.\n\n2. Robust fairness methods (e.g., ABSLD) demonstrate their effectiveness by showing significant improvements in worst-case robustness or NSD while maintaining comparable or slightly improved average robustness relative to strong baselines. This isolates the contribution specifically to fairness. However, FERD's results show a large increase in average robustness alongside the worst-case improvement (e.g., Table 1, RN-18/CIFAR-10: Avg. AA +3% vs. Worst AA +1% compared to DFHL). This large average gain makes it difficult to determine if the worst-case improvement is a direct outcome of the fairness mechanisms or simply a byproduct of the model becoming significantly stronger overall. While NSD improves, the paper does not isolate the fairness contribution by comparing against a baseline with similar average robustness, thus obscuring whether FERD primarily enhances fairness or just general robustness, which happens to also lift the worst-case performance.\n\n3. As a pioneering work in DFRD fairness, the paper's evaluation could be strengthened by providing additional context. Including teacher's own fairness profile and student performance under data-available adversarial distillation—using both standard (e.g., RSLAD) and fairness-aware (e.g., ABSLD) methods—would offer valuable reference optimal points. Without these comparisons, and alongside the significant average robustness gains (Weakness 2), it remains somewhat challenging to fully gauge the significance of FERD's fairness improvements specifically within the constraints of the data-free setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TfoRq9c4cs", "reviewer": "Reviewer_VYHq", "summary": "This paper proposes a new framework, FERD, to address the robustness and fairness issues in data-free adversarial robustness distillation. The authors find that, even with uniformly sampled synthetic data, the robustness of the student model in traditional DFRD methods still varies significantly across categories, and the success rate of adversarial attacks also varies depending on the target class. FERD addresses this issue through two strategies: Robustness-guided class reweighting: This increases the generation of samples for weakly robust classes; Fairness-aware sample and uniform target adversarial sample generation: This improves the class coverage of adversarial samples by uniformly constraining feature levels and uniformizing attack targets.", "questions": "See the weakness", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is easy to follow.\n\n2. The proposed method is novel. A class reweighting strategy is proposed to enhance robustness against weak classes. FAEs and UTAEs are designed to improve fairness at both the feature and attack target levels.\n\n3. The experimental result demonstrates the proposed method can effectively alleviate fairness issues in data-free ARD.\n\n4. This paper also provides an explanation from a theoretical level.", "weaknesses": "1. This paper only selected one teacher model for one data set. I am curious whether the method is effective when different teacher models are selected.\n\n2. The experimental result in newer architectures, such as VIT, can further verify the effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)