Indirect Gradient Matching for Adversarial Robust Distillation
Authors: Hongsin Lee
Seungju Cho
Changick Kim
Venue: ICLR 2025 Poster (2025)
Keywords: Adversarial Robustness
Adversarial Training
Adversarial Distillation
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Adversarial training significantly improves adversarial robustness, but superior performance is primarily attained with large models. 
This substantial performance gap for smaller models has spurred active research into adversarial distillation (AD) to mitigate the difference. 
Existing AD methods leverage the teacher’s logits as a guide.
In contrast to these approaches, we aim to transfer another piece of knowledge from the teacher, the input gradient.
In this paper, we propose a distillation module termed Indirect Gradient Distillation Module (IGDM) that indirectly matches the student’s input gradient with that of the teacher.
Experimental results show that IGDM seamlessly integrates with existing AD methods, significantly enhancing their performance.
Particularly, utilizing IGDM on the CIFAR-100 dataset improves the AutoAttack accuracy from 28.06\% to 30.32\% with the ResNet-18 architecture and from 26.18\% to 29.32\% with the MobileNetV2 architecture when integrated into the SOTA method without additional data augmentation.

Published as a conference paper at ICLR 2025
INDIRECT GRADIENT MATCHING FOR ADVERSARIAL ROBUST DISTILLATION
Hongsin Lee", Seungju Cho", Changick Kim
Korea Advanced Institute Science and Technology (KAIST) (hongsin04, joyga, changick @aackkkkkkkkak @kaist.akk............. kr
ABSTRACT
Adversarial training significantly improves adversarial robustness, but superior performance is primarily attained with large models. This substantial performance gap for smaller models has spurred active research into adversarial distillation (AD) to mitigate the difference. Existing AD methods leverage the teacher's log- its as guide. In contrast to these approaches, we aim to transfer another piece of knowledge from the teacher, the input gradient. In this paper, we propose a distillation module termed Indirect Gradient Distillation Module (IGDM) that in- directly matches the student's 's input gradient with that of the teacher. Experimental results show that IGDM seamlessly integrates with existing AD methods, signifi- cantly enhancing their performance. Particularly, utilizing IGDM on the CIFAR- 100 dataset improves the AutoAttack accuracy from 28.06% to 30.32% with the ResNet- 18 architecture and from 18% to 29.32% with the MobileNetV2 archi- tecture when integrated into the SOTA method without additional data augmenta- tion.
INTRODUCTION
Recently, adversarial attacks have revealed the vulnerabilities of deep boarning-based models (Good-
fellow et al. 2014; Carlini Wagner 2017: Madry et al. 2017). raising critical concerns in safety-
important applications (Grigoreseuu et al. 2020 Ma et al. 2021 Wang et al. (023a). Thus, much
research has been done on defense technologies to make deep learning more reliable (Das et al.
2017 Carmon et al., 2019; Cohen et al. 2019 Xie et al. 2019 Zhang et al. 2022 Jin al. (023). Among adversarial defense mechanisms, adversarial training is one of the most effective methods enhance adversarial robustness (Pang al. 2020 Bai et al. 2021 Wei al.) 2023). However, there is a significant performance gap between large and small models in adversarial training. Since light models with less computational complexity are preferred in practical applications, increasing the robustness of light models is necessary. To address this, adversarial training incorporates dis- tillation methodologies which are commonly employed to boost the performance of light or small models (Goldblum et al. 2020 Zhu et al. 2021 Zi et al., [021] Huang et al. 2023).
In the teacher-student architecture of knowledge distillation, prevailing methods leverage the teacher's features or logits (Hinton al.] 20155] Ji et al., [2021] Kim al. [02T]] Yang et al.] 2022). Adversarial distillation (AD) approaches extend this paradigm by incorporating the teacher's log- its guide within their adversarial training framework (Goldblum al.) 2020 Zhu et al.) 2021 Maroto et al.) 2022). Recent studies have specifically focused on tailoring the inner maximization problem of adversarial training, particularly by involving the teacher model in the generation of ad- versarial examples (Zi et al. 2021: Huang et al., 2023). We leverage input gradients, which have been studied in relation robustness in adversarially trained models (Tsipras al.) 2018 Engstrom
et al. 2019 Srinivas et al. 2023).
We additionally demonstrate that matching the input gradients between teacher and student con- tributes to point-wise alignment, which has been studied in state-of-the-art AD methods (Huang al. 2023). In the top right of Figure Ial if the input gradients between the teacher and the student match as depicted in the red line, then the output of the points located on the red line, denoted as
"These authors contributed equally.

---

maximization problem as Published as a conference paper at ICLR 2025
30
A1,00
AutoNttack accuracy (86)
Tncchrgradas
27
+5.40
24
2 8
2
x
gradict
Folx 6
falx 6 1800
Indired Gradient
Distillatoon Modulee
Clean accuracy (X5
Toacher outgue space
#5C0Myaa-AT
Suckett oper
*100Masus
(a) Conceptual diagram of IGDM
(b) Performance of IGDM
Figure 1: (a) We match the gradient in the imput space with knowledge distillation. Since adversarial robust models are locally linear, matching the gradient has the effect of matching the output points of the surrounding input marked with 'stars', as illustrated in the top right figure. Consequently, gradient matching involves matching the teacher and student point by point in the output space, as depicted in the bottom right. (b) Clean and Autoattack accuracy on ResNet-18 with BDM-AT teacher CIFAR- 100 dataset. IGDM demonstrates significantly improved AutoAttack accuracy compared with other adversarial distillation methods.
stars, will also match. We show that the output of random points around x also becomes similar to the teacher as illustrated in the bottom right of Figure Ia This alignment causes the student to mimic the teacher and reduces the capacity gap between teacher and student.
In this paper, we propose Indirect Gradient Distillation Module (IGDM), which indirectly aligns the student's gradient with that of the teacher. We easily match input gradients through Taylor expansion, leveraging the locally-linear property of adversarial training. Since existing AD meth- ods are mainly designed match logits, IGDM can be used in conjunction with them. Through extensive experimental results, we verify that our method successfully complements the robustness of existing AD methods. We significantly improve existing AD methods depicted in Figure lb Our contributions are as follows:
We propose methodology to transfer the gradient information from the teacher to the student through the exploitation of the local linearity inherent in adversarial training. This stands in contrast to prevailing AD methods, which primarily concentrate on the distillation of logits.
Based on the analysis, we propose the Indirect Gradient Distillation Module (IGDM), which indirectly distills the gradient information. modular design allows easy inte- gration into existing AD methods.
We empirically demonstrate that IGDM significantly improves robustness against various attack scenarios, datasets, student models, and teacher models.
RELATED WORK
2.1
ADVERSARIAL TRAINING
Adversarial training (AT) involves generating adversarial examples during the training process to improve model robustness. The adversarial training is a follows:
arg min Exxy)~D [emin (Jo(x + 0), where 8 = arg max Lmaz (So(x + 8'),))
||
Here, 0 is parameters of model f, is data distribution of x and corresponding labels 5 is pertur- bation causing the largest loss within I, norm e, Lmaz is an inner maximiazation loss, and L,vin an outer minimization loss. Multi-step PGD attack (Madry et al. 2017) is commonly utilized to
2

---

Published as a conference paper at ICLR 2025
solve the inner maximization loss, and various regularization loss functions have been introduced for the outer minimization loss. TRADES (Zhang et al. 2019) incorporates the Kullback- Leibler (KL)
divergence loss between the predictions of clean and adversarial images. MART Wang et al. 2020
introduces per-sample weights based on the confidence of each sample. Due to their simplicity and
effectiveness, TRADES and MART commonly used as baseline methods in adversarial training
(Wu et al., 2020; Bai et al. 2021b; Jin et al., [022 Tack et al.) [0222] Jin et al.] [023]] Wei et al.
2023 Qin et 2019). Moreover, several strategies such data augmentation (Rebuffi al. 2021
Li & Spratling 2023), and diverse loss functions (Wu et al., 2020), have been introduced.
Although defending against strong adversarial attack strategies(Croce & Hein 2020aa b An- driushchenko et al., (2200) is challenging, highly robust models have also been developed. Low Temperature Distillation (LTD) (Chen & Lee (021) points out the shortcomings of one-hot labels in adversarial training and advocates the use of soft labels as an alternative approach. Better Dif- fusion Models for Adversarial Training (BDM-AT) (Wang et al. 2023b) explores methods for the more efficient utilization of diffusion models within the context of adversarial training. Improved Kullback-Leible Adversarial Training (IKL-AT) (Cui et al. 2023) inspects the mechanism of KL divergence loss and proposes the Decoupled Kullback- divergence loss. However, given the use of large architectures in these models, there a necessity to enhance adversarial robustness in smaller models.
2.2 ADVERSARIAL ROBUST DISTILLATION
Adversarial distillation (AD) is an effective technique to distill the robustness from large teacher model to small student model Goldblum et al.) 2020: Zhu et al. 202T] Maroto et al. 2022; Zi
et al. T [Huang et al. 2023 Kuang et al. 2023 Jung et al. [024]] Yin et al. 2024 Dong et al.,
2025 Adversarial Robustness Distillation (ARD) (Goldblum et al. (020) reveals that students can
more effectiivey acquire robustness when guided by robust teacher within an adversarial training framework. RSLAD (Zi et al. 2021) emphasizes the importance of smooth teacher logits in robust distillation, integrating the teacher's s guidance directly into the adversarial image generation process. Introspective Adversarial Distillation (IAD) (Zhu et al., (20T]) concentrates on assessing the relia- bility of the teacher and introduces a confidence score of the information provided by the teacher. AdaAD (Huang el al. 2023) generates more sophisticated adversarial images through the integra- tion of the teacher during the inner maximization process. Previous AD methods utilize the teacher's logits as guide, but our approach also incorporates the distillation of gradient information.
2.3 GRADIENT DISTILLATION AND INPUT GRADIENT
In knowledge distillation, gradient information has been used in various ways (Czarnecki et al, 2017 Du et al. 2020 Zhu & Wang 2021 Lan & Tian 2023 Wang et al. (022). computed in either input space or weight space. For example, an exploration of the diversity among teacher models in the weight gradient space aids in identifying an optimal direction for training the student network (Du et al. 2020). Another study examines the capacity gap between teachers and students, focusing on the perspective of weight gradients similarity (Zhu & Waag 2021). Conversely, input space gradients find applications in knowledge distillation for tasks such as classification (Czarnecki et al., 2017). object detection (Lan & Tian| 2023). or language model (Wang et al., 2022). These methods all compute the gradient directly, which is different from our approach.
Additionally, input gradients of adversarially trained models contain semantically meaningful infor- mation, aligning with human perception, where image modifications for a specific class resemble human-recognized features (Tsipras et al.] [2088) [Engstrom et al., 2019] (Srinivas et al.) (023). This suggests that robust models capture more human- features, underlining the potential of dis- tilling such valuable insights from robust teacher models to enhance student model performance in adversarial settings.
METHOD
In this section, we elaborate on how to distill the input gradients from the robust teacher models. In the analysis of Appendix [A.4] direct gradient matching is shown to be ineffective. Therefore, we propose novel approach for gradient matching that avoids direct gradient computation.
3

---

Published as a conference paper at ICLR 2025
3.1 LOCAL LINEARITY OF ADVERSARIAL TRAINING
First, we show that adversarially trained models are capable of first-order Taylor expansion on the input unlike natural training. For small noise e, the output of can be expressed follows.
@ 8x
fxx+e) = f(x) +
e+ ++ +(IE)))
mesiidee
first-order approxinatiin
remainder
proportion
0.3
nottraaiingg
Here, f denotes the model, where (2) for brevity instead of fg. and E is suf- ficiently small noise with the same dimen- sion as the x. To explore the influence of the remainder, we calculate the proportion occupied by the remainder in Equation (2) by introducing uniform noise with mag- nitude of 8/255 as the perturbation c. We calculate the remainder proportion as the ra- tio of the remainder to the total value, i.e.
0.2
002
0.2
50
90
836--T
90.01
eporh
(a)
(b)
Figure 2: The proportion occupied by the remainder in Taylor expansion. (a) The remainder proportion of adversarially robust teacher model on CIFAR-100 dataset.
(b) The proportion along with training epochs in natural training, adversarial training, and adversarial distillation using ResNet-18 18 CIFAR-100 dataset.
[[((xte)-((x-((( (1())
We first examine the remainder proportion in an adversari- ally well-trained model: LTD (Chen & Leel
[[(x+e]]
2021]. BDM-AT [Wang et al. 2023b), and IKL-AT (Cui et al. 2023) where we summarized the per- formance in Table 1] In Figure 2al the remainder proportions computed to be very small, with values of 0.012, 0.012, and 0.016 for three adversarially-Be trrinned models LTD, BDM-AT, and IKL- AT, respectively. In other words. f(x + can be approximated the first-order Taylor expansion since the remainder proportion is negligible. Thus, we utilize this local linearity of the adversarially well-trained models to match input gradients.
Next, we investigate whether the model retains the local linearity in adversarial training, not in the case of a fully-trained model. We test three training strategies: natural training, adversarial training via ten steps of PGD (Madry et al. (017). and adversarial distillation (Goldblum et al. (020) using BDM-AT teacher. In Figure 2b] remainder proportion continuously increases in natural training. while it consistently remains small in both adversarial training and adversarial distillation. Therefore, employing first-order Taylor expansion is feasible during adversarial training. In the following section, we demonstrate how the locally linear property enables the matching of input gradients between the teacher and the student.
3.2 GRADIENT MATCHING VIA OUTPUT DIFFERENCES
For input x, we define xc, and N,2 as follows:
x,
61, Xr, -
where E1 and E2 represent small random perturbations of the same dimension as the input x. For adversarially trained or training model f. the output x., and Xxy can be approximated using first- order Taylor expansion through the input space:
(0f(x) Ox
c1, f(x.,) f(x) + Of((x)
T
f(x.,) ~ f(x) +
E2.
Ox
where we neglect the remainder term based on the observations Section align the gradients between the student and the teacher models, we utilize the output differences as follows:
3.1 To extract and
L = D (fs(x.,) fs(x.,) fr( (X.,))
where fs and fr represent the student and teacher models, while denotes a discrepancy metric like L2 or KL divergence loss. This loss can be reformulated using Equation 4 as follows:
(( (Dfs(x) Ix
(0fr(x)
T
L L=D
(E1 -E2,,
(61 62)
(6)
0x
4

---

Published as a conference paper at ICLR 2025
If we consider c1 <2 is arbitrary perturbation vector, varying through the optimization process, minimizing this loss encourages the alignment of the gradients between the student and the teacher models, mcaning
(x) 8x
afr(()
8x
3.3 INDIRECT GRADIENT DISTILLATION MODULE (IGDM)
To effectively integrate the gradient matching through output differences into AD methods, we select the 61 and 62 constant multiples of the adversary perturbation d from the AD methods, i.e. 61 62 x Since the adversarial perturbations obtained during the training process continuously change, the gradients can be aligned. Specifically, as the adversarial perturbation ô employs gradient information, the inner product of gradient and é1 in Equation
(6) can be effectively increased
with the adversarial perturbation. In other words, (0أs(S) (e Ez) [ y remains large, 5x
facilitating the enhanced alignment of gradients using loss function in Equation 6 during training.
Finally, Indirect Gradient Distillation Module (IGDM) loss is formulated as:
LIGDM = D (Js(x
8) fx (x(xx
-x( (x(x-
Since IGDM complements the loss function by matching the gradients between the teacher and student models, the integration of IGDM with other AD methods becomes feasible in the following manner.
Lmi = LAD + LIGDM.
where a is a hyperparameter and LAD stands for the outer minimization loss of other AD methods such ARD (Goldblum et al. 2020), RSLAD Zi al. 2021,, AdaAD (Huang et al. 2023), etc.
4
EXPERIMENTS
We explain the experimental setup, followed by comparative performance evaluations of the pro- posed IGDM method against various AT and AD methods.
4.1 SETTINGS
Teacher and Student Models We selected three teacher models including LTD (Chen & Lee 2021), BDM-AT (Wang et al. 2023b), and IKI-AT (Cui et al. 2023) for CIFAR-10/T00, The LTD model widely adopted in prior AD research, while the others have achieved high-ranking perfor- mance in RobustBench (Croce et al. 2021), demonstrating superior robustness against AutoAttack (Croce Hein 2020a). For Tiny-Imm ImageNet, we employed a pre-trained RiFT model (Zhu al.) 2023), and for S SVHN, we trained a WideResNeet34-100 (Zagoruyko & Komodakis 2016) with PGD. AT. For student models, we employed the ResNet-18 (He | (0)6a) and MobileNetV2 (Sandler et al. 2018) for 10/10010/100, ResNet-18 for SVHN, and PreActResNet-11 18 (He et al. 201 for Tiny-I ImageNe.
Table I: Performance (%) of the teacher models. Experiments with teacher models in italics are provided in the supplementary material.
Dataset
Teacher name
Architecture
Clean
PGD
AA
BDM-AT (Wang al.
2023b
WideResNet-28-10
72.58
44.24
38,83
CIFAR-100
LTD (Chn & Lee 202T
WideResNeet34-10
64.07
36.61
30.57
IKL-AT CCui et
2023
WideResNNet-2--10
73.80
44.14
39.18
LTD Chen & Lee 2021
WideResNNet34-110
85.21
60.89
56.94
CIFAR-10
BDM-AT Wang et al.
2023b
WideResNNet-2.
92.44
70.63
67.31
IKL-AT (Cui el al.
2023)
WideResNNet28-100 10
92.16
71.09
67.73
SVHN
PGD-AT (Madry al.
2017)
WideResNet-34-10
93.90
61.78
54.28
Tiny-Iman ImageNett
RiFT (Zhu et al.
2023
W0deResNNet34-110
52.54
25.52
21.78
5

---

Published as a conference paper at ICLR 2025
Table Adversarial distillation result on ResNet-18 with teacher models on CIFAR-I00. The Clean, PGD, FGSM, C&W, and AA each indicate performance (%). Bold indicates cases where IGDM improved accuracy by more than 0.5 percentage points.
CIFAR-100 with BDM-AT teacher
CIFAR-100 with LTD teacher
Method
Clean
FGSM
PGD
C&W
AA
Clean
FGSM
PGD
C&W
AA
PGD-AT
55,80
23.51
19.88
20.46
18.86
55.80
23.51
19.88
20.46
18.86
+IGDM
60.83
38.11
35.09
30.21
29.16
60.49
37.22
33.72
29.61
28.02
TRADES
53.56
29.85
25.85
23.32
22.02
53.56
29.85
25.85
23.92
22.02
+IGDM
60.88
36.26
32.26
26.50
25.50
59.29
36.08
32.08
26.10
26.10
ARD
61.51
34.23
30.23
26.97
24.77
61.34
35.19
31.19
27.74
25.74
+IGDM
61.62
39.75
35.75
30.99
28.79
61.58
37.45
33.45
29.94
27.84
IAD
59.92
35.47
31.47
26.91
25.15
60.12
36.91
32.91
28.29
26.29
+IGDM
62.99
37.32
34.76
29.55
27.76
62.73
37.69
33.69
29.65
27.49
AKD
60.27
35.38
31.38
26.29
25.09
60.46
35.53
31.53
27.29
25.37
+IGDM
60.42
37.63
33.62
29.27
28.86
60.31
38.15
34.15
29.44
28.44
RSLAD
60.22
36.16
32.16
27.96
26.76
60.01
36.39
32.39
28.94
26.94
+IGDM
62.06
39.67
35.67
30.98
29.78
60.43
38.02
34.02
29.94
28.51
AdaAD
64.43
37.33
33.21
29.53
28.06
63.34
37.39
33.39
29.73
27.81
AdalAD
64.13
37.33
33.33
29.02
27.82
63.24
37.45
33.45
29.04
27.83
+IGDM
64.44
39.31
36.19
31.75
30.32
63.44
38.23
34.23
31.09
28.87
Table Adversarial distillation result on ResNet-18 on CIFAR-10 and SVHN. The Clean, PGD, FGSM, C&W W, and AA each indicate performance (%). Bold indicates cases where IGDM improved accuracy by more than 0.5 percentage points.
CIFAR-I0 with LTD teacher
SVHN with PGD-AT teacher
Method
Clean
FGSM
PGD
C&W
AA
Clean
FGSM
PGD
C&W
AA
PGD-AT
84.52
52.42
42.80
42.98
41.12
91.62
65.93
48.07
48.34
42.46
+IGDM
84.15
59.21
53.70
51.19
49.63
92.57
72.19
60.55
56.10
52.12
TRADES
82.46
56.97
49.13
47.98
47.09
89.91
69.81
57.52
51.32
50.74
+IGDM
83.50
60.84
54.64
49.34
48.83
91.98
71.85
60.09
56.68
54.15
ARD
85.04
60.31
53.27
50.27
49.49
92.32
70.46
55.62
53.07
48.24
+IGDM
85.18
61.24
54.75
51.31
50.20
92.19
72.01
60.07
56.56
52.95
IAD
84.33
61.21
54.24
50.97
50.09
91.62
70.82
56.42
53.01
47.76
+IGDM
84.49
62.45
56.55
53.01
51.09
93.03
71.85
58.86
54.66
51.45
AKD
85.10
59.07
51.53
49.13
48.04
92.49
70.44
56.49
53.91
50.17
+IGDM
84.94
61.36
54.89
51.55
50.87
92.50
71.15
59.38
55.97
52.51
RSLAD
83.59
60.97
55.98
53.15
52.13
90.52
62.74
53.80
50.01
48.41
+IGDM
83.67
62.41
57.34
54.00
53.10
90.71
64.71
56.14
53.98
50.69
AdaAD
84.74
61.87
56.78
53.51
52.79
93.27
67.14
57.43
54.88
52.93
AdaIAD
84.75
61.98
57.04
53.57
52.88
93.39
67.10
57.26
54.76
52.74
+IGDM
84.83
62.54
57.61
55.09
54.02
93.48
68.34
58.74
56.03
53.89
Evaluation Metrics After training, we evaluate performance using five metrics: Clean, FGSM, PGD, C&W, and AutoAttack (AA) accuracy. Clean refers to the accuracy on the test dataset. We measure FGSM and PGD accuracy against fast gradient sign method (FGSM) (Goodfellow et al.) 2014) and 20-step projected gradient descent (PGD) attacks (Madry et al. 2017) respectively. The C&W attack measures accuracy against Carlini & Wagner (2017). while AA evaluates accuracy against the AutoAttack method (Croce & Hein 2020a). All attacks were conducted within an 4x- norm bound of 8/ 8/255.

---

Published as a conference paper at ICLR 2025
Implementation We utilized the CIFAR- 10/100 (Krizhevsky al. 2009). SVHN Netzer al. 2011]. and Iny-ImageNet {Le & Yang 2015) datasets for our experiments. Random crop and random horizontal flip were applied, while other augmentations were not utilized. Our training methods encompassed conventional adversarial training methods, PGD-AT (Madry et al. 2017) and TRADES (Zhang al., 2019), as well adversarial distillation techniques including ARD (Goldblum et al., 2020). IAD (Zhu et al. 2021). AKD (Maroto et al. 2022), RSLAD (Zi et al. 2021). and AdaAD (Huang et al. 2023). In our comparative analysis, we integrated IGDM into all of these methods, and we used shortened notations to represent the methods with IGDM. For example, when IGDM is combined with ARD, we denote it as ARD IGDM or IGDMARD. We employed the recommended inner loss functions for generating adversarial examples as outlined in each baseline as specified in Appendix [ We fixed the hyperparameters of all given methods used in the experiment as original paper. Then, we adjusted the hyperparameter of IGDM. For the surrogate loss of IGDM, we employed KL divergence loss, but using alternative surrogate losses, such as LI and L2 loss, yields nearly indistinguishable results. The detailed experimental setting can be found in Appendix B
4.2 RESULTS
Main Results We present the comprehensive results of integrating IGDM into other baseline methods and their original versions in Table [] Table 3 More experimental results are in the appendix: distillation results on MobileNerV2 student (Appendix I.I). distillation with IKL-AT teacher (Appendix (.2)). and experiments on the Tiny-ImagNet dataset (Appendix (A.3). IGDM sig- nificantly improves robustness against various attacks with consistent clean accuracy, regardless of the original methods, datasets, student models, or teacher models. IGDM demonstrates notably en- hanced AA robustness the CIFAR-10 100 dataset, achieving 30.32% ResNet-18 18 with BDM-AT teacher.
Gradient Alignment We justify that IGDM can align the student's input gradient with the teacher's input gradient. To quan- tify this alignment between the two imput gradients, we introduce two metrics: mean Gradient Distance (GD) and mean Gradient Cosine similarity (GC). GD quantifies the average L2 distance between the input gra- dients of the teacher and student models for all test samples, with smaller values indi- cating closer alignment. GC measures the
Table 4: Gradient alignment on CIFAR-100, using the ResNet-18 student model. The numbers in bold indi- cate enhanced gradient alignment.
BDM-AT
LTD
Method
AA
GD$
GC$
AA
GDJ
GC1
ARD
24.77
0.142
0.439
25.74
0.108 0.592
+IGDM
28.79
0.101
0.571
27.84 0.082 0.688
IAD
25.15
0.135
0.443
26.29
0.102 0.596
+IGDM
27.76
0.104
0.549
27.49
0.086 0.674
cosine similarity between these gradients, where value closer to one signifies bet-
AKD
25.09
0.127 0.438
25.37
0.113 0.584
+IGDM
28.86
0.114 0.513 28.44
0.078 0.693
ter alignment. In Table 4 IGDM improves gradient alignment with the teacher model, with improved robustness. As shown in Figure the robustness of the student model increases as GD decreases and GC rises, confirming the positive correlation be- tween gradient alignment and student model performance.
RSLAD
26.76
0.118 0.492 26.94
0.089 0.658
+IGDM 29.78
0.096 0.582 28.51 0.077 0.709
AdaAD 28.06 0.107 0.567 27.81 0.077 0.736
AdalAD 27.82 0.107 0.568
27.83
0.077 0.733
+IGDM 30.32 0.086 0.643 28.87
0.070 0.769
Point-wise Alignment Existing adversarial distillation methods have largely overlooked the im-
portance of input gradients, instead focusing on point-wise alignment. For example, RSLAD [Zi al., )02I) and AdaAD (Huang al. 2023) explicitly aim to align clean and adversarial outputs with the teacher's clean output. Thus, we reinterpret point-wise alignment in terms of input gradient matching, where IGDM demonstrates superior performance compared to existing AD methods in
achieving this alignment. We define point-wise distance D(x, 8) || (x 5) - fs + +)]. Then given x, the upper bound of D(x, 8) for sufficiently small 8 as follows.
(Df((x)
fxs
D(x, 8) < [1(T(x) - fs(x)]l +
8
(10)
Ox
8x
7

---

Published as a conference paper at ICLR 2025
AutoAttack acuracy (%6
AutoAttack acuracy (%)
30
0.837
25
25-
20
20
0.35
0.40
0.45
0.50
0.55
0.60
0.65
0.08
0.10
0.12
0.14
0.16
0.18
0.20
GC
IGDM
AD methads
Others
Regression
Figure 3: Correlation between GC and AA (le and between GD and AA (right). All results were obtained using ResNet-18 18 and BDM-AT teacher CIFAR-I00. The values for IGDM and AD methods match those in Table while 'Others' represent results from additional experiments under the same configuration. p denotes the correlation coefficient.
Hence, the better the gradient matching, the smaller the upper bound D(x, 6) becomes, aligning the teacher and student in point-wise manner.
To empirically substantiate this assertion, we compared the distances between the teacher and the student models. We calculated Dxfs(x+ +)) (r(x+-00 6)) with 1.2 distance for random noise from the uniform distribution and adversarial perturbation in Table 5 For the adversarial noise, we conducted an adversarial attack using the inner maximization loss corresponding to each method, with fixed number of steps set to 20. In all cases, we observed an improvement in alignment when using IGDM. In particular, IGDM enhanced alignment against not only adversarial noise but also random noise. This outcome demonstrates that our module contributes to point- wine alignment, improving overall robustness by significantly reducing local invariance during training, as asserted Huang al. 2023).
Table 5: Point-wise alignment with IGDM on CIFAR-100, using the ResNet-18 student model and two teacher models. The ~ indicates uniformly selected from 87[-8/20 8/255 8/255]. The numbers in bold indicate enhanced point alignment.
BDM-AT Teacher
LTD teacher
IKL-AT teacher
Method
8 ~ U
8 = adv
δ ~ U
= adv
8 ~ U
8 == adv
ARD
0,2392
0.2848
0.1822
0.2176
0.4030
0.4734
+IGDM
0.1312
0.1533
0.1301
0.1501
0.1504
0.1690
IAD
0.2374
0.2811
0.1905
0.2190
0.3939
0.4568
+IGDM
0.1342
0.1684
0.1194
0.1395
0.1524
0.1971
AKD
0.3184
0.3622
0.2017
0.2323
0.4325
0.4810
+IGDM
0.1704
0.2081
0.1156
0.1312
0.1521
0.1698
RSLAD
0.1903
0,2375
0.1686
0.2019
0.2227
0.2856
+IGDM
0.1233
0.1516
0.1219
0.1424
0.1361
0.1681
AdaAD
0.1794
0,2529
0.1348
0.2028
0.2499
0.3400
AdalAD
0.1803
0,2518
0.1331
0.2010
0.2504
0,3479
+IGDM
0.1294
0.1857
0.1253
0.1833
0.1338
0.2002
4.3 COMPARISON WITH THE STATE-OF-THE-ART METHOD: ADAAD
Simple Inner Maximization with IGDM AdaAD (Huang et al. 2023) employs teacher model for inner maximization to enhance student model robustness. However, this approach significantly increases computational overhead due to the teacher model's large architecture. contrast, IGDM achieves competitive robustness without teacher model in inner maximization, simplifying the overall training process and reducing training time. In Table 6 we compare performance with and without the teacher model for inner maximization. When AdaAD's inner maximization replaced with PGD attack on the student model alone, robustness drops significantly. However, IGDMAGAD achieves superior robustness than the original AdaAD method, with reduced training time by
8

---

Published as a conference paper at ICLR 2025
factor of three. This highlights the efficiency of IGDM, offering both robustness and flexibility by eliminating the need for a teacher model on inner maximization.
Table 6: AutoAttack accuracy (%) and computational overhead of AdaAD and IGDM AduAD with different inner loss CIFAR-100 with the ResNet-18 student. 'T/E' and 'Mem' refer to time per epoch (in minutes) and memory usage. The 'w/o Tin indicates that the inner loss is computed using a PGD attack solely the student model, without the teacher's involvement.
BDM-AT teacher
LTD teacher
IKL-AT teacher
Method
AA
T/E.
Mem
AA
T/E
MemlJ
AA
T/E.
Mem
AdaAD AdaAD w/o Tin
28.06
10.62
4711M
27.81
12.12
4900M
26.89
10.91
4711M
25.62
2.26
2063M
26.57
2.42
2634M
23.37
2.27
2063M
IGDMAdAD
30.32
11.21
4711M
28.87
13.27
4900M
29.22
11.30
4711M
IGDMAdaD w/o Tin
29.34
3.36
2611M
28.12
3.64
3187M
28.94
3.38
2611M
Comparison of IGDM and AdaAD under Same Experimental Conditions We replicated AdaAD's experimental setup using the same teacher-student model configuration for a fair compar- ison. Specifically, we used the CIFAR-I00 dataset with the LTD teacher and the CIFAR- 10 dataset with the LTD₂ teacher. (In the AdaAD paper, the LTD teacher was trained on WideResNNet-34-2 for CIFAR-10 10, so we refer a ResNet-18 student, shown in Table 7 While the AdaAD paper proposed two methods, AdaAD and AdalAD, and noted a performance gap of about percentage point, our experiments showed smaller difference. As result, we focused on applying the module specifically to AdaAD in our implementation. Notably, IGDM AdaAD consistently demonstrated a significant improvement ro- bustness accuracy. This demonstrates that even in fair comparison, the IGDM module consistently enhances adversarial robustness, underscoring its effectiveness.
LTD,, to reflect this difference.) Both setups were evaluated with
Table 7: Comparison between AdaAD paper results and our implementation results with identical experimental settings.
Result from
CIFAR-100 with LTD
CIFAR-10 with LTD,
Method
Huang et al. 2023)
Ours
Clean
AA
Clean
AA
AdaAD
62.19
26.74
85.58
51.37
AdaIAD
62.49
27.98
85,04
52.96
AdaAD
63.34
27.81
85,47
52.47
AdaIAD
63.24
27.83
85.34
52.30
IGDMAAaD
63.44
28.87
85.50
53.45
Linitations of Hyperparameter Tuning in AdaAD for Gradient Alignment AdaAD utilizes a single hyperparameter, X, to con- trol the balance between the distillation of ad- versarial and clean inputs, as À (L(Js(x + 6) lfr((x + 6)) + (1 - A) (x) However, in their implementation, À is always set one. Initially, we hypothesized that ad- justing X could facilitate the distillation of both adversarial and clean inputs, potentially align- ing the gradients. Yet, as shown in Table modifying À degraded both robustness and gradient matching. This suggests that while point- wise matching is achieved in AdaAD, simply distilling two points is insufficient for capturing the teacher's s gradients. In contrast, our approach enables effective gradient alignment and better distil- lation of the teacher model's robustness.
Table 8: Comparison between AdaAD with vari- ous hyperparameter.
Method
Clean
AA
GD4
GCf
AdaAD(A 0.25)
66.48
25.51
0.118
0.526
AdaAD(A 0.50)
65.44
27.24
0.108 0.564
AdaAD(A 0.75)
64.98
27.42
0.109
0.560
AdaAD(A 1.0)
64.43
27.82
0.107
0.568
IGDMABAD
64.44
30.32 0.086 0.643

---

Published as a conference paper at ICLR 2025
4.4 ABLATION STUDIES
In this section, we conduct more extensive experiments, including robustness against unseen attacks and distillation results with various teacher models. Here, we chose the CIFAR-100 dataset with ResNet- 18 architecture of student model.
Robustness against Unseen Attacks Table 9 we measured performance against the different attack scenarios in out-of-distribution (OOD) to demonstrate that effectively distill the robustness of a teacher model. OODRobustBench (Li et al. 2023) is designed to simulate real-world distribution shifts and evaluate adversarial robustness. focuses on types of shifts: dataset shifts (OODa) and threat shifts (OOD,), offering a more comprehen- sive assessment compared to relying solely on Au- toAttack accuracy. The term OODa encompasses natural and corruption shifts, which consist of vari- ant datasets of CIFAR-100 110 and common corruptions such noise and blur. Meanwhile, OOD, considers six unforeseen attacks, such the Recolor (Laidlaw)
Table 9: The CIFAR-100 performance, eval- uated with OODRobustBench.
Clean Acc (%)
Robust Acc (%)
Method
OODs
OOD₂
OOD;
ARD
49.92
15.70
14.91
IGDM
50.65
16.98
17.36
IAD
48.75
16.21
15.03
IGDM
51.43
18.20
17.35
AKD
49.74
15.95
14.96
IGDM
49.83
18.52
17.07
RSLAD
48.96
17.41
16.46
IGDM
50.45
19.66
18.53
Feizi, 2019) and StAdv (Xiao et al.) 2018), etc. Overall, module improves the performance of existing adversarial distillation methods against var- ious noise and unseen attacks.
AdaAD
51.53
17.77
17.32
AdaIAD
51.18
17.75
17.26
IGDMM
51.52
19.66
18.65
Adversarial Distillation with Various Teacher Model We conducted adversarial distillation using several teacher models to further validate the effectiveness of approach. We utilized multiple teacher models that were adversarially trained on the CIFAR-100 dataset. All these models are publicly available in RobustBench (Croce et 202T), and we selected them based on varying levels of robustness against AutoAt- tack (Croce & Hein, 2020a). As shown in the Figure IGDM consistently outperforms other distillation methods across different teacher se- tups. Notably, even when using less robust teachers, there is a substantial performance gap. In contrast, while AdaAD demonstrates signif- icant improvements with strong teachers com- pared other baselines, shows minimal differences when employing weaker teachers. This high- lights that our approach maintains meaningful performance advantages across all teacher models.
-60
Auwattaaa Accracy
AED
Figure 4: Performance comparison of different adversarial distillation methods across various teacher models.
5 CONCLUSION
We have proposed novel method Indirect Gradient Distillation Module (IGDM) for adversarial distillation. In contrast to conventional adversarial distillation methods that primarily focus on dis- tilling the logits the teacher model, we concentrate on distilling the gradient information of the teacher model. We obtain these gradients indirectly by leveraging the locally linear property, characteristic of adversarially trained models. Notably, IGDM can be seamlessly applied to exist- ing adversarial distillation methods. Extensive experimental results demonstrate that the student model with IGDM successfully follows the gradients of the teacher model, resulting in significantly enhanced robustness.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square at- tack: a query-efficient black-box adversarial attack via random search. European conference computer vision, pp. 484-501. Springer, 2020.
Tao Bai, Jinqi Luo, Jun Zhao, Bihan Wen, and Qian Wang. Recent advances in adversarial training for adversarial robustness. arXiv preprint arXiv:2102.01336 2021a.
Yang Bai, Yuyuan Zeng, Yong Jiang, Shu-Tao Xia, Xingjun Ma, and Yisen Wang. Improving adversarial robustness via channel-wise activation suppressing. arXiv preprint arX1v:2103.00300, 2021b.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pp. 39-57. Ieee, 2017.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. Unlabeled data improves adversarial robustness. Advances in neural information processing systems, 32, 2019.
Alvin Chan, Yi Tay, and Yew-Soon Ong. What it thinks is important is important: Robustness transfers through input gradients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 332-341, 2020.
Erh- Erh-Chung Chen and Che Che-Rung Lee. LTD: low temperature distillation for robust adversarial train- ing. CoRR, abs//l11 0102111100231,, 2021. URL https://aaxii.../////// org/ hwtpg//b////// 2111. 101111.000 02331
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In international conference machine leurning, pp. 1310-1320. PMLR, 2019.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference machine learning, pp. 2206- 2216. PMLR, 2020a.
Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with fast adaptive boundary attack. International Conference on Machine Learning, pp. 2196-2205. PMLR, 2020b.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flam- marion, Mung Chiang, Prateek Mittal, and Mathhis Hein. Robusthench: standardized adver- sarial robustness benchmark. In Thirty-fifth Conference on Neural Information Processing Sys- tems Datasets and Benchmarks Track, 2021. URL https::/openzer//// http://eeettffoteaa///eee forum?
Ad=SSxXPJCE
Jiequan Cui, Zhuotao Tian, Zhisheng Zhong, Xiaojuan Qi, Bei Yu, and Hanwang Zhang, Decoupled kullback-leibler divergence loss. arXiv preprint arXiv:2305. 13948, 2023.
Wojeiech M Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszez, and Razvan Pascanu. Sobolev training for neural networks. Advances in neural information processing systems, 30, 2017.
Nilaksh Das, Madhuri Shanbhogue, Shang-Tse Chen, Fred Hohman, Li Chen, Michael E Kounavis, and Duen Horng Chau, Keeping the bad guys out: Protecting and vaceinating deep learning with jpeg compression. arXiv preprint arXiv: 0001100009000 7755009900 2017.
Junhao Dong, Piotr Koniusz, Junxi Chen, and Yew-Soon Ong. Adversarially robust distillation by reducing the student-teacher variance gap. In European Conference Computer Vision, pp- 92-111. Springer, 2025.
Shangchen Du, Shan You, Xiaojie Li. Jianlong Wu, Fei Wang. Chen Qian, and Changshui Zhang. Agree to disagree: Adaptive ensemble knowledge distillation in gradient space. advances in neural information processing systems, 33:12345-12333 2020.

---

Published as a conference paper at ICLR 2025
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Alek- sander Madry. Adversarial robustness as a prior for learned representations. arXiv preprint arXiv:1906.0094 2019.
Micah Goldblum, Liam Fowl, Soheil Feizi, and Tom Goldstein. Adversarially robust distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3996 4003, 2020.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: arXiv: 1412.6572, 2014.
Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, and Gigel Macesanu. A survey of deep learning techniques for autonomous driving. Journal of Field Robotics, 37(3):362-386, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual net- works. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Nether- lands, October 11-14, 2016, Proceedings, Part IV 14, pp. 630-645. Springer, 2016b.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv: /503.000 025331, 2015.
Bo Huang, Mingyang Chen, Yi Wang, Junda Lu, Minhao Cheng, and Wei Wang. Boosting accuracy and robustness of student models via adaptive adversarial distillation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 24668-26222 246678-2477,, 2023.
Mingi Ji, Byeongho Heo, and Sungrae Park. Show, attend and distill: Knowledge distillation via coantion-based feature matching. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 35, pp. 7945-7952, 2021.
Gaojie Jin, Xinping Yi, Wei Huang, Sven Schewe, and Xiaowei Huang. Enhancing adversarial training with second-order statistics of weights. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15273-15283, 2022.
Gaojie Jin, Xinping Yi, Dengyu Wu, Ronghui Mu, and Xiaowei Huang. Randomized adversarial training via taylor expansion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16447-16457, 2023.
Jaewon Jung, Hongsun Jang, Jaeyong Song, and Jinho Lee. Peeraid: Improving adversarial distil- lation from a specialized peer tutor. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 24482-24491, 2024.
Jangho Kim, Minsung Hyun, Inseop Chung, and Nojun Kwak. Feature fusion for online mutual knowledge distillation. In 2020 25th International Conference on Puttern Recognition (ICPR), pp. 4619-4625 IEEE, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Huafeng Kuang, Hong Liu, Yongjian Wu, Shin'ichi Satoh, and Rongrong Ji. Improving adversarial robustness via information bottleneck distillation. Advances Neural Information Processing Systems, 36:10796-10813, 2023.
Cassidy Laidlaw and Soheil Feizi. Functional Adversarial Attacks. In Neural Information Process- ing Systems (NeurIPS), 2019.
Qizhen Lan and Qing Tian. Gradient-guided knowledge distillation for object detectors. arXiv preprint arXiv:2303.0422 2023.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 23/N, 7(7):3, 2015.
Lin Li and Michael Spratling. Data augmentation alone can improve adversarial training. arXiv preprint arXiv:2301 109.9990 2023.
12

---

Published as a conference paper at ICLR 2025
Lin Li, Yifei Wang, Chawin Sitawarin, and Michael Spratling. Oodrobustbench: benchmarking and analyzing adversarial robustness under distribution shift. arXiv preprint arXiv:2310.1279) 2023.
Xingjun Ma, Yuhao Niu, Lin Gu, Yisen Wang, Yitian Zhao, James Bailey, and Feng Lu. Under- standing adversarial attacks on deep learning based medical image analysis systems. Pattern Recognition, 110:107333 2021.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks, arXiv preprint 1rX6v:1706.06000 2017.
Javier Maroto, Guillermo Ortiz-Jiménez, and Pascal Frossard. On the benefits of knowledge distilla- tion for adversarial robustness. CoRR, httpp////....022202 2022. doi: 10.48550/ARXX122200.07755 httpht:ss://do /
48550/ arxvv wwt.:riwwwi...i........
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits natural images with unsupervised feature learning. In NIPS workshop on deep learning and featt learming, volume 2011, pp. 4. Granada, 2011.
Tianyu Pang, Xiao Yang. Yinpeng Dong, Hang Su, and Jun Zhu. Bag of tricks for adversarial training. arXiv preprint arXiv:2010.00467, 2020.
Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial robustness through local linearization. Advances in Neural Information Processing Systems, 32, 2019.
Sylvestre-Alvise Rebuffi, Sven Gowal, Dan Calian, Florian Stimberg, Olivia Wiles, and Tim- othy Mann. Fixing data augmentation improve adversarial robustness. arXiv preprint arXiv:2103.01946, 2021.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and paitern recognition, pp. 4510 4520, 2018.
Suraj Srinivas, Sebastian Bordt, and Himabindu Lakkaraju. Which models have perceptually- aligned gradients? an explanation via off-manifold robustness. Advances neural information processing systems,
2023.
Jihoon Tack, Sihyun Yu, Jongheon Jeong, Minseon Kim, Sung Ju Hwang, and Jinwoo Shin. Con- sistency regularization for adversarial robustness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8414-8422, 2022.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. arXiv preprint 18X5 12152, 2018.
Lean Wang, Lei Li, and Xu Sun. Gradient knowledge distillation for pre-trained language models. arXiv preprint arXiv:2211.0107 2022.
Ningfei Wang, Yunpeng Luo, Takami Sato, Kaidi Xu, and Qi Alfred Chen. Does physical adversarial example really matter to autonomous driving? towards system-level effect of adversarial object evasion attack. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4412 4423, 2023a.
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In International Conference on Learning Representations, 2020.
Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. In International Conference on Machine Learning (ICML), 2023b.
Zeming Wei, Yifei Wang, Yiwen Guo, and Yisen Wang. Cfa: Class wise calibrated fair adversarial training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 8193-8201, 2023.
13

---

Published as a conference paper at ICLR 2025
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust gener- alization. Advances in Neural Information Processing Systems, 33:2958- 0112998-296, 2020.
Chaowei Xiao, Jun Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially Trans- formed Adversarial Examples. In International Conference on Learning Representations (ICLR), 2018.
Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan L Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 501-509, 2019.
Zhendong Yang, Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao, and Chun Yuan. Focal and global knowledge distillation for detectors. In Proceedings of the IEEE/CVF Confer- on Computer Vision and Pattern Recognition, pp. 4643 4652, 2022. 2022.
Shenglin Yin, Zhen Xiao, Mingxuan Song, and Jieyi Long. Adversarial distillation based on slack matching and attribution region alignment. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 14605-246115 2024.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. Edwin Hancock Richard C. Wilson and William A. P. Smith (eds.), Proceedings of the British Machine Vision Conference (BMVC), pp. 87.1-87.12. BMVA Press, September 2016. ISBN 1-901725-59-6. doi: 10.5244/C. 30.87. URL httht:ss:://k doi. /t0.g//.... //.244/C..0 30.87
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning. pp. 7472-7482. PMLR, 2019.
Qingzhao Zhang, Shengtuo Hu, Jiachen Sun, Qi Alfred Chen, and 7. Morley Mao. On adversarial robustness of trajectory prediction for autonomous vehicles. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 15159-15168, 2022.
Jianing Zhu, Jiangchao Yao, Bo Han, Jingfeng Zhang. Tongliang Liu, Gang Niu, Jingren Zhou, Jianliang Xu, and Hongxia Yang. Reliable adversarial distillation with unreliable teachers. arXiv preprint arXiv:2 /06.006.000 046.0488, 2021.
Kaijie Zhu, Xixu Hu, Jindong Wang. Xing Xie, and Ge Yang. Improving generalization of ad- versarial training via robust critical fine-tuning, In Proceedings of the IÉEE/CVF International Conference on Computer Vision, pp. 1424-4334, 2023.
Yichen Zhu and Yi Wang. Student customized knowledge distillation: Bridging the gap between student and teacher. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5057-5066, 2021.
Bojia Zi, Shihao Zhao, Xingjun Ma, and Yu-Gang Jiang. Revisiting adversarial robustness distil- lation: Robust soft labels make student better. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. (6443-16455 2021.
14

---

Published as a conference paper at ICLR 2025
A FURTHER EXPERIMENTS
We conduct further experiments to corroborate our main contribution. These include distillation results on MobileNetV2 student (Appendix [..I]. distillation with IKL-AT teacher (Appendix A..). and experiments on the Tiny-ImagNet dataset (Appendix (A.3). We also analyze the drawbacks of direct gradient matching (Appendix (A.4) and role of logit difference (Appendix (A.5). Finally,
hyperparameter tuning experiments are detailed in Appendix A.6
A.1 ADVERSARIAL DISTILLATION MOBILENETV2 ARCHITECTURE
Table 10: Adversarial distillation results MobileNett2 with BDM-AT and LTD teacher models on CIFAR-100. Bold indicates cases where IGDM improved accuracy by more than 0.5 percentage points better gradient matching.
CIFAR- 100 with BDM-AT teacher
CIFAR-100 with LTD teacher
Method
Clean
PGD
AA
GD
GC1
Clean
PGD
AA
GD
GC1
PGD-AT
59.23
24.04
21.58
0.204
0.408
59.23
24.04
21.58
0.194
0.486
+IGDM
59.52
32.88
27.28
0.109
0.528
59.70
32.59
27.14
0.091
0.636
TRADES
51.05
24.83
20.62
0.132
0.377
51.05
24.83
20.62
0.119
0.460
+IGDM
57.80
29.47
22.05
0.097
0.469
56.81
29.87
23.38
0.081
0.640
ARD
60.74
29.92
24.33
0.137
0.445
60.55
30.82
25.28
0.111
0.569
+IGDM
60.83
33.55
27.50
0.107
0.534
60.44
33.36
27.59
0.082
0.681
IAD
56.35
28.96
23.43
0.129
0.430
56.11
29.55
24.22
0.106
0.547
+IGDM
57.65
31.97
25.60
0.106
0.506
58.60
31.36
25.64
0.090
0.617
AKD
60.84
29.37
24.22
0.135
0.428
60.65
29.84
24.85
0.112
0.564
+IGDM
60.39
33.62
28.04
0.104
0.538
59.94
33.70
28.12
0.077
0.715
RSLAD
61.29
31.74
26.18
0.121
0.490
60.43
32.37
26.85
0.092
0.632
+IGDM
61.48
35.22
29.32
0.101
0.560
60.36
33.99
28.00
0.079
0.696
AdaAD
61.89
29.54
23.80
0.118
0.497
61.43
30.58
25,03
0.091
0.649
AdalAD
60.83
29.94
23.83
0.116
0.499
61.22
30.78
25.22
0.088
0.652
+IGDM
61.43
33.51
27.43
0.097
0.583
61.57
33.14
27.65
0.076
0.721
Table 11: Adversarial distillation results on MobileNetV2 with BDM-AT and LTD teacher models on CIFAR-10. Bold indicates cases where IGDM improved accuracy by more than 0.5 percentage points or better gradient matching.
CIFAR-10 10 with BDM-AT teacher
CIFAR-10 with LTD teacher
Method
Clean
PGD
AA
GDJ
GC1
Clean
PGD
AA
GDI
GC1
PGD-AT
83.52
44.47
41.19
0.175
0.406
83.52
44.47
41.19
0.171
0.505
+IGDM
82.78
51.54
47.13
0.074
0.501
81.73
52.63
48.51
0.048
0.671
TRADES
81.57
50.49
46.88
0.074
0.492
81.57
50.49
46.88
0.062
0.605
+IGDM
82.33
52.94
47.70
0.050
0.502
81.82
52.96
47.47
0.039
0.645
ARD
84.38
48.26
44.02
0.091
0.371
84.18
52.16
48.11
0.054
0.578
+IGDM
84.53
52.45
45.06
0.071
0.467
84.25
54.03
49.53
0.042
0.684
IAD
83.79
48.36
44.02
0.088
0.385
83.38
52.71
48.45
0.051
0.592
+IGDM
84.27
52.42
45.08
0.063
0.497
83.65
54.57
50.14
0.038
0.704
AKD
83.75
47.15
43.43
0.093
0.368
84.46
50.84
46.96
0.057
0.566
+IGDM
83.29
53.29
48.41
0.064
0.527
83.98
54.40
49.99
0.040
0.691
RSLAD
84.69
50.94
47.38
0.074
0.491
82.60
54.55
50.44
0.040
0.706
+IGDM
84.64
52.33
49.27
0.068
0.515
82.61
55.02
51.01
0.039
0.706
AdaAD
84.51
50.67
46.56
0.069
0.516
84.24
55.47
51.36
0.034
0.777
AdalAD
85.04
51.33
47.62
0.068
0.519
84.14
55.58
51.48
0.034
0.778
+IGDM
84.75
52.35
48.34
0.065
0.526
84.27
56.15
52.15
0.033
0.790
15

---

Published as a conference paper at ICLR 2025
In Table 10 and Table I1 present additional experiments the CIFAR-10 10 and CIFAR-100 datasets using MobileNetV2 (Sandler et al., 2018) as the student model, with two distinct teacher models: LTD and BDM-AT. Across both datasets, applying IGDM consistently improved robust- ness under PGD and AttoAttack, compared to the baseline adversarial training methods. Notably, IGDM enhanced the model's performance
both CIFAR-10 and CIFAR-100, particularly in ducing the Gradient Distance (GD) and improving Gradient Cosine similarity (GC), These results underscore the effectiveness of IGDM in improving adversarial robustness and gradient alignment across different datasets and architectures.
A.2 ADVERSARIAL DISTILLATION WITH IKL-AT TEACHER MODEL.
Adversarial distillation results on ResNet-18 and MobileNetV2 with a IKL-AT teacher model on CIFAR-I00. Bold indicates cases where IGDM improved accuracy by more than 0.5 percentage points or better gradient matching.
Table 12:
CIFAR- 100 with ResNet-18 student
CIFAR-IO6 100 with MobileNetV2 student
Method
Clean
PGD
AA
GD
GC1
Clean
PGD
AA
GD
GC1
PGD-AT
55.80
19.88
18.86
0.460
0.306
59.23
24.04
21.58
0.211
0.392
+IGDM
62.91
35,25
28.89
0.122
0.534
59.75
33.59
27.02
0.120
0.517
TRADES
53.56
25.85
22.02
0.199
0.354
51.05
24.83
20.62
0.148
0.351
+IGDM
62.41
31.79
24.02
0.116
0.467
59.34
29.36
22.40
0.120
0.444
ARD
61.38
27.59
23.18
0.178
0.400
61.52
28.00
23.40
0.164
0.420
+IGDM
61.55
35.24
28.87
0.117
0.546
61.95
32.26
26.48
0.132
0.498
IAD
61.09
29.25
23.61
0.173
0.403
58.39
28.40
23.06
0.147
0.418
+IGMM
64.20
33.71
27.03
0.132
0.506
59.58
30.96
25.42
0.128
0.478
AKD
61.90
29.23
23.96
0.171
0.412
61.69
27.87
23.60
0.159
0.431
+IGDM
61.94
33.16
27.82
0.131
0.511
61.56
33.62
27.32
0.117
0.519
RSLAD
61.18
30.54
25.27
0.147
0.444
61.95
30.16
25.09
0.157
0,441
+IGDM
63.55
35.26
29.10
0.119
0.541
62.48
35.25
28.82
0.118
0.547
AdaAD 65.36
32.29
26.89
0.133
0.525
62.35
28.51
23.01
0.140
0.470
AdaIAD
65.29
32.47
26.80
0.133
0.527
62.17
28.48
23.39
0.139
0.473
+IGDM
66.00
34.47
29.22
0.119
0.582
62.02
32.85
26.11
0.119
0.533
Table 13:
Adversarial distillation results on ResNet-18 and MobileNetV2 with a IKL-AT teacher model on CIFAR-IO. Bold indicates cases where IGDM improved accuracy by more than 0.5 per- centage points or better gradient matching.
CIFAR- 10 with ResNet-18 18 student
CIFAR-10 with MobileNetV2 student
Method
Clean
PGD
AA
GDJ
GC1
Clean
PGD
AA
GDI
GC
PGD-AT
84.52
42.80
41.12
0.409
0.286
83.52
44.47
41.19
0.175
0.401
+IGDM
83.66
54.38
49.85
0.072
0.541
82.11
51.13
47.00
0.063
0.507
TRADES
82.46
49.13
47.09
0.131
0.425
81.57
50.49
46.88
0.073
0.489
+IGDM
83.14
54.91
48.76
0.098
0.455
81.67
52.32
47.40
0.057
0.492
ARD
85.41
49.36
45.32
0.087
0.389
84.68
48.03
44.11
0.087
0.376
+IGDM
85.78
54.37
46.52
0.072
0.467
84.72
50.89
46.13
0.070
0.473
IAD
85.22
49.70
45,96
0.083
0.393
84.25
48.58
44.15
0.082
0.390
+IGDM
83.78
54.77
50.24
0.061
0.521
83.89
51.46
46.48
0.069
0.458
AKD
83.62
51.81
46.01
0.080
0.395
84.52
47.56
43.27
0.154
0.421
+IGDM
83.70
53.69
49.45
0.067
0.539
84.33
50.36
45.37
0.089
0.438
RSLAD
85.55
52.13
48.83
0.069
0.53
84.62
51.43
47.69
0.072
0.503
+IGDM
85.67
53.30
49.58
0.065
0.541
84.30
53.14
48.91
0.066
0.523
AdaAD
86.59
54.25
50.86
0.062
0.570
85.31
52.10
48.03
0.066
0.524
AdalAD
86.57
54.59
50.61
0.062
0.572
85.47
52.05
47.96
0.066
0.523
+IGDM
86.10
55.41
51.29
0.060
0.587
85.11
53.07
49.08
0.061
0.541
16

---

Published as a conference paper at ICLR 2025
In Table 12 and Table 13 conduct additional experiments the CIFAR-100 and CIFAR-10 datasets with the IKL AT Cui et al. 2023) teacher model. Similar to the outcomes observed with the LTD (Chen & Lee 2021 and BDM-AT (Wang et al. 2023b) teachers those datasets, IGDM demonstrates substantial enhancements in robustness with the IKL-AT teacher, regardless of the original methods, datasets, student models, while maintaining consistent clean accuracy. Further- more, consistent results are observed across all experiments, with a decrease in GD and an increase GC upon applying IGDM.
A.3 ADVERSARIAL DISTILLATION TINY-IMAGENET
Adversarial distillation result on PreActResNet-I8 with a WideResNat-34-10 teacher model on Imageen. Bold indicates cases where IGDM improved accuracy by more than 0.5 percentage points or better gradient matching.
Table 14:
Method
Clean
PGD
AA
GD
GC1t
PGD-AT Madry et al.
2017)
50.13
17.10
14.58
0.126
0.490
+IGDM
51.46
25.04
20.10
0.071
0.672
TRADES (Zhagg al.
2019
46.51
19.51
14.89
0.094
0.447
+IGDM
51.31
24.10
18.92
0.073
0.649
ARD (Goldblum al.
2020
50.81
23.37
19.48
0.081
0.613
+IGDM
50.86
25.39
20.57
0.069
0.683
IAD (Zhu et al.
2021
49.89
23.89
19.10
0.079
0.614
+IGDM
50.02
25.57
20.69
0.066
0.704
AKD (Maroto al 2022
51.70
23.86
19.69
0.082
0.612
+IGDM
51.38
25.25
20.99
0.063
0.720
RSLAD (Zi et al. 2021
47.12
22.14
17.65
0.081
0.567
+IGDM
47.54
23.95
18.35
0.074
0.620
AdaAD (Huang et al.
2023
51.54
24.65
20.60
0.064
0.738
+IGDM
51.21
25.73
21.17
0.060
0.764
The results in Table 14 confirm the effectiveness of IGDM in enhancing adversarial robustness, especially on the challenging Imag-lm dataset & Yang (2015). Consistent with findings on other datasets, IGDM improves robustness metrics such as PGD and AA accuracy, while maintaining comparable clean accuracy. This improvement is particularly significant given the complexity of Imaeeem,,,e,, where achieving high robustness is often challenging. Moreover, IGDM facilitates better alignment between teacher and student models, evidenced by reduction in mean Gradient Distance (GD) and increase in mean Gradient Cosine similarity (GC), further underscoring the impact of gradient matching of our method.
A.4 DRAWBACKS OF DIRECT GRADIENT MATCHING IN CONTRAST TO IGDM
IGDM achieves gradient matching indirectly by leveraging the differences logits. One might assume that distilling directly calculated gradients could achieve gradient matching more intuitively, as expressed by the following equation:
Lpirect = T(a) D @ff((x)
0fr(x) Ox
(1I)
&x
In Table 15 we conduct experiment on IGDMARD and direct gradient matching (DirectARD). To ensure fair comparison between the two methods, we kept all other factors the same, varying only the gradient calculation method (logits differences direct gradient distillation fails to meaningfully enhance robustness compared to the original ARD method. Although it shows slight enhancement in gradient matching, its performance significantly lags behind IGDM.
direct calculation). The results reveal that
To provide comprehensive interpretation, analyze GD and training loss training time, epoch, and step in Figure The limitations direct gradient matching are evident due to optimiza- tion challenges; small fails to match the gradient, while large results in poor convergence of
17

---

Published as a conference paper at ICLR 2025
Table 15: Adversarial distillation result of ARD variant methods on ResNet-18 with BDM-AT
teacher on CIFAR-100. DIreCtARD represents the direct distillation of gradients, and the a is the hyperparameter of the gradient distillation loss.
Method
Clean
PGD
AA
GD
GC1
DirectARD (
60.92
30.56
25.23
0.136
0,447
DirectARD a
60.98
30.29
24.88
0.137
0.444
DIReCtARD a 6)
61.13
30.34
25.27
0.135
0,448
ARD (a 10)
61.05
30.80
25.33
0.132
0.453
DIR a 30)
61.43
30.45
25.29
0.127
0.455
ARD a 60)
61.14
30.17
24.73
0.124
0.450
DireCtARD a 100)
60.36
30.48
24.62
0,121
0,445
DRD a 300)
59.63
29.72
23.99
0.116
0,440
DiRectARD a 600)
58.79
29.76
24.06
0.111
0,441
AIRECTARD a 1000)
57.08
29.89
24.14
0.108
0.445
DIreCTARD a 3000)
$1,30
29.13
22.85
0.104
0.441
ARD 6000)
46.95
27.33
20.70
0.105
0.424
DIDECTARD a = 10000)
43.22
25.67
19.18
0.105
0.404
ARD (Goldblum al. 2020
61.51
30.23
24.77
0.142
0.439
+IGDM
61.62
35.75
28.79
0.101
0.571
the training loss. More interestingly, even with large a, direct matching results in GD values higher than those achieved by IGDM. We interpret these optimization issues arising from the low-level features of the input gradient, similar to the input itself. In other words, directly matching low-level feature gradients is comparable to matching pixel-wise image differences. This leads to the model's inability to effectively capture the training objective, failing to match the gradient and, consequently, diminished robustness. One potential solution to address these optimization challenges is to employ an additional discriminator model, as proposed in (Chan et al. 2020). However, this approach re- quires training additional discriminator model, whereas IGDM does not necessitate any other model or training procedure. In summary, IGDM significantly enhances robustness by indirectly matching the gradient using high-level feature logits, thereby achieving superior robustness.
IGDM
Direct(a=100)
Direct (Ø=10000)
prd Noma A/ll you Willi wn
0.125
1.0
0.12
Training Loss
0.8
GD
0.115
0.6
0.11
0.4
0.105
0.2
100
150
200
Training epoch
Training step
Figure Comparision between IGDM and direct gradient matching. GD and training loss are measured at the training epoch and step, respectively.
18

---

Published as a conference paper at ICLR 2025
A.5 IMPORTANT ROLE OF LOGIT DIFFERENCE IN IGDM
Adversarial training aims to make the network output almost constant within a ball around the input point. Therefore, might assume that the distilling logit difference of the adversarially trained teacher model in IGDM loss close to distilling O. In other words, IGDM can be simplified as expressed by the following equation:
LIGDMMADES iiz T(a) ((fs(x + -
0)
(12)
T(a) D(fs(x 6). fs(x-5)),
We refer to this method as IGDMTRADES-ING because it resembles a TRADES Zhang et al., 2019) regularization loss (TRADES,g). and empirically, its performance closely aligns with the addition of the TRADES regularization loss:
TRADES T(a) -(fe))(f((++))) fs(x)).
(13)
Table 16: Adversarial distillation result of ARD variant methods on ResNet- 18 with BDM-AT teacher on CIFAR-100. TRADES represents the regularization loss of TRADES. and the a is the hyperparameter of the gradient distillation loss.
Method
Clean
PGD
AA
GD
GC1
ARD TRADES reg
57.86
31.44
25.95
0.125
0.459
ARD + TRADES reg (t 5)
58.07
32.13
26.06
0.117
0.466
ARD TRADES rey a 10)
53.58
32.01
26.27
0.116
0.463
ARD + TRADES reg a 15)
52.05
31.53
26.23
0.115
0.464
ARD TRADES reg a 20)
50.87
31.92
26.30
0.115
0.462
ARD + TRADES reg 25)
50.28
31.92
26.15
0.115
0.458
ARD TRADES, a 30)
49.20
31.93
25.87
0.115
0.456
ARD + IGDM ARAEE- ( =
57.73
31.72
25.64
0.119
0.461
ARD IGDM TR ARRAAEES.AAR like (a 5)
54.18
31.71
25.79
0.117
0.466
ARD + IGDM TR like
= 10)
52.21
32.08
25.98
0.118
0,464
ARD IGDM TRADES- like a 15)
50.50
31.85
25.91
0.118
0.457
ARD + IGDM like (a =20)
49.67
31.76
25.78
0.118
0,452
ARD + IGDM TRADES like 25)
48.21
31.33
25.38
0.119
0.445
ARD IGDMTRADES TRADES like (a 30)
47.61
31.04
25.05
0.119
0.439
ARD Goldblum et 2020
61.51
30.23
24.77
0.142
0.439
+IGDM
61.62
35.75
28.79
0.101
0.571
In Table 16 we conduct experiments on these two methods conjunction with the ARD (Gold- blum et al. 2020) method. For TRADES the regularization term strengthens, the robustness slightly increases; however, it lags far behind IGDM in terms of robustness. IGDMTRADES-HL fol- lows similar trend the TRADES method with the increase in hyperparameter. These results indicate that the logit difference of the teacher model provides valuable information for teaching the student model achieve robustness through gradient matching, resulting in superior robustness. This finding corroborates our main contribution in the main paper.
A.6 ANALYSIS ON HYPERPARAMETER SENSITIVITY OF IGDM
The proposed IGDM loss is defined as follows:
LIGDM = T(a) DD-DDDD DJs((x 8) fs(x- 8), fr(x + fr ())
Table 17 and Table [8] we analyze the sensitivity hyperparameters a using IGDMPGD-AT and IGDMARD. In these tables, value of 0 for represents the original methods, PGD-AT and ARD, respectively. As a increases, gradient matching becomes more prominent, leading to enhanced robustness for both methods. However, excessively large values of do not provide additional enhancements and may even reduce robustness. Based on the insights from the hyperparameter sensitivity analysis, employed grid search adapt the IGDM module for all experiments.
19

---

Published as a conference paper at ICLR 2025
Table 17: Analysis of hyperparameter of IGDM using PGD-AT variant methods on ResNet-18 with BDM-AT teacher on CIFAR-100. is the hyperparameter of IGDM loss. The gray row indicates the reported values.
Method
Clean
PGD
AA
GDU
GCB
PGD-AT (Madry al.
2017
55.80
19.88
18.86
0,452
0.389
IGDMPaD-Ar(a 20]
56.95
28.10
24.02
0.157
0,427
IGDMPGD-Ar(a 40)
58.23
30.36
25.71
0.139
0.457
IGDMPGDAATE AT(a 60)
58.74
31.69
26.57
0.131
0,483
IGDM PTDMAT(TTTTC AT(a 80)
59.01
32.02
27.08
0.123
0.504
IGDM PT (a 100)
59.20
33.09
27.48
0.118
0.519
PG = 120)
59.42
33.60
27.95
0.114
0.531
IGDMPGD-AT(E 140)
59.32
34.18
28.26
0,111
0.542
PUDMPUD-ATCC (a 160)
60.06
34.38
28.47
0.108
0.550
IGDM 180)
60.53
34.47
28.51
0.105
0.556
IGDMPGD AT 200)
60.83
34.90
28.84
0.104
0.562
IGDMPGD (a 220)
60.68
35.09
29.16
0.102
0.569
GGDMPGD-ATC (a -44
59.65
35.34
29.10
0.101
0.572
IGDMPGD-ATC -260)
59.71
35.37
29.09
0.100
0.573
PTDMPGD-ATC = 280)
59.83
35,23
28.84
0.099
0,576
IGDMPGD-AT(a 300)
59.95
35.39
29.13
0.098
0.579
Table 18: Analysis of hyperparameter of IGDM using ARD variant methods on ResNet-18 with BDM-AT teacher on C1FAR-100. a is the hyperparameter of IGDM loss. The gray row indicates the reported values.
Method
Clean
PGD
AA
GDI
GC1
ARD Goldblum et al.
2020
61.51
30.23
24.77
0.1422
0.439
GGDMARD(Q = 20)
59.74
33.97
27.85
0.112
0,523
IGDM (R =40)
60.13
34.21
28.56
0.106
0.546
(GDMARD(C = 600)
61.08
34.54
28.42
0.104
0.557
IGDMARD(a 80))
60.78
35.21
28.73
0.102
0.568
IGDMARD(( 100)
61.18
35.75
28.79
0.101
0.571
IGDMARD(Q 120)
61.15
35.17
28.67
0.099
0.578
IGDMARD = 140)
61.24
35.32
28.75
0.098
0.583
@GDMRRDC 160)
60.76
35.81
28.71
0.096
0.584
IGDM (R 180)
60.19
35.67
28.78
0.095
0.585
IGDM ARD (a 200)
59.95
35.19
28.67
0.095
0.585
IGDM 220)
59,42
35.24
28.76
0.096
0.588
IGDM (a 240)
59.76
34.78
28.74
0.095
0.587
IGDMARD (a 260)
59.04
35.01
28.36
0.095
0.586
IGDM 280)
59.56
35.21
28.59
0.095
0.589
IGDMARD(Q 300)
58.91
34.78
28.58
0.095
0.582
B TRAINING DETAILS
B. SETTINGS
We utilized the CIFAR-10, CIFAR-100 (Krizhevsky al. 2009). SVHN Netzer et al. (2011), and Tiny- ImageNet (Le & Yangl 2015) datasets with random crop and random horizontal flips, exclud- ing other augmentations.
trained all AT, AD, and IGDM incorporated methods using an SGD momentum optimizer with the same initial learning rate of 0.1, momentum of 0.9, and weight decay of 5e-4.
For CIFAR-10 and CIFAR-100, adhered to the training settings of other adversarial distillation methods, training for 200 epochs, except for RSLAD and IGDMRSLAD. which were trained for 300 epochs. RSLAD suggested that increasing the number of training epochs could enhance model
20

---

Published as a conference paper at ICLR 2025
robustness; thus, we followed their recommendation to train for 300 epochs. The learning rate scheduler reduced the learning rate by a factor of 10 at the 100th and 150th epochs. However, for RSLAD and IGDMRSLAAD the learning rate decreased by 10 at the 215th, 260th, and 285th epochs, as suggested in the original paper. For SVHN, we trained for 50 epochs with the learning rate decayed by factor of 10 at the 40th and 45th epochs for all methods. For Imay--m,geNet, we trained for 100 epochs with the learning rate decayed by factor of 10 at the 50th and 80th epochs for all methods.
The adversarial perturbation settings were as follows: the number of iterations for inner maximiza- tion was set to 10, with step size of 2/255, and a total perturbation bound of Loe 8/255. Specifically, we employed the recommended inner loss functions as outlined in the original paper: PGD attack on student model for PGD-AT, ARD, and IAD; TRADES attack on student model for TRADES; RSLAD inner loss for RSLAD;; and AdaAD inner loss for AdaAD, Moreover, the IGDMM incorporated method followed the inner maximization method of the original AT or AD method. For formulaic representation as follows,
PGD attack: 8 arg max CE(fs(x 0'),
(14)
TRADES attack:
arg max KL[Js(x + 8'), Sc
(15)
RSLAD attack: arg max [T((f(x + (T()), Se
(16)
AdaAD attack: 8 arg max (L(fs(( 8'), fx + 8'))
(17)
Here, CE represents cross-entropy loss, KL represents KL-divergence loss.
B.2 HYPERPARAMETER
The parameters of AT, AD, and the AD component in IGDM -icoooooooportted methods were strictly set to the value suggested in the original paper. On the other hand, parameter of IGDM varied depend- ing on the original AT or AD method, dataset, teacher model, and student model. We experimentally employed an increasing hyperparameter function, defined as T(a) = varied in each training scenario. The following paragraphs provide detailed information on the in each experimental setting, determined through grid search results. IGDMAdaDD and IGDM have same value of hyperparameter in all cases.
Current Fpach
Total Total Epochs ac, and the a value
B.2.1 RESNET- 18 MODEL TRAINED ON CIFAR- 100 DATASET
For IGDMPGD-AT. to 100 for LTD and IKL-AT, 220 for BDM-AT. For IGDMTRADES. 5 for all teach- ers. For IGDMARD. 50 for LTD and 100 for BDM-AT and IKL-AT. For IGDMAAD. 20 for LTD and 50 for BDM-AT and IKL-AT. For IGDMAKD. 50 for LTD, 25 for BDM-AT, and 40 for IKL-AT. For IGDMRSLAD 3 for LTD and 10 for BDM-AT and IKL-AT. For IGDMAdRAD. 15 for LTD. 70 for BDM-AT, and 50 for IKL-AT.
B.2.2 RESNET- 18 MODEL TRAINED ON THE CIFAR-10 DATASET
For IGDMPGD-AT- 60 for LTD and 70 for BDM-AT and IKL-AT. For IGDMTRADES. 20 for all teach- ers. For IGDMARD. 5 for LTD and 10 for BDM-AT and IKL-AT. For IGDMIAD. 50 for LTD and 30 for BDM-AT and IKL-AT. For IGDMARD. for LTD, 10 for BDM-AT, and 15 for IKL-AT, For IGDMRSLAD- 0.9 for LTD and BDM-AT, and I for IKL-AT. For IGDM 5 for LTD and IKL-AT, and 10 for BDM-AT.
B.2.3 MOBILENETV2 MODEL TRAINED ON CIFAR- DATASET
For IGDMFGD-AT- 150 for LTD, 160 for BDM-AT, and IKL-AT. For IGDMIRADES: 10 for LTD and BDM-AT, and for IKL-AT. For IGDMARD. 70 for all teachers. For IGDMADD 20 for LTD and 50 for BDM-AT and IKL-AT. For IGDMAKD- 30 for all teachers. For IGDMRSLAD. for LTD, 4 for BDM-AT, and 10 for IKL-AT, For IGDMALADD 20 for LTD, 50 for BDM-AT, and 40 for IKL-AT.
21

---

Published as a conference paper at ICLR 2025
B.2.4 MOBILENETV2 MODEL TRAINED ON CIFAR-IP 10 DATASET
For IGDMPGD-AT. 70 for LTD, 40 for BDM-AT, and 50 for IKL-AT. For IGDMTRADES. for LTD, 0.5 for BDM-AT, and 0.2 for IKL-AT. For ARD. 10 for LTD, 8 for BDM-AT, and for IKL-AT. For IGDMAD, 50 for all teachers. For IGDMAKD. 15 for all teachers. For IGDMRSLAD. 0.5 for LTD and 0.7 for BDM-AT and IKL-AT. For IGDMAMAD, 10 for all teachers.
B.2.5 RESNET-18 MODEL TRAINED ON THE SVHN DATASET
We set 60 for IGDMPGD-AT. 30 for IGDMMRADDS. 50 for IGDMARD. 40 for IGDMAD 10 for IGDMAKD. for IGDMRSLAD. and for IGDMAdaAD-
B.2.6 PREACTRESNET- 18 MODEL TRAINED ON TINY-IMAGENET DATASET
We set 30 for IGDMPGD-AT. 20 for IGDMIRADES. 10 for IGDMARD. 20 for IGDMAAD 10 for AGDMAKD, for IGDMRSLAD, and 10 for IGDMAdaAD-
C MAIN ALGORITHM
Algorithm Main Algorithm
Input:
Robust teacher model fr. inner loss (Lmar) and outer loss (LAD) of base AT or AD method, and batched training dataset {(x, y)} with batch size. Robust stduent model fs
Output:
Randomly initialize 0, the weights of ss repeat
8 arg max (Lmax (x + +881))) Se
LIGNM (x, 8) T(a) D(Ss(x+ fs(x- 8), fr(x + +6 -T(x 8))
Lmin (x. v) = LAD(X, y) + LIGDM (x, 8)
0 0 Σラ= VoLmin (X,, Us)
until training converged
22


Reviews:
[{"review_id": "t2gShiEULK", "reviewer": "Reviewer_ECU4", "summary": "This paper aims to improve the adversarial robustness of lightweight or small models using adversarial distillation. Unlike directly using the teacher's logits as a guide, the authors focus on distilling the input gradients of the teacher model to achieve point-wise alignment between the teacher and student. The proposed IGDM can integrate with other AD methods and has demonstrated consistent performance improvements across different datasets.", "questions": "See Weaknesses", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Aligning the input gradients of the teacher and student is an insightful and interesting approach.\n\n2. IGDM is a simple and effective method that can be easily applied as a regularization term to other AD methods, achieving consistent performance improvements.\n\n3. Comprehensive experimental results and ablation studies demonstrate the effectiveness of IGDM.\n\n4. The paper is well-written and easy to follow.", "weaknesses": "Overall, this paper is logically coherent and well-argued. The effectiveness of IGDM is based on the assumption of the locally linear property of adversarial robust models, and preliminary experiments in Section 3.1 support this assumption. However, the data and models used in the experiments are relatively simple. Validating the effectiveness of IGDM on more general models, such as ViT or CLIP, and on more complex tasks, such as detection and segmentation, could provide more universal insights for future research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JocIC6xTBi", "reviewer": "Reviewer_avUv", "summary": "The paper focuses on improving the adversarial robustness of DNNs through adversarial distillation. Specifically, it addresses the challenge that smaller models, which are preferred for their computational efficiency, lag behind larger models in terms of robustness against adversarial attacks. The authors propose the Indirect Gradient Distillation Module (IGDM), which aims to transfer the input gradient knowledge from a robust teacher model to a student model, indirectly matching the student’s input gradient with the teacher's.  The experimental results demonstrate the effectiveness of the proposed approach in significantly improving adversarial robustness.\n\n\n\nBased on the detailed response by the authors, I decided to improve my rating!", "questions": "1. I am confused about Figure 1(a) in the paper. In the distillation architecture, the teacher model is already trained, while the student model is not. In the feature space, the manifolds of the models are different. Even if the directions of the gradients are the same, the manifold spaces are inconsistent, and the paths they take should also be different. Therefore, the adversarial samples of the teacher model and the student model cannot be consistent. One way to prove this is to use the adversarial samples generated by the teacher model and the student model to attack another model to see if the attack effects are consistent.\n\n2  In Figure 2, in adversarial training and adversarial distillation, why is it that in the early stages of training, when the student model should not be robust yet, why does the remainder proportion of the model not increase?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors propose a novel distillation module called the Indirect Gradient Distillation Module (IGDM) and provide a theoretical foundation for why gradient matching through output differences works, leveraging the local linearity of adversarially trained models. But, I have some doubts about the theoretical assumptions here. The fragility of neural networks is attributed to the linear properties of neural networks[1], and the paper assumes this local linear property to solve the gradient, which is contrary to the conclusions of previous related research. So, does this method really simulate the gradient of the network?\n\n2. Extensive experiments across different datasets, student models, and teacher models validate the effectiveness of IGDM in improving robustness against various attack scenarios.\n\n[1] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. ICLR, 2015.", "weaknesses": "1.  Although the paper did a lot of experiments, I think it is not fair enough. For example, the teacher model used in the paper is usually better than the teacher models of other methods. So if other methods also use the same teacher model, can they also achieve good robustness?\n\n2. Line 160 of the paper mentions that using gradients directly is difficult, which is important for the motivation of the paper, so it needs to be reflected in the main context rather than being placed in the Appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c2SJ0vKX7U", "reviewer": "Reviewer_yGhV", "summary": "This paper proposes a new method named Indirect Gradient Distillation Module (IGDM) for adversarial distillation. Different from the traditional adversarial distillation methods that mainly focus on distilling the logits of the teacher model, this method emphasizes distilling the gradient information of the teacher model. By taking advantage of the local linear characteristics of the adversarially trained model, these gradients are obtained indirectly, thereby significantly enhancing robustness.", "questions": "Please check the details in weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The author obtains the gradients indirectly by taking advantage of the local linear characteristics of the adversarially trained model, thereby significantly enhancing robustness.\n2. The modular design of IGDM enables it to be easily integrated with existing adversarial distillation methods.\n3. Through a large number of experimental results, the author has verified that IGDM can successfully enhance the robustness of the existing adversarial distillation methods.", "weaknesses": "The loss function of the Indirect Gradient Distillation Module (IGDM) proposed in this paper is derived from the first-order Taylor function expansion, which is similar to the situation of making the training loss become low-curvature [in d - f]. In fact, the research works in [e, f] also used similar techniques, but they mainly studied the problems in traditional adversarial training. If the teacher network in this paper is changed to only input natural images, the proposed loss in the paper may degenerate into an evolutionary target of the aforementioned works. Therefore, the author needs to conduct a comparative analysis between their method and the existing works that are also based on the first-order Taylor function expansion, and analyze the differences. Moreover, what will be the result if the regularization loss of the existing works is added to the adversarial distillation target?\n\nThe authors utilize the internal loss to generate adversarial perturbations. It would be better to elaborate on its formulaic representation in the main paper. Is the cross-entropy used directly? How much impact do different internal losses have on the results? What would be the effect if the adversarial perturbations are generated using Equation 9?\n\nSome of more recent works, that are about the adversarial distillation, are missing, such as [a-c]. \n\na. Adversarially Robust Distillation by Reducing the Student-Teacher Variance Gap. ECCV 2024\n\nb. PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutor. CVPR 2024\n\nc. Improving Adversarial Robustness via Information Bottleneck Distillation. NeurIPS 2023\n\nd. Efficient Training of Low-Curvature Neural Networks. NeurIPS 2022.\n\ne. Adversarial Robustness through Local Linearization. NeurIPS 2019.\n\nf. Relating Adversarially Robust Generalization to Flat Minima. ICCV 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tGkx0vBTJ7", "reviewer": "Reviewer_Pxd3", "summary": "The paper presents an Indirect Gradient Distillation Module (IGDM) for adversarial training via adversarial distillation methods. \nIt claims to match the input space gradient between teacher and student models with knowledge distillation. The main claim is \"indirect\" gradient matching/alignment compared to direct gradient alignment, which is available in the literature. Central to the paper is equation 2 (first-order Taylor expansion of adversarially trained models) and equation 8, which is the formulation of IGMD. The paper performs an exhaustive list of experiments. However, there is no indication of making code available for reproducibility is indicated in the paper.", "questions": "Figure 2a says that the \"remainder proportion\" of the adversarial robust teachers is shown in Table 1. However, One would wonder, comparing Figure 2a and Table 1, how to make sense of these two different metrics. Figure 2a y-axis is \"proportion,\" Table 1 has clean accuracy and attacked accuracy values. How can one calculate \"proportion\" from Table 1 to make sense of Figure 2a?\n\nIt has not been demonstrated how preparation values 0.012 and 0.016 have arrived. It is not clear if the proposition nose value is the accuracy value. From the equation, it appeared to be a noise term, but making reference to Table 1 makes one feel it is an accuracy term.\n\nFigure 2b also under explains as to how one can relate Equation 2 and Figure 2b. \n\nThe paper claims to do an \"input gradient\" matching Figure 1 and Line 151. However, the gradient matching is done via the discrepancy between student and teacher models model outputs. It is unclear how this method differs from other methods that do gradient distillation.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main strength of the paper is the extensive set of experiments on Indirect Gradient Distillation Modules (IGDM) for adversarial training via adversarial distillation methods. The IGDM is use used as an additive loss for adversarial knowledge distillation (Equation 9), which is claimed to be the first Indirect gradient matching compared to direct gradient matching.", "weaknesses": "The paper does not make it clear how this way gradient matching is \"indirect.\" From Figure 1 and Figure 2 and the related explanation, it is not clear why this is \"indirect\" gradient distillation.  There is no comparative picture presented to clarify the comparison/contrast between  Indirect Gradient  and Direct Gradient distillation. \n\nExcept for Table 7, It is not explicitly mentioned or clear that they run the other (SOTA) algorithms themselves or that the results were taken from the reported results. If experiments were re-run, other hyperparameter setting descriptions were not given or not referred to which hyperparameter setting were used, e.g., home many epoch os training.\n\nThere are some inconsistencies in making results bold for best results in some tables and not in some tables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)