Reliable Adversarial Distillation with Unreliable Teachers
Authors: Jianing Zhu
Jiangchao Yao
Bo Han
Jingfeng Zhang
Tongliang Liu
Gang Niu
Jingren Zhou
Jianliang Xu
Hongxia Yang
Venue: ICLR 2022 Poster (2022)

Abstract: In ordinary distillation, student networks are trained with soft labels (SLs) given by pretrained teacher networks, and students are expected to improve upon teachers since SLs are stronger supervision than the original hard labels. However, when considering adversarial robustness, teachers may become unreliable and adversarial distillation may not work: teachers are pretrained on their own adversarial data, and it is too demanding to require that teachers are also good at every adversarial data queried by students. Therefore, in this paper, we propose reliable introspective adversarial distillation (IAD) where students partially instead of fully trust their teachers. Specifically, IAD distinguishes between three cases given a query of a natural data (ND) and the corresponding adversarial data (AD): (a) if a teacher is good at AD, its SL is fully trusted; (b) if a teacher is good at ND but not AD, its SL is partially trusted and the student also takes its own SL into account; (c) otherwise, the student only relies on its own SL. Experiments demonstrate the effectiveness of IAD for improving upon teachers in terms of adversarial robustness.

Published as a conference paper at ICLR 2022
RELIABLE ADVERSARIAL DISTILLATION WITH UNRELIABLE TEACHERS
Jianing Zhu'
Jiangchao Yao?
Bo Han
Jingfeng Zhang' Jianliang Xu'
Tongliang Liu
Gang N Jingren Zhou?
Hongxia Yang²
Hong Kong Baptist University 2 Alibaba Group SRIKEN Center for Advanced Intelligence Project "The University of Sydney
{esjnzhu, bhanml, xujl} @ @coop.hkbu.eoe (jiangchao.y] yje, singrenzhho,, vaxg-yhx) @alibaba-inc.com rrgfeng.zhang @riken..pp tongliang. liu @sydney.edu.au gang.niu.ml gang.niu.l@ggail.comm
ABSTRACT
In ordinary distillation, student networks are trained with soft labels (SLs) given by pretrained teacher networks, and students are expected to improve upon teachers since SLs are stronger supervision than the original hard labels. However, when considering adversarial robustness, teachers may become unreliable and adversar- ial distillation may not work: teachers are pretrained on their own adversarial data, and is too demanding to require that teachers are also good at every adversarial data queried by students. Therefore, in this paper, we propose reliable introspective adversarial distillation (IAD) where students partially instead of fully trust their teachers. Specifically, IAD distinguishes between three cases given a query of a natural data (ND) and the corresponding adversarial data (AD): (a) if a teacher is good at AD, its SL fully trusted; (b) if a teacher good ND but not AD, its SL is partially trusted and the student also takes its own SL into account; (c) otherwise, the student only relies on its own SL. Experiments demonstrate the effectiveness of IAD for improving upon teachers in terms of adversarial robustness.
INTRODUCTION
Deep Neural Networks (DNNs) have shown excellent performance range of tasks in computer vision (He et al. 2016) and natural language processing (Devlin et al. 2019). Nevertheless, Szegedy cet al. 20144); Goodfellow et al. (2015) demonstrated that DNNs could be easily fooled by adding small number of perturbations on natural examples, which increases the concerns on the robustness of DNNs in the sustworthy-sensitii areas, e.g., finance (Kumar et al. (2020) and autonomous driving (Litman] 2017). To overcome this problem, adversarial training Goodffllow et al., 2015)
Madry al. 2018) proposed and has shown effectiveness to acquire the adversarially robust DNNs.
Most existing adversarial training approaches focus on learning from data directly. For example, the popular adversarial training (AT) (Madry al.) 2018) leverages multi-step projected gradient descent (PGD) to generate the adversarial examples and feed them into the standard training. Zhang al. 22019) developed TRADES on the basis of AT to balance the standard accuracy and robust performance. Recently, there are several methods under this paradigm are developed to improve the model robustness (Wang al. 2019 Alayrac et al. 2019 | Carmon et al.) 2019 Zhang al.. 2020) Jiang et al. 2020 Ding et al.) 2020 Du et al. 2021| Zhang et al. (021). However, directly learning from the adversarial examples is challenging task on the complex datasets since the loss with hard labels is difficult to be optimized (Liu et al. 2020), which limits us to achieve higher robust accuracy.
To mitigate this issue, one emerging direction is distilling robustness from the adversarially pre- trained model intermediately, which has shown promise in the recent study (Zi et al. 2021 Shu et al.) 202T). For example, lyas et al. (2019) used an adversarially pre-trained model to build a "robustified" dataset to learn robust DNN. Fan et al. (022) Salman et al. (2020) explored to boost the model
Corresponding author (bhanml @comp www.e/owww......u...ooo

---

Published as a conference paper at ICLR 2022
Outgut
Advenarial Output
Adhvessaiial Output
Output
64-
On student models
On natura ditt
Beacher Guidence
Minic
Partially
Sey-sstroopectoo
Ninie
62-
Teachor
Student
Student
Itself
Natural data
Naturall data
25
50 75
100
125
156
175 200
Epocts
(a) Unreliability issue
(b) Overview Introspective Adversarial Distillation (IAD)
Comparison of the teacher model's s accuracy on the natural data and the student model's adversarial data from CIFAR-IO datasets. Different from the ordinary distillation in which the natural data is unchanged and the teacher model has the consistent standard performance, the teacher model's robust accuracy on the student model's adversarial data is decreasing during distillation, which means the guidance of the teacher model in adversarial distillation is progressively unreliable. (b) Overview of Introspective Adversarial Distillation (IAD): The student partially trusts the teacher guidance and partially trusts self introspection during adversarial distillation. Specifically, the student model generates the adversarial data by itself and mimics the outputs of the teacher model and itself partially in LAD. Note that, the concrete intuition about the student introspection is referred to the analysis and the evidence in Figure 2 and Figure 3
Figure 1: (a) Unreliability issue:
robustness through fine-tuning or transfer learning from adversarially pre-traines models. Goldblum et al. (2020)and Chen et al. (202T) investigated distilling the robustness from adversarially pre-trained models, termed as adversarial distillation for simplicity, where they encouraged student models to mimic the outputs (i.e., soft labels) of the adversarially pre- trtinaiied teachers.
However, one critical difference is: in the conventional distillation, the teacher model and the student model share the natural training data; while in the adversarial distillation, the adversarial training data of the student model and that of the teacher model are egocentric (respectively generated by themselves) and becoming more adversarial challenging during training. Given this distinction, are the soft labels acquired from the teacher model in adversarial distillation always reliable and informative guidance? To answer this question, we take a closer look at the process of adversarial distillation. As shown in Figure| [(a] we discover that along with the training. the teacher model progressively fails to give a correct prediction for the adversarial data queried by the student model. The reason could be that with the students being more adversarially robust and thus the adversarial data being harder, is too demanding to require the teachers become always good at every adversarial data queried by the student model, as the teacher model has never seen these data its pre- pre-training. In contrast, for the conventional distillation, student models are expected to distill the "static" " knowledge from the teacher model, since the soft labels for the natural data from the teacher model are always fixed.
The observation in Figure I(a) raises the challenge: how to conduct reliable adversarial distillation with unreliable teachers? To solve this problem, we can categorize the training data according to the prediction natural and student generated adversarial data into three cases. First, if the teacher model can correctly classify both natural and adversarial data, it is reliable; Second, if the teacher model can correctly classify the natural but not adversarial data, it should be partially trusted, and the student model is suggested to trust itself to enhance model robustness as the adversarial regularization (Zhang) al. 2019); Third, if the teacher model cannot correctly classify both natural and adversarial data, the student model is recommended to trust itself totally. According to this intuition, we propose an Introspective Adversarial Distillation (IAD) to effectively utilize the knowledge from an adversarially pre-trained teacher model. The framework of our proposed IAD can be seen in Figure [(b) Briefly, the student model is encouraged to partially instead of fully trust the teacher model, and gradually trust itself more as being more adversarially robust. We conduct extensive experiments on the benchmark CIFAR- CO/CA-ROOCIFAR--- -000 and the more challenging Tiny-ImaeeNe datasets to evaluate the efficiency of our IAD. The main contributions of our work can be summarized as follows.
1. We take closer look at adversarial distillation under the teacher-student paradigm. Con- sidering adversarial robustness, we discover that the guidance from the teacher model is progressively unreliable along with the adversarial training.
2. We construct the reliable guidance for adversarial distillation by flexibly utilizing the robust knowledge from the teacher model: (a) if a teacher is good at adversarial data, its soft labels can be fully trusted; (b) if teacher is good at natural data but not adversarial data, its soft
2

---

Published as a conference paper at ICLR 2022
labels should be partially trusted and the student also takes its soft labels into account; (c) otherwise, the student only relies on its own soft labels.
3. We propose Introspective Adversarial Distillation (IAD) to automatically realize the intuition of the previous reliable guidance during the adversarial distillation. The exper- imental results confirmed that our approach can improve adversarial robustness across variety of training settings and evaluations, and also on the challenging (consider adversarial robustness) datasets (e.g., CIFAR-100 (Krizhevsky 2009)) and Tiny-mageNNer (Le & Yang 2015) or using large models (e.g., WideResNet (Zagoruyko & Komodakis 2016)).
RELATED WORK
2.1 ADVERSARIAL TRAINING.
Adversarial examples Goodffelow et al. 2015) motivate many defensive approaches developed in the last few years. Among them, adversarial training has been demonstrated as the most effective method to improve the robustness of DNNs (Cai et al. [2018) Wang et al., 2200)) Jiang et al. 2020
Chen et al. [02T]] Sriramanan et al. 202T). The formulation of the popular AT (Madry et al.
2018
and its variants can be summarized as the minimization of the following loss:
min ! foEF
a Σ fffo (2)).
(1)
where n is the number of training examples, T. is the adversarial example within the c-ball (bounded by normm centered natural example T4,, yi is the associated label, fe is the DNN with parameter and f(-) is the standard classification loss, e.g., the cross-entropy loss. Adversarial training leverages adversarial examples to smooth the small neighborhood, making the model prediction locally invariant. To generate the adversarial examples, AT employs PGD method (Madry et al. 2018). Concretely, given a sample (0) E X and the step size B > 0, PGD recursively searches
(++1) 11 (5(1) + 3 sign( V ato ((fo(())))))))
(2)
until a certain stopping criterion is satisfied. In Eq, equation 2 l E N, é is the loss function, a(() is adversarial data at step 4, y is the corresponding label for natural data, and [Tg.(2))() is the projection function that projects the adversarial data back into the e-ball centered at (()),
2.2 KNOWLEDGE DISTILLATION
The idea of distillation from other models can be dated back to (Craven & Shavlik, 1996), and re-introduced by (Hinton al. 2015) knowledge distillation (KD). It has been widely studied in recent years ( Yao et al. 2021 and works well in numerous applications like model compression and transfer learning. For adversarial defense, few studies have explored obtaining adversarial robust models by distillation. Papernot et al.] (2016) proposed defensive distillation which utilizes the soft labels produced by a standard pre-trained teacher model, while this method proved be not resistant to the C& attacks (Carlini & Wagner 2017);; Goldblum et al. (2020) combined AT with KD to transfer robustness to student models, and they found that the distilled models can outperform adversarially pre-trained teacher models of identical architecture in terms of adversaris robustness; Chen al. (202T) utilized distillation as regularization for adversarial training, which employed robust and standard pre-trained teacher models to address the robust overfitting (Rice et al. (2220).
Nonetheless, all these related methods fully trust teacher models and do not consider that whether the guidance of the teacher model in distillation is reliable or not. In this paper, different from the previous studies, we find that the teacher model in adversarial distillation is not always trustworthy. Formally, adversarial distillation suggests to minimize Exe8(z) (S 7) (T))], where T(ū(T) is not a constant soft supervision along with the adversarial training and affected by the adversarial data generated by the dynamically evolved student network. Based on that, propose reliable IAD to encourage student models to partially instead of fully trust teacher models, which effectively utilizes the knowledge from the adversarially pre-trained models.
3

---

Published as a conference paper at ICLR 2022
45000
44140
32
42000)
29
5000
5000
2040
Table Grsisrnge
(a) Toy illustration
(b) Number changes on the three kinds data during distillation
Figure 2: (a) Toy illustration: An illustration of three situations about the prediction of the teacher model. The blue/orange areas represent the decision region (in which the model can give the correct predictions) of the teacher/student model, and the red dashed box represents the unit-norm ball AT. 1) z/j The generated adversarial example from the natural example æ is located in the blue area, where the teacher model can correctly predict; 2) x². The generated adversarial example from ø out of both orange and blue areas, where the teacher model has the wrong prediction; 3) Tg. The natural example that the teacher model cannot correctly predict. (b) Number changes on the three kinds of data: We trace the number of three types of data during adversarial distillation on CIFAR- dataset. Noted that, different from the consistent number of examples like the number of examples like x' is decreasing and that of examples like x²f increasing during adversarial distillation.
3 A CLOSER LOOK AT ADVERSARIAL DISTILLATION
discuss the unreliable issue of adversarial distillation, i.e., the guidance of the teacher model is progressively unreliable along with adversarial training. In Section [.2 we partition the training examples into three parts and analyze them part by part. Specifically, we expect that the student model should partially instead of fully trust the teacher model and gradually trust itself more along with adversarial training.
In Section 3.1
3.1 FULLY TRUST: PROGRESSIVELY UNRELIABLE GUIDANCE
As aforementioned in the Introduction, previous methods (Goldblum et al. 2020 Chen al. 2021
fully trust the teacher model when distilling robustness from adversarially pre-trained models. Taking Adversarial Robust Distillation (ARD) (Goldblum al., 2020)
example, illustrate its procedure in the left part of Figure [(b) the student model generates its adversarial data and then optimizes the prediction of them to mimic the output of the teacher model. However, although the teacher model is well optimized on the adversarial data queried by itself, we argue that it might not always be good the more and more challenging adversarial data queried by the student model.
As shown in Figure [aa] different from the ordinary distillation in which the teacher model has the consistent standard performance on the natural data, robust accuracy on the student model's adversarial data is decreasing during distillation. The guidance of the teacher model gradually fails to give the correct output on the adversarial data queried by the student model.
3.2 PARTIALLY TRUST: CONSTRUCTION OF RELIABLE GUIDANCE
The unreliable issue of the teacher model in adversarial distillation raises the challenge of how to conduct reliable adversarial distillation with unreliable teachers? Intuitively, this requires us to re-consider the guidance of adversarially pre-trained models along with the adversarial training, For simplicity, we use T(x) (T(T) to represent the predicted label of the teacher model on the natural (adversarial) examples, and use y to represent the targeted label. We partition the adversarial samples into following parts as shown in the toy illustration (Figure 2(a).. and analyze them part by part.
1) T(x) = y n T(ă) = 3/: can be seen in Figure 2(a) that this part of data whose adversarial variants like 21 is the most trustworthy among the three parts, since the teacher model performs well on both natural and adversarial data. this case, we could choose to trust the guidance of the teacher model on this part of the data. However, shown in Figure 2(b) we find that the sample number of this part decreasing along with the adversarial training. That is, what we can rely on from the teacher model in adversarial distillation is progressively reduced.
4

---

Published as a conference paper at ICLR 2022
2) T(x)= ynT(() * y: In Figure 2(b) adversarial variants like zÿ Corresponding to the previous category, the number of this kind of data is increasing during distillation. Since the teacher model's outputs queried natural data are not always correct, its knowledge may not be robust and the guidance for the student model is not reliable. Think back the reason for the decrease in the robust accuracy of the teacher model, the student model itself may also be trustworthy since it becomes gradually adversarial robust during distillation.
we also check the number change of the part of data whose
the small neighborhood of the
3) (()) + yn & y. As for the data which are like 22 in Figure 2(a) the guidance of the teacher model is totally unreliable since the predicted labels on the natural data are wrong. The student model may also trust itself to encourage the outputs to mimic that of their natural data rather than the wrong outputs from the teacher model. First, removes the potential threat that the teacher's guidance may be a kind of noisy labels for training. Second, as an adversarial regularization ZZang et al. 2019), can improve the model robustness through enhancing the stability of the model's outputs on the natural and the corresponding adversarial data.
4) (()) + y (T) = y: Considering the generation process of the adversarial data, i.e. 5* arg maX;EB, (()(T),)) Once the original prediction is wrong, i..., Txx) A y, the generation of H' only make the prediction worse. Thus, this group doesn't 't exist.
To sum up, we suggest employing reliable guidance from the teacher model and encouraging the student model to trust itself more as the teacher model's s guidance being progressively unreliable and the student model gradually becoming more adversarialyy robust.
4 INTROSPECTIVE ADVERSARIAL DISTILLATION
Based previous analysis about the adversarial distillation, we propose the Introspective Adversarial Distillation (LAD) to better utilize the guidance from the adversarially pre- treinra model. Concretely, we have the following KD-style loss, but composite with teacher guidance and student introspection.
LIDD =
((A);a)
+y (x
Label & Teacher Guidance
Student Introspection
where (ADD:aa is the previous adversarial distillation baseline, e.g. ARD (Goldblum al., 2020) or AKD Chen et al. 2021). weighted by the hyper-parameter a is weight for the student introspection, ((-1T) [T is a Softmax operator with the temperature T E (0, +00), e.g., ((xx
En exp( S() is the conventional Softmax with the temperature r = 1, T(-T)) is the tempered variant of the teacher output T(-). ā is the adversarial data generated from the natural data x, y the
exp(m*/T)
hard label and leg is the Cross Entropy loss and ε is the KL-divergence loss. As for the annealing parameter a E [0, 1] that is used to balance the effect of the teacher model in adversarial distillation, based on the analysis about the reliability of adversarial supervision in Section 3 we define it as,
a (2lv)))
where Pr(-1y) (y) is the prediction probability of the teacher model about the targeted label and 8 is hyperparameter sharpen the prediction. The intuition behind IAD is calibrate the guidance from the teacher model automatically based on the prediction of adversarial data. Our a naturally corresponds to the construction in Section 3.2] since the prediction probability of the teacher model for the ad adversarial data can well represent the categorical information. As for B, we have plot the specific values of a under its adjustment in the left of Figure 4
Intuitively, the student model can trust the teacher model when a approaches 1, which means that the teacher model is good both natural and adversarial data. However, when a approaches 0,
corresponds that the teacher model is good at natural but not adversarial data, or even not good at both, and thus the student model should take its self-introspection into account. In Figure 3 we check the reliability of the student model itself. According to the left panel of Figure 3 we can see that the student model is progressively robust to the adversarial data. And if we incorporate the student introspection into the adversarial distillation, the results in the middle of Figure conffirm its
Note that, do use when AD, ARD. Please refer the Appendix A.2 for the ablation study.
5

---

Published as a conference paper at ICLR 2022
FOD T 22g Accomaa (952
With Susssccci
Only tust Racher Modd
70
20
Teacher Model
Qudeent Model
33
75
130
125
150
175
200
25
7.5
1.00
150
175
200
25
30
163
125
150
175
200
Epectt
Spacts
Figure 3: Reliability certification about self-introspection of the student model: Left, PGD-10 10 training accuracy of teacher/student model; Middle, PGD-10 training aceuracy of teacher model and that combined with the seff-introspection of the student model; Right, PGD-10 test accuracy of the student model which only trusts the soft labels and the student model which also considers seff-introspection.
Algorithm Introspective Adversarial Distillation (IAD)
Input: student model S, teacher model T, training dataset {( y: 1-) learning rate n. number of epochs N, batch size m, number of batches M, temperature parameter т, the annealing parameter on teacher model's 's predicted probability a, adjustable parameter X, )1, X2, Ag and y.
Output: adversarially robust model S,
for epoch = 1...., N do
for mini-batch 1, ...... M do Sample a mini-batch { y1 p from D for ................................ m (in parallel) do
Obtain adversarial data ã; of by PGD based on Eq. equation 2
Compute a for each adversarial data based on Eq. equation end for
(C))(S())))) (1- A) ((x[(((ā)))) + ( IAD-I: + -
}
or
A + A2 .2. (a (a-4xu(5((1)))) +\3t (at ((
IAD-II: 0 + 0 0
end for end for
potential benefits improve the accuracy of the guidance. Moreover, as shown in the right panel of Figure adding self-introspection results in better improvement in model robustness compared to only using the guidance of the teacher model. Therefore, ADD automatically encourages the outputs of the student model to mimic more reliable guidance in adversarial distillation.
Algorithm T summarizzss the implementation of Introspective Adversarial Distillation (IAD). Specifi- cally, IAD first leverages PGD to generate the adversarial data for the student model. Secondly, IAD computes the outputs of the teacher model and the student model on the natural data. Then, IAD mimics the outputs of the student model with that of itself and the teacher model partially based on the probability the teacher model on the adversarial data.
Warming-up period. During training, we add a varming-up period activate the student model, where a (in Eq. equation 33 is hardcoded to 1. This is because the student itself is not trustworthy in the early stage (refer to the left panel of Figure 3}. Through that, we expect the student model first evolve into relatively reliable learner and then conducts the procedure of IAD.
4.1 COMPARISON WITH RELATED METHODS
In this section, we discuss the difference between IAD and other related approaches the perspective of the loss functions. Table summarizes all of them.
As shown in Table AT (Madry et al. 2018) utilizes the hard labels to supervise adversarial training: TRADES (Zhang et al., 2019) decomposes the loss function of AT into two terms, one for standard training and the other one for adversarial training with the soft supervision: Motivated by KD (Hinton et al. 2015). Goldblum al. 2002)) proposed ARD conduct the adversarial distillation, which fully

---

Published as a conference paper at ICLR 2022
Table 1: Loss comparison. The checkmark means that the term is considered in the model objective.
Knowledge
Tabel
Seir
Toucher
Loss Term (ox(S(s). ()
(
AT TRADES
ARD IAD
AKD IAD-I1
Table 2: Test accuracy (%) CIFAR-IO/CIFAR 100 datasets using ResNer-18.
Datasets
CIFAR-10
CIFAR-100
Teacher
Method
Natural
FGSM
PGD-20
CW sc
Natural
FGSM
PGD-20
CW2
AA
AT
83.06%
63.53%
50.21%
49.22%
46,70%
56.21%
33,94%
24.60%
23.35%
21.55%
ARD
83.13%
64.11%
51.36%
50.37%
48.05%
56.65%
35.82%
27.04%
25.30%
23.34%
AT
AKD2
83.52%
63.91%
51.36%
50.36%
48.08%
56.46%
35.76%
27.18%
25.33%
23.47%
IAD-I
82.09%
63.20%
52.14%
50.74%
48.66% %
55.53%
35.69%
27 27.40%
25.80 %
23.66%
IAD-1I
83.21%
63.54%
51.85%
50.67%
48.58%
57.26%
35.94%
27.50 %
25. 68.
24.06%
TRADES
81.26%
63.12%
52.98%
50.29%
49.47%
54.01%
35.23%%
28.00 18.00%
24.26%
23.46%
ARD
81.50%
63.38%
53.38%
51.27%
49.92%
82%)
37.56% %
30.21 1%
26.76%
25.36%
TRADES
AKD
83.49%
64.00%
51.89%
50.25%
48.52%
57.46%
37.32%
28 28.62%
25. 25.76%
24.36%
IAD-I IAD-II
80.63%
63.14%
53.84%
51.60%%
50.17%
55. 19%
37.47%
30.60 %
27.24 %
25.84%
83.76%
64.17%
52.16%
50.59%
48.91%
57.08%
36.94%
29.08%
25.831
24.45%
trusts the outputs of the teacher model to learn the student model. As indicated by the experiments in Goldblum et al. (2020), a larger À resulted in less robust student models. Thus, they generally set À O in their experiments; Chen et al.] (202I) utilized distillation as a regularization to avoid the robust overfitting issue, which employed both the adversarially pre-trained teacher model and the standard pre-trained model. Thus, there are two KL-divergence loss and for simplicity, term their method as AKD?; Regarding IAD, there are two types of implementations that are respectively based ARD or AKD?. We term them as IAD-I and IAD-II, and their difference with previous ARD and AKD? is an additional self-intrroppcctonn term. Besides, we also apply a to downweight the dependency on the term (KL (S which has been explained in previous sections.
5 EXPERIMENTS
We conduct extensive experiments to evaluate the effectiveness of IAD. In Scction 5.1 we compare IAD with benchmark adversarial training methods (AT and TRADES) and some related methods which utilize adversarially pre-trained models via KD (ARD and AKD?) on CIFAR- -O/A--O///FAR- 100 (Krizhevsky] (2009) datasets. In Section 5.2 we compare the previous methods with IAD on more challenging dataset Niny-mageNNet (Le & Yang 2015). In Section the ablation studies are conducted analyze the effects of the byper-parameter 3 and different warming-up periods for IAD.
Regarding the measures, we compute the natural accuracy on the natural test data and the robust accuracy on the adversarial test data generated by FGSM, PGD-20, and C&W attacks following (Wang et al. 2019) Moreover, we estimate the performance under AutoAttack (termed as AA).
5.1 EVALUATION ON CIFAR-IO/CIFAR-100 DATASETS
Experiment Sctup. In this part, we follow the setup (learning rate, optimizer, weight decay, momentum) Goldblum et al. (2020) implement the adversarial distillation experiments on the CIFAR-IO/CIFAR 100 datasets. Specifically. we train ResNet- 18 under different methods using SGD with 0.9 momentum for 200 epochs. The initial learning rate 0.1 divided by 10 at Epoch 100 and Epoch 150 respectively, and the weight decay=0.00m 0000.. In the settings of adversarial training, we set the perturbation bound = 8/255, the PGD step size o = 2/ 2/255, and PGD step numbers K = 10. In 1 and use models pre-trained by AT and TRADES which have
the settings of distillation, we use the best PGD-10 test accuracy as the teacher models for ARD, AKD2 and our IAD. For ARD, we set its hyper- apper-parameter À 0 as recommend in Goldblum et al. (020)) for gaining better robustness. For Xxper- we set A = 0.25, X2 = 0.5 and Ag = 0.25 as recommanded in Chen et al. (2021). For IAD-I and IAD-II, we respectively set the warming-up period as 60/80 and 40/80 epochs train on
7

---

Published as a conference paper at ICLR 2022
Table 3: Test accuracy (%) on CIFAR-IO/CIFAR-ION datasets using WideResNNer34-11.
Datasets
CIFAR-10
CIFAR-100
Teacher
Method
Natural
FGSM
PGD 20
CW.,
AA
Natural
FGSM
PGD:20
CW.,
AA
AT ARD
85.24%
66.07%
53.36%
52.85%
50.37%
59.75%
39.67%
31.14%
29.99%
27.46%
83.39%
64.43%
54.06%
53.27%
51.57%
58. 84%
40.68%
32.18%
30.36%
27.96%
AT
AKD
86.25%
67.65%
55.05%
54.07%
51.70%
61.57%
41.14%
31.90%
30.45%
27.80%
IAD-I IAD-II
84.21%
66.62%
55.07%
54.38%
52.11%
57.78%
39.95%
32.43%
30.46 %
28.39%
85.09%
66.54%
55.45 %
54.63%
52.29% %
60.72%
40.67%
32.33%
30.40%
27.89%
TRADES
82 90%%
65.55%
55.50%
53.04%
52.00%
59. 14%
38.81%
31.21%
27. 77.6%%
26.53%
ARD
83.60%
65.67%
55.32%
53.23%
52.08%
63%
40.20%
32.41%
86.886%
27.92%
TRADES
AKD
84.95%
67.14%
55,38%
53.38%
12%
61.48%
40.52% %
32.34%
29.84%
27.84%
IAD-I
83.06%
66.51%
56.17%
$3.99%
$2.68% %
57.61%
39.955%
32.87 %
30.13%
28, 28.63%
IAD-II
85.68%
67.39%
55.45%
53.77%
51.85%
60.882 82.
40.43%
32.39%
30.01%
28.01%
Table 4: Test accuracy (%) on Tiny-ImageNe dataset using MrrAcctiv-ResNer..
Teacher
Method
Natural
FGSM
PGD-20
CW.c
AA
AT
46.16%
28. 20%
22.16%
19.52%
18.04%
ARD AKD
47.34%
29.56%
22.60%
19.96%
17.60%
AT
49.48%
30.10%
22.70%
20.10%
18.18%
IAD-I IAD-II
46.14%
29. 29.38%
23.40%
20.74%
19.16%
49.52%
30.40 %
23.42%
21.26 %
18.64%
TRADES
46.12%
27. 74% 29.98%
21.00%
16.86%
15.86%
ARD
49.60%
22.76%
19.16%
17.26%
TRADES
AKD2
47.72%
28%12%
22.10%
18.28%
17.13%
IAD-1
47.58%
29 .2% 30.54% %
23.12% 23.72%
19.12% 19.78 %
17 66.6 17.82%
IAD-II
48.10%
CIFAR-IO/CIFAR-IO0 100. Regarding the computation of a, we set À 0, 0.1. For 7, we currently set = a and more ablation study about its setting can be found in the Appendix A.3
We report the results in Table 2, where the results of AT and TARDES are listed in the first and fifth rows of Table 2 and the other methods use these models as the teacher models in distillation. On CIFAR-10 and CIFAR- improvements on adversarial robustness in terms of PGD-20, CW and AA accuracy compared with the student models distilled by ARD AKI and the adversarially pre-trained teacher models. Besides, IAD-II generally performs better than IAD-I when the teacher is trained by AT, which means AKD2 in this case could be better starting point than ARD. However, when the teacher model is trained by TRAEDS, the advantage about the robustness of IAD-I over that of IAD-II in reverse. Considering their distillation philosophy, i.e., ( Ea (((()))) of IAD-II, it might be up to which of T(x(T) and T(ëTT) is more informative in adversarial distillation from the diverse teachers. The natural accuracy of LAD sometimes lower than others, but the performance drop is not very significant compared to IAD-II.
Results.
note that our IAD-I IAD-II has obtained consistent
of IAD-I and
Experiment Setup. In this part, we evaluate these methods by the model with a larger capacity, ie, WideResNer--44.1 The teacher network is trained by AT and TRADES, following the settings of (Zhang et al., 2021). We keep the most settings of baselines same as the previous experiment. For IAD-I and IAD-II. we set 5/10 epochs warming-up up period on CIFAR-IOICIFAR-100
The results is summarized in Tables 3 Similarly, on CIFAR-10 10 and CIFAR-100, our method can also achieve better model robustness than ARD, AKD and the original teacher models. Moreover, our IAD methods do not sacrifice much standard performance compared with the original teacher models. Since AKD2 externally utilizes a standard pre-trained teacher model, IAD-II can achieve the consistently better natural performance than IAD-I. However, in terms of the robustness, IAD-I generally achieves the comparable or even better than IAD-II under both AT and TRADES.
Results.
5.2 EVALUATION ON Tiny-ImageNet DATASET
Experiment Setup. In this part, we evaluate these methods more challenging Timy- ImageNet dataset. For these adversarially pre-trained models, we follow the settings of (Chen et al., 202T) to train AT and TRADES. To be specific, we train Presctive-ResNe---P 18 using SGD with 0.9 momentum
8

---

Published as a conference paper at ICLR 2022
82.5
Natural accuracy
83
62.0
81.4
80
0.0
90.5
9500
0.44
0 0.02
PSD-20 accuracy
0 0.05
0.2
0 0.1
0.5
1.0
$1.5
530
0.
0.4
10
0.1
605
0.1
0.5
20
60
50
P:dd0)
Werm ngsup peroo (tprecs)
Figure 4: Analysis about using different and warming-up up periods in IAD: Left, the values of a (Eq, under different 3: Middle, Natural and PGD-20 accuracy of teacher model using different 3; Right, Natural and PGD-20 accuracy of the teacher model with different warming-up up periods.
for 100 epochs. The initial learning rate 0.1 divided by 10 at Epoch 50 and 80 respectively, and the weight decay=0.0005. For distillation baselines, we keep most settings the same Section For ARD and IAD-I, here we set its = 0.9 to deal with the complex task following (Goldblum et al.) 2020
And for both IAD-I and IAD-II, we use 0.1, 0.1 and 10 warming-up epochs.
Results. We report the results in Table Overall, our IAD-I or IAD-II can still achieve better model robustness than other methods. Specifically, on Tiny-ImugeNet, IAD-II can approximately improve both the natural accuracy and the robust accuracy compared to IAD-I and other baselines.
5.3 ABLATION STUDIES
To give comprehensive understanding of our proposed IAD method, have conducted series of experiments (in Appendix(A) including the ablation study on using different related to student introspection, different 7 related adversarial distillation and the loss terms of IAD-II well the comparison of the computational cost. In the following, we only study the effect of 3 and warming-up up periods for our student introspection. More ablation study can be found in the Appendix.
Experiment Setup. To understand the effects of different / and different warming-up periods on CIFAR-10 dataset, we conduct the ablation study in this part. Specifically, we choose the ResNet- the backbone model, and keep the experimental settings the same Section [. In the first experiments, we set no warming-up period and study the effect of using different B. Then, in the second experiments, set 0.1 and investigate different warming-up periods.
Results. We report part of the results on IAD-I method in Figure 4 The complete results with other evaluation metries as well as that on IAD-II method can be found Appendix A. In Figure 4 we first visualize the values of the using different in the left panel, which shows the proportion of the teacher guidance and student introspection in adversarial distillation. The bigger the beta corresponds to larger proportion of the student introspection. In the middle panel, we plot the natural and PGD-20 accuracy of the student models distilled by different B. We note that the PGD-20 accuracy is improved when the student model trusts itself more with the larger 3 value. However, the natural accuracy is decreasing along with the increasing of the 3 value. Similarly, we adjust the length of warming-up periods and check the natural and PGD-20 accuracy in the right panel of Figure We find that setting the student model partially trust itself at the beginning of the training process Teads to inadequate robustness improvements. An appropriate warming-up period at the early stage can improve the student model performance on the adversarial examples.
6 CONCLUSION
In this paper, we study distillation from adversarially pre-trained models. We take a closer look at adversarial distillation and discover that the guidance of teacher model is progressively unreliable by considering the robustness. Hence, we explore the construction of reliable guidance in adversarial distillation and propose method for distillation from unreliable teacher models, i..., Introspective Adversarial Distillation. Our methods encourages the student model partially instead fully trust the guidance of the teacher model and gradually trust its self-introspection
to improve robustness.

---

Published as a conference paper at ICLR 2022
7 ACKNOWLEDGEMENT
JNZ and BH were supported by the RGC Early Career Scheme No. 22200720, NSFC Young Scientists Fund No. 62006202 and HKBU CSD Departmental Incentive Grant. JCY and HXY was supported by NSFC No. U20A20222. JFZ was supported by JST, ACT-X Grant Number JPMJAX21AF, TLL was supported by Australian Research Council Projects DE-190101473 and DP-220102121. JLX was supported by RGC Grant C6030- 18GF.
ETHICS STATEMENT
This paper does not raise any ethics concerns. This study does not involve any human subjects, practices to data set releases, potentially harmful insights, methodologies and applications, potential conflicts of interest and sponsorship, inserinination/iias/ainee concerns, privacy and security issues, legal compliance, and research integrity issues.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of experimental results, our code is available at at/ttps://githu com/ Fam//Fancyy/aa IAD
REFERENCES
Jean-Band Cean-Baptiste Alayrae, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli. Are labels required for improving adversarial robustness? In NeurlPS, 2019.
Qi-Zhi Cai, Chang Liu, and Dawn Song. Curriculum adversarial training. In IJCAI, 2018.
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness neural networks. In IEEE Symposiun on Security and Privacy (SP), 2017.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C. Duchi. Unlabeled data improves adversarial robustness. In NewrlPS, 2019.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Robust overfitting may mitigated by properly learned smoothening. ICIR, 2021.
Mark Craven and Jude Shavlik. Extracting tree-structured representations of trained networks. In NeurlPS, 1996.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Mma training: Direct input space margin maximization through adversarial training. In ICLR, 2020.
Xuefeng Du, Jingfeng Zhang, Bo Han, Tongliang Liu, Yu Rong, Gang Niu, Junzhou Huang, and Masashi Sugiyama. Learning diverse-structurred networks for adversarial robustness. In ICML, 2021.
Lijie Fan, Sijia Liu, Pin-Yu Chen, Gaoyuan Zhang, and Chuang Gan. When does contrastive learning preserve adversarial robustness from pretraining to finetuning? In NeurlPS, 2021.
Micah Goldblum, Liam Fowl, Soheil Feizi, and Tom Goldstein. Adversarially robust distillation. In AAAI, 2020.
Ian J. Goodfellow. Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
10

---

Published as a conference paper at ICLR 2022
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In arXiv, 2015.
Andrew Ilyss, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In NeurlPS, 2019.
Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Robust pre-training by adversarial contrastive learning. In NeurIPS, 2020.
Alex Krizhevsky. Learning multiple layers of features from tiny images. In arXiv, 2009.
Ram Shankar Siva Kumar, Magnus Nyström, John Lambert, Andrew Marshall, Mario Goertzel, Andi Comissoneru, Matt Swann, and Sharon Xia. Adversarial machine bearning-induste perspectives. In IEEE Symposium on Security and Privacy Workshops (SPW), 2020.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. In CS231N course, 2015.
Todd Litman. Autonomous vehicle implementation predictions. Victoria Transport Policy Institute Victoria, Canada, 2017.
Chen Liu, Mathieu Salzmann, Tao Lin, Ryota Tomioka, and Sabine Süsstrunk. On the loss landscape of adversarial training: Identifying challenges and how to overcome them. NeurIPS, 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In /CLR, 2018.
Nicolas Papernot, Patrick McDaniel, Wu, Somesh Jha, and Ananthram Swami. Distillation as defense to adversarial perturbations against deep neural networks. In IEEE Symposium on Security and Privacy (SP), 2016.
Leslie Rice, Eric Wong, and J Zico Kolter. Overfitting in adversarially robust deep learning. In ICML, 2020.
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversari- ally robust imagenet models transfer better? In NeurlPS, 2020.
Manli Shu, Zuxuan Wu, Micah Goldblum, and Tom Goldstein. Encoding robustness to image style via adversarial feature perturbations. In NeurIPS, 2021.
Gaurang Sriramanan, Sravanti Addepalli, Arya Baburaj, and Venkatesh Babu Radhakrishnan. Towards efficient and effective adversarial training. In NeurIPS, 2021.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
Haotao Wang. Tianlong Chen, Shupeng Gui, King-Kuei Hu, Ji Liu, and Zhangyang Wang. Once- for-all adversarial training: In-situ tradeoff between robustness and accuracy for free. In NeurIPS, 2020.
Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In JCMI., 2019.
Jiangchao Yao, Feng Wang, Kunyang Jia, Bo Han, Jingren Zhou, and Hongxia Yang. Device-cloud collaborative learning for recommendation. In KDD, 2021.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In arXiv, 2016.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing. Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019.
Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan Kankan- halli. Attacks which do not kill training make adversarial learning stronger. In ICML, 2020.
Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. Geometry-aware instance-reweighted adversarial training. In ICLR, 2021.
Bojia Zi, Shihao Zhao, Xingjun Ma. and Yu- Ga-g Jiang. Revisiting adversarial robustness distillation: Robust soft labels make student better. In ICCV, 2021.

---

Published as a conference paper at ICLR 2022
APPENDIX
A COMPREHENSIVE EXPERIMENTS
A.1 COMPLETE RESULTS OF ABLATION STUDIES.
In this section, we report the complete results of our ablation studies Tables 5 (about warming-up periods).
(about ()
and
Table 5: Test accuracy (%) of IAD-I and IAD-II using different & with ResNet- 18 on CIFAR-10.
Natural
FGSM
PGD-20
CW
AA
0.01
82.76%
63.58%
51.49%
10.16%
48.26%
0.05
82.72%
64.09 %
51.51%
50.43%
48.73%
IAD-I
0.1
82.70%
63.57%
51.75%
50.51 %
48.53%
80.66%
62.79%
52.36%
19%
48.43%
1.0
80.79%
63.55%
52.64%
50.40%
48.61%
0.01
83.25 %
64.46%
51.59%
50.62%
48.39%
0.05
83.13%
63.76%
52.23%
50.81%
48.75%
IAD-II
0.1
83.07%
64.12%
52.07%
50.63%
48.36%
0.5
82.08%
63.44%
52.53%
50.86%
49.02% %
1.0
81.54%
63.82%
52.72%
50.76%
48.75%
Table 6: Test accuracy (%) of IAD-I and IAD-Il using different warming-up up periods with ResNet-18 on CIFAR-10.
warming-up
Natural
FGSM
PGD-20
CW
AA
0 epochs
82.70%
63.57%
51.75%
50.51%
48.53%
20 epochs
82.75%
63.82%
51.50%
50.44%
48.37%
IAD-I
40 epochs
82.82 %
63.96%
51.71%
50.67%
48.44%
60 epochs
82.09%
63.20%
52.14%
50.74%
48.66%
80 epochs
82.17%
63.40%
52.03%
50.59%
48.57%
0 epochs 83.07%
64.12%
52.07%
50.63%
48.36%
20 epochs
84.22% %
64.07%
51.60%
50.38%
48.47%
TAD-II
40 epochs
83.21 %
63.54%
51.85%
50.67%
48.58 %
60 epochs
83.18%
63.97%
52.08
50.68 %
48.51 %%
80 epochs 83.07%
63.92%
51.96%
50.30%
48.09%
In Table 5 we can see that the natural and FGSM accuracy will decrease, and the robust accuracy (PGD-20, CWoo, AA) will increase with the rise of B. In Table we adjust length of warming-up periods. We can that letting the student network partially trust itself at the beginning of the training process would result in inadequate robustness improvements. that there is trade-off in Table 5 achieve both larger natural accuracy and larger robustness in adversarial distillation, which also similar to the standard adversarial training. We can slightly sacrifice the robustness (adjust the adjust the warming-uy up periods) to acquire better natural accuracy and FGSM.
summary, we can find
A.2 FULLY TRUST PARTIALLY TRUST THE TEACHER GUIDANCE
In this section, compare the method with fully trusted or partially trusted teacher guidance. Regarding the variant of AKD that replaces the second term in AKD2 by our "partially trust" KL term (but without the introspection term), find the robust improvement in Table 7
12

---

Published as a conference paper at ICLR 2022
Table 7: Comparison between AKD and a "partially trust" variant with ResNet- 18 on CIFAR-10.
Natural
FGSM
PGD-20
CW
AKD
83.52 % 83.37%
63.91% 63.95%
51.36%
50.36% 50.40 %
"partially trust"
51.49 %
In the "partially trust" variant of AKD?, we just down weight the part of teacher guidance which has wrong prediction results with the hard labels. The results show that fit this part "unreliable" teacher guidance may improve the natural performance but consistently drop the robust accuracy.
In the following, we also conduct a range of experiments to compare IAD-I and variant IAD-I- Down that applys our downweighting on ( ((] rather than constant 1.0 like ARD, According to the results, we can see that IAD-I consistently achieves better robustness while sacrifices little natural accuracy. Hence, will choose the constant 1.0 on (x(((( for ARD in the main part of our experiments on those benchmark datasets.
Table 8: Comparison between IAD-I and IAD-I-Down. IAD-I-Down variant of LAD-I, where the weight on (5( is downweighted with rather than constant 1.0 like ARD.
Dataset
Method
Natural
FGSM
PGD-20
CW
IAD-I
82.09%
63.20%
52.14%
50.74%
CIFAR-10
IAD-I-Down
83.33
63.90%
51.77%
50.63%
IAD-I
55.53%
35.69%
27.40 %
25.80 %
CIFAR-100
IAD-I-Down
55.88
35.68%
27.32%
25.60%
IAD-I
46. 14%
29.38%
23.40%
20.74%
Finy-ImageNee
IAD-I-Down
46.66 %
29.30%
22.68%
19.64%
A.3 THE EFFECTS OF y FOR STUDENT INTROSPECTION
In this section, check the effects of the Student Introspection via adjusting the for LAD methods. Here, we also use the constant y in the experiments. We find that using the constant can further boost the performance on model robustness, but there another problem that the natural accuracy will sacrifice a little bit more than our previous dynamical design, i.e.,
a.
Table 9:
Test accuracy (%) IAD-I and IAD-II using different coefficients Student Introspection with R8 on CIFAR-10.
coefficient
Natural
FGSM
PGD-20
CW mc
1-
82.09 %
63.20%
52.14%
50.74%
0.5
81.46%
63.61 %
52.77%
50.67%
IAD-I
1.0
80.04%
63.52%
54.14%
50.88%
2.0
77.59%
62.86%
55.64%
50.72%
4.0
77.34%
62.68%
55.78%
51.29%
1-a
83.21
63.54%
51.85%
50.67%
0.5
82.76%
63.91%
52.50%
50.67%
IAD-II
1.0
81.33%
63.48%
53.32%
50.84%
2.0
80.95%
63.90%
54.02%
51.13%
4.0
80.43%
64.05%
55.27 %
51.97%
According to the results in Table 9 we can hardly to find an optimal coefficient for the student introspection term to achieve both the best natural accuracy and the best robustness. However,
13

---

Published as a conference paper at ICLR 2022
there is one trend that increasing the coefficient will gain more robustness with losing natural accuracy. With the hyper- pprammerreeter ,, we may flexibly instantiated by some constants or some strategic schedules to pursue the robustness the natural accuracy.
A.4 THE EFFECTS OF FOR ADVERSARIAL DISTILLATION
In this section, we check the effects of for adversarial distillation under the same training configura- tions and also for TRADES. have listed the results of TRADES in Tabel TO and the results of IAD-I and ARD in Table
Table 10: Test accuracy (%) of TRADES with different with ResNet- on CIFAR-10.
20
60
100
Natural PGD-20
87.14% 43.26%
81.26%
84.57%
83.95%
84.43%
52.98%
51.18%
51.17%
50.73%
As for TRADES, think it may not need the temperature т which is especially designed for distillation, since the KL term in TRADES is aims to encourage the output of the model on adversarial data to being the similar with that natural data. Intuitively, it encourage the output be more stable which leads to better robust performance, As result, based on the test accuracy in Table IO maybe the best for TRADES achieve better robustness and enlarging (or decreasing) can disturb the original output logits and may result in lower robustness (i.e., = 0.1: PGD-20 43 43.26% )
Table 11: Test accuracy (%) of IAD-I and ARD with different with ResNet-18 18 on CIFAR-10.
0.1
20
60
100
Natural PGD-20
82.76%
82.09 %
82.22%
82.12%
81.95%
IAD-I
55.71% %
52.14%
50.74%
50.83%
51.02%
Natural PGD-20
83.11 13.11% %
13%
83.18%
82.39%
82.89%
ARD
55.81 %
51.36%
50.44%
50.37%
50.15%
According to the comparison in Table 11 with the proper (e.g., + 2 1), TAD-I can achieve the comparable natural accuracy and better robustness. Note that in the experiments of IAD-I, we keep the r = 1 for the student introspection term according to the results of TRADES in Table 10
A.5 COMPARISON WITH TRADES AND ARD
In this section, we compared the performance of TRADES, ARD and our IAD-I with different weights in their each loss terms. We summarized the results in Table 12
Table 12: Test accuracy (%) of using different weights for the loss terms in TRADES and ARD.
Method
Label (cz(S(z),))
Self txc (S(
Teacher
Accuracy
(S(a)))
Natural
PGD-20
TRADES
81.26%
52.98%
ARD
0
83.13%
51.36%
O
85.94 %
48. 48.58%
0
74.95%
55.89 %
79.73%
54.07%
0
I-a
83.33%
51.77%
IAD-I
0
82.09%
52.14%
According to the results, IAD-I can approximately achieve the better robust accuracy than ARD, and the better natural accuracy than TRADES with less drop PGD-20 accuracy. With varying the
14

---

Published as a conference paper at ICLR 2022
hyper-parametee the loss terms, also find that natural accuracy reach to 85.94% loss of robustness to 48.5 58.58% and robustness can reach to 55.89% at loss of natural accuracy to 74.95%. In summary, through adjust those weight, our IAD-I can flexibly make good balance among them by considering both the natural accuracy and the robustness.
A.6 ABLATION STUDY ON THE LOSS TERMS OF IAD-II
In this section, we conduct the ablation study in Table 13 about the four loss terms in IAD-II which is based on AKD
Table 13: The ablation study about the different terms in IAD-II with ResNet- 18 on CIFAR 10.
Label
Self
Teacher
Accuracy
(Sn(5()). (s(())))))))(((l))) (S(
Natural
PGD-20
84.30 %
50.67%
82.66%
51.72%
83.52%
51.36%
83.21%
51.85 %
Specifically, according to the results, we find & ( (T can help model gain more natural accuracy, while ( (T can help model gain more robustness. AKD achieves good balance between two aspects by combining above two terms, and IAD-II further boosts the model robustness by incorporating the self- seff-introspection term with less drop in natural performance.
A.7 COMPUTATIONAL COST COMPARISON
In this section, we check and compare the computational cost of ARD, AKD2, IAD-I and IAD-II II in terms of the averaged training time in each epoch, as well the GPU Memory-Usage The detailed results are summarized in Table 14
Table 14: Time and Memory cost of adversarial distillation methods with ResNet-18 on CIFAR-10.
Method
Time (Avg. Epoch)
GPU Memory-Usage
ARD
156.122
3253MiB
IAD-I
182.01s
3977MiB
2KD2
147.10s
2397MiB
IAD-II
167.23s
3249MiB
According to the results, IAD-I and IAD-II correspondingly consume bit more time and memory than ARD and AKD due to the additional self-introspection.. Besides, an interesting phenomenon is that ARD has less terms than AKD?, but consumes more time and GPU memory. This is because ARD has to deal with both and ă, while AKD? only needs to deal with ž.
15


Reviews:
[{"review_id": "v3gxaSi5Oln", "reviewer": "Reviewer_xcEE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In summary, the discovery of the specific degeneration in adversarial distillation is interesting, and the authors design an elaborate loss that takes such a drawback in the adversarial distillation into account. A range of experiments on different network architectures, adversarial training backbones (AT or TRADES) and three widely-used datasets demonstrate its superiority in improving the model robustness. Besides, the ablation study regarding the annealing temperature and the warming up has been conducted to provide the view on its working mechanism. \n\nHowever, there are still some concerns about this work.\nMajor Concerns\n1)  In Section 3.2, the authors analyze the adversarial data by partitioning them into three groups. Is there the fourth group where the teacher model has wrong predictions on natural data but right prediction on its adversarial counterpart? If it almost does not exist, it will be complete that the authors at least mention this in the paragraph. Besides, for the third group, it is not absolutely consistent with the case of x_2 where it also requires the wrong prediction of the student model. \n2) In Section 4, Eq.(4) can approximately reflect the confidence on the adversarial data. But have the authors considered the difference between group 2 and group 3 in Eq.(4), since they may have different scale and corresponds to different cases.\n3) In fact, the proposed Introspective Adversarial Distillation can degenerate to TRADES and ARD by setting some extreme hyperparameters. It will be better to conduct an experiment to plot a accuracy (natural acc and robust acc) surface with varying hyperparameters that includes these three methods, which helps us to better understand the position your method.\n\nMinor Concerns\n1) Table 1 is messy. The authors could list each distance term as a column name and compare baselines by considering it or not.\n2) The experiments of IAD + AKD^2 is not well explained. I guess it is to show how IAD can help improve the natural accuracy but the authors have not clarified this.\n", "summary_of_the_paper": "This paper studies a specific distillation scenario, adversarial distillation, to improve the model robustness. Different from the constant sort supervision in the ordinary distillation, the teacher model will become progressively unreliable along with training, since the adversarial data are dynamically searched by the student model and might not be well identified by the teacher model. Therefore, the authors introduce an introspective adversarial distillation, which considers to bootstrap the learning from both the teacher model and itself. A range of experiments demonstrate its superiority on improving the model robustness.", "summary_of_the_review": "This paper investigates an interesting problem about adversarial distillation, which has some distinct with the ordinary distillation. The authors provides an introspective adversarial distillation to solve the problem. Despite simple, it shows consistent improvement on the model robustness while does not sacrifices the natural accuracy too much. However, the writing about the motivation and the loss designed for each group as well as the experimental parts are not very clear and need to be further improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "mJDfftZQHGX", "reviewer": "Reviewer_2bpg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The paper is well written and easy to follow. The main claims and key observations are clearly stated.\n\n2. The key observation is very inspiring: The (adversarially pretrained) teacher model's accuracy on adversarial images generated by the student model gradually drops. This provides solid foundation for the main claim of the paper: The student model shouldn't be fully convinced by the soft labels provided by the teacher model in adversarial distillation. This is something important that previous works overlooked. \n\n3. AKD2+IDA does generally outperform previous methods, although the margin is not that significant. I suggest the authors to try the modification I mentioned in the second bullet in \"Cons\" if haven't already. It may enlarge the performance improvement based on my intuitions.\n\nCons:\n\nAs I mentioned before, I'm totally convinced with the intuition to partially trust the teacher model's soft label in adversarial distillation. Bellow are some minor concerns.\n\n1. The observation in Figure 1 (a) and Figure 3 (left column) tells us that the teacher model's predictions on the **adversarial** images generated by the student model shouldn't be fully trusted. However, the previous method ARD uses the soft label generated by the teacher network on **clean** images. So, it is not proper to claim that ARD has issue because it fully trusted the teach model's unreliable soft labels. In fact, the soft labels used in ARD are generated by teacher on **clean** images and has been shown to be generally reliable by yourself in the blue curve in Figure 1 (a). With that said, ADK2 does use the soft-label on **adversarial** images. I think it is better to use ADK2 as the starting point to motivate your method, instead of from ARD as in Sec 3.1 of current version. An even more direct way to motivate IDA is to replace the second loss term (ie the \"fully trust\" KL term) in ADK2 to your \"partially trust\" KL term, while keeping other terms unchanged. \n\n2. Most importantly, just like in ARD, IAD in Eq. (3) uses soft labels from teacher on **clean** images, instead of **adversarial** images. As I mentioned in bullet 1, the soft labels on **clean** images are generally trustworthy, as shown by yourself in the paper. So why does IAD partially trust it? It is never motivated, if I understand correctly. \n\n3. To support your intuitions, the current design of IAD in Eq. (3) might not be the best choice. Specifically, the \"Student Introspection\" KL loss is weighted by (1-\\alpha). This is not so intuitive for me. You have justified the \"Teacher Guidance\" is not always trustworthy so it is good to weight its loss term with \\alpha. However, a trustworthy \"Teacher Guidance\" doesn't mean an untrustworthy \"Student Introspection\". So why do you down-weight \"Student Introspection\" KL loss when \\alpha is large? In fact, the  \"Student Introspection\" KL loss term is just an annealed version of the smoothness loss term in TRADES. As shown in TRADES paper, it regularizes the smoothness of the model which helps robustness. In my view, based on your key motivation to partially trust the **teacher**, the **student** introspection loss term should have a constant weight and instead of dynamically down-weighted as in Eq (3). \n\n4. There are four loss terms in the best performing method AKD2+IDA. It would be nice if ablation studies can be provided on all those loss terms. \n\nOverall, I think the key observations in this paper is very inspiring. If proper modifications can be made to address the above issues, I think it would be a good paper. ", "summary_of_the_paper": "This paper proposes a new knowledge distillation (KD) method for adversarial training. The authors first observed that the soft-labels provided by the teacher gradually becomes less and less reliable during the adversarial training of student model. Based on this observation, they propose to partially trust the soft labels provided by the adversarily pretrained teacher.  ", "summary_of_the_review": "Interesting observation and good intuition, but there is a noticeable gap between the designed method and their intuition. I agree the proposed IAD empirically works, but the intuitions provided in the paper to motivate and interpret IAD is problematic. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "d3zC9kP-oEm", "reviewer": "Reviewer_Q98Y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n+ The research direction is promising and more practical than adversarial training in the real world. As I demonstrated in the summary part, in many ML-driven scenarios, users might not want to retrain their big deep networks using adversarial training due to huge computational cost. Thus, it is, IMHO, urgent to consider to distil the robustness of a robust model. \n+ Although the high-level idea is mentioned in previous papers, this paper considers this problem in detail and find a failure case of previous papers. By considering this failure case, this paper proposes a novel framework to complete this task, which is novel and significant to the development of the field.\n+ This paper is well-written and easy to follow. Experiments are enough to support the claims made in this paper. A plus should be that TRADES and AT are both considered in the experiments, which verifies that the proposed framework is suitable for existing adversarial training methods.\n\nCons:\n- Since the student model is different from the teacher model, it is expected that adversarial data generated by the student model is different from the data used to train the teacher model. This kind of performance drop also happens in other fields, such as domain adaptation, even adversarial machine learning itself. Thus, it is not clear what the point of Figure 1(a) is. I am not sure if 1(a) is necessary. It is more like the transferability of adversarial attacks.\n- Based on the above drawback, it is also unclear if there are other points that make the DISTILLATION fails (except for the distributional discrepancy).\n- The formal problem setting is missing. It is better to provide the formal problem setting to make readers understand the problem clearly.\n- What is the difference between S(\\circ|\\tau) and S(\\circ)? I did not see any definition of S(\\circ|\\tau). How does \\tau effect on S(\\circ|\\tau)?\n- I would like to see how \\tau effects the performance of the proposed method. If we remove \\tau, then the “Student Introspection” term is actually from TRADES. Meanwhile, how does \\tau effect the performance of TRADES? Are there any papers discussing this point?\n- From the experimental results, IAD performs much better than ARD (IMHO, the most direct baseline to IAD), which is very good. I would like to see how \\tau effects the performance of IAD compared to ARD (giving the same \\tau for ARD and IAD).\n- The computational-cost comparison is missing, which is important for distillation-based method.\n- Figure 1(b) is a clear figure, but I cannot see the advantages of your method in this figure. In other words, it is unclear why your method can improve the performance when reading 1(b).\n", "summary_of_the_paper": "Adversarial training has been developed for many years, and it aims to provide reliable classifiers in the era of deep learning. However, in many ML-driven scenarios, users might not want to retrain their big deep networks using adversarial training. The main reason is that training a reliable/adversarial-robust classifier will cost many computational resources. To address this issue, some methods are proposed to distil the robustness from adversarially pre-trained models, which is more practical than the direct adversarial training. This is also the point this paper focuses on. In my humble opinion, this is a very promising research direction and should be paid more attentions in future. \nCompared to existed works, this paper argues that the adversarial training data of the student model and that of the teacher model are egocentric (respectively generated by themselves) and becoming more adversarial challenging during training, which causes failure of existing works. To address this issue, this paper proposes an Introspective Adversarial Distillation (IAD) to effectively utilize the knowledge from an adversarially pre-trained teacher model. Extensive experiments are conducted on CIFAR-10/CIFAR-100 and the more challenging Tiny-ImageNet datasets to evaluate the efficiency of our IAD.\n", "summary_of_the_review": "In general, considering the significance of the researched problem, this paper can be accepted by the ICLR2022. However, some points should be clarified and strengthened in the revision. I would like to strongly support this paper if my concerns can be fully addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "6DJY1VSUa8f", "reviewer": "Reviewer_svED", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros.\n\nThe overall idea is interesting and the proposed method seems simple and practical as well since we only need to adjust the annealing parameter adaptively. Besides, the paper is quite readable and experimental results seem promising. \n\nCons. \n\nThere are some issues to be addressed.\n\n- The results of Natural and FGSM are not good. The tradeoff between Natural performance and robustness should be investigated more. \n\n- In tiny-imagenet and cifar datasets, why IAD + AKD^2 is better than IAD for AA? This phenomenon should be explained more.\n\n- The current evaluation is based on image datasets, like cifar and tiny-imagenet. However, is IAD still working for language datasets?\n\n- It is good to discuss some theoretical justification for IAD, which will make IAD stronger in theory.", "summary_of_the_paper": "In this paper, a new method called introspective adversarial distillation (IAD) is proposed for conventional adversarial training. Concretely, it targets the unreliable teacher case, where teacher is good at adversarial / natural data or none of it. Experimental results validate the effectiveness of the proposed method towards enhancing adversarial robustness. ", "summary_of_the_review": "Basically, this paper is well written and the proposed method seems promising. I thus would like to lean on the acceptance side. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new knowledge distillation (KD) method for adversarial training. The key observation is inspiring: soft-labels provided by the teacher gradually becomes less and less reliable during the adversarial training of student model. Based on that,  they propose to partially trust the soft labels provided by the teacher in adversarial distillation. 

Reviewers unanimously agree that this paper has clear motivation, well-sorted logic, and neat writing. While some reviewers initially posed concerns on evaluation completeness and detail clarification, they were well addressed during the rebuttal. AC reads the paper/discussion thread and agrees this is a worthy work to get accepted.
Decision: Accept (Poster)