Abstract:
In ordinary distillation, student networks are trained with soft labels (SLs) given by pretrained teacher networks, and students are expected to improve upon teachers since SLs are stronger supervision than the original hard labels. However, when considering adversarial robustness, teachers may become unreliable and adversarial distillation may not work: teachers are pretrained on their own adversarial data, and it is too demanding to require that teachers are also good at every adversarial data queried by students. Therefore, in this paper, we propose reliable introspective adversarial distillation (IAD) where students partially instead of fully trust their teachers. Specifically, IAD distinguishes between three cases given a query of a natural data (ND) and the corresponding adversarial data (AD): (a) if a teacher is good at AD, its SL is fully trusted; (b) if a teacher is good at ND but not AD, its SL is partially trusted and the student also takes its own SL into account; (c) otherwise, the student only relies on its own SL. Experiments demonstrate the effectiveness of IAD for improving upon teachers in terms of adversarial robustness.

heading: 1 INTRODUCTION
text: Deep Neural Networks (DNNs) have shown excellent performance on a range of tasks in computer vision (He et al., 2016) and natural language processing (Devlin et al., 2019). Nevertheless, Szegedy et al. (2014); Goodfellow et al. (2015) demonstrated that DNNs could be easily fooled by adding a small number of perturbations on natural examples, which increases the concerns on the robustness of DNNs in the trustworthy-sensitive areas, e.g., finance (Kumar et al., 2020) and autonomous driving (Litman, 2017). To overcome this problem, adversarial training (Goodfellow et al., 2015; Madry et al., 2018) is proposed and has shown effectiveness to acquire the adversarially robust DNNs.
Most existing adversarial training approaches focus on learning from data directly. For example, the popular adversarial training (AT) (Madry et al., 2018) leverages multi-step projected gradient descent (PGD) to generate the adversarial examples and feed them into the standard training. Zhang et al. (2019) developed TRADES on the basis of AT to balance the standard accuracy and robust performance. Recently, there are several methods under this paradigm are developed to improve the model robustness (Wang et al., 2019; Alayrac et al., 2019; Zhang et al., 2020; Jiang et al., 2020; Ding et al., 2020; Wang et al., 2020b; Du et al., 2021; Tian et al., 2021; Zhang et al., 2021). However, directly learning from the adversarial examples is a challenging task on the complex datasets since the loss with hard labels is difficult to be optimized, which limits us to achieve higher robust accuracy.
To mitigate this issue, one emerging direction is distilling robustness from the adversarially pre-trained model intermediately, which has shown promise in the recent study. For example, Ilyas et al. (2019) used an adversarially pre-trained model to build a “robustified” dataset to learn a robust DNN. Chen et al. (2020); Salman et al. (2020) explored to boost the model robustness through fine-tuning or transfer learning from adversarially pre-trained models. Goldblum et al. (2020)and Chen et al. (2021) investigated distilling the robustness from adversarially pre-trained models, termed as adversarial distillation for simplicity, where they encouraged student models to mimic the outputs (i.e., soft labels) of the adversarially pre-trained teachers.
However, one critical difference is: in the conventional distillation, the teacher model and the student model share the natural training data; while in the adversarial distillation, the adversarial training data of the student model and that of the teacher model are egocentric (respectively generated by themselves) and becoming more adversarial challenging during training. Given this distinction, are the
soft labels acquired from the teacher model in adversarial distillation always reliable and informative guidance? To answer this question, we take a closer look at the process of adversarial distillation. As shown in Figure 1(a), we discover that along with the training, the teacher model progressively fails to give a correct prediction for the adversarial data queried by the student model. The reason could be that with the students being more adversarially robust and thus the adversarial data being harder, it is too demanding to require the teachers become always good at every adversarial data queried by the student model, as the teacher model has never seen these data in its pre-training. In contrast, for the conventional distillation, student models are expected to distill the “static” knowledge from the teacher model, since the soft labels for the natural data from the teacher model are always fixed.
The observation in Figure 1(a) raises the challenge: how to conduct reliable adversarial distillation with unreliable teachers? To solve this problem, we can categorize the training data according to the prediction on natural and student generated adversarial data into three cases. First, if the teacher model can correctly classify both natural and adversarial data, it is reliable; Second, if the teacher model can correctly classify the natural but not adversarial data, it should be partially trusted, and the student model is suggested to trust itself to enhance model robustness as the adversarial regularization (Zhang et al., 2019); Third, if the teacher model cannot correctly classify both natural and adversarial data, the student model is recommended to trust itself totally. According to this intuition, we propose an Introspective Adversarial Distillation (IAD) to effectively utilize the knowledge from an adversarially pre-trained teacher model. The framework of our proposed IAD can be seen in Figure 1(b). Briefly, the student model is encouraged to partially instead of fully trust the teacher model, and gradually trust itself more as being more adversarially robust. We conduct extensive experiments on the benchmark CIFAR-10/CIFAR-100 and the more challenging Tiny-ImageNet datasets to evaluate the efficiency of our IAD. The main contributions of our work can be summarized as follows.
1. We take a closer look at adversarial distillation under the teacher-student paradigm. Considering adversarial robustness, we discover that the guidance from the teacher model is progressively unreliable along with the adversarial training.
2. We construct the reliable guidance for adversarial distillation by flexibly utilizing the robust knowledge from the teacher model: (a) if a teacher is good at adversarial data, its soft labels can be fully trusted; (b) if a teacher is good at natural data but not adversarial data, its soft labels should be partially trusted and the student also takes its own soft labels into account; (c) otherwise, the student only relies on its own soft labels.
3. We propose an Introspective Adversarial Distillation (IAD) to automatically realize the intuition of the previous reliable guidance during the adversarial distillation. The experimental results confirmed that our approach can improve adversarial robustness across a variety of training settings and evaluations, and also on the challenging (consider adversarial
robustness) datasets (e.g., CIFAR-100 (Krizhevsky, 2009) and Tiny-ImageNet (Le & Yang, 2015)) or using large models (e.g., WideResNet (Zagoruyko & Komodakis, 2016)).

heading: 2 RELATED WORK

heading: 2.1 ADVERSARIAL TRAINING.
text: Adversarial examples (Goodfellow et al., 2015) motivate many defensive approaches developed in the last few years. Among them, adversarial training has been demonstrated as the most effective method to improve the robustness of DNNs (Cai et al., 2018; Wang et al., 2020a;b; Jiang et al., 2020; Bai et al., 2021; Chen et al., 2021; Wu et al., 2020). The formulation of the popular AT (Madry et al., 2018) and its variants can be summarized as the minimization of the following loss:
min fθ∈F
1
n n∑ i=1 `(fθ(x̃i), yi), (1)
where n is the number of training examples, x̃i is the adversarial example within the -ball (bounded by an Lp-norm) centered at natural example xi, yi is the associated label, fθ is the DNN with parameter θ and `(·) is the standard classification loss, e.g., the cross-entropy loss. Adversarial training leverages adversarial examples to smooth the small neighborhood, making the model prediction locally invariant. To generate the adversarial examples, AT employs a PGD method (Madry et al., 2018). Concretely, given a sample x(0) ∈ X and the step size β > 0, PGD recursively searches
x̃(t+1) = ΠB[x̃(0)] ( x̃(t) + β sign(∇x̃(t)`(fθ(x̃(t)), y)) ) , (2)
until a certain stopping criterion is satisfied. In Eq. equation 2, t ∈ N, ` is the loss function, x̃(t) is adversarial data at step t, y is the corresponding label for natural data, and ΠB [x̃0](·) is the projection function that projects the adversarial data back into the -ball centered at x̃(0).

heading: 2.2 KNOWLEDGE DISTILLATION
text: The idea of distillation from other models can be dated back to (Craven & Shavlik, 1996), and reintroduced by (Hinton et al., 2015) as knowledge distillation (KD). It has been widely studied in recent years and works well in numerous applications like model compression and transfer learning. For adversarial defense, a few studies have explored obtaining adversarial robust models by distillation. Papernot et al. (2016) proposed defensive distillation which utilizes the soft labels produced by a standard pre-trained teacher model, while this method is proved to be not resistant to the C&W attacks (Carlini & Wagner, 2017); Goldblum et al. (2020) combined AT with KD to transfer robustness to student models, and they found that the distilled models can outperform adversarially pre-trained teacher models of identical architecture in terms of adversarial robustness; Chen et al. (2021) utilized distillation as a regularization for adversarial training, which employed both robust and standard pre-trained teacher models to address the robust overfitting (Rice et al., 2020).
Nonetheless, all these related methods fully trust teacher models and do not consider that whether the guidance of the teacher model in distillation is reliable or not. In this paper, different from the previous studies, we find that the teacher model in adversarial distillation is not always trustworthy. Based on that, we propose reliable IAD to encourage student models to partially instead of fully trust teacher models, which effectively utilizes the knowledge from the adversarially pre-trained models.

heading: 3 A CLOSER LOOK AT ADVERSARIAL DISTILLATION
text: In Section 3.1, we discuss the unreliable issue of adversarial distillation, i.e., the guidance of the teacher model is progressively unreliable along with adversarial training. In Section 3.2, we partition the training examples into three parts and analyze them part by part. Specifically, we expect that the student model should partially instead of fully trust the teacher model and gradually trust itself more along with adversarial training.

heading: 3.1 FULLY TRUST: PROGRESSIVELY UNRELIABLE GUIDANCE
text: As aforementioned in the Introduction, previous methods (Goldblum et al., 2020; Chen et al., 2021) fully trust the teacher model when distilling robustness from adversarially pre-trained models. Taking Adversarial Robust Distillation (ARD) (Goldblum et al., 2020) as an example, we illustrate its procedure in the left part of Figure 1(b): the student model generates its adversarial data and then optimizes the prediction of them to mimic the output of the teacher model. However, although the teacher model is well optimized on the adversarial data queried by itself, we argue that it might not always be good at the more and more challenging adversarial data queried by the student model.
As shown in Figure 1(a), different from the ordinary distillation in which the teacher model has the consistent standard performance on the natural data, its robust accuracy on the student model’s adversarial data is decreasing during distillation. The guidance of the teacher model gradually fails to give the correct output on the adversarial data queried by the student model.

heading: 3.2 PARTIALLY TRUST: CONSTRUCTION OF RELIABLE GUIDANCE
text: The unreliable issue of the teacher model in adversarial distillation raises the challenge of how to conduct reliable adversarial distillation with unreliable teachers? Intuitively, this requires us to re-consider the guidance of adversarially pre-trained models along with the adversarial training. For simplicity, we use T (x) (T (x̃)) to represent the predicted label of the teacher model on the natural (adversarial) examples, and use y to represent the targeted label. We partition the adversarial samples into following parts as shown in the toy illustration (Figure 2(a)), and analyze them part by part.
1) T (x) = y ∩ T (x̃) = y: It can be seen in Figure 2(a) that this part of data whose adversarial variants like x′1 is the most trustworthy among the three parts, since the teacher model performs well on both natural and adversarial data. In this case, we could choose to trust the guidance of the teacher model on this part of the data. However, as shown in Figure 2(b), we find that the sample number of this part is decreasing along with the adversarial training. That is, what we can rely on from the teacher model in adversarial distillation is progressively reduced.
2) T (x) = y ∩ T (x̃) 6= y: In Figure 2(b), we also check the number change of the part of data whose adversarial variants like x′′1 . Corresponding to the previous category, the number of this kind of data is increasing during distillation. Since the teacher model’s outputs on the small neighborhood of the queried natural data are not always correct, its knowledge may not be robust and the guidance for the student model is not reliable. Think back to the reason for the decrease in the robust accuracy of the teacher model, the student model itself may also be trustworthy since it becomes gradually adversarial robust during distillation.
3) T (x) 6= y∩T (x̃) 6= y: As for the data which are like x2 in Figure 2(a), the guidance of the teacher model is totally unreliable since the predicted labels on the natural data are wrong. The student model may also trust itself to encourage the outputs to mimic that of their natural data rather than the wrong outputs from the teacher model. First, it removes the potential threat that the teacher’s guidance may be a kind of noisy labels for training. Second, as an adversarial regularization (Zhang et al., 2019), it can improve the model robustness through enhancing the stability of the model’s outputs on the natural and the corresponding adversarial data.
4) T (x) 6= y ∩ T (x̃) = y: Considering the generation process of the adversarial data, i.e., x̃∗ = arg maxx̃∈B (x) `(f(x̃), y), Once the original prediction is wrong, i.e.,, T (x) 6= y, the generation of x̃∗ only make the prediction worse. Thus, this group doesn’t exist.
To sum up, we suggest employing reliable guidance from the teacher model and encouraging the student model to trust itself more as the teacher model’s guidance being progressively unreliable and the student model gradually becoming more adversarially robust.

heading: 4 INTROSPECTIVE ADVERSARIAL DISTILLATION
text: Based on previous analysis about the adversarial distillation, we propose the Introspective Adversarial Distillation (IAD) to better utilize the guidance from the adversarially pre-trained model. Concretely, we have the following KD-style loss, but composite with teacher guidance and student introspection.
`IAD = O(ADi;α)︸ ︷︷ ︸ Label & Teacher Guidance +γ `KL(S(x̃|τ)||S(x|τ))︸ ︷︷ ︸ Student Introspection ), (3)
where O(ADi;α) is the previous adversarial distillation baseline, e.g., ARD (Goldblum et al., 2020) or AKD2 (Chen et al., 2021), weighted by the hyper-parameter α1, γ is a weight for the student introspection, S(·|τ) is a Softmax operator with the temperature τ ∈ (0,+∞), e.g., S(xk|τ) = exp(xk/τ)∑ k′ exp(xk′/τ)
, S(·) is the conventional Softmax with the temperature τ = 1, T (·|τ) is the tempered variant of the teacher output T (·), x̃ is the adversarial data generated from the natural data x, y is the hard label and `CE is the Cross Entropy loss and `KL is the KL-divergence loss. As for the annealing parameter α ∈ [0, 1] that is used to balance the effect of the teacher model in adversarial distillation, based on the analysis about the reliability of adversarial supervision in Section 3, we define it as,
α = (PT (x̃|y))β , (4)
where PT (·|y) is the prediction probability of the teacher model about the targeted label y and β is a hyperparameter to sharpen the prediction. The intuition behind IAD is to calibrate the guidance from the teacher model automatically based on the prediction of adversarial data. Our α naturally corresponds to the construction in Section 3.2, since the prediction probability of the teacher model for the adversarial data can well represent the categorical information. As for β, we have plot the specific values of α under its adjustment in the left of Figure 4.
1Note that, we do not use α when ADi is ARD. Please refer to the Appendix A.2 for the ablation study.
Algorithm 1 Introspective Adversarial Distillation (IAD) Input: student model S, teacher model T , training dataset D = {(xi, yi)}ni=1, learning rate η, number of epochs N , batch size m, number of batches M , temperature parameter τ , the annealing parameter on teacher model’s predicted probability α, adjustable parameter λ, λ1, λ2, λ3 and γ. Output: adversarially robust model Sr for epoch = 1, . . . , N do
for mini-batch = 1, . . . , M do Sample a mini-batch {(xi, yi)}mi=1 from D for i = 1, . . . , m (in parallel) do
Obtain adversarial data x̃i of xi by PGD based on Eq. equation 2. Compute α for each adversarial data based on Eq. equation 4.
end for IAD-I: θ ← θ − η∇θ {
λ`CE(S(x), y) + (1− λ) · τ2· (`KL(S(x̃|τ)||T (x|τ)) + γ`KL(S(x̃|τ)||S(x|τ))) } or
IAD-II: θ ← θ−η∇θ  λ1`CE(S(x̃), y) + λ2 · τ2· (α · `KL(S(x̃|τ)||Tat(x̃|τ))+λ3τ2`KL(S(x̃|τ)||Tst(x̃|τ))+
γ`KL(S(x̃|τ)||S(x|τ)))  end for
end for
Intuitively, the student model can trust the teacher model when α approaches 1, which means that the teacher model is good at both natural and adversarial data. However, when α approaches 0, it corresponds that the teacher model is good at natural but not adversarial data, or even not good at both, and thus the student model should take its self-introspection into account. In Figure 3, we check the reliability of the student model itself. According to the left panel of Figure 3, we can see that the student model is progressively robust to the adversarial data. And if we incorporate the student introspection into the adversarial distillation, the results in the middle of Figure 3 confirms its potential benefits to improve the accuracy of the guidance. Moreover, as shown in the right panel of Figure 3, adding self-introspection results in better improvement in model robustness compared to only using the guidance of the teacher model. Therefore, `IAD automatically encourages the outputs of the student model to mimic more reliable guidance in adversarial distillation.
Algorithm 1 summarizes the implementation of Introspective Adversarial Distillation (IAD). Specifically, IAD first leverages PGD to generate the adversarial data for the student model. Secondly, IAD computes the outputs of the teacher model and the student model on the natural data. Then, IAD mimics the outputs of the student model with that of itself and the teacher model partially based on the probability of the teacher model on the adversarial data.
Warming-up period. During training, we add a warming-up period to activate the student model, where α (in Eq. equation 3) is hardcoded to 1. This is because the student itself is not trustworthy in the early stage (refer to the left panel of Figure 3). Through that, we expect the student model to first evolve into a relatively reliable learner and then conducts the procedure of IAD.

heading: 4.1 COMPARISON WITH RELATED METHODS
text: In this section, we discuss the difference between IAD and other related approaches in the perspective of the loss functions. Table 1 summarizes all of them.
As shown in Table 1, AT (Madry et al., 2018) utilizes the hard labels to supervise adversarial training; TRADES (Zhang et al., 2019) decomposes the loss function of AT into two terms, one for standard training and the other one for adversarial training with the soft supervision; Motivated by KD (Hinton et al., 2015), Goldblum et al. (2020) proposed ARD to conduct the adversarial distillation, which fully trusts the outputs of the teacher model to learn the student model. As indicated by the experiments in Goldblum et al. (2020), a larger λ resulted in less robust student models. Thus, they generally set λ = 0 in their experiments; Chen et al. (2021) utilized distillation as a regularization to avoid the robust overfitting issue, which employed both the adversarially pre-trained teacher model and
the standard pre-trained model. Thus, there are two KL-divergence loss and for simplicity, we term their method as AKD2; Regarding IAD, there are two types of implementations that are respectively based on ARD or AKD2. We term them as IAD-I and IAD-II, and their difference with previous ARD and AKD2 is an additional self-introspection term. Besides, we also apply α to downweight the dependency on the term `KL(S(x̃|τ)||Tat(x̃|τ)), which has been explained in previous sections.

heading: 5 EXPERIMENTS
text: We conduct extensive experiments to evaluate the effectiveness of IAD. In Section 5.1, we compare IAD with benchmark adversarial training methods (AT and TRADES) and some related methods which utilize adversarially pre-trained models via KD (ARD and AKD2) on CIFAR-10/CIFAR100 (Krizhevsky, 2009) datasets. In Section 5.2, we compare the previous methods with IAD on a more challenging dataset Tiny-ImageNet (Le & Yang, 2015). In Section 5.3, the ablation studies are conducted to analyze the effects of the hyper-parameter β and different warming-up periods for IAD.
Regarding the measures, we compute the natural accuracy on the natural test data and the robust accuracy on the adversarial test data generated by FGSM, PGD-20, and C&W attacks following (Wang et al., 2019) Moreover, we estimate the performance under AutoAttack (termed as AA).
5.1 EVALUATION ON CIFAR-10/CIFAR-100 DATASETS
Experiment Setup. In this part, we follow the setup (learning rate, optimizer, weight decay, momentum) of Goldblum et al. (2020) to implement the adversarial distillation experiments on the CIFAR-10/CIFAR-100 datasets. Specifically, we train ResNet-18 under different methods using SGD with 0.9 momentum for 200 epochs. The initial learning rate is 0.1 divided by 10 at Epoch 100 and Epoch 150 respectively, and the weight decay=0.0002. In the settings of adversarial training, we set the perturbation bound = 8/255, the PGD step size σ = 2/255, and PGD step numbers K = 10. In the settings of distillation, we use τ = 1 and use models pre-trained by AT and TRADES which have the best PGD-10 test accuracy as the teacher models for ARD, AKD2 and our IAD. For ARD, we set its hyper-parameter λ = 0 as recommend in Goldblum et al. (2020) for gaining better robustness. For AKD2, we set λ1 = 0.25, λ2 = 0.5 and λ3 = 0.25 as recommanded in Chen et al. (2021). For IAD-I and IAD-II, we respectively set the warming-up period as 60/80 and 40/80 epochs to train on CIFAR-10/CIFAR-100. Regarding the computation of α, we set λ = 0, β = 0.1. For γ, we currently set γ = 1− α and more ablation study about its setting can be found in the Appendix A.3.
Results. We report the results in Table 2, where the results of AT and TARDES are listed in the first and fifth rows of Table 2, and the other methods use these models as the teacher models in distillation. On CIFAR-10 and CIFAR-100, we note that our IAD-I or IAD-II has obtained consistent improvements on adversarial robustness in terms of PGD-20, CW∞ and AA accuracy compared with the student models distilled by ARD or AKD2 and the adversarially pre-trained teacher models. Besides, IAD-II generally performs better than IAD-I when the teacher is trained by AT, which means AKD2 in this case could be a better starting point than ARD. However, when the teacher model is trained by TRAEDS, the advantage about the robustness of IAD-I over that of IAD-II is in reverse. Considering their distillation philosophy, i.e., `kl(S(x̃|τ)||T (x|τ))) of IAD-I and `kl(S(x̃|τ)||T (x̃|τ))) of IAD-II, it might be up to which of T (x|τ) and T (x̃|τ) is more informative in adversarial distillation from the diverse teachers. The natural accuracy of IAD sometimes is lower than others, but the performance drop is not very significant compared to IAD-II.
Experiment Setup. In this part, we evaluate these methods by the model with a larger capacity, i.e, WideResNet-34-10. The teacher network is trained by AT and TRADES, following the settings of (Zhang et al., 2021). We keep the most settings of baselines same as the previous experiment. For IAD-I and IAD-II, we set 5/10 epochs warming-up period on CIFAR-10/CIFAR-100.
Results. The results is summarized in Tables 3. Similarly, on CIFAR-10 and CIFAR-100, our method can also achieve better model robustness than ARD, AKD2 and the original teacher models. Moreover, our IAD methods do not sacrifice much standard performance compared with the original teacher models. Since AKD2 externally utilizes a standard pre-trained teacher model, IAD-II can achieve the consistently better natural performance than IAD-I. However, in terms of the robustness, IAD-I generally achieves the comparable or even better than IAD-II under both AT and TRADES.
5.2 EVALUATION ON Tiny-ImageNet DATASET
Experiment Setup. In this part, we evaluate these methods on a more challenging Tiny-ImageNet dataset. For these adversarially pre-trained models, we follow the settings of (Chen et al., 2021) to train AT and TRADES. To be specific, we train PreActive-ResNet-18 using SGD with 0.9 momentum
for 100 epochs. The initial learning rate is 0.1 divided by 10 at Epoch 50 and 80 respectively, and the weight decay=0.0005. For distillation baselines, we keep most settings the same as Section 5.1. For ARD and IAD-I, here we set its λ = 0.9 to deal with the complex task following (Goldblum et al., 2020). And for both IAD-I and IAD-II, we use λ = 0.1, β = 0.1 and 10 epochs as the warming-up period.
Results. We report the results in Table 4. Overall, our IAD-I or IAD-II can still achieve better model robustness than other methods. Specifically, on Tiny-ImageNet, IAD-II can approximately improve both the natural accuracy and the robust accuracy compared to IAD-I and other baselines.

heading: 5.3 ABLATION STUDIES
text: To give a comprehensive understanding of our proposed IAD method, we have conducted a series of experiments (in Appendix A), including the ablation study on using different γ related to student introspection, different τ related to adversarial distillation and the loss terms of IAD-II as well as the comparison of the computational cost. In the following, we only study the effect of β and warming-up periods for our student introspection. More ablation study can be found in the Appendix.
Experiment Setup. To understand the effects of different β and different warming-up periods on CIFAR-10 dataset, we conduct the ablation study in this part. Specifically, we choose the ResNet-18 as the backbone model, and keep the experimental settings the same as Section 5.1. In the first experiments, we set no warming-up period and study the effect of using different β. Then, in the second experiments, we set β = 0.1 and investigate different warming-up periods.
Results. We report part of the results in Figure 4. The complete results with other evaluation metrics can be found in Appendix A.1. In Figure 4, we first visualize the values of the α using different β in the left panel, which shows the proportion of the teacher guidance and student introspection in adversarial distillation. The bigger the beta corresponds to a larger proportion of the student introspection. In the middle panel, we plot the natural and AA accuracy of the student models distilled by different β. We note that the AA accuracy is improved when the student model trusts itself more with the larger β value. However, the natural accuracy is decreasing along with the increasing of the β value. Similarly, we adjust the length of warming-up periods and check the natural and AA accuracy in the right panel of Figure 4. We find that setting the student model partially trust itself at the beginning of the training process leads to inadequate robustness improvements and more sacrifice on natural accuracy. An appropriate warming-up period at the early stage can improve the student model performance on the adversarial examples.

heading: 6 CONCLUSION
text: In this paper, we study distillation from adversarially pre-trained models. We take a closer look at adversarial distillation and discover that the guidance of teacher model is progressively unreliable by considering the robustness. Hence, we explore the construction of reliable guidance in adversarial distillation and propose a method for distillation from unreliable teacher models, i.e., Introspective Adversarial Distillation. Our methods encourages the student model partially instead of fully trust the guidance of the teacher model and gradually trust its self-introspection more to improve robustness.

heading: 7 ETHICS STATEMENT
text: This paper does not raise any ethics concerns. This study does not involve any human subjects, practices to data set releases, potentially harmful insights, methodologies and applications, potential conflicts of interest and sponsorship, discrimination/bias/fairness concerns, privacy and security issues, legal compliance, and research integrity issues.

heading: 8 REPRODUCIBILITY STATEMENT
text: To ensure the reproducibility of experimental results, we will provide a link for an anonymous repository about the source codes of this paper in the discussion forums.

heading: A COMPREHENSIVE EXPERIMENTS
text: A.1 COMPLETE RESULTS OF ABLATION STUDIES.
In this section, we report the complete results of our ablation studies in Tables 5 (about β) and 6 (about warming-up periods).
In Table 5, we can see that the natural and FGSM accuracy will decrease, and the robust accuracy (PGD-20, CW∞, AA) will increase with the rise of β. In Table 6, we adjust the length of warming-up periods. We can see that letting the student network partially trust itself at the beginning of the training process would result in inadequate robustness improvements. In summary, we can find that there is a trade-off in Table 5 to achieve both larger natural accuracy and larger robustness in adversarial distillation, which is also similar to the standard adversarial training. We can slightly sacrifice the robustness (adjust the β or adjust the warming-up periods) to acquire a better natural accuracy and FGSM.
A.2 FULLY TRUST OR PARTIALLY TRUST THE TEACHER GUIDANCE
In this section, we compare the method with fully trusted or partially trusted teacher guidance. Regarding the variant of AKD2 that replaces the second term in AKD2 by our ”partially trust“ KL term (but without the introspection term), we find the robust improvement in Table 7.
In the ”partially trust” variant of AKD2, we just down weight the part of teacher guidance which has wrong prediction results with the hard labels. The results show that fit this part ”unreliable” teacher guidance may improve the natural performance but consistently drop the robust accuracy.
In the following, we also conduct a range of experiments to compare IAD-I and its variant IAD-I-cons that replaces the downweighting with a constant 1.0 like ARD. According to the results, we can
see that IAD-I-cons consistently achieves better robustness while sacrifices a little natural accuracy. Hence, we will choose the constant 1.0 on `kl(S(x̃|τ)||T (x|τ)) for ARD in the main part of our experiments on those benchmark datasets.
A.3 THE EFFECTS OF γ FOR STUDENT INTROSPECTION
In this section, we check the effects of the Student Introspection via adjusting the γ for our IAD method. Here, we also use the constant γ in the experiments. We find that using the constant γ can further boost the performance on model robustness, but there is another problem that the natural accuracy will sacrifice a little bit more than our previous dynamical design, i.e., γ = 1− α.
According to the results in Table 9, we can hardly to find an optimal coefficient for the student introspection term to achieve both the best natural accuracy and the best robustness. However, there is one trend that increasing the coefficient will gain more robustness with losing more natural accuracy. With the hyper-parameter γ, we may flexibly instantiated by some constants or some strategic schedules to pursue the robustness or the natural accuracy.
A.4 THE EFFECTS OF τ FOR ADVERSARIAL DISTILLATION
In this section, we check the effects of τ for adversarial distillation under the same training configurations and also for TRADES. We have listed the results of TRADES in Tabel 10, and the results of IAD and ARD in Table 11.
As for TRADES, we think it may not need the temperature τ which is especially designed for distillation, since the KL term in TRADES is aims to encourage the output of the model on adversarial data to being the similar with that on natural data. Intuitively, it encourage the output to be more stable which leads to better robust performance. As a result, based on the test accuracy in Table 10,
τ = 1 maybe the best for TRADES to achieve a better robustness and enlarging (or decreasing) τ can disturb the original output logits and may result in lower robustness (i.e., τ = 0.1: PGD-20 43.26% ).
According to the comparison in Table 11, with the proper τ (e.g., τ ≥ 1), IAD can achieve the comparable or better natural accuracy and better robustness.
A.5 COMPARISON WITH TRADES AND ARD
In this section, we compared the performance of TRADES, ARD and our IAD with different weights in their each loss terms. We summarized the results in Table 12.
According to the results, IAD can achieve the better natural accuracy than ARD with simultaneously better robustness, and the better natural accuracy than TRADES with less drop on PGD-20 accuracy. With varying the hyper-parameters on the loss terms, we also find that natural accuracy can reach to 85.94% at a loss of robustness to 48.58% and robustness can reach to 55.89% at a loss of natural accuracy to 74.95%. In summary, our IAD makes a good balance among them by considering both the natural accuracy and the robustness.
A.6 ABLATION STUDY ON THE LOSS TERMS OF IAD-II
In this section, we conduct the ablation study in Table 13 about the four loss terms in IAD-II which is based on AKD2.
Specifically, according to the results, we find `KL(S(x̃|τ)||Tst(x̃|τ)) can help model gain more natural accuracy, while `KL(S(x̃|τ)||Tst(x̃|τ)) can help model gain more robustness. AKD2 achieves a good balance between two aspects by combining above two terms, and IAD-II further boosts the model robustness by incorporating the self-introspection term with less drop in natural performance.
A.7 COMPUTATIONAL COST COMPARISON
In this section, we check and compare the computational cost of ARD, AKD2, IAD-I and IAD-II in terms of the averaged training time in each epoch, as well as the GPU Memory-Usage. The detailed results are summarized in Table 14.
According to the results, IAD-I and IAD-II correspondingly consume a bit more time and memory than ARD and AKD2 due to the additional self-introspection. Besides, an interesting phenomenon is that ARD has less terms than AKD2, but consumes more time and GPU memory. This is because ARD has to deal with both x and x̃, while AKD2 only needs to deal with x.

Review summaries:
This paper proposes a new knowledge distillation (KD) method for adversarial training. The authors first observed that the soft-labels provided by the teacher gradually becomes less and less reliable during the adversarial training of student model. Based on this observation, they propose to partially trust the soft labels provided by the adversarily pretrained teacher.  

In this paper, a new method called introspective adversarial distillation (IAD) is proposed for conventional adversarial training. Concretely, it targets the unreliable teacher case, where teacher is good at adversarial / natural data or none of it. Experimental results validate the effectiveness of the proposed method towards enhancing adversarial robustness. 

This paper studies a specific distillation scenario, adversarial distillation, to improve the model robustness. Different from the constant sort supervision in the ordinary distillation, the teacher model will become progressively unreliable along with training, since the adversarial data are dynamically searched by the student model and might not be well identified by the teacher model. Therefore, the authors introduce an introspective adversarial distillation, which considers to bootstrap the learning from both the teacher model and itself. A range of experiments demonstrate its superiority on improving the model robustness.

Adversarial training has been developed for many years, and it aims to provide reliable classifiers in the era of deep learning. However, in many ML-driven scenarios, users might not want to retrain their big deep networks using adversarial training. The main reason is that training a reliable/adversarial-robust classifier will cost many computational resources. To address this issue, some methods are proposed to distil the robustness from adversarially pre-trained models, which is more practical than the direct adversarial training. This is also the point this paper focuses on. In my humble opinion, this is a very promising research direction and should be paid more attentions in future.  Compared to existed works, this paper argues that the adversarial training data of the student model and that of the teacher model are egocentric (respectively generated by themselves) and becoming more adversarial challenging during training, which causes failure of existing works. To address this issue, this paper proposes an Introspective Adversarial Distillation (IAD) to effectively utilize the knowledge from an adversarially pre-trained teacher model. Extensive experiments are conducted on CIFAR-10/CIFAR-100 and the more challenging Tiny-ImageNet datasets to evaluate the efficiency of our IAD.