Reliable Adversarial Distillation with Unreliable Teachers

1 INTRODUCTION . Deep Neural Networks ( DNNs ) have shown excellent performance on a range of tasks in computer vision ( He et al. , 2016 ) and natural language processing ( Devlin et al. , 2019 ) . Nevertheless , Szegedy et al . ( 2014 ) ; Goodfellow et al . ( 2015 ) demonstrated that DNNs could be easily fooled by adding a small number of perturbations on natural examples , which increases the concerns on the robustness of DNNs in the trustworthy-sensitive areas , e.g. , finance ( Kumar et al. , 2020 ) and autonomous driving ( Litman , 2017 ) . To overcome this problem , adversarial training ( Goodfellow et al. , 2015 ; Madry et al. , 2018 ) is proposed and has shown effectiveness to acquire the adversarially robust DNNs . Most existing adversarial training approaches focus on learning from data directly . For example , the popular adversarial training ( AT ) ( Madry et al. , 2018 ) leverages multi-step projected gradient descent ( PGD ) to generate the adversarial examples and feed them into the standard training . Zhang et al . ( 2019 ) developed TRADES on the basis of AT to balance the standard accuracy and robust performance . Recently , there are several methods under this paradigm are developed to improve the model robustness ( Wang et al. , 2019 ; Alayrac et al. , 2019 ; Zhang et al. , 2020 ; Jiang et al. , 2020 ; Ding et al. , 2020 ; Wang et al. , 2020b ; Du et al. , 2021 ; Tian et al. , 2021 ; Zhang et al. , 2021 ) . However , directly learning from the adversarial examples is a challenging task on the complex datasets since the loss with hard labels is difficult to be optimized , which limits us to achieve higher robust accuracy . To mitigate this issue , one emerging direction is distilling robustness from the adversarially pre-trained model intermediately , which has shown promise in the recent study . For example , Ilyas et al . ( 2019 ) used an adversarially pre-trained model to build a “ robustified ” dataset to learn a robust DNN . Chen et al . ( 2020 ) ; Salman et al . ( 2020 ) explored to boost the model robustness through fine-tuning or transfer learning from adversarially pre-trained models . Goldblum et al . ( 2020 ) and Chen et al . ( 2021 ) investigated distilling the robustness from adversarially pre-trained models , termed as adversarial distillation for simplicity , where they encouraged student models to mimic the outputs ( i.e. , soft labels ) of the adversarially pre-trained teachers . However , one critical difference is : in the conventional distillation , the teacher model and the student model share the natural training data ; while in the adversarial distillation , the adversarial training data of the student model and that of the teacher model are egocentric ( respectively generated by themselves ) and becoming more adversarial challenging during training . Given this distinction , are the soft labels acquired from the teacher model in adversarial distillation always reliable and informative guidance ? To answer this question , we take a closer look at the process of adversarial distillation . As shown in Figure 1 ( a ) , we discover that along with the training , the teacher model progressively fails to give a correct prediction for the adversarial data queried by the student model . The reason could be that with the students being more adversarially robust and thus the adversarial data being harder , it is too demanding to require the teachers become always good at every adversarial data queried by the student model , as the teacher model has never seen these data in its pre-training . In contrast , for the conventional distillation , student models are expected to distill the “ static ” knowledge from the teacher model , since the soft labels for the natural data from the teacher model are always fixed . The observation in Figure 1 ( a ) raises the challenge : how to conduct reliable adversarial distillation with unreliable teachers ? To solve this problem , we can categorize the training data according to the prediction on natural and student generated adversarial data into three cases . First , if the teacher model can correctly classify both natural and adversarial data , it is reliable ; Second , if the teacher model can correctly classify the natural but not adversarial data , it should be partially trusted , and the student model is suggested to trust itself to enhance model robustness as the adversarial regularization ( Zhang et al. , 2019 ) ; Third , if the teacher model can not correctly classify both natural and adversarial data , the student model is recommended to trust itself totally . According to this intuition , we propose an Introspective Adversarial Distillation ( IAD ) to effectively utilize the knowledge from an adversarially pre-trained teacher model . The framework of our proposed IAD can be seen in Figure 1 ( b ) . Briefly , the student model is encouraged to partially instead of fully trust the teacher model , and gradually trust itself more as being more adversarially robust . We conduct extensive experiments on the benchmark CIFAR-10/CIFAR-100 and the more challenging Tiny-ImageNet datasets to evaluate the efficiency of our IAD . The main contributions of our work can be summarized as follows . 1 . We take a closer look at adversarial distillation under the teacher-student paradigm . Considering adversarial robustness , we discover that the guidance from the teacher model is progressively unreliable along with the adversarial training . 2 . We construct the reliable guidance for adversarial distillation by flexibly utilizing the robust knowledge from the teacher model : ( a ) if a teacher is good at adversarial data , its soft labels can be fully trusted ; ( b ) if a teacher is good at natural data but not adversarial data , its soft labels should be partially trusted and the student also takes its own soft labels into account ; ( c ) otherwise , the student only relies on its own soft labels . 3 . We propose an Introspective Adversarial Distillation ( IAD ) to automatically realize the intuition of the previous reliable guidance during the adversarial distillation . The experimental results confirmed that our approach can improve adversarial robustness across a variety of training settings and evaluations , and also on the challenging ( consider adversarial robustness ) datasets ( e.g. , CIFAR-100 ( Krizhevsky , 2009 ) and Tiny-ImageNet ( Le & Yang , 2015 ) ) or using large models ( e.g. , WideResNet ( Zagoruyko & Komodakis , 2016 ) ) . 2 RELATED WORK . 2.1 ADVERSARIAL TRAINING .. Adversarial examples ( Goodfellow et al. , 2015 ) motivate many defensive approaches developed in the last few years . Among them , adversarial training has been demonstrated as the most effective method to improve the robustness of DNNs ( Cai et al. , 2018 ; Wang et al. , 2020a ; b ; Jiang et al. , 2020 ; Bai et al. , 2021 ; Chen et al. , 2021 ; Wu et al. , 2020 ) . The formulation of the popular AT ( Madry et al. , 2018 ) and its variants can be summarized as the minimization of the following loss : min fθ∈F 1 n n∑ i=1 ` ( fθ ( x̃i ) , yi ) , ( 1 ) where n is the number of training examples , x̃i is the adversarial example within the -ball ( bounded by an Lp-norm ) centered at natural example xi , yi is the associated label , fθ is the DNN with parameter θ and ` ( · ) is the standard classification loss , e.g. , the cross-entropy loss . Adversarial training leverages adversarial examples to smooth the small neighborhood , making the model prediction locally invariant . To generate the adversarial examples , AT employs a PGD method ( Madry et al. , 2018 ) . Concretely , given a sample x ( 0 ) ∈ X and the step size β > 0 , PGD recursively searches x̃ ( t+1 ) = ΠB [ x̃ ( 0 ) ] ( x̃ ( t ) + β sign ( ∇x̃ ( t ) ` ( fθ ( x̃ ( t ) ) , y ) ) ) , ( 2 ) until a certain stopping criterion is satisfied . In Eq . equation 2 , t ∈ N , ` is the loss function , x̃ ( t ) is adversarial data at step t , y is the corresponding label for natural data , and ΠB [ x̃0 ] ( · ) is the projection function that projects the adversarial data back into the -ball centered at x̃ ( 0 ) . 2.2 KNOWLEDGE DISTILLATION . The idea of distillation from other models can be dated back to ( Craven & Shavlik , 1996 ) , and reintroduced by ( Hinton et al. , 2015 ) as knowledge distillation ( KD ) . It has been widely studied in recent years and works well in numerous applications like model compression and transfer learning . For adversarial defense , a few studies have explored obtaining adversarial robust models by distillation . Papernot et al . ( 2016 ) proposed defensive distillation which utilizes the soft labels produced by a standard pre-trained teacher model , while this method is proved to be not resistant to the C & W attacks ( Carlini & Wagner , 2017 ) ; Goldblum et al . ( 2020 ) combined AT with KD to transfer robustness to student models , and they found that the distilled models can outperform adversarially pre-trained teacher models of identical architecture in terms of adversarial robustness ; Chen et al . ( 2021 ) utilized distillation as a regularization for adversarial training , which employed both robust and standard pre-trained teacher models to address the robust overfitting ( Rice et al. , 2020 ) . Nonetheless , all these related methods fully trust teacher models and do not consider that whether the guidance of the teacher model in distillation is reliable or not . In this paper , different from the previous studies , we find that the teacher model in adversarial distillation is not always trustworthy . Based on that , we propose reliable IAD to encourage student models to partially instead of fully trust teacher models , which effectively utilizes the knowledge from the adversarially pre-trained models . 3 A CLOSER LOOK AT ADVERSARIAL DISTILLATION . In Section 3.1 , we discuss the unreliable issue of adversarial distillation , i.e. , the guidance of the teacher model is progressively unreliable along with adversarial training . In Section 3.2 , we partition the training examples into three parts and analyze them part by part . Specifically , we expect that the student model should partially instead of fully trust the teacher model and gradually trust itself more along with adversarial training . 3.1 FULLY TRUST : PROGRESSIVELY UNRELIABLE GUIDANCE . As aforementioned in the Introduction , previous methods ( Goldblum et al. , 2020 ; Chen et al. , 2021 ) fully trust the teacher model when distilling robustness from adversarially pre-trained models . Taking Adversarial Robust Distillation ( ARD ) ( Goldblum et al. , 2020 ) as an example , we illustrate its procedure in the left part of Figure 1 ( b ) : the student model generates its adversarial data and then optimizes the prediction of them to mimic the output of the teacher model . However , although the teacher model is well optimized on the adversarial data queried by itself , we argue that it might not always be good at the more and more challenging adversarial data queried by the student model . As shown in Figure 1 ( a ) , different from the ordinary distillation in which the teacher model has the consistent standard performance on the natural data , its robust accuracy on the student model ’ s adversarial data is decreasing during distillation . The guidance of the teacher model gradually fails to give the correct output on the adversarial data queried by the student model .

Summary: Adversarial training has been developed for many years, and it aims to provide reliable classifiers in the era of deep learning. However, in many ML-driven scenarios, users might not want to retrain their big deep networks using adversarial training. The main reason is that training a reliable/adversarial-robust classifier will cost many computational resources. To address this issue, some methods are proposed to distil the robustness from adversarially pre-trained models, which is more practical than the direct adversarial training. This is also the point this paper focuses on. In my humble opinion, this is a very promising research direction and should be paid more attentions in future.  Compared to existed works, this paper argues that the adversarial training data of the student model and that of the teacher model are egocentric (respectively generated by themselves) and becoming more adversarial challenging during training, which causes failure of existing works. To address this issue, this paper proposes an Introspective Adversarial Distillation (IAD) to effectively utilize the knowledge from an adversarially pre-trained teacher model. Extensive experiments are conducted on CIFAR-10/CIFAR-100 and the more challenging Tiny-ImageNet datasets to evaluate the efficiency of our IAD.