Reliable Adversarial Distillation with Unreliable Teachers

1 INTRODUCTION . Deep Neural Networks ( DNNs ) have shown excellent performance on a range of tasks in computer vision ( He et al. , 2016 ) and natural language processing ( Devlin et al. , 2019 ) . Nevertheless , Szegedy et al . ( 2014 ) ; Goodfellow et al . ( 2015 ) demonstrated that DNNs could be easily fooled by adding a small number of perturbations on natural examples , which increases the concerns on the robustness of DNNs in the trustworthy-sensitive areas , e.g. , finance ( Kumar et al. , 2020 ) and autonomous driving ( Litman , 2017 ) . To overcome this problem , adversarial training ( Goodfellow et al. , 2015 ; Madry et al. , 2018 ) is proposed and has shown effectiveness to acquire the adversarially robust DNNs . Most existing adversarial training approaches focus on learning from data directly . For example , the popular adversarial training ( AT ) ( Madry et al. , 2018 ) leverages multi-step projected gradient descent ( PGD ) to generate the adversarial examples and feed them into the standard training . Zhang et al . ( 2019 ) developed TRADES on the basis of AT to balance the standard accuracy and robust performance . Recently , there are several methods under this paradigm are developed to improve the model robustness ( Wang et al. , 2019 ; Alayrac et al. , 2019 ; Zhang et al. , 2020 ; Jiang et al. , 2020 ; Ding et al. , 2020 ; Wang et al. , 2020b ; Du et al. , 2021 ; Tian et al. , 2021 ; Zhang et al. , 2021 ) . However , directly learning from the adversarial examples is a challenging task on the complex datasets since the loss with hard labels is difficult to be optimized , which limits us to achieve higher robust accuracy . To mitigate this issue , one emerging direction is distilling robustness from the adversarially pre-trained model intermediately , which has shown promise in the recent study . For example , Ilyas et al . ( 2019 ) used an adversarially pre-trained model to build a “ robustified ” dataset to learn a robust DNN . Chen et al . ( 2020 ) ; Salman et al . ( 2020 ) explored to boost the model robustness through fine-tuning or transfer learning from adversarially pre-trained models . Goldblum et al . ( 2020 ) and Chen et al . ( 2021 ) investigated distilling the robustness from adversarially pre-trained models , termed as adversarial distillation for simplicity , where they encouraged student models to mimic the outputs ( i.e. , soft labels ) of the adversarially pre-trained teachers . However , one critical difference is : in the conventional distillation , the teacher model and the student model share the natural training data ; while in the adversarial distillation , the adversarial training data of the student model and that of the teacher model are egocentric ( respectively generated by themselves ) and becoming more adversarial challenging during training . Given this distinction , are the soft labels acquired from the teacher model in adversarial distillation always reliable and informative guidance ? To answer this question , we take a closer look at the process of adversarial distillation . As shown in Figure 1 ( a ) , we discover that along with the training , the teacher model progressively fails to give a correct prediction for the adversarial data queried by the student model . The reason could be that with the students being more adversarially robust and thus the adversarial data being harder , it is too demanding to require the teachers become always good at every adversarial data queried by the student model , as the teacher model has never seen these data in its pre-training . In contrast , for the conventional distillation , student models are expected to distill the “ static ” knowledge from the teacher model , since the soft labels for the natural data from the teacher model are always fixed . The observation in Figure 1 ( a ) raises the challenge : how to conduct reliable adversarial distillation with unreliable teachers ? To solve this problem , we can categorize the training data according to the prediction on natural and student generated adversarial data into three cases . First , if the teacher model can correctly classify both natural and adversarial data , it is reliable ; Second , if the teacher model can correctly classify the natural but not adversarial data , it should be partially trusted , and the student model is suggested to trust itself to enhance model robustness as the adversarial regularization ( Zhang et al. , 2019 ) ; Third , if the teacher model can not correctly classify both natural and adversarial data , the student model is recommended to trust itself totally . According to this intuition , we propose an Introspective Adversarial Distillation ( IAD ) to effectively utilize the knowledge from an adversarially pre-trained teacher model . The framework of our proposed IAD can be seen in Figure 1 ( b ) . Briefly , the student model is encouraged to partially instead of fully trust the teacher model , and gradually trust itself more as being more adversarially robust . We conduct extensive experiments on the benchmark CIFAR-10/CIFAR-100 and the more challenging Tiny-ImageNet datasets to evaluate the efficiency of our IAD . The main contributions of our work can be summarized as follows . 1 . We take a closer look at adversarial distillation under the teacher-student paradigm . Considering adversarial robustness , we discover that the guidance from the teacher model is progressively unreliable along with the adversarial training . 2 . We construct the reliable guidance for adversarial distillation by flexibly utilizing the robust knowledge from the teacher model : ( a ) if a teacher is good at adversarial data , its soft labels can be fully trusted ; ( b ) if a teacher is good at natural data but not adversarial data , its soft labels should be partially trusted and the student also takes its own soft labels into account ; ( c ) otherwise , the student only relies on its own soft labels . 3 . We propose an Introspective Adversarial Distillation ( IAD ) to automatically realize the intuition of the previous reliable guidance during the adversarial distillation . The experimental results confirmed that our approach can improve adversarial robustness across a variety of training settings and evaluations , and also on the challenging ( consider adversarial robustness ) datasets ( e.g. , CIFAR-100 ( Krizhevsky , 2009 ) and Tiny-ImageNet ( Le & Yang , 2015 ) ) or using large models ( e.g. , WideResNet ( Zagoruyko & Komodakis , 2016 ) ) . 2 RELATED WORK . 2.1 ADVERSARIAL TRAINING .. Adversarial examples ( Goodfellow et al. , 2015 ) motivate many defensive approaches developed in the last few years . Among them , adversarial training has been demonstrated as the most effective method to improve the robustness of DNNs ( Cai et al. , 2018 ; Wang et al. , 2020a ; b ; Jiang et al. , 2020 ; Bai et al. , 2021 ; Chen et al. , 2021 ; Wu et al. , 2020 ) . The formulation of the popular AT ( Madry et al. , 2018 ) and its variants can be summarized as the minimization of the following loss : min fθ∈F 1 n n∑ i=1 ` ( fθ ( x̃i ) , yi ) , ( 1 ) where n is the number of training examples , x̃i is the adversarial example within the -ball ( bounded by an Lp-norm ) centered at natural example xi , yi is the associated label , fθ is the DNN with parameter θ and ` ( · ) is the standard classification loss , e.g. , the cross-entropy loss . Adversarial training leverages adversarial examples to smooth the small neighborhood , making the model prediction locally invariant . To generate the adversarial examples , AT employs a PGD method ( Madry et al. , 2018 ) . Concretely , given a sample x ( 0 ) ∈ X and the step size β > 0 , PGD recursively searches x̃ ( t+1 ) = ΠB [ x̃ ( 0 ) ] ( x̃ ( t ) + β sign ( ∇x̃ ( t ) ` ( fθ ( x̃ ( t ) ) , y ) ) ) , ( 2 ) until a certain stopping criterion is satisfied . In Eq . equation 2 , t ∈ N , ` is the loss function , x̃ ( t ) is adversarial data at step t , y is the corresponding label for natural data , and ΠB [ x̃0 ] ( · ) is the projection function that projects the adversarial data back into the -ball centered at x̃ ( 0 ) . 2.2 KNOWLEDGE DISTILLATION . The idea of distillation from other models can be dated back to ( Craven & Shavlik , 1996 ) , and reintroduced by ( Hinton et al. , 2015 ) as knowledge distillation ( KD ) . It has been widely studied in recent years and works well in numerous applications like model compression and transfer learning . For adversarial defense , a few studies have explored obtaining adversarial robust models by distillation . Papernot et al . ( 2016 ) proposed defensive distillation which utilizes the soft labels produced by a standard pre-trained teacher model , while this method is proved to be not resistant to the C & W attacks ( Carlini & Wagner , 2017 ) ; Goldblum et al . ( 2020 ) combined AT with KD to transfer robustness to student models , and they found that the distilled models can outperform adversarially pre-trained teacher models of identical architecture in terms of adversarial robustness ; Chen et al . ( 2021 ) utilized distillation as a regularization for adversarial training , which employed both robust and standard pre-trained teacher models to address the robust overfitting ( Rice et al. , 2020 ) . Nonetheless , all these related methods fully trust teacher models and do not consider that whether the guidance of the teacher model in distillation is reliable or not . In this paper , different from the previous studies , we find that the teacher model in adversarial distillation is not always trustworthy . Based on that , we propose reliable IAD to encourage student models to partially instead of fully trust teacher models , which effectively utilizes the knowledge from the adversarially pre-trained models . 3 A CLOSER LOOK AT ADVERSARIAL DISTILLATION . In Section 3.1 , we discuss the unreliable issue of adversarial distillation , i.e. , the guidance of the teacher model is progressively unreliable along with adversarial training . In Section 3.2 , we partition the training examples into three parts and analyze them part by part . Specifically , we expect that the student model should partially instead of fully trust the teacher model and gradually trust itself more along with adversarial training . 3.1 FULLY TRUST : PROGRESSIVELY UNRELIABLE GUIDANCE . As aforementioned in the Introduction , previous methods ( Goldblum et al. , 2020 ; Chen et al. , 2021 ) fully trust the teacher model when distilling robustness from adversarially pre-trained models . Taking Adversarial Robust Distillation ( ARD ) ( Goldblum et al. , 2020 ) as an example , we illustrate its procedure in the left part of Figure 1 ( b ) : the student model generates its adversarial data and then optimizes the prediction of them to mimic the output of the teacher model . However , although the teacher model is well optimized on the adversarial data queried by itself , we argue that it might not always be good at the more and more challenging adversarial data queried by the student model . As shown in Figure 1 ( a ) , different from the ordinary distillation in which the teacher model has the consistent standard performance on the natural data , its robust accuracy on the student model ’ s adversarial data is decreasing during distillation . The guidance of the teacher model gradually fails to give the correct output on the adversarial data queried by the student model .

Summary: In this paper, a new method called introspective adversarial distillation (IAD) is proposed for conventional adversarial training. Concretely, it targets the unreliable teacher case, where teacher is good at adversarial / natural data or none of it. Experimental results validate the effectiveness of the proposed method towards enhancing adversarial robustness.