SELF-KNOWLEDGE DISTILLATION ADVERSARIAL ATTACK

1 INTRODUCTION . Neural networks have been shown to be susceptible to adversarial examples . They can be misled when we add small perturbation to a clean image , even though the perturbation may be invisible to the human eye . Adversarial examples are widely used in different physical attack scenarios , including face recognition , voice recognition , and autonomous driving . In the physical world , however , it is difficult to obtain the structure of neural networks . This creates a need for attack algorithms in black-box scenarios . As attack algorithms are created for defense purposes , early discovery of adversarial examples that harm the physical world can help us defend against unknown threats . Many researchers have observed that adversarial examples can be transferred between different networks . Papernot et al . ( 2016 ) create adversarial examples which can be transferred to black-box scenarios by attacking a constructed substitute model . Existing attack algorithms have high attack success rate in white-box scenarios and good transferability in non-targeted attacks . However , there is a very low transferability in targeted black-box scenes which is more harmful . Due to the tricky degree of targeted attack transferability problem , almost all existing researches to improve the transferability focus on the untargeted attack part . In fact , it is easy to understand that an untargeted attack only needs to move the original image away from its category , while a targeted attack needs to reach the target category while penetrating the classification boundary . Liu et al . ( 2017 ) consider gradient based method only searching attacks in a 1-D subspace . In this approach , the subspace contains just a small subset of all target labels . They improve transferability by using ensemble neural networks in the optimization based method . They also prove that the approach of ensemble models is not effective for the gradient based method . For their experimental results and hypotheses , there is no difference between attacking ensemble neural networks and attacking a neural network for the transferability of gradient based method . We will show this is not correct by our study . There are two problems in existing gradient based methods for targeted attacks . Firstly , the attack success rate of gradient based method can not be improved after a certain number of iterations . Secondly , targeted attacks do not make full use of the information between different categories . Existing adversarial attack algorithms usually ensemble neural networks to produce adversarial examples . Dong et al . ( 2018b ) use an approach namely “ fuse in logits ” to ensemble models . “ Fuse in logit ” performs a simple weighted average of logits before the softmax layer of the neural network , and then calculates the cross entropy loss between logits and the one-hot vector of the target labels . The ensemble attack algorithm produces adversarial examples in each ensembled model , which can achieve a good attack success rate . But this is not black-box transferability , since every model structure which is ensembled can be acquired by an attack algorithm . Black-box transferability refers to the fact that adversarial examples can still attack successfully against the networks with unknown structures . In this paper , the term transferability will refer uniformly to the black-box and targeted attack scenarios . Hinton et al . ( 2015 ) put forward the concept of knowledge distillation . Knowledge distillation is used to extract useful dark knowledge from bloated models to train a small and lightweight network . Knowledge distillation puts forward the concept of soft targets and hard targets . Through the cumbersome model of probability distribution divided by a value greater than 1 , the original probability distribution can be transformed to soft targets . When the soft targets have high entropy , they provide much more information . Instead of training a small model from multiple large models , we simply take advantage of the fact that knowledge distillation can effectively extract knowledge . Actually , we are not attacking a lightweight model extracted from a bloated model . It is simpler and more efficient to produce transferable adversarial examples in targeted attack , which we call self-distillation adversarial attack . 2 BACKGROUND . 2.1 ADVERSARIAL EXAMPLES . Deep neural networks are proved vulnerable to adversarial attacks since Szegedy et al . ( 2014 ) , which shows that neural networks can be misclassified by adding small perturbations to a clean picture . The existing of adversarial examples has proven to be a huge threat to the entire application of deep learning algorithms , including speech recognition , face recognition , and even the recognition of road signs in the physical world . As a result , more and more attention has been paid to adversarial examples , and many defensive researches have been made to improve the robustness of neural networks . At present , the research on the attack and defense of neural network is also in a state of game and spiral . The research on adversarial examples is also a good starting points for the explainability of neural network . 2.2 GRADIENT-BASED ADVERSARIAL ATTACK METHODS . The current methods of generating adversarial examples are mainly gradient based and optimization based . Here , we briefly introduce the gradient based method . Goodfellow et al . ( 2015 ) come up with a way to get adversarial examples by applying the sign of the gradient to a real example only once . They call this method the Fast Gradient Sign Method ( FGSM ) . Kurakin et al . ( 2017 ) move the adversarial example in the direction of the sign of the gradient in each iteration . Dong et al . ( 2018b ) integrate momentum into iterative FGSM ( I-FGSM ) , and this can help attack algorithm stabilizing update directions and escaping from poor local maxima . This method which is called MI-FGSM solves the problem that I-FGSM is easy to obtain local extreme value and “ overfit ” model . Madry et al . ( 2018 ) propose a saddle point formulation , which can convert the attack problems and defense problems to inner maximization problems and outer minimization problems , respectively . They suppose projected gradient descent ( PGD ) could be the strongest universal attack method . 2.3 THE WEAKNESS OF GRADIENT BASED METHOD . The first problem with gradient based methods is that they lose their effectiveness after a certain number of iterations . In neural networks , softmax is usually used as the activation function to get probability output for multi-classification problems . In gradient based adversarial attacks , the probability output of image in a neural network is usually concerned . Therefore , most adversarial attack methods also use softmax activation function to obtain the probability and calculate the cross entropy with the target category , and finally optimize the cross entropy loss function to get the adversarial example . However , softmax has saturation problems . Softmax ’ s probability output sum is 1 , so there is competition among each output . In extreme cases , it becomes winner-take-all , and when one output “ approaches 1 ” , the others “ approach 0 ” . Another downside of softmax is that it can cause either overspill or underspill problems . Although this downside can be effectively alleviated by subtracting the maximum value of all inputs from the input value of softmax , it causes the output value of softmax to be driven by the maximum input value . When the input value is the maximum , the output value will be close to 1 . On the contrary , the large difference can cause softmax ’ s output to be saturated to 0 . Liu et al . ( 2017 ) prove that the classification boundaries between white-box models and black-box models do not coincide . And for the model which is not used to generate the adversarial image , its classification area is much smaller . This means that when a successful attack on neural network whose structure is known , adversarial examples can not penetrates the classification boundaries of other neural networks unknown structure precisely . It is more important to obtain information of different categories . The gradient update stop caused by softmax saturation makes gradient based adversarial examples unable to accurately penetrate classification boundaries . So , insufficient information acquisition for different categories and premature stop of gradient update are the reasons why gradient based methods have low transferability in targeted black-box scenes . The solution to this problem is very simple . We introduce knowledge distillation into gradient based methods . 2.4 DISTILL IN NEURAL NETWORKS . In machine learning tasks , researchers always train many different kinds of models on the same dataset , and combine the results of each single model into a final result ( Dietterich ( 2000 ) ) . A simple combined method is to average the predictions , but the cost of this ensemble method is too expensive to deploy . Hinton et al . ( 2015 ) find that instead of simply averaging the predictions , it will get much better performance if converting the logits by : qi = exp ( zi/T ) ∑ j exp ( zj/T ) ( 1 ) where T is called a temperature . Using a higher T can produce a softer probability distribution over classes . And it will be normal softmax-layer when T is set by 1 . Distillation in neural networks can make the combination of multiple networks perform better , and the knowledge of the cumbersome model can tansfer to a small model in an effective way . The current research on distillation is mainly used to train a smaller model from a bloated model set . However , distillation can effectively help to get the target information in adversarial attacks . 3 METHODOLOGY . 3.1 THE SOLVE OF SATURATION PROBLEM . Here , we prove that knowledge distillation can effectively reduce the distance between classes and solve the saturation problem . If we use the ratio ρ to measure the distance between the softmaxlayer ’ s outputs , we can get the distance of two hard targets below . ρ = q ( xa ) q ( xb ) = exa∑ i e i exb∑ i e i = exa exb ( 2 ) If we distill the logits with a temperature T like formula-1 , the distance will be as follows . ρ′ = q ( xa/T ) q ( xb/T ) = exa/T∑ i e i/T exb/T∑ i e i/T = exa/T exb/T ( 3 ) So the relationship between ρ′ and ρ is : ρ′ = T √ ρ ( 4 ) The distillation processing is essentially the dispersion of the softmax results among different classifications using the T -th power of the square root , so that the different classification softmax values distribution are closer . 3.2 GET MORE ACCURATE DIRECTION . It may be a good thing for untargeted attacks when data classes are close to each other , because it makes easy for adversarial examples to randomly move from the original category to a target category and the attack succeeds . But for targeted attacks , this is a problem , making it more likely to attack the wrong class . Liu et al . ( 2017 ) observe that the decision boundaries of all models are very consistent with each other . They think this explains why non-targeted adversarial images can transfer among models . But for targeted attack , decision boundaries of models do not generate adversarial examples in a small area , thus tiny change of gradient will make adversarial examples change to wrong target categories . We assume that a gradient based method can obtain a more accurate gradient direction by knowledge distillation . The distance between the gradient of a target class and the gradient of a wrong class can be give by : ∆ = ∂Ctarget ∂zi − ∂Cwrong ∂zi = ( pTrue − qtarget ) − ( pTrue − qwrong ) = qwrong − qtarget ( 5 ) where Ctarget means the cross entropy about target class , Cwrong means the cross entropy about wrong probability . ∆ means the distance of two gradient . We define δ = ∆/ptrue , so it is easy to get : φ = δ′ δ = ∆′/p′true ∆/ptrue = ( q′wrong − q′target ) /p′true ( qwrong − qtarget ) /ptrue = ewrong/T−etarget/T∑ i e i/T / etrue/T∑ i e i/T ewrong−etarget∑ i e i / etrue∑ i e i = ewrong/T − etarget/T etrue/T / ewrong − etarget etrue ( 6 ) Compared to the true class , the values of the wrong class and the target class tend to zero . We can approximate that : φ ≈ wrong/T − target/T etrue/T / wrong − target etrue ≈ e true etrue/T > > 1 ( 7 ) It means distillation can effectively increase the distance between the gradient of the target class and the gradient of a wrong class . In this way , the gradient based attack method can attack to a targeted class more precisely .

Summary: This paper proposes distillation attacks to generate transferable targeted adversarial examples. The technique itself is pretty simple: instead of only using the raw logits L(x) to compute the cross entropy loss for optimization, they also use the distilled logits L(x)/T to generate adversarial examples. Their evaluation setup largely follows the style of Liu et al., but they construct a different subset of ILSVRC validation set, and some of the model architectures in their ensemble are different from Liu et al. Their results show that by including the distilled logits when computing the gradient, the generated adversarial examples can transfer better among different models using both single-model and ensemble-based attacks.