Title: SELF-KNOWLEDGE DISTILLATION ADVERSARIAL AT-TACK

Abstract: Neural networks show great vulnerability under the threat of adversarial examples. By adding small perturbation to a clean image, neural networks with high classification accuracy can be completely fooled. One intriguing property of the adversarial examples is transferability. This property allows adversarial examples to transfer to networks of unknown structure, which is harmful even to the physical world. The current way of generating adversarial examples is mainly divided into optimization based and gradient based methods. Liu et al. (2017) conjecture that gradient based methods can hardly produce transferable targeted adversarial examples in black-box attack. However, in this paper, we use a simple technique to improve the transferability and success rate of targeted attacks with gradient based methods. We prove that gradient based methods can also generate transferable adversarial examples in targeted attacks. Specifically, we use knowledge distillation for gradient based methods, and show that the transferability can be improved by effectively utilizing different classes of information. Unlike the usual applications of knowledge distillation, we did not train a student network to generate adversarial examples. We take advantage of the fact that knowledge distillation can soften the target and obtain higher information, and combine the soft target and hard target of the same network as the loss function. Our method is generally applicable to most gradient based attack methods.

1. INTRODUCTION:

Neural networks have been shown to be susceptible to adversarial examples. They can be misled when we add small perturbation to a clean image, even though the perturbation may be invisible to the human eye. Adversarial examples are widely used in different physical attack scenarios, including face recognition, voice recognition, and autonomous driving. In the physical world, however, it is difficult to obtain the structure of neural networks. This creates a need for attack algorithms in black-box scenarios. As attack algorithms are created for defense purposes, early discovery of adversarial examples that harm the physical world can help us defend against unknown threats. Many researchers have observed that adversarial examples can be transferred between different networks. Papernot et al. (2016) create adversarial examples which can be transferred to black-box scenarios by attacking a constructed substitute model. Existing attack algorithms have high attack success rate in white-box scenarios and good transferability in non-targeted attacks. However, there is a very low transferability in targeted black-box scenes which is more harmful. Due to the tricky degree of targeted attack transferability problem, almost all existing researches to improve the transferability focus on the untargeted attack part. In fact, it is easy to understand that an untargeted attack only needs to move the original image away from its category, while a targeted attack needs to reach the target category while penetrating the classification boundary. Liu et al. (2017) consider gradient based method only searching attacks in a 1-D subspace. In this approach, the subspace contains just a small subset of all target labels. They improve transferability by using ensemble neural networks in the optimization based method. They also prove that the approach of ensemble models is not effective for the gradient based method. For their experimental results and hypotheses, there is no difference between attacking ensemble neural networks and attacking a neural network for the transferability of gradient based method. We will show this is not correct by our study.

There are two problems in existing gradient based methods for targeted attacks. Firstly, the attack success rate of gradient based method cannot be improved after a certain number of iterations. Secondly, targeted attacks do not make full use of the information between different categories.

Existing adversarial attack algorithms usually ensemble neural networks to produce adversarial examples. Dong et al. (2018b) use an approach namely "fuse in logits" to ensemble models. "Fuse in logit" performs a simple weighted average of logits before the softmax layer of the neural network, and then calculates the cross entropy loss between logits and the one-hot vector of the target labels. The ensemble attack algorithm produces adversarial examples in each ensembled model, which can achieve a good attack success rate. But this is not black-box transferability, since every model structure which is ensembled can be acquired by an attack algorithm. Black-box transferability refers to the fact that adversarial examples can still attack successfully against the networks with unknown structures. In this paper, the term transferability will refer uniformly to the black-box and targeted attack scenarios. Hinton et al. (2015) put forward the concept of knowledge distillation. Knowledge distillation is used to extract useful dark knowledge from bloated models to train a small and lightweight network. Knowledge distillation puts forward the concept of soft targets and hard targets. Through the cumbersome model of probability distribution divided by a value greater than 1, the original probability distribution can be transformed to soft targets. When the soft targets have high entropy, they provide much more information.

Instead of training a small model from multiple large models, we simply take advantage of the fact that knowledge distillation can effectively extract knowledge. Actually, we are not attacking a lightweight model extracted from a bloated model. It is simpler and more efficient to produce transferable adversarial examples in targeted attack, which we call self-distillation adversarial attack.

2. BACKGROUND 2.1 ADVERSARIAL EXAMPLES:

Deep neural networks are proved vulnerable to adversarial attacks since Szegedy et al. (2014) , which shows that neural networks can be misclassified by adding small perturbations to a clean picture. The existing of adversarial examples has proven to be a huge threat to the entire application of deep learning algorithms, including speech recognition, face recognition, and even the recognition of road signs in the physical world. As a result, more and more attention has been paid to adversarial examples, and many defensive researches have been made to improve the robustness of neural networks. At present, the research on the attack and defense of neural network is also in a state of game and spiral. The research on adversarial examples is also a good starting points for the explainability of neural network.

2.2. GRADIENT-BASED ADVERSARIAL ATTACK METHODS:

The current methods of generating adversarial examples are mainly gradient based and optimization based. Here, we briefly introduce the gradient based method. Goodfellow et al. (2015) come up with a way to get adversarial examples by applying the sign of the gradient to a real example only once. They call this method the Fast Gradient Sign Method (FGSM). Kurakin et al. (2017) move the adversarial example in the direction of the sign of the gradient in each iteration. Dong et al. (2018b) integrate momentum into iterative FGSM (I-FGSM), and this can help attack algorithm stabilizing update directions and escaping from poor local maxima. This method which is called MI-FGSM solves the problem that I-FGSM is easy to obtain local extreme value and "overfit" model. Madry et al. (2018) propose a saddle point formulation, which can convert the attack problems and defense problems to inner maximization problems and outer minimization problems, respectively. They suppose projected gradient descent (PGD) could be the strongest universal attack method.

2.3. THE WEAKNESS OF GRADIENT BASED METHOD:

The first problem with gradient based methods is that they lose their effectiveness after a certain number of iterations. In neural networks, softmax is usually used as the activation function to get probability output for multi-classification problems. In gradient based adversarial attacks, the probability output of image in a neural network is usually concerned. Therefore, most adversarial attack methods also use softmax activation function to obtain the probability and calculate the cross entropy with the target category, and finally optimize the cross entropy loss function to get the adversarial example.

However, softmax has saturation problems. Softmax's probability output sum is 1, so there is competition among each output. In extreme cases, it becomes winner-take-all, and when one output "approaches 1", the others "approach 0". Another downside of softmax is that it can cause either overspill or underspill problems. Although this downside can be effectively alleviated by subtracting the maximum value of all inputs from the input value of softmax, it causes the output value of softmax to be driven by the maximum input value. When the input value is the maximum, the output value will be close to 1. On the contrary, the large difference can cause softmax's output to be saturated to 0. Liu et al. (2017) prove that the classification boundaries between white-box models and black-box models do not coincide. And for the model which is not used to generate the adversarial image, its classification area is much smaller. This means that when a successful attack on neural network whose structure is known, adversarial examples cannot penetrates the classification boundaries of other neural networks unknown structure precisely. It is more important to obtain information of different categories. The gradient update stop caused by softmax saturation makes gradient based adversarial examples unable to accurately penetrate classification boundaries. So, insufficient information acquisition for different categories and premature stop of gradient update are the reasons why gradient based methods have low transferability in targeted black-box scenes. The solution to this problem is very simple. We introduce knowledge distillation into gradient based methods.

2.4. DISTILL IN NEURAL NETWORKS:

In machine learning tasks, researchers always train many different kinds of models on the same dataset, and combine the results of each single model into a final result (Dietterich (2000) ). A simple combined method is to average the predictions, but the cost of this ensemble method is too expensive to deploy. Hinton et al. (2015) find that instead of simply averaging the predictions, it will get much better performance if converting the logits by:

EQUATION

where T is called a temperature. Using a higher T can produce a softer probability distribution over classes. And it will be normal softmax-layer when T is set by 1.

Distillation in neural networks can make the combination of multiple networks perform better, and the knowledge of the cumbersome model can tansfer to a small model in an effective way. The current research on distillation is mainly used to train a smaller model from a bloated model set. However, distillation can effectively help to get the target information in adversarial attacks.

3. METHODOLOGY:



3.1. THE SOLVE OF SATURATION PROBLEM:

Here, we prove that knowledge distillation can effectively reduce the distance between classes and solve the saturation problem. If we use the ratio ρ to measure the distance between the softmaxlayer's outputs, we can get the distance of two hard targets below.

ρ = q(x a ) q(x b ) = e xa i e i e x b i e i = e xa e x b

(2)

If we distill the logits with a temperature T like formula-1, the distance will be as follows.

ρ = q(x a /T ) q(x b /T ) = e xa /T i e i /T e x b /T i e i /T = e xa/T e x b /T (3)

So the relationship between ρ and ρ is:

ρ = T √ ρ (4)

The distillation processing is essentially the dispersion of the softmax results among different classifications using the T -th power of the square root, so that the different classification softmax values distribution are closer.

3.2. GET MORE ACCURATE DIRECTION:

It may be a good thing for untargeted attacks when data classes are close to each other, because it makes easy for adversarial examples to randomly move from the original category to a target category and the attack succeeds. But for targeted attacks, this is a problem, making it more likely to attack the wrong class. Liu et al. (2017) observe that the decision boundaries of all models are very consistent with each other. They think this explains why non-targeted adversarial images can transfer among models. But for targeted attack, decision boundaries of models do not generate adversarial examples in a small area, thus tiny change of gradient will make adversarial examples change to wrong target categories. We assume that a gradient based method can obtain a more accurate gradient direction by knowledge distillation.

The distance between the gradient of a target class and the gradient of a wrong class can be give by:

∆ = ∂C target ∂z i - ∂C wrong ∂z i = (p T rue -q target ) -(p T rue -q wrong ) = q wrong -q target (5)

where C target means the cross entropy about target class, C wrong means the cross entropy about wrong probability. ∆ means the distance of two gradient. We define δ = ∆/p true , so it is easy to get: Any gradient based adversarial attack algorithm can be easily extended to include distillation. For example, FGSM can be simply extended to Distillation FGSM (D-FGSM) as follows.

φ = δ δ = ∆ /p true ∆/p true = (q wrong -q target )/p true (q wrong -q target )/

X adv = x real + • sign( x J( L(x) T , y))

Here, x J is the gradient of loss function, L(x) means the logits of image x produced by neural network, and sign(•) is sign function limits the size of the disturbance.

Momentum iteration fast gradient sign method can be simply extended to Distillation MI-FGSM (D-MIFGSM). Liu et al. (2017) ensemble neural networks to improve transferability in optimize based method. They consider there is no difference between single model attack and ensemble based attack in fast gradient based method, and both these two methods can not produce transferable adversarial examples. Dong et al. (2018b) 's ensemble method is a simple weighted average of logits generated by multiple neural networks, which they call "fuse in logits".

g t+1 = µ • g t + ∇ x J( L(x) T , y) ∇ x J( L(x) T , y) 1 x adv t+1 = x adv t + α • sign(g t+1 ) 4.2 ENSEMBLE BASED METHOD

We have a better ensemble approach. By distilling logits generated by n neural networks, n softened logits can be obtained. In the gradient based attack method, n logits can obtain n cross entropy losses, and the cross entropy losses generated by the softened logits and the original logits, namely the hardened logits, can produce a total of 2n cross entropy losses. A simple weighted average of these cross entropy losses can extend the gradient based approach to the ensemble attack method that includes knowledge distillation:

EQUATION

where L(x) is the logit produced by different neural networks, and J(x, y) is the cross entropy between the logit x and the attack target y.

We have carried out experiments on different adversarial attack algorithms and found that the momentum iterative fast gradient sign method based on knowledge distillation (D-MIFGSM) is the most effective method to produce transferable adversarial examples.

We summarize our ensemble based method in Algorithm 1, and the rest of our experiments are based on this algorithm. Specifically, when we input a clean image x and get K logits from K classifiers, we distill these N logits using a pre-defined temperature T . Then, we perform Equation 8 "fuse in cross entropy" on these distilled logits. The rest of the algorithm is the same as MI-FGSM.

5. EXPERIMENT:



5.1. SETUP:

To see how distillation works on targeted attack and what is the best distillation environment, we design a series of experiments. We choose four networks, Inception V 3, Inception V 4 (Szegedy 

5.3. ENSEMBLE BASED METHOD:

We find that attacking V GG networks can generate adversarial examples with high transferability in a single model attack. Moreover, in the optimization-based approach, ensemble networks have been proved to be able to produce transferable adversarial examples.

We test the impact of ensemble on transferability of different networks. In our test, we take off one of the four known networks each time, and the results are shown on Table 2 .

Since it is possible to generate a large number of adversarial examples with high transferability in the distillation based method through ensemble models, we perform different parameter tests on this method.

5.4. THE EFFECT OF TEMPERATURE:

Firstly, we want to know how different distillation temperatures affect the attack success rate, and which is the best temperature to distill. We test different temperatures while the iteration number is 20 and the max disturb limit is 32. The success rate of D-MIFGSM in ensemble based method. Cell (i, j) indicates that the accuracy of the targeted adversarial examples generated for the ensemble of the four models except model i (row) is predicted as the target label by model j (column). In each line ,means the network behind is not used in ensemble attack.

Figure 1 shows the effect of different distillation temperatures on black-box targeted attack, where the temperature equals 1 means there's no distillation processing. It is clear that the distillation can make effects on both single model attack and ensemble based method. And the distillation temperature makes best effects when it reaches about 16. But in our method, as Figure 2 shows, the more iterations, the much transferability we can get, until it is large enough to converge. In another word, to certain extent, we avoid the overfit problem. However, high iterations also mean long computing time. In our experiment, we suggest 30 to 50 iterations to be a better choice.

:



Figures and Tables:

Figure 1: Effect of Different Temperature

Figure 2: Effect of Different Iterations

It means distillation can effectively increase the distance between the gradient of the target class and the gradient of a wrong class. In this way, the gradient based attack method can attack to a targeted class more precisely.

Table content: <table><tr><td colspan="6">4 DISTILLATION ADVERSARIAL ATTACK</td><td/></tr><tr><td colspan="6">4.1 SINGLE MODEL GRADIENT-BASED ATTACK</td><td/></tr><tr><td/><td/><td/><td/><td/><td>p true</td><td/></tr><tr><td>=</td><td colspan="2">e wrong/T -e target/T i e i/T e wrong -e target i e i / e true / e true/T i e i/T i e i</td><td colspan="2">=</td><td colspan="3">e wrong/T -e target/T e true/T</td><td>/</td><td>e true e wrong -e target</td><td>(6)</td></tr><tr><td/><td>φ ≈</td><td colspan="2">wrong/T -target/T e true/T</td><td>/</td><td>wrong -target e true</td><td>≈</td><td>e true e true/T &gt;&gt; 1</td><td>(7)</td></tr></table>

Algorithm 1 Ensemble D-MIFGSMInput: The logits of K classifiers l1, l2, ..., l K ;Temprature T ; a real example x and target label y * ; Input: The size of perturbation ε; iterations N and decay factor µ. Output: An adversarial example x * with x * -x inf ≤ ε ResN et V 1 50, ResN et V 1 152(He et al. (2016)) and Inception Resnet(Szegedy et al. (2017)) as the networks in black-box attacks to test the transferability of the adversarial examples.To verify the true effectiveness, we tend to choose a bigger dataset. So, we randomly choose 10,000 images from ILSVRC 2012 validation set, with 10 images in each class, and make sure they can be all classified correctly by the networks we use. We define the disturb of adversarial examples by L inf -distance.5.2 SINGLE MODEL ATTACKForLiu et al. (2017), attacking one model is the same as attacking multiple models in fast gradient method, neither of which can produce adversarial examples with high transferability. We test the performance of different networks in a single model attack. Table1shows that the transferability in single model attack can reach about 30% by attacking V GG networks. And we find that only attacking V GG networks can produce transferable adversarial examples. The success rate of D-MIFGSM in single model attack. The cell (i, j) indicates the accuracy of the adversarial examples generated for model i (row) evaluated over model j (column).

Table content: <table><tr><td>1: α = ε/N ;</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>2: for i = 0 to N -1 do</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="7">3: Input x  *  i and output l k (x  *  i ) for k = 1, 2, ..., K; l(x  *  i ) 4: Get different logits as l(x  *  i ) = T l(x  *  i ) 5: Fuse the softmax cross-entropy loss J(x  *  i , y  *  ) based on Equation 8. 6: Obtain the gradient ∇xJ(x  *  i , y  *  ); 7: Update gi+1 by accumulating the velocity vector in the gradient direction as gi+1 = µ • gi + 8: Update x  *  i+1 by applying the sign gradient as x  *  i+1 = x  *  i -α • sign(gi+1);</td><td colspan="2">∇xJ(x  *  i , y  *  ) ∇xJ(x  *  i , y  *  ) 1</td><td>;</td></tr><tr><td>9: end for</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>10: return x  *  = x  *  N .</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="9">et al. (2016)), V GG16 and V GG19 (Simonyan &amp; Zisserman (2015)), to produce the adversarial</td></tr><tr><td colspan="2">examples. We use IncV 3</td><td>IncV 4</td><td>V GG16</td><td>V GG19</td><td>Resnet 50</td><td colspan="2">Resnet 152</td><td>Inc Res</td></tr><tr><td>IncV 3</td><td>0.999</td><td>0.018</td><td>0.009</td><td>0.007</td><td>0.006</td><td/><td>0.005</td><td>0.020</td></tr><tr><td>IncV 4</td><td>0.059</td><td>0.991</td><td>0.125</td><td>0.010</td><td>0.005</td><td/><td>0.005</td><td>0.027</td></tr><tr><td>V GG16</td><td>0.241</td><td>0.205</td><td>1.0</td><td>0.929</td><td>0.340</td><td/><td>0.235</td><td>0.201</td></tr><tr><td>V GG19</td><td>0.233</td><td>0.197</td><td>0.918</td><td>1.0</td><td>0.333</td><td/><td>0.241</td><td>0.191</td></tr></table>



Table content: <table><tr><td/><td>IncV 3</td><td>IncV 4</td><td>V GG16</td><td>V GG19</td><td>Resnet 50</td><td>Resnet 152</td><td>Inc Res</td></tr><tr><td>All</td><td>0.949</td><td>0.982</td><td>1.0</td><td>1.0</td><td>0.471</td><td>0.385</td><td>0.334</td></tr><tr><td>-IncV 3</td><td>0.499</td><td>0.991</td><td>1.0</td><td>1.0</td><td>0.575</td><td>0.462</td><td>0.386</td></tr><tr><td>-IncV 4</td><td>0.937</td><td>0.294</td><td>1.0</td><td>1.0</td><td>0.437</td><td>0.355</td><td>0.219</td></tr><tr><td>-V GG16</td><td>0.939</td><td>0.980</td><td>0.803</td><td>1.0</td><td>0.232</td><td>0.176</td><td>0.192</td></tr><tr><td>-V GG19</td><td>0.941</td><td>0.979</td><td>1.0</td><td>0.787</td><td>0.228</td><td>0.173</td><td>0.188</td></tr></table>

Peer review (iclr2020):
This paper proposes distillation attacks to generate transferable targeted adversarial examples. The technique itself is pretty simple: instead of only using the raw logits L(x) to compute the cross entropy loss for optimization, they also use the distilled logits L(x)/T to generate adversarial examples. Their evaluation setup largely follows the style of Liu et al., but they construct a different subset of ILSVRC validation set, and some of the model architectures in their ensemble are different from Liu et al. Their results show that by including the distilled logits when computing the gradient, the generated adversarial examples can transfer better among different models using both single-model and ensemble-based attacks. I think their proposed attack is interesting due to its simplicity and effectiveness. However, I would like to see clarification of some evaluation details, as well as more experiments to compare with Liu et al.: 1. To assess the effectiveness of targeted attacks, it is important to ensure that the semantic meaning of target label is far from the ground truth label. Some of the 1000 ImageNet labels have very similar meanings to each other, thus different choices of the target label would dramatically affect the difficulty of the attacks. In Liu et al., they manually inspect the image-target pairs to ensure that the target label is very different from the ground truth in its meaning. To enable a fair comparison, it would be helpful to provide results on the same image-target pairs constructed by Liu et al., which could be found in the public repo linked in their paper. 2. For ensemble attacks, is including both the raw and the distilled logits crucial in obtaining a good performance? What is the performance of including distilled logits only? How do different values of lambda_1 and lambda_2 in (8) affect the attack performance? 3. Could you visualize some generated adversarial examples, so that we can view the qualitative results? 4. In general this paper lacks empirical analysis on why distillation helps improve the transferability. Some more discussion would be helpful. ------------- Post-rebuttal comments Thanks for your response! I think this paper still misses a more in-depth analysis, and thus I keep my original assessment. -------------