Title: SELF-KNOWLEDGE DISTILLATION ADVERSARIAL AT-TACK

Abstract: Neural networks show great vulnerability under the threat of adversarial examples. By adding small perturbation to a clean image, neural networks with high classification accuracy can be completely fooled. One intriguing property of the adversarial examples is transferability. This property allows adversarial examples to transfer to networks of unknown structure, which is harmful even to the physical world. The current way of generating adversarial examples is mainly divided into optimization based and gradient based methods. Liu et al. (2017) conjecture that gradient based methods can hardly produce transferable targeted adversarial examples in black-box attack. However, in this paper, we use a simple technique to improve the transferability and success rate of targeted attacks with gradient based methods. We prove that gradient based methods can also generate transferable adversarial examples in targeted attacks. Specifically, we use knowledge distillation for gradient based methods, and show that the transferability can be improved by effectively utilizing different classes of information. Unlike the usual applications of knowledge distillation, we did not train a student network to generate adversarial examples. We take advantage of the fact that knowledge distillation can soften the target and obtain higher information, and combine the soft target and hard target of the same network as the loss function. Our method is generally applicable to most gradient based attack methods.

1. INTRODUCTION:

Neural networks have been shown to be susceptible to adversarial examples. They can be misled when we add small perturbation to a clean image, even though the perturbation may be invisible to the human eye. Adversarial examples are widely used in different physical attack scenarios, including face recognition, voice recognition, and autonomous driving. In the physical world, however, it is difficult to obtain the structure of neural networks. This creates a need for attack algorithms in black-box scenarios. As attack algorithms are created for defense purposes, early discovery of adversarial examples that harm the physical world can help us defend against unknown threats. Many researchers have observed that adversarial examples can be transferred between different networks. Papernot et al. (2016) create adversarial examples which can be transferred to black-box scenarios by attacking a constructed substitute model. Existing attack algorithms have high attack success rate in white-box scenarios and good transferability in non-targeted attacks. However, there is a very low transferability in targeted black-box scenes which is more harmful. Due to the tricky degree of targeted attack transferability problem, almost all existing researches to improve the transferability focus on the untargeted attack part. In fact, it is easy to understand that an untargeted attack only needs to move the original image away from its category, while a targeted attack needs to reach the target category while penetrating the classification boundary. Liu et al. (2017) consider gradient based method only searching attacks in a 1-D subspace. In this approach, the subspace contains just a small subset of all target labels. They improve transferability by using ensemble neural networks in the optimization based method. They also prove that the approach of ensemble models is not effective for the gradient based method. For their experimental results and hypotheses, there is no difference between attacking ensemble neural networks and attacking a neural network for the transferability of gradient based method. We will show this is not correct by our study.

There are two problems in existing gradient based methods for targeted attacks. Firstly, the attack success rate of gradient based method cannot be improved after a certain number of iterations. Secondly, targeted attacks do not make full use of the information between different categories.

Existing adversarial attack algorithms usually ensemble neural networks to produce adversarial examples. Dong et al. (2018b) use an approach namely "fuse in logits" to ensemble models. "Fuse in logit" performs a simple weighted average of logits before the softmax layer of the neural network, and then calculates the cross entropy loss between logits and the one-hot vector of the target labels. The ensemble attack algorithm produces adversarial examples in each ensembled model, which can achieve a good attack success rate. But this is not black-box transferability, since every model structure which is ensembled can be acquired by an attack algorithm. Black-box transferability refers to the fact that adversarial examples can still attack successfully against the networks with unknown structures. In this paper, the term transferability will refer uniformly to the black-box and targeted attack scenarios. Hinton et al. (2015) put forward the concept of knowledge distillation. Knowledge distillation is used to extract useful dark knowledge from bloated models to train a small and lightweight network. Knowledge distillation puts forward the concept of soft targets and hard targets. Through the cumbersome model of probability distribution divided by a value greater than 1, the original probability distribution can be transformed to soft targets. When the soft targets have high entropy, they provide much more information.

Instead of training a small model from multiple large models, we simply take advantage of the fact that knowledge distillation can effectively extract knowledge. Actually, we are not attacking a lightweight model extracted from a bloated model. It is simpler and more efficient to produce transferable adversarial examples in targeted attack, which we call self-distillation adversarial attack.

2. BACKGROUND 2.1 ADVERSARIAL EXAMPLES:

Deep neural networks are proved vulnerable to adversarial attacks since Szegedy et al. (2014) , which shows that neural networks can be misclassified by adding small perturbations to a clean picture. The existing of adversarial examples has proven to be a huge threat to the entire application of deep learning algorithms, including speech recognition, face recognition, and even the recognition of road signs in the physical world. As a result, more and more attention has been paid to adversarial examples, and many defensive researches have been made to improve the robustness of neural networks. At present, the research on the attack and defense of neural network is also in a state of game and spiral. The research on adversarial examples is also a good starting points for the explainability of neural network.

2.2. GRADIENT-BASED ADVERSARIAL ATTACK METHODS:

The current methods of generating adversarial examples are mainly gradient based and optimization based. Here, we briefly introduce the gradient based method. Goodfellow et al. (2015) come up with a way to get adversarial examples by applying the sign of the gradient to a real example only once. They call this method the Fast Gradient Sign Method (FGSM). Kurakin et al. (2017) move the adversarial example in the direction of the sign of the gradient in each iteration. Dong et al. (2018b) integrate momentum into iterative FGSM (I-FGSM), and this can help attack algorithm stabilizing update directions and escaping from poor local maxima. This method which is called MI-FGSM solves the problem that I-FGSM is easy to obtain local extreme value and "overfit" model. Madry et al. (2018) propose a saddle point formulation, which can convert the attack problems and defense problems to inner maximization problems and outer minimization problems, respectively. They suppose projected gradient descent (PGD) could be the strongest universal attack method.

2.3. THE WEAKNESS OF GRADIENT BASED METHOD:

The first problem with gradient based methods is that they lose their effectiveness after a certain number of iterations. In neural networks, softmax is usually used as the activation function to get probability output for multi-classification problems. In gradient based adversarial attacks, the probability output of image in a neural network is usually concerned. Therefore, most adversarial attack methods also use softmax activation function to obtain the probability and calculate the cross entropy with the target category, and finally optimize the cross entropy loss function to get the adversarial example.

However, softmax has saturation problems. Softmax's probability output sum is 1, so there is competition among each output. In extreme cases, it becomes winner-take-all, and when one output "approaches 1", the others "approach 0". Another downside of softmax is that it can cause either overspill or underspill problems. Although this downside can be effectively alleviated by subtracting the maximum value of all inputs from the input value of softmax, it causes the output value of softmax to be driven by the maximum input value. When the input value is the maximum, the output value will be close to 1. On the contrary, the large difference can cause softmax's output to be saturated to 0. Liu et al. (2017) prove that the classification boundaries between white-box models and black-box models do not coincide. And for the model which is not used to generate the adversarial image, its classification area is much smaller. This means that when a successful attack on neural network whose structure is known, adversarial examples cannot penetrates the classification boundaries of other neural networks unknown structure precisely. It is more important to obtain information of different categories. The gradient update stop caused by softmax saturation makes gradient based adversarial examples unable to accurately penetrate classification boundaries. So, insufficient information acquisition for different categories and premature stop of gradient update are the reasons why gradient based methods have low transferability in targeted black-box scenes. The solution to this problem is very simple. We introduce knowledge distillation into gradient based methods.

2.4. DISTILL IN NEURAL NETWORKS:

In machine learning tasks, researchers always train many different kinds of models on the same dataset, and combine the results of each single model into a final result (Dietterich (2000) ). A simple combined method is to average the predictions, but the cost of this ensemble method is too expensive to deploy. Hinton et al. (2015) find that instead of simply averaging the predictions, it will get much better performance if converting the logits by:

EQUATION

where T is called a temperature. Using a higher T can produce a softer probability distribution over classes. And it will be normal softmax-layer when T is set by 1.

Distillation in neural networks can make the combination of multiple networks perform better, and the knowledge of the cumbersome model can tansfer to a small model in an effective way. The current research on distillation is mainly used to train a smaller model from a bloated model set. However, distillation can effectively help to get the target information in adversarial attacks.

3. METHODOLOGY:



3.1. THE SOLVE OF SATURATION PROBLEM:

Here, we prove that knowledge distillation can effectively reduce the distance between classes and solve the saturation problem. If we use the ratio ρ to measure the distance between the softmaxlayer's outputs, we can get the distance of two hard targets below.

ρ = q(x a ) q(x b ) = e xa i e i e x b i e i = e xa e x b

(2)

If we distill the logits with a temperature T like formula-1, the distance will be as follows.

ρ = q(x a /T ) q(x b /T ) = e xa /T i e i /T e x b /T i e i /T = e xa/T e x b /T (3)

So the relationship between ρ and ρ is:

ρ = T √ ρ (4)

The distillation processing is essentially the dispersion of the softmax results among different classifications using the T -th power of the square root, so that the different classification softmax values distribution are closer.

3.2. GET MORE ACCURATE DIRECTION:

It may be a good thing for untargeted attacks when data classes are close to each other, because it makes easy for adversarial examples to randomly move from the original category to a target category and the attack succeeds. But for targeted attacks, this is a problem, making it more likely to attack the wrong class. Liu et al. (2017) observe that the decision boundaries of all models are very consistent with each other. They think this explains why non-targeted adversarial images can transfer among models. But for targeted attack, decision boundaries of models do not generate adversarial examples in a small area, thus tiny change of gradient will make adversarial examples change to wrong target categories. We assume that a gradient based method can obtain a more accurate gradient direction by knowledge distillation.

The distance between the gradient of a target class and the gradient of a wrong class can be give by:

∆ = ∂C target ∂z i - ∂C wrong ∂z i = (p T rue -q target ) -(p T rue -q wrong ) = q wrong -q target (5)

where C target means the cross entropy about target class, C wrong means the cross entropy about wrong probability. ∆ means the distance of two gradient. We define δ = ∆/p true , so it is easy to get: Any gradient based adversarial attack algorithm can be easily extended to include distillation. For example, FGSM can be simply extended to Distillation FGSM (D-FGSM) as follows.

φ = δ δ = ∆ /p true ∆/p true = (q wrong -q target )/p true (q wrong -q target )/

X adv = x real + • sign( x J( L(x) T , y))

Here, x J is the gradient of loss function, L(x) means the logits of image x produced by neural network, and sign(•) is sign function limits the size of the disturbance.

Momentum iteration fast gradient sign method can be simply extended to Distillation MI-FGSM (D-MIFGSM). Liu et al. (2017) ensemble neural networks to improve transferability in optimize based method. They consider there is no difference between single model attack and ensemble based attack in fast gradient based method, and both these two methods can not produce transferable adversarial examples. Dong et al. (2018b) 's ensemble method is a simple weighted average of logits generated by multiple neural networks, which they call "fuse in logits".

g t+1 = µ • g t + ∇ x J( L(x) T , y) ∇ x J( L(x) T , y) 1 x adv t+1 = x adv t + α • sign(g t+1 ) 4.2 ENSEMBLE BASED METHOD

We have a better ensemble approach. By distilling logits generated by n neural networks, n softened logits can be obtained. In the gradient based attack method, n logits can obtain n cross entropy losses, and the cross entropy losses generated by the softened logits and the original logits, namely the hardened logits, can produce a total of 2n cross entropy losses. A simple weighted average of these cross entropy losses can extend the gradient based approach to the ensemble attack method that includes knowledge distillation:

EQUATION

where L(x) is the logit produced by different neural networks, and J(x, y) is the cross entropy between the logit x and the attack target y.

We have carried out experiments on different adversarial attack algorithms and found that the momentum iterative fast gradient sign method based on knowledge distillation (D-MIFGSM) is the most effective method to produce transferable adversarial examples.

We summarize our ensemble based method in Algorithm 1, and the rest of our experiments are based on this algorithm. Specifically, when we input a clean image x and get K logits from K classifiers, we distill these N logits using a pre-defined temperature T . Then, we perform Equation 8 "fuse in cross entropy" on these distilled logits. The rest of the algorithm is the same as MI-FGSM.

5. EXPERIMENT:



5.1. SETUP:

To see how distillation works on targeted attack and what is the best distillation environment, we design a series of experiments. We choose four networks, Inception V 3, Inception V 4 (Szegedy 

5.3. ENSEMBLE BASED METHOD:

We find that attacking V GG networks can generate adversarial examples with high transferability in a single model attack. Moreover, in the optimization-based approach, ensemble networks have been proved to be able to produce transferable adversarial examples.

We test the impact of ensemble on transferability of different networks. In our test, we take off one of the four known networks each time, and the results are shown on Table 2 .

Since it is possible to generate a large number of adversarial examples with high transferability in the distillation based method through ensemble models, we perform different parameter tests on this method.

5.4. THE EFFECT OF TEMPERATURE:

Firstly, we want to know how different distillation temperatures affect the attack success rate, and which is the best temperature to distill. We test different temperatures while the iteration number is 20 and the max disturb limit is 32. The success rate of D-MIFGSM in ensemble based method. Cell (i, j) indicates that the accuracy of the targeted adversarial examples generated for the ensemble of the four models except model i (row) is predicted as the target label by model j (column). In each line ,means the network behind is not used in ensemble attack.

Figure 1 shows the effect of different distillation temperatures on black-box targeted attack, where the temperature equals 1 means there's no distillation processing. It is clear that the distillation can make effects on both single model attack and ensemble based method. And the distillation temperature makes best effects when it reaches about 16. But in our method, as Figure 2 shows, the more iterations, the much transferability we can get, until it is large enough to converge. In another word, to certain extent, we avoid the overfit problem. However, high iterations also mean long computing time. In our experiment, we suggest 30 to 50 iterations to be a better choice.

:



Figures and Tables:

Figure 1: Effect of Different Temperature

Figure 2: Effect of Different Iterations

It means distillation can effectively increase the distance between the gradient of the target class and the gradient of a wrong class. In this way, the gradient based attack method can attack to a targeted class more precisely.

Table content: <table><tr><td colspan="6">4 DISTILLATION ADVERSARIAL ATTACK</td><td/></tr><tr><td colspan="6">4.1 SINGLE MODEL GRADIENT-BASED ATTACK</td><td/></tr><tr><td/><td/><td/><td/><td/><td>p true</td><td/></tr><tr><td>=</td><td colspan="2">e wrong/T -e target/T i e i/T e wrong -e target i e i / e true / e true/T i e i/T i e i</td><td colspan="2">=</td><td colspan="3">e wrong/T -e target/T e true/T</td><td>/</td><td>e true e wrong -e target</td><td>(6)</td></tr><tr><td/><td>φ ≈</td><td colspan="2">wrong/T -target/T e true/T</td><td>/</td><td>wrong -target e true</td><td>≈</td><td>e true e true/T &gt;&gt; 1</td><td>(7)</td></tr></table>

Algorithm 1 Ensemble D-MIFGSMInput: The logits of K classifiers l1, l2, ..., l K ;Temprature T ; a real example x and target label y * ; Input: The size of perturbation ε; iterations N and decay factor µ. Output: An adversarial example x * with x * -x inf ≤ ε ResN et V 1 50, ResN et V 1 152(He et al. (2016)) and Inception Resnet(Szegedy et al. (2017)) as the networks in black-box attacks to test the transferability of the adversarial examples.To verify the true effectiveness, we tend to choose a bigger dataset. So, we randomly choose 10,000 images from ILSVRC 2012 validation set, with 10 images in each class, and make sure they can be all classified correctly by the networks we use. We define the disturb of adversarial examples by L inf -distance.5.2 SINGLE MODEL ATTACKForLiu et al. (2017), attacking one model is the same as attacking multiple models in fast gradient method, neither of which can produce adversarial examples with high transferability. We test the performance of different networks in a single model attack. Table1shows that the transferability in single model attack can reach about 30% by attacking V GG networks. And we find that only attacking V GG networks can produce transferable adversarial examples. The success rate of D-MIFGSM in single model attack. The cell (i, j) indicates the accuracy of the adversarial examples generated for model i (row) evaluated over model j (column).

Table content: <table><tr><td>1: α = ε/N ;</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>2: for i = 0 to N -1 do</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="7">3: Input x  *  i and output l k (x  *  i ) for k = 1, 2, ..., K; l(x  *  i ) 4: Get different logits as l(x  *  i ) = T l(x  *  i ) 5: Fuse the softmax cross-entropy loss J(x  *  i , y  *  ) based on Equation 8. 6: Obtain the gradient ∇xJ(x  *  i , y  *  ); 7: Update gi+1 by accumulating the velocity vector in the gradient direction as gi+1 = µ • gi + 8: Update x  *  i+1 by applying the sign gradient as x  *  i+1 = x  *  i -α • sign(gi+1);</td><td colspan="2">∇xJ(x  *  i , y  *  ) ∇xJ(x  *  i , y  *  ) 1</td><td>;</td></tr><tr><td>9: end for</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>10: return x  *  = x  *  N .</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="9">et al. (2016)), V GG16 and V GG19 (Simonyan &amp; Zisserman (2015)), to produce the adversarial</td></tr><tr><td colspan="2">examples. We use IncV 3</td><td>IncV 4</td><td>V GG16</td><td>V GG19</td><td>Resnet 50</td><td colspan="2">Resnet 152</td><td>Inc Res</td></tr><tr><td>IncV 3</td><td>0.999</td><td>0.018</td><td>0.009</td><td>0.007</td><td>0.006</td><td/><td>0.005</td><td>0.020</td></tr><tr><td>IncV 4</td><td>0.059</td><td>0.991</td><td>0.125</td><td>0.010</td><td>0.005</td><td/><td>0.005</td><td>0.027</td></tr><tr><td>V GG16</td><td>0.241</td><td>0.205</td><td>1.0</td><td>0.929</td><td>0.340</td><td/><td>0.235</td><td>0.201</td></tr><tr><td>V GG19</td><td>0.233</td><td>0.197</td><td>0.918</td><td>1.0</td><td>0.333</td><td/><td>0.241</td><td>0.191</td></tr></table>



Table content: <table><tr><td/><td>IncV 3</td><td>IncV 4</td><td>V GG16</td><td>V GG19</td><td>Resnet 50</td><td>Resnet 152</td><td>Inc Res</td></tr><tr><td>All</td><td>0.949</td><td>0.982</td><td>1.0</td><td>1.0</td><td>0.471</td><td>0.385</td><td>0.334</td></tr><tr><td>-IncV 3</td><td>0.499</td><td>0.991</td><td>1.0</td><td>1.0</td><td>0.575</td><td>0.462</td><td>0.386</td></tr><tr><td>-IncV 4</td><td>0.937</td><td>0.294</td><td>1.0</td><td>1.0</td><td>0.437</td><td>0.355</td><td>0.219</td></tr><tr><td>-V GG16</td><td>0.939</td><td>0.980</td><td>0.803</td><td>1.0</td><td>0.232</td><td>0.176</td><td>0.192</td></tr><tr><td>-V GG19</td><td>0.941</td><td>0.979</td><td>1.0</td><td>0.787</td><td>0.228</td><td>0.173</td><td>0.188</td></tr></table>

Peer review (iclr2020):
The paper suggests to use temperature scaling in adversarial attack design for improving transferability under black-box attack setting. Based on this, the paper proposes several new attacks: D-FGSM, D-MIFGSM, and their ensemble versions. Experimental results found that the proposed methods improves transferability from VGG networks, compared to the non-distillated counterparts. In overall, I liked its novel motivation and simplicity of the method, but it seems to me the manuscript should be improved to meet the ICLR standard. Firstly, the presentation of the method is not that clear to me. The mathematical notations are quite confusing for me as most of them are used without any definitions. I am still not convinced that the arguments in Section 3.1 and 3.2 are indeed relevant to the actual practice of black-box adversarial attacks, which usually includes extremely non-smooth boundaries with multiple gradient steps. Even though the experiments show effectiveness partially on VGGNets, but the overall improvements are not sufficient for me to claim the general effectiveness of the method unless the paper could provide additional results on broader range of architectures and threat models. - I feel Section 2.3 is too subjective with vague statements. The following statement was particularly unclear to me: *The first problem with gradient based methods is that they lose their effectiveness after a certain number of iterations.*: Does the term *effectiveness* indicate some relative effectiveness compared to other methods, e.g. optimization-based attacks? Is this really a general phenomenon in gradient-based attacks? Also, please elaborate more on *So, insufficient information acquisition for different categories and premature stop of gradient update are the reasons ...* - Regarding that the softmax is the problem, one could try to directly minimize the logit layers skipping the softmax, i.e., gradient on logits? This is actually one of common techniques and there are many simple tricks in the context of adversarial attack, so the paper may include comparisons with such of tricks as well. - It is important to specify the exact threat model used throughout the experiments, e.g. perturbation constraints and attack details. Demonstrating the effectiveness on a variety of threat models could also strengthen the manuscript. - Table 1 and 2 may include other baseline (black-box attack) methods for comparison. This would much help to understand the method better.