Year: 2019

Abstract:
Neural networks show great vulnerability under the threat of adversarial examples. By adding small perturbation to a clean image, neural networks with high classification accuracy can be completely fooled. One intriguing property of the adversarial examples is transferability. This property allows adversarial examples to transfer to networks of unknown structure, which is harmful even to the physical world. The current way of generating adversarial examples is mainly divided into optimization based and gradient based methods. Liu et al. (2017) conjecture that gradient based methods can hardly produce transferable targeted adversarial examples in black-box attack. However, in this paper, we use a simple technique to improve the transferability and success rate of targeted attacks with gradient based methods. We prove that gradient based methods can also generate transferable adversarial examples in targeted attacks. Specifically, we use knowledge distillation for gradient based methods, and show that the transferability can be improved by effectively utilizing different classes of information. Unlike the usual applications of knowledge distillation, we did not train a student network to generate adversarial examples. We take advantage of the fact that knowledge distillation can soften the target and obtain higher information, and combine the soft target and hard target of the same network as the loss function. Our method is generally applicable to most gradient based attack methods.

heading: 1 INTRODUCTION
text: Neural networks have been shown to be susceptible to adversarial examples. They can be misled when we add small perturbation to a clean image, even though the perturbation may be invisible to the human eye. Adversarial examples are widely used in different physical attack scenarios, including face recognition, voice recognition, and autonomous driving. In the physical world, however, it is difficult to obtain the structure of neural networks. This creates a need for attack algorithms in black-box scenarios. As attack algorithms are created for defense purposes, early discovery of adversarial examples that harm the physical world can help us defend against unknown threats.
Many researchers have observed that adversarial examples can be transferred between different networks. Papernot et al. (2016) create adversarial examples which can be transferred to black-box scenarios by attacking a constructed substitute model. Existing attack algorithms have high attack success rate in white-box scenarios and good transferability in non-targeted attacks. However, there is a very low transferability in targeted black-box scenes which is more harmful. Due to the tricky degree of targeted attack transferability problem, almost all existing researches to improve the transferability focus on the untargeted attack part. In fact, it is easy to understand that an untargeted attack only needs to move the original image away from its category, while a targeted attack needs to reach the target category while penetrating the classification boundary.
Liu et al. (2017) consider gradient based method only searching attacks in a 1-D subspace. In this approach, the subspace contains just a small subset of all target labels. They improve transferability by using ensemble neural networks in the optimization based method. They also prove that the approach of ensemble models is not effective for the gradient based method. For their experimental results and hypotheses, there is no difference between attacking ensemble neural networks and attacking a neural network for the transferability of gradient based method. We will show this is not correct by our study.
There are two problems in existing gradient based methods for targeted attacks. Firstly, the attack success rate of gradient based method cannot be improved after a certain number of iterations. Secondly, targeted attacks do not make full use of the information between different categories.
Existing adversarial attack algorithms usually ensemble neural networks to produce adversarial examples. Dong et al. (2018b) use an approach namely “fuse in logits” to ensemble models. “Fuse in logit” performs a simple weighted average of logits before the softmax layer of the neural network, and then calculates the cross entropy loss between logits and the one-hot vector of the target labels. The ensemble attack algorithm produces adversarial examples in each ensembled model, which can achieve a good attack success rate. But this is not black-box transferability, since every model structure which is ensembled can be acquired by an attack algorithm. Black-box transferability refers to the fact that adversarial examples can still attack successfully against the networks with unknown structures. In this paper, the term transferability will refer uniformly to the black-box and targeted attack scenarios.
Hinton et al. (2015) put forward the concept of knowledge distillation. Knowledge distillation is used to extract useful dark knowledge from bloated models to train a small and lightweight network. Knowledge distillation puts forward the concept of soft targets and hard targets. Through the cumbersome model of probability distribution divided by a value greater than 1, the original probability distribution can be transformed to soft targets. When the soft targets have high entropy, they provide much more information.
Instead of training a small model from multiple large models, we simply take advantage of the fact that knowledge distillation can effectively extract knowledge. Actually, we are not attacking a lightweight model extracted from a bloated model. It is simpler and more efficient to produce transferable adversarial examples in targeted attack, which we call self-distillation adversarial attack.

heading: 2 BACKGROUND

heading: 2.1 ADVERSARIAL EXAMPLES
text: Deep neural networks are proved vulnerable to adversarial attacks since Szegedy et al. (2014), which shows that neural networks can be misclassified by adding small perturbations to a clean picture. The existing of adversarial examples has proven to be a huge threat to the entire application of deep learning algorithms, including speech recognition, face recognition, and even the recognition of road signs in the physical world. As a result, more and more attention has been paid to adversarial examples, and many defensive researches have been made to improve the robustness of neural networks. At present, the research on the attack and defense of neural network is also in a state of game and spiral. The research on adversarial examples is also a good starting points for the explainability of neural network.

heading: 2.2 GRADIENT-BASED ADVERSARIAL ATTACK METHODS
text: The current methods of generating adversarial examples are mainly gradient based and optimization based. Here, we briefly introduce the gradient based method.
Goodfellow et al. (2015) come up with a way to get adversarial examples by applying the sign of the gradient to a real example only once. They call this method the Fast Gradient Sign Method (FGSM).
Kurakin et al. (2017) move the adversarial example in the direction of the sign of the gradient in each iteration.
Dong et al. (2018b) integrate momentum into iterative FGSM (I-FGSM), and this can help attack algorithm stabilizing update directions and escaping from poor local maxima. This method which is called MI-FGSM solves the problem that I-FGSM is easy to obtain local extreme value and “overfit” model.
Madry et al. (2018) propose a saddle point formulation, which can convert the attack problems and defense problems to inner maximization problems and outer minimization problems, respectively. They suppose projected gradient descent (PGD) could be the strongest universal attack method.

heading: 2.3 THE WEAKNESS OF GRADIENT BASED METHOD
text: The first problem with gradient based methods is that they lose their effectiveness after a certain number of iterations. In neural networks, softmax is usually used as the activation function to get probability output for multi-classification problems. In gradient based adversarial attacks, the probability output of image in a neural network is usually concerned. Therefore, most adversarial attack methods also use softmax activation function to obtain the probability and calculate the cross entropy with the target category, and finally optimize the cross entropy loss function to get the adversarial example.
However, softmax has saturation problems. Softmax’s probability output sum is 1, so there is competition among each output. In extreme cases, it becomes winner-take-all, and when one output “approaches 1”, the others “approach 0”. Another downside of softmax is that it can cause either overspill or underspill problems. Although this downside can be effectively alleviated by subtracting the maximum value of all inputs from the input value of softmax, it causes the output value of softmax to be driven by the maximum input value. When the input value is the maximum, the output value will be close to 1. On the contrary, the large difference can cause softmax’s output to be saturated to 0.
Liu et al. (2017) prove that the classification boundaries between white-box models and black-box models do not coincide. And for the model which is not used to generate the adversarial image, its classification area is much smaller. This means that when a successful attack on neural network whose structure is known, adversarial examples cannot penetrates the classification boundaries of other neural networks unknown structure precisely. It is more important to obtain information of different categories. The gradient update stop caused by softmax saturation makes gradient based adversarial examples unable to accurately penetrate classification boundaries.
So, insufficient information acquisition for different categories and premature stop of gradient update are the reasons why gradient based methods have low transferability in targeted black-box scenes. The solution to this problem is very simple. We introduce knowledge distillation into gradient based methods.

heading: 2.4 DISTILL IN NEURAL NETWORKS
text: In machine learning tasks, researchers always train many different kinds of models on the same dataset, and combine the results of each single model into a final result (Dietterich (2000)). A simple combined method is to average the predictions, but the cost of this ensemble method is too expensive to deploy.
Hinton et al. (2015) find that instead of simply averaging the predictions, it will get much better performance if converting the logits by:
qi = exp(zi/T )∑ j exp(zj/T )
(1)
where T is called a temperature. Using a higher T can produce a softer probability distribution over classes. And it will be normal softmax-layer when T is set by 1.
Distillation in neural networks can make the combination of multiple networks perform better, and the knowledge of the cumbersome model can tansfer to a small model in an effective way. The current research on distillation is mainly used to train a smaller model from a bloated model set. However, distillation can effectively help to get the target information in adversarial attacks.

heading: 3 METHODOLOGY

heading: 3.1 THE SOLVE OF SATURATION PROBLEM
text: Here, we prove that knowledge distillation can effectively reduce the distance between classes and solve the saturation problem. If we use the ratio ρ to measure the distance between the softmaxlayer’s outputs, we can get the distance of two hard targets below.
ρ = q(xa)
q(xb) =
exa∑ i e i exb∑ i e i = exa exb (2)
If we distill the logits with a temperature T like formula-1, the distance will be as follows.
ρ′ = q(xa/T )
q(xb/T ) =
exa/T∑ i e i/T exb/T∑ i e i/T = exa/T exb/T (3)
So the relationship between ρ′ and ρ is:
ρ′ = T √ ρ (4)
The distillation processing is essentially the dispersion of the softmax results among different classifications using the T -th power of the square root, so that the different classification softmax values distribution are closer.

heading: 3.2 GET MORE ACCURATE DIRECTION
text: It may be a good thing for untargeted attacks when data classes are close to each other, because it makes easy for adversarial examples to randomly move from the original category to a target category and the attack succeeds. But for targeted attacks, this is a problem, making it more likely to attack the wrong class.
Liu et al. (2017) observe that the decision boundaries of all models are very consistent with each other. They think this explains why non-targeted adversarial images can transfer among models. But for targeted attack, decision boundaries of models do not generate adversarial examples in a small area, thus tiny change of gradient will make adversarial examples change to wrong target categories. We assume that a gradient based method can obtain a more accurate gradient direction by knowledge distillation.
The distance between the gradient of a target class and the gradient of a wrong class can be give by:
∆ = ∂Ctarget ∂zi − ∂Cwrong ∂zi = (pTrue − qtarget)− (pTrue − qwrong) = qwrong − qtarget (5)
where Ctarget means the cross entropy about target class, Cwrong means the cross entropy about wrong probability. ∆ means the distance of two gradient. We define δ = ∆/ptrue, so it is easy to get:
φ = δ′
δ = ∆′/p′true ∆/ptrue = (q′wrong − q′target)/p′true (qwrong − qtarget)/ptrue
=
ewrong/T−etarget/T∑ i e i/T / etrue/T∑ i e i/T
ewrong−etarget∑ i e i / etrue∑ i e i
= ewrong/T − etarget/T etrue/T / ewrong − etarget etrue
(6)
Compared to the true class, the values of the wrong class and the target class tend to zero. We can approximate that:
φ ≈ wrong/T − target/T etrue/T / wrong − target etrue ≈ e true etrue/T >> 1 (7)
It means distillation can effectively increase the distance between the gradient of the target class and the gradient of a wrong class. In this way, the gradient based attack method can attack to a targeted class more precisely.

heading: 4 DISTILLATION ADVERSARIAL ATTACK

heading: 4.1 SINGLE MODEL GRADIENT-BASED ATTACK
text: Any gradient based adversarial attack algorithm can be easily extended to include distillation. For example, FGSM can be simply extended to Distillation FGSM (D-FGSM) as follows.
Xadv = xreal + · sign(5xJ( L(x)
T , y))
Here, 5xJ is the gradient of loss function, L(x) means the logits of image x produced by neural network, and sign(·) is sign function limits the size of the disturbance. Momentum iteration fast gradient sign method can be simply extended to Distillation MI-FGSM (D-MIFGSM).
gt+1 = µ · gt + ∇xJ(L(x)T , y)
‖ ∇xJ(L(x)T , y) ‖1 xadvt+1 = x adv t + α · sign(gt+1)

heading: 4.2 ENSEMBLE BASED METHOD
text: Liu et al. (2017) ensemble neural networks to improve transferability in optimize based method. They consider there is no difference between single model attack and ensemble based attack in fast gradient based method, and both these two methods can not produce transferable adversarial examples. Dong et al. (2018b)’s ensemble method is a simple weighted average of logits generated by multiple neural networks, which they call “fuse in logits”.
We have a better ensemble approach. By distilling logits generated by n neural networks, n softened logits can be obtained. In the gradient based attack method, n logits can obtain n cross entropy losses, and the cross entropy losses generated by the softened logits and the original logits, namely the hardened logits, can produce a total of 2n cross entropy losses. A simple weighted average of these cross entropy losses can extend the gradient based approach to the ensemble attack method that includes knowledge distillation:
J(L(x), y) = λ1 · n∑ i=1 J( L(xi) T , y) + λ2 · n∑ i=1 J(L(xi), y) (8)
where L(x) is the logit produced by different neural networks, and J(x, y) is the cross entropy between the logit x and the attack target y.
We have carried out experiments on different adversarial attack algorithms and found that the momentum iterative fast gradient sign method based on knowledge distillation (D-MIFGSM) is the most effective method to produce transferable adversarial examples.
We summarize our ensemble based method in Algorithm 1, and the rest of our experiments are based on this algorithm. Specifically, when we input a clean image x and get K logits from K classifiers, we distill these N logits using a pre-defined temperature T . Then, we perform Equation 8 “fuse in cross entropy” on these distilled logits. The rest of the algorithm is the same as MI-FGSM.

heading: 5 EXPERIMENT

heading: 5.1 SETUP
text: To see how distillation works on targeted attack and what is the best distillation environment, we design a series of experiments. We choose four networks, Inception V 3, Inception V 4 (Szegedy
Algorithm 1 Ensemble D-MIFGSM Input: The logits ofK classifiers l1, l2, ..., lK ;Temprature T ; a real example x and target label y∗; Input: The size of perturbation ε; iterationsN and decay factor µ. Output: An adversarial example x∗ with ‖ x∗ − x ‖inf≤ ε 1: α = ε/N ; 2: for i = 0 toN − 1 do 3: Input x∗i and output lk(x ∗ i ) for k = 1, 2, ..., K;
4: Get different logits as l(x∗i ) =
{ l(x∗i )
T
l(x∗i )
5: Fuse the softmax cross-entropy loss J(x∗i , y ∗) based on Equation 8. 6: Obtain the gradient∇xJ(x∗i , y ∗); 7: Update gi+1 by accumulating the velocity vector in the gradient direction as gi+1 = µ · gi + ∇xJ(x∗i , y ∗)
‖ ∇xJ(x∗i , y∗) ‖1 ;
8: Update x∗i+1 by applying the sign gradient as x ∗ i+1 = x ∗ i − α · sign(gi+1); 9: end for 10: return x∗ = x∗N .
et al. (2016)), V GG16 and V GG19 (Simonyan & Zisserman (2015)), to produce the adversarial examples. We use ResNet V 1 50, ResNet V 1 152 (He et al. (2016)) and Inception Resnet (Szegedy et al. (2017)) as the networks in black-box attacks to test the transferability of the adversarial examples.
To verify the true effectiveness, we tend to choose a bigger dataset. So, we randomly choose 10,000 images from ILSVRC 2012 validation set, with 10 images in each class, and make sure they can be all classified correctly by the networks we use. We define the disturb of adversarial examples by Linf -distance.

heading: 5.2 SINGLE MODEL ATTACK
text: For Liu et al. (2017), attacking one model is the same as attacking multiple models in fast gradient method, neither of which can produce adversarial examples with high transferability. We test the performance of different networks in a single model attack. Table 1 shows that the transferability in single model attack can reach about 30% by attacking V GG networks. And we find that only attacking V GG networks can produce transferable adversarial examples.

heading: 5.3 ENSEMBLE BASED METHOD
text: We find that attacking V GG networks can generate adversarial examples with high transferability in a single model attack. Moreover, in the optimization-based approach, ensemble networks have been proved to be able to produce transferable adversarial examples.
We test the impact of ensemble on transferability of different networks. In our test, we take off one of the four known networks each time, and the results are shown on Table 2.
Since it is possible to generate a large number of adversarial examples with high transferability in the distillation based method through ensemble models, we perform different parameter tests on this method.

heading: 5.4 THE EFFECT OF TEMPERATURE
text: Firstly, we want to know how different distillation temperatures affect the attack success rate, and which is the best temperature to distill. We test different temperatures while the iteration number is 20 and the max disturb limit is 32.
Figure 1 shows the effect of different distillation temperatures on black-box targeted attack, where the temperature equals 1 means there’s no distillation processing. It is clear that the distillation can make effects on both single model attack and ensemble based method. And the distillation temperature makes best effects when it reaches about 16.

heading: 5.5 THE EFFECT OF THE ITERATION
text: After the test of temperature, we want to know the effect of the iteration number on the success rate of the attack in the distillation method. Dong et al. (2018a) suppose that the adversarial examples generated by iterative methods have a tendency to overfit, which means iteration is negative to the transferability of adversarial examples.
But in our method, as Figure 2 shows, the more iterations, the much transferability we can get, until it is large enough to converge. In another word, to certain extent, we avoid the overfit problem.
However, high iterations also mean long computing time. In our experiment, we suggest 30 to 50 iterations to be a better choice.

heading: 5.6 THE EFFECT OF THE DISTURB LIMIT
text: The distance between origin images and adversarial examples is also an important indicator to judge the performance of attacking methods. In both MIFGSM and D-MIFGSM, the distance depends on the max disturb limit which is a hyperparameter in attack processing. It is also an important task to find out the best disturb limit.
As Figure 3 shows, lower disturb causes lower attack success rate. But at the mean time, too big disturb limit also has a negative effect on the success rate of adversarial attack. On the other hand, bigger disturb means bigger noise, which is more likely to be realized by human. Above all, in our experiment, we prefer to choose the disturb limit between 16 and 32.

heading: 5.7 THE COMPARISON BETWEEN MIFGSM AND D-MIFGSM
text: We compare the attack success rate in both black-box and white-box attack scenarios between MIFGSM and our D-MIFGSM. The setup of the temperature is 16, the iterations is 20 and the disturb limit is 32.
Table 3 shows that in both two kinds of attack scenarios D-MIFGSM provides a significant boost to MI-FGSM.

heading: 6 CONCLUSION
text: In this paper, we show that gradient based methods can also produce adversarial examples with high transferability. In order to generate adversarial examples with high transferability in targeted attack, we introduce knowledge distillation into gradient based methods. Our method can be easily implemented in any method based on fast gradient. In targeted attack, the ensemble of multiple networks and optimization based methods is proved to produce adversarial examples with high transferability. But optimization-based methods are notoriously slow. We introduce knowledge distillation into the momentum iterative fast gradient method to produce adversarial examples of about 30% to 50% transferability in the experiment. At the same time, we find out that the network structure also has an effect on transferability, and it will be our future work to explore neural network structures that can produce transferability.

Review summaries:
This paper proposes distillation attacks to generate transferable targeted adversarial examples. The technique itself is pretty simple: instead of only using the raw logits L(x) to compute the cross entropy loss for optimization, they also use the distilled logits L(x)/T to generate adversarial examples. Their evaluation setup largely follows the style of Liu et al., but they construct a different subset of ILSVRC validation set, and some of the model architectures in their ensemble are different from Liu et al. Their results show that by including the distilled logits when computing the gradient, the generated adversarial examples can transfer better among different models using both single-model and ensemble-based attacks.