SELF-KNOWLEDGE DISTILLATION ADVERSARIAL ATTACK
Authors: Ma Xiaoxiong[1]
Wang Renzhi[1]
Tian Cong
Dong Zeqian
Duan Zhenhua
Venue:  (2020)
Keywords: Adversarial Examples
Transferability
black-box targeted attack
Distillation

Abstract: Neural networks show great vulnerability under the threat of adversarial examples.
   By adding small perturbation to a clean image, neural networks with high classification accuracy can be completely fooled.
   One intriguing property of the adversarial examples is transferability.  This property allows adversarial examples to transfer to networks of unknown structure, which is harmful even to the physical world.
   The current way of generating adversarial examples is mainly divided into optimization based and gradient based methods.
   Liu et al. (2017) conjecture that gradient based methods can hardly produce transferable targeted adversarial examples in black-box-attack.
   However, in this paper, we use a simple technique to improve the transferability and success rate of targeted attacks with gradient based methods.
   We prove that gradient based methods can also generate transferable adversarial examples in targeted attacks.
   Specifically, we use knowledge distillation for gradient based methods, and show that the transferability can be improved by effectively utilizing different classes of information.
   Unlike the usual applications of knowledge distillation, we did not train a student network to generate adversarial examples.
   We take advantage of the fact that knowledge distillation can soften the target and obtain higher information, and combine the soft target and hard target of the same network as the loss function.
   Our method is generally applicable to most gradient based attack methods.

Under review as a conference paper at ICLR 2020
SELF-KNOWLEDG DISTILLATION ADVERSARIAL AT- TACK
Anonymous authors
Paper under double-blind review
ABSTRACT
Neural networks show great vulnerability under the threat of adversarial exam- ples. By adding small perturbation to clean image, neural networks with high classification accuracy can be completely fooled. One intriguing property of the adversarial examples is transferability. This property allows adversarial exam- ples transfer to networks of unknown structure, which is harmful even the physical world. The current way of generating adversarial examples mainly divided into optimization based and gradient based methods. Liu al. (2017) conjecture that gradient based methods can hardly produce transferable targeted adversarial examples in black-box attack. However, in this paper, technique to improve the transferability and success rate of targeted attacks with gradient based methods. We prove that gradient based methods can also generate transferable adversarial examples in targeted attacks. Specifically, we use knowl- edge distillation for gradient based methods, and show that the transferability can be improved by effectively utilizing different classes of information. Unlike the usual applications of knowledge distillation, we did not train a student network to generate adversarial examples. We take advantage of the fact that knowledge distillation can soften the target and obtain higher information, and combine the soft target and hard target of the same network as the loss function. Our method is generally applicable to most gradient based attack methods.
simple
INTRODUCTION
Neural networks have been shown to susceptible to adversarial examples. They can be misled when we add small perturbation to a clean image, even though the perturbation may be invisible to the human eye. Adversarial examples are widely used in different physical attack scenarios, includ- ing face recognition, voice recognition, and autonomous driving. In the physical world, however, is difficult to obtain the structure of neural networks. This creates a need for attack algorithms in black-box scenarios. As attack algorithms created for defense purposes, early discovery of adversarial examples that harm the physical world can help us defend against unknown threats.
Many researchers have observed that adversarial examples be transferred between different net- works. Paaperntt et al.] (2016) create adversarial examples which can be transferred to black-box scenarios by attacking a constructed substitute model. Existing attack algorithms have high attack success rate in white-box scenarios and good transferability in non-targeted attacks. However, there very low transferability in targeted black-box scenes which more harmful. Due to the tricky degree of targeted attack transferability problem, almost all existing researches to improve the trans- ferability focus on the untargeted attack part. fact, is easy to understand that an untargeted attack only needs to move the original image away from its category, while targeted attack needs to reach the target category while penetrating the classification boundary.
Liu et al. (017) consider gradient based method only searching attacks in 1-D subspace. In this approach, the subspace contains just a small subset of all target labels. They improve transferabil- ity by using ensemble neural networks in the optimization based method. They also prove that the approach of ensemble models is not effective for the gradient based method. For their experimental results and hypotheses, there is difference between attacking ensemble neural networks and at- tacking a neural network for the transferability gradient based method. We will show this is not correct by our study.

---

Under review as a conference paper at ICLR 2020
There are two problems in existing gradient based methods for targeted attacks. Firstly, the attack success rate of gradient based method cannot be improved after certain number of iterations. Secondly, targeted attacks do not make full use of the information between different categories.
Existing adversarial attack algorithms usually ensemble neural networks to produce adversarial ex- amples. Dong al. (2018b) use approach namely "fuse in logits" to ensemble models. "Fuse in logit" performs a simple weighted average logits before the softmax layer of the neural network, and then calculates the cross entropy loss between logits and the one-hot vector of the target labels. The ensemble attack algorithm produces adversarial examples in each ensembled model, which can achieve good attack success rate. But this not black-box transferability, since every model struc- ture which is ensembled can be acquired by an attack algorithm. Black-box transferability refers to the fact that adversarial examples can still attack successfully against the networks with unknown structures. In this paper, the term transferability will refer uniformly to the black-box and targeted attack scenarios.
Hinton et al. (2015) put forward the concept of knowledge distillation. Knowledge distillation is used to extract useful dark knowledge from bloated models to train a small and lightweight network. Knowledge distillation puts forward the concept of soft targets and hard targets. Through the cum- bersome model probability distribution divided by a value greater than 1, the original probability distribution can be transformed to soft targets. When the soft targets have high entropy, they provide much more information.
Instead of training small model from multiple large models, simply take advantage of the fact that knowledge distillation can effectively extract knowledge. Actually, we are not attacking lightweight model extracted from a bloated model. It is simpler and more efficient produce transferable adversarial examples in targeted attack, which we call self-distillation adversarial attack.
BACKGROUND
2.1 ADVERSARIAL EXAMPLES
Deep neural networks are proved vulnerable to adversarial attacks Since/SZzeedyy et al. (2014), which shows that neural networks be misclassified by adding small perturbations to a clean picture. The existing of adversarial examples has proven to he a huge threat to the entire application of deep learning algorithms, including speech recognition, face recognition, and even the recognition of road signs in the physical world. As result, more and more attention has been paid to adversarial exam- ples, and many defensive researches have been made to improve the robustness of neural networks. At present, the research on the attack and defense of neural network is also in state of game and spiral. The research on adversarial examples also good starting points for the explainability of neural network.
2.2 GRADIENT-BASED ADVERSARIAL ATTACK METHOBS
The current methods of generating adversarial examples are mainly gradient based and optimization based. Here, we briefly introduce the gradient based method.
Goodfellow al. (2015) come up with way to get adversarial examples by applying the sign of the gradient to a real example only once. They call this method the Fast Gradient Sign Method (FGSM).
Kurakin et al. 2017) move the adversarial example in the direction of the sign of the gradient in each iteration.
Dong et al. (2018b) integrate momentum into iterative FGSM (I-FGSM), and this can help attack algorithm stabilizing update directions and escaping from poor local maxima. This method which is called MI-FGSM solves the problem that I-FGSM is easy to obtain local extreme value and "overfit" model.
Madry et al. (2018) propose saddle point formulation, which can convert the attack problems and defense problems to inner maximization problems and outer minimization problems, respectively. They suppose projected gradient descent (PGD) could be the strongest universal attack method.
2

---

Under review as a conference paper at ICLR 2020
2.3 THE WEAKNESS OF GRADIENT BASED METHOD
The first problem with gradient based methods is that they lose their effectiveness after a certain number of iterations. In neural networks, softmax is usually used the activation function to get probability output for multi-classification problems. In gradient based adversarial attacks, the probability output of image in neural network usually concerned. Therefore, most adversarial attack methods also use softmax activation function to obtain the probability and calculate the cross entropy with the target category, and finally optimize the cross entropy loss function to get the adversarial example.
However, softmax has saturation problems. Softmax's probability output sum 1, so there is com- petition among each output. In extreme cases, becomes winner-take-all, and when one output "approaches 1", the others "approach 0". Another downside of softmax that can cause either overspill or underspill problems. Although this downside can be effectively alleviated by subtract- ing the maximum value of all inputs from the input value of softmax, causes the output value of softmax to be driven by the maximum input value. When the input value is the maximum, the output value will be close to On the contrary, the large difference can cause softmax's 's output to be saturated to O.
Liu et al. (2017) prove that the classification boundaries between white-box models and black-box models do coincide. And for the model which is not used to generate the adversarial image, its classification area is much smaller. This means that when a successful attack on neural network whose structure is known, adversarial examples cannot penetrates the classification boundaries of other neural networks unknown structure precisely. It is more important to obtain information of different categories. The gradient update stop caused by softmax saturation makes gradient based adversarial examples unable to accurately penetrate classification boundaries.
So, insufficient information acquisition for different categories and premature stop of gradient update are the reasons why gradient based methods have low transferability in targeted black-box scenes. The solution this problem very simple. We introduce knowledge distillation into gradient based methods.
2.4 DISTILL IN NEURAL NETWORKS
In machine learning tasks, researchers always train many different kinds models on the same dataset, and combine the results of each single model into final result (Dietterich (2000)). A simple combined method is to average the predictions, but the cost of this ensemble method is too expensive to deploy.
Hinton et al. (2033) find that instead simply averaging the predictions, will get much better performance converting the logits by:
eap(z: (T))
/: E, exp(2;/T)
where I' is called temperature. Using a higher T can produce a softer probability distribution over classes. And it will be normal softmax-layer when T is set by 1.
Distillation in neural networks can make the combination of multiple networks perform better, and the knowledge of the cumbersome model can tansfer to small model in an effective way. The current research on distillation mainly used to train a smaller model from bloated model set. However, distillation can effectively help to get the target information in adversarial attacks.
3
METHODOLOGY
3.1 THE SOLVE OF SATURATION PROBLEM
Here, prove that knowledge distillation effectively reduce the distance between classes and solve the saturation problem. we use the ratio to measure the distance between the softmax- layer's outputs, we can get the distance of two hard targets below.
3

---

Under review as a conference paper at ICLR 2020
p ((Fa) g(36))
6
2
Σ e
we distill the logits with a temperature T like formula-] the distance will be as follows.
#af7 E
1' g(aa/T) ((bb/T)
e
(3)
e=//7 E/eTT
eab/T
So the relationship between p' and is:
d'= v/
The distillation processing essentially the dispersion softmax results among different clas- sifications using the T-thh power of the square root, so that the different classification softmax values distribution are closer.
3.2 GET MORE ACCURATE DIRECTION
It may be good thing for untargeted attacks when data classes close each other, because makes easy for adversarial examples to randomly move from the original category to a target category and the attack succeeds. But for targeted attacks, this problem, making it more likely to attack the wrong class.
Liu et al. 2017) observe that the decision boundaries of all models very consistent with each other. They think this explains why non-targeted adversarial images can transfer among models. But for targeted attack, decision boundaries of models do not generate adversarial examples in small area, thus tiny change of gradient will make adversarial examples change to wrong target categories. We assume that gradient based method can obtain a more accurate gradient direction by knowledge distillation.
The distance between the gradient target class and the gradient of wrong class can be give by:
A ACtarget
@Cwrong
Oz
dz4
(Prrue
Qtorget) (pIrue qwrong) Qwrong
aarret
where Ctarget means the entropy about target class, Curong means the cross entropy about wrong probability. A means the distance of two gradient. We define A/Ptrues so is easy to
get:
/1/Ttrue
(g' wrong (Awrong
4 target ) /1/ true Qtarget) Ptrue
8
A/Ptrue
sronghT Jarges/T
Arue/T of/T
6
wronggT starget/T /T
eorong
target
etrue/T
etrve
Compared to the true class, the values of the wrong class and the target class tend to zero. We can approximate that:
wrong/T target/T -targett// TT wrong target
true
etruc/T >>
etrue/T
etrue
4

---

Under review as a conference paper at ICLR 2020
It means distillation effectively increase the distance between the gradient of the target class and the gradient of a wrong class. In this way, the gradient based attack method can attack to a targeted class more precisely.
4 DISTILLATION ADVERSARIAL ATTACK
4.1 SINGLE MODEL GRADIENT-BASHE ATTACK
Any gradient based adversarial attack algorithm can be easily extended include distillation. For example, FGSM can be simply extended to Distillation FGSM (D-FGSM) as follows.
X greal sign(V=][ L(x)
T
Here, VaJ is the gradient loss function, L(x) means the logits of image x produced by neural network, and sign(-) is sign function limits the size of the disturbance.
Momentum iteration fast gradient sign method can be simply extended to Distillation MI-FGSM (D-MIFGSM).
M(x V
y)
padv F+1 = adv + a sign(g(+1)
9++1 -
V
J
Lix) T y)
4.2
ENSEMBLE BASED METHOD
Liu et al. (2017) ensemble neural networks to improve transferability in optimize based method. They consider there is no difference between single model attack and ensemble based attack in fast gradient based method, and both these methods
produce transferable adversarial examples. Dong et al. (2018b) 8by's ensemble method is a simple weighted average of logits generated by multiple neural networks, which they call "fuse in logits".
We have a better ensemble approach. By distilling logits generated by n neural networks, n softened logits can be obtained. the gradient based attack method, logits can obtain n cross entropy losses, and the cross entropy losses generated by the softened logits and the original logits, namely the hardened logits, can produce total of 2n cross entropy losses. simple weighted average of these cross entropy losses can extend the gradient based approach to the ensemble attack method that includes knowledge distillation:
T J(L()),) = J( 9) + A2
L(xe)
n
Σ J(L(x)).)) 1=1
where L(x) is the logit produced by different neural networks, and [ y) is the cross entropy between the logit and the attack target y.
We have carried out experiments on different adversarial attack algorithms and found that the mo- mentum iterative fast gradient sign method based on knowledge distillation (D-MIFGSM) is the most effective method to produce transferable adversarial examples.
We summarize our ensemble based method in Algorithm I and the rest of our experiments are based on this algorithm. Specifically, when we input clean image x and get K logits from K classifiers, we distill these N logits using pre-defined temperature T. Then, we perform Equation g "fuse in cross entropy" on these distilled logits. The rest of the algorithm the same as MI-FGSM.
EXPERIMENT
5.1
SETUP
To see how distillation works on targeted attack and what the best distillation environment, we design series of experiments. We choose four networks, Inception V3, Inception V4 (Szegedy

---

Under review as a conference paper at ICLR 2020
Algorithm Ensemble D-MIFGSM
Input: logits classifiers 11: 22; .... Temprature 7; example and target label ;
Input: The size of perturbation et iterations N and decay factor
Output: An adversarial example x= with x² laff
e/N; 2: for à = to N do
Input zi output (x))
1, 2,,,,.,
different logits (x2?)=
3
((=7)
Fuse the softmax cressentropy loss [( y² based Equation
6: Obtain the gradient V, vx
V.J(a
Update 5c+1 by accumulating the velocity vector the gradient direction gi+1 p.
Updale si44 by applying the sign gradient as 2(41 = mi sign(g++1);
end for 10: return xt a.v
et al. (2016)). VGG16 and VGG19 (Simonyan & Zisserman (2013)), produce the adversarial examples. We use ResNellVi 50, ResNel. Vi.NVe.VV.... V1_152 (He et al. (2016)) and Inceplion. Resnel (Szegedy al. (2017)) the networks in black-box attacks test the transferability of the adver-
sarial examples.
To verify the true effectiveness, tend choose a bigger dataset. So, we randomly choose 10,000 images from ILSVRC 2012 validation set, with 10 images in each class, and make sure they can be all classified correctly by the networks we use. We define the disturb of adversarial examples by Liuf- distaniit
5.2 SINGLE MODEL ATTACK
For Liu et al. 2017), attacking one model is the same as attacking multiple models in fast gradient method, neither of which can produce adversarial examples with high transferability. We test the performance of different networks in single model attack. Table I shows that the transferability single model attack can reach about 30% by attacking VGG networks. And we find that only attacking VGG networks can produce transferable adversarial examples.
IncV3
IncV4
VGG16
VGG19
Resnet 50
Resnet 152
Inc Res
TReV3
0.999
0.018
0.009
0.007
0.006
0.005
0.020
IneV4
0.059
0.991
0.125
0.010
0.005
0.005
0.027
VGG16
0.241
0.205
1.0
0.929
0.340
0.235
0.201
VGG19
0.197
0.918
Table The success rate of D-MIFGSM in single model attack. The cell (i.j) indicates the accuracy of the adversarial examples generated for model (row) evaluated over model j (column).
5.3 ENSEMBLE BASED METHOD
We find that attacking VGG networks can generate adversarial examples with high transferability in single model attack. Moreover, in the optimization-base approach, ensemble networks have been proved to be able to produce transferable adversarial examples.
We test the impact of ensemble transferability of difffeent networks. In our test, take off one of the four known networks each time, and the results are shown on Table 2
Since it is possible to generate large number of adversarial examples with high transferability in the distillation based method through ensemble models, we perform different parameter tests on this method.
5.4 THE EFFECT OF TEMPERATURE
Firstly, want to know how different distillation temperatures affect the attack success rate, and which is the best temperature to distill. We test different temperatures while the iteration number is 20 and the max disturb limit is 32.

---

Under review as a conference paper at ICLR 2020
TacV3
TnoV4
PGGT6
VGGIS
Resner_K0
Resnet_152
Tec_Res
AB
0.949
0.982
1.0
1.0
0.471
0.385
0.334
IneV3
0.991
1.0
IncV4
0.937
0.294
1.0
1.0
0.437
0.355
0.219
VGG16
0.803
VGG19
0.941
0.979
1.0
0.787
0.228
0.173
0.188
Table 2: The success rate of D-MIFGSM in ensemble based method. Cell (f,j) indicates that the accuracy of the targeted adversarial examples generated for the ensemble of the four models except model (row) is predicted as the target label by model j (column). In each line means the network behind is not used in ensemble attack.
Figure shows the effect of different distillation temperatures black-box targeted attack, where temperature equals means there's no distillation processing. clear that distillation can make effects both single model attack and ensemble based method. And the distillation temperature makes best effects when reaches about 16.
330
2.00
110
300
25
12.
ni 0
25
7.
18.
(a) Different Temperature in Single Models
(b) Different Temperature in Ensemble Models
Figure I: Effeect of Different Temperature
5.5 THE EFFECT OF THE ITERATION
After the test temperature, we want to know the effect the iteration number on the success rate of the attack in the distillation method. Dong et al. (2018a) suppose that the adversarial examples generated by iterative methods have a tendency to overfit, which means iteration is negative to the transferability of adversarial examples.
But in our method, as Figure shows, the more iterations, the much transferability we can get, until is large enough converge. another word, certain extent, avoid the overfit problem.
330
Geoger 113
Baizee 10)
Inreatin bogee
lnvert Seare
30
120
(a) Different Iterations in Single Models
(b) Different Iterations in Ensemble Models
Figure 2: Effect of Different Iterations
However, high iterations also mean long computing time. In our experiment, we suggest 30 to 50 iterations be better choice.

---

Under review as a conference paper at ICLR 2020
5.6
THE EFFECT OF THE DISTURB LIMIT
Rond
370
100
(a) Different Disturb Limit in Single Models
(b) Different Disturb Limit in Single Models
Figure 3: Effect of Different Disturb Limit
The distance between origin images and adversarial examples is also an important indicator to judge the performance of attacking methods. In both MIFGSM and D-MIFGSM, the distance depends the max disturb limit which is a hyperparameter attack processing. It is also an important task to find out the best disturb limit.
As Figure 3 shows, lower disturb causes lower attack success rate. But at the mean time, too big disturb limit also has negative effect the success of adversarial attack. On the other hand, bigger disturb means bigger noise, which is more likely to be realized by human. Above all, in our experiment, prefer choose the disturb limit between 16 and 32.
5.7 THE COMPARISON BETWEEN MIFGSM AND D-MIFGSM
We compare the attack success rate in both black-box and white-box attack scenarios between MIFGSM and our D-MIFGSM. The setup of the temperature is 16, the iterations is 20 and the disturb limit is 32.
Table 3 shows that in both two kinds of attack scenarios D-MIFGSM provides a significant boost to MI-FGSM.
Black-Box Resmet 152
White-Box
Resnet 50
Inc Res
IncV3
IncV
VGG16
VGG19
Single
MIFGSM
0.072
0.050
0.002
1.0
Ensemble
MIFGSM
0.201
0.154
0.170
0.944
0.789
0,993
0.988
Models
MIFGSM
0.471
0.385
0.334
0.982
1.0
1.0
0.949
Table 3: The success rate of MIFGSM and our D-MIFGSM. The single model attack VGG16 to generate adverarial examples. Ensemble based method use all four models to generate adverarial examples in 5.1
CONCLUSION
In this paper, we show that gradient based methods can also produce adversarial examples with high transfcrability. In order to generate adversarial examples with high transferability in targeted attack, we introduce knowledge distillation into gradient based methods. Our method can be easily imple- mented in any method based on fast gradient. In targeted attack, the ensemble of multiple networks and optimization based methods proved to produce adversarial examples with high transferability. But optimization-based methods are notoriously slow. We introduce knowledge distillation into the momentum iterative fast gradient method to produce adversarial examples of about 30% to 50% transferability in the experiment. At the same time, find out that the network structure also has an effect on transferability, and it will be our future work to explore neural network structures that can produce transferability.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL
Thomas G. Dietterich. Ensemble methods in machine learning. In Josef Kittler and Fabio Roli (eds.), Multiple Classifier Systems, First International Workshop, MCS 2000, Cagliari, Italy, June 21- 23, 2000, Proceedings, volume 1857 of Lecture Notes in Computer Science, pp. 1-15. Springer. 2000. ISBN 3-540-67704-6.
doi: 00.100773-540-4551449,, URL https://doi.org/100.000
3-540-45014-9_1
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boost- ing adversarial attacks with momentum. In 2018 IEEE Conference on Computer Vision and Pat- tern Recognition, CVPR 2018, Salt Lake City, UT. USA, June 18-22, 2018. pp. 9185-9193. IEEE Computer Society, 2018a. doi: 10. 109/CVPR 20. 2..0009777.070... URL http:///ppeaaccesstteeeee thecvf.com// content.cvpr 20tprr2211//ha 201m///m//Doogggggg Beegggooooootooiieeeee Adversarial Attacks CVPR 2018 1a1188pppeettmll
Yinpeng Dong, Fangzhou Liao, Tianyu Pang. Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boost- ing adversarial attacks with momentum. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018b.
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Bengio & LeCun (2115). URL http://aartw.orr//b////112220
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016- 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pp. 630-645. Springer, 2016. ISBN 978-3-319-46492-- 3-119778--331--44933
doi: 10.10077978-3331994450
38. URL https://dooi.or///0
Geoffrey F. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CoRR, abs/ 0B5/1500002511, 2015. URL http://arxxvvoor//bb////00 553.002531
Alexey Kurakin, lan J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net, 2017. URL. httpps://openeeview.neet forum HJGGUURRol
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial exam- ples and black-box attacks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017. URL httpss//ppeeeeeeeee.e///oomom//d
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openeeriiennee///mmm FJZlBfZAb
Nicolas Papernot, Patrick D. McDaniel, Ian J. Goodfellow, Somesh Jha, Z. Berkay Celik, and Anan- thram Swami. Practical black-box attacks against deep learning systems using adversarial exam- ples. CoRR, abs/1602.02697, 2016. URL http:///axxivoorrgaa//6600000.00
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Bengio & LeCunn (2015). URL http::/tt///arriioorgaa//////// 1009..1561166666
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Good- fellow, and Rob Fergus. Intriguing properties of neural networks. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference Learning Representations, ICLR 2014, Banff AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL BBLLhhttp//aariioor//////11111 6199.

---

Under review as a conference paper at ICLR 2020
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shiens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer
Vision and Pattern Recognition. CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2818- 2826. IEEE Computer Sociery, 2016. ISBN 978-1-4673-8851-11 doi: 10.1109/CVPR.20663000 URL http:///wo..rr..111100.////....
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A. Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Satinder P. Singh and
Shaul Markovitch (eds.), Proceedings of the Thirty-First AAAI Conference on Artificial Intelli- gence, Febrwary 4-9, 2017, San Francisco, California, USA., pp. 2278-488, AAAI Press, 2017.
URL http://aaiiaigcccco.m...//ll/A//
10


Reviews:
[{"review_id": "Hkl_08-49S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper suggests to use temperature scaling in adversarial attack design for improving transferability under black-box attack setting. Based on this, the paper proposes several new attacks: D-FGSM, D-MIFGSM, and their ensemble versions. Experimental results found that the proposed methods improves transferability from VGG networks, compared to the non-distillated counterparts. \n\nIn overall, I liked its novel motivation and simplicity of the method, but it seems to me the manuscript should be improved to meet the ICLR standard. Firstly, the presentation of the method is not that clear to me. The mathematical notations are quite confusing for me as most of them are used without any definitions. I am still not convinced that the arguments in Section 3.1 and 3.2 are indeed relevant to the actual practice of black-box adversarial attacks, which usually includes extremely non-smooth boundaries with multiple gradient steps. Even though the experiments show effectiveness partially on VGGNets, but the overall improvements are not sufficient for me to claim the general effectiveness of the method unless the paper could provide additional results on broader range of architectures and  threat models. \n\n- I feel Section 2.3 is too subjective with vague statements. The following statement was particularly unclear to me: \"The first problem with gradient based methods is that they lose their effectiveness after a certain number of iterations.\": Does the term \"effectiveness\" indicate some relative effectiveness compared to other methods, e.g. optimization-based attacks? Is this really a general phenomenon in gradient-based attacks? Also, please elaborate more on \"So, insufficient information acquisition for different categories and premature stop of gradient update are the reasons ...\"\n\n- Regarding that the softmax is the problem, one could try to directly minimize the logit layers skipping the softmax, i.e., gradient on logits? This is actually one of common techniques and there are many simple tricks in the context of adversarial attack, so the paper may include comparisons with such of tricks as well. \n\n- It is important to specify the exact threat model used throughout the experiments, e.g. perturbation constraints and attack details. Demonstrating the effectiveness on a variety of threat models could also strengthen the manuscript.\n\n- Table 1 and 2 may include other baseline (black-box attack) methods for comparison. This would much help to understand the method better. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BylnNnhiYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes distillation attacks to generate transferable targeted adversarial examples. The technique itself is pretty simple: instead of only using the raw logits L(x) to compute the cross entropy loss for optimization, they also use the distilled logits L(x)/T to generate adversarial examples. Their evaluation setup largely follows the style of Liu et al., but they construct a different subset of ILSVRC validation set, and some of the model architectures in their ensemble are different from Liu et al. Their results show that by including the distilled logits when computing the gradient, the generated adversarial examples can transfer better among different models using both single-model and ensemble-based attacks.\n\nI think their proposed attack is interesting due to its simplicity and effectiveness. However, I would like to see clarification of some evaluation details, as well as more experiments to compare with Liu et al.:\n\n1. To assess the effectiveness of targeted attacks, it is important to ensure that the semantic meaning of target label is far from the ground truth label. Some of the 1000 ImageNet labels have very similar meanings to each other, thus different choices of the target label would dramatically affect the difficulty of the attacks. In Liu et al., they manually inspect the image-target pairs to ensure that the target label is very different from the ground truth in its meaning. To enable a fair comparison, it would be helpful to provide results on the same image-target pairs constructed by Liu et al., which could be found in the public repo linked in their paper.\n\n2. For ensemble attacks, is including both the raw and the distilled logits crucial in obtaining a good performance? What is the performance of including distilled logits only? How do different values of \\lambda_1 and \\lambda_2 in (8) affect the attack performance?\n\n3. Could you visualize some generated adversarial examples, so that we can view the qualitative results?\n\n4. In general this paper lacks empirical analysis on why distillation helps improve the transferability. Some more discussion would be helpful.\n\n-------------\nPost-rebuttal comments\n\nThanks for your response! I think this paper still misses a more in-depth analysis, and thus I keep my original assessment.\n-------------", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeNLz1sFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an attack method to improve the transferability of targeted adversarial examples. The proposed method uses a temperature T to convert the logit of the network, and calculates the gradient based on the new logit, yielding the distillation-based attack method. It has been integrated into FGSM and MI-FGSM.\n\nOverall, this paper has the poor quality based on the writing, presentation, significance of the algorithm, insufficient experiments. The detailed comments are provided below.\n\n1. The writing of this paper is poor. There are a lot of typos in the paper. The notations are used without definitions. These make the paper hard to read and understand.\n\n2. Based on my understanding of the paper, the motivation of the proposed method is that the softmax function on top of neural networks can make the gradient unable to accurately penetrate classification boundaries. And the distillation-based method is proposed to reduce the magnitude of the logits to make the gradient more stable. However, if the argument were true, we could use the C&W loss to perform the attack, which is defined on the logit layer without affected by the softmax function.\n\n3. There are a lot of recent attack methods proposed to improve the transferability of adversarial example, e.g., \"Improving transferability of adversarial examples with input diversity\" (Xie et al., 2019); \"Evading defenses to transferable adversarial example by translation-invariant attacks\" (Dong et al., 2019). The authors are encouraged to compare the proposed methods with previous works.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an attack method to improve the transferability of adversarial examples under black-box attack settings.

Despite the simplicity of the proposed idea, reviewers and AC commonly think that the paper is far from being ready to publish in various aspects: (a) the presentation/writing quality, (b) in-depth analysis and (c) experimental results.

Hence, I recommend rejection.
Decision: Reject