Towards Disentangling Non-Robust and Robust Components in Performance Metric
Authors: Yujun Shi
Benben Liao
Guangyong Chen
Yun Liu
Ming-ming Cheng
Jiashi Feng
Venue:  (2020)
Keywords: adversarial examples
robust machine learning

Abstract: The vulnerability to slight input perturbations is a worrying yet intriguing property of deep neural networks (DNNs). Though some efforts have been devoted to investigating the reason behind such adversarial behavior, the relation between standard accuracy and adversarial behavior of DNNs is still little understood. In this work, we reveal such relation by first introducing a metric characterizing the standard performance of DNNs. Then we theoretically show this metric can be disentangled into an information-theoretic non-robust component that is related to adversarial behavior, and a robust component. Then, we show by experiments that DNNs under standard training rely heavily on optimizing the non-robust component in achieving decent performance. We also demonstrate current state-of-the-art adversarial training algorithms indeed try to robustify DNNs by preventing them from using the non-robust component to distinguish samples from different categories. Based on our findings, we take a step forward and point out the possible direction of simultaneously achieving decent standard generalization and adversarial robustness. It is hoped that our theory can further inspire the community to make more interesting discoveries about the relation between standard accuracy and adversarial robustness of DNNs.

Under review as a conference paper at ICLR 2020
TOWARDS DISENTANGLING NON-ROBUST AND Ro- BUST COMPONENTS IN PERFORMANCE METRIC
Anonymous authors Paper under double-blind review
ABSTRACT
The vulnerability to slight input perturbations is a worrying yet intriguing property of deep neural networks (DNNs). Though some efforts have been devoted investigating the reason behind such adversarial behavior, the relation between standard accuracy and adversarial behavior of DNNs is still little understood. In this work, we reveal such relation by first introducing a metric characterizing the standard performance of DNNs. Then we theoretically show this metric can be disentangled into an informatioonthhoreed to adversarial behavior, and robust component. Then, we show by experiments that DNNs under standard training rely heavily on optimizing the non-robust component in achieving decent performance. We also demonstrate current state-of- the-art adversarial training algorithms indeed try to robustify DNNs by preventing them from using the non-robust component to distinguish samples from different categories. Based possible direction of simultaneously achieving decent standard generalization and adversarial robustness. It is hoped that our theory further inspire the community to make more interesting discoveries about the relation between standard accuracy and adversarial robustness of DNNs.
non-robust component that is related
findings, take step forward and point out the
INTRODUCTION
Deep neural networks (DNNs) have achieved wide success over the last decade. In literature, the majority of deep learning models pursue boosted performance from different aspects (Krizhessky et al. 2012 Simonyan & Zisserman 2014 Szegedy et al., 2015 He et al.) 2016 2015 loffe & Szegedy 2015). However,
found these powerful models are susceptible to perturbations, even those imperceptible to humans (Szegedy et al. 2013). Therefore beyond the main research stream, there are also works devoted to investigating effective attacks (Goodfellow et al., 2014 Moosavi Dezfooli al., 2016 Kurakin et 2016 Zhao et al.) 2018) and designing adversarial robust models (Goodfellow et al. 2014 Miyato et al. 2015 Madry et al. 2017 Zhang et al.) 2019).
It is observed that the adversarial robustness is usually achieved at the cost of non-trivial degradation of standard performance. Previous efforts (Schmidt et af.] [0018] Tsipras et al.) 2018 Nakkiran 2019) trying to understand this phenomenon are usually based on simple toy models or heavy assumptions, and do not provide a general theoretic framework that explicitly shows how adversarial robustness is related standard generalization. The work (Iyys et al., 2019) empirically shows the adversarial samples might be human imperceptible features that could help generalization, but it does not provide any general theoretic framework properly explain this phenomenon. In another work (Zhang et al. 2019), though bound on the gap between standard accuracy and adversarial accuracy is derived so that the gap between these two quantities could be explicitly controlled, the underlying reason of such trade-off is still not fully characterized.
In this work, we start from a new perspective and consider the performance of deep classification models with new metric Cross Category Kullback-Leibler divergence (CCKL), namely the Kull- back-Leibles (KL) divergence between the model's output distributions over input data from different categories, instead of the traditional metric of accuracy. Interestingly, by applying Taylor expansion on CCKL, we show that it can be disentangled into a lower order non-robust component, which is related to causing adversarial behavior, and higher order robust component. By applying such

---

Under review as a conference paper at ICLR 2020
disentanglement, are able to reveal the relation between standard generalization and adversarial robustness of the deep learning model in a relatively general setting.
Furthermore, demonstrate by experiments that current deep learning models rely heavily optimizing the lower order non-robust component to generalize, which is a major underlying reason for the adversarial behavior. We also show the state-of-the-ar adversarial training algorithms in fact trying to constrain the model from using the lower order non-robust component to discriminate data of different categories. Based on these findings, we claim that enabling the model to rely more on the higher order robust component instead of the adversary-prons lower order component might be the key achieving decent standard accuracy and adversarial robustness simultaneously.
all
Our contributions are summarized as follows:
We propose a new metric, Cross Category Kullback- divergence (CCKL), to char- acterize the standard performance. This metric be more naturally connected with the adversarial behavior of DNNs than the traditional metric of accuracy.
By applying simple Taylor expansion on CCKL, we theoretically reveal the relation between standard generalization and adversarial behavior DNNs general way without relying on toy models.
Based on the above novelties, we take a further step and point out the possible direction for simultaneously achieving decent standard accuracy and adversarial robustness.
2
RELATED WORK
To study the adversarial behavior of DNNs, gradient-base algorithms Goodfellow et al., | Papernot
Adversarial Attack and Defense
et al. 2016a; Moosavi-Dezfooli et al. 2016 Carlini & Wagnne 2017 Kurakin et al. 2016 Zhao et al. 2018) are developed to find the most effective adversarial perturbations, and on the other hand, model based defense algorithms (Papernot et al. 2016b; Madry et al., 2017 Miyato et al., 2015 Zhang al. (019)) also explored for enhancing model robustness against the adversarial perturbations. For example, in (Miyato et al.] [211]] Zhang al. 2019), the model is regularized by the KI. divergenc between outputs of clean and adversarial samples, which is also adopted in our theoretic framework.
Fisher Information Deep Learning Fisher information is used as a tool to study machine learning topics. Many previous works (Chaudhry et al. 2018 Pascanu & Bengio 2013 Desjardins et al. 2015 Andrychowicz et al. 2016 Achille et 2019) view the parameters of the model as those of the Fisher information, while some other (Miyato et al. 2015 Zhao al. (0188) take the model's input as the parameters of Fisher information, making it applicable in the context of adversarial
robustness. We adopt the same definition for Fisher information as (Miyato al. 2015 Zhao et al.. 2018) in our later analysis.
Explaining Adversarial Behavior of DNNs Previous works trying to explain the robust model's degradation standard performance (Tsipras al. 2018 Schmidt et al., 2018 Nakkiran 2019) are mostly based on some simple toy models that cannot generalize to real world settings. Recently, (Iyas et al. 2019) empirically shows that adversarial noise actually features that can help the generalization of DNNs. However, this work does not provide general theoretic framework to explain this phenomenon. ZZhag et al. (20T9) provides bound that characterizes the gap between standard accuracy and adversarial robustness, but they do not clearly explain the underlying reason of such trade-off. our work, we analyze the underlying reason for the adversarial behavior of DNNs by directly disentangling the proposed CCKL. In this way, not only theoretically demonstrate the reasoning behind the trade-off between adversarial robustness and standard generalization, but also point out the possible direction for achieving both objectives simultaneously.
NEW PERFORMANCE METRIC AND ITS DISENTANGLEMENT
In this section, we first propose to consider and measure the performance deep classification models from the perspective of CCKL instead of accuracy. Then we explain finding that the performance
2

---

Under review as a conference paper at ICLR 2020
of DNN model is determined by two components that are disentangled from the obtained objective based on KL divergence, one of which related to the model's ' adversarial behavior and the other not. After that, to help understand our idea, we provide some complementary understandings from the viewpoint of information geometry.
3.1 PROPOSED PERFORMANCE METRIC
Normally, DNNs are trained by minimizing the following cross-entropy loss:
L = (tt=a))->xxx(KK(eelfff
(I)
where X' x } is the distribution where the data-label pair (x, y) is sampled from and f(x is the prediction of the model. Note that y and f(x) are both discrete distribution categories here. Correspondingly, the prediction accuracy is adopted to measure the performance of DNN model. However, such metric hides the connection between the model performance and its adversarial behavior. Since adversarial attack is concerned with changing the prediction of the model on data point to another, we propose shift the performance metric from investigating one single data point to comparing the data point with its counterparts from other categories, which can help reveal the above connection. Therefore, we propose the following Cross Category KL Divergence (CCKL) metric:
CCKL BKccaalllesssaeee/////////alll
where S is defined
(((1)) Wa), w;)) Vi,j N, + My Tz. 25 E X3.
With the definition above, we can derive the following theorem:
Theorem 3.1 Denoting (pll)) as the Jensen-Shunnon divergence between two distributions p and have
CCKL 2 (e
CoL,
(4)
where I, is the cross-entropy loss defined in Equation 7 and col. (KT( the co-object for L..
We defer the proof and more comments of Theorem 3.1 to Appendix A due to space limit.
Taking closer look at Theorem 3.1
know that (,),,,,
[y;)] remains unchanged
during training, and L and C are supposed to decrease when the model performance increases. Therefore, the lower bound on CCKL is supposed to increase with the rise of model performance, which will asymptotically result in the increase of CCKL. Thus, the CCKL, which characterizes how well the model discriminates input data from different categories, can serve as performance metric.
To further expound on our point, we provide visualization of how CCKL evolves with test accuracy According to our results, the variation of
and test cross entropy loss during training Figure 2 CCKL correlates well with the test accuracy and test cross entropy loss during training.
3.2 CONNECTIONS BETWEEN ADVERSARIAL BEHAVIOR AND CCKL
We now explain how the above CCKL objective 2 correlates with the model's adversarial behavior. Consider single data-label pair (x, y)- The tollowing cross entropy loss of the model fo adversarial samples (with perturbation 17):
Lo(z, 7) (L(ylff n)) s.t. || <c
(5)
is adopted to study the adversarial behavior in most previous literature (Goodfellow et al. 2014 Kurakin et al.) 2016, Madry et al. 2017). However, it is not easy to build connection between this objective and the standard generalization of the model. Interestingly, in some recent works (Miyato et al. 2015| Zhao al., 2018 Zhang et al. 2019), another objective is adopted to characterize the model's 's adversarial behavior:
La(æ, m) KL(
+n) s.t. ||m|| S.
Based on the above objective ( (Zhao et al. 2018) reports sfate-of-the-as results in the task of adversarial attack and (Zhang al. 2019) reports state-of-the-art results in adversarial robustness.
3

---

Under review as a conference paper at ICLR 2020
X1 Y/1
(x)
Strained (x,,
X2
Y2
fini((2)
fraanedd (x²)
Figure 1: Illustration on how output distributions f(x ) and (x22 evolve during training. Column 2: ground-truth label. Column 3: network outputs at initialization. Column 4: network outputs when trained to converge. The value of (ffaa (21)) (21) fiat (22)) in the graph is 0.053, and the value of K [ftrammed (itrraameeeiillffarraeeaa (rraseed is 3.197.
test entropy loss
test accuracy
kl diverrencce
123
25
10
35 is
12
8
training epoch
training epoch
training epoch
Figure 2: Visualization of how accuracy (left), cross entropy loss (middle) and CCKL (right) test set evolve when training VGGI3 on CIFAR-10. It is clear that CCKL well indicates test accuracy and test cross entropy loss. For more experiment results on relation between CCKL and cross entropy loss, please refer to Appendix B
More importantly, as will be shown later in the paper, the adversarial objective (6) can be easily linked to the standard generalization the model. Therefore, we adopt the objective (6) the rest of the paper. The corresponding adversarial training objective is thus formulated as follows:
min max Lo 17) s.t. ||>/|| S.
"
Given the definition of Lo(x, 1),, applying Taylor expansion yields the following:
1 "" F.n + Σ (n) s.t. |i S.
max LL 7) = max "
(8)
where Fz is the Fisher information of f() w.r.t. x and ((()) is the following k-th order Taylor expansion term. Let f() (x) be the j-th entry f()) and n be the dimension of f fxx) Then F, can be calculated by
F. [ log (o)))))))
log (o)
j=0
When is sufficiently small, the higher order terms in the above would vanish and Equation could be simplified into
max (o 1) max rj Fan s.t. I| Se. " We could therefore construct the lagrangian of the above problem:
(10)
L(x, m) 2 Fun 2 A(1 2))
(11)
By setting V Lx, 17) = O, we obtain F, = >max where Amax is the maximum eigenvalue F,. Therefore, the most effective adversarial perturbation corresponds to the leading eigenvector of Fz.
4

---

Under review as a conference paper at ICLR 2020
Consequently, have
max Lo 2 Amaaxx..
(12)
7
Note that Amax here is also the spectral norm of the Fisher information matrix Fz. The above theory shows that the local adversarial behavior of the model f around the input determined by the spectral norm of the Fisher information matrix: the adversarial behavior around r would be more severe if the spectral norm of Fz is larger.
Uy)) randomly sampled from S. We Now we analyze CCKL by considering one tuple ((z4, (z)) rewrite (L(((=)))
(L(f(z)) (L(f(=)))f(z; + (z, xx))) = Lo(d;, 2, - =x;)-
(13)
Then, we apply the same Taylor expansion as above and obtain
(I() = Lo(os, Ty -x;) [x) - x;) x3) Fr, (og - 3z) +
T (xy - -ii). (14)
2
Comparing Equation 14 and Equation (10 we notice they share the same Fisher information Fz, at data point z4. Here, we show the adversarial behavior each data point can be connected to the performance objective CCKL via Fisher information, which lays the foundation for the
disentanglement the next subsection.
3.3 DISENTANGLEMEN OF CCKL
As mentioned above, we build the connection between adversarial behavior and CCKL. We here explain how the non-robust and robust component in CCKL can be disentangled. We first denote
(x, 24 Fr, (x, 24) in Equation (14) as G1 and the following terms Σk= T2. (xz x;) as G2. Thus KL(f(Z)) can be formulated as
KL(S(>))))(ss))
Gi + G2.
(15)
Taking a closer look at Equation (15), we notice the increase of G1 and G2 can both contribute to the rise of K (f=))l)) which contribute to CCKL. In addition, since Xmaz is the maximum eigen value of PR.. we have the following bound:
|7 x
|
Amax ||
Fs:
(a;
2z)) -G
(16)
||
| - x3
With the above bound, we know that the rise of G1 would asymptotically result in the rise of the norm Fs,, which corresponds to the adversarial behavior.
Therefore, if the model relies heavily on the increase of G1 to boost performance, the norm of Fz, would also increase drastically. According to Equation (12) and our analysis above, the rise in the The trade-off between standard
norm of F2; means more severe adversarial behavior around performance and adversarial behavior is thus characterized as follows: the model can rely on G to boost performance, but it comes with the side effect of more severe adversarial behavior; the other hand, G2 contributes to the CCKL but not involved in the adversarial objective, thus would not cause adversarial behavior rely on terms in G2 to distinguish n, from data belonging to other categories. In this way, we can disentangle the non-robust and robust component in the overall performance objective CCKL.
To further understand the role of Gj in classification, we visualize how F-norm of Fisher information evolves during training. We visualize F-norm instead of spectral norm because all norms equivalent and spectral norm is not computationally feasible in our case. We first empirically show how the average F-norm of Fisher information and standard accuracy of VGG13 model on CIFAR- 10 test set vary during normal training in Figure 4 We observe that the norm of Fisher information increases drastically with the rise of accuracy, which indicates DNNs rely heavily on the non-robust component G to boost performance.
Note that (a,- 214) is the difference between two input data instead of small perturbation, the higher order terms would not vanish here.
5

---

Under review as a conference paper at ICLR 2020
Log Average Fisher F-norm
67-
8.5
Baseline Madry PGD
Baseline Madry PGD
TRADES
TRADES
61
10
50
10
40
50
training epoch
training epoch
Figure 3: Visualization of how standard test accuracy (left) and average F-norm (right) of Fisher Information Matrix on test set vary during normal training and adversarial training with VGG13 on CIFAR-10 10 (for the same experiments on ResNet, see Appendix D). We take the nature logarithm to better visualize the average F-norm of Fisher information.
Then we compare normal training with the two state-of-the-art adversarial training algorithms (Madry al. 2017 Zhang al. 2019) using the same visualization method, and show results in Figure It is clear that during adversarial training. although the Fisher information's 's aver- age F-norm also rises with standard accuracy, value is significantly smaller than its counterpart during normal training. That is, the adversar- ial training process effectively constrain the model from relying on Fisher information to boost performance.
Log Average Fisher F-norm
accuracy
02-
02
training epoch
Our experiments reveal the widely known but seldom understood fact that the standard accu- racy of models under the two adversarial train- ing algorithms are significantly lower than that under normal training. According to our theory, can be explained these models trained over adversarial samples are unable to effectively rely on adversary- prone non-robust components such as Fisher information to distinguish the input data from those belonging to other categories.
Figure 4: Visualization of how standard accuracy and average F-norm of Fisher information matrix test vary during normal training. The periment is conducted with a VGG13 model and CIFAR-10 10 data set (for the same experiment on ResNet, see Appendix D). We take the nature log- arithm better visualize the average F-norm of Fisher information.
Till now, theoretically and empirically demonstrate the relation between standard performance and adversarial robustness using the proposed disentanglement theory.
3.4 EXPLANATION FROM GEOMETRY POINT VIEW
We provide some explanations on the above findings from the information geometry viewpoint. We note the model doing maximum likelihood estimation by learning to fit the label distribution over training data. This can be viewed as a process of the log-likelihood landscape of the model on input data gradually transiting into state where the model can well distinguish data of different categories. Since the training data are sparsely sampled from the whole distribution, the smoothness prior does not hold during the formation of the model's log-likelihood landscape. Consequently, the model tends to use lower order local geometric descriptors such as Fisher information -the local curvature of the log-likelln likelihood landscape to form an overly simplified adversary -ponne log- liglliillhood landscape. When applying adversarial training, strong smoothness constraint is enforced and the model would have to rely on higher-order global geometric descriptors that vanish locally to form the whole landscape. Thus the landscape would be more robust adversarial perturbations.

---

Under review as a conference paper at ICLR 2020
Table 1: Standard accuracy and adversarial robustness of models of different capacity trained by ZZang et al. 2019). We set the maximum allowed L.o norm of attack noise = 8/ 8/255. The learning rate is 0.01 for all VGG13 models and 0.1 for all resnet models. All models are trained with SGD for 160 epochs with a decay of 10x in learning rate at 80th and 120th epochs. The SGD's momentum 0.9. For adversarial training settings, the coefficient for the regularization term in (Zhang et al.. 2019 is x 5.0, the step size for projected gradient descent is 2/255 and the number of step is 10. The weight decay during training is le-4. For evaluation against adversarial attack, the step size for PGD attack is 2/255, the number of steps is 20. The CW attack objective's coefficient is 5e2 and also solved by PGD with the same optimization parameters. All experiments are conducted on NVIDIA Tesla V100 GPUs.
VGG13 pure linear
VGG13 half linear
VGG13 normal
resnet20x1
resnet32x10 10
accuracy (standard)
35.14
70.48
72.36
72.88
79.45
accuracy (PGD attack, c = 8/255)
17.06
38.52
42.29
44.94
49.52
accuracy (CW attack, = 8/255)
13.94
34.12
38.96
41.49
47.67
3.5 ANALYSIS FOR e NOT SUFFICIENTLY SMALL
Researchers have found that the local linearity does not hold when the norm of perturbation is allowed to be relatively larger (Kurakin et al. 2016; Madry et al. (017). In this case, some higher order terms may also involve in the adversarial objective. However, since the norm of perturbation is still considered to be small, the Taylor expansion terms of K (=(f)))]f(() + n)) would still vanish up to certain order K.. Therefore, the adversarial objective can be rewritten as
max Lo Σ (n) s.t. ||l|| Se.
(17)
Comparing (14) and (17). we know that the functional of the first K, - terms could still jointly
connect adversarial objective and CCKL. Also. we conduct numeric experiments in Appendix C to show that the Fisher induced term is the dominant component in [f)((=)) 7)) within practical range of perturbation. We leave more detailed theoretic analysis on higher order terms involved in the adversarial objective to future work.
TOWARDS SIMULTANBOUS GOOD PERFORMANCE AND ROBUSTNESS
With the disentanglement theory introduced above, we now consider how to achieve simultancous decent standard accuracy and adversarial robustness. According our analysis, if relying on the robust component alone can effectively distinguish data of different categories, then obtaining an adversarial robust model with high standard accuracy is possible.
Since the expansion terms in the robust component are all higher order terms, we claim that the key to achieving the two desired qualities simultaneously that it would be able to rely more on higher order terms for prediction. In this way, the model would not have to rely heavily on Fisher information and still have decent standard performance with higher order terms (the robust component). Our disentanglement provides thcoretical justification for the importance of model capacity in achieving adversarial robustness and decent standard performance.
increase the expressive power of the model so
We experiment on CIFAR-10 to validate our strategy of achieving the two objectives simultaneously. We train the following models with TRADES algorithm ZZang et al., 2019): a pure linear VGG13 model without ReLU and with average pooling, half linear VGG13 model without ReLU but using max pooling, normal VGG 13 model, a normal resnet20 model and a resnet32 model with 10x more channels. We evaluate these models standard samples and adversarial samples produced

---

Under review as a conference paper at ICLR 2020
030
rene
02204
Figure 5: We sampled 25896 pairs of images (æi, 2ÿ) in CIFAR-IO test set (o; and z, belong to different categories) and visualized the distribution of K L(((aa)) (left), distribution of G2 (middle), and distribution Gz/ (()))L(()))lll( (right) resnet20x and resnet32x10.
by Loe PGD attack (Madry et al. 2017) and CW attack (Carlini & Wagner 2017). The results provided in Table
We first compare VGG models. Since adversarial training impedes the model from using the adversary-prone lower order terms to discriminate input data, and the pure linear model is not capable of utilizing higher order information, the standard generalization and adversarial robustness of the pure linear model are both very poor. However, as the capacity of the model increases (half linear and normal VGG13 models). the performance on standard and adversarial samples improves simultancously. For resnet models, when the model is shallower and narrower (resnet20x1) I), the performance in both standard and adversarial settings is relatively low. However, with a deeper and wider resnet model (resnet32x1 10), more higher order information is explored for prediction, so the performance on standard samples and adversarial samples increases significantly.
In addition, we sampled 25896 pairs of images (z,, 2j) in CIFAR-10 test set (z; and zj belong to different categories) and visualized the distribution of (L(f(=))l/<;;))) G2, and with the above mentioned adversarially trained resnet20x and resnet32x10
G2/
model in Figure 5
According our results in Figure 5 we have the following observations. 1) The value of CCKL of the resnet32x10 model on CIFAR-10 test set is much higher than that resnet20x1, which means the resnet32x10 can better distinguish input data from different categories. 2) The value of G2 of resnet32x10 on the CIFAR-10 10 test set is significantly higher than that of resmet20x which shows G2 plays a much more important role in resnet32x10 10 than resnet20x 1. 3) The ratio of G2/ K (==))))) of resnet32x10 on CIFAR-10 10 test set is much higher than that of resnet20x1, meaning the relative importance of G2 of resnet32x10 10 is much greater than that of resnet20x1.
These observations validate our claim that a model with better capacity can more effectively leverage higher order information achieve higher standard accuracy and also adversarial robustness under adversarial training.
5 CONCLUSION
In this work, we provide a novel viewpoint on the relation of standard accuracy and adversarial robustness of deep learning models. We propose a new CCKL metric measure the model perfor- mance instead of accuracy. With CCKL, the overall performance objective can be disentangled into an adversary-prone non-robust component, and a robust component. Based on such disentanglement, we then claim that for achieving both adversarial robustness and decent standard accuracy, a DNN model should rely more on the robust component to generalize. Our findings are well validated. In the future, we will dig deeper along the geometric properties of the log- likelihooood landscape formed by DNNs on input data, and try to better characterize the relation between standard accuracy and adversarial robustness.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji. Charless Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. arXiv preprint (tt.:::9902.0054444 2019.
Marein Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Plau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in Neural Information Processing Systems, pp. 3981-3989, 2016.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), pp. 39-57. IEEE, 2017.
Arslan Chaudhry, Puneet Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings the European Conference on Computer Vision (ECCV), pp. 532-547, 2018.
Guillaume Desjardins, Karen Simonyan, Razvan Pascanu, et al. Natural neural networks. In Advances in Neural Information Processing Systems, pp. 2071-2079, 2015.
Dominik Maria Endres and Johannes E Schindelin. A new metric for probability distributions. IEEE Transactions on Information theory, 2003.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6577 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEFE international conference on computer vision, pp. 1026-10344 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are bugs, they are features. arXiv:1905.02177. 00t.:1995.0215,,0 2019.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: arXiv:/502.033 1502.03167, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105. 2012.
Alexey Kurakin, lan Goodfellow, and Samy Bengio. Adversarial machine learning scale. arXiv preprint arXiv:1611.01236, 2016.
Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1):145-151, 1991.
Aleksander Madry. Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1706.060836 2017.
Takeru Miyato. Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. Distributional smoothing with virtual adversarial training, arXiv preprint arXiv:1507.0067. 2015.
Seyed-Mohsen DeosavviDezfoool, Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2574-2582, 2016.
Preetum Nakkiran. Adversarial robustness may be at odds with simplicity. arXiv preprint 000532, 2019.

---

Under review as a conference paper at ICLR 2020
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European Symposium Security and Privacy (EuroS&P), pp. 372-387. IEEE, 2016a.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation defense to adversarial perturbations against deep neural networks, In 2016 IEEE Symposium on Security and Privacy (SP), pp. 582-597. IEEE, 2016b.
Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. arXiv preprint arXiv:1301.3584, 2013.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adver- sarially robust generalization requires more data. In Advances in Neural Information Processing Systems, pp- 5014-5024 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXivv1409.1556 2014.
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties neural networks. arXiv preprint 131.6693,,, 2013.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2016. URL oz httpgggabs/11122 051.100662100
http:://axxvvvvo org
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry, Robustness may be odds with accuracy. stat, 1050: 11, 2018.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael I Jordan. Theoretically principled trade-off between robustness and accuracy. arXiv preprint arXiv:1901.08572 2019.
Chenxiao Zhao, P Thomas Fletcher, Mixue Yu, Yaxin Peng, Guixu Zhang, and Chaomin Shen. The ad- versarial attack and detection under the fisher information metric. arXiv preprint arXiv:1810.088 2018.
10

---

Under review as a conference paper at ICLR 2020
A PROOF AND COMMENTS OF THEOREM 3.1
In practical scenario, one cannot be too confident about a label y due to human annotation errors, and often replaces it during training with smoothed version y using the label smoothing technique Szegedy et al. (2016). Namely,
y ay + (1
K
where K is the number of classes, and [0, 1] is the confidence level. The label smoothing version of Theorem 3.1 still holds.
Proof of Theorem
data-label pair (1),))), (1z))))), where ys + yı, since JS1/2 satisfies triangular inequality (Endres
& Schindelin 2003 we have:
2 (f
(18)
2 JSu (1,)
Also, we have the inequality:
+2). (a
(19)
With
18 and
19
we have:
2 (applying (18)
(applying
19
(20)
2 JS((yll)))
In addition, we have Lin's incquality Lin 1991):
JS(plla) s ((Lll)) + ((lll)))
(21)
Apply
20
we have:
(S(nillu))
s +0US1teellffe))---00o/4)4))-----
(22)
+ [[ulll((aa)+
+ + K (z(((x))))))
Taking the expectation over the data set in 22 we have:
CCKL 2 Ev,, [J (g)
(23)
which proves the theorem 3.1
B ADDITIONAL RESULTS ON CCKL AS PERFORMANCE METRIC
We further empirically validate the relation between CCKL and standard cross entropy loss on test set in Figure. 6 We trained a ResNet-20 on CIFAR-10 and CIFAR-100. The Pearson Correlation Coefficients of CCKL and cross entropy loss on test set across the whole training period are -0.6744 and -0,8018 in these settings, which further justify our use of CCKL as performance metric.
C APPROXIMATE KL DIVERGENCE LOCALLY WITH FISHER
Experiment in Tab. shows the relative numerical enror of approximating KL locally with Fisher induced term with VGG13 model adversarially trained by (Zhang et al. 2019) on CIFAR-10 10. We observe that when is within practical range (S 8/255), the relative numeric emor is small.
11

---

Under review as a conference paper at ICLR 2020
Test Cross Entropy Loss
Test Cross Entropy Loss
2.-
113
CCKL
3.01
30
20
26
48 Training Epach
Training Epoch
Figure 6: relation between CCKL and test loss of resnet20 on CIFAR-I0 and CIFAR-100
Table 2:
and G1
(We denote G1 1/1
F2 here.)
1/255
2/255
3/255
4/255
5/255
6/255
7/255
8/255
KL - Gil/KL
0.058
0.098
0.135
0.170
0.202
0.234
0.263
0.290
D MORE EXPERIMENTS ON THE ROLE OF FISHER INFORMATION
We conduct more visualization experiments about the role of Fisher information in standard perfor- mance of DNN. The results are shown in Figure 7 and Figure The experiments are conducted on resnet20 model. The same conclusion could be drawn according to statistics.
Log Average Fisher F-norm
0.8
accuracy
0.6
0.5
0.3
0.1
10
training epoch
Figure 7: Visualization how standard accuracy and average F-norm of Fisher information matrix on test set vary during training. The experiment is conducted on resnet20 model and CIFAR-10 10 data set. We take the nature logarithm better visualize the average F-norm of Fisher information
Also, to better understand the geometry of adversarially trained model around robust samples and non-robust samples, we visualized the F-norm of Fisher information at correct and robust samples and correct but robust samples of VGG13 model. According to results in Figure [9] could see that although the overall Fisher information is small, the Fisher information around robust samples still significantly smaller than that of non-robust samples.
12

---

Under review as a conference paper at ICLR 2020
69-
Log Average Fisher F-norm
6.
Baseline Madry PGD
TRADES
Baseline Madry PGD
TRADES
61
10
30
10
50
training epoch
training epoch
Figure 8: Visualization of how (left) standard test accuracy and (right) average F-norm of Fisher Information Matrix on test set vary during normal training and adversarial training, with resmet20 on CIFAR-10. 10. We take the nature logarithm to better visualize the average F-norm of Fisher information
4.0
robust
3.5
non-robust
3.0
2.5
2.0
1.5
1.0
0.5
0.0
0.0
0.5
1.0
1.5
2.0
Figure Distribution of F-norm of Fisher information robust and non-robust samples on CIFAR-10
test set.
E INTERPRETATION FROM CRAMÉR-RAO BOUND POINT VIEW
According to the main text, adversarial training constrains the input-output Fisher information of DNN model. This constrain is a criteria of good DNN model due to the following reasons. Recall the well-known Cramér-Rao bound
var (a) 21
says that if we try to use the output probability f(x) to statistics è to reconstruct the input æ, the uncertainty in terms of variance var(i) bounded below by the inverse of Fisher information F.- For DNN model that represents the reality, when it classifies an image with a correct label, say dog, the label does not have any information about the environments what color the dog is, where is the dog, adversarial perturbation, etc. Therefore, one cannot use the information contained in the label reconstruct the original image. This means that the variance var(f) of any statistics & derived from output distribution f(x) is relatively large for a good DNN model. In view of Cramer-Rao bound, this implies that the Fisher information DNN relatively small value.
13


Reviews:
[{"review_id": "SkevaUL15S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is built around the theme of adversarial attacks. The papers titled \"Towards...\" are usually records of failed attempts of an attack on a grand challenge. The reviewer generally believes, that such records can be of a value.\nIn case of this paper an alternative loss function is proposed, based on information theory. The authors claim and support their claims with graphs that this loss tracks the accuracy much more faithfully than the \"standard\" cross entropy. A natural question would be therefore - what happens if we use this new measure as a training loss? Would it lead to more adversarially-robust models? Should evidence for that be supported for that the reviewer would be strongly in favour of accepting the paper. In the present shape the reviewer does not see why such measure should be of an interest to community. The impact of \"inspiring the community to make interesting discoveries\" is in the reviewers opinion not sufficient to justify publication at a venue of such importance as ICLF.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1ea-6ssFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new performance metric, CCKL, as an alternative to standard cross-entropy. At a high level the goal of this objective is to ensure that the classifier is confidently separating samples from different classes. They decompose this objective into its Taylor expansion terms and argue that, since adversarial vulnerability is a local property, it only depends on the first of these terms. They finally propose training models of higher capacity as a way of relying more on higher-order terms of the objective and hence improving adversarial robustness.\n\nI found the core idea potentially interesting. It is an attempt at rigorously studying the tension between local stability (necessary for robustness) and the fact that different classes should be confidently separated. However, I have several concerns regarding the theoretical derivation and experimental evaluation.\n\nFirst of all, I did not find the proposed CCKL objective well motivated. Theorem 3.1 only shows that standard training improves a _lower bound_ on the proposed objective. However, this only implies that this objective is being optimized as a byproduct of standard cross-entropy optimization. It is unclear for instance if this is a good performance indicator or whether training with this objective will even lead to a good classifier (for instance the classifier could learn a permutation of the labels instead). Therefore, I do not find the theoretical and experimental evidence in favor of studying this proposed objective convincing.\n\nMoreover, the experimental evidence presented is quite weak. While the authors compare various quantities related to standard and robust models, they do not perform the ablations necessary to understand whether these properties are causally related to robustness of simply a byproduct of the specific training scheme.\n\nFinally, the conclusion that larger capacity can improve both model accuracy and robustness has already been made in prior work (Kurakin et al. 2016, Madry et al. 2017). The intuition that added capacity helps by allowing the model to leverage higher order terms of the loss is interesting but a rather incremental contribution.\n\nOverall, while the ideas presented in the paper could be of interest, they are not rigorously established and rather serve as high-level intuition at this moment. Moreover, the writing of the paper could be significantly improved. It took me quite some time to distill the above intuition despite the fact that it is a rather elementary consequence of the Taylor expansion (I am not sure what the discussion about information geometry adds to this work). I would thus encourage the authors to continue working on these ideas and their presentation. However, for now, I will have to recommend rejection.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "B1x0RyRZcB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work explores the problem of robustness and adversarial\nattacks in NN. In a multiclass prediction setting the idea\nis to use a taylor expansion of a loss coined CCKL which\nis the KL divergence between predictions for pairs of samples\nfrom different classes.\n\nThe papers seems to find a convoluted route to arrive\nto something like this: when the Fisher information matrix\nhas a strong eigenvalue the model is not robust. In other words\nit says that if the landscape close to convergence has\nvalleys, or fast changes, the model is not robust.\nThis appears quite obvious and related to previous similar\nstudies.\n\nThis statement is then empirically evaluated on CIFAR-10.\n\nThe mathematical derivations should be made more rigorous.\nFor example the paragraph on Cramer-Rao bound is very handwavy.\n\nTypos\n\n-  is found these  ->  is found that these ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers suggest rejection. Beyond that, the more knowledgable two have consistent questions about the motivation for using the CCKL objective. As such, the exposition of this paper, and justification of the work could use improvement, so that experienced reviewers understand the contributions of the paper.
Decision: Reject