Towards Disentangling Non-Robust and Robust Components in Performance Metric

The vulnerability to slight input perturbations is a worrying yet intriguing property of deep neural networks ( DNNs ) . Though some efforts have been devoted to investigating the reason behind such adversarial behavior , the relation between standard accuracy and adversarial behavior of DNNs is still little understood . In this work , we reveal such relation by first introducing a metric characterizing the standard performance of DNNs . Then we theoretically show this metric can be disentangled into an information-theoretic non-robust component that is related to adversarial behavior , and a robust component . Then , we show by experiments that DNNs under standard training rely heavily on optimizing the non-robust component in achieving decent performance . We also demonstrate current state-ofthe-art adversarial training algorithms indeed try to robustify DNNs by preventing them from using the non-robust component to distinguish samples from different categories . Based on our findings , we take a step forward and point out the possible direction of simultaneously achieving decent standard generalization and adversarial robustness . It is hoped that our theory can further inspire the community to make more interesting discoveries about the relation between standard accuracy and adversarial robustness of DNNs . 1 INTRODUCTION . Deep neural networks ( DNNs ) have achieved wide success over the last decade . In literature , the majority of deep learning models pursue boosted performance from different aspects ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2014 ; Szegedy et al. , 2015 ; He et al. , 2016 ; 2015 ; Ioffe & Szegedy , 2015 ) . However , it is found these powerful models are susceptible to perturbations , even those imperceptible to humans ( Szegedy et al. , 2013 ) . Therefore beyond the main research stream , there are also works devoted to investigating effective attacks ( Goodfellow et al. , 2014 ; MoosaviDezfooli et al. , 2016 ; Kurakin et al. , 2016 ; Zhao et al. , 2018 ) and designing adversarial robust models ( Goodfellow et al. , 2014 ; Miyato et al. , 2015 ; Madry et al. , 2017 ; Zhang et al. , 2019 ) . It is observed that the adversarial robustness is usually achieved at the cost of a non-trivial degradation of standard performance . Previous efforts ( Schmidt et al. , 2018 ; Tsipras et al. , 2018 ; Nakkiran , 2019 ) trying to understand this phenomenon are usually based on simple toy models or heavy assumptions , and do not provide a general theoretic framework that explicitly shows how adversarial robustness is related to standard generalization . The work ( Ilyas et al. , 2019 ) empirically shows the adversarial samples might be human imperceptible features that could help generalization , but it does not provide any general theoretic framework to properly explain this phenomenon . In another work ( Zhang et al. , 2019 ) , though a bound on the gap between standard accuracy and adversarial accuracy is derived so that the gap between these two quantities could be explicitly controlled , the underlying reason of such trade-off is still not fully characterized . In this work , we start from a new perspective and consider the performance of deep classification models with a new metric Cross Category Kullback-Leibler divergence ( CCKL ) , namely the Kullback–Leibler ( KL ) divergence between the model ’ s output distributions over input data from different categories , instead of the traditional metric of accuracy . Interestingly , by applying Taylor expansion on CCKL , we show that it can be disentangled into a lower order non-robust component , which is related to causing adversarial behavior , and a higher order robust component . By applying such disentanglement , we are able to reveal the relation between standard generalization and adversarial robustness of the deep learning model in a relatively general setting . Furthermore , we demonstrate by experiments that current deep learning models rely heavily on optimizing the lower order non-robust component to generalize , which is a major underlying reason for the adversarial behavior . We also show the state-of-the-art adversarial training algorithms are all in fact trying to constrain the model from using the lower order non-robust component to discriminate data of different categories . Based on these findings , we claim that enabling the model to rely more on the higher order robust component instead of the adversary-prone lower order component might be the key to achieving decent standard accuracy and adversarial robustness simultaneously . Our contributions are summarized as follows : • We propose a new metric , Cross Category Kullback-Leibler divergence ( CCKL ) , to characterize the standard performance . This metric can be more naturally connected with the adversarial behavior of DNNs than the traditional metric of accuracy . • By applying a simple Taylor expansion on CCKL , we theoretically reveal the relation between standard generalization and adversarial behavior of DNNs in a general way without relying on toy models . • Based on the above novelties , we take a further step and point out the possible direction for simultaneously achieving decent standard accuracy and adversarial robustness . 2 RELATED WORK . Adversarial Attack and Defense To study the adversarial behavior of DNNs , gradient-based algorithms ( Goodfellow et al. , 2014 ; Papernot et al. , 2016a ; Moosavi-Dezfooli et al. , 2016 ; Carlini & Wagner , 2017 ; Kurakin et al. , 2016 ; Zhao et al. , 2018 ) are developed to find the most effective adversarial perturbations , and on the other hand , model-based defense algorithms ( Papernot et al. , 2016b ; Madry et al. , 2017 ; Miyato et al. , 2015 ; Zhang et al. , 2019 ) are also explored for enhancing model robustness against the adversarial perturbations . For example , in ( Miyato et al. , 2015 ; Zhang et al. , 2019 ) , the model is regularized by the KL divergence between outputs of clean and adversarial samples , which is also adopted in our theoretic framework . Fisher Information in Deep Learning Fisher information is used as a tool to study machine learning topics . Many previous works ( Chaudhry et al. , 2018 ; Pascanu & Bengio , 2013 ; Desjardins et al. , 2015 ; Andrychowicz et al. , 2016 ; Achille et al. , 2019 ) view the parameters of the model as those of the Fisher information , while some other ( Miyato et al. , 2015 ; Zhao et al. , 2018 ) take the model ’ s input as the parameters of Fisher information , making it applicable in the context of adversarial robustness . We adopt the same definition for Fisher information as ( Miyato et al. , 2015 ; Zhao et al. , 2018 ) in our later analysis . Explaining Adversarial Behavior of DNNs Previous works trying to explain the robust model ’ s degradation in standard performance ( Tsipras et al. , 2018 ; Schmidt et al. , 2018 ; Nakkiran , 2019 ) are mostly based on some simple toy models that can not generalize to real world settings . Recently , ( Ilyas et al. , 2019 ) empirically shows that adversarial noise is actually features that can help the generalization of DNNs . However , this work does not provide a general theoretic framework to explain this phenomenon . Zhang et al . ( 2019 ) provides a bound that characterizes the gap between standard accuracy and adversarial robustness , but they do not clearly explain the underlying reason of such trade-off . In our work , we analyze the underlying reason for the adversarial behavior of DNNs by directly disentangling the proposed CCKL . In this way , we not only theoretically demonstrate the reasoning behind the trade-off between adversarial robustness and standard generalization , but also point out the possible direction for achieving both objectives simultaneously . 3 NEW PERFORMANCE METRIC AND ITS DISENTANGLEMENT . In this section , we first propose to consider and measure the performance of deep classification models from the perspective of CCKL instead of accuracy . Then we explain our finding that the performance of a DNN model is determined by two components that are disentangled from the obtained objective based on KL divergence , one of which related to the model ’ s adversarial behavior and the other not . After that , to help understand our idea , we provide some complementary understandings from the viewpoint of information geometry . 3.1 PROPOSED PERFORMANCE METRIC . Normally , DNNs are trained by minimizing the following cross-entropy loss : L = E ( x , y ) ∼X×Y [ KL ( y‖f ( x ) ) ] , ( 1 ) where X × Y is the distribution where the data-label pair ( x , y ) is sampled from and f ( x ) is the prediction of the model . Note that y and f ( x ) are both discrete distribution over categories here . Correspondingly , the prediction accuracy is adopted to measure the performance of a DNN model . However , such a metric hides the connection between the model performance and its adversarial behavior . Since adversarial attack is concerned with changing the prediction of the model on a data point to another , we propose to shift the performance metric from investigating one single data point to comparing the data point with its counterparts from other categories , which can help reveal the above connection . Therefore , we propose the following Cross Category KL Divergence ( CCKL ) metric : CCKL = E ( ( xi , yi ) , ( xj , yj ) ) ∈S [ KL ( f ( xi ) ||f ( xj ) ) ] , ( 2 ) where S is defined as S = { ( ( xi , yi ) , ( xj , yj ) ) | ∀i , j ∈ N , yi 6= yj , xi , xj ∈ X } . ( 3 ) With the definition above , we can derive the following theorem : Theorem 3.1 Denoting JS ( p‖q ) as the Jensen-Shannon divergence between two distributions p and q , we have CCKL ≥ E ( ( xi , yi ) , ( xj , yj ) ) ∈S [ JS ( yi‖yj ) ] − L− coL , ( 4 ) where L is the cross-entropy loss defined in Equation ( 1 ) and coL = E ( x , y ) ∼ ( X , Y ) [ KL ( f ( x ) ‖y ) ] is the co-object for L. We defer the proof and more comments of Theorem 3.1 to Appendix A due to space limit . Taking a closer look at Theorem 3.1 , we know that E ( ( xi , yi ) , ( xj , yj ) ) ∈S [ JS ( yi‖yj ) ] remains unchanged during training , and L and coL are supposed to decrease when the model performance increases . Therefore , the lower bound on CCKL is supposed to increase with the rise of model performance , which will asymptotically result in the increase of CCKL . Thus , the CCKL , which characterizes how well the model discriminates input data from different categories , can serve as a performance metric . To further expound on our point , we provide a visualization of how CCKL evolves with test accuracy and test cross entropy loss during training in Figure 2 . According to our results , the variation of CCKL correlates well with the test accuracy and test cross entropy loss during training . 3.2 CONNECTIONS BETWEEN ADVERSARIAL BEHAVIOR AND CCKL . We now explain how the above CCKL objective ( 2 ) correlates with the model ’ s adversarial behavior . Consider a single data-label pair ( x , y ) . The following cross entropy loss of the model fθ on adversarial samples ( with perturbation η ) : Lθ ( x , η ) = KL ( y‖fθ ( x+ η ) ) s.t . ‖η‖ ≤ ( 5 ) is adopted to study the adversarial behavior in most previous literature ( Goodfellow et al. , 2014 ; Kurakin et al. , 2016 ; Madry et al. , 2017 ) . However , it is not easy to build connection between this objective and the standard generalization of the model . Interestingly , in some recent works ( Miyato et al. , 2015 ; Zhao et al. , 2018 ; Zhang et al. , 2019 ) , another objective is adopted to characterize the model ’ s adversarial behavior : Lθ ( x , η ) = KL ( fθ ( x ) ‖fθ ( x+ η ) ) s.t . ‖η‖ ≤ . ( 6 ) Based on the above objective ( 6 ) , ( Zhao et al. , 2018 ) reports state-of-the-art results in the task of adversarial attack and ( Zhang et al. , 2019 ) reports state-of-the-art results in adversarial robustness . More importantly , as will be shown later in the paper , the adversarial objective ( 6 ) can be easily linked to the standard generalization of the model . Therefore , we adopt the objective ( 6 ) in the rest of the paper . The corresponding adversarial training objective is thus formulated as follows : min θ max η Lθ ( x , η ) s.t . ‖η‖ ≤ . ( 7 ) Given the definition of Lθ ( x , η ) , applying Taylor expansion yields the following : max η Lθ ( x , η ) = max η 1 2 η > Fxη + ∞∑ k=3 T ( k ) x ( η ) s.t . ‖η‖ ≤ , ( 8 ) where Fx is the Fisher information of f ( x ) w.r.t . x and T ( k ) x is the following k-th order Taylor expansion term . Let f ( j ) ( x ) be the j-th entry of f ( x ) and n be the dimension of f ( x ) . Then Fx can be calculated by Fx = n∑ j=0 f ( j ) ( x ) ( ∇x log f ( j ) ( x ) ) ( ∇x log f ( j ) ( x ) ) > . ( 9 ) When is sufficiently small , the higher order terms in the above would vanish and Equation ( 8 ) could be simplified into max η Lθ ( x , η ) = 1 2 max η η > Fxη s.t . ‖η‖ ≤ . ( 10 ) We could therefore construct the lagrangian of the above problem : L ( x , η ) = 1 2 η > Fxη − 1 2 λ ( η > η − 2 ) . ( 11 ) By setting∇ηL ( x , η ) = 0 , we obtain Fxη = λmaxη , where λmax is the maximum eigenvalue of Fx . Therefore , the most effective adversarial perturbation corresponds to the leading eigenvector of Fx . Consequently , we have max η Lθ ( x , η ) = 1 2 λmax 2 . ( 12 ) Note that λmax here is also the spectral norm of the Fisher information matrix Fx . The above theory shows that the local adversarial behavior of the model f around the input x is determined by the spectral norm of the Fisher information matrix : the adversarial behavior around x would be more severe if the spectral norm of Fx is larger . Now we analyze CCKL by considering one tuple ( ( xi , yi ) , ( xj , yj ) ) randomly sampled from S . We rewrite KL ( f ( xi ) ‖f ( xj ) ) as KL ( f ( xi ) ‖f ( xj ) ) = KL ( f ( xi ) ‖f ( xi + ( xj − xi ) ) ) = Lθ ( xi , xj − xi ) . ( 13 ) Then , we apply the same Taylor expansion as above and obtain KL ( f ( xi ) ‖f ( xj ) ) = Lθ ( xi , xj − xi ) = 1 2 ( xj − xi ) > Fxi ( xj − xi ) + ∞∑ k=3 T ( k ) xi ( xj − xi ) . ( 14 ) Comparing Equation ( 14 ) 1 and Equation ( 10 ) , we notice they share the same Fisher information Fxi at data point xi . Here , we show the adversarial behavior at each data point can be connected to the performance objective CCKL via Fisher information , which lays the foundation for the disentanglement in the next subsection .

Summary: The authors propose a new performance metric, CCKL, as an alternative to standard cross-entropy. At a high level the goal of this objective is to ensure that the classifier is confidently separating samples from different classes. They decompose this objective into its Taylor expansion terms and argue that, since adversarial vulnerability is a local property, it only depends on the first of these terms. They finally propose training models of higher capacity as a way of relying more on higher-order terms of the objective and hence improving adversarial robustness.