Distilling Robust and Non-Robust Features in Adversarial Examples by Information Bottleneck

1 Introduction . Deep neural networks ( DNNs ) have achieved remarkable performances in a wide variety of machine learning tasks . Despite the breakthrough outcomes , DNNs are easily fooled from adversarial attacks , with crafted perturbations [ 1 , 2 , 3 , 4 , 5 , 6 , 7 ] . These perturbations are imperceptible to human eyes , but simply adding them to clean images ( i.e. , adversarial examples ) can effectively deceive classifiers . Such a vulnerability affects security problems [ 8 , 9 , 10 , 11 ] , bringing in the weak reliability of DNNs . Previous works have broadly investigated the reason for the widespread of such adversarial examples . Goodfellow et al . [ 2 ] have argued that adversarial vulnerability is induced from the excessive linearity nature of DNNs in high-dimensional spaces . Several works [ 1 , 12 ] have regarded the primary cause of the adversarial examples as statistical variation with aberrations in data manifold . Schmidt et al . [ 13 ] have suggested that the pervasiveness of the examples should not be considered as a drawback of training methods for DNNs , since the available dataset may not be large enough to train them robustly . In recent years , Tsipras et al . [ 14 ] have suggested an intriguing analysis that the disagreement between standard and adversarial accuracy stems from differently trained feature representation . In this literature , Ilyas et al . [ 15 ] further have demonstrated the adversarial examples are inevitable results of standard supervised training and arisen from well-generalized features in the dataset . They have analyzed the adversarial examples are originated from brittle and unintelligible features ( i.e. , non-robust features ) that are arbitrarily manipulated with the imperceptible noise , and shown that the ∗Equal contribution . † Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . robust features still can provide precise accuracy even in the existence of adversarial perturbation . They have argued that the non-robust features can not show reliable accuracy in the adversarial setting and could provoke incomprehensible properties . Nonetheless , the underlying reason for the existence and pervasiveness of adversarial examples can not derive common consensus in the research field and still remains unclear [ 16 ] . To clarify where the adversarial brittleness truly comes from , we need to figure out how the robust and non-robust features in data manifold subtly manipulate feature representation and fool model prediction , by directly handling them in the feature space . To address it , we propose a way to precisely distill intermediate features into robust and non-robust features by employing Information Bottleneck ( IB ) [ 17 , 18 , 19 ] . In the sense that semantic information is included in the units of intermediate feature representation [ 20 , 21 , 22 , 23 ] , we utilize the bottleneck to regulate the information flow in the feature space by explicitly adding noise to the feature units . Then , we estimate how each feature unit contaminated with the noise affects model prediction with assigned information . Based on the prediction sensitivity of the noise intervention , we assort the feature units either robust or brittle , and disentangle the feature representation into robust or non-robust features , respectively . Through extensive analysis of the distilled features , we corroborate that the pervasiveness of the adversarial brittleness is derived from the non-robust features , and they have a high correlation with the adversarial prediction . In addition , in order to understand the semantic information of distilled features , we directly visualize them in the feature space and provide their visual interpretation . Consequently , we reveal that both of the robust and non-robust features indeed have semantic information in terms of human-perception by themselves . Based on our observation , we theoretically describe the negative impact of the non-robust features for the model prediction and introduce an approach of amplifying the gradients of non-robust features to break the model prediction . In this paper , our contributions can be summarized into three-fold as follows : • We propose a novel way to explicitly distill intermediate features into the robust and nonrobust features using Information Bottleneck , and interpret the disentangled features in terms of human-perception by directly visualizing them in the feature space . • By analyzing how the distilled features affect the intermediate feature representation under adversarial perturbation , we demonstrate that the non-robust features are highly correlated with the adversarial prediction . • We present an attack mechanism manipulating the non-robust features by strengthening their gradients , and validate its effectiveness of breaking model prediction . 2 Distilling Robust and Non-robust Features in Intermediate Feature Space . Problem Setup and Notations . Let X denote clean images and Y denote ( one-hot encoded ) target labels corresponding to the clean images . Then , adversarial examples Xadv can be created by the following equations : max δ E ( X , Y ) [ L ( f ( X + δ ) , Y ) ] , where δ denotes an adversarial perturbation , and L denotes a pre-defined loss for machine learning tasks . The adversarial examples can be made by Xadv = X + δ . When a given model f is adversarially trained against PGD attack [ 7 ] , it can be written as follows : min w max ‖δ‖∞≤γ E ( X , Y ) [ L ( f ( X + δ ) , Y ) ] , ( 1 ) where w represents the parameters of f , which are learned to be robust against adversarial attacks . Here , ‖·‖∞ ≤ γ describes L∞ norm , and γ-ball means the perturbation magnitude . In this paper , we adversarially train the model f on γ = 0.03 for the standard adversarial attack . Note that once adversarially trained , the parameters of the model f are no longer covered.1 As notation of variables that we will use in this paper , Z and Z̄ indicate the intermediate features of the model f such that Z = fl ( X ) and Z̄ = fl ( Xadv ) , where fl ( · ) describes l-th layer outputs of the given model . Similarly , fl+ ( · ) represents subsequent network after the l-th layer , thus intermediate features can be propagated to the last output layer , such that Ŷ = fl+ ( Z ) and Ŷadv = fl+ ( Z̄ ) . The 1Previous works [ 14 , 15 ] have demonstrated that the distinction between the robust and non-robust features arises in adversarial settings . In the sense that adversarially trained networks learn robust representation [ 24 ] , we set the robust classifier as default . Please see the analysis of the standard training in Appendix F. model f can be expressed as f = fl+ ◦ fl , satisfying Ŷ = f ( X ) for the given clean images X . Also , Ŷadv = f ( Xadv ) is denoted by model propagation of the adversarial examples Xadv . Note that we designate l-th layer as the last convolutional layer , and regard l+ as the rest of the layers in the model . 2.1 Information Bottleneck for Distilling Informative Features . In adversarial settings , we focus on separating robust and non-robust features in the intermediate layer . Recall that robust features are literally robust on the noise ( variation ) and invariant to the existence of the adversarial perturbation , but non-robust features are not . Our approach aims to distill feature units that affect target prediction under the noise perturbation in the intermediate feature space . We follow that the semantic information is inherently included in the feature units of DNNs [ 20 , 22 , 25 ] . From this perspective , we utilize Information Bottleneck ( IB ) to distill the robust and non-robust features on the given intermediate features Z . Information Bottleneck [ 17 , 18 ] proposed to encode maximally informative representation for target labels , restraining input information , concurrently . Using the bottleneck , we suggest a way to assess feature importance and quantify information flow for the target prediction . The objective function of IB can be written as follows : max Z I ( Z , Y ) − βI ( Z , X ) , ( 2 ) where I denotes mutual information , and β represents the degree of restraining input information . The first term I ( Z , Y ) allows the intermediate features Z to be predictive on the target label Y , and the second term I ( Z , X ) encourages Z to compress the information of the given images X in the bottleneck . Here , the second term requires a true feature probability p ( Z ) = ∫ X p ( Z | X ) p ( X ) dX to expand it , but it is computationally intractable due to a high dimensional dependency of the dataset probability p ( X ) . Thus , several works [ 18 , 19 ] modified the IB ’ s objective function to make it possible to learn DNNs without the true feature probability as follows : minLCE + βLI ( see Appendix B ) . In this formulation , LCE indicates cross-entropy loss , and LI represents information loss computed by KL divergence [ 26 ] between a feature likelihood p ( Z | X ) and an approximate feature probability q ( Z ) . It is radically a closed-form approximation for the true feature probability p ( Z ) . Firstly , we deliberately inject noise variation into Z to estimate the prediction sensitivity of each feature unit along the noise intervention . To do so , we newly design an approximate feature probability using noise variation σ , such that qσ ( Z ) = N ( fl ( X ) , σ2 ) . Then , we sample random variables from qσ ( Z ) and define informative features ZI as follows : ZI = fl ( X ) + σ · , ( 3 ) where ZI ∼ qσ ( Z ) . Note that the operator · denotes Hadamard product , and stands for Gaussian noise sampled from N ( 0 , I ) . Here , the noise variation measures a correlation between intermediate features and model prediction based on the fact that robustness means high correlation on model prediction and non-robustness are opposite [ 14 , 15 ] in adversarial settings . Since the correlation E ( X , Y ) [ Y ·f ( X ) ] in output layer can be expressed as a variance measure Cov ( Y , f ( X ) ) , we consider the correlation in intermediate layer as the noise variation . From this perspective , if a feature unit is highly predictive despite large noise variation ( high correlation ) , the unit can robustly predict target labels , while a non-robust unit can not . Once the informative features ZI are acquired from the noise variation σ , we propagate ZI to the last output layer and estimate the feature importance of each unit for model prediction . Then , we deal with the information loss LI in order to alleviate feature heterogeneity between Z and ZI . Through aforementioned modification [ 18 , 19 ] , our objective function can be written as follows : min σ LX ( σ ) = −Y log fl+ ( fl ( X ) + σ · ) ︸ ︷︷ ︸ LCE +β DKL [ p ( Z | X ) || qσ ( Z ) ] ︸ ︷︷ ︸ LI , ( 4 ) where the feature likelihood p ( Z | X ) is set to N ( fl ( X ) , σ2z ) . Here , σz indicates inherent feature variation of the intermediate features Z for each unit . The second term LI makes the informative features ZI resemble the intermediate features Z , while minimizing the cross-entropy LCE . This second term can be written as LI = 12 ∑C k=1 [ σ2zk σ2k + log σ2k σ2zk − 1 ] , where k denotes an index of the noise variation σ = [ σ1 , σ2 , · · · , σC ] ( the optimizing parameters ) and the feature variation σz = [ σz1 , σz2 , · · · , σzC ] ( the given parameters ) . Here , only of the variation σ is updated such that σ ← σ − ∂∂σLX ( σ ) . In brief , we summarize overall procedure of our bottleneck concept in Fig . 1 . Moreover , we mention that β in Eq . ( 4 ) controls the amount of information that flows into the feature representation . Specifically , when β is set to zero , IB loss is equivalent to cross-entropy loss , which means that ZI can accommodate even unimportant features to predict target labels . In contrast , excessively large β only focuses on compressing input information , thus IB may can not filter out important features to predict target labels . Accordingly , we empirically control β to distill informative features ZI based on the noise variation σ ( Please see section 3.4 for analysis of information flow ) .

Summary: This study aims at distilling feature representations (mainly the last convolution layer) into robust and non-robust features using information bottleneck. For every layer, the feature maps are partitioned into robust and non-robust for each individual example. After finding this, the paper first illustrates that selectively propagating robust features of adversarial examples built for a robust network will not degrade the accuracy but propagating non-robust features does which to some extent illustrates that the information bottleneck method has correctly identified the robust and non robust features. They also illustrate that the adversarial class label has high correlation to the Non-robust features.