Distilling Robust and Non-Robust Features in Adversarial Examples by Information Bottleneck

1 Introduction . Deep neural networks ( DNNs ) have achieved remarkable performances in a wide variety of machine learning tasks . Despite the breakthrough outcomes , DNNs are easily fooled from adversarial attacks , with crafted perturbations [ 1 , 2 , 3 , 4 , 5 , 6 , 7 ] . These perturbations are imperceptible to human eyes , but simply adding them to clean images ( i.e. , adversarial examples ) can effectively deceive classifiers . Such a vulnerability affects security problems [ 8 , 9 , 10 , 11 ] , bringing in the weak reliability of DNNs . Previous works have broadly investigated the reason for the widespread of such adversarial examples . Goodfellow et al . [ 2 ] have argued that adversarial vulnerability is induced from the excessive linearity nature of DNNs in high-dimensional spaces . Several works [ 1 , 12 ] have regarded the primary cause of the adversarial examples as statistical variation with aberrations in data manifold . Schmidt et al . [ 13 ] have suggested that the pervasiveness of the examples should not be considered as a drawback of training methods for DNNs , since the available dataset may not be large enough to train them robustly . In recent years , Tsipras et al . [ 14 ] have suggested an intriguing analysis that the disagreement between standard and adversarial accuracy stems from differently trained feature representation . In this literature , Ilyas et al . [ 15 ] further have demonstrated the adversarial examples are inevitable results of standard supervised training and arisen from well-generalized features in the dataset . They have analyzed the adversarial examples are originated from brittle and unintelligible features ( i.e. , non-robust features ) that are arbitrarily manipulated with the imperceptible noise , and shown that the ∗Equal contribution . † Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . robust features still can provide precise accuracy even in the existence of adversarial perturbation . They have argued that the non-robust features can not show reliable accuracy in the adversarial setting and could provoke incomprehensible properties . Nonetheless , the underlying reason for the existence and pervasiveness of adversarial examples can not derive common consensus in the research field and still remains unclear [ 16 ] . To clarify where the adversarial brittleness truly comes from , we need to figure out how the robust and non-robust features in data manifold subtly manipulate feature representation and fool model prediction , by directly handling them in the feature space . To address it , we propose a way to precisely distill intermediate features into robust and non-robust features by employing Information Bottleneck ( IB ) [ 17 , 18 , 19 ] . In the sense that semantic information is included in the units of intermediate feature representation [ 20 , 21 , 22 , 23 ] , we utilize the bottleneck to regulate the information flow in the feature space by explicitly adding noise to the feature units . Then , we estimate how each feature unit contaminated with the noise affects model prediction with assigned information . Based on the prediction sensitivity of the noise intervention , we assort the feature units either robust or brittle , and disentangle the feature representation into robust or non-robust features , respectively . Through extensive analysis of the distilled features , we corroborate that the pervasiveness of the adversarial brittleness is derived from the non-robust features , and they have a high correlation with the adversarial prediction . In addition , in order to understand the semantic information of distilled features , we directly visualize them in the feature space and provide their visual interpretation . Consequently , we reveal that both of the robust and non-robust features indeed have semantic information in terms of human-perception by themselves . Based on our observation , we theoretically describe the negative impact of the non-robust features for the model prediction and introduce an approach of amplifying the gradients of non-robust features to break the model prediction . In this paper , our contributions can be summarized into three-fold as follows : • We propose a novel way to explicitly distill intermediate features into the robust and nonrobust features using Information Bottleneck , and interpret the disentangled features in terms of human-perception by directly visualizing them in the feature space . • By analyzing how the distilled features affect the intermediate feature representation under adversarial perturbation , we demonstrate that the non-robust features are highly correlated with the adversarial prediction . • We present an attack mechanism manipulating the non-robust features by strengthening their gradients , and validate its effectiveness of breaking model prediction . 2 Distilling Robust and Non-robust Features in Intermediate Feature Space . Problem Setup and Notations . Let X denote clean images and Y denote ( one-hot encoded ) target labels corresponding to the clean images . Then , adversarial examples Xadv can be created by the following equations : max δ E ( X , Y ) [ L ( f ( X + δ ) , Y ) ] , where δ denotes an adversarial perturbation , and L denotes a pre-defined loss for machine learning tasks . The adversarial examples can be made by Xadv = X + δ . When a given model f is adversarially trained against PGD attack [ 7 ] , it can be written as follows : min w max ‖δ‖∞≤γ E ( X , Y ) [ L ( f ( X + δ ) , Y ) ] , ( 1 ) where w represents the parameters of f , which are learned to be robust against adversarial attacks . Here , ‖·‖∞ ≤ γ describes L∞ norm , and γ-ball means the perturbation magnitude . In this paper , we adversarially train the model f on γ = 0.03 for the standard adversarial attack . Note that once adversarially trained , the parameters of the model f are no longer covered.1 As notation of variables that we will use in this paper , Z and Z̄ indicate the intermediate features of the model f such that Z = fl ( X ) and Z̄ = fl ( Xadv ) , where fl ( · ) describes l-th layer outputs of the given model . Similarly , fl+ ( · ) represents subsequent network after the l-th layer , thus intermediate features can be propagated to the last output layer , such that Ŷ = fl+ ( Z ) and Ŷadv = fl+ ( Z̄ ) . The 1Previous works [ 14 , 15 ] have demonstrated that the distinction between the robust and non-robust features arises in adversarial settings . In the sense that adversarially trained networks learn robust representation [ 24 ] , we set the robust classifier as default . Please see the analysis of the standard training in Appendix F. model f can be expressed as f = fl+ ◦ fl , satisfying Ŷ = f ( X ) for the given clean images X . Also , Ŷadv = f ( Xadv ) is denoted by model propagation of the adversarial examples Xadv . Note that we designate l-th layer as the last convolutional layer , and regard l+ as the rest of the layers in the model . 2.1 Information Bottleneck for Distilling Informative Features . In adversarial settings , we focus on separating robust and non-robust features in the intermediate layer . Recall that robust features are literally robust on the noise ( variation ) and invariant to the existence of the adversarial perturbation , but non-robust features are not . Our approach aims to distill feature units that affect target prediction under the noise perturbation in the intermediate feature space . We follow that the semantic information is inherently included in the feature units of DNNs [ 20 , 22 , 25 ] . From this perspective , we utilize Information Bottleneck ( IB ) to distill the robust and non-robust features on the given intermediate features Z . Information Bottleneck [ 17 , 18 ] proposed to encode maximally informative representation for target labels , restraining input information , concurrently . Using the bottleneck , we suggest a way to assess feature importance and quantify information flow for the target prediction . The objective function of IB can be written as follows : max Z I ( Z , Y ) − βI ( Z , X ) , ( 2 ) where I denotes mutual information , and β represents the degree of restraining input information . The first term I ( Z , Y ) allows the intermediate features Z to be predictive on the target label Y , and the second term I ( Z , X ) encourages Z to compress the information of the given images X in the bottleneck . Here , the second term requires a true feature probability p ( Z ) = ∫ X p ( Z | X ) p ( X ) dX to expand it , but it is computationally intractable due to a high dimensional dependency of the dataset probability p ( X ) . Thus , several works [ 18 , 19 ] modified the IB ’ s objective function to make it possible to learn DNNs without the true feature probability as follows : minLCE + βLI ( see Appendix B ) . In this formulation , LCE indicates cross-entropy loss , and LI represents information loss computed by KL divergence [ 26 ] between a feature likelihood p ( Z | X ) and an approximate feature probability q ( Z ) . It is radically a closed-form approximation for the true feature probability p ( Z ) . Firstly , we deliberately inject noise variation into Z to estimate the prediction sensitivity of each feature unit along the noise intervention . To do so , we newly design an approximate feature probability using noise variation σ , such that qσ ( Z ) = N ( fl ( X ) , σ2 ) . Then , we sample random variables from qσ ( Z ) and define informative features ZI as follows : ZI = fl ( X ) + σ · , ( 3 ) where ZI ∼ qσ ( Z ) . Note that the operator · denotes Hadamard product , and stands for Gaussian noise sampled from N ( 0 , I ) . Here , the noise variation measures a correlation between intermediate features and model prediction based on the fact that robustness means high correlation on model prediction and non-robustness are opposite [ 14 , 15 ] in adversarial settings . Since the correlation E ( X , Y ) [ Y ·f ( X ) ] in output layer can be expressed as a variance measure Cov ( Y , f ( X ) ) , we consider the correlation in intermediate layer as the noise variation . From this perspective , if a feature unit is highly predictive despite large noise variation ( high correlation ) , the unit can robustly predict target labels , while a non-robust unit can not . Once the informative features ZI are acquired from the noise variation σ , we propagate ZI to the last output layer and estimate the feature importance of each unit for model prediction . Then , we deal with the information loss LI in order to alleviate feature heterogeneity between Z and ZI . Through aforementioned modification [ 18 , 19 ] , our objective function can be written as follows : min σ LX ( σ ) = −Y log fl+ ( fl ( X ) + σ · ) ︸ ︷︷ ︸ LCE +β DKL [ p ( Z | X ) || qσ ( Z ) ] ︸ ︷︷ ︸ LI , ( 4 ) where the feature likelihood p ( Z | X ) is set to N ( fl ( X ) , σ2z ) . Here , σz indicates inherent feature variation of the intermediate features Z for each unit . The second term LI makes the informative features ZI resemble the intermediate features Z , while minimizing the cross-entropy LCE . This second term can be written as LI = 12 ∑C k=1 [ σ2zk σ2k + log σ2k σ2zk − 1 ] , where k denotes an index of the noise variation σ = [ σ1 , σ2 , · · · , σC ] ( the optimizing parameters ) and the feature variation σz = [ σz1 , σz2 , · · · , σzC ] ( the given parameters ) . Here , only of the variation σ is updated such that σ ← σ − ∂∂σLX ( σ ) . In brief , we summarize overall procedure of our bottleneck concept in Fig . 1 . Moreover , we mention that β in Eq . ( 4 ) controls the amount of information that flows into the feature representation . Specifically , when β is set to zero , IB loss is equivalent to cross-entropy loss , which means that ZI can accommodate even unimportant features to predict target labels . In contrast , excessively large β only focuses on compressing input information , thus IB may can not filter out important features to predict target labels . Accordingly , we empirically control β to distill informative features ZI based on the noise variation σ ( Please see section 3.4 for analysis of information flow ) .

Summary: This paper categorizes the intermediate features of a DNN into robust ones and non-robust ones. Through visualization, the non-robust ones display different semantic information from the true label, and thus potentially leads to the existence of AE.  The paper proposes a new AE generation algorithm targeting the non-robust feature, and beats the common attack baselines.