Visual Interpretability Alone Helps Adversarial Robustness

1 INTRODUCTION . It has become widely known that convolutional neural networks ( CNNs ) are vulnerable to adversarial examples , namely , perturbed inputs with intention to mislead networks ’ prediction ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ; Papernot et al. , 2016a ; Carlini & Wagner , 2017 ; Chen et al. , 2018 ; Su et al. , 2018 ) . The vulnerability of CNNs has spurred extensive research on adversarial attack and defense . To design adversarial attacks , most work has focused on creating either imperceptible input perturbations ( Goodfellow et al. , 2015 ; Papernot et al. , 2016a ; Carlini & Wagner , 2017 ; Chen et al. , 2018 ) or adversarial patches robust to the physical environment ( Eykholt et al. , 2018 ; Brown et al. , 2017 ; Athalye et al. , 2017 ) . Many defense methods have also been developed to prevent CNNs from misclassification when facing adversarial attacks . Examples include defensive distillation ( Papernot et al. , 2016b ) , training with adversarial examples ( Goodfellow et al. , 2015 ) , input gradient or curvature regularization ( Ross & Doshi-Velez , 2018 ; Moosavi-Dezfooli et al. , 2019 ) , adversarial training via robust optimization ( Madry et al. , 2018 ) , and TRADES to trade adversarial robustness off against accuracy ( Zhang et al. , 2019 ) . Besides studying adversarial effects on network prediction decisions , this work explores the connection between adversarial robustness and network interpretability , and provides novel insights on when and how interpretability helps the robustness . Having a prediction might not be enough for many real-world machine learning applications . It is also crucial to demystify why CNNs make certain decisions . Thus , the problem of network interpretation arises . Various methods have been proposed to understand the mechanism of decision making by CNNs . One category of methods justify a prediction decision by assigning importance values to reflect the influence of individual pixels or image sub-regions on the final classification . Examples include pixel-space sensitivity map methods ( Simonyan et al. , 2013 ; Zeiler & Fergus , 2014 ; Springenberg et al. , 2014 ; Smilkov et al. , 2017 ; Sundararajan et al. , 2017 ) and class-discriminative localization methods ( Zhou et al. , 2016 ; Selvaraju et al. , 2017 ; Chattopadhay et al. , 2018 ; Petsiuk et al. , 2018 ) , where the former evaluates the sensitivity of a network classification decision to pixel variations at the input , and the latter localizes which parts of an input image were looked at by the network for making a classification decision . Sensitivity map methods include vanilla gradient ( Simonyan et al. , 2013 ) , deconvolution ( Zeiler & Fergus , 2014 ) , guided backpropagation ( Springenberg et al. , 2014 ) , SmoothGrad ( Smilkov et al. , 2017 ) , integrated gradient ( IG ) ( Sundararajan et al. , 2017 ) to name a few . They highlight fine-grained details in the image but are not class-discriminative for visual explanation . By contrast , localization approaches like class activation map ( CAM ) ( Zhou et al. , 2016 ) , GradCAM ( Selvaraju et al. , 2017 ) , GradCAM++ ( Chattopadhay et al. , 2018 ) and RISE ( Petsiuk et al. , 2018 ) are highly class-discriminative , namely , localizing image sub-regions reasoned for a prediction class . We refer readers to Sec . 2 for some representative interpretation methods . Besides interpreting CNNs via feature importance maps , some methods zoom into the internal response of neural networks . Examples include network dissection ( Bau et al. , 2017 ) , which evaluates the alignment between individual hidden units and semantic concepts , and learning perceptuallyaligned representations from robust training ( Engstrom et al. , 2019 ) . Very recently , some works ( Xu et al. , 2019b ; a ; Zhang et al. , 2018 ; Subramanya et al. , 2018 ; Ghorbani et al. , 2019 ; Dombrowski et al. , 2019 ; Chen et al. , 2019 ) are beginning to study adversarial robustness by exploring the spectrum between classification accuracy and network interpretability . It was shown in ( Xu et al. , 2019b ; a ) that an imperceptible adversarial perturbation to fool classifiers can lead to a significant change in a class-specific network interpretability map , e.g. , CAM . Thus , it was argued that such an interpretability discrepancy can be used as a helpful metric to differentiate adversarial examples from benign inputs . However , the work ( Zhang et al. , 2018 ; Subramanya et al. , 2018 ) showed that under certain conditions , generating an attack ( which we call interpretability sneaking attack , ISA ) that fools the classifier as well as its coupled interpreter ( in terms of keeping interpretability map highly similar to that of benign input ) is not significantly more difficult than generating adversarial inputs deceiving the classifier only . Besides investigating robustness in classification through the lens of interpretability , the work ( Ghorbani et al. , 2019 ; Dombrowski et al. , 2019 ) studied the robustness in network interpretation maps , showing that which can significantly be manipulated via imperceptible input perturbations but keeping the classifier ’ s decision intact . We call this type of threat model attack against interpretability ( AAI ) . The existing work had no agreement on the relationship between robustness in interpretation and robustness in classification . Spurred by that , we attempt to explore this relationship from both attack and defense perspectives . The most relevant work to ours is ( Chen et al. , 2019 ) , which robustified network interpretation with the aid of integrated gradient ( IG ) , an axiomatic attribution map . It proposed a robust attribution training , which was shown as a principled generalization of previous formulations of robust classification and an effective defense against AAI . In this paper , we first investigate when ISA is possible , and then relate our insights on ISA to robust classification and robust interpretability . Different from the previous work , our paper contains the following contributions . 1 . We provide an answer to the question of when adversarial examples can bypass interpretability discrepancy . We show that enforcing stealthiness of adversarial examples from network interpretation could be challenging . Its difficulty relies on how one measures the interpretability discrepancy caused by input perturbations . 2 . We propose an ` 1 norm based 2-class interpretability discrepancy measure and theoretically show that constraining it helps adversarial robustness . 3 . We develop an interpretability-aware robust training method and empirically show that interpretability alone can be used to defend adversarial attacks for both misclassifcation and misinterpretation . Compared to the IG-based robust attribution training ( Chen et al. , 2019 ) , our approach is simpler in implementation , and provides better robustness even as facing a strong adversary . 2 PRELIMINARIES AND MOTIVATION : INTERPRETABILITY OF CNNS FOR JUSTIFYING A CLASSIFICATION DECISION . To explain what and why CNNs predict , we consider two types of network interpretation methods : a ) class activation map ( CAM ) ( Zhou et al. , 2016 ; Selvaraju et al. , 2017 ; Chattopadhay et al. , 2018 ) and b ) pixel sensitivity map ( PSM ) ( Simonyan et al. , 2013 ; Springenberg et al. , 2014 ; Smilkov et al. , 2017 ; Sundararajan et al. , 2017 ; Yeh et al. , 2019 ) . Let f ( x ) ∈ RC denote a CNN-based predictor that maps an input x ∈ Rd to a probability vector of C classes . Here fc ( x ) , the cth element of f ( x ) , denotes the classification score ( given by logit before the softmax ) for class c. Let L ( x , c ) denote an interpreter ( CAM or PSM ) that reflects where in x contributes to the classifier ’ s decision on c. CAM-type methods . CAM ( Zhou et al. , 2016 ) produces a class-discriminative localization map for CNNs , which performs global averaging pooling over convolutional feature maps prior to the softmax . Let the penultimate layer output K feature maps , each of which is denoted by a vector representation Ak ∈ Ru for channel k ∈ [ K ] . Here [ K ] represents the integer set { 1 , 2 , . . . , K } . The ith entry of CAM LCAM ( x , c ) is given by [ LCAM ( x , c ) ] i = 1 u ∑ k∈ [ K ] wckAk , i , i ∈ [ u ] , ( 1 ) where wck is the linear classification weight that associates the channel k with the class c , and Ak , i denotes the ith element of Ak . The rationale behind ( 1 ) is that the classification score fc ( x ) can be written as the average of CAM values ( Zhou et al. , 2016 ) , fc ( x ) = ∑u i=1 [ LCAM ( x , c ) ] i . For visual explanation , LCAM ( x , c ) is often up-sampled to the input dimension d using bi-linear interpolation . GradCAM ( Selvaraju et al. , 2017 ) generalizes CAM for CNNs without the architecture ‘ global average pooling → softmax layer ’ over the final convolutional maps . Specifically , the weight wck in ( 1 ) is given by the gradient of the classification score fc ( x ) with respect to ( w.r.t . ) the feature map Ak , wck = 1 u ∑u i=1 ∂fc ( x ) ∂Ak , i . GradCAM++ ( Chattopadhay et al. , 2018 ) , a generalized formulation of GradCAM , utilizes a more involved weighted average of the ( positive ) pixel-wise gradients but provides a better localization map if an image contains multiple occurrences of the same class . In this work , we focus on CAM since it is computationally light and our models used in experiments follow the architecture ‘ global average pooling→ softmax layer ’ . PSM-type methods . PSM uses calculations with gradients to assign importance scores to individual pixels toward explaining the classification decision about an input . Examples of commonly-used approaches include vanilla gradient ( Simonyan et al. , 2013 ) , guided backpropogation ( Springenberg et al. , 2014 ) , SmoothGrad ( Smilkov et al. , 2017 ) , and integrated gradient ( IG ) ( Sundararajan et al. , 2017 ) . In particular , IG satisfies the completeness attribution axiom that PSM ought to obey . Specifically , it averages gradient saliency maps for interpolations between an input x and a baseline image a : [ LIG ( x , c ) ] i = ( xi − ai ) ∫ 1 α=0 ∂fc ( a+ α ( x− a ) ) ∂xi dα ≈ ( xi − ai ) m∑ i=1 ∂fc ( a+ i m ( x− a ) ) ∂xi 1 m , i ∈ [ d ] , ( 2 ) where m is the number of steps in the Riemman approximation of the integral . The completeness axiom ( Sundararajan et al. , 2017 , Proposition 1 ) states that ∑d i=1 [ LIG ( x , c ) ] i = fc ( x ) − fc ( a ) , where the baseline image a is often chosen such that fc ( a ) ≈ 0 , e.g. , the black image . Note that CAM also satisfies the completeness axiom . PSM is able to highlight fine-grained details in the image , but is computationally intensive and not very class-discriminative compared to CAM ( Selvaraju et al. , 2017 ) . Interpretability discrepancy caused by adversarial perturbation . Let x′ = x + δ represent an adversarial example w.r.t . x , where δ denotes an adversarial perturbation . By replacing the input image x with x′ , the CNN will be fooled from the true label t to the target ( incorrect ) label t′ . It was recently shown in ( Xu et al. , 2019b ; a ) that the adversary could introduce an evident interpretability discrepancy w.r.t . both the true and the target label in terms of L ( x , t ) vs. L ( x′ , t ) , and L ( x , t′ ) vs. L ( x′ , t′ ) . An illustrative example is provided in Figure 1 . We see that an adversary suppresses the network interpretation w.r.t . the true label but promotes the interpretation w.r.t . the target label . We also observe that compared to IG , CAM and GradCAM++ better localize class-specific discriminative regions . These results reveal two observations on how measuring interpretation discrepancy affects classification robustness : a ) interpretability discrepancy may be used to detect adversarial examples , b ) interpretability discrepancy itself may be vulnerable to adversarial perturbations . In what follows , we explore the spectrum between adversarial robustness and interpretability from a unified perspective considering both the adversarial vulnerability of interpretability discrepancy and the value of interpretability discrepancy in a defense .

Summary: The present work considers adversarial attacks that also yield similar outputs for "interpretability methods", which are methods that output some vector corresponding to a given classification (usually the vector is e.g. an image or a similar object). It also shows that by regularizing nearby inputs to have similar interpetations (instead of similar classifications), robustness can be achieved similar to adversarial training.