Visual Interpretability Alone Helps Adversarial Robustness

1 INTRODUCTION . It has become widely known that convolutional neural networks ( CNNs ) are vulnerable to adversarial examples , namely , perturbed inputs with intention to mislead networks ’ prediction ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ; Papernot et al. , 2016a ; Carlini & Wagner , 2017 ; Chen et al. , 2018 ; Su et al. , 2018 ) . The vulnerability of CNNs has spurred extensive research on adversarial attack and defense . To design adversarial attacks , most work has focused on creating either imperceptible input perturbations ( Goodfellow et al. , 2015 ; Papernot et al. , 2016a ; Carlini & Wagner , 2017 ; Chen et al. , 2018 ) or adversarial patches robust to the physical environment ( Eykholt et al. , 2018 ; Brown et al. , 2017 ; Athalye et al. , 2017 ) . Many defense methods have also been developed to prevent CNNs from misclassification when facing adversarial attacks . Examples include defensive distillation ( Papernot et al. , 2016b ) , training with adversarial examples ( Goodfellow et al. , 2015 ) , input gradient or curvature regularization ( Ross & Doshi-Velez , 2018 ; Moosavi-Dezfooli et al. , 2019 ) , adversarial training via robust optimization ( Madry et al. , 2018 ) , and TRADES to trade adversarial robustness off against accuracy ( Zhang et al. , 2019 ) . Besides studying adversarial effects on network prediction decisions , this work explores the connection between adversarial robustness and network interpretability , and provides novel insights on when and how interpretability helps the robustness . Having a prediction might not be enough for many real-world machine learning applications . It is also crucial to demystify why CNNs make certain decisions . Thus , the problem of network interpretation arises . Various methods have been proposed to understand the mechanism of decision making by CNNs . One category of methods justify a prediction decision by assigning importance values to reflect the influence of individual pixels or image sub-regions on the final classification . Examples include pixel-space sensitivity map methods ( Simonyan et al. , 2013 ; Zeiler & Fergus , 2014 ; Springenberg et al. , 2014 ; Smilkov et al. , 2017 ; Sundararajan et al. , 2017 ) and class-discriminative localization methods ( Zhou et al. , 2016 ; Selvaraju et al. , 2017 ; Chattopadhay et al. , 2018 ; Petsiuk et al. , 2018 ) , where the former evaluates the sensitivity of a network classification decision to pixel variations at the input , and the latter localizes which parts of an input image were looked at by the network for making a classification decision . Sensitivity map methods include vanilla gradient ( Simonyan et al. , 2013 ) , deconvolution ( Zeiler & Fergus , 2014 ) , guided backpropagation ( Springenberg et al. , 2014 ) , SmoothGrad ( Smilkov et al. , 2017 ) , integrated gradient ( IG ) ( Sundararajan et al. , 2017 ) to name a few . They highlight fine-grained details in the image but are not class-discriminative for visual explanation . By contrast , localization approaches like class activation map ( CAM ) ( Zhou et al. , 2016 ) , GradCAM ( Selvaraju et al. , 2017 ) , GradCAM++ ( Chattopadhay et al. , 2018 ) and RISE ( Petsiuk et al. , 2018 ) are highly class-discriminative , namely , localizing image sub-regions reasoned for a prediction class . We refer readers to Sec . 2 for some representative interpretation methods . Besides interpreting CNNs via feature importance maps , some methods zoom into the internal response of neural networks . Examples include network dissection ( Bau et al. , 2017 ) , which evaluates the alignment between individual hidden units and semantic concepts , and learning perceptuallyaligned representations from robust training ( Engstrom et al. , 2019 ) . Very recently , some works ( Xu et al. , 2019b ; a ; Zhang et al. , 2018 ; Subramanya et al. , 2018 ; Ghorbani et al. , 2019 ; Dombrowski et al. , 2019 ; Chen et al. , 2019 ) are beginning to study adversarial robustness by exploring the spectrum between classification accuracy and network interpretability . It was shown in ( Xu et al. , 2019b ; a ) that an imperceptible adversarial perturbation to fool classifiers can lead to a significant change in a class-specific network interpretability map , e.g. , CAM . Thus , it was argued that such an interpretability discrepancy can be used as a helpful metric to differentiate adversarial examples from benign inputs . However , the work ( Zhang et al. , 2018 ; Subramanya et al. , 2018 ) showed that under certain conditions , generating an attack ( which we call interpretability sneaking attack , ISA ) that fools the classifier as well as its coupled interpreter ( in terms of keeping interpretability map highly similar to that of benign input ) is not significantly more difficult than generating adversarial inputs deceiving the classifier only . Besides investigating robustness in classification through the lens of interpretability , the work ( Ghorbani et al. , 2019 ; Dombrowski et al. , 2019 ) studied the robustness in network interpretation maps , showing that which can significantly be manipulated via imperceptible input perturbations but keeping the classifier ’ s decision intact . We call this type of threat model attack against interpretability ( AAI ) . The existing work had no agreement on the relationship between robustness in interpretation and robustness in classification . Spurred by that , we attempt to explore this relationship from both attack and defense perspectives . The most relevant work to ours is ( Chen et al. , 2019 ) , which robustified network interpretation with the aid of integrated gradient ( IG ) , an axiomatic attribution map . It proposed a robust attribution training , which was shown as a principled generalization of previous formulations of robust classification and an effective defense against AAI . In this paper , we first investigate when ISA is possible , and then relate our insights on ISA to robust classification and robust interpretability . Different from the previous work , our paper contains the following contributions . 1 . We provide an answer to the question of when adversarial examples can bypass interpretability discrepancy . We show that enforcing stealthiness of adversarial examples from network interpretation could be challenging . Its difficulty relies on how one measures the interpretability discrepancy caused by input perturbations . 2 . We propose an ` 1 norm based 2-class interpretability discrepancy measure and theoretically show that constraining it helps adversarial robustness . 3 . We develop an interpretability-aware robust training method and empirically show that interpretability alone can be used to defend adversarial attacks for both misclassifcation and misinterpretation . Compared to the IG-based robust attribution training ( Chen et al. , 2019 ) , our approach is simpler in implementation , and provides better robustness even as facing a strong adversary . 2 PRELIMINARIES AND MOTIVATION : INTERPRETABILITY OF CNNS FOR JUSTIFYING A CLASSIFICATION DECISION . To explain what and why CNNs predict , we consider two types of network interpretation methods : a ) class activation map ( CAM ) ( Zhou et al. , 2016 ; Selvaraju et al. , 2017 ; Chattopadhay et al. , 2018 ) and b ) pixel sensitivity map ( PSM ) ( Simonyan et al. , 2013 ; Springenberg et al. , 2014 ; Smilkov et al. , 2017 ; Sundararajan et al. , 2017 ; Yeh et al. , 2019 ) . Let f ( x ) ∈ RC denote a CNN-based predictor that maps an input x ∈ Rd to a probability vector of C classes . Here fc ( x ) , the cth element of f ( x ) , denotes the classification score ( given by logit before the softmax ) for class c. Let L ( x , c ) denote an interpreter ( CAM or PSM ) that reflects where in x contributes to the classifier ’ s decision on c. CAM-type methods . CAM ( Zhou et al. , 2016 ) produces a class-discriminative localization map for CNNs , which performs global averaging pooling over convolutional feature maps prior to the softmax . Let the penultimate layer output K feature maps , each of which is denoted by a vector representation Ak ∈ Ru for channel k ∈ [ K ] . Here [ K ] represents the integer set { 1 , 2 , . . . , K } . The ith entry of CAM LCAM ( x , c ) is given by [ LCAM ( x , c ) ] i = 1 u ∑ k∈ [ K ] wckAk , i , i ∈ [ u ] , ( 1 ) where wck is the linear classification weight that associates the channel k with the class c , and Ak , i denotes the ith element of Ak . The rationale behind ( 1 ) is that the classification score fc ( x ) can be written as the average of CAM values ( Zhou et al. , 2016 ) , fc ( x ) = ∑u i=1 [ LCAM ( x , c ) ] i . For visual explanation , LCAM ( x , c ) is often up-sampled to the input dimension d using bi-linear interpolation . GradCAM ( Selvaraju et al. , 2017 ) generalizes CAM for CNNs without the architecture ‘ global average pooling → softmax layer ’ over the final convolutional maps . Specifically , the weight wck in ( 1 ) is given by the gradient of the classification score fc ( x ) with respect to ( w.r.t . ) the feature map Ak , wck = 1 u ∑u i=1 ∂fc ( x ) ∂Ak , i . GradCAM++ ( Chattopadhay et al. , 2018 ) , a generalized formulation of GradCAM , utilizes a more involved weighted average of the ( positive ) pixel-wise gradients but provides a better localization map if an image contains multiple occurrences of the same class . In this work , we focus on CAM since it is computationally light and our models used in experiments follow the architecture ‘ global average pooling→ softmax layer ’ . PSM-type methods . PSM uses calculations with gradients to assign importance scores to individual pixels toward explaining the classification decision about an input . Examples of commonly-used approaches include vanilla gradient ( Simonyan et al. , 2013 ) , guided backpropogation ( Springenberg et al. , 2014 ) , SmoothGrad ( Smilkov et al. , 2017 ) , and integrated gradient ( IG ) ( Sundararajan et al. , 2017 ) . In particular , IG satisfies the completeness attribution axiom that PSM ought to obey . Specifically , it averages gradient saliency maps for interpolations between an input x and a baseline image a : [ LIG ( x , c ) ] i = ( xi − ai ) ∫ 1 α=0 ∂fc ( a+ α ( x− a ) ) ∂xi dα ≈ ( xi − ai ) m∑ i=1 ∂fc ( a+ i m ( x− a ) ) ∂xi 1 m , i ∈ [ d ] , ( 2 ) where m is the number of steps in the Riemman approximation of the integral . The completeness axiom ( Sundararajan et al. , 2017 , Proposition 1 ) states that ∑d i=1 [ LIG ( x , c ) ] i = fc ( x ) − fc ( a ) , where the baseline image a is often chosen such that fc ( a ) ≈ 0 , e.g. , the black image . Note that CAM also satisfies the completeness axiom . PSM is able to highlight fine-grained details in the image , but is computationally intensive and not very class-discriminative compared to CAM ( Selvaraju et al. , 2017 ) . Interpretability discrepancy caused by adversarial perturbation . Let x′ = x + δ represent an adversarial example w.r.t . x , where δ denotes an adversarial perturbation . By replacing the input image x with x′ , the CNN will be fooled from the true label t to the target ( incorrect ) label t′ . It was recently shown in ( Xu et al. , 2019b ; a ) that the adversary could introduce an evident interpretability discrepancy w.r.t . both the true and the target label in terms of L ( x , t ) vs. L ( x′ , t ) , and L ( x , t′ ) vs. L ( x′ , t′ ) . An illustrative example is provided in Figure 1 . We see that an adversary suppresses the network interpretation w.r.t . the true label but promotes the interpretation w.r.t . the target label . We also observe that compared to IG , CAM and GradCAM++ better localize class-specific discriminative regions . These results reveal two observations on how measuring interpretation discrepancy affects classification robustness : a ) interpretability discrepancy may be used to detect adversarial examples , b ) interpretability discrepancy itself may be vulnerable to adversarial perturbations . In what follows , we explore the spectrum between adversarial robustness and interpretability from a unified perspective considering both the adversarial vulnerability of interpretability discrepancy and the value of interpretability discrepancy in a defense .

Summary: In summary, this paper studies if interpretation robustness (i.e., similar examples should have similar interpretation) can help enhance the robustness of the model, especially in terms of adversarial attacks. The study direction itself is interesting and very useful for the interpretation and adversarial attack community. Moreover, some promising results can be observed in part of the empirical study. However, this paper can be improved a lot as follows.