Certifiably Robust Interpretation in Deep Learning

Deep learning interpretation is essential to explain the reasoning behind model predictions . Understanding the robustness of interpretation methods is important especially in sensitive domains such as medical applications since interpretation results are often used in downstream tasks . Although gradient-based saliency maps are popular methods for deep learning interpretation , recent works show that they can be vulnerable to adversarial attacks . In this paper , we address this problem and provide a certifiable defense method for deep learning interpretation . We show that a sparsified version of the popular SmoothGrad method , which computes the average saliency maps over random perturbations of the input , is certifiably robust against adversarial perturbations . We obtain this result by extending recent bounds for certifiably robust smooth classifiers to the interpretation setting . Experiments on ImageNet samples validate our theory . 1 INTRODUCTION . 1.1 MOTIVATION . The growing use of deep learning in a wide range of highly-sensitive applications , such as autonomous driving , medicine , finance , and even the legal system ( Teichmann et al. , 2018 ; Quellec et al. , 2017 ; Fischer & Krauss , 2018 ; Nissan , 2017 ) , raises concerns about human trust in machine learning systems . Interpretations , which explain why certain predictions are made , are critical for establishing this trust between users and the machine learning system . Moreover , the interpretation results themselves are often used for downstream tasks . For example , gradient-based saliency maps have been used in medical applications : Quellec et al . ( 2017 ) uses gradient-based interpretation techniques to highlight signs of retinal disease in retinal fundus photographs ( see Figure 1-b . ) Similarly , BenTaieb et al . ( 2016 ) uses a gradient-based saliency map as part of a pipeline to automatically analyse histopathology slides of colon tumours . Techniques like these are also used in medical research to annotate datasets ( Lahiani et al. , 2019 ) . Outside of the medical field , gradientbased interpretations can be used in general-purpose image segmentation tasks ( Hong et al. , 2015 ) . However , Ghorbani et al . ( 2019 ) has shown that several gradient-based interpretation methods are sensitive to adversarial examples , obtained by adding a small perturbation to the input image . These adversarial examples maintain the original class label while greatly distorting the saliency map ( Figure 1-a : see ‘ Gradient ’ interpretation method ) . Although adversarial attacks and defenses on image classification have been studied extensively in recent years ( Szegedy et al. , 2014 ; Uesato et al. , 2018 ; Goodfellow et al. , 2015 ; Athalye et al. , 2018a ; b ; Buckman et al. , 2018 ; Kurakin et al. , 2017 ; Papernot & McDaniel , 2016 ; Carlini & Wagner , 2017 ; Madry et al. , 2018 ) , less attention has been paid to effectively defending deep learning interpretation against adversarial examples ( Lee et al. , 2019 ; Etmann et al. , 2019 ) . This is partially due to the difficulty of protecting high-dimensional saliency maps compared to defending a class label , as well as to the lack of a ground truth for interpretation . Interestingly , we have observed that the standard adversarial training used for the classification robustness does not lead to robust interpretations ( see Section 4 ) . This calls for better understanding the robustness of interpretation methods and developing defenses with performance guarantees . 1.2 PROPOSED APPROACH AND MAIN CONTRIBUTIONS . In the last couple of years , several approaches have been proposed for interpreting neural network outputs ( Simonyan et al. , 2014 ; Sundararajan et al. , 2017 ; Alvarez-Melis & Jaakkola , 2018 ; Adebayo et al. , 2018 ; Huang et al. , 2017 ) . Specifically , Simonyan et al . ( 2014 ) computes the elementwise absolute value of the gradient of the largest class score with respect to the input . To define some notation , let g ( x ) be this most basic form of the gradient-based saliency map , for an input image x ∈ Rn . For simplicity , we also assume that elements of g ( x ) have been linearly normalized to be between 0 and 1. g ( x ) represents , to a first order linear approximation , the importance of each pixel in determining the class label . Numerous variations of this method have been introduced , which we review in the appendix . A popular saliency map method which extends the basic gradient method is SmoothGrad ( Smilkov et al. , 2017 ) , which takes the average gradient over random perturbations of the input . Formally , we define the smoothing function as : ḡ ( x ) ∶= E [ g ( x + ) ] , ( 1.1 ) where has a normal distribution ( i.e . ∼ N ( 0 , σ2I ) ) . We will discuss other smoothing functions in Section 2.3 while the empirical smoothing function which computes the average over finitely many perturbations of the input will be discussed in Section 2.2 . We refer to the basic method described in the above equation as the scaled SmoothGrad 1 . Since a ground truth label for interpretation is not available , we use a similarity metric between the original and perturbed saliency maps as an estimate of the interpretation robustness . Ghorbani et al . ( 2019 ) introduced a top-K overlap metric for this purpose : define R ( x , x̃ , K ) as the number of overlapping elements between top K largest elements of saliency maps of x and the perturbed image x̃ . Ghorbani et al . ( 2019 ) developed an L∞ attack on R ( x , x̃ , K ) , which we adapt to the L2 case to test empirical robustness : see the appendix for details . This top-K overlap metric is naturally motivated : absolute magnitude has little meaning in saliency maps , which are nearly always presented as normalized . Moreover , it is common ( Sundararajan et al. , 2017 ) to clip the highest values when displaying saliency maps , so that more than a few pixels are clearly visible . This suggests that relative ranks of salience values ( and not their absolute values ) are important for interpretation . Note that for an input x , R ( x , x̃ , K ) depends on the specific perturbation x̃ . We define R∗ ( x , K ) as the robustness measure with respect to the worst perturbation of x . That is , R∗ ( x , K ) ∶= min x̃ R ( x , x̃ , K ) ( 1.2 ) ∥x̃ − x∥2 ≤ ρ . For deep learning models , this optimization is non-convex in general . Thus , characterizing the true robustness of interpretation methods is likely intractable for most modern classifier models . In this paper , we show that a lower bound on the true robustness value of an interpretation method ( i.e . a robustness certificate ) can be computed efficiently . In other words , for a given input x , we compute a robustness certificateRcert such thatRcert ( x , K ) ≤ R∗ ( x , K ) . To establish the robustness certificate for saliency map methods , we prove the following result for a general function h ( . ) whose range is between 0 and 1 : Theorem 2 . Let h ( x ) be the output of an interpretation method whose range is between 0 and 1 and let h̄ be its smoothed version defined as in equation 1.1 . Let h̄i ( x ) and h̄ [ i ] ( x ) be the i-th element and the i-th largest elements of h̄ ( x ) , respectively . Let Φ be the cdf of the normal distribution . If Φ ( Φ−1 ( h̄ [ i ] ( x ) ) − 2ρ σ ) ≥ h̄ [ 2K−i ] ( x ) , ( 1.3 ) then for the smoothed interpretation method , we have Rcert ( x , K ) ≥ i . Intuitively , this means that , if there is a sufficiently large gap between the i-th largest element of the smoothed saliency map and its ( 2K − i ) -th largest element , then we can certify that at least i elements in the top K largest elements of the original smoothed saliency map will also be in the top K elements of adversarially perturbed saliency map . We present a more general version of this result with empirical expectations for smoothing in Section 2 , as well as another rank-based robustness certificate in Section 3 . The proof of this bound relies on an extension of Cohen et al . ( 2019 ) which addresses certified robustness in the classification case . Proofs for all theorems are given in the appendix . Evaluating the robustness certificate for the scaled SmoothGrad method on ImageNet samples produces vacuous bounds ( Figure 1-c ) . This motivated us to develop variations of SmoothGrad with larger robustness certificates . One such variation is Sparsified SmoothGrad which is defined by smoothing a sparsification function that maps the largest elements of g ( x ) to one and the rest to zero . Sparsified SmoothGrad obtains a considerably larger value of the robustness certificate ( Figure 1-c ) while producing high-quality saliency maps ( Figure 3 ) . We study other variations of Sparsified SmoothGrad in Section 2 . A considerably different certification result for robust interpretation is provided by Lee et al . ( 2019 ) . That work provides a certified L2 radius in which a first-order gradient saliency map is exactly identical to the saliency map for the unperturbed image . The method depends on the network structure being locally linear ( ReLU-based ) : the inner radius of the locally linear region is directly calculated . 1The original definition of SmoothGrad does not normalize and take the absolute values of gradient elements before averaging . We start with the definition of equation 1.1 because it can more easily be compared to our certifiably robust interpretation scheme discussed in Section 2 . By contrast , our method makes no demands on the structure of the classifier , and could be applied to saliency maps generated by methods other than computing simple gradients : any bounded saliency function can be used as h ( x ) in our Theorem 1 . Additionally , our certification method is computationally inexpensive enough to allow certificates to be computed on ImageNet-scale samples . 2 SMOOTHING FOR CERTIFIABLY ROBUST INTERPRETATION . 2.1 NOTATION . We introduce the following notations to indicate Gaussian smoothing : for a function h , we define population and empirical smoothed functions , respectively , as : h̄ ( x ) =E ∼N ( 0 , σ2I ) [ h ( x + ) ] , h̃ ( x ) = 1 q q ∑ i=1 h ( x + i ) , i ∼ N ( 0 , σ 2I ) . ( 2.1 ) In other words , h̄ ( x ) represents the expected value of h ( x ) when smoothed under normal perturbations of with some standard deviation σ while h̃ ( x ) represents an empirical estimate of h̄ ( x ) using q samples . We call σ2 the smoothing variance and q the number of smoothing perturbations . We use vi to denote the ith element of the vector v. Similarly hi ( x ) denotes the ith element of the output h ( x ) . We also define , for any h ( x ) , rank ( h ( x ) , i ) as the ordinal rank of hi ( x ) in h ( x ) ( in the descending order ) : rank ( h ( x ) , i ) = j denotes that hi ( x ) is the jth largest element in h ( x ) . We use x [ i ] to denote the ith largest element in x . If i is not an integer , the ceiling of i is used .

Summary: This paper introduces an extension of Cohen et al. (2019)’s result that allows one to derive robustness certificates for interpretation methods, as well as a bound on the top-K overlap of saliency methods. These results motivate the introduction of Sparsified SmoothGrad and a relaxation of this method that has differentiable elements. These introduced approaches adapt previous methods so the derived bounds are applicable. The proposed methods are shown to perform as well as Quadratic SmoothGrad (Smilkov et al. 2017) in CIFAR-10 experiments.