Robust saliency maps with distribution-preserving decoys

1 INTRODUCTION . Deep neural networks ( DNNs ) deliver remarkable performance in an increasingly wide range of application domains , but they often do so in an inscrutable fashion , delivering predictions without accompanying explanations . In a practical setting such as automated analysis of pathology images , if a patient sample is classified as malignant , then the physician will want to know which parts of the image contribute to this diagnosis . Thus , in general , a DNN that delivers explanations alongside its predictions will enhance the credibility and utility of its predictions for end users ( Lipton , 2016 ) . Two primary classes of methods have been developed for interpreting CNN models . The first class of methods , often referred to as saliency maps , use gradients of classification outcome as a per-pixel importance score to characterize the influence of individual pixels to the prediction ( Simonyan et al. , 2013 ; Selvaraju et al. , 2016 ; Binder et al. , 2016 ; Shrikumar et al. , 2016 ; Smilkov et al. , 2017 ; Sundararajan et al. , 2017 ; Levine et al. , 2019 ) . Despite providing apparently meaningful interpretations in practice , these methods exhibit fundamental limitations . First , saliency maps evaluate pixel-wise importance in an isolated fashion by design , implicitly assuming that other pixels are fixed ( Singla et al. , 2019 ) . Worse still , the presence of gradient saturation often breaks the assumption that important features in general correspond to large gradients ( Sundararajan et al. , 2016 ; Shrikumar et al. , 2016 ; Smilkov et al. , 2017 ) . Specifically , an important pixel may contribute a strong joint effect on the prediction when considered in conjunction with neighboring pixels . However , that important feature may have a tiny gradient locally , in the sense that its marginal contribution is diminishing due to the flattened output of the layer . Finally , Ghorbani et al . ( 2017 ) and Kindermans et al . ( 2017 ) systematically revealed the fragility of widely-used saliency methods by showing that even an imperceivable perturbation or simple shift transformation of the input data can lead to a large change in the resulting saliency scores . In light of this observation , a key challenge for any saliency method is ensuring that the saliency scores are robust to gradient saturation and perturbations . A second class of interpretability methods , often based on counterfactual , perturb ( i.e. , nullifying , blurring , adding noise , or inpainting ) small regions of the image to modify the classifier prediction ( Fong & Vedaldi , 2017 ; Dabkowski & Gal , 2017 ; Chang et al. , 2017 ; Fan et al. , 2017 ; Chang et al. , 2019 ; Yousefzadeh & O ’ Leary , 2019 ; Goyal et al. , 2019 ) . Despite identifying meaningful regions in practice , these methods exhibit several limitations . First , counterfactual-based methods implicitly assume that regions containing the object are the ones most contributed to the prediction ( Fan et al. , 2017 ) . However , Moosavi-Dezfooli et al . ( 2017 ) revealed that counterfactual-based methods are also vulnerable to the adversarial attacks , which force these explanation methods to output nonrelated backgrounds rather than the meaningful objects as important subregions . In addition , the counterfactual images may be potentially far away from the distribution from which the training samples were drawn ( Burns et al. , 2019 ) , causing the classifier behavior ill-defined . Though some efforts have been made to train the saliency extractor and the classifier simultaneously ( Fan et al. , 2017 ; Zołna et al. , 2019 ) , the robustness of the resulting explanations are not guaranteed in most cases ( Hendrycks & Dietterich , 2019 ) . In light of this , for any perturbation-based method , a key challenge is ensuring that the perturbations are effective yet preserving the training distribution . In this work , we propose a data-driven technique to infer robust saliency maps with distributionpreserving perturbations . Given an image of interest , the core idea of this method is to generate a population of perturbed images , referred to as decoy images , that resemble the neural network ’ s intermediate representation of the original image but are conditionally independent of its label . The resulting decoy images capture the variation of the image data originating from either sensor noise or adversarial attacks . The major differences between the proposed and counterfactual-based methods are threefold . First , unlike counterfactual images that seek the minimum set of features to exclude so as to minimize the prediction score or to include so as to maximize the prediction score ( Fong & Vedaldi , 2017 ) , saliency map methods like the one we propose in this paper aim to characterize the influence of each feature on the prediction score . Second , unlike counterfactual images that are optimized toward the aforementioned objective with respect to the decision boundary , decoys are designed to be independent of labels , without changing the decision boundary at all . Finally , unlike counterfactual images which could potentially be out-of-distribution by adding noise ( Smilkov et al. , 2017 ) , rescaling ( Sundararajan et al. , 2017 ) , blurring ( Fong & Vedaldi , 2017 ) , or inpainting the image ( Chang et al. , 2017 ) , decoys are plausibly constructed in the sense that their intermediate representations are non-discriminative from the original input data by design . In addition , we propose a novel decoy-enhanced saliency score , which considers an ensemble of saliency maps across multiple decoy images . We also derive a closed-form formula for the decoyenhanced saliency score generated from a single-layer convolutional neural network . This analysis shows that the proposed saliency score for each pixel can be decomposed into two components : the range of values for a given pixel among the decoys and the number of neurons jointly activated by that pixel and its neighbors . The former compensates for gradient saturation ( Sundararajan et al. , 2016 ) in the sense that an important pixel can vary without strongly affecting the prediction , due to the joint effect of nearby pixels . The latter compensates for the way in which saliency maps treat each pixel independently ( Singla et al. , 2019 ) in the sense that the importance of a pixel is jointly determined by the importance of pixels in its vicinity . This closed-form result may provide insights to help design better defensive strategies against adversarial attacks . We apply the proposed method to the ImageNet dataset ( Russakovsky et al. , 2015 ) in conjunction with three standard saliency methods . We demonstrate empirically that decoy-enhanced saliency scores perform better than the original saliency scores , both qualitatively and quantitatively , even in the presence of various adversarial perturbations to the image . 2 PROBLEM SETTING . Consider a multi-label classification task in which a pre-trained neural network model implements a function F : Rd 7→ RC that maps from the given input x ∈ Rd to C predicted classes . The score for each class c ∈ { 1 , · · · , C } is F c ( x ) , and the predicted class is the one with maximum score , i.e. , arg maxc∈ { 1 , ··· , C } F c ( x ) . A common instance of this setting is image classification , in which case the input x corresponds to the pixels of an image . A saliency method aims to find the subset of features in the input x that most strongly lead the neural network to make its prediction . Thus , each pixel is assigned a saliency score , encoded in an explanation map E : Rd 7→ Rd , in which the pixels with higher scores represent higher “ importance ” to the final prediction . A variety of saliency methods have been proposed in the literature . Some , such as edge or other generic feature detectors ( Adebayo et al. , 2018 ) , are independent of the predictive model . In this work , we focus on three methods that do depend on the predictor . The vanilla gradient method ( Simonyan et al. , 2013 ) simply calculates the gradient of the class score with respect to the input x , which is defined as Egrad ( x ) = OxF c ( x ) . The SmoothGrad method ( Smilkov et al. , 2017 ) seeks to reduce noise in the saliency map by averaging over explanations of noisy copies of an input , defined as Esg ( x ) = 1 N ∑N i=1Egrad ( x+gi ) with noise vectors gi ∼ N ( 0 , σ2 ) . The integrated gradients method ( Sundararajan et al. , 2017 ) aims to tackle the problem of gradient saturation . The method starts from a baseline input x0 and sums over the gradient with respect to scaled versions of the input ranging from the baseline to the observed input , defined as Eig ( x ) = ( x− x0 ) × ∫ 1 0 OxF c ( x0 + α ( x− x0 ) ) dα . In this setting , we are given three inputs : a pre-trained neural network model F with L layers , an image of interest x , and a saliency method E such that E ( x ; F ) is a saliency map of the same dimensions as x . As illustrated in Figure 1 , the robust saliency map can be obtained in two steps : 1 ) constructing decoy images , and 2 ) computing decoy-enhanced saliency scores . 3 DEFINITION OF DECOYS Say that F ` : Rd 7→ Rd ` is the function instantiated by the given network , which maps from an input image x ∈ Rd to its intermediate representation F ` ( x ) ∈ Rd ` at layer ` ∈ { 1 , 2 , · · · , L } . At a specified layer ` , the random vector x̃ ∈ Rd is said to be a decoy of x ∈ Rd if the following swappable condition is satisfied : F ` ( x ) = F ` ( xswap ( x̃ , P ) ) , for image patches P ⊂ { 1 , · · · , d } , ( 1 ) The swap ( x̃ , P ) operation swaps image patches between x and x̃ ( Figure 2 ) . Hence , if P = { 10 } , then xswap ( x̃ , P ) is a new image which is almost equivalent to x except that the 10th patch is exchanged from x̃ . A “ valid ” patch is a local region receptive to the convolutional filter of a given CNN , for example , the 3 × 3 area in VGGNet ( Simonyan & Zisserman , 2014 ) . The swappable condition ensures that the original image x and its decoy x̃ are indistinguishable in terms of the intermediate representation at layer ` . Note in particular that the construction of decoys relies solely on the first ` layers of the neural network F1 , F2 , · · · , F ` and is independent of the succeeding layers F ` +1 , · · · , FL . In other words , x̃ is conditionally independent of the classification task F ( x ) given the input x ; i.e. , x̃ |= F ( x ) |x . See Supplementary Section A6 for the explanation why decoys exist .

Summary: Given a model and an image, the proposed method generates perturbations of the image, such that the model output in intermediate layers of the network does not change. Their method first generates such perturbed images, and then uses those to generate a saliency map in order to interpret the classification.  The idea is overall interesting, but its relationship with many other methods is not adequately examined. Significant portion of recent literature is ignored, especially studies on counterfactuals and decision boundaries in the context of image classification.  For numerical results, they compare their results with raw scores generated by three other methods, but the results are quite thin and disorganized with respect to adversarial robustness and interpretability.