Rectified Gradient: Layer-wise Thresholding for Sharp and Coherent Attribution Maps

Saliency map , or the gradient of the score function with respect to the input , is the most basic means of interpreting deep neural network decisions . However , saliency maps are often visually noisy . Although several hypotheses were proposed to account for this phenomenon , there is no work that provides a rigorous analysis of noisy saliency maps . This may be a problem as numerous advanced attribution methods were proposed under the assumption that the existing hypotheses are true . In this paper , we identify the cause of noisy saliency maps . Then , we propose Rectified Gradient , a simple method that significantly improves saliency maps by alleviating that cause . Experiments showed effectiveness of our method and its superiority to other attribution methods . Codes and examples for the experiments will be released in public . 1 INTRODUCTION . The gradient of the score function with respect to the input , also called the saliency map ( Erhan et al. , 2009 ; Baehrens et al. , 2010 ; Simonyan et al. , 2014 ) , is the most basic means of interpreting deep neural networks ( DNNs ) . It is also a baseline method for other advanced attribution-based methods . However , our understanding of saliency maps is still poor . Previous studies such as Springenberg et al . ( 2015 ) and Selvaraju et al . ( 2017 ) have noted that saliency maps tend to be visually noisy . To explain this phenomenon , Sundararajan et al . ( 2016 ) and Smilkov et al . ( 2017 ) suggested saturation and discontinuous gradients as the causes ( see Section 2.1 for further explanation ) . There were several studies attempting to improve saliency maps by tackling these hypothesized causes ( Bach et al. , 2015 ; Montavon et al. , 2017 ; Sundararajan et al. , 2016 ; Shrikumar et al. , 2017 ; Smilkov et al. , 2017 ; Sundararajan et al. , 2017 ) . Even though such attribution methods generally produce better visualizations , we find troubling that the hypotheses regarding noisy saliency maps have not been rigorously verified ( see Section 2.2 for more detail on attribution methods ) . In other words , numerous attribution methods were built upon unproven claims that gradient discontinuity or saturation truly causes saliency maps to be noisy . This situation gives rise to two major problems . First , if the hypotheses regarding noisy saliency maps are incorrect , current and future works based on those hypotheses will also be erroneous . Second , as we do not know precisely why saliency maps are noisy , we have to rely on heuristics and guessworks to develop better attribution methods . In this paper , we address these problems by identifying saliency maps are noisy because DNNs do not filter out irrelevant features during forward propagation . We then introduce Rectified Gradient , or RectGrad in short , a simple technique that significantly improves the quality of saliency maps by alleviating the cause through layer-wise thresholding during backpropagation . Finally , we demonstrate that RectGrad produces attributions qualitatively superior and quantitatively comparable to other attribution methods . Specifically , we have the following key contributions : • We explain why saliency maps are noisy . Noise occurs in saliency maps when irrelevant features have positive pre-activation values and consequently pass through ReLU activation functions . This causes gradients to be nonzero at unimportant regions . We perform experiments with networks trained on CIFAR-10 to justify our claims ( Section 3 ) . • We introduce Rectified Gradient , a method that removes noise from saliency maps by thresholding irrelevant units at ReLU binary gates during backpropagation ( Section 4 ) . We first explain the rationale behind Rectified Gradient ( Section 4.1 ) . We then prove that Rectified Gradient generalizes Deconvolution and Guided Backpropagation ( Section 4.2 ) . In addition , we discuss two techniques that enhance the visual quality of Rectified Gradient attribution maps ( Appendix C ) . • We first investigate the effect of threshold level on attribution maps produced by Rectified Gradient ( Section 5.1 ) . Then , we apply Rectified Gradient to networks trained on CIFAR10 and ImageNet to demonstrate that it produces qualitatively superior attribution maps ( Section 5.2 ) . We also compare Rectified Gradient with other attribution methods using several quantitative metrics ( Section 5.3 ) . 2 BACKGROUND OVERVIEW . Let S : Rd 7→ R|C| be an image classification network , where x ∈ Rd is a single image instance and C is the set of image classes . Then , we can define a score function Sc : Rd 7→ R for each class c ∈ C and the final class of the image x is given by class ( x ) = argmaxc∈C Sc ( x ) . A typical score function is constructed by alternately composing affine transformations and nonlinear activation functions . A squashing function such as softmax is applied to the final layer . Since functions comprising Sc are differentiable or piecewise linear , the score function is also piecewise differentiable . Using this fact , Erhan et al . ( 2009 ) , Baehrens et al . ( 2010 ) and Simonyan et al . ( 2014 ) proposed the saliency map , or the gradient of Sc with respect to x , to highlight features within x that the network associates with the given class . In an ideal case , saliency maps highlight objects of interest . However , previous studies such as Springenberg et al . ( 2015 ) and Selvaraju et al . ( 2017 ) have pointed out that saliency maps tend to be visually noisy , as verified by Figure 1 . Three hypotheses were proposed to account for this phenomenon . We describe them in the next section . 2.1 PREVIOUS HYPOTHESES . Saliency Maps are Truthful . Smilkov et al . ( 2017 ) suggested that noisy saliency maps are faithful descriptions of what the network is doing . That is , pixels scattered seemingly at random are actually crucial to how the network makes a decision . In short , this hypothesis claims that noise is actually informative . Discontinuous Gradients . Smilkov et al . ( 2017 ) and Shrikumar et al . ( 2017 ) proposed that saliency maps are noisy due to the piece-wise linearity of the score function . Specifically , since typical DNNs use ReLU activation functions and max pooling , the derivative of the score function with respect to the input will not be continuously differentiable . Under this hypothesis , noise is caused by meaningless local variations in the gradient . Saturating Score Function . Shrikumar et al . ( 2017 ) and Sundararajan et al . ( 2017 ) suggested that important features may have small gradient due to saturation . In other words , the score function can flatten in the proximity of the input and have a small derivative . This hypothesis explains why informative features may not be highlighted in the saliency map even though they contributed significantly to the decision of the DNN . 2.2 PREVIOUS WORKS ON IMPROVING SALIENCY MAPS . DNN interpretation methods that assign a signed attribution value to each input feature are collectively called attribution methods . Attributions are usually visualized as a heatmap by arranging them to have the same shape as the input sample . Such heatmaps are called attribution maps . We now describe attribution methods that have been proposed to improve saliency maps . Attribution Methods Addressing Discontinuity . SmoothGrad ( Smilkov et al. , 2017 ) attempts to smooth discontinuous gradient with a Gaussian kernel . Since calculating the local average in a high dimensional space is intractable , the authors proposed a stochastic approximation which takes random samples in a neighborhood of the input x and then averages their gradients . Attribution Methods Addressing Saturation . Since saliency maps estimate the local importance of each input feature , they are vulnerable to saturation . Therefore , attribution methods such as Gradient * Input ( Shrikumar et al. , 2017 ) , Layer-wise Relevance Propagation ( LRP ) ( Bach et al. , 2015 ) , DeepLIFT ( Shrikumar et al. , 2017 ) and Integrated Gradient ( Sundararajan et al. , 2017 ) attempt to alleviate saturation by estimating the global importance of each pixel ( Ancona et al. , 2018 ) . Ancona et al . ( 2018 ) has also shown that several global attribution methods are closely related under certain conditions . Other Attribution Methods . Some attribution methods take a different approach to improving saliency maps . Deconvolution ( Zeiler & Fergus , 2014 ) and Guided Backpropagation ( Springenberg et al. , 2015 ) remove negative gradient during backpropagation . Due to this imputation procedure , Deconvolution and Guided Backpropagation yield attribution maps sharper than those of other methods . However , Nie et al . ( 2018 ) has recently proven that these methods are actually doing partial image recovery which is unrelated to DNN decisions . 3 OUR EXPLANATION FOR NOISY SALIENCY MAPS . For brevity , we refer to pixels on the background as background features and pixels on the object as object features . Then , noise in a saliency map corresponds to background gradient , or gradient that highlights background features . We assume the DNN uses ReLU activation functions . Under this condition , nonzero background gradient indicates the presence of at least one positive pre-activation in each network layer corresponding to background features . To verify this , we visualized intermediate layer activations of a convolutional neural network ( CNN ) trained on CIFAR-10 . Figure 2b shows convolutional layer feature maps for an image that produced a noisy saliency map . Since CNN filters act as feature extractors , we expected the CNN to remove most background feature activations through convolutions . However , we found significant amounts of background feature activations in all convolution layers . As the last convolution layer is connected to fully connected layers , the majority of activations in the last convolution layer will have nonzero gradient . Hence , the gradient flowed through background feature activations up to the input . This gradient flow caused background gradient , as shown in Figure 2a . From our perspective , the answer to “ why are saliency maps noisy ? ” is trivial . Saliency maps are noisy because background features pass through ReLU activation functions . Therefore , rather than asking why saliency maps are noisy , we ask “ do activations of background features highlighted by background gradient have nontrivial influence on the decision ? ” If the answer is yes , noise in saliency maps is informative as suggested by Smilkov et al . ( 2017 ) , and saliency maps do not need any major improvement . However , if the answer is no , we should find a way to remove background gradient . We investigated this question through two experiments . Feature Map Occlusion . We evaluated the significance of background features by occluding activations at intermediate layers . Then , we analyzed the effect of this perturbation on the final decision . Note that this is different from the Sensitivity metric ( Bach et al. , 2015 ; Samek et al. , 2017 ) . Sensi- tivity measures the impact of occlusion in the data space ( e.g . pixel occlusion ) while we measured the impact of occlusion in each feature space . We first created a background mask that covers background features in the images . We then plotted the average class logits as we incrementally occluded intermediate layer activations that fell on the background mask . We carried out occlusion following a random ordering and took the average over 50 trials . Figures 3a and 3b give an example of a background mask and a completely occluded feature map respectively . Figure 3c shows that the final decision did not change throughout the occlusion process for all convolution layers . Moreover , the difference between the top label logit and the next largest logit remained constant . Therefore , background feature activations are irrelevant to the classification task . To further support this claim , we conducted a larger-scale version of this experiment , and we describe the procedure and results in Appendix A.1 . Training Dataset Occlusion . Next , we show that gradient can be nonzero for completely uninformative features . We occluded the upper left corner of all images in the training dataset with a 10× 10 random patch and trained a randomly initialized CNN on the modified dataset . We used the same patch for all images . Since the test accuracy did not change significantly ( 79.4 % to 79.3 % ) , we expected the CNN to have learned to extract important features and ignore irrelevant ones . However , Figure 4 shows that gradient is nonzero for the patch although it is completely irrelevant to the classification task . We can draw three conclusions from these experiments : 1 . DNNs do not filter out irrelevant features during forward propagation . 2 . DNNs are capable of making correct decisions even if we occlude the majority of back- ground feature activations in intermediate layers . This implies that most background feature activations are irrelevant to the classification task . 3 . Since DNNs do not remove irrelevant features through ReLU activation functions , zero threshold at ReLU binary gates during backpropagation also allow irrelevant information to flow through the gradient . With the conclusions above , we can refute the first of three previous hypotheses . As for the second hypothesis , we can interpret meaningless local variation in the gradient as a side effect of irrelevant features contaminating the gradient . Why the network does not learn to filter out irrelevant features is a matter of optimization , which is out of scope of this paper . However , we believe it is a phenomenon worth investigating .

Summary: This paper studies how to better visually interpret a deep neural network. It proposes a new method to produce less noisy saliency maps, named RectGrad. RectGrad thresholds gradient during backprop in a layer-wise fashion in a similar manner to a previous work called Guided Backprop. The difference is that Guided Backprop employs a constant threshold, i.e. 0, while RectGrad uses an adaptive threshold based on a percentile hyper-parameter. The paper is well-written, including a comprehensive review of previous related works, an meaningful meta-level discussion for motivation, and a clear explanation of the proposed method.