Efficient Saliency Maps for Explainable AI

We describe an explainable AI saliency map method for use with deep convolutional neural networks ( CNN ) that is much more efficient than popular gradient methods . It is also quantitatively similar and better in accuracy . Our technique works by measuring information at the end of each network scale which is then combined into a single saliency map . We describe how saliency measures can be made more efficient by exploiting Saliency Map Order Equivalence . Finally , we visualize individual scale/layer contributions by using a Layer Ordered Visualization of Information . This provides an interesting comparison of scale information contributions within the network not provided by other saliency map methods . Since our method only requires a single forward pass through a few of the layers in a network , it is at least 97x faster than Guided Backprop and much more accurate . Using our method instead of Guided Backprop , class activation methods such as Grad-CAM , Grad-CAM++ and Smooth Grad-CAM++ will run several orders of magnitude faster , have a significantly smaller memory footprint and be more accurate . This will make such methods feasible on resource limited platforms such as robots , cell phones and low cost industrial devices . This will also significantly help them work in extremely data intensive applications such as satellite image processing . All without sacrificing accuracy . Our method is generally straight forward and should be applicable to the most commonly used CNNs 1 . 1 INTRODUCTION . Deep neural networks ( DNN ) have provided a new burst of research in the machine learning community . However , their complexity obfuscates the underlying processes that drive their inferences . This has lead to a new field of explainable AI ( XAI ) . A variety of tools are being developed to enable researchers to peer into the inner workings of DNNs . One such tool is the XAI saliency map . It is generally used with image or video processing applications and is supposed to show what parts of an image or video frame are most important to a network ’ s decisions . The seemingly most popular methods derive a gradient saliency map by back-propagating a gradient from the end of the network and project it onto an image plane ( Simonyan et al. , 2014 ; Zeiler & Fergus , 2014 ; Springenberg et al. , 2015 ; Sundararajan et al. , 2017 ; Patro et al. , 2019 ) . The gradient can typically be from a loss function , layer activation or class activation . Thus , it requires storage of the data necessary to compute a full backward pass on the input image . Several newer methods attempt to iteratively augment the image or a mask in ways that affect the precision of the results ( Fong & Vedaldi , 2017 ; Chang et al. , 2018 ) . Additionally , saliency map encoders can be trained within the network itself ( Dabkowski & Gal , 2017 ) . Both of these methods have a distinct advantage of being more self-evidently empirical when compared with gradient techniques . Class Activation Map ( CAM ) methods ( Selvaraju et al. , 2017 ; Chattopadhyay et al. , 2018 ; Omeiza et al. , 2019 ) efficiently map a specified class to a region in an image , but the saliency map is very coarse . They generally use a method like Guided Backprop ( Springenberg et al. , 2015 ) to add finer pixel level details . This requires a full backwards pass through the network , and it adds significant memory and computational overhead to CAM solutions relative to just computing the CAM alone . Several of the CAM methods compute gradients aside from the use of Guided Backprop , but we will differentiate them by referring to them as CAM methods . 1For reproducibility , full source code is available at http : //www.anonymous.submission.com 1.1 EFFICIENCY AND WHY IT MATTERS HERE . Most saliency map methods require many passes through the network in order to generate results or train . The gradient methods hypothetically would require just one backwards pass , but often require as many as 15 in order to give an accurate rendering ( Hooker et al. , 2018 ) . This is not always a problem in the lab when a person has a powerful GPU development box . However , what if one would like to see the workings of the network at training time or even run time ? This can be a serious hurdle when running a network on mobile or embedded platforms . It is not uncommon for hardware to be barely fast enough to process each image from a video source . Running a full backward pass can lead to dropped frames . Additionally , viewing or saving the saliency map for each frame is infeasible . Another problem is that some platforms may not have the memory capacity to save all the information required for a backward pass . Gradient based methods can not work in these instances . Sometimes this can even be the case with powerful hardware . Satellite images can be vary large and potentially exhaust generous resources . An efficient method would enable developers in these areas to get feedback in the field and aid in debugging or understanding the behavior of a network . Here we show a method of computing an XAI saliency map which is highly efficient . The memory and processing overhead is several orders of magnitude lower than the commonly used gradient methods . This makes it feasible in any resource limited environment . Also , since we demonstrate empirically that our method is either similar or more accurate than the most commonly used gradient methods , it can be used it to speed up run-time in any situation . It is fast enough that we already use it automatically when training networks . We notice very little degradation of training speed . 2 METHODS . 2.1 SALIENCY MAP DERIVATION . We were looking for a method to compute saliency maps based on certain conditions and assumptions . 1 . The method used needs to be relatively efficient to support rapid analysis at both test time and during DNN training . 2 . The method should have a reasonable information representation . As a DNN processes data , the flow of information should become localized to areas which are truly important . 3 . The method should capture the intuition that the informativeness of a location is proportional to the overall activation level of all the filters as well their variance . That is , informative activations should have a sparse pattern with strong peaks . Our approach works by creating saliency maps for the output layer of each scale in a neural network and then combines them . We can understand scale by noting that the most commonly used image processing DNNs work on images with filter groups at the same scale which down-sample the image and pass it to the group of filters at the next scale , and so on . Given a network like ResNet-50 ( He et al. , 2015 ) with in input image size of 224x224 , we would have five scale groups of size 112x112 , 56x56 , 28x28 , 14x14 and 7x7 . It is at the end of these scale groups where we are interested in computing saliency . In this way , our approach is efficient and is computed during the standard forward pass through the network . No extra pass is needed . To achieve localization of information , we measure statistics of activation values arising at different input locations . Given an output activation tensor T ∈ R+ , p×q×r with spatial indices i , j ∈ p , q and depth index k ∈ r from some layer T = l ( X ) . In our case l ( . ) is a ReLU ( Nair & Hinton , 2010 ) . We apply a function to each column at i , j over all depths k. This yields a 2D saliency map S ∈ R+ , p×q where S = ϕ ( T ) . We process the tensor after it has been batch-normalized ( Ioffe & Szegedy , 2015 ) and processed by the activation function . When we compute Truncated Normal statistics as an alternative in later section , we take the tensor prior to the activation function . Finally , to capture our intuition about the informativeness of an output activation tensor , we derived ϕ ( . ) by creating a special simplification of the maximum likelihood estimation Gamma Scale parameter ( Choi & Wette , 1969 ) . One way we can express it is : . ϕ ( . ) = 1 r · r∑ k=1 xk · [ log2 ( 1 r · r∑ k=1 xk ) − 1 r · r∑ k=1 log2 ( xk ) ] ( 1 ) To avoid log zero , we add 1e − 06 to each xk . How mean and variance relate seems readily apparent with the square bracketed part being the computational formula for the standard deviation with values taken to log2 ( . ) rather than squared . This is preceded by a typical mean estimate . This meets the third requirement we mentioned . This simplification is Saliency Map Order Equivalent ( SMOE ) to the full iterative ( and expensive ) scale parameter estimation . We define SMOE as follows . Given saliency map Sa ∈ R+ , p×q and Sb ∈ R+ , p×q where we may have Sa 6= Sb , if we sort the pixels by value , then Sa will be sorted in exactly the same order as Sb . That means that the most salient location i , j is exactly the same in both Sa and Sb . This also means that if we create a binary mask of the n % most salient pixels , the mask for both Sa and Sb will also be exactly the same . SMOE is preserved if for instance , we apply independent monotonic functions to a saliency map . As such , we may as well strip these away to save on computation . Tie values may create an imperfect equivalence , but we assert that these should be very rare and not affect results by a measurable amount . Using µ as the mean of each column r in T , we can see the information relation more clearly if we simplify Eq 1 further which gives us our SMOE Scale method : ϕ ( . ) = 1 r · r∑ k=1 µ · log2 ( µ xk ) ( 2 ) The resemblance to conditional entropy is apparent . However , since the values in Eq 2 are not probabilities , this does not fit the precise definition of it . On the other hand , the interpretation is fairly similar . It is the mean activation multiplied by the information we would gain if we knew the individual values which formed the mean . Put in traditional terms , it is the information in the mean conditioned on the individual values . Numerical examples of this method at work can be seen in Appendix Table 3 along with more information on the derivation . To create a 2D saliency map S ∈ R+ , p×q , we simply apply Eq 2 at each spatial location i , j ∈ p , q with column elements k ∈ r in the 3D activation tensor T for a given input image . 2.2 COMBINED SALIENCY MAP GENERATION . For each input image , we derive five saliency maps . For different networks , this number may vary . Given a network such as a ResNet ( He et al. , 2015 ) , AlexNet ( Krizhevsky et al. , 2013 ) , VGG Net ( Simonyan & Zisserman , 2015 ) or DenseNet ( Huang et al. , 2017 ) we compute saliency on the final tensor computed at each spatial scale . Recall that most image classification networks process images in a pipeline that processes an image in consecutive groups of convolution layers where each group downsamples the image by 1/2x before passing it onto the next . It is just prior to the downsampling that we compute each saliency map . Computing saliency across image scales is a classical technique ( Itti et al. , 1998 ) . This is also similar to the method used in the XAI saliency technique described in ( Dabkowski & Gal , 2017 ) . . To make our maps easier to visualize or combine together , we normalize them from 0 to 1 by squashing them with the normal cumulative distribution function γ ( s ; µ , σ ) . Here mean and standard deviation are computed independently over each saliency map . We then create a combined saliency map by taking the weighted average of the maps . Since they are at different scales , they are upsampled via bilinear interpolation to match the dimensions of the input image . Given r saliency maps that have been bilinear interpolated ( upsampled ) to the original input image size p , q , they are then combined as : ci , j = ∑r k=1 γ ( si , j , k ; µk , σk ) · wk∑r k=1 wk ( 3 ) Note that technically , we compute γ ( s ; µ , σ ) before we upsample . Weighting is very useful since we expect that saliency maps computed later in the network should be more accurate than saliency maps computed earlier as the network has reduced more irrelevant information in deeper layers , distilling relevant pixels from noise ( Tishby et al. , 2000 ) . Network activity should be more focused on relevant locations as information becomes more related to the message . We observe this behavior which can be seen later in Figure 4 . A saliency map generation example can be seen in Figure 1 with many more examples in Appendix Figures 9 and 10 . The advantages of creating saliency maps this way when compared with most gradient methods are : • Pro : This is relatively efficient , requiring processing of just five low cost layers during a standard forward pass . • Pro : We can easily visualize the network at different stages ( layers ) . • Con : The current method does not have a class specific activation map ( CAM ) ( Selvaraju et al. , 2017 ; Chattopadhyay et al. , 2018 ; Omeiza et al. , 2019 ) , but we discuss how this can be done later .

Summary: This paper presents a method for creating saliency maps reflecting what a network deems important while also proposing an interesting method for visualizing this. The central premise for the method of characterizing relative importance of information represented by the network is based on an information theoretic measure. A variety of results are presented to examine the impact of keeping or removing information deemed important by this measure and a comparison is made to existing approaches as a justification for the proposed methods.