Where do Models go Wrong? Parameter-Space Saliency Maps for Explainability

1 INTRODUCTION . With the widespread deployment of deep neural networks in high-stakes applications such as medical imaging ( Kang et al. , 2017 ) , credit score assessment ( West , 2000 ) , and facial recognition ( Deng et al. , 2019 ) , practitioners need to understand why their models make the decisions they do . In fact , “ right to explanation ” legislation in the European Union and the United States dictates that relevant public and private organizations must be able to justify the decisions their algorithms make ( United States Congress Senate Committee on Banking and Housing and Urban Affairs , 1976 ; European Commission , 2018 ) . Diagnosing the causes of system failures is particularly crucial for understanding the flaws and limitations of models we intend to employ . Conventional saliency methods focus on highlighting sensitive pixels ( Simonyan et al. , 2014 ) or image regions that maximize specific activations ( Erhan et al. , 2009 ) . However , such maps may not be useful in diagnosing undesirable model behaviors as they do not necessarily identify areas that specifically cause bad performance since the most sensitive pixels may not be the ones responsible for triggering misclassification . We develop an alternative approach to saliency which highlights network parameters that influence decisions rather than input features . These parameter saliency maps yield a number of useful analyses : • Nearest neighbors in parameter saliency space share common semantic information . That is , samples which are misclassified for similar reasons and cause similar parameters to malfunction are semantically similar . • By first identifying the network parameters responsible for an erroneous classification , we can then visualize the image regions that interact with those parameters and trigger the identified misbehavior . • We verify that identified salient parameters are indeed responsible for misclassification by showing that turning these parameters off improves predictions on the associated samples , more than pruning the same number of random or least salient parameters . • We further validate the link between salient parameters and network misclassification errors by observing that fine-tuning a small number of the most salient parameters on a single sample results in error correction on other samples which were misclassified for similar reasons . After carefully delineating our methodology and experimentally validating the meaningfulness of our parameter saliency maps , we showcase the practical utility of this paradigm as an explainability tool with a case study in which we are able to uncover a neural network ’ s reliance on a spurious correlation which causes interpretable failures . 1.1 RELATED WORK . Neural network interpretability and parameter importance . A major line of work in neural network interpretability focuses on convolutional neural networks . Works visualizing , interpreting , and analysing feature maps ( Zeiler & Fergus , 2014 ; Yosinski et al. , 2015 ; Olah et al. , 2017 ; Mahendran & Vedaldi , 2015 ) provide insight into the role of individual convolutional filters . These methods , together with other approaches for filter explainability ( Bau et al. , 2017 ; Zhou et al. , 2018 ; 2019 ) find that individual convolutional filters often are responsible for specific tasks such as edge , shape , and texture detection . The idea of measuring neural network parameter importance has been studied in multiple contexts . Notions of neuron and parameter importance have been used for AI explainability ( Srinivas & Fleuret , 2019 ; Selvaraju et al. , 2017 ; Morcos et al. , 2018 ; Shrikumar et al. , 2017 ; Shrikumar et al . ) , manipulating model behavior ( Bau et al. , 2018 ) , and parameter pruning ( Abbasi-Asl & Yu , 2017 ; Liu & Wu , 2019 ) . Input space saliency maps . A considerable amount of literature focuses on identifying input features that are important for neural network decisions . These methods include using deconvolution approaches ( Zeiler & Fergus , 2014 ) and data gradient information ( Simonyan et al. , 2014 ) . Several works build on these ideas and propose improvements such as Integrated Gradients ( Sundararajan et al. , 2017 ) , SmoothGrad ( Smilkov et al. , 2017 ) , and Guided Backpropagation ( Springenberg et al. , 2015 ) which result in sharper and more localized saliency maps . Other approaches focus on the use of class activation maps ( Zhou et al. , 2016 ) with improvements incorporating gradient information ( Selvaraju et al. , 2017 ) and more novel approaches to weighting the activation maps ( Wang et al. , 2020 ) . In addition , various saliency methods are based on manipulating the input image ( Fong & Vedaldi , 2017 ; Zeiler & Fergus , 2014 ) . Another line of work is aimed at evaluating the effectiveness of saliency maps ( Adebayo et al. , 2018 ; Alqaraawi et al. , 2020 ) . Although extensive work studies how different regions of images affect a network ’ s predictions , limited work ( Srinivas & Fleuret , 2019 ) aims to distinguish important network parameters . Our work combines the ideas of saliency maps and parameter importance and evaluates saliency directly on model parameters by aggregating their absolute gradients on a filter level . We leverage the resulting parameter saliency profiles as an explainability tool and develop an input-space saliency counterpart which highlights image features that cause specific filters to malfunction to study the interaction between the image features and the erroneous filters . 2 METHOD . It is known that different network filters are responsible for identifying different image properties and objects ( Zeiler & Fergus , 2014 ; Yosinski et al. , 2015 ; Olah et al. , 2017 ; Mahendran & Vedaldi , 2015 ) . This motivates the idea that mistakes made on wrongly classified images can be understood by investigating the network parameters , rather than only the pixels , that played a role in making a decision . We develop parameter-space saliency methods geared towards identifying and analyzing neural network parameters that are responsible for making erroneous decisions . Central to our method is the use of gradient information of the loss function as a measure of parameter sensitivity and optimality of the network at a given point in image space . 2.1 PARAMETER SALIENCY PROFILE . Let x be a sample in the validation set D with label y , and suppose a trained classifier has parameters θ that minimize a loss function L. We define the parameter-wise saliency profile of x as a vector s ( x , y ) with entries s ( x , y ) i : = |∇θiLθ ( x , y ) | , the magnitudes of the gradient of the loss with respect to each model parameter . Because the gradients on training data for a model trained to convergence are near zero , it is important to specify that D be a validation , or holdout , set . Intuitively , a larger gradient norm at the point ( x , y ) indicates a greater inefficiency in the network ’ s classification of sample x , and thus each entry of s ( x , y ) measures the suboptimality of individual parameters . Aggregation of parameter saliency . Convolutional filters are known to specialize in tasks such as edge , shape , and texture detection ( Yosinski et al. , 2015 ; Bau et al. , 2017 ; Olah et al. , 2017 ) . We therefore choose to aggregate saliency on the filter-wise basis by averaging the gradient magnitudes of parameters corresponding to each convolutional filter . This allows us to isolate filters to which the loss is most sensitive ( i.e . those which , when corrected , lead to the greatest reduction in loss ) . Formally , for each convolutional filter Fk in the network , consider its respective index set αk , which gives the indices of parameters corresponding to the filter Fk . The filter-wise saliency profile of x is defined to be a vector s ( x , y ) with entries s ( x , y ) k : = 1 |αk| ∑ i∈αk s ( x , y ) i , ( 1 ) the parameter-wise saliency profile aggregated by averaging on the filter level . Standardizing parameter saliency . Figure 1 exhibits the ResNet-50 ( He et al. , 2016 ) filter-wise saliency profile averaged over the ImageNet ( Deng et al. , 2009 ) validation set , where filters within each layer are sorted from highest to lowest saliency . One clear observation is the difference in the scale of gradient magnitudes – shallower filters are more salient than deeper filters . This phenomenon might occur for a number of reasons . First , early filters encode low-level features , such as edges and textures , which are active across a wide spectrum of images . Second , typical networks have fewer filters in shallow layers than in deep layers , making each individual filter more influential at shallower layers . Third , the effects of early filters cascade and accumulate as they pass through a network . To isolate filters that uniquely cause erroneous behavior on particular samples , we find filters that are abnormally salient for a sample , x , but not for others . That is , we further standardize the saliency profile of x with respect to all filter-wise saliency profiles of D. Formally , let µ be the average filter-wise saliency profile across all x ∈ D , and let σ be an equallength vector with the corresponding standard deviation for each entry . We use these statistics to produce the standardized filter-wise saliency profile as follows : ŝ ( x , y ) : = |s ( x , y ) − µ| σ . ( 2 ) The resulting tensor ŝ ( x , y ) is of length equal to the number of convolutional filters in the network , and we henceforth call it the saliency profile for sample x . By standardizing saliency profiles , we create a saliency map that activates when the importance of a filter is unusually strong relative to other samples in the dataset . This prevents the saliency map from highlighting filters that are uniformly important for all images , and instead focuses saliency on filters that are uniquely important and serve an image-dependent role . In the rest of the paper , unless explicitly noted otherwise , we use ŝ ( x , y ) and refer to it as parameter saliency . Incorrectly classified samples are more salient . Empirically , we observe the saliency profiles of incorrectly classified samples exhibit , on average , greater values than those of correctly classified examples . This bolsters the intuition that salient filters are precisely those malfunctioning — if the classification is correct , there should be few malfunctioning filters or none at all . Moreover , we see deeper parts of the network appear to be most salient for the incorrectly classified samples while earlier layers are often the most salient for correctly classified samples . An example of these behaviors for ResNet-50 is shown in Figure 2 which presents standardized filter-wise saliency profiles averaged over the correctly and incorrectly classified examples from the ImageNet validation set . Additionally , we note the improved relative scale of the standardized saliency profile across different layers compared to the absolute gradient magnitudes in Figure 1 . Saliency profiles for other architectures could be found in Appendix A . Henceforth , we will focus specifically on saliency profiles of misclassified samples in order to explore how neural networks make mistakes . 2.2 INPUT-SPACE SALIENCY FOR VISUALIZING HOW FILTERS MALFUNCTION . The parameter saliency profile allows us to identify filters that are most responsible for mistakes and erroneous network behavior . In this section , we develop an input-space counterpart to our parameter saliency method to understand which features of the image affect the saliency of particular filters . Geiping et al . ( 2020 ) show that the gradient information of a network is invertible , providing a link between input space and parameter saliency space . This work , along with existing input-space saliency map tools ( Simonyan et al. , 2014 ; Springenberg et al. , 2015 ; Smilkov et al. , 2017 ; Zhou et al. , 2016 ; Selvaraju et al. , 2017 ) , inspires our method . Given a parameter saliency profile ŝ = ŝ ( x , y ) for an image x with label y , our goal is to highlight the input features that drive large filter saliency values . That is , we would like to identify image pixels altering which can make filters more salient . To this end , we first select some set F of the most salient filters that we would like to explore . Then , we create a boosted saliency profile s′F by increasing the entries of ŝ corresponding to the chosen filters F ( e.g. , multiplying by a large constant ) . Now , we can find pixels that are important for making the chosen filters F more salient and , equivalently , making the filter saliency profile ŝ ( x , y ) close to the boosted saliency profile s′F by taking the following gradients : MF = |∇xDC ( ŝ ( x , y ) , s′F ) | , ( 3 ) where DC ( · , · ) is cosine distance . The resulting input saliency map MF contains input features ( pixels ) that affect the saliency of the chosen filters F the most .

Summary: This paper devises an analytic method for explainability based on the observation of filter-wise parameter saliency distribution, and tests on several models. And several experiments are conducted to deminstrate the conjecture. The motivation is straightforward and easy to understand.