Aggregating explanation methods for neural networks stabilizes explanations

Despite a growing literature on explaining neural networks , no consensus has been reached on how to explain a neural network decision or how to evaluate an explanation . Our contributions in this paper are twofold . First , we investigate schemes to combine explanation methods and reduce model uncertainty to obtain a single aggregated explanation . The aggregation is more robust and aligns better with the neural network than any single explanation method . Secondly , we propose a new approach to evaluating explanation methods that circumvents the need for manual evaluation and is not reliant on the alignment of neural networks and humans decision processes . 1 INTRODUCTION . Despite the great success of neural networks especially in classic visual recognition problems , explaining the networks ’ decisions remains an open research problem Samek et al . ( 2019 ) . This is due in part to the complexity of the visual recognition problem and in part to the basic ’ ill-posedness ’ of the explanation task . This challenge is amplified by the fact that there is no agreement on what a sufficient explanation is and how to evaluate an explanation method . Many different explanation strategies and methods have been proposed ( Simonyan et al. , 2013 ; Zeiler & Fergus , 2014 ; Bach et al. , 2015 ; Selvaraju et al. , 2017 ; Smilkov et al. , 2017 ; Sundararajan et al. , 2017 ) . Focusing on visual explanations for individual decisions , most methods either use a backpropagation approach or aim to construct a simpler linear model with an intuitive explanation . The plethora of explanation approaches is a signature of the high-level epistemic uncertainty of the explanation task . This paper is motivated by a key insight in machine learning : Ensemble models can reduce both bias and variance compared to applying a single model . A related approach was pursued for functional visualization in neuroimaging ( Hansen et al. , 2001 ) . Here we for the first time explore the potential of aggregating explanations of individual visual decisions in reducing epistemic uncertainty for neural networks . We test the hypothesis that ensembles of multiple explanation methods are more robust than any single method . This idea is analyzed theoretically and evaluated empirically . We discuss the properties of the aggregate explanations and provide visual evidence that they combine features , hence are more complete and less biased than individual schemes . Based on this insight , we propose two ways to aggregate explanation methods , AGG-Mean and AGG-Var . In experiments on Imagenet , MNIST , and FashionMNIST , the aggregates identify relevant parts of the image more accurately than any single method . Second , we introduce IROF ( Iterative Removal Of Features ) as a new approach to quantitatively evaluate explanation methods without relying on human evaluation . We circumvent the problems of high correlation between neighbor pixels as well as the human bias that are present in current evaluation methods . 2 RELATED WORK . 2.1 EXPLANATION METHODS . The open problem of explainability is reflected in a lot of recent work ( Kindermans et al. , 2017 ; Selvaraju et al. , 2017 ; Bach et al. , 2015 ; Zhang et al. , 2018 ; Zhou et al. , 2016 ; Ancona et al. , 2018 ; Ribeiro et al. , 2016 ; Rieger et al. , 2018 ; Kim et al. , 2018 ; Lundberg & Lee , 2017 ; Zintgraf et al. , 2017 ; Simonyan et al. , 2013 ; Zeiler & Fergus , 2014 ; Selvaraju et al. , 2017 ; Smilkov et al. , 2017 ; Sundararajan et al. , 2017 ; Shrikumar et al. , 2017 ; Montavon et al. , 2017 ; Chang et al. , 2018 ) . We focus on generating visual explanations for single samples . The first work in this direction was Simonyan et al . ( 2013 ) with Saliency Maps ( SM ) that proposed backpropagating the output onto the input to gain an understanding of a neural network decision . The relevance for each input dimension is extracted by taking the gradient of the output w. r. t. to the input . This idea was extended by Springenberg et al . ( 2014 ) into Guided Backpropagation ( GM ) by applying ReLU non-linearities after each layer during the backpropagation . Compared to Saliency , this removes visual noise in the explanation . Grad-CAM ( GC ) from Selvaraju et al . ( 2017 ) is an explanation method , developed for use with convolutional neural networks . By backpropagating relevance through the dense layers and up-sampling the evidence for the convolutional part of the network , the method obtains coarse heatmaps that highlight relevant parts of the input image . Integrated Gradients ( IG ) Sundararajan et al . ( 2017 ) sums up the gradients from linearly interpolated pictures between a baseline , e.g . a black image , and the actual image . SmoothGrad ( SG ) filters out noise from a basic saliency map by creating many samples of the original input with Gaussian noise Smilkov et al . ( 2017 ) . The final saliency map is the average over all samples . Finally , we also consider LIME Ribeiro et al . ( 2016 ) . In contrast to the other methods , LIME is not based on backpropagation . Instead , it approximates the neural network with a linear model locally around the input to be explained . The coefficients of the linear model for the respective input dimensions give the importance of each dimension . Compared to the other methods this is much more computationally expensive as it requires many passes through the neural network . 2.2 EVALUATION OF EXPLANATION METHODS . The evaluation of explanation methods is a relatively recent topic with few systematic approaches ( Bau et al. , 2017 ; Ancona et al. , 2018 ; Hooker et al. , 2018 ; Adebayo et al. , 2018 ; Fong & Vedaldi , 2017 ) . To our knowledge , Bach et al . ( 2015 ) proposed the first quantitative approach to evaluate an explanation method by flipping pixels to their opposite and comparing the decrease in output with the relevance attributed to the pixel by the explanation method . As the authors note , this only works for low-dimensional input . This work was followed up upon in Samek et al . ( 2016 ) . By dividing high-dimensional images into squares , they make the method feasible for high-dimensional inputs . Squares with high relevance ( as measured by the explanation method ) consecutively get replaced with noise sampled from the uniform distribution . The difference between the original output and the output for the degraded images indicates the quality of the explanation method . Hooker et al . ( 2018 ) proposes another quantitative approach to evaluate explanation methods called ROAR . For each explanation method they extract the relevance maps over the entire training set . They degrade the training set by setting different percentages of the pixels with the highest relevance to the mean and retrain the network . Each retrained network is evaluated on the test set . The accuracy on the test set decreases dependent on the percentage of pixels set to the mean . This requires retraining the same architecture multiple times for each explanation method at a high computational cost . Ancona et al . ( 2018 ) proposed a different approach to evaluate explanation methods , called Sensitivityn , based on the notion that the decrease in output when a number of inputs are canceled out should be equal to the sum of their relevances . For a range of n ( between 1 and the total number of inputs ) they sample a hundred subsets of the input . For each n , the Pearson Correlation Coefficient ( PCC ) between the decrease in output , when the subset of features is removed , and the sum of their relevances is reported . The result is a curve of the PCC dependent on the percentage of the input being removed . For a good explanation method , the PCC will decrease slowly . 3 METHODS . 3.1 AGGREGATING EXPLANATION METHODS TO REDUCE VARIANCE . All currently available explanation methods have weaknesses that are inherent to the approach and include significant uncertainty in the resulting heatmap ( Kindermans et al. , 2017 ; Adebayo et al. , 2018 ; Smilkov et al. , 2017 ) . A natural way to mitigate this issue and reduce noise is to combine multiple explanation methods . Ensemble methods have been used for a long time to reduce the variance and bias of machine learning models . We apply the same idea to explanation methods and build an ensemble of explanation methods . We assume a neural network F : X 7→ y with X ∈ Rm×m and a set of explanation methods { ej } Jj=1 with ej : X , y , F 7→ E with E ∈ Rm×m . We write Ej , n for the explanation obtained for Xn with method ej and denote the mean aggregate explanation as ē with Ēn = 1J ∑J j=1Ej , n . While we assume the input to be an image ∈ Rm×m , this method is generalizable to inputs of other dimensionalities as well . To get a theoretical understanding of the benefit of aggregation , we hypothesize the existence of a ’ true ’ explanation Ên . This allows us to quantify the error of an explanation method as the mean squared difference between the ’ true ’ explanation and an explanation procured by an explanation method , i.e . the MSE . For clarity we subsequently omit the notation for the neural network . We write the error of explanation method j on image Xn as errj , n = ||Ej , n − Ên||2 with MSE ( Ej ) = 1 N ∑ n errj , n and MSE ( Ē ) = 1N ∑ n ||Ēn− Ên||2 is the MSE of the aggregate . The typical error of an explanation method is the mean error over all explanation methods MSE = 1 J ∑ j MSE ( Ej ) . With these definitions we can do a standard bias-variance decomposition ( Geman et al. , 1992 ) . Accordingly we can show the error of the aggregate will be less that the typical error of explanation methods , MSE = 1N ∑ n 1 J ∑ j ||Ên − Ej , n||2 ( 1 ) = 1N ∑ n ||Ên − Ēn||2 + 1 NJ ∑ n , j ||Ēn − Ej , n||2 , hence , MSE = 1 J ∑ j 1 N ∑ n ||Ēn − Ej , n||2︸ ︷︷ ︸ epistemic uncertainty +MSE ( Ē ) ≥ MSE ( Ē ) . A detailed calculation is given in appendix A.1 . The error of the aggregate MSE ( Ē ) is less than the typical error of the participating methods . The difference - a ‘ variance ’ term - represents the epistemic uncertainty and only vanishes if all methods produce identical maps . By taking the average over all available explanation methods , we reduce the variance of the explanation compared to using a single method . To obtain this average , we normalize all input heatmaps such that the relevance over all pixels sum up to one . This reflects our initial assumption that all individual explanation methods are equally good estimators . We refer to this approach as AGG-Mean . EAgg-Mean , n = 1 J J∑ j=1 Ej , n This estimator however does not take into account the estimate of the local epistemic uncertainty , i.e . the disagreement between methods . A way to incorporate this information is to form an ’ effect size ’ map by dividing the mean aggregate locally with its standard deviation ( Sigurdsson et al. , 2004 ) . Intuitively , this will assign less relevance to segments with high disagreement between methods . For stability , we divide not directly by the local variance but add a constant to the estimate of the local variance . This can be interpreted as a smoothing regularizer or a priori information regarding epistemic and aleatoric uncertainties . We refer to this approach as AGG-Var . EAGG-Var , n = 1 J J∑ j=1 Ej , n σ ( Ej∈J , n ) + where σ ( Ej∈J , n ) is the point-wise standard deviation over all explanations j ∈ J for Xn In section 4 we will evaluate and compare AGG-Mean and AGG-Var against basic explanation methods .

Summary: This paper, inspired by the established technique of model ensembling, proposes two methods (AGG-Mean and AGG-Var) for aggregating different model explanations into a single unified explanation. The authors mathematically prove that the derived explanation is guaranteed to be more truthful than the average performance of the constituent explanations. In practice, the aggregation consistently outperforms *all* individual explanations, not just their aggregated performance. Additionally, the paper introduces a new quantitative evaluation metric for explanations, free of human intervention: IROF (Incremental Removal of Features) incrementally grays out the segments deemed as relevant by an explanation method and observes how quickly the end-task performance is degraded (good explanations will cause fast degradation). Solid validation confirms that the IROF metric is sound.