Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysis

1 Introduction . Deep neural networks are now being deployed in numerous domains including medicine , transportation , security or finances with broad societal implications . Yet , these networks have become nearly inscrutable , and for most real-world applications , these systems are used to make critical decisions – often without any explanation . In recent years , numerous explainability methods have been proposed [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 ] . In addition to helping improve people ’ s trust in these systems , these methods have helped identify and correct biases in datasets [ 4 , 5 , 9 ] . This has , in turn , helped improve these systems ’ robustness and accelerate their broad deployment . An important limitation of standard explainability methods is that they require access to the system ’ s internal states including hidden layer activations or input gradients [ 1 , 5 , 6 , 10 ] . As a result , these so-called white-box methods can not be applied in the most general situations for which the internal states of network are not publicly accessible . For instance , it is common for companies to use neural networks provided by third parties ( e.g. , through web APIs or specialized hardware ) . However , only a handful of so-called black-box methods have been proposed to address this challenge with limited successes [ 2 , 4 , 8 ] . It is thus critical to develop more general methods that can reliably interpret and characterize the underlying decision processes of a wider array of models . Common approaches to explaining a model ’ s prediction consists of attributing a score for each input dimension such as image pixels for computer vision systems or individual words for natural language processing . Shown in Fig . 1 is an example image and associated importance map for an image categorization system , whereby scores for individual pixels are displayed as heatmaps where * Equal contribution †Work done before April 2021 and joining Tesla 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . hotter locations correspond to pixels that contribute the most to the system ’ s final prediction . In the context of black-box models , a core challenge is to derive these heatmaps using only the output predictions available through the network ’ s forward pass . A simple approach consists in applying a given perturbation at a specific location on the input image to then measure how the corresponding prediction is affected . In the case of image models , pixel intensities are simply set to a default value corresponding to a pure black or gray value ; in the case of language models , individual words are removed entirely from the text [ 11 , 12 ] . However , evaluating the impact of these perturbations one dimension at a time fails to identify all of the non-linear interactions between input variables that are known to prevail in a complex system such as a deep neural network . However , estimating the combined effect of perturbations across multiple locations quickly becomes combinatorially intractable . Methods have been recently proposed to try to address some of these issues by grouping dimensions together , such as by grouping pixels within a neighborhood of the image ( superpixel ) [ 2 ] or sampling perturbation masks that affect multiple regions of the input [ 4 , 8 ] . A first limitation of these approaches includes the use of Monte Carlo sampling methods which require a high number of forward passes – making these approaches computationally expensive . A second limitation is that they rely on relatively simple perturbations such as flipping pixels on or off [ 2 , 4 , 8 ] . This severely constrains the space of perturbations considered and limits the efficiency with which the space of perturbations can be explored . We address these limitations by introducing an attribution method that leverages variance-based sensitivity analysis techniques , and more specifically Sobol indices [ 13 ] . These methods were initially introduced to help identify the input variables that have the most influence on the variance of the output of a non-linear mathematical system [ 14 ] . These were traditionally used in physics and economics to estimate the part of the variance induced by a group of variables on a system ’ s output [ 15 , 16 , 17 ] . Our main contribution is a general framework to explain predictions from black-box models by adapting Sobol indices to be used in conjunction with perturbation masks . One of the originalities of our attribution method is that it is designed to support standard perturbations and real-valued intensity perturbations that can generate a continuous range of perturbations . Our second contribution is a tractable method for the calculation of Sobol indices . This is done by first sampling the perturbation masks following Quasi-Monte Carlo sequences , which efficiently covers the space of perturbations . This can be done best by leveraging the most efficient estimator borrowed from the sensitivity analysis literature [ 18 , 19 , 20 , 21 ] . As a result , the proposed method can be efficiently applied to high-dimensional inputs such as images . It produces on par with or better explanations than the state-of-the-art with at least half the number of forward passes . Another benefit of our method is that it allows to characterize not only the main effect of image regions but also higher-order interactions by decomposing the variance of the system ’ s prediction using the Sobol indices . We run extensive experiments to demonstrate the benefits of the proposed method on several recent complementary benchmarks including the “ Pointing Game ” and “ Deletion ” . Our primary focus is image classification , but our method is general and we include results on text classification benchmarks as well . 2 Related work . Attribution methods for white-box models A widely used family of attribution methods uses gradient computations via the backpropagation algorithm . The first such method was introduced in [ 22 ] and later refined in [ 1 ] in the context of deep convolutional networks . The idea is to backpropagate the gradient from a classification unit all the way down to the input pixels . The resulting gradient image indicates which pixels affect the decision score the most . Other gradientbased methods including DeConvNet [ 2 ] and Guided Backprop [ 3 ] were specifically developed to deal with certain activation functions such as ReLU . However , this family of methods are limited by the fact that they focus on the influence of individual pixels in an infinitesimal neighborhood around the input image in the image space . For instance , it has been shown that gradients often vanish when the prediction score to be explained is near the maximum value [ 7 ] . Integrated Gradient [ 7 ] partially addresses this issue by accumulating gradients along a straight interpolation path from a baseline state to the original image . Yet another class of gradient-based methods [ 6 , 23 ] iteratively optimize a perturbation mask over regions of an image . It leads to a different type of explanation given by the final perturbation mask . Such an approach might leverage more meaningful perturbations but it turns out to be computationally very slow with 20 seconds per explanation – limiting its broad applicability . In comparison , our proposed approach also works on image regions but it does not require iterative gradient computation – and can thus be parallelized . Another family of attribution methods relies on the neural network ’ s activations . Popular examples include CAM [ 24 ] which computes an attribution score based on a weighted sum of feature channels activities – right before the classification layer . GradCAM [ 5 ] extends CAM via the use of gradients to reweigh each feature channel to take into account their importance for the predicted class . These methods also work on image regions because they are computed on the output of feature maps which are themselves at a coarser resolution than the original image . In comparison , our proposed approach is model-agnostic and hence does not require access to internal computations . Attribution methods for black-box models Most similar to our approach are attribution methods that can be used to explain the predictions of truly black-box models . These methods probe a neural network ’ s responses to perturbations over image regions and combine the resulting predictions into an influence score for each individual pixel or group of pixels . The simplest method , “ Occlusion ” [ 2 ] , masks individual image regions – one at a time – with an occluding mask set to a baseline value and assigns the corresponding prediction scores to all pixels within the occluded region . Then the explanation is given by these prediction scores and can be easily interpreted . However , occlusion fails to account for the joint ( higher-order ) interactions between multiple image regions . For instance , occluding two image regions – one at a time – may only decrease the model ’ s prediction minimally ( say a single eye or mouth component on a face ) while occluding these two regions together may yield a substantial change in the model ’ s prediction if these two regions interact non-linearly as is expected for a deep neural network . Our work , together with related methods such as LIME [ 4 ] and RISE [ 8 ] , addresses this problem by randomly perturbating the input image in multiple regions at a time . Obviously , perturbating multiple image locations simultaneously leads to a combinatorial explosion in the number of combinations and methods have been proposed to make these approaches more tractable . For instance , a popular method , LIME [ 4 ] , takes superpixels as regions to perturbate instead of individual pixels . An influence score is then computed for a set of connected pixel patches indicating how strongly a patch is correlated to the model predictions . RISE [ 8 ] relies on Monte Carlo sampling to generate a set of binary masks , each value in the masks representing a pixel region . By probing the model with randomly masked versions of the input , RISE [ 8 ] produces a importance map by considering the average of the masks weighted by their associated prediction scores . Instead of using binary masks , our method considers a continuous range of perturbations which allows for a finer exploration of the model ’ s response . Our method can still use the same perturbations as used in Occlusion [ 2 ] , LIME [ 4 ] and RISE [ 8 ] , but it also enables the use of more advanced perturbation functions that take continuous inputs . More importantly , the aforementioned methods lack a rigorous framework . Here , we introduce a theoretical framework that decomposes the influence score of each individual region between multiple orders of influence . The first-order approximates Occlusion [ 2 ] by considering the influence of one region at a time , while the second-order considers two regions at a time , etc . The decomposition also includes higher-orders . Variance-based sensitivity analysis Our attribution method builds on the variance-based sensitivity analysis framework . The approach was introduced in the 70s [ 25 ] and reached a cornerstone with the Sobol indices [ 26 ] . Sobol indices are currently used in many fields ( including those that are said to be safety-critical ) , especially for the analysis of physical phenomena [ 16 ] . They are used to identify the input dimensions that have the highest influence on the output of a model or a mathematical system . Several statistical estimators to compute these indices are available [ 15 , 27 , 20 , 28 , 29 ] and have asymptotic guarantees [ 20 , 30 , 31 ] . We build on this literature by adapting these Sobol indices in the context of black-box models to compute the influence of regions of an image on the output predictions using perturbation masks .

Summary: This paper proposes a black-box method to assess the contribution of image pixels and groups of pixels (regions) to the classification decision. The proposed approach is based on Sobol indices, that allow to characterize the influence of an input subset (image pixel/group of pixels or region) as its own variance with respect to the total variance.   To employ these indices in an efficient way, they are approximated by empirical Sobol indices. Specifically, from an input image, a set of real-valued masks is drawn from a Quasi-Monte-Carlo Sequence to generate perturbed inputs. The Sobol indices and total Sobol indices are estimated empirically from the variance of model predictions of these images. The first-order Sobol indices indicate the importance of each pixel in isolation, the total Sobol  indices also encompass higher-order interactions. The method is evaluated on standard benchmarks for vision and NLP and shows competitive performance to black box and white box methods.