Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysis

1 Introduction . Deep neural networks are now being deployed in numerous domains including medicine , transportation , security or finances with broad societal implications . Yet , these networks have become nearly inscrutable , and for most real-world applications , these systems are used to make critical decisions – often without any explanation . In recent years , numerous explainability methods have been proposed [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 ] . In addition to helping improve people ’ s trust in these systems , these methods have helped identify and correct biases in datasets [ 4 , 5 , 9 ] . This has , in turn , helped improve these systems ’ robustness and accelerate their broad deployment . An important limitation of standard explainability methods is that they require access to the system ’ s internal states including hidden layer activations or input gradients [ 1 , 5 , 6 , 10 ] . As a result , these so-called white-box methods can not be applied in the most general situations for which the internal states of network are not publicly accessible . For instance , it is common for companies to use neural networks provided by third parties ( e.g. , through web APIs or specialized hardware ) . However , only a handful of so-called black-box methods have been proposed to address this challenge with limited successes [ 2 , 4 , 8 ] . It is thus critical to develop more general methods that can reliably interpret and characterize the underlying decision processes of a wider array of models . Common approaches to explaining a model ’ s prediction consists of attributing a score for each input dimension such as image pixels for computer vision systems or individual words for natural language processing . Shown in Fig . 1 is an example image and associated importance map for an image categorization system , whereby scores for individual pixels are displayed as heatmaps where * Equal contribution †Work done before April 2021 and joining Tesla 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . hotter locations correspond to pixels that contribute the most to the system ’ s final prediction . In the context of black-box models , a core challenge is to derive these heatmaps using only the output predictions available through the network ’ s forward pass . A simple approach consists in applying a given perturbation at a specific location on the input image to then measure how the corresponding prediction is affected . In the case of image models , pixel intensities are simply set to a default value corresponding to a pure black or gray value ; in the case of language models , individual words are removed entirely from the text [ 11 , 12 ] . However , evaluating the impact of these perturbations one dimension at a time fails to identify all of the non-linear interactions between input variables that are known to prevail in a complex system such as a deep neural network . However , estimating the combined effect of perturbations across multiple locations quickly becomes combinatorially intractable . Methods have been recently proposed to try to address some of these issues by grouping dimensions together , such as by grouping pixels within a neighborhood of the image ( superpixel ) [ 2 ] or sampling perturbation masks that affect multiple regions of the input [ 4 , 8 ] . A first limitation of these approaches includes the use of Monte Carlo sampling methods which require a high number of forward passes – making these approaches computationally expensive . A second limitation is that they rely on relatively simple perturbations such as flipping pixels on or off [ 2 , 4 , 8 ] . This severely constrains the space of perturbations considered and limits the efficiency with which the space of perturbations can be explored . We address these limitations by introducing an attribution method that leverages variance-based sensitivity analysis techniques , and more specifically Sobol indices [ 13 ] . These methods were initially introduced to help identify the input variables that have the most influence on the variance of the output of a non-linear mathematical system [ 14 ] . These were traditionally used in physics and economics to estimate the part of the variance induced by a group of variables on a system ’ s output [ 15 , 16 , 17 ] . Our main contribution is a general framework to explain predictions from black-box models by adapting Sobol indices to be used in conjunction with perturbation masks . One of the originalities of our attribution method is that it is designed to support standard perturbations and real-valued intensity perturbations that can generate a continuous range of perturbations . Our second contribution is a tractable method for the calculation of Sobol indices . This is done by first sampling the perturbation masks following Quasi-Monte Carlo sequences , which efficiently covers the space of perturbations . This can be done best by leveraging the most efficient estimator borrowed from the sensitivity analysis literature [ 18 , 19 , 20 , 21 ] . As a result , the proposed method can be efficiently applied to high-dimensional inputs such as images . It produces on par with or better explanations than the state-of-the-art with at least half the number of forward passes . Another benefit of our method is that it allows to characterize not only the main effect of image regions but also higher-order interactions by decomposing the variance of the system ’ s prediction using the Sobol indices . We run extensive experiments to demonstrate the benefits of the proposed method on several recent complementary benchmarks including the “ Pointing Game ” and “ Deletion ” . Our primary focus is image classification , but our method is general and we include results on text classification benchmarks as well . 2 Related work . Attribution methods for white-box models A widely used family of attribution methods uses gradient computations via the backpropagation algorithm . The first such method was introduced in [ 22 ] and later refined in [ 1 ] in the context of deep convolutional networks . The idea is to backpropagate the gradient from a classification unit all the way down to the input pixels . The resulting gradient image indicates which pixels affect the decision score the most . Other gradientbased methods including DeConvNet [ 2 ] and Guided Backprop [ 3 ] were specifically developed to deal with certain activation functions such as ReLU . However , this family of methods are limited by the fact that they focus on the influence of individual pixels in an infinitesimal neighborhood around the input image in the image space . For instance , it has been shown that gradients often vanish when the prediction score to be explained is near the maximum value [ 7 ] . Integrated Gradient [ 7 ] partially addresses this issue by accumulating gradients along a straight interpolation path from a baseline state to the original image . Yet another class of gradient-based methods [ 6 , 23 ] iteratively optimize a perturbation mask over regions of an image . It leads to a different type of explanation given by the final perturbation mask . Such an approach might leverage more meaningful perturbations but it turns out to be computationally very slow with 20 seconds per explanation – limiting its broad applicability . In comparison , our proposed approach also works on image regions but it does not require iterative gradient computation – and can thus be parallelized . Another family of attribution methods relies on the neural network ’ s activations . Popular examples include CAM [ 24 ] which computes an attribution score based on a weighted sum of feature channels activities – right before the classification layer . GradCAM [ 5 ] extends CAM via the use of gradients to reweigh each feature channel to take into account their importance for the predicted class . These methods also work on image regions because they are computed on the output of feature maps which are themselves at a coarser resolution than the original image . In comparison , our proposed approach is model-agnostic and hence does not require access to internal computations . Attribution methods for black-box models Most similar to our approach are attribution methods that can be used to explain the predictions of truly black-box models . These methods probe a neural network ’ s responses to perturbations over image regions and combine the resulting predictions into an influence score for each individual pixel or group of pixels . The simplest method , “ Occlusion ” [ 2 ] , masks individual image regions – one at a time – with an occluding mask set to a baseline value and assigns the corresponding prediction scores to all pixels within the occluded region . Then the explanation is given by these prediction scores and can be easily interpreted . However , occlusion fails to account for the joint ( higher-order ) interactions between multiple image regions . For instance , occluding two image regions – one at a time – may only decrease the model ’ s prediction minimally ( say a single eye or mouth component on a face ) while occluding these two regions together may yield a substantial change in the model ’ s prediction if these two regions interact non-linearly as is expected for a deep neural network . Our work , together with related methods such as LIME [ 4 ] and RISE [ 8 ] , addresses this problem by randomly perturbating the input image in multiple regions at a time . Obviously , perturbating multiple image locations simultaneously leads to a combinatorial explosion in the number of combinations and methods have been proposed to make these approaches more tractable . For instance , a popular method , LIME [ 4 ] , takes superpixels as regions to perturbate instead of individual pixels . An influence score is then computed for a set of connected pixel patches indicating how strongly a patch is correlated to the model predictions . RISE [ 8 ] relies on Monte Carlo sampling to generate a set of binary masks , each value in the masks representing a pixel region . By probing the model with randomly masked versions of the input , RISE [ 8 ] produces a importance map by considering the average of the masks weighted by their associated prediction scores . Instead of using binary masks , our method considers a continuous range of perturbations which allows for a finer exploration of the model ’ s response . Our method can still use the same perturbations as used in Occlusion [ 2 ] , LIME [ 4 ] and RISE [ 8 ] , but it also enables the use of more advanced perturbation functions that take continuous inputs . More importantly , the aforementioned methods lack a rigorous framework . Here , we introduce a theoretical framework that decomposes the influence score of each individual region between multiple orders of influence . The first-order approximates Occlusion [ 2 ] by considering the influence of one region at a time , while the second-order considers two regions at a time , etc . The decomposition also includes higher-orders . Variance-based sensitivity analysis Our attribution method builds on the variance-based sensitivity analysis framework . The approach was introduced in the 70s [ 25 ] and reached a cornerstone with the Sobol indices [ 26 ] . Sobol indices are currently used in many fields ( including those that are said to be safety-critical ) , especially for the analysis of physical phenomena [ 16 ] . They are used to identify the input dimensions that have the highest influence on the output of a model or a mathematical system . Several statistical estimators to compute these indices are available [ 15 , 27 , 20 , 28 , 29 ] and have asymptotic guarantees [ 20 , 30 , 31 ] . We build on this literature by adapting these Sobol indices in the context of black-box models to compute the influence of regions of an image on the output predictions using perturbation masks .

Summary: In this work, the authors propose to use Sobol indices for black-box explanations of deep neural networks on computer vision and NLP tasks. The method is based on efficiently estimating Sobol indices and Total Sobol indices with a Quasi-Monte Carlo sampling strategy. The experiments on the tasks of pointing game, deletion, and word deletion show that the proposed method achieves comparable results to state-of-the-art attribution models.