Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysis
Year: 2021

Abstract:
We describe a novel attribution method which is grounded in Sensitivity Analysis and uses Sobol indices. Beyond modeling the individual contributions of image regions, Sobol indices provide an efficient way to capture higher-order interactions between image regions and their contributions to a neural network’s prediction through the lens of variance. We describe an approach that makes the computation of these indices efficient for high-dimensional problems by using perturbation masks coupled with efficient estimators to handle the high dimensionality of images. Importantly, we show that the proposed method leads to favorable scores on standard benchmarks for vision (and language models) while drastically reducing the computing time compared to other black-box methods – even surpassing the accuracy of state-of-the-art white-box methods which require access to internal representations. Our code is freely available: github.com/fel-thomas/ Sobol-Attribution-Method.

heading: 1 Introduction
text: Deep neural networks are now being deployed in numerous domains including medicine, transportation, security or finances with broad societal implications. Yet, these networks have become nearly inscrutable, and for most real-world applications, these systems are used to make critical decisions – often without any explanation. In recent years, numerous explainability methods have been proposed [1, 2, 3, 4, 5, 6, 7, 8]. In addition to helping improve people’s trust in these systems, these methods have helped identify and correct biases in datasets [4, 5, 9]. This has, in turn, helped improve these systems’ robustness and accelerate their broad deployment. An important limitation of standard explainability methods is that they require access to the system’s internal states including hidden layer activations or input gradients [1, 5, 6, 10]. As a result, these so-called white-box methods cannot be applied in the most general situations for which the internal states of network are not publicly accessible. For instance, it is common for companies to use neural networks provided by third parties (e.g., through web APIs or specialized hardware). However, only a handful of so-called black-box methods have been proposed to address this challenge with limited successes [2, 4, 8]. It is thus critical to develop more general methods that can reliably interpret and characterize the underlying decision processes of a wider array of models.
Common approaches to explaining a model’s prediction consists of attributing a score for each input dimension such as image pixels for computer vision systems or individual words for natural language processing. Shown in Fig. 1 is an example image and associated importance map for an image categorization system, whereby scores for individual pixels are displayed as heatmaps where
* Equal contribution †Work done before April 2021 and joining Tesla
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
hotter locations correspond to pixels that contribute the most to the system’s final prediction. In the context of black-box models, a core challenge is to derive these heatmaps using only the output predictions available through the network’s forward pass. A simple approach consists in applying a given perturbation at a specific location on the input image to then measure how the corresponding prediction is affected. In the case of image models, pixel intensities are simply set to a default value corresponding to a pure black or gray value; in the case of language models, individual words are removed entirely from the text [11, 12]. However, evaluating the impact of these perturbations one dimension at a time fails to identify all of the non-linear interactions between input variables that are known to prevail in a complex system such as a deep neural network. However, estimating the combined effect of perturbations across multiple locations quickly becomes combinatorially intractable. Methods have been recently proposed to try to address some of these issues by grouping dimensions together, such as by grouping pixels within a neighborhood of the image (superpixel) [2] or sampling perturbation masks that affect multiple regions of the input [4, 8]. A first limitation of these approaches includes the use of Monte Carlo sampling methods which require a high number of forward passes – making these approaches computationally expensive. A second limitation is that they rely on relatively simple perturbations such as flipping pixels on or off [2, 4, 8]. This severely constrains the space of perturbations considered and limits the efficiency with which the space of perturbations can be explored.
We address these limitations by introducing an attribution method that leverages variance-based sensitivity analysis techniques, and more specifically Sobol indices [13]. These methods were initially introduced to help identify the input variables that have the most influence on the variance of the output of a non-linear mathematical system [14]. These were traditionally used in physics and economics to estimate the part of the variance induced by a group of variables on a system’s output [15, 16, 17]. Our main contribution is a general framework to explain predictions from black-box models by adapting Sobol indices to be used in conjunction with perturbation masks. One of the originalities of our attribution method is that it is designed to support standard perturbations and real-valued intensity perturbations that can generate a continuous range of perturbations. Our second contribution is a tractable method for the calculation of Sobol indices. This is done by first sampling the perturbation masks following Quasi-Monte Carlo sequences, which efficiently covers the space of perturbations. This can be done best by leveraging the most efficient estimator borrowed from the sensitivity analysis literature [18, 19, 20, 21]. As a result, the proposed method can be
efficiently applied to high-dimensional inputs such as images. It produces on par with or better explanations than the state-of-the-art with at least half the number of forward passes. Another benefit of our method is that it allows to characterize not only the main effect of image regions but also higher-order interactions by decomposing the variance of the system’s prediction using the Sobol indices. We run extensive experiments to demonstrate the benefits of the proposed method on several recent complementary benchmarks including the “Pointing Game” and “Deletion”. Our primary focus is image classification, but our method is general and we include results on text classification benchmarks as well.

heading: 2 Related work
text: Attribution methods for white-box models A widely used family of attribution methods uses gradient computations via the backpropagation algorithm. The first such method was introduced in [22] and later refined in [1] in the context of deep convolutional networks. The idea is to backpropagate the gradient from a classification unit all the way down to the input pixels. The resulting gradient image indicates which pixels affect the decision score the most. Other gradientbased methods including DeConvNet [2] and Guided Backprop [3] were specifically developed to deal with certain activation functions such as ReLU. However, this family of methods are limited by the fact that they focus on the influence of individual pixels in an infinitesimal neighborhood around the input image in the image space. For instance, it has been shown that gradients often vanish when the prediction score to be explained is near the maximum value [7]. Integrated Gradient [7] partially addresses this issue by accumulating gradients along a straight interpolation path from a baseline state to the original image.
Yet another class of gradient-based methods [6, 23] iteratively optimize a perturbation mask over regions of an image. It leads to a different type of explanation given by the final perturbation mask. Such an approach might leverage more meaningful perturbations but it turns out to be computationally very slow with 20 seconds per explanation – limiting its broad applicability. In comparison, our proposed approach also works on image regions but it does not require iterative gradient computation – and can thus be parallelized.
Another family of attribution methods relies on the neural network’s activations. Popular examples include CAM [24] which computes an attribution score based on a weighted sum of feature channels activities – right before the classification layer. GradCAM [5] extends CAM via the use of gradients to reweigh each feature channel to take into account their importance for the predicted class. These methods also work on image regions because they are computed on the output of feature maps which are themselves at a coarser resolution than the original image. In comparison, our proposed approach is model-agnostic and hence does not require access to internal computations.
Attribution methods for black-box models Most similar to our approach are attribution methods that can be used to explain the predictions of truly black-box models. These methods probe a neural network’s responses to perturbations over image regions and combine the resulting predictions into an influence score for each individual pixel or group of pixels. The simplest method, “Occlusion” [2], masks individual image regions – one at a time – with an occluding mask set to a baseline value and assigns the corresponding prediction scores to all pixels within the occluded region. Then the explanation is given by these prediction scores and can be easily interpreted. However, occlusion fails to account for the joint (higher-order) interactions between multiple image regions. For instance, occluding two image regions – one at a time – may only decrease the model’s prediction minimally (say a single eye or mouth component on a face) while occluding these two regions together may yield a substantial change in the model’s prediction if these two regions interact non-linearly as is expected for a deep neural network.
Our work, together with related methods such as LIME [4] and RISE [8], addresses this problem by randomly perturbating the input image in multiple regions at a time. Obviously, perturbating multiple image locations simultaneously leads to a combinatorial explosion in the number of combinations and methods have been proposed to make these approaches more tractable. For instance, a popular method, LIME [4], takes superpixels as regions to perturbate instead of individual pixels. An influence score is then computed for a set of connected pixel patches indicating how strongly a patch is correlated to the model predictions.
RISE [8] relies on Monte Carlo sampling to generate a set of binary masks, each value in the masks representing a pixel region. By probing the model with randomly masked versions of the input,
RISE [8] produces a importance map by considering the average of the masks weighted by their associated prediction scores. Instead of using binary masks, our method considers a continuous range of perturbations which allows for a finer exploration of the model’s response. Our method can still use the same perturbations as used in Occlusion [2], LIME [4] and RISE [8], but it also enables the use of more advanced perturbation functions that take continuous inputs.
More importantly, the aforementioned methods lack a rigorous framework. Here, we introduce a theoretical framework that decomposes the influence score of each individual region between multiple orders of influence. The first-order approximates Occlusion [2] by considering the influence of one region at a time, while the second-order considers two regions at a time, etc. The decomposition also includes higher-orders. Variance-based sensitivity analysis Our attribution method builds on the variance-based sensitivity analysis framework. The approach was introduced in the 70s [25] and reached a cornerstone with the Sobol indices [26]. Sobol indices are currently used in many fields (including those that are said to be safety-critical), especially for the analysis of physical phenomena [16]. They are used to identify the input dimensions that have the highest influence on the output of a model or a mathematical system. Several statistical estimators to compute these indices are available [15, 27, 20, 28, 29] and have asymptotic guarantees [20, 30, 31]. We build on this literature by adapting these Sobol indices in the context of black-box models to compute the influence of regions of an image on the output predictions using perturbation masks.

heading: 3 Sobol attribution method
text: In this work, we formulate the feature attribution problem as quantifying the contribution of a collection of d real-valued features x = (x1, ..., xd) with respect to a model decision. Specifically, we consider a black-box decision function f : X → Rk whose internal states and analytical form are unknown (for instance, f can score the probability for the input to belong to a specific class). Our goal is to quantify the importance of each feature to the decision score f(x), not just individually but also collectively. To capture these higher-order interactions, our method consists in estimating the Sobol indices of the features x by randomly perturbating them and evaluating the impact of these perturbations on the prediction of the black-box model (Fig. 1).
Considering variations of f(x) in response to meaningful perturbations of the input x is a natural way to interpret the local behavior of a the decision function f around x. Several methods build on this idea, e.g., by removing one or a group of input features [2, 4, 6, 8, 23] or by back-propagating the gradient to the input space through the model [1, 7, 32, 5]. Most of these methods use the model’s internal representations and/or require computing the gradient w.r.t. the input, which makes them unusable in a black-box setting. Moreover, these methods focus on estimating the intrinsic contribution of each feature, neglecting the combinatorial components. Our method applies perturbations directly on the input in order to deal with a black-box scenario, and allows us to estimate higher-order interactions between the features x = (x1, ..., xd) that contribute to f(x).

heading: 3.1 Random Perturbation
text: Formally, let us define a probability space (Ω,X , P ) of possible input perturbations and a random vector X = (X1, ..., Xd) as a stochastic perturbation of the original input x (see Fig. 1). There are several ways to define random perturbations corresponding to different coverage of the data manifold around x. For instance, we can consider the perturbation mask operator Φ : X ×M → X which combines a stochastic mask M = (M1, ...,Md) ∈M (i.e., an i.i.d sequence of real-valued random variables on [0, 1]d) with the original input x. This formulation encompasses Inpainting perturbations: Φ(x,M) = x M + (1−M)µ with µ ∈ R a baseline value, and the Hadamard product. This consists in linearly varying the pixel intensities towards a baseline intensity such as a pure black with a value of zero [6, 4, 2, 8]. Similarly, Blurring consists of applying a blur operator with various intensities to certain regions of the image [6]. Different perturbation domains can be considered for other types of data such as textual or tabular data that we discuss further in the experimental section. In the next section, we explain how we adapt the Sobol-based sensitivity analysis using a class of perturbations to explain the predictions of a black-box model.

heading: 3.2 Sensitivity analysis using Sobol indices
text: We first briefly review the classical Sobol-Hoeffding decomposition from [33] and introduce the Sobol indices. Let (X1, ..., Xd) be independent variables and assume that f belongs to L2(X , P ).
Moreover we denote the set U = {1, ..., d}, u a subset of U , its complementary ∼u and E(·) the expectation over the perturbation space. The Hoeffding decomposition allows us to express the function f into summands of increasing dimension, denoting fu the partial contribution of variables Xu = (Xi)i∈u to the score f(X):
f(X) = f∅ + d∑ i fi(Xi) + ∑ 16i<j6d fi,j(Xi, Xj) + · · ·+ f1,...,d(X1, ..., Xd)
= ∑ u⊆U fu(Xu)
(1)
Eq. 1 consists of 2d terms and is unique under the following orthogonality constraint:
∀(u,v) ⊆ U2 s.t. u 6= v, E(fu(Xu)fv(Xv)) = 0 (2)
Furthermore, orthogonality yields the characterization fu(X) = E(f(X)|Xu) − ∑
v⊂u fv(X) and allows us to decompose the model variance as:
Var(f(X)) = d∑ i Var(fi(Xi)) + ∑ 16i<j6d Var(fi,j(Xi, Xj)) + ...+ Var(f1,...,d(X1, ..., Xd))
= ∑ u⊆U Var(fu(Xu))
(3) Building from Eq. 3, it is natural to characterize the influence of any input subset u as its own variance w.r.t. the total variance. This yields, after normalization by Var(f(X)), the general definition of Sobol indices.
Definition 1 (Sobol indices [26]) The sensitivity index Su which measures the contribution of the variable set Xu to the model response f(X) in terms of fluctuation is given by:
Su = Var(fu(Xu))
Var(f(X)) =
Var(E(f(X)|Xu))− ∑
v⊂u Var(E(f(X)|Xv)) Var(f(X))
(4)
Sobol indices give a quantification of the importance of any subset of features with respect to the model decision, in the form of a normalized measure of the model output deviation from f(X). Thus, Sobol indices sum to one : ∑ u⊆U Su = 1.
For each subset of variables Xu, the associated Sobol index Su describes the proportion of the model’s output variance explained by this subset. In particular, the first-order Sobol indices Si capture the intrinsic share of total variance explained by a particular variable, without taking into account its interactions. Many attribution methods construct such intrinsic importance estimator. However, the framework of Sobol indices enables us to capture higher-order interactions between features. In this view, we define the Total Sobol indices.
Definition 2 (Total Sobol indices [34]) The total Sobol index STi which measures the contribution of the variable Xi as well as its interactions of any order with any other input variables to the model output variance is given by:
STi = ∑ u⊆U i∈u Su = 1− VarX∼i(EXi(f(X)|X∼i)) Var(f(X)) = EX∼i(VarXi(f(X)|X∼i)) Var(f(X))
(5)
Where EX∼i(VarXi(f(X)|X∼i)) is the expected variance that would be left if all variables but Xi were to be fixed. STi is the sum of the Sobol indices for the all the possible groups of variables where i appears, i.e. first and higher order interactions of variable Xi.
Since the total interaction index contains the first order index, it is natural that it is greater than or equal to the first order index. We thus note the property which can easily be deduced: ∀i, 0 ≤ Si ≤ STi ≤ 1. We will now see why these two indices and the difference between them make them relevant for the explainability of a black-box model.
These statistics quantify the intrinsic (first-order indices) and relational (total indices) impact of each variable to the model output. A variable with a low total Sobol index is therefore not important to explain the model decision. Also, a variable has a weak interaction with other variables when STi ≈ Si, while it has a strong interaction when the difference between its two indices is high. A strong interaction means that the effect of one variable on the variation of the model output depends on other variables. Thus, using Sobol indices allows to describe fine grained interactions between inputs which leads to the model decision. We next present an efficient method to estimate these indices.

heading: 3.3 Efficient estimator
text: As models are becoming more and more complex, the proposed estimator must take into account the computational cost of model evaluation. Many efficient estimators have been proposed in the literature [16]. In this work, we use the Jansen [19] estimator which is often considered as one of the most efficient [21]. Jansen is typically used with a Monte Carlo sampling strategy. We improve over Monte Carlo by using a Quasi-Monte Carlo (QMC) sampling strategy which generates low-discrepancy sample sequences allowing a faster and more stable convergence rate [35]. For more information, see appendix D. We will now describe the procedure to implement these estimators.
We start by drawing two independent matrices of size N × d of perturbation masks from a Sobol low discrepancy LPτ sequences. N will be our number of designs and we recall that d is our dimensions (e.g, 121 for 11 by 11 masks). Once the perturbation operator is applied to x with these masks, we obtain two matrices A and B of the same size as the perturbed inputs (i.e., partially masked images). We note Aji and Bji the elements of the matrices such that i = 1, ..., d the number of variables studied and j = 1, ..., N the number of samples in each matrix. We form the new matrix C(i) in the same way as A except for the fact that the column corresponding to the variable i is now replaced by the column of B. We denote f∅ = 1N ∑N j=0 f(Aj) and the empirical variance
V̂ = 1N−1 ∑N j=0(f(Aj)− f∅)2. The empirical estimators for first (Ŝi) and total order (ŜTi) can be formulated as:
Ŝi = V̂− 12N
∑N j=1(f(Bj)− f(C (i) j )) 2
V̂ ŜTi =
1 2N ∑N j=1(f(Aj)− f(C (i) j )) 2
V̂ (6)
Hence, to compute the set of first order and total indices, it is necessary to perform N(d+2) forwards of the model. We study in section 3.3 how to choose a sufficient number of forwards (N ). To ease understanding and demonstrate that these estimators can be easily implemented, we show in Algorithm 1 a minimal pythonic implementation of the total order estimator that outputs ŜTi indices. The input Y contains the prediction scores of the N ∗ (d + 2) forwards. The scores are ordered following the same QMC sampling ordering of their associated mask. The output STis contains d importance scores, one for each dimension of the mask. In the case of images, we obtain our final explanation map by applying a bilinear upsampling to match the dimensions of the input image.
def total_order_estimator(Y, N=32, d=11*11): fA, fB, fC = Y[:N], Y[N:N*2], [Y[N*2+N*i:N*2+N*(i+1)] for i in range(d)] f0 = mean(fA)
V = sum([(val - f0)**2 for val in fA]) / (len(fA) - 1) STis = [sum((fA - fC[i])**2) / (2 * N) / V for i in range(d)] return STis
Algorithm 1: Pythonic implementation of the Total Order indices (ŜTi ) calculation.

heading: 3.3.1 Signed estimator
text: Although the proposed Sobol-based attribution method allows us to determine the impact of any variables for a given prediction and thus to identify diagnostic ones, it lacks the ability to highlight the type of contributions made, whether positive or negative. Simple methods such as “Occlusion” typically include this information. Hence, we propose a variant that combines the importance scores of the total Sobol indices with the sign of the occlusion. We compute the difference in score between the prediction on the original input x and a partial version x\i with the variable xi occluded. Intuitively, this provides an estimate of the direction of the variations generated by the variables studied with respect to a reference state.
Ŝ∆Ti = ŜTi × sign(f(x)− f(x\i)) (7)

heading: 4 Experiments
text: To evaluate the benefits and the reliability of the Sobol attribution method, we performed multiple systematic experiments on vision and natural language models using common explainability metrics.
For our vision experiments, we compared the plausibility of the explanations produced on the Pointing Game [36] benchmark. We evaluate the fidelity of our explanations using the Deletion metric for 4 representative models commonly used in explainability studies: ResNet50V2 [37] , VGG16 [38], EfficientNet [39] and MobileNetV2 [40] trained on ILSVRC-2012 [41]. In addition, we also compared the speed of convergence of the proposed estimator with that of the leading approach, RISE [8], on the same models. For our NLP experiments, we fine-tuned a Bert model and trained a bi-LSTM on the IMDB sentiment analysis dataset [42] before comparing fidelity scores using word-deletion for representative methods.
Throughout this work, explanations were generated using the Sobol total estimator ŜTi on the target class output. In the supplementary material, we demonstrate the effectiveness of modeling higher-order interactions between image regions by comparing ŜTi against Ŝi which only models the main effects. For the experiments involving images, the masks were generated at a resolution of d′ = 11× 11 pixels, then upsampled with a nearest-neighbor interpolation method before being applied with the Inpainting perturbation function. Finally, N was set to 32 which is equivalent to 3, 936 forward passes (see Section 3.3 for details). For Ŝ∆Ti , an occlusion using the same resolution as the masks was used to sign ŜTi , with zero as baseline. For RISE [8], we have followed the recommendations of the original paper with 8, 000 forward passes for all models.

heading: 4.1 Pointing game
text: Different evaluation methods have been proposed to compare attribution methods and their explanations [43, 44, 45, 46]. The first common approach consists in measuring the plausibility of an explanation as the correlation between attribution maps and human-provided semantic annotations. Here, we focused on the Pointing Game used in [36, 6, 23, 8]. For each attribution method, we compute a contribution score for each pixel of a given class of objects, e.g., bike or car. We then calculated the percentage of times the pixel with the highest score is included in the bounding box surrounding the object of interest. In this benchmark, a good attribution method should point to the most important evidence of the object appearance in accordance with a human user.
In Table 1, a report results for the Pascal VOC [47] and MS COCO [48] datasets using VGG16 [38] and ResNet50[37]. In the last column we report the computation times for each method averaged over 100 MS COCO samples for the ResNet50 model. We subdivided explanation methods into two
categories: white-box methods which require the use of backpropagation, such as Gradient [2] and Extremal Perturbation [23], and/or access to the internal states of the model, such as GradCAM [5] versus black-box methods such as Occlusion [2], RISE [8], or the proposed Sobol method STi which only require the final model predictions. The proposed method outperforms RISE [8] on all of the tested cases, while reducing the number of forward passes by half. Surprisingly, white-box methods do not always lead to higher scores, and indeed Ŝ∆Ti is the leading method for Pascal VOC / VGG16 and our two estimators Ŝ∆Ti , ŜTi prevail on COCO / VGG16. Also note that our signed version of the estimator obtains higher scores overall. This might be due to the fact that images from VOC and COCO often feature several types of objects. Thus, the maximum variance in the output is not always induced by the object of interest but can be due to the masking of another object in the image. This result suggests that our signed version Ŝ∆Ti should be used on multi-label datasets, while ŜTi should be used on multi-class datasets. We indeed confirm this in the next set of experiments on a multi-class dataset.

heading: 4.2 Fidelity
text: There is a broad consensus that measuring the plausibility of an explanation alone is insufficient [49, 50]. Indeed, if an explanation is used to make a critical decision, users expect an explanation to reflect the true underlying decision process of the model and not just a consensus with humans. Failures to do so could have disastrous consequences. A first major limitation of current evaluation methods based on human-provided groundtruth such as the pointing game is that they do not work when a model prediction is wrong. In this case, an explanation method can be penalized for not pointing to the correct evidence even though explaining prediction errors is a critical use case for explanation methods. Another limitation of these evaluation methods is that they make the implicit assumption that the models should be relying on the same image regions than humans for recognition [51, 52, 53], which is likely to be an incorrect assumption. We thus use the fidelity metric as a complementary type of evaluation. This metric assumes that the more faithful an explanation is, the quicker the prediction score should drop when pixels that are considered important are reset to a baseline value (e.g., gray values).
In Table S1, we report results for the Deletion Metric [8] (or 1− AOPC [43]) for 4 different pretrained models: ResNet50 [37] , VGG16 [38], EfficientNet [39] and MobileNet [40] on 2,000 images sampled from the ImageNet validation set. TensorFlow [54] and the Keras [55] API were used to run the models. Several baseline values can be used [56], but we chose the standard approach with gray values. We observe that the proposed Sobol ŜTi is the most faithful black-box methods with the lowest deletion scores across all models. Overall ŜTi is able to match the scores of the most faithful white-box method, namely Integrated Gradients [7], and gets the lowest score on ResNet50V2 with 0.121 against 0.123 (lower is better). We also report that our signed version Ŝ∆Ti is less faithful that the standard Sobol ŜTi. This can be explained by the fact that ImageNet images contains only one object and therefore the main variance area generally coincides with the class to be explained. This confirms our observation on the previous pointing game benchmark that ŜTi should be preferred in a multi-class setup and Ŝ∆Ti in a multi-label setup. Another metric called Insertion has been proposed by the authors of RISE [8]. Instead of deleting pixels in the original image like with Deletion, Insertion consists in adding pixels on a baseline image, e.g. one gray image, starting with pixels that are associated with the highest importance scores for a given explanation method. An issue with Insertion is that the score computed along the insertion path is highly influenced by the first inserted pixels which contributes disproportionately. A good score on this metric therefore requires exploring a region very far from the original image and closer to the baseline. For this reason, we rather preferred to focus our study on Deletion than Insertion. However, we also report results on Insertion in the supplementary material using the same hyperparameters as used in Deletion.

heading: 4.3 Efficiency
text: The black-box methods presented so far compete with white-box methods that do not require access to the internal representation of the model at the cost of a large number of forward passes, e.g., around 8, 000 for RISE [8]. This weakness leads us to take a more serious look at the performance of the proposed method. It seems critical for the deployment of black-box methods to lower the amount of compute required to produce correct explanations. We describe an experiment to
show that beyond producing higher quality explanations, our estimator converges quickly. We first generate an explanation with a high number of forward passes that is large enough to reach convergence, e.g. 10, 000 forward passes. Then we compare this explanation that “converged” to other explanations obtained with lower numbers of forward passes. It allows us to measure the stability and rate of convergence towards this explanation that “converged”, but more practically to find the proper trade-off between the amount of compute and the quality of explanations. This procedure requires defining a measure of similarity between two explanations. Since the proper interpretation method is to rank the features most sensitive to the model’s decision, it seems natural to consider the Spearman rank correlation [58] to compare the similarity between explanations. Prior work has provided theoretical and experimental arguments in line with this choice [50, 49, 59, 46].
In Fig. 2, we compare the proposed Sobol attribution method ŜTi against RISE [8], which is the current state-of-the-art for black-box methods. We use their respective gold explanation generated after 10, 000 forwards. To allow a fair comparison, both methods use masks generated in 7×7 dimensions (as recommended by RISE [8]). We report the average results and variance over 500 images from the ImageNet validation set using EfficientNet, a convolutional neural network optimized for fast forward computing times. We observe that our method exhibit higher convergence rate by getting higher Spearman’s rank correlation of 0.8 after only 1,000 forwards against 0.65 for RISE, and consistently obtain higher scores until reaching 0.97 with 7,000 forwards against 0.73 for RISE. Additionally, we observe that Sobol has a more stable convergence by getting an overall lower
variance than RISE. This implies that the number of forward passes used in Sobol can be greatly reduced to accommodate computational resources constraints compared to RISE. Indeed, RISE proposes to use 8, 000 forwards, but our method is faster and reaches better results with half the number of passes. We report similar results for other neural network architectures in the supplementary material D. Finally, we perform an ablation study of Sobol to show the impact of lowering the number of forwards on the Deletion benchmark. We report competitive scores with 16 times fewer number of forwards than RISE by reaching 0.151 in Deletion score with 492 forwards. For reference, Sobol was reaching state-of-the-art results of 0.121 with 3,936 forwards. We report additional scores in the supplementary materials S2.

heading: 4.4 Word deletion
text: For NLP, black-box methods require the use of perturbations that can be applied to the space of characters, words or sentences. For instance, a common perturbation consists in simply removing one word of the sentence to be explained. Therefore, the Inpainting perturbation that we used with Sobol to reduce the intensity of pixels in a continuous manner cannot be directly applied in this context. Instead, we adapt it by binarizing the masks such that Φ(x,M) = x dM− 0.5e, i.e., if the value is greater than 0.5 the word is kept, otherwise it is removed. We then verify that our Sobol method can be used to identify words that support a specific decision of a text classifier. Inspired by previous work [11, 12, 60], we introduce an experimental benchmark on the IMDB Review dataset [61]. It is similar to the previous Deletion benchmark for images in that it focuses on assessing the faithfulness of the explanation and does not require specific human annotations. More precisely, we first trained two models: a bi-LSTM from scratch and a BERT model fine-tuned for the task (see appendix H for details). We generate explanations on 1, 000 sentences from the validation dataset. An explanation associates an importance score to each word. Similar to Deletion, we use these scores to successively remove the most relevant word of the sentence and measure the corresponding drop in the prediction score of the model.
In Table 3, we report results for explanation methods that are commonly used in NLP. Both our two Sobol methods have a better faithfulness than all the tested gradient-based white-box methods including Saliency, Grad-Input, SmoothGrad, and Integr-Grad. For instance, Sobol Ŝ∆Ti even reaches a low Word deletion scores of 0.461 (lower is better) for the bi-LSTM compared to 0.529 for the best white-box approach. However, the proposed methods are only the second and third most faithful methods. Occlusion (often called Omit-1 in NLP) reaches the lowest score of 0.440 for LSTM and 0.531 for BERT, against 0.461 and 0.598 for Sobol Ŝ∆Ti . This is due to the fact that the default distribution of masks used in Sobol is centered around 0.5, which corresponds to removing on average half of the words as opposed to a single word for Occlusion. In IMDB, this causes the frequent removal of critical words that support the model decision. Indeed, we report comparable results with Occlusion (e.g., 0.527 for BERT) for a lower threshold of 0.05 to remove far fewer words. Since Sobol can model higher-order interactions between words, we believe that it could successfully be used for NLP tasks that are more complex than sentiment classification on IMDB.

heading: 5 Conclusion
text: We have presented a novel explainability method to study and understand the predictions of a blackbox model. This approach is grounded within the theoretical framework of sensitivity analysis using Sobol indices. A non-trivial contribution of this work was to make the approach tractable and efficient for high-dimensional data such as images. For this purpose, we have introduced a method using perturbation masks coupled with efficient estimators from the sensitivity analysis literature. One additional benefit of the approach is that it provides a way to study the importance of not just the main effects of input variables but also higher-order interactions between them. We showed that our method can be efficiently used to explain the decisions of image classifiers. It reaches performance on par with or better than the current best black-box methods while being twice as fast. It even reaches comparable results to the best white-box methods without requiring access to internal states. We also showed that our method could be applied to language models and reported initial competitive results. It is our hope that this work will guide further efforts in the search for more general and efficient explainability methods for black-box models and that further links will be made with the field of sensitivity analysis.

Review summaries:
In this work, the authors propose to use Sobol indices for black-box explanations of deep neural networks on computer vision and NLP tasks. The method is based on efficiently estimating Sobol indices and Total Sobol indices with a Quasi-Monte Carlo sampling strategy. The experiments on the tasks of pointing game, deletion, and word deletion show that the proposed method achieves comparable results to state-of-the-art attribution models. 

This paper proposes a black-box method to assess the contribution of image pixels and groups of pixels (regions) to the classification decision. The proposed approach is based on Sobol indices, that allow to characterize the influence of an input subset (image pixel/group of pixels or region) as its own variance with respect to the total variance.   To employ these indices in an efficient way, they are approximated by empirical Sobol indices. Specifically, from an input image, a set of real-valued masks is drawn from a Quasi-Monte-Carlo Sequence to generate perturbed inputs. The Sobol indices and total Sobol indices are estimated empirically from the variance of model predictions of these images. The first-order Sobol indices indicate the importance of each pixel in isolation, the total Sobol  indices also encompass higher-order interactions. The method is evaluated on standard benchmarks for vision and NLP and shows competitive performance to black box and white box methods.

This paper proposes a new method to understand the input-output causal relationship of a deep network. The proposed method is based on input perturbation (black-box), Quasi-Monte Carlo is used to sample two real-value masks for each input. Sobol indices is applied to measure the variance discrepancy between all the feature and a subset of the total. The sampling method for perturbation is efficient, and the sobol indices can capture both first- and high-order information of the variance. The proposed method can better visualise and understand the causal relationship of a learned network.