Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysis
Authors: Thomas FEL
Remi Cadene
Mathieu Chalvidal
Matthieu Cord
David Vigouroux
Thomas Serre
Venue: NeurIPS 2021 Poster (2021)
Keywords: deep learning
machine learning
explainability
interpretability
computer vision
nlp
black box
sobol

Abstract: We describe a novel attribution method which is grounded in Sensitivity Analysis and uses  Sobol indices. Beyond modeling the individual contributions of image regions,  Sobol indices provide an efficient way to capture higher-order interactions between image regions and their contributions to a neural network's prediction through the lens of variance.
We describe an approach that makes the computation of these indices efficient for high-dimensional problems by using perturbation masks coupled with efficient estimators to handle the high dimensionality of images.
Importantly, we show that the proposed method leads to favorable scores on standard benchmarks for vision (and language models) while drastically reducing the computing time compared to other black-box methods -- even surpassing the accuracy of state-of-the-art white-box methods which require access to internal representations. Our code is freely available:
github.com/fel-thomas/Sobol-Attribution-Method.

Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysis
Thomas 1..4.
Rémi Cadène 1.2
Mathieu Chalvidal
Matthieu 2.5
David Vigouroux 3,1
Thomas Serre
Cammey Institute for Brain Science, Brown University, USA 2Sorbonne Université, CNRS, France Artificial and Natural Intelligence Toulouse Institute, Université de Toulouse, France
Institut de Recherche Technologique Saint-Exupeny France 5 Valeo.ai
{ tthomas_feee fel, remi remi_eemmeeeeeee cadene @brown.
Abstract
We describe novel attribution method which is grounded in Sensitivity Analysis and uses Sobol indices. Beyond modeling the individual contributions of image regions, Sobol indices provide an efficient way to capture higher-order interactions between image regions and their contributions to neural network's prediction through the lens variance. We describe an approach that makes the computation of these indices efficient for high-dimensional problems by using perturbation masks coupled with efficient estimators to handle the high dimensionality images. Importantly, show that the proposed method leads to favorable scores standard benchmarks for vision (and language models) while drastically reducing the computing time compared other black-box methods accuracy of state-of-the-ee white-box methods which require access to internal representations. Our code is freely available: github. com/ fe fel-tele tomaahhhhmm// Met hod.
surpassing the
Aebol-Attrinn
Introduction
Deep neural networks portation, security or finances with broad societal implications. Yet, these networks have become nearly inscrutable, and for most real-world applications, these systems are used to make critical decisions often without any explanation. In recent years, numerous explainability methods have been proposed [1, 3, 4, 5, 8]. addition helping improve people's trust in these systems, these methods have helped identify and correct biases in datasets [4, 5, 9]. This has, in turn, helped improve these systems' robustness and accelerate their broad deployment. An important limitation of standard explainability methods is that they require access to the system's 's internal states including hidden layer activations or input gradients [1, 5, 6, 10J. As a result, these so-called white-box methods cannot be applied in the most general situations for which the internal states of network are not publicly accessible. For instance, it common for companies to use neural networks provided by third parties (e.g., through web APIs or specialized hardware). However, only handful of so-called black-box methods have been proposed to address this challenge with limited successes [2, 4, 8]. It is thus critical to develop more general methods that can reliably interpret and characterize the underlying decision processes of a wider array models.
being deployed in numerous domains including medicine, trans-
Common approaches to explaining a model's prediction consists of attributing score for each input dimension such as image pixels for computer vision systems individual words for natural language processing. Shown in Fig. is an example image and associated importance map for an image categorization system, whereby scores for individual pixels are displayed as heatmaps where
Equal contribution "ork done before April 2021 and joining Tesla
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

S
Sys
X
0.8
M QMC
Black-box box
0.2
S, Sobol indices
S n
0.6
Figure I: (Left) Sobol Attribution Method overview. Our method aims to explain the prediction of black-box model for given image. We first sample set of real-valued masks M drawn from Quasi-Monte Carlo (QMC) sequence. We apply these masks to the input image through a perturbation function & (here the Inpainting function) to form perturbed inputs X that we forward to the black box f to obtain prediction scores. Using the masks M and the associated prediction scores, we finally produce explanation Sy, which characterizes the importance of each region by estimating the total order Sobol indices. While Sy, encompasses the effects of first and all higher-order non-linear interactions between pixel regions, the importance of a region in isolation (e.g.. the eyes of the cats). (Right) Sample explanations for ResNet50V2. Comparing explanations produced with S; and ST; helps highlight the importance of individual image regions in isolation vs. jointly (e.g., the lynx tips are important but conditioned on the presence of the presence of eye).
can also produce the first-order Sobol indices S; that reflect
hotter locations correspond to pixels that contribute the most to the system's final prediction. In the context of black-box models, core challenge is to derive these heatmaps using only the output predictions available through the network's 's forward pass. A simple approach consists in applying a given perturbation at specific location on the input image to then measure how the corresponding prediction is affected. the case of image models, pixel intensities are simply set to a default value corresponding pure black gray value; in the case of language models, individual words removed entirely from the text [11, 12]. However, evaluating the impact of these perturbations one dimension at a time fails to identify all of the non-linear interactions between input variables that are known to prevail in complex system such as a deep neural network. However, estimating the combined effect of perturbations across multiple locations quickly becomes combinatorially intractable. Methods have been recently proposed to try to address some of these issues by grouping dimensions together, such by grouping pixels within a neighborhood of the image (superpixel) [2] or sampling perturbation masks that affect multiple regions of the input [4, 8J. A first limitation of these approaches includes the use of Monte Carlo sampling methods which require high number of forward passes making these approaches computationally expensive. A second limitation is that
they rely on relatively simple perturbations such as flipping pixels on or off [2, 4, 8]. This severely constrains the space of perturbations considered and limits the efficiency with which the space of perturbations can be explored.
We address these limitations by introducing an attribution method that leverages variance-based sensitivity analysis techniques, and more specifically Sobol indices [13]. These methods were initially introduced to help identify the input variables that have the most influence on the variance of the output of non-linear mathematical system [14]. These were traditionally used in physics and economics to estimate the part of the variance induced by a group of variables on system's 's output [15, 16, 17]. Our main contribution is general framework to explain predictions from black-box models by adapting Sobol indices to be used in conjunction with perturbation masks. One of the originalities of our attribution method is that is designed to support standard perturbations and real-valued intensity perturbations that can generate continuous range of perturbations. Our second contribution is a tractable method for the calculation of Sobol indices. This is done by first sampling the perturbation masks following Quasi-Monte Carlo sequences, which efficiently covers the space of perturbations. This can be done best by leveraging the most efficient estimator borrowed from the sensitivity analysis literature [18, 19, 20, 21]. As a result, the proposed method can be

---

efficiently applied to sigh-dimensional inputs such as images. It produces on par with or better explanations than the state-of-thes with at least half the number of forward passes. Another benefit of our method is that it allows to characterize not only the main effect of image regions but also higher-order interactions by decomposing the variance of the system's 's prediction using the Sobol indices. We run extensive experiments to demonstrate the benefits of the proposed method on several recent complementary benchmarks including the "Pointing Game" and "Deletion". Our primary focus is image classification, but our method is general and we include results on text classification benchmarks as well.
2 Related work
Attribution methods for white-box models A widely used family of attribution methods uses gradient computations via the backpropagation algorithm. The first such method was introduced in [22] and later refined in [1] in the context of deep convolutional networks. The idea is to backpropagate the gradient from classification unit all the way down the input pixels. The resulting gradient image indicates which pixels affect the decision score the most. Other gradient- based methods including DeConvNet [2] and Guided Backprop [3] were specifically developed to deal with certain activation functions such as ReL.U. However, this family of methods are limited by the fact that they focus on the influence of individual pixels in an infinitesimal neighborhood around the input image in the image space. For instance, it has been shown that gradients often vanish when the prediction score to be explained is near the maximum value [7]. Integrated Gradient [7] partially addresses this issue by accumulating gradients along straight interpolation path from a baseline state to the original image.
Yet another class of gradient-based methods [6. 23] iteratively optimize a perturbation mask over regions of an image. It leads different type of explanation given by the final perturbation mask. Such an approach might leverage more meaningful perturbations but it turns out to be computationally very slow with 20 seconds per explanation limiting its broad applicability. In comparison, our proposed approach also works on image regions but it does not require iterative gradient computation and can thus be parallclized.
Another family of attribution methods relies on the neural network's s activations. Popular examples include CAM [24] which computes attribution score based on weighted sum of feature channels activities right before the classification layer. GradCAM [5] extends CAM via the use of gradients to reweigh each feature channel to take into account their importance for the predicted class. These methods also work on image regions because they are computed on the output of feature maps which themselves at coarser resolution than the original image. In comparison, proposed approach is model-agnostic and hence does not require access to internal computations.
Attribution methods for black-box models Most similar to our approach are attribution methods that can be used to explain the predictions of truly black-box models. These methods probe a neural network's 's responses perturbations over image regions and combine the resulting predictions into an influence score for each individual pixel or group of pixels. The simplest method, "Occlusion" [2], masks individual image regions- one at time with an occluding mask set to a baseline value and assigns the corresponding prediction scores to all pixels within the occluded region. Then the explanation is given by these prediction scores and can be easily interpreted. However, occlusion fails to account for the joint (higher-order) interactions between multiple image regions. For instance, occluding two image regions- one at time may only decrease the model's 's prediction minimally (say single eye or mouth component on a face) while occluding these two regions together may yield substantial change in the model's 's prediction if these two regions interact non-linearly as is expected for a deep neural network.
Our work, together with related methods such LIME [4] and RISE [8], addresses this problem by randomly perturbating the input image in multiple regions at a time. Obviously, perturbating multiple image locations simultaneously leads to a combinatorial explosion in the number of combinations and methods have been proposed to make these approaches more tractable. For instance, a popular method, LIME [4], takes superpixels regions to perturbate instead of individual pixels. An influence score is then computed for a set of connected pixel patches indicating how strongly a patch is correlated to the model predictions.
RISE [8] relies on Monte Carlo sampling to generate a set of binary masks, each value the masks representing pixel region. By probing the model with randomly masked versions of the input,
3

---

RISE [8] produces a importance map by considering the average of the masks weighted by their associated prediction scores, Instead of using binary masks, our method considers a continuous range of perturbations which allows for a finer exploration of the model's 's response. Our method can still use the same perturbations as used in Occlusion [2], LIME [4] and RISE [8] but it also enables the use of more advanced perturbation functions that take continuous inputs.
More importantly, the aforementioned methods lack a rigorous framework. Here, we introduce theoretical framework that decomposes the influence score of each individual region between multiple orders of influence. The first-order approximates Occlusion [2] by considering the influence of one region at a time, while the second-order considers two regions at a time, etc. The decomposition also includes higher-orders.
Variance- based sensitivity analysis Our attribution method builds on the variance-based sensitiv- ity analysis framework. The approach was introduced in the 70s [25] and reached a cornerstone with the Sobol indices [26]. Sobol indices are currently used in many fields (including those that said to be safety-criticall especially for the analysis of physical phenomena [16]. They are used to identify the input dimensions that have the highest influence on the output of model or a mathematical system. Several statistical estimators to compute these indices are available [15, 27, 20, 28, 29] and have asymptotic guarantees [20, 30, 31]. We build this literature by adapting these Sobol indices in the context of black-box models to compute the influence of regions of an image on the output predictions using perturbation masks.
3 Sobol attribution method
In this work, we formulate the feature attribution problem as quantifying the contribution of collection of d real-valued features æ = (II, Ta) with respect to model decision. Specifically, we consider a black-box decision function f : : + IR whose internal states and analytical form are unknown (for instance, can score the probability for the input to belong to specific class). Our goal is to quantify the importance of each feature to the decision score f(z), not just individually but also collectively. To capture these higher-order interactions, our method consists in estimating the Sobol indices of the features x by randomly perturbating them and evaluating the impact of these perturbations on the prediction of the black-box model (Fig. 1).
Considering variations of f() in response to meaningful perturbations of the input æ is a natural way to interpret the local behavior of the decision function f around x. Several methods build on this idea, c.g., by removing one or group of input features [2, 4, 6, 8, 23] or by back- propagaingg the gradient to the input space through the model [1. 7, 32, 5]. Most of these methods use the model's internal representations and/or require computing the gradient w.r.t. the input, which makes them unusable in black-box setting. Moreover, these methods focus on estimating the intrinsic contribution of each feature, neglecting the combinatorial components. Our method applies perturbations directly on the input in order to deal with black-box scenario, and allows to estimate higher-order interactions between the features æ = (II, ...; Ta) that contribute to fæx
3.1 Random Perturbation
Formally, let us define a probability space (2, P) of possible input perturbations and a random vector X = (X1,.. Xa) as a stochastic perturbation of the original input x: (see Fig. 1). There are several ways to define random perturbations corresponding to different coverage of the data manifold around æ. For instance, we can consider the perturbation mask operator &b X' x M - X' which combines a stochastic mask M- = (M1, .... Ma) E .M (i.e., an i.i.d sequence of real-valued random variables on [0. 1]d) with the original input x. This formulation encompasses Inpainting perturbations: ((x, M) = @ M + (1 - M) ) with μ E IR a baseline value, and @ the Hadamard product. This consists in linearly varying the pixel intensities towards a baseline intensity such as a pure black with value of zero [6, 4, 2, 8). Similarly, Blurring consists of applying a blur operator with various intensities to certain regions of the image [6]. Different perturbation domains can be considered for other types of data such as textual or tabular data that we discuss further in the experimental section. In the next section, we explain how we adapt the Sobol-based sensitivity analysis using a class of perturbations to explain the predictions of a black-box model.
3.2 Sensitivity analysis using Sobol indices
We first briefly review the classical Sobol-Hoeffding decomposition from [33] and introduce the Sobol indices. Let (X (X1,..., Xa) be independent variables and assume that f belongs to IL (LY

---

Moreover we denote the set 4 = [1..... d). u a subset of 24, its complementary ~uu and E(-) the expectation over the perturbation space. The Hoeffding decomposition allows to express the function f into summands of increasing dimension, denoting Su the partial contribution of variables X, (Xi) to the score f(X):
f(X) = fo + Σ f(X)) E
fx,(X,, +.. f Xa) IEijjed
(u
uCH
Eq. consists 2d terms and is unique under the following orthogonalitty constraint:
V(u, v) C u s.t. u v, fo =0
(2)
Furthermore, orthogonality yields the characterization fu (X) [E((()) - Eveu sx(() and allows to decompose the model variance
Var Var(f(X)) Var(f:(X,)
Σ LEKiEE
Var Var(fis(X,,X)) +... Var Xa))
Σ Var(fu(X..)) uCu
(3)
Building from Eq. 3, is natural to characterize the influence of any input subset u as its variance w.r.t. the total variance. This yields, after normalization by Var (f()). the general definition of Sobol indices.
Definition (Sobol indices [26]) The sensitivity index Su which measures the contribution of the variable set Xu the model response fXX) in terms of fluctuation is given by:
Var(fu(X.)) Xx)()(((X))xx)) - -ooc)))-oooatttttt Var(B(((X))x.)) Var(f(X))
S_ Var Var(f(X))
Sobol indices give a quantification of the importance of any subset of features with respect to the model decision, in the form of normalized measure of the model output deviation from f(X). Thus, Sobol indices sum to one Eugu Sn
For each subset of variables Xu, the associated Sobol index S., describes the proportion of the model's output variance explained by this subset. In particular, the first-order Sobol indices S capture the intrinsic share of total variance explained by particular variable, without taking into account its interactions. Many attribution methods construct such intrinsic importance estimator. However, the framework of Sobol indices enables capture higher-order interactions between features. this view, we define the Total Sobol indices.
Definition (Total Sobol indices [34]) The total Sobol index Sr, which measures the contribution of the variable X, as well as its interactions of any order with any other input variables to the model output variance given by:
Varx (Exx, (f(X))x..) Ex., (Varx, (f(X) Var(f(X))
ST. >>=>s,m
=1 S.=11 Ss
(5)
Var(f(X))
uCH
Where Ex (Varx, (f(X) is the expected variance that would be left if all variables but X; were to be fixed. Sy; is the sum of the Sobol indices for the all the possible groups variables where i appears, i.e. first and higher order interactions of variable X,.
Since the total interaction index contains the first order index, is natural that is greater than or equal to the first order index. We thus note the property which can easily be deduced: Vi, 0 < S < Sy, 1. We will now see why these two indices and the difference between them make them relevant for the explainability of black-box model.
5

---

These statistics quantify the intrinsic (first-order indices) and relational (total indices) impact of each variable to the model output. A variable with a low total Sobol index therefore not important to explain the model decision. Also, variable has weak interaction with other variables when Sr, S;, while has a strong interaction when the difference between its two indices is high. A strong interaction means that the effect of one variable on the variation of the model output depends on other variables. Thus, using Sobol indices allows to describe fine grained interactions between inputs which leads to the model decision. We next present an efficient method to estimate these indices.
3.3 Efficient estimator
As models are becoming more and more complex, the proposed estimator must take into account the computational cost of model evaluation. Many efficient estimators have been proposed in the literature [16]. In this work,
the Jansen [19] estimator which is often considered as one of the most efficient [21]. Jansen is typically used with a Monte Carlo sampling strategy. We improve over Monte Carlo by using Quasi-Monte Carlo (QMC) sampling strategy which generates low-diserepancy sample sequences allowing a faster and more stable convergence rate [35]. For more information, see appendix D. We will now describe the procedure to implement these estimators.
We start by drawing two independent matrices of size x of perturbation masks from a Sobol low discrepancy LP, sequences. N will be our number of designs and recall that is our dimensions (e.g, 121 for 11 by 11 masks). Once the perturbation operator is applied to æ with these masks, obtain two matrices A and B of the same size the perturbed inputs (i.e., partially masked images). We note A and Bys the elements of the matrices such that = =1,... d the number of variables studied and j 1,... N the number of samples in each matrix. We form the new matrix C6) in the same way as A except for the fact that the column corresponding to the variable is now replaced by the column of B. We denote fe N/ Ey=o f(A,) and the empirical variance fo)?. The empirical estimators for first (S,) and total order (ST, can be
V= N-T D)=o(f(A))))
formulated as:
V 2N E ((())))(())"' ST.
1
J 2N EM(AA -F(C)))2
(6)
Hence, to compute the set of first order and total indices, it is necessary to perform (dd+ forwards of the model. We study in section 3.3 how to choose sufficient number of forwards (N). To ease understanding and demonstrate that these estimators can be easily implemented, we show in Algorithm minimal pythonic implementation of the total order estimator that outputs Sr, indices. The input Y contains the prediction scores of the N * (d + 2) forwards. The scores are ordered following the same QMC sampling ordering of their associated mask. The output STis contains d importance scores, one for each dimension of the mask. In the case of images, we obtain our final explanation map by applying bilinear upsampling to match the dimensions of the input image.
def @eeell_oder__ttiin fA, fB, fC = Y[:N], Y[N:N+2), [Y [N[N+2+N) : (1+1)) for i in range (d) ]
(Y, N-32, d-11+11):
E0 = mean (EA)
v sum (1 (val f0) for val in fA]) (1en (fA) 1)
STis [sumt (fA- fc(i)) (2 * N) v for in range (d)
return STis
Algorithm 1: Pythonic implementation of the Total Order indices (St, ) calculation.
3.3.1 Signed estimator
Although the proposed Sobol-based attribution method allows us to determine the impact of any variables for a given prediction and thus identify diagnostic ones, it lacks the ability to highlight the type of contributions made, whether positive or negative. Simple methods such as "Occlusion" typically include this information. Hence, we propose variant that combines the importance scores of the total Sobol indices with the sign of the occlusion. We compute the difference in score between the prediction the original input 2: and a partial version Di with the variable 2; occluded. Intuitively, this provides an estimate of the direction of the variations generated by the variables studied with respect to reference state.
ST. = Sr, sign(f(æ) f(x())

---

VOCO7 Test (AIl/Diff)
COC014 Val (AIl/Diff)
Method
Time (s)
VGG16
ResNet50
VGG16
ResNer50
Baseline Center
69.6 42.4
69.6 42.4
27.8 19.5
27.8 / 19.5
Saliency [1]
76.3 56.9
72.3 56.8
37.7 31.4
35.0/29.4
0.031
DeconvNet [2]
67.5 44.2
44.7
23.0
30.0/21.9
0.029
White box
Guided-Backpreg [3]
75.9 / 53.0
7 59.4
39.1/31.4
42.1/35.3
0.051
MWP [36]
77.1 / 56.6
84.4
39.8/32.8
49.6/43.9
0.039
cMWP [36]
79.9 / 66.5
90.7/82.1
49.7/
58.5/53.6
0.040
GradCAM [5]
86.6 74.0
90.4/82.3
54.2/ 49.0
57.3/
0.015
Extremallertturbation [23]
88.0 / 76.1
/8.9
51.5/45.9
56.5/51.5
26.48
Black box
RISE [8]
86.9 / 75.1
86.4
50.8/45.
54.7/50.0
13.19
Sobol (Sr,) (ours)
87.7 / 75.3
89.6/802.
54.6/ 49.6
57.3 52.7
6.381
Sobol signed (SA)) (ours)
89.8 82.4
86.7 73..0
5 49.9
57.5 53.1
6.721
Table 1: Pointing game. Accuracy over the full test and subset of difficult images (defined in [36]). The first and second best results are bolded and underlined. Results are based on Py PyToch re-implementations using the TorchRay package. The reported execution time is an average over 100 runs on ResNet50 using an Nvidia Tesla PI00 on Google Colab and a batch size of 64, Lower execution time can be reached with higher batch size.
Experiments
To evaluate the benefits and the reliability of the Sobol attribution method, we performed multiple systematic experiments vision and natural language models using common explainability metrics.
For our vision experiments, we compared the plausibility of the explanations produced on the Pointing Game [36] benchmark. We evaluate the fidelity of our explanations using the Deletion metric for representative models commonly used explainability studies: ResNe150V2 [37]. VGG16 [38], EfficientNet [39] and MobileNetV2 [40] trained on ILSVRC-2012 [41]. J. In addition, we also compared the speed of convergence of the proposed estimator with that of the leading approach, RISE [8]. the same models. For our NLP experiments, we fine-tuned Bert model and trained a bi-LSTM on the IMDB sentiment analysis dataset [42] before comparing fidelity scores using word-deletion for representative methods.
Throughout this work, explanations were generated using the Sobol total estimator Sr. the target class output. In the supplementary material, we demonstrate the effectiveness of modeling higher-order interactions between image regions by comparing Sr, against S; which only models the main effects. For the experiments involving images, the masks were generated at resolution of d' x pixels, then upsampled with a nearest-neighbor interpolation method before being applied with the Inpainting perturbation function. Finally, N was set to 32 which is equivalent to 3. 936 forward passes (see Section 3.3 for details). For SA an occlusion using the same resolution as the masks used to sign ST. with as baseline. For RISE [8], have followed the recommendations the original paper with 8, 000 forward passes for all models.
4.1 Pointing game
Different evaluation methods have been proposed to compare attribution methods and their expla- nations [43, 44, 45, 46]. The first common approach consists in measuring the plausibility of an explanation the correlation between attribution maps and human- provided semantic annotations. Here, we focused on the Pointing Game used in [36, 6, 23, 8]. For each attribution method, we compute contribution score for each pixel of given class of objects, e.g., bike car. We then calculated the percentage of times the pixel with the highest score is included the bounding box surrounding the object of interest. In this benchmark, good attribution method should point to the most important evidence of the object appearance in accordance with a human user.
human-provided
In Table 1, report results for the Pascal VOC [47] and MS COCO [48] datasets using VGG16 [38] and [377. the last column we report the computation times for each method averaged over 100 MS COCO samples for the ResNet50 model. We subdivided explanation methods into two
7

---

white-box methods which require the use of backpropagation, such as Gradient [2] and Extremal Perturbation [23], and/or access to the internal states of the model, such as GradCAM [5] versus black-box methods such as Occlusion [2], RISE [8], or the proposed Sobol method Sr; which
categories:
only require the final model predictions. The proposed method outperforms RISE [8] on all of the tested cases, while reducing the number of forward passes by half. Surprisingly, white-box methods do not always lead to higher scores, and indeed S is the leading method for Pascal VOC / VGGI6
and our two estimators S Sr. prevail on COCO VGG16. Also note that our signed version of the estimator obtains higher scores overall. This might be due to the fact that images from VOC and COCO often feature several types of objects. Thus, the maximum variance in the output is not always induced by the object of interest but can be due to the masking of another object in the image. This result suggests that our signed version S should be used on multi-label datasets, while Sr, should be used on multi-class datasets. We indeed confirm this in the next set of experiments on a multi-class dataset.
4.2 Fidelity
There is broad consensus that measuring the plausibility of an explanation alone is insufficient [49, 50J. Indeed, if an explanation used to make a critical decision, users expect an explanation to reflect the true underlying decision process of the model and not just consensus with humans. Failures to do so could have disastrous consequences. A first major limitation of current evaluation methods based numan-providee groundtruth such as the pointing game is that they do not work when model prediction is wrong. In this case, an explanation method can be penalized for not pointing to the correct evidence even though explaining prediction errors is critical use case for explanation methods. Another limitation of these evaluation methods is that they make the implicit assumption that the models should be relying the same image regions than humans for recognition [51, 52, 53], which is likely to be an incorrect assumption. We thus use the fidelity metric as a complementary type of evaluation. This metric assumes that the more faithful explanation is, the quicker the prediction score should drop when pixels that are considered important are reset to a baseline value (e.g.. gray values).
In Table S1, we report results for the Deletion Metric [8] (or - AOPC [43]) for 4 different pre- trained models: ResNet50 [37]. VGG16 [38], EfficientNet [39] and MobileNet [40] on 2,000 images sampled from the ImageNet validation set. TensorFlow [54] and the Keras [55] API were used to run the models. Several baseline values can be used [56], but chose the standard approach with gray values. We observe that the proposed Sobol ST; is the most faithful black-box methods with the lowest deletion scores across all models. Overall S7. is able match the scores of the most faithful white-box method, namely Integrated Gradients [7], and gets the lowest score on ResNet50V2 with 0.121 against 0.123 (lower is better). We also report that our signed version S is less faithful that the standard Sobol ST, This can be explained by the fact that ImageNet images contains only one object and therefore the main variance area generally coincides with the class to be explained. This confirms our observation on the previous pointing game benchmark that ST, should be preferred in multi-class setup and بي in a multi-label setup.
Another metric called Insertion has been proposed by the authors of RISE [8]. Instead of deleting pixels in the original image like with Deletion, Insertion consists in adding pixels on a baseline image, e.g. one gray image, starting with pixels that are associated with the highest importance scores for a given explanation method. An issue with Insertion is that the score computed along the insertion path highly influenced by the first inserted pixels which contributes disproportionately. this metric therefore requires exploring region very far from the original image and closer to the baseline. For this reason, we rather preferred to focus our study on Deletion than Insertion. However, we also report results on Insertion in the supplementary material using the same hyperparameters as used in Deletion.
good score on
4.3 Efficiency
The black-box methods presented so far compete with white-box methods that do not require access to the internal representation of the model at the cost of a large number of forward passes, e.g., around 8, 000 for RISE [8]. This weakness leads us to take more serious look at the performance of the proposed method. It seems critical for the deployment of black-box methods to lower the amount of compute required to produce correct explanations. We describe experiment to
8

---

Method
ResNer50V2
VGG16
EfficientNet
MobileNerV2
Baseline Random (ours)
0.235
0.168
0.124
0.137
Saliency [1]
0.174
0.134
0.105
0.125
Shiided-Backprop [3]
0.142
0.138
0.105
0.102
White box
DeconvNet [2]
0.159
0.146
0.105
0.111
Grad.-Input [57]
0.140
0.096
0.093
0.103
Integ.-Grad [7]
0.123
0.095
0.091
0.093
SmoothGrad [32]
0.130
0.106
0.094
0.098
GradCAM [5]
0.141
0.118
0.130
0.122
Black box
Occlusion [2]
0.350
0.357
0.252
0.357
RISE [8]
0.127
0.121
0.119
0.114
Sobol (Sr,) (ours)
0.121
0.109
0.104
0.107
Sobol signed ( (ours)
0.145
0.114
0.147
0.141
Table 2: Deletion scores obtained on 2,000 ImageNet validation set images. Lower is better. Random consists in removing pixels each step random. The first and second best results are bolded and underlined.
show that beyond producing higher quality explanations, our estimator converges quickly. We first generate explanation with high number of forward passes that is large enough to reach convergence, e.g. 10, 000 forward passes. Then we compare this explanation that "converged" to other explanations obtained with lower numbers of forward passes. It allows to measure the stability and rate of convergence towards this explanation that "converged", but more practically to find the proper trade-off between the amount of compute and the quality of explanations. This procedure requires defining a measure of similarity between two explanations. Since the proper interpretation method is rank the features most sensitive to the model's decision, it seems natural to consider the Spearman rank correlation [58] to compare the similarity between explanations. Prior work has provided theoretical and experimental arguments in line with this choice [50, 49, 59, 46]. In Fig. 2, we compare the proposed Sobol attri- bution method Sy, against RISE [8], which is the current oftt-oofthhe-ar for black-box methods. We use their respective gold explanation generated af- ter 10, 000 forwards. To allow a fair comparison, both methods use masks generated in 7 x 7 dimen- sions (as recommended by RISE [8). We report the average results and variance over 500 images from the ImageNet validation set using Efficient- Net, convolutional neural network optimized for fast forward computing times. We observe that our method exhibit higher convergence rate by getting higher Spearman's rank correlation of 0.8 after only 1,000 forwards against 0.65 for RISE, and consistently obtain higher scores until reaching 0.97 with 7,000 forwards against 0.73 for RISE. Additionally, we observe that Sobol has a more stable convergence by getting an overall lower variance than RISE. This implies that the number of forward passes used in Sobol can be greatly reduced accommodate computational resources constraints compared RISE. Indeed, RISE pro- poses to use 8, 000 forwards, but our method is faster and reaches better results with half the number of passes. We report similar results for other neural network architectures in the supplementary material D. Finally, we perform an ablation study of Sobol to show the impact of lowering the number of forwards on the Deletion benchmark. We report competitive scores with times fewer number of forwards than RISE by reaching 0.151 in Deletion score with 492 forwards. For reference, Sobol was reaching tate-of-theear results of 0.121 with 3,936 forwards. We report additional scores the supplementary materials S2.
0.9.
0.7
0.6
Rise
Sobol
1000
Forwards
Figure 2: Comparison between the rate and sta- bility of convergence of Sobol Sr, and RISE [8]. A high Spearman correlation rank corresponds to producing explanation that is similar to the explanation that "converged", i.c. with 10, 000 forward passes. We report mean and variance computed over 500 images from the ImageNet dataset using EfficientNet.

---

Saliency
Grad- GraddInput
SmoothGrad
Integ-Graat
Occlusion
Sr;
ST;
BERT 0.684
0.682
0.682
0.689
0.531 0.440
0.662
0.598 0,461
LSTM
0.541
0.529
0.541
0.538
0.523
Table 3: Word deletion scores, obtained on 1,000 sentences. Delete up to 20 words per sentence accordingly to their relevance and track the impact on the classification performance. Lower is better. The first and second best results are bolded and underlined.
4.4 Word deletion
For NLP, black-box methods require the use of perturbations that can be applied to the space of characters, words or sentences. For instance, a common perturbation consists in simply removing one word of the sentence to be explained. Therefore, the Inpainting perturbation that we used with Sobol to reduce the intensity of pixels in a continuous manner cannot be directly applied in this context. Instead, we adapt by binarizing the masks such that (<x, M) 2 [M 0.5), i.c., if the value is greater than 0.5 the word is kept, otherwise is removed. We then verify that our Sobol method can be used to identify words that support a specific decision of text classifier. Inspired by previous work [11. 12, 60], we introduce an experimental benchmark on the IMDB Review dataset [61]. It is similar to the previous Deletion benchmark for images in that it focuses on assessing the faithfulness of the explanation and does not require specific human annotations. More precisely. we first trained two models: bi-LSTM from scratch and BERT model fine-tuned for the task (see appendix H for details). We generate explanations on 1, 000 sentences from the validation dataset. An explanation associates importance score each word. Similar Deletion, we use these scores to successively remove the most relevant word of the sentence and measure the corresponding drop in the prediction score of the model.
In Table 3, we report results for explanation methods that are commonly used in NLP. Both our two Sobol methods have a better faithfulness than all the tested gradient-based white-box methods including Saliency, Grad-Input, SmoothGrad, and integr-Grad. For instance, Sobol SA even reaches a low Word deletion scores of 0.461 (lower is better) for the bi-L.STM compared to 0.529 for the best white-box approach. However, the proposed methods are only the second and third most faithful methods. Occlusion (often called Omit-1 in NLP) reaches the lowest score of 0.440 for LSTM and 0.531 for BERT, against 0.461 and 0.598 for Sobol SA distribution of masks used in Sobol is centered around 0.5, which corresponds to removing on average half of the words as opposed to a single word for Occlusion. In IMDB, this causes the frequent removal of critical words that support the model decision. Indeed, we report comparable results with Occlusion (e.g., 0.527 for BERT) for a lower threshold of 0.05 to remove far fewer words. Since Sobol can model higher-order interactions between words, we believe that it could successfully be used for NLP tasks that are more complex than sentiment classification on IMDB.
I This due to the fact that the default
5 Conclusion
We have presented a novel explainability method to study and understand the predictions of a black- box model. This approach is grounded within the theoretical framework of sensitivity analysis using Sobol indices. A non-trivial contribution of this work was to make the approach tractable and efficient for high-dimensional data such as images. For this purpose, we have introduced a method using perturbation masks coupled with efficient estimators from the sensitivity analysis literature, One additional benefit of the approach is that provides a way to study the importance of not just the main effects of input variables but also higher-orde interactions between them. We showed that our method can be efficiently used to explain the decisions of image classifiers. It reaches performance on par with or better than the current best black-box methods while being twice as fast. It even reaches comparable results to the best white-box methods without requiring access to internal states. We also showed that our method could be applied to language models and reported initial competitive results. our hope that this work will guide further efforts in the search for more general and efficient explainability methods for black-box models and that further links will be made with the field of sensitivity analysis.
10


Reviews:
[{"review_id": "hIJp_tOrTu", "reviewer": "Reviewer_n1GW", "summary": "This paper proposes a black-box method to assess the contribution of image pixels and groups of pixels (regions) to the classification decision. The proposed approach is based on Sobol indices, that allow to characterize the influence of an input subset (image pixel/group of pixels or region) as its own variance with respect to the total variance.  \nTo employ these indices in an efficient way, they are approximated by empirical Sobol indices. Specifically, from an input image, a set of real-valued masks is drawn from a Quasi-Monte-Carlo Sequence to generate perturbed inputs. The Sobol indices and total Sobol indices are estimated empirically from the variance of model predictions of these images. The first-order Sobol indices indicate the importance of each pixel in isolation, the total Sobol  indices also encompass higher-order interactions.\nThe method is evaluated on standard benchmarks for vision and NLP and shows competitive performance to black box and white box methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a new method to asses the contribution of image pixels and regions to the classification score of an image, of the contribution of words in the case of NLP applications. The proposed paper/approach has several strengths:\n\n- The motivation to consider the relative variance of the prediction is clear and interesting.\n- The combination with the occlusion approach yields practically good results and allows to assess positive vs. negative impact to a score.\n- The paper is well written and mathematically clear.\n- The experimental evaluation for vision and language applications is thorough.\n- The efficiency is also properly evaluated\n\nWeaknesses and Questions: \n- A discussion on the discrepancy between the approximated, empirical variance and the actual variance would be interesting. Also, the accuracy of this approximation depends on the number of samples drawn. An ablation study on the needed number of samples would be interesting.\n- considering the result in the pointing game, the a version of the proposed approach outperformes RISE in every case. Yet, the signed Sobol variant is not always the best. \n- in the NLP evaluation, the Occlusion method works better than the proposed approach. I think this needs more discussion. For example, what is the role of masking during training in this context?\n\nOverall, the paper proposes an efficient model that can be useful to the community and that is well motivated. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3h"}, "extra_text": {}}, {"review_id": "M175RfVZfin", "reviewer": "Reviewer_Szkm", "summary": "This paper proposes a new method to understand the input-output causal relationship of a deep network. The proposed method is based on input perturbation (black-box), Quasi-Monte Carlo is used to sample two real-value masks for each input. Sobol indices is applied to measure the variance discrepancy between all the feature and a subset of the total. The sampling method for perturbation is efficient, and the sobol indices can capture both first- and high-order information of the variance. The proposed method can better visualise and understand the causal relationship of a learned network.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have seen some interpretability studies proposed based on variance or entropy, but they can only show first-order information. The proposed method can capture high-order information. From Fig 1, the high-order information could be different from the first-order info. Given the receptive field of a model is bigger than the input in most cases, the high-order information could play a more significant role in decision making. Thus, this high-order information study is more intuitive and reasonable to me. \n\nOne question in my mind is about the real-value mask. The use of binary mask is intuitive that some of the features(or input) will be completely ignored. When applying a real-value mask, does this actually create new patterns, e.g., a specific pixel or region could have low values due to the mask. Is it safe to consider the low-real-value region is somehow \"masked out\" for the variance computation, or could the weight will be increased to compensate this mask? I might miss some background for this setup, so I hope I can see some explanations in the rebuttal. If this is not a common setting, it is better to discuss a little in the paper. (L111 did not answer this question.)\n\nGenerally, the idea of applying sobol indices is interesting to me.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I personally like the clarity of this draft, the aim of this study is stated clearly in Section 1(a black-box method for causal relationship analysis), and the method is explained in details in Section 3 (sobol indices can deliver high-order information and Quasi_monte carlo can reduce the sampling complexity). And to my knowledge, this study of high-order information is important to better understand which regions(based on their interaction) are used for making a prediction. (other reviewers can complete the literature I missed.) If the rebuttal can explain the question I raised in the last comment, I would like to accept this study.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "three"}, "extra_text": {}}, {"review_id": "L1SSlt0tPDC", "reviewer": "Reviewer_Xxm2", "summary": "In this work, the authors propose to use Sobol indices for black-box explanations of deep neural networks on computer vision and NLP tasks. The method is based on efficiently estimating Sobol indices and Total Sobol indices with a Quasi-Monte Carlo sampling strategy. The experiments on the tasks of pointing game, deletion, and word deletion show that the proposed method achieves comparable results to state-of-the-art attribution models.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ The proposed approach is based on the Sobol indices that have been used for sensitivity analysis in many fields. \n+ The authors propose to use the Quasi-Monte Carlo sampling strategy for efficient estimation of Sobol indices\n+ The baselines and experiments are comprehensive.\n\nWeaknesses:\n- The technical novelty appears to be limited. The proposed method is the adoption of existing techniques.\n- The experimental results do not show that the method has outstanding performance, according to Table 1,2,3.\n- I do not see why black-box attribution models are necessary for practice applications. Can authors say some applications with black-box attribution models?\n- The experiments have not demonstrated that high-order information is effective in the proposed model. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitation is not clearly explained yet. The potential negative societal impact may include wrong explanation to deep learning models. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: A method for assessing the influence of image regions on neural network predictions is proposed.  Reviewers n1GW and Szkm are positive about the approach and its reported experimental performance.  In discussion following the author response, Reviewer Xxm2 stated the response's ablation study sufficiently addressed concerns.

Decision: Accept (Poster)