SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Siu Lun Chau  
CISPA
Helmholtz Center for Information Security 66123 Saarbrücken, Germany siu-lun.chau@cispa.de
Krikamol Muandet* CISPA
Helmholtz Center for Information Security 66123 Saarbrücken, Germany muandet@cispa.de
Dino Sejdinovic*
School of Computer and Mathematical Sciences
Adelaide, South Australia, 5005 Australia
dino.sejdinovic@adelaide.edu.au
We present a novel approach for explaining Gaussian processes (GPs) that can utilize the full analytical covariance structure present in GPs. Our method is based on the popular solution concept of Shapley values extended to stochastic cooperative games, resulting in explanations that are random variables. The GP explanations generated using our approach satisfy similar favorable axioms to standard Shapley values and possess a tractable covariance function across features and data observations. This covariance allows for quantifying explanation uncertainties and studying the statistical dependencies between explanations. We further extend our framework to the problem of predictive explanation, and propose a Shapley prior over the explanation function to predict Shapley values for new data based on previously computed ones. Our illustrations demonstrate the effectiveness of the proposed approach.
Shapley values [1], a solution concept derived from cooperative game theory, are an essential tool in explainable AI that helps understand and interpret the prediction of complex machine learning models. In particular, SHapley-Additive-exPlanation (SHAP) [2] algorithms have been applied in various fields, including finance [3], healthcare [4], and robotics [5]. SHAP algorithms compute Shapley values by formulating cooperative game payoffs based on the function obtained from the learning algorithm, e.g., a regressor. This function is often treated as fixed, resulting in deterministic game payoffs. In contrast, we argue that uncertainty in model predictions also plays an equally important role for trustworthy machine learning models as it enables users to make more informed decisions [6], assess the model's confidence [7], and identify areas where more data or improved features are needed [8]. Since uncertainty in model prediction can propagate to downstream explanation, explainability tools should also help users better understand the model's confidence or lack thereof in its predictions, allowing them to calibrate their trust in the explanations. By providing transparency into both the model's predictions and uncertainty, AI systems can become more reliable, interpretable, and trustworthy, facilitating broader adoption across domains where trust is imperative.
Gaussian processes (GPs) [9] are a natural model class for developing an explanation method that can account for predictive uncertainty. While there exist model-specific explanation methods for popular models such as LinearSHAP [10] for linear models, TreeSHAP [11] for trees, DeepSHAP [2] for deep networks, and RKHS-SHAP [12] for kernel methods, there is a lack of GP-specific Shapley value based explanation methods despite the ubiquity of GPs in machine learning. While feature importance can be studied via automatic relevance determination (ARD) lengthscales [9] of the covariance kernel, they only reveal global notions of importance which are not associated to the specific predictions, and are limited to the specific classes of kernels. Yoshikawa and Iwata [13] have proposed using a GP with a local linear regression component for interpretability, but this approach results in a specific interpretable GP model rather than a general GP explanation algorithm. Recently, Marx et al. [14] proposed to sample multiple realisations of the GP function and to apply a model-agnostic explanation algorithm to each sample, thus getting a distribution over explanations. While this in principle can give uncertainties in explanations, it is computationally expensive and not tailored for GPs. It does not take advantage of the various important properties that GPs enjoy, such as the fact that posterior means are reproducing kernel Hilbert space (RKHS) [15] functions and that we have a fully tractable covariance structure. These properties, as we later demonstrate, can lead to more effective estimation of Shapley values and help to understand the statistical dependencies across these explanations. Additionally, this covariance structure allows us to formulate explanations themselves as GPs and thus are able to predict explanations of unseen data.
To address this gap, we present the GP-SHAP algorithm. In Section 2, we contextualize our work by introducing the concepts of stochastic cooperative games and stochastic Shapley values. Both are studied thoroughly by game theorists [16, 17], but have not been utilized by the explanation community. Section 3 characterizes the stochastic cooperative game induced by GPs through the use of the conditional mean process [18, 8], a GP of conditional expectations. We use the weighted least squares formulation of Shapley values [19, 2] to express explanations as multivariate Gaussians with a tractable covariance structure. We then introduce the GP-SHAP algorithm as a method for estimating these stochastic Shapley values. In addition, we extend the GP-SHAP algorithm by combining it with the Bayesian weighted least squares approach by Slack et al. [20]. The extension we call BayesGP-SHAP integrates two sources of uncertainty: GP predictive posterior uncertainty as well as the Shapley value estimation uncertainty (in the cases where Shapley values are too expensive to compute exactly). Section 4 expands beyond GP explanations, demonstrating the applicability of our framework to other predictive models. We consider the setting of predictive explanations and propose a Shapley prior to represent explanations for more general predictive models as GPs. This allows us to predict explanations for new data without relying on the standard Shapley value procedure and provide predictive uncertainty to more general explanations beyond GP models, such as explanations from TreeSHAP, DeepSHAP, and KernelSHAP. Illustrations of the effectiveness of GP-SHAP, BayesGP-SHAP, and the Shapley prior for predictive explanations are provided in Section 5 and we conclude the paper in Section 6. All proofs in this paper are given in the appendix.
2 Stochastic cooperative games and their Shapley values
In this section, we review the concepts of stochastic cooperative games, denoted as s-games, and their corresponding stochastic Shapley values, referred to as SSVs. These concepts provide the necessary language to introduce GP-SHAP in the coming section. It is important to note that game theorists have studied s-games in various forms [16, 17, 21], but they have not been adequately introduced in the explanation literature. While Covert and Lee [22] briefly discussed s-games, they focused on computing deterministic Shapley values (DSVs) of the mean of the s-game, overlooking the inherent uncertainty in s-games and their potential application to uncertainty-aware explanations.
Problem formulation. The results presented below are analogous to the original work of Shapley's [1], but adapted to games with random outcomes. Formally, let  $\Omega$  denote the player set and let  $\nu : 2^{\Omega} \to \mathcal{L}_2(\mathbb{R})$  be a s-game with random variable payoff. We restrict our attention to stochastic payoffs with finite second moments as we aim to characterise the variance of the corresponding stochastic Shapley values. However, this is not a necessary condition to show existence and uniqueness of the SSVs. We further require  $\nu(\emptyset) = \delta_{\nu_0}$  where  $\delta_{\nu_0}$  is the Dirac measure at some constant  $\nu_0 \in \mathbb{R}$ . We introduce the following concepts to formalise the axioms for SSVs:
Definition 1 (Carrier). A carrier of  $\nu$  is any  $N\subseteq \Omega$  such that  $\nu (S) = \nu (N\cap S)$  for all  $S\subseteq \Omega$
Definition 2 (Permutation s-game). Denote  $\Pi(\Omega)$  the set of permutations on  $\Omega$ . For  $\pi \in \Pi(\Omega)$ , the induced permutation s-game is  $\nu_{\pi}(\pi S) \coloneqq \nu(S)$  for all  $S \subseteq \Omega$ , where  $\pi S$  is the image of  $S$  under  $\pi$ .
Definition 3 (Stochastic value allocation). Denote  $\mathcal{G}$  the space of  $s$ -games for  $\Omega$ . We say  $\phi: \mathcal{G} \to \mathcal{L}_2(\mathbb{R})^{|\Omega|}$  is a value allocation of  $\nu$  that assigns to each player  $i \in \Omega$  a stochastic value  $\phi_i(\nu)$ .
The natural extension of the original axioms by Shapley are extended to s-games as follows:
1. (s-symmetry) For any  $\pi \in \Pi (\Omega),\phi_{\pi i}(\nu_{\pi}) = \phi_{i}(\nu)$
2. (s-efficiency) For each carrier  $N$  of  $\nu$ ,  $\sum_{i\in N}\phi_i(\nu) = \nu(N)$ .
3. (s-linearity) For any  $\nu, \omega \in \mathcal{G}$ ,  $\phi(\nu + \omega) = \phi(\nu) + \phi(\omega)$  where addition of s-games are defined as  $(\nu + \omega)(S) \coloneqq \nu(S) + \omega(S)$  for all  $S \subseteq \Omega$ .
Note that all equality happens at the random variable level. Unsurprisingly, there is a unique stochastic value allocation that satisfies these three axioms:
Theorem 4 (Stochastic Shapley values). The only stochastic value allocation  $\phi$  of  $\nu$  satisfying  $s$ -symmetry,  $s$ -efficiency, and  $s$ -linearity takes the following form,
$$
\phi_ {i} (\nu) = \sum_ {S \subseteq N \backslash \{i \}} c _ {| S |} \left(\nu (S \cup i) - \nu (S)\right) \tag {1}
$$
where  $N$  is the smallest carrier set of  $\Omega$ ,  $c_{|S|} = \frac{1}{|N|}\binom{|N|-1}{|S|}^{-1}$  and  $\phi_i(\nu)$  is the  $i^{th}$  SSV of  $s$ -game  $\nu$ .
Note that Equation (1) is equivalent to the DSVs, except it is written in terms of random variables. We emphasise that this result has been proven in Ma et al. [16] using a top-down approach, i.e. starting with Equation (1) and verify it satisfies the stochastic axioms and uniqueness. In the appendix, we offer a contrasting perspective where we mirror Shapley's original bottom-up derivation, i.e. began with the stochastic axioms and subsequently determined the unique solution.
2.1 Variances of stochastic Shapley values are not Shapley values of variance games
As (1) is now defined by summing over a weighted differences between random variables, we can analyse the corresponding mean and variances across the SSVs. In the following, denote  $\bar{\phi}:\bar{\mathcal{G}}\to \mathbb{R}^{|\Omega |}$  as the deterministic Shapley value allocation where  $\bar{\mathcal{G}} := \{\bar{\nu}:\Omega \to \mathbb{R}\}$  is the space of deterministic cooperative games, referred to as d-games from now on.
Proposition 5. Given the player set  $\Omega$ , let  $\nu$  be a stochastic game,  $\phi$  a stochastic Shapley value allocation, and  $\bar{\phi}$  a deterministic Shapley value allocation. Suppose that  $\mathbb{E}[\nu]$  and  $\mathbb{V}[\nu]$  are the corresponding mean and variance  $d$ -games, respectively. Then,  $\mathbb{E}[\phi(\nu)] = \bar{\phi}(\mathbb{E}[\nu])$ , but  $\mathbb{V}[\phi(\nu)] \neq \bar{\phi}(\mathbb{V}[\nu])$ . In particular, the SSV variance is given by
$$
\mathbb {V} [ \phi_ {i} (\nu) ] = \sum_ {S \subseteq N \setminus \{i \}} \sum_ {S ^ {\prime} \subseteq N \setminus \{i \}} c _ {| S |} c _ {| S ^ {\prime} |} \big (\mathbb {C} [ \nu_ {S \cup i}, \nu_ {S ^ {\prime} \cup i} ] - \mathbb {C} [ \nu_ {S \cup i}, \nu_ {S ^ {\prime}} ] - \mathbb {C} [ \nu_ {S}, \nu_ {S ^ {\prime} \cup i} ] + \mathbb {C} [ \nu_ {S}, \nu_ {S ^ {\prime}} ] \big),
$$
where  $\nu_{S} = \nu(S)$  and  $\mathbb{C}$  is the covariance function between the stochastic payoffs.
Proposition 5 highlights the difference between variances of stochastic Shapley values, which capture the propagated uncertainties through the s-game, and deterministic Shapley values of variance games [23, 24], i.e. deterministic game  $\bar{\nu}(S) = \mathbb{V}[\nu(S)]$  for all  $S \subseteq \Omega$  for some stochastic game  $\nu$ . Therefore, in the context of model explanations, variance-based allocation techniques such as the Shapley effects [25] and the work of Fryer et al. [26], where they computed Shapley values on games constructed using the variance and coefficient of determination  $R^2$  respectively, cannot be used to capture uncertainty of explanations, as those approaches are themselves explaining the model variance instead. Besides stochastic Shapley values, it is possible to leverage the framework of coalition interval games [27] and interval Shapley values [28] to provide confidence intervals for feature explanations, as Napolitano et al. [29] demonstrated. However, their approach is based on confidence interval of predictive models, thus this frequentist approach is not applicable to Bayesian models such as GPs and the obtained uncertainty around explanations have very different interpretations as well.
At first glance, computing the variance of each stochastic Shapley value seems cumbersome as it requires summing over all possible coalitions twice. However, there exists a compact expression for the full covariance matrix of stochastic Shapley values across players for stochastic games constructed using a Gaussian process model, which we demonstrate in the following section.
3 Explaining GPs with GP-SHAP
This section is dedicated to introducing GP-SHAP, where the key idea is about efficiently computing stochastic Shapley values for stochastic games that models conditional expectations of Gaussian processes. Specifically, in Section 3.1, we review the process of constructing Shapley values for model explanations using deterministic cooperative games. We then proceed to formalize the stochastic cooperative game induced by pushing the GP functions through the d-game and demonstrate that the resulting stochastic Shapley values follow a multivariate Gaussian distribution. Section 3.2 is dedicated to the estimation procedure and the introduction of two algorithms for computing GP explanations: GP-SHAP and BayesGP-SHAP. GP-SHAP provides explanations that incorporate the GP predictive uncertainty, while BayesGP-SHAP extends this further by accounting for the additional estimation uncertainty arising in the weighted least squares procedure. This extension is similar to how BayesSHAP introduced by Slack et al. [20] extends SHAP.
3.1 Formulating stochastic Shapley values for GP models
Let  $X, Y$  be random variables taking values in the  $d$ -dimensional instance space  $\mathcal{X} \subseteq \mathbb{R}^d$  and label space  $\mathcal{Y}$  (could be in  $\mathbb{R}$  or discrete) respectively with the joint distribution  $p(X, Y)$ . We use  $[d] := \{1, \dots, d\}$  to denote feature index set and  $S \subseteq [d]$  as the feature index subset. For supervised learning with GPs, the usual goal is to model  $P(Y \mid X = x) = \psi(f(\mathbf{x}))$  [9] where  $f$  is the GP function of interest and  $\psi$  is a problem-specific transformation, e.g.,  $\psi$  is the identity map for regression and sigmoid transformation for classification problem. We then posit a prior  $f \sim \mathcal{GP}(0, k)$  where  $k: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$  is a covariance kernel, and compute the posterior distribution (or its approximation)  $p(f \mid \mathbf{D})$ , which is typically also a GP with posterior mean  $\tilde{m}$  and kernel  $\tilde{k}$ .
Explaining deterministic  $f$  with d-game. To explain a deterministic function  $f$ , we create a d-game by treating each feature as a player and constructing the game with  $f$ . The resulting DSVs then provide feature attributions that satisfy various favourable properties, such as the sum of the attribution equals to the prediction made at that specific point, identical features receive the same attribution, and null feature receives zero attribution [2]. The d-game for local attribution on an observation  $\mathbf{x}$  typically involves computing the expected value of  $f(X)$  using a reference distribution  $r(X \mid X_S = \mathbf{x}_S)$  for different feature subsets  $S \subseteq [d]$ ,
$$
\bar {\nu} _ {f} (\mathbf {x}, S) := \mathbb {E} _ {r} [ f (X) \mid X _ {S} = \mathbf {x} _ {S} ]. \tag {2}
$$
This approach, also known as removal-based explanation [30], has been used in various explanation algorithms [2, 11, 12, 31]. These d-games determine the "worth" of features in  $S$  by looking at the expectation after "removing" the contribution of other features in  $[d] \backslash S$  through integration. The choice of reference distribution leads to explanations with different properties, such as improved locality of estimation [32], promotion of fairness [33], or the incorporation of causal knowledge [34, 35]. In this paper, we focus on the scenario where  $r$  corresponds to the data distribution, i.e.,  $r(X \mid X_S) = p(X \mid X_S)$ , as it is one of the most frequently used approaches in the literature [36, 37, 12, 31]. Although different reference functions result in different estimation procedures, our formulation of s-games and SSVs for GPs still holds.
Induced s-game by stochastic  $f$ . Now suppose  $f$  is a random function with a posterior distribution  $p(f \mid \mathbf{D})$ . The corresponding stochastic cooperative game with players  $[d]$  can be defined analogously to the deterministic case as  $\nu_{p(f|\mathbf{D})} : \mathcal{X} \times 2^{[d]} \to \mathcal{L}_2(\mathbb{R})$  such that, for a given observation  $\mathbf{x} \in \mathcal{X}$  and feature subset  $S \subseteq [d]$ , the stochastic payoff is
$$
\nu_ {p (f | \mathbf {D})} (\mathbf {x}, S) = \mathbb {E} _ {X} [ f (X) \mid X _ {S} = \mathbf {x} _ {S} ]. \tag {3}
$$
For brevity, we write  $\nu_{p(f|\mathbf{D})}$  as  $\nu_{f}$ , bearing in mind that  $\nu_{f}$  is a random function since  $f$  is a random function. In particular, when  $f$  is a GP with mean function  $\tilde{m}$  and covariance function  $\tilde{k}$ , the stochastic payoff  $\nu_{f}$  is also a GP indexed by  $\mathbf{x}$  and  $S$ .
Proposition 6 (Stochastic game  $\nu_{f}$  as induced GP). Let  $f\sim \mathcal{GP}(\tilde{m},\tilde{k})$  with integrable sample paths, i.e.  $\int_{\mathcal{X}}|f|dp_X < \infty$  almost surely. The stochastic payoff function  $\nu_{f}$  induced by  $f$  is a Gaussian process with the following mean and covariance functions:
$$
m _ {\nu} (\mathbf {x}, S) := \mathbb {E} _ {X} [ \tilde {m} (X) \mid X _ {S} = \mathbf {x} _ {S} ], \tag {4}
$$
$$
k _ {\nu} \left(\left(\mathbf {x}, S\right), \left(\mathbf {x} ^ {\prime}, S ^ {\prime}\right)\right) := \mathbb {E} _ {X, X ^ {\prime}} \left[ \tilde {k} \left(X, X ^ {\prime}\right) \mid X _ {S} = \mathbf {x} _ {S}, X _ {S ^ {\prime}} ^ {\prime} = \mathbf {x} _ {S ^ {\prime}} ^ {\prime} \right]. \tag {5}
$$
In other words, the stochastic game  $\nu_{f}$  now assigns a payoff distribution to each feature subset  $S$ , where the variability arises from taking conditional expectation on a random function  $f$ . We note that this GP is also known as the conditional mean process, previously studied in Chau et al. [8, 18].
Stochastic Shapley values as multivariate Gaussians for GPs. Given the s-game  $\nu_{f}$ , we can characterise the corresponding SSVs using the weighted least squares formulation of DSVs [19, 2]. For each coalition  $S_{j} \subseteq [d]$ , let  $z_{j} \in \{0,1\}^{d}$  be the binary vector such that  $z_{j}[i] = 1$  if  $i \in S_{j}$  and  $\mathbf{Z} \in \{0,1\}^{2^{d} \times d}$  the concatenation of all  $z_{j}$  vectors. When the context is clear, we use  $\mathbf{Z}$  and  $[d]$  interchangeably. Let  $\mathbf{W} \in \mathbb{R}^{2^{d} \times 2^{d}}$  be the diagonal matrix with entries  $W_{jj} = w(S_{j}) = \frac{d-1}{(|S_{j}|)|S_{j}|(d-S_{j}|)}$  for all subsets with size  $0 < |S_{j}| < d$ . When  $S_{j}$  has 0 or  $d$  elements, the weights are set to  $\infty$  to enforce the efficiency axiom.
Theorem 7 (Stochastic Shapley values of  $\nu_{f}$ ). Let  $\nu_{f}$  be an induced stochastic game from the  $GP$ $f\sim \mathcal{GP}(\tilde{m},\tilde{k})$  and denote  $\mathbf{v}_{\mathbf{x}}\coloneqq [\nu_f(\mathbf{x},S_1),\dots \nu_f(\mathbf{x},S_{2^d})]^{\top}$  the vector of stochastic payoffs across all coalitions, then the corresponding stochastic Shapley values  $\phi (\nu_{f}(\mathbf{x},\cdot))$  follows a  $d$ -dimensional multivariate Gaussian distribution,
$$
\phi \left(\nu_ {f} (\mathbf {x}, \cdot)\right) \sim \mathcal {N} \left(\mathbf {A} \mathbb {E} \left[ \mathbf {v} _ {\mathbf {x}} \right], \mathbf {A} \mathbb {V} \left[ \mathbf {v} _ {x} \right] \mathbf {A} ^ {\top}\right) \quad w i t h \quad \mathbf {A} := \left(\mathbf {Z} ^ {\top} \mathbf {W} \mathbf {Z}\right) ^ {- 1} \mathbf {Z} ^ {\top} \mathbf {W}, \tag {6}
$$
where  $\mathbb{E}[\mathbf{v}_x] \in \mathbb{R}^{2^d}$  and  $\mathbb{V}[\mathbf{v}_x] \in \mathbb{R}^{2^d \times 2^d}$  are the corresponding mean vector and covariance matrix of the payoffs.
The derivation is straightforward using the fact that the stochastic payoffs  $\mathbf{v}_{\mathbf{x}}$  are multivariate Gaussian random variables and the matrix  $\mathbf{A}$  is obtained from the weighted regression formulation of Shapley values. The resulting variance of  $\phi (\nu_f(\mathbf{x},\cdot))$  can then be interpreted as the uncertainties around the explanations, propagated from the posterior variance of  $f$ .
3.2 Estimation algorithms: the GP-SHAP and the BayesGP-SHAP
In this section, we introduce our main algorithm GP-SHAP and its variant BayesGP-SHAP to estimate the stochastic Shapley values for GP posterior distributions.
Formulating GP-SHAP. Given a set of observations  $\mathbf{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^n = (\mathbf{X}, \mathbf{y})$  with GP prior  $f \sim \mathcal{GP}(0, k)$ , we can obtain the posterior GP  $f \mid \mathbf{D} \sim \mathcal{GP}(\tilde{m}, \tilde{k})$  by using the Gaussian conditioning rule for regression or e.g. the variational or Laplace approximation for classification. In fact, the conditional expectations of mean and covariance function from a posterior GP can be estimated using conditional mean embeddings [38] without the need of explicit density estimations, following derivations of Chau et al. [8, 18]. We provide technical discussion on conditional mean embeddings in the appendix.
Proposition 8 (Estimating  $\nu_{f}$ ). Given  $\mathbf{D} = (\mathbf{X},\mathbf{y})$  and the posterior  $GPf\mid \mathbf{D}\sim \mathcal{GP}(\tilde{m},\tilde{k})$ , the mean and covariance function of the stochastic cooperative game  $\nu_{f}$  can be estimated as,
$$
\hat {m} _ {\nu} (\mathbf {x}, S) = \mathbf {b} (\mathbf {x}, S) ^ {\top} \tilde {m} (\mathbf {X}), \quad \hat {k} _ {\nu} \left(\left(\mathbf {x}, S\right), \left(\mathbf {x} ^ {\prime}, S ^ {\prime}\right)\right) = \mathbf {b} (\mathbf {x}, S) ^ {\top} \tilde {\mathbf {K}} _ {\mathbf {X X}} \mathbf {b} \left(\mathbf {x} ^ {\prime}, S ^ {\prime}\right), \tag {7}
$$
where  $\mathbf{b}(\mathbf{x},S) := (\mathbf{K}_{\mathbf{X}_S\mathbf{X}_S} + \lambda I)^{-1}k_S(\mathbf{X}_S,\mathbf{x}_S)$ ,  $\tilde{m}(\mathbf{X}) = [\tilde{m}(\mathbf{x}_1),\dots,\tilde{m}(\mathbf{x}_n)]^\top$ , and  $k_S: \mathcal{X}_S \times \mathcal{X}_S \to \mathbb{R}$  is the kernel defined on the sub-feature space of  $\mathcal{X}$  and we write  $k_S(\mathbf{x}_S,\mathbf{X}_S) := [k_S(\mathbf{x}_S,\mathbf{x}_{1S}),\dots,k_S(\mathbf{x}_S,\mathbf{x}_{nS})]$  and  $\mathbf{K}_{\mathbf{XX}}$  and  $\tilde{\mathbf{K}}_{\mathbf{XX}}$  as the gram matrix of  $\mathbf{X}$  using kernel  $k$  and  $\tilde{k}$  respectively. The parameter  $\lambda > 0$  is a fixed hyperparameter to stabilise the inversion.
It is worth noting that the estimation  $\hat{m}_{\nu}$  coincides with the one deployed in RKHS-SHAP [12] since we are also utilising conditional mean embeddings for the estimation of conditional expectation of RKHS functions. In addition, we obtain the analytical covariance function that is used to estimate the covariance of the stochastic Shapley values in the following proposition:
Proposition 9 (GP-SHAP). Let the matrix  $\mathbf{A}$  be defined as in Theorem 7. The mean and covariance for the multivariate stochastic Shapley values can be estimated as,