Title: Explaining the Uncertain: Stochastic Shapley Values for Gaussian Process Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Gaussian Processes

Shapley values

Uncertainty Modelling
Abstract: We present a novel approach for explaining Gaussian processes (GPs) that can utilize the full analytical covariance structure present in GPs. Our method is based on the popular solution concept of Shapley values extended to stochastic cooperative games, resulting in explanations that are random variables. The GP explanations generated using our approach satisfy similar favorable axioms to standard Shapley values and possess a tractable covariance function across features and data observations. This covariance allows for quantifying explanation uncertainties and studying the statistical dependencies between explanations. We further extend our framework to the problem of predictive explanation, and propose a Shapley prior over the explanation function to predict Shapley values for new data based on previously computed ones. Our extensive illustrations demonstrate the effectiveness of the proposed approach.

Explaining the Uncertain:

Abstract

We present a novel approach for explaining Gaussian processes (GPs) that can utilize the full analytical covariance structure present in GPs. Our method is based on the popular solution concept of Shapley values extended to stochastic cooperative games, resulting in explanations that are random variables. The GP explanations generated using our approach satisfy similar favorable axioms to standard Shapley values and possess a tractable covariance function across features and data observations. This covariance allows for quantifying explanation uncertainties and studying the statistical dependencies between explanations. We further extend our framework to the problem of predictive explanation , and propose a Shapley prior over the explanation function to predict Shapley values for new data based on previously computed ones. Our illustrations demonstrate the effectiveness of the proposed approach.

1 Introduction

Shapley values [1], a solution concept derived from cooperative game theory, are an essential tool in explainable AI that helps understand and interpret the prediction of complex machine learning models. In particular, SH apleyA dditive-ex P lanation (SHAP) [2] algorithms have been applied in various fields, including finance [3], healthcare [4], and robotics [5]. SHAP algorithms compute Shapley values by formulating cooperative game payoffs based on the function obtained from the learning algorithm, e.g., a regressor. This function is often treated as fixed, resulting in deterministic game payoffs. In contrast, we argue that uncertainty in model predictions also plays an equally important role for trustworthy machine learning models as it enables users to make more informed decisions [6], assess the model's confidence [7], and identify areas where more data or improved features are needed [8]. Since uncertainty in model prediction can propagate to downstream explanation, explainability tools should also help users better understand the model's confidence or lack thereof in its predictions, allowing them to calibrate their trust in the explanations. By providing transparency into both the model's predictions and uncertainty, AI systems can become more reliable, interpretable, and trustworthy, facilitating broader adoption across domains where trust is imperative.

Gaussian processes (GPs) [9] are a natural model class for developing an explanation method that can account for predictive uncertainty. While there exist model-specific explanation methods for popular models such as LinearSHAP [10] for linear models, TreeSHAP [11] for trees, DeepSHAP [2] for deep networks, and RKHS-SHAP [12] for kernel methods, there is a lack of GP-specific Shapley value based explanation methods despite the ubiquity of GPs in machine learning. While feature importance can be studied via automatic relevance determination (ARD) lengthscales [9] of the covariance kernel, they only reveal global notions of importance which are not associated to the specific predictions, and are limited to the specific classes of kernels. Yoshikawa and Iwata [13] have proposed using a GP with a local linear regression component for interpretability, but this approach results in a specific interpretable GP model rather than a general GP explanation algorithm. Recently, Marx et al. [14] proposed to sample multiple realisations of the GP function and to apply a model-agnostic explanation algorithm to each sample, thus getting a distribution over explanations. While this in principle can give uncertainties in explanations, it is computationally expensive and not tailored for GPs. It does not take advantage of the various important properties that GPs enjoy, such as the fact that posterior means are reproducing kernel Hilbert space (RKHS) [15] functions and that we have a fully tractable covariance structure. These properties, as we later demonstrate, can lead to more effective estimation of Shapley values and help to understand the statistical dependencies across these explanations. Additionally, this covariance structure allows us to formulate explanations themselves as GPs and thus are able to predict explanations of unseen data.

To address this gap, we present the GP-SHAP algorithm. In Section 2, we contextualize our work by introducing the concepts of stochastic cooperative games and stochastic Shapley values. Both are studied thoroughly by game theorists [16, 17], but have not been utilized by the explanation community. Section 3 characterizes the stochastic cooperative game induced by GPs through the use of the conditional mean process [18, 8], a GP of conditional expectations. We use the weighted least squares formulation of Shapley values [19, 2] to express explanations as multivariate Gaussians with a tractable covariance structure. We then introduce the GP-SHAP algorithm as a method for estimating these stochastic Shapley values. In addition, we extend the GP-SHAP algorithm by combining it with the Bayesian weighted least squares approach by Slack et al. [20]. The extension we call BayesGP-SHAP integrates two sources of uncertainty: GP predictive posterior uncertainty as well as the Shapley value estimation uncertainty (in the cases where Shapley values are too expensive to compute exactly). Section 4 expands beyond GP explanations, demonstrating the applicability of our framework to other predictive models. We consider the setting of predictive explanations and propose a Shapley prior to represent explanations for more general predictive models as GPs. This allows us to predict explanations for new data without relying on the standard Shapley value procedure and provide predictive uncertainty to more general explanations beyond GP models, such as explanations from TreeSHAP, DeepSHAP, and KernelSHAP. Illustrations of the effectiveness of GP-SHAP, BayesGP-SHAP, and the Shapley prior for predictive explanations are provided in Section 5 and we conclude the paper in Section 6. All proofs in this paper are given in the appendix.

2 Stochastic cooperative games and their Shapley values

In this section, we review the concepts of stochastic cooperative games, denoted as s-games, and their corresponding stochastic Shapley values, referred to as SSVs. These concepts provide the necessary language to introduce GP-SHAP in the coming section. It is important to note that game theorists have studied s-games in various forms [16, 17, 21], but they have not been adequately introduced in the explanation literature. While Covert and Lee [22] briefly discussed s-games, they focused on computing deterministic Shapley values (DSVs) of the mean of the s-game, overlooking the inherent uncertainty in s-games and their potential application to uncertainty-aware explanations.

Problem formulation. The results presented below are analogous to the original work of Shapley's [1], but adapted to games with random outcomes. Formally, let Ω denote the player set and let ν : 2 Ω →L 2 ( R ) be a s-game with random variable payoff. We restrict our attention to stochastic payoffs with finite second moments as we aim to characterise the variance of the corresponding stochastic Shapley values. However, this is not a necessary condition to show existence and uniqueness of the SSVs. We further require ν ( ∅ ) = δ ν 0 where δ ν 0 is the Dirac measure at some constant ν 0 ∈ R . We introduce the following concepts to formalise the axioms for SSVs:

Definition 1 (Carrier) . A carrier of ν is any N ⊆ Ω such that ν ( S ) = ν ( N ∩ S ) for all S ⊆ Ω .

Definition 2 (Permutation s-game) . Denote Π(Ω) the set of permutations on Ω . For π ∈ Π(Ω) , the induced permutation s-game is ν π ( πS ) := ν ( S ) for all S ⊆ Ω , where πS is the image of S under π .

Definition 3 (Stochastic value allocation) . Denote G the space of s-games for Ω . We say ϕ : G → L 2 ( R ) | Ω | is a value allocation of ν that assigns to each player i ∈ Ω a stochastic value ϕ i ( ν ) .

The natural extension of the original axioms by Shapley are extended to s-games as follows:

(s-symmetry) For any π ∈ Π(Ω) , ϕ πi ( ν π ) = ϕ i ( ν ) .

(s-efficiency) For each carrier N of ν , ∑ i ∈ N ϕ i ( ν ) = ν ( N ) .

(s-linearity) For any ν, ω ∈ G , ϕ ( ν + ω ) = ϕ ( ν ) + ϕ ( ω ) where addition of s-games are defined as ( ν + ω )( S ) := ν ( S ) + ω ( S ) for all S ⊆ Ω .

Note that all equality happens at the random variable level. Unsurprisingly, there is a unique stochastic value allocation that satisfies these three axioms:

Theorem 4 (Stochastic Shapley values) . The only stochastic value allocation ϕ of ν satisfying s-symmetry, s-efficiency, and s-linearity takes the following form,

where N is the smallest carrier set of Ω , c | S | = 1 | N | ( | N |-1 | S | ) -1 and ϕ i ( ν ) is the i th SSV of s-game ν .

Note that Equation (1) is equivalent to the DSVs, except it is written in terms of random variables. We emphasise that this result has been proven in Ma et al. [16] using a top-down approach, i.e. starting with Equation (1) and verify it satisfies the stochastic axioms and uniqueness. In the appendix, we offer a contrasting perspective where we mirror Shapley's original bottom-up derivation, i.e. began with the stochastic axioms and subsequently determined the unique solution.

2.1 Variances of stochastic Shapley values are not Shapley values of variance games

As (1) is now defined by summing over a weighted differences between random variables, we can analyse the corresponding mean and variances across the SSVs. In the following, denote ¯ ϕ : ¯ G → R | Ω | as the deterministic Shapley value allocation where ¯ G := { ¯ ν : Ω → R } is the space of deterministic cooperative games, referred to as d-games from now on.

̸

Proposition 5. Given the player set Ω , let ν be a stochastic game, ϕ a stochastic Shapley value allocation, and ¯ ϕ a deterministic Shapley value allocation. Suppose that E [ ν ] and V [ ν ] are the corresponding mean and variance d-games, respectively. Then, E [ ϕ ( ν )] = ¯ ϕ ( E [ ν ]) , but V [ ϕ ( ν )] = ¯ ϕ ( V [ ν ]) . In particular, the SSV variance is given by

where ν S = ν ( S ) and C is the covariance function between the stochastic payoffs.

Proposition 5 highlights the difference between variances of stochastic Shapley values, which capture the propagated uncertainties through the s-game, and deterministic Shapley values of variance games [23, 24], i.e. deterministic game ¯ ν ( S ) = V [ ν ( S )] for all S ⊆ Ω for some stochastic game ν . Therefore, in the context of model explanations, variance-based allocation techniques such as the Shapley effects [25] and the work of Fryer et al. [26], where they computed Shapley values on games constructed using the variance and coefficient of determination R 2 respectively, cannot be used to capture uncertainty of explanations, as those approaches are themselves explaining the model variance instead. Besides stochastic Shapley values, it is possible to leverage the framework of coalition interval games [27] and interval Shapley values [28] to provide confidence intervals for feature explanations, as Napolitano et al. [29] demonstrated. However, their approach is based on confidence interval of predictive models, thus this frequentist approach is not applicable to Bayesian models such as GPs and the obtained uncertainty around explanations have very different interpretations as well.

At first glance, computing the variance of each stochastic Shapley value seems cumbersome as it requires summing over all possible coalitions twice. However, there exists a compact expression for the full covariance matrix of stochastic Shapley values across players for stochastic games constructed using a Gaussian process model, which we demonstrate in the following section.

3 Explaining GPs with GP-SHAP

This section is dedicated to introducing GP-SHAP, where the key idea is about efficiently computing stochastic Shapley values for stochastic games that models conditional expectations of Gaussian processes. Specifically, in Section 3.1, we review the process of constructing Shapley values for model explanations using deterministic cooperative games. We then proceed to formalize the stochastic cooperative game induced by pushing the GP functions through the d-game and demonstrate that the resulting stochastic Shapley values follow a multivariate Gaussian distribution. Section 3.2 is dedicated to the estimation procedure and the introduction of two algorithms for computing GP explanations: GP-SHAP and BayesGP-SHAP. GP-SHAP provides explanations that incorporate the GP predictive uncertainty, while BayesGP-SHAP extends this further by accounting for the additional estimation uncertainty arising in the weighted least squares procedure. This extension is similar to how BayesSHAP introduced by Slack et al. [20] extends SHAP.

3.1 Formulating stochastic Shapley values for GP models

Let X,Y be random variables taking values in the d -dimensional instance space X ⊆ R d and label space Y (could be in R or discrete) respectively with the joint distribution p ( X,Y ) . We use [ d ] := { 1 , ..., d } to denote feature index set and S ⊆ [ d ] as the feature index subset. For supervised learning with GPs, the usual goal is to model P ( Y | X = x ) = ψ ( f ( x )) [9] where f is the GP function of interest and ψ is a problem-specific transformation, e.g., ψ is the identity map for regression and sigmoid transformation for classification problem. We then posit a prior f ∼ GP (0 , k ) where k : X × X → R is a covariance kernel, and compute the posterior distribution (or its approximation) p ( f | D ) , which is typically also a GP with posterior mean ˜ m and kernel ˜ k .

Explaining deterministic f with d-game. To explain a deterministic function f , we create a d-game by treating each feature as a player and constructing the game with f . The resulting DSVs then provide feature attributions that satisfy various favourable properties, such as the sum of the attribution equals to the prediction made at that specific point, identical features receive the same attribution, and null feature receives zero attribution [2]. The d-game for local attribution on an observation x typically involves computing the expected value of f ( X ) using a reference distribution r ( X | X S = x S ) for different feature subsets S ⊆ [ d ] ,

This approach, also known as removal-based explanation [30], has been used in various explanation algorithms [2, 11, 12, 31]. These d-games determine the 'worth' of features in S by looking at the expectation after 'removing' the contribution of other features in [ d ] \ S through integration. The choice of reference distribution leads to explanations with different properties, such as improved locality of estimation [32], promotion of fairness [33], or the incorporation of causal knowledge [34, 35]. In this paper, we focus on the scenario where r corresponds to the data distribution, i.e., r ( X | X S ) = p ( X | X S ) , as it is one of the most frequently used approaches in the literature [36, 37, 12, 31]. Although different reference functions result in different estimation procedures, our formulation of s-games and SSVs for GPs still holds.

Induced s-game by stochastic f . Now suppose f is a random function with a posterior distribution p ( f | D ) . The corresponding stochastic cooperative game with players [ d ] can be defined analogously to the deterministic case as ν p ( f | D ) : X × 2 [ d ] →L 2 ( R ) such that, for a given observation x ∈ X and feature subset S ⊆ [ d ] , the stochastic payoff is

For brevity, we write ν p ( f | D ) as ν f , bearing in mind that ν f is a random function since f is a random function. In particular, when f is a GP with mean function ˜ m and covariance function ˜ k , the stochastic payoff ν f is also a GP indexed by x and S .

Proposition 6 (Stochastic game ν f as induced GP) . Let f ∼ GP ( ˜ m, ˜ k ) with integrable sample paths, i.e. ∫ X | f | dp X < ∞ almost surely. The stochastic payoff function ν f induced by f is a Gaussian process with the following mean and covariance functions:

In other words, the stochastic game ν f now assigns a payoff distribution to each feature subset S , where the variability arise from taking conditional expectation on a random function f . We note that this GP is also known as the conditional mean process, previously studied in Chau et al. [8, 18].

Stochastic Shapley values as multivariate Gaussians for GPs. Given the s-game ν f , we can characterise the corresponding SSVs using the weighted least squares formulation of DSVs [19, 2]. For each coalition S j ⊆ [ d ] , let z j ∈ { 0 , 1 } d be the binary vector such that z j [ i ] = 1 if i ∈ S j and Z ∈ { 0 , 1 } 2 d × d the concatenation of all z j vectors. When the context is clear, we use Z and [ d ] interchangeably. Let W ∈ R 2 d × 2 d be the diagonal matrix with entries W jj = w ( S j ) = d -1 ( d | S j | ) | S j | ( d -| S j | ) for all subsets with size 0 < | S j | < d . When S j has 0 or d elements, the weights are set to ∞ to enforce the efficiency axiom.

Theorem 7 (Stochastic Shapley values of ν f ) . Let ν f be an induced stochastic game from the GP f ∼ GP ( ˜ m, ˜ k ) and denote v x := [ ν f ( x , S 1 ) , . . . ν f ( x , S 2 d )] ⊤ the vector of stochastic payoffs across all coalitions, then the corresponding stochastic Shapley values ϕ ( ν f ( x , · )) follows a d -dimensional multivariate Gaussian distribution,

where E [ v x ] ∈ R 2 d and V [ v x ] ∈ R 2 d × 2 d are the corresponding mean vector and covariance matrix of the payoffs.

The derivation is straightforward using the fact that the stochastic payoffs v x are multivariate Gaussian random variables and the matrix A is obtained from the weighted regression formulation of Shapley values. The resulting variance of ϕ ( ν f ( x , · )) can then be interpreted as the uncertainties around the explanations, propagated from the posterior variance of f .

3.2 Estimation algorithms: the GP-SHAP and the BayesGP-SHAP

In this section, we introduce our main algorithm GP-SHAP and its variant BayesGP-SHAP to estimate the stochastic Shapley values for GP posterior distributions.

Formulating GP-SHAP. Given a set of observations D = { ( x i , y i ) } n i =1 = ( X , y ) with GP prior f ∼ GP (0 , k ) , we can obtain the posterior GP f | D ∼ GP ( ˜ m, ˜ k ) by using the Gaussian conditioning rule for regression or e.g. the variational or Laplace approximation for classification. In fact, the conditional expectations of mean and covariance function from a posterior GP can be estimated using conditional mean embeddings [38] without the need of explicit density estimations, following derivations of Chau et al. [8, 18]. We provide technical discussion on conditional mean embeddings in the appendix.

Proposition 8 (Estimating ν f ) . Given D = ( X , y ) and the posterior GP f | D ∼ GP ( ˜ m, ˜ k ) , the mean and covariance function of the stochastic cooperative game ν f can be estimated as,

where b ( x , S ) := ( K X S X S + λI ) -1 k S ( X S , x S ) , ˜ m ( X ) = [ ˜ m ( x 1 ) , . . . , ˜ m ( x n )] ⊤ , and k S : X S ×X S → R is the kernel defined on the sub-feature space of X and we write k S ( x S , X S ) := [ k S ( x S , x 1 S ) , ..., k S ( x S , x nS )] and K XX and ˜ K XX as the gram matrix of X using kernel k and ˜ k respectively. The parameter λ > 0 is a fixed hyperparameter to stabilise the inversion.

It is worth noting that the estimation ˆ m ν coincides with the one deployed in RKHS-SHAP [12] since we are also utilising conditional mean embeddings for the estimation of conditional expectation of RKHS functions. In addition, we obtain the analytical covariance function that is used to estimate the covariance of the stochastic Shapley values in the following proposition:

Proposition 9 ( GP-SHAP ) . Let the matrix A be defined as in Theorem 7. The mean and covariance for the multivariate stochastic Shapley values can be estimated as,

The complete algorithm, along with a discussion of computational techniques deployed to reduce computation time, such as vectorisation across X using tensor operations and incorporation of the sparse GP formulation to speed up computations of quantities from Propositions 8 and 9, is provided in the appendix. It is worth noting that subsampling coalitions to reduce computational cost while estimating Shapley values is also a standard approach in SHAP algorithm implementations [2]. Empirically, this procedure has been shown to give an unbiased estimate of the true DSVs, with the variance decreasing at a rate of O ( 1 ℓ ) , where ℓ is the number of coalition samples [22]. However, this approach incurs an additional source of uncertainty, due to estimation, and we propose to incorporate that as well into our framework by utilising the BayesSHAP approach proposed by Slack et al. [20].

Formulating BayesGP-SHAP. To capture the estimation uncertainty, Slack et al. [20] proposed BayesSHAP and reformulated the WLS procedure as a hierarchical Bayesian weighted least square and studied the corresponding posterior distribution over the deterministic Shapley values. We provide the hierarchical data generation process below, with an abuse of notations, for a deterministic f , we have

where w and z are the weight function and binary vector respectively introduced in Theorem 7, and ℓ 0 , σ 2 0 are two hyperparameters, typically set to small values to keep priors uninformative. Slack et al. [20] showed that the posterior distribution on σ 2 and ¯ ϕ follows a scaled Invχ 2 and normal respectively, due to their corresponding conjugacies with the likelihood.

Proposition 10 (BayesSHAP [20]) . Given the data generation above, the posterior distribution on ¯ ϕ and σ 2 follows:

where ℓ is the number of coalitions S = { S j } ℓ j =1 we sample uniformly from 2 [ d ] , Z ℓ is the binary matrix representing S , and W ℓ is the corresponding weight matrix, and A ℓ = ( Z ⊤ ℓ W ℓ Z ℓ ) -1 Z ⊤ ℓ W ℓ is the WLS matrix, ¯ v x = [¯ ν f ( x , S 1 ) , ..., ¯ ν f ( x , S ℓ )] ⊤ is the vector of deterministic payoffs, and

measures the average weighted error in the regression and the norm of the mean explanations.

It is important to highlight that while both GP-SHAP and BayesSHAP have a notion of the 'posterior of Shapley values', the two sources of uncertainty which these approaches capture are very different: GP-SHAP corresponds to the predictive uncertainty induced by the GP posterior p ( f | D ) , whereas BayesSHAP is the uncertainty due to having to estimate Shapley values when their exact computation is infeasible due to the exponential number of coalitions - and is in Slack et al. [20] proposed for a deterministic f . Nonetheless, by integrating the BayesSHAP posterior of Shapley values through the posterior GP p ( f | D ) , we can in fact incorporate both sources of uncertainty, leading to our second algorithm, BayesGP-SHAP.

Proposition 11 ( BayesGP-SHAP ) . Continuing from Propositions 9 and 10, the posterior distribution of the stochastic Shapley values can be estimated using the Bayesian WLS approach as,

We note that in the above proposition, instead of integrating p ( σ 2 | Z ℓ , f, x , D ) with respect to the posterior GP, which leads to a complex scaled mixture of Gaussians, we simplify the expression and construct a scaled inverse chi-square distribution with s 2 ( E [ v x ]) instead, which represents the error of the weighted regression with respect to the mean payoffs E [ v x ] .

Conditionally on σ 2 , the posterior variance in BayesGP-SHAP is therefore the sum of the variance from GP-SHAP and BayesSHAP due to Gaussian conjugacies.

4 Predictive explanations using the Shapley prior

In this section, we move beyond standard GP explanations by formulating explanation functions of a broader class of models as Gaussian processes. We introduce a Shapley prior over the space of vector-valued explanation functions ϕ : X → R d , which correspond to Shapley values for arbitrary functions f . By treating previously obtained Shapley values as regression labels, we can predict explanations for new data without relying on the standard procedure to compute Shapley values. Our framework is not limited to explanations generated from GP-SHAP, but can also be applied to other explanation methods such as TreeSHAP, DeepSHAP, or model-agnostic KernelSHAP. The proposed approach learns the direct mapping from X to the space of Shapley values, without the need to access the underlying model f during training, which is different from previous predictive approaches such as FastSHAP [39]. In contrast to prior work, Hill et al. [40] also considered fitting a GP regression model directly to explanation functions, but their focus is on incorporating the corresponding predictive uncertainty to explanations from black-box classifiers rather than predicting Shapley values for unseen data.

To achieve this, we leverage the key ideas behind the GP-SHAP algorithm, which uses the facts that GPs remain tractable under conditional expectations (to build the s-game) and linear combinations (to compute the stochastic Shapley values). By applying these properties to a GP prior GP (0 , k ) instead of a posterior, we have an induced prior over the space of Shapley functions ϕ .

Proposition 12 (The Shapley prior over ϕ ) . The prior f ∼ GP (0 , k ) and the corresponding stochastic game ν f ( x , S ) = E [ f ( X ) | X S = x S ] induce a vector-valued GP prior over the explanation functions ϕ ∼ GP (0 , κ ) where κ : X × X → R d × d is the matrix-valued covariance kernel

where Ψ( x ) = [ E [ k ( · , X ) | X S 1 = x S 1 ] , . . . , E [ k ( · , X ) | X S 2 d = x S 2 d ] ] , and the ⊤ sign refers to taking inner products in the RKHS of k .

By utilizing the Shapley function prior, we can apply our approach to predict explanations for a wide range of models, including trees, deep neural networks, or RKHS functions, by treating their explanations as noisy samples from this prior and using them as regression labels. Our approach is based on the perspective that there exists a true data generating mechanism f ∗ : X → Y and that any model f we use is an approximation of f ∗ . Therefore, any explanations derived from f can be seen as an attempt to reveal the true explanations under f ∗ . This perspective has also been adopted in the work of Chen et al. [36] and Marx et al. [14]. We present the predictive posterior below.

Proposition 13 (Predictive explanations as multi-output GPs) . Given D ϕ = { ( x i , ϕ i ) } n i =1 = ( X , Φ X ) where ϕ i ∈ R d are the Shapley values computed under predictive model f and Φ X = [ ϕ 1 , ..., ϕ n ] ⊤ , the predictive explanations for new data x ′ is distributed as,

where ˜ m ϕ ( x ′ ) = b κ ( x ′ , X ) ⊤ vec(Φ X ) , b κ ( x ′ , X ) := ( κ XX + σ 2 ϕ I ) -1 κ ( X , x ′ ) , κ XX is the gram matrix for kernel κ of size nd × nd , κ ( x ′ , X ) = [ κ ( x ′ , x 1 ) , . . . , κ ( x ′ , x n )] is of size d × nd and σ 2 ϕ is the noise parameter for regression.

In fact, we see that the posterior mean ˜ m ϕ ( x ′ ) from (15) are Shapley values of the following payoff vector ˜ v x ′ , computed based on the observed explanations Φ X ,

Proposition 14 (Posterior mean as Shapley values for payoff vector ˜ v x ′ ) . The posterior mean ˜ m ϕ ( x ′ ) corresponds to Shapley values for the payoff vector ˜ v x ′ , i.e., ˜ m ϕ ( x ′ ) = A ˜ v x ′ , where ˜ v x ′ = ∑ n i =1 Ψ( x ′ ) ⊤ Ψ( x i ) A ⊤ α i and α i ∈ R d is the [ i, ..., i + ( d -1)] subvector of ( κ XX + σ 2 ϕ I ) -1 vec( Φ X ) .

5 Illustrations

We demonstrate the proposed approaches through three sets of illustrations. We first conduct an ablation study to examine how predictive and estimation uncertainty captured by our explanation algorithms vary under different configurations. We then move on to discuss various exploratory tools that can assist practitioners in utilizing the stochastic explanations. Finally, we demonstrate the

1

0

1

SSVs

25% data, 50% coalition used

25% data, 100% coalition used

MedInc

HouseAge

AveRooms

AveBedrms

Population

AveOccup

Latitude

Longitude

1

0

1

SSVs

100% data, 50% coalition used

MedInc

HouseAge

AveRooms

AveBedrms

Population

AveOccup

Latitude

Longitude

100% data, 100% coalition used

bayesSHAP

GPSHAP

BayesGPSHAP

effectiveness of the Shapley prior for the predictive explanations problem. Our code is included in the supplementary material, and we provide implementation details in the appendix. For all experiments, we pick the radial basis function (RBF) as our covariance kernel k for the GPs.

5.1 Ablation study on different notions of uncertainties captured

We conducted a comparison between GP-SHAP, BayesSHAP, and BayesGP-SHAP to demonstrate the differences between model predictive uncertainty and estimation uncertainty captured in the stochastic Shapley values. BayesSHAP is applied to the predictive posterior mean of GP to obtain its explanation. For this purpose, we used the California housing dataset [41] from the StatlLib repository, which includes 20640 instances and 8 numerical features, with the goal of predicting the median house value for California districts, expressed in hundreds of thousands of dollars. We trained our GP model using 25% and 100% of the data and calculated local stochastic explanations from GP-SHAP, BayesSHAP, and BayesGP-SHAP using 50% and 100% of coalitions. The results, shown in Figure 1, demonstrate that the magnitude of BayesSHAP uncertainties (green bars) are uniform across features as it is designed to capture the overall estimation performance and not feature-specific uncertainty. In contrast, GP-SHAP (red bars) exhibits varying uncertainties across features due to the propagation of variation from the posterior GP f to the attributions. This allows practitioners to make more granular statements about the uncertainty around specific feature explanations. We also observe that increasing the number of coalitions and training data generally leads to a decrease in both estimation and predictive uncertainties, but the estimation uncertainty drops more significantly. BayesGP-SHAP (yellow bars) consistently provides a more conservative uncertainty estimation by considering both predictive and estimation uncertainties.

5.2 Exploratory analysis of the stochastic explanations

We propose several exploratory analysis methods to aid practitioners in comprehending the stochastic explanations generated for their downstream tasks. To this end, we use BayesGP-SHAP to explain a Gaussian process model trained on the breast cancer dataset [42]. The dataset consists of 569 patients, 30 numeric features, and the objective is to predict whether a tumor is malignant or benign.

Local explanation. To visualize local explanations, we can plot the mean and standard deviations of stochastic Shapley values in a bar plot. This approach allows us to not only understand the degree of contribution a feature has to the prediction but also the corresponding credible interval as a measure of explanation uncertainty. Figure 2a displays that both 'worst fractal dimension' and 'worst perimeter' features have a similar contribution in terms of their absolute mean stochastic Shapley values. However, the model is much more uncertain about the former, allowing the user to calibrate their trust in model explanations.

1.0

0.8

0.6

0.4

0.2

0.0

0.2

0.4

0.6

stochastic Shapley values

radius error

worst symmetry

texture error

concavity error

fractal dimension error

compactness error

worst concave points

worst texture

worst perimeter

worst fractal dimension

Top 10 explanations from BayesGPSHAP for patient 1

(a) Local explanation

0.0

0.2

0.4

0.6

0.8

1.0

explanation score

concavity error

perimeter error

mean concave points

compactness error

worst perimeter

texture error

symmetry error

worst texture

worst concave points

worst fractal dimension

Top 10 global explanations from BayesGPSHAP

Mean of absolute SSVs

Std of absolute SSVs

Absolute mean SSVs

(b) Global explanation

(worst fractal dimension)

(worst perimeter)

(worst texture)

(worst concave points)

(compactness error)

(fractal dimension error)

(concavity error)

(texture error)

(worst symmetry)

(radius error)

Correlation between top 10 explanations for patient 1

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

(c) Local explanation correlation

(radius error)

(worst symmetry)

(texture error)

(concavity error)

(fractal dimension error)

(compactness error)

(worst concave points)

(worst texture)

(worst perimeter)

(worst fractal dimension)

Sparsity=0.9

Graphical model for top 10 explanations for patient 1

(d) Local explanation graphical model

Heuristic global explanation as means of absolute stochastic Shapley values. It is common to compute the mean absolute DSVs as a proxy to global feature contributions [43] based on computed local explanations. Although this heuristic does not result in any explanations that are themselves DSVs of any game at the global level, they provide a quick summary to practitioners about the overall contributions. We can similarly compute the average of the absolute stochastic Shapley values, which are now distributed according to a folded multivariate Gaussian distribution [44] with a tractable covariance structure. However, it is important to highlight a key distinction from the previous approach: when computing the mean of absolute SSVs , we consider the uncertainty in the local stochastic explanations, while directly calculating the absolute values of mean SSVs (equivalent to computing absolute DSVs) disregards this uncertainty. We believe the former approach is more appropriate than the latter as practitioners utilising Gaussian processes would want their global explanations to take the predictive uncertainty into account. The blue and yellow bars in Figure 2b depict these two approaches, respectively. Notably, we observe that the blue bars suggest that the "worst fractal dimension" feature is considered more influential than "worst concave points" because it accounts for the higher variability in the former feature's explanation. In contrast, the yellow bar overlooks this variability, leading to a different conclusion.

Correlations and dependencies across explanations. As we have access to the explanation covariance, we can explore their correlation and visualise them as in Figure 2c. For instance, the stochastic Shapley values of 'concavity error' are less correlated with other features. Moreover, as our explanations are multivariate Gaussians, we can build an undirected Gaussian graphical model [45] using the precision matrix (inverse of the covariance matrix) to study the independence across the local stochastic explanations. We visualise the corresponding graphical model for explanations from patient 1 in Figure 2d by setting a sparsity threshold of 90% , following the approach in [46].

5.3 Predictive explanations

Finally, we showcase the effectiveness of our Shapley prior by comparing the regression performances of a multi-output GP using the Shapley prior with multi-output random forest and neural networks when predicting withheld explanations generated from GP-SHAP, TreeSHAP, and DeepSHAP, respectively. We use the diabetes dataset [47] from the UCI repository, which contains 442 patients with 10 numerical features and the goal is to predict disease progression. We first train a GP, a random forest, and a neural network, to obtain the

subsequent explanations from GP-SHAP, TreeSHAP, and DeepSHAP. Next, we feed 70% of the explanations to our predictive model as training data and the remaining 30% as test data.

Werepeat this process over 10 seeds and compute the root mean squared error between the predicted explanation and the exact explanations for respective models, as shown in Figure 3. We observe that our GP-model with the Shapley prior consistently outperforms random forest and neural network model for predicting all three types of explanations. This demonstrates that the inductive bias that comes from our choice of covariance structure κ allows to build more accurate predictive explanation models. We also observe that the overall average prediction error for explanations generated from GP-SHAP is lower than that of TreeSHAP [11] and DeepSHAP [2], suggesting that GP-SHAP produce explanations that are easier to learn. This follows our intuition as GPs are typically smoother functions than trees and neural networks.

6 Discussion

In this work, we presented a novel and principled approach for explaining Gaussian process models using stochastic Shapley values. The proposed algorithm GP-SHAP, and its variant BayesGP-SHAP, allow practitioners to consider both predictive and estimation uncertainties when reasoning about the explanations and calibrate their trust in the model accordingly. Furthermore, we consider the setting of predictive explanations, where we introduced a Shapley prior over explanation functions, enabling us to model and predict Shapley values for a wider range of predictive models. We demonstrated the effectiveness of our methods through a number of illustrations and discussed various exploratory tools for practitioners to analyse the stochastic explanations.

Limitations and future outlook. While the general framework of using stochastic Shapley values for stochastic explanations can be applied beyond Gaussian process models, the specific estimation algorithms presented in this paper are tailored to GPs and may not be directly applicable to other probabilistic models like Bayesian neural networks. These alternative models would require different estimation procedures as the resulting explanations would not be GPs anymore. Another promising avenue for future research involves leveraging the uncertainty obtained from predicting explanations and incorporating it into Bayesian optimization for guiding experimental design. For example, this could allow practitioners to explore data regions that produce significant Shapley values for specific features. At last, we would like to highlight that although SHAP is popular, as an explanation tool it contains several limitations, specifically Kumar et al. [48] have shown that SHAP violates several expected properties of explanations from a human-centric perspective.

References

Lloyd S Shapley. A value for n-person games. Contributions to the Theory of Games , 2(28): 307-317, 1953.

Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in neural information processing systems , pages 4765-4774, 2017.

Philippe Bracke, Anupam Datta, Carsten Jung, and Shayak Sen. Machine learning explainability in finance: an application to default risk analysis. Bank of England Working Paper , 2019.

Konstantin D Pandl, Fabian Feiland, Scott Thiebes, and Ali Sunyaev. Trustworthy machine learning for health care: scalable data valuation with the shapley value. In Proceedings of the Conference on Health, Inference, and Learning , pages 47-57, 2021.

Sindre Benjamin Remman, Inga Strümke, and Anastasios M Lekkas. Causal versus marginal shapley values for robotic lever manipulation controlled using deep reinforcement learning. In 2022 American Control Conference (ACC) , pages 2683-2690. IEEE, 2022.

Umang Bhatt, Javier Antorán, Yunfeng Zhang, Q Vera Liao, Prasanna Sattigeri, Riccardo Fogliato, Gabrielle Melançon, Ranganath Krishnan, Jason Stanley, Omesh Tickoo, et al. Uncertainty as a form of transparency: Measuring, communicating, and using uncertainty. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , pages 401-413, 2021.

Eyke Hüllermeier and Willem Waegeman. Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods. Machine Learning , 110(3):457-506, March 2021. ISSN 0885-6125, 1573-0565. doi: 10.1007/s10994-021-05946-3. URL http: //arxiv.org/abs/1910.09457 . arXiv:1910.09457 [cs, stat].

Siu Lun Chau, Jean-Francois Ton, Javier González, Yee Teh, and Dino Sejdinovic. BayesIMP: Uncertainty Quantification for Causal Data Fusion. In Advances in Neural Information Processing Systems , volume 34, pages 3466-3477. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/hash/ 1ca5c750a30312d1919ae6a4d636dcc4-Abstract.html .

Carl Edward Rasmussen, Christopher KI Williams, et al. Gaussian processes for machine learning , volume 1. Springer, 2006.

Erik Štrumbelj and Igor Kononenko. Explaining prediction models and individual predictions with feature contributions. Knowledge and information systems , 41(3):647-665, 2014.

Scott M Lundberg, Gabriel G Erion, and Su-In Lee. Consistent individualized feature attribution for tree ensembles. arXiv preprint arXiv:1802.03888 , 2018.

Siu Lun Chau, Robert Hu, Javier Gonzalez, and Dino Sejdinovic. RKHS-SHAP: Shapley values for kernel methods. Advances in Neural Information Processing Systems , 35:13050-13063, 2022.

Yuya Yoshikawa and Tomoharu Iwata. Gaussian Process Regression with Local Explanation. IEEE Transactions on Neural Networks and Learning Systems , pages 1-15, 2021. ISSN 2162237X, 2162-2388. doi: 10.1109/TNNLS.2021.3131234. URL http://arxiv.org/abs/ 2007.01669 . arXiv:2007.01669 [cs, stat].

Charles Marx, Youngsuk Park, Hilaf Hasson, Yuyang Wang, Stefano Ermon, and Luke Huan. But are you sure? an uncertainty-aware perspective on explainable ai. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , pages 7375-7391. PMLR, 25-27 Apr 2023. URL https://proceedings. mlr.press/v206/marx23a.html .

Vern I Paulsen and Mrinal Raghupathi. An introduction to the theory of reproducing kernel Hilbert spaces , volume 152. Cambridge university press, 2016.

Ying Ma, Zuofeng Gao, Wei Li, Ning Jiang, Lei Guo, et al. The shapley value for stochastic cooperative game. Modern Applied Science , 2(4):1-76, 2008.

Jinwu Gao, Xiangfeng Yang, and Di Liu. Uncertain Shapley value of coalitional game with application to supply chain alliance. Applied Soft Computing , 56:551-556, July 2017. ISSN 15684946. doi: 10.1016/j.asoc.2016.06.018. URL https://linkinghub.elsevier.com/ retrieve/pii/S1568494616302988 .

Siu Lun Chau, Shahine Bouabid, and Dino Sejdinovic. Deconditional Downscaling with Gaussian Processes. In Advances in Neural Information Processing Systems , volume 34, pages 17813-17825. Curran Associates, Inc., 2021. URL https://papers.nips.cc/paper_ files/paper/2021/hash/94aef38441efa3380a3bed3faf1f9d5d-Abstract.html .

A Charnes, B Golany, M Keane, and J Rousseau. Extremal principle solutions of games in characteristic function form: core, chebychev and shapley value generalizations. Econometrics of planning and efficiency , pages 123-133, 1988.

Dylan Slack, Sophie Hilgard, Sameer Singh, and Himabindu Lakkaraju. Reliable Post hoc Explanations: Modeling Uncertainty in Explainability, November 2021. URL http://arxiv. org/abs/2008.05030 . arXiv:2008.05030 [cs, stat].

Boyang Dai, Xiangfeng Yang, and Xiaoyue Liu. Shapley Value of Uncertain Coalitional Game based on Hurwicz Criterion with Application to Water Resource Allocation. Group Decision and Negotiation , 31(1):241-260, February 2022. ISSN 0926-2644, 15729907. doi: 10.1007/s10726-021-09767-6. URL https://link.springer.com/10.1007/ s10726-021-09767-6 .

Ian Covert and Su-In Lee. Improving KernelSHAP: Practical Shapley Value Estimation Using Linear Regression. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , pages 3457-3465. PMLR, March 2021. URL https: //proceedings.mlr.press/v130/covert21a.html . ISSN: 2640-3498.

Riccardo Colini-Baldeschi, Marco Scarsini, and Stefano Vaccari. Variance Allocation and Shapley Value. Methodology and Computing in Applied Probability , 20(3):919-933, September 2018. ISSN 1387-5841, 1573-7713. doi: 10.1007/s11009-016-9540-5. URL http://arxiv. org/abs/1606.09424 . arXiv:1606.09424 [cs, math, stat].

Marcello Galeotti and Giovanni Rabitti. On the comparison of Shapley values for variance and standard deviation games. Journal of Applied Probability , 58(3):609-620, September 2021. ISSN 0021-9002, 1475-6072. doi: 10.1017/jpr.2020.106. URL https://www.cambridge. org/core/product/identifier/S0021900220001060/type/journal_article .

Art B Owen. Sobol'indices and shapley value. SIAM/ASA Journal on Uncertainty Quantification , 2(1):245-251, 2014.

Daniel Fryer, Inga Strümke, and Hien Nguyen. Shapley value confidence intervals for attributing variance explained. Frontiers in Applied Mathematics and Statistics , 6:587199, 2020.

Rodica Branzei, Oana Branzei, S Zeynep Alparslan Gök, and Stef Tijs. Cooperative interval games: a survey. Central European Journal of Operations Research , 18:397-411, 2010.

SZ Alparslan Gök, Rodica Branzei, and Stef Tijs. The interval shapley value: an axiomatization. Central European Journal of Operations Research , 18:131-140, 2010.

Davide Napolitano, Lorenzo Vaiani, and Luca Cagliero. Learning Confidence Intervals for Feature Importance: A Fast Shapley-based Approach. Workshop Proceedings of the EDBT/ICDT 2023 Joint Conference (March 28-March 31, 2023, Ioannina, Greece) .

Ian Covert, Scott Lundberg, and Su-In Lee. Explaining by removing: A unified framework for model explanation. Journal of Machine Learning Research , 22(209):1-90, 2021.

Robert Hu, Siu Lun Chau, Jaime Ferrando Huertas, and Dino Sejdinovic. Explaining preferences with shapley values. Advances in Neural Information Processing Systems , 35:27664-27677, 2022.

Sahra Ghalebikesabi, Lucile Ter-Minassian, Karla DiazOrdaz, and Chris C Holmes. On locality of local explanation models. Advances in Neural Information Processing Systems , 34, 2021.

Masayoshi Mase, Art B Owen, and Benjamin B Seiler. Cohort shapley value for algorithmic fairness. arXiv preprint arXiv:2105.07168 , 2021.

Christopher Frye, Ilya Feige, and Colin Rowat. Asymmetric shapley values: incorporating causal knowledge into model-agnostic explainability. arXiv preprint arXiv:1910.06358 , 2019.

Tom Heskes, Evi Sijben, Ioan Gabriel Bucur, and Tom Claassen. Causal shapley values: Exploiting causal knowledge to explain individual predictions of complex models. Advances in neural information processing systems , 33:4778-4789, 2020.

Hugh Chen, Joseph D Janizek, Scott Lundberg, and Su-In Lee. True to the model or true to the data? arXiv preprint arXiv:2006.16234 , 2020.

Christopher Frye, Damien de Mijolla, Laurence Cowton, Megan Stanley, and Ilya Feige. Shapley-based explainability on the data manifold. arXiv preprint arXiv:2006.01272 , 2020.

Krikamol Muandet, Kenji Fukumizu, Bharath Sriperumbudur, Bernhard Schölkopf, et al. Kernel mean embedding of distributions: A review and beyond. Foundations and Trends® in Machine Learning , 10(1-2):1-141, 2017.

Neil Jethani, Mukund Sudarshan, Ian Covert, Su-In Lee, and Rajesh Ranganath. FastSHAP: Real-Time Shapley Value Estimation, March 2022. URL http://arxiv.org/abs/2107. 07436 . arXiv:2107.07436 [cs, stat].

Davin Hill, Aria Masoomi, Sandesh Ghimire, Max Torop, and Jennifer Dy. Boundary-Aware Uncertainty for Feature Attribution Explainers, February 2023. URL http://arxiv.org/ abs/2210.02419 . arXiv:2210.02419 [cs].

R Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters , 33(3):291-297, 1997.

WNick Street, William H Wolberg, and Olvi L Mangasarian. Nuclear feature extraction for breast tumor diagnosis. In Biomedical image processing and biomedical visualization , volume 1905, pages 861-870. SPIE, 1993.

Ian Covert, Scott Lundberg, and Su-In Lee. Understanding global feature contributions with additive importance measures. Advances in Neural Information Processing Systems , 33, 2020.

Raymond Kan and Cesare Robotti. On moments of folded and truncated multivariate normal distributions. Journal of Computational and Graphical Statistics , 26(4):930-934, 2017.

David Edwards. Introduction to graphical modelling . Springer Science & Business Media, 2012.

Salar Fattahi and Somayeh Sojoudi. Graphical lasso and thresholding: Equivalence and closed-form solutions. Journal of Machine Learning Research , 20(10):1-44, 2019. URL http://jmlr.org/papers/v20/17-501.html .

Bradley Efron, Trevor Hastie, Iain Johnstone, and Robert Tibshirani. Least Angle Regression. The Annals of Statistics , 32(2), April 2004. ISSN 0090-5364. doi: 10.1214/009053604000000067. URL http://arxiv.org/abs/math/0406456 . arXiv:math/0406456.

I Elizabeth Kumar, Suresh Venkatasubramanian, Carlos Scheidegger, and Sorelle Friedler. Problems with shapley-value-based explanations as feature importance measures. In International Conference on Machine Learning , pages 5491-5500. PMLR, 2020.

Michalis Titsias and Neil D Lawrence. Bayesian gaussian process latent variable model. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 844-851. JMLR Workshop and Conference Proceedings, 2010.

Le Song, Kenji Fukumizu, and Arthur Gretton. Kernel embeddings of conditional distributions: A unified kernel framework for nonparametric inference in graphical models. IEEE Signal Processing Magazine , 30(4):98-111, 2013.

Mauricio A. Alvarez, Lorenzo Rosasco, and Neil D. Lawrence. Kernels for Vector-Valued Functions: a Review, April 2012. URL http://arxiv.org/abs/1106.6251 . arXiv:1106.6251 [cs, math, stat].

Bernhard Schölkopf, Ralf Herbrich, and Alex J Smola. A generalized representer theorem. In Computational Learning Theory: 14th Annual Conference on Computational Learning Theory, COLT 2001 and 5th European Conference on Computational Learning Theory, EuroCOLT 2001 Amsterdam, The Netherlands, July 16-19, 2001 Proceedings 14 , pages 416-426. Springer, 2001.

SUPPLEMENTARY MATERIAL: Explaining the uncertain: Stochastic Shapley values for Gaussian process models

A The GP-SHAP algorithm and discussion on computation techniques

We present the complete algorithm for both GP-SHAP and BayesGP-SHAP in Algorithm 1.

Algorithm 1 GP-SHAP / BayesGP-SHAP

Input: Posterior mean function ˜ m , posterior covariance function ˜ k , inducing locations ˜ X , explanation instances X , number of coalition samples n Z , hyperparameter λ, n 0 , σ 2 0 , base kernel k , algorithm algo ,

1: Compute n I = number of inducing location, n = number of explanation instances, d = number of features.

3: Sample coalitions S = { S 1 , ..., S n Z } from [ d ] , build binary matrix Z = { 0 , 1 } n Z × d from S , and compute weights W = diag[ w 1 , ..., w n Z ] with w i = d -1 ( d | | ) | S i | ( d -| S i | ) .

4: Compute A = ( Z ⊤ W Z ) -1 Z ⊤ W

▷

Shape:

d

×

n

Z

5: Compute B ( X , S ) = [( K ˜ X S ˜ X S + λI ) -1 k S ( ˜ X S , X S ) for S in S ] ▷ Shape: n Z × n I × n

6: Compute

L

j,l

▷

Shape:

n

Z

×

n

×

n

I

Q

where

Q

i,l,k

=

∑

j

B

(

X

,

S

)

i,j,k

7: Compute R where R i,k,l = ∑ j A i,j Q j,k,l ▷ Shape: d × n × n I

8: Compute

9: Compute m,l,n

R

i,j,k

R

S

˜

(

)

j

Shape:

▷

d

×

d

×

Shape:

n

n

Z

×

×

n

n

V

E

where where

V

E

i,m,k,n i,k

=

=

∑

j

∑

B

(

j,l

X

,

)

i,j,k

m

10: Compute Φ = AE ▷ The mean stochastic Shapley values of shape d × n 11: if algo = GP-SHAP then

12: return mean explanations Φ and covariance V between d features and n instances

13: else if algo = BayesGP-SHAP then

14:

Compute

s

2

=

diag

(

(

E

-

Z

Φ)

⊤

W

(

E

-

Z

Φ)

)

+diag(Φ

⊤

Φ)

▷

Shape:

n

×

1

15: Sample σ 2 from Scaled-Invχ 2 ( n 0 + n Z , n 0 σ 2 0 + n Z s 2 n 0 + n Z ) ▷ Shape: n × 1

16: return mean explanations Φ and covariance V +( Z ⊤ WZ ) -1 σ 2 17: end if

Computational considerations. In terms of computational complexity, one of the most demanding operations in the algorithm is the computation of conditional mean embeddings in step 5 . Instead of naively inverting an n × n matrix, which would have a computational cost of O ( n 3 ) , we employ the conjugate gradient method to reduce the computation of the conditional mean embedding component to O ( n 2 a ) , where a ≪ n represents the number of conjugate gradient iterations. Additionally, to further reduce runtime, we utilize the variational sparse GP model [49]. This model learns a set of inducing locations ˜ X with a size of n I ≪ n , which can be reused for the estimation of conditional mean embeddings in the algorithm. Consequently, the computation of the conditional expectation is reduced from O ( n 2 a ) to O ( n 2 I a ) . Another computational burden arises from the computation of the full covariance matrix across d features and n instances, which requires storage of a n 2 d 2 matrix. However, since the full covariance matrix can be factorized into the R component from step 7 of the algorithm, we can store this low-rank component and compute covariances between specific instances when necessary. It is worth noting that this decomposition of the covariance matrix allows us to avoid redundant computations when computing the covariance component, as we no longer need to iterate over all possible coalitions twice. Finally, we can further speed up our computational by parallelising computation across the sub-sampled coalitions in step 5 .

▷

2: Compute Cholesky decomposition on posterior covariance LL ⊤ = ˜ K ˜ X ˜ X

S i

B Proofs and derivations

B.1 Section 2 proofs: Stochastic Shapley values

We include the full proof of the derivation of stochastic Shapley values for completeness. The proof is analogous to the original work of Shapley's [1] but extended to random variable payoffs. Ma et al. [16] has also proved the same theorem but used a different proving strategy. They started with the solution and showed it satisfies the axioms and then prove uniqueness, whereas the following proof starts from the characterisation of s-games and derive the solution from a bottom-up fashion.

To facilitate the proof, we first introduce the concept of stochastic symmetric game.

Proposition 15 (s-symmetric games) . Let C be a real-valued random variables, then the symmetric game ν C,R ( S ) := C 1 [ R ⊆ S ] gets a stochastic shapley value as,

where r = | R | .

Proof. Take any i, j ∈ R , pick a permutation π ∈ Π( U ) so that πR = R and πi = j , so the induced game πν C,R = ν C,R , and therefore by the s-symmetry axiom,

Now by the s-efficiency axiom,

for any i ∈ R .

Now we can characterise the form of any stochastic game as follows:

Proposition 16. All s-games with finite carrier can be written as a linear combination of s-symmetric games,

̸

̸

holds for all S ⊆ U , and for any finite carrier N of ν . If S ⊆ N , then we can rewrite the expression as,

where in the last equation we used the fact that ∑ s r = t ( -1) r -t ( s -t r -t ) is a binomal expansion of (1 + ( -1)) s -t , therefore the only non-zero expression is when t = s .

where

Proof. We start by verifying

We can now prove the uniqueness of stochastic Shapley values,

Theorem 4 (Stochastic Shapley values) . The only stochastic value allocation ϕ of ν satisfying s-symmetry, s-efficiency, and s-linearity takes the following form,

where N is the smallest carrier set of Ω , c | S | = 1 | N | ( | N |-1 | S | ) -1 and ϕ i ( ν ) is the i th SSV of s-game ν .

Proof. First, let us denote

Applying the s-linearity axiom on ϕ to the characterisation of ν from the previous propositions leads us to the following,

̸

̸

where in (32) we used the following observation: given i / ∈ S ′ ⊆ N , and S = S ′ ∪ { i } , then γ i ( S ) = -γ i ( S ′ ) .

It satisfies uniqueness by construction.

Proposition 5. Given the player set Ω , let ν be a stochastic game, ϕ a stochastic Shapley value allocation, and ¯ ϕ a deterministic Shapley value allocation. Suppose that E [ ν ] and V [ ν ] are the corresponding mean and variance d-games, respectively. Then, E [ ϕ ( ν )] = ¯ ϕ ( E [ ν ]) , but V [ ϕ ( ν )] = ¯ ϕ ( V [ ν ]) . In particular, the SSV variance is given by

̸

where ν S = ν ( S ) and C is the covariance function between the stochastic payoffs.

Proof. The equivalence between mean of stochastic Shapley values and deterministic Shapley values of mean game is trivial to show leveraging the linearity of expectation. The variance of V [ ϕ i ( ν )] can be shown by repeatedly applying the standard identity V [ X + Y ] = V [ X ] + V [ Y ] + 2 C [ X,Y ] for random variables X,Y . Now consider the deterministic Shapley values of variance game V [ ν ] ,

Comparing to the expression of V [ ϕ i ( ν )] from the lemma,

even if we assume mutual independence across all payoff random variables, leading to C [ ν ( S ∪ i ) , ν ( S )] = 0 for all S , we still would not subtract but instead sum the variance of V [ ν ( S ∪ i )] and V [ ν ( S )] . Therefore the variances of stochastic Shapley values is not the same as the deterministic Shapley values of the variance game.

B.2 Section 3.1 proofs on the stochastic Shapley values for induced stochastic game from GP

Proposition 6 (Stochastic game ν f as induced GP) . Let f ∼ GP ( ˜ m, ˜ k ) with integrable sample paths, i.e. ∫ X | f | dp X < ∞ almost surely. The stochastic payoff function ν f induced by f is a Gaussian process with the following mean and covariance functions:

Proof. This is a direct application of Chau et al. [18, Proposition 3.2] to the distribution P ( X | X S = x S ) .

Theorem 7 (Stochastic Shapley values of ν f ) . Let ν f be an induced stochastic game from the GP f ∼ GP ( ˜ m, ˜ k ) and denote v x := [ ν f ( x , S 1 ) , . . . ν f ( x , S 2 d )] ⊤ the vector of stochastic payoffs across all coalitions, then the corresponding stochastic Shapley values ϕ ( ν f ( x , · )) follows a d -dimensional multivariate Gaussian distribution,

where E [ v x ] ∈ R 2 d and V [ v x ] ∈ R 2 d × 2 d are the corresponding mean vector and covariance matrix of the payoffs.

Proof. Recall from Lundberg and Lee [2, Theorem 2], for deterministic Shapley values, given a deterministic payoff ¯ v x for all 2 d coalitions, the expression of Shapley values for each i ∈ [ d ] ,

can be written compactly as the following vector,

We can therefore similarly write down the form of the stochastic Shapley values using this linear operator A , acting now on a vector of random variable output stochastic payoff vector v x ,

Nonetheless, as Proposition 8 implies that v x is a multivariate Gaussian, therefore ϕ x is also multivariate Gaussian with mean and covariance the following,

B.3 Section 3.2 proofs on estimation

To proceed, we first introduce the concepts of conditional mean embedding as a tool to estimate conditional expectation of functions living in their corresponding RKHSs,

Definition 17 (Conditional mean embedding [38]) . Let X,Y be random variables and k : X → X → R a kernel on X , then we define the following as the conditional mean embedding of p ( X | Y = y ) ,

Proposition 18 (Conditional Mean estimation) . For random variable X,Y , and a kernel k : X → X → R on X and a kernel l : Y → Y → R on Y . Given observations D = { X , y } , the empirical conditional mean embedding can be estimated as

where l ( y, y ) = [ l ( y, y 1 ) , . . . , l ( y, y n )] ⊤ and k ( · , X ) = [ k ( · , x 1 ) , . . . , k ( · , x n )] ⊤ , the parameter λ > 0 is there to stablise the inversion. Now for f ∈ H k , the conditional expectation can then be estimated as,

where f = [ f ( x 1 ) , . . . , f ( x n )] ⊤ .

Proof. This is standard result from literature, please read Song et al. [50], Muandet et al. [38] for more details.

Now we can apply these propositions to estimate the mean and covariance functions of the induced stochastic game from GP,

Proposition 8 (Estimating ν f ) . Given D = ( X , y ) and the posterior GP f | D ∼ GP ( ˜ m, ˜ k ) , the mean and covariance function of the stochastic cooperative game ν f can be estimated as,

where b ( x , S ) := ( K X S X S + λI ) -1 k S ( X S , x S ) , ˜ m ( X ) = [ ˜ m ( x 1 ) , . . . , ˜ m ( x n )] ⊤ , and k S : X S ×X S → R is the kernel defined on the sub-feature space of X and we write k S ( x S , X S ) := [ k S ( x S , x 1 S ) , ..., k S ( x S , x nS )] and K XX and ˜ K XX as the gram matrix of X using kernel k and ˜ k respectively. The parameter λ > 0 is a fixed hyperparameter to stabilise the inversion.

Proof. Without loss of generality, we will demonstrate this proposition with ˜ m, ˜ k obtained via standard GP regression, i.e.,

Starting with the mean function,

We can replace the population conditional mean embedding with the empirical version, and expand,

Analogously, the conditional expectation of the posterior covariance function, i.e., E [ ˜ k ( X,X ′ ) | X S = x S , X ′ S = x ′ S ] , can be estimated following the steps above,

After replacing the population conditional mean embedding as their empirical estimates, we can arrive at the solution.

Proposition 9 ( GP-SHAP ) . Let the matrix A be defined as in Theorem 7. The mean and covariance for the multivariate stochastic Shapley values can be estimated as,

Proof. The result follows directly from the previous proposition. Recall ϕ (ˆ ν f ( x , · )) = A ˆ v x for ˆ v x the vector of stochastic payoffs for each coalition. To estimate the mean, we

Recall V [ v x ] i,j = ˆ k ν (( x , S i ) , ( x , S j )) = b ( x , S i ) ⊤ ˜ K XX b ( x , S j ) , the derivation for the covariance matrix then follows analogously as the derivation for the mean,

Proposition 10 (BayesSHAP [20]) . Given the data generation above, the posterior distribution on ¯ ϕ and σ 2 follows:

where ℓ is the number of coalitions S = { S j } ℓ j =1 we sample uniformly from 2 [ d ] , Z ℓ is the binary matrix representing S , and W ℓ is the corresponding weight matrix, and A ℓ = ( Z ⊤ ℓ W ℓ Z ℓ ) -1 Z ⊤ ℓ W ℓ is the WLS matrix, ¯ v x = [¯ ν f ( x , S 1 ) , ..., ¯ ν f ( x , S ℓ )] ⊤ is the vector of deterministic payoffs, and

measures the average weighted error in the regression and the norm of the mean explanations.

Proof.

Proposition 11 ( BayesGP-SHAP ) . Continuing from Propositions 9 and 10, the posterior distribution of the stochastic Shapley values can be estimated using the Bayesian WLS approach as,

Proof. We drop the bar notation of ¯ ϕ to unify notations. Given the posterior GP f | D ∼ GP ( ˜ m, ˜ k )

Using a standard Gaussian conjugacy procedure, we can derive the variance as the sum of variances from GP-SHAP and BayesSHAP. While it is possible to integrate p ( σ 2 | Z ℓ , f, x , D ) with respect to the posterior, this leads to a complex scaled mixture of normals that is difficult to model. Instead, we construct a scaled inverse chi-square distribution with s 2 ( E [ v x ]) , which represents the error of the weighted regression with respect to the mean payoffs E [ v x ] . We sample σ 2 from the following distribution:

B.4 Proofs for section 4 on predictive explanation and Shapley prior

Proposition 12 (The Shapley prior over ϕ ) . The prior f ∼ GP (0 , k ) and the corresponding stochastic game ν f ( x , S ) = E [ f ( X ) | X S = x S ] induce a vector-valued GP prior over the explanation functions ϕ ∼ GP (0 , κ ) where κ : X × X → R d × d is the matrix-valued covariance kernel

where Ψ( x ) = [ E [ k ( · , X ) | X S 1 = x S 1 ] , . . . , E [ k ( · , X ) | X S 2 d = x S 2 d ] ] , and the ⊤ sign refers to taking inner products in the RKHS of k .

Proof. The proof is similar to how we proved previous propositions but applied to prior GP f ∼ GP (0 , k ) instead. If we set,

then ν f is a GP on the joint space of data and coalitions with mean 0 , and covariance function,

Since ϕ = Av x for v x the vector of stochastic payoff from the game induced by the GP prior, the mean stays 0 , and the covariance is,

therefore we have a matrix-valued covariance kernel κ to build a prior over the induced Shapley values.

Proposition 13 (Predictive explanations as multi-output GPs) . Given D ϕ = { ( x i , ϕ i ) } n i =1 = ( X , Φ X ) where ϕ i ∈ R d are the Shapley values computed under predictive model f and Φ X = [ ϕ 1 , ..., ϕ n ] ⊤ , the predictive explanations for new data x ′ is distributed as,

where ˜ m ϕ ( x ′ ) = b κ ( x ′ , X ) ⊤ vec(Φ X ) , b κ ( x ′ , X ) := ( κ XX + σ 2 ϕ I ) -1 κ ( X , x ′ ) , κ XX is the gram matrix for kernel κ of size nd × nd , κ ( x ′ , X ) = [ κ ( x ′ , x 1 ) , . . . , κ ( x ′ , x n )] is of size d × nd and σ 2 ϕ is the noise parameter for regression.

Proof. Follows from standard vector-valued Gaussian process regression results. See Alvarez et al. [51] for a detailed discussion on regression with matrix-valued kernels.

Proposition 14 (Posterior mean as Shapley values for payoff vector ˜ v x ′ ) . The posterior mean ˜ m ϕ ( x ′ ) corresponds to Shapley values for the payoff vector ˜ v x ′ , i.e., ˜ m ϕ ( x ′ ) = A ˜ v x ′ , where ˜ v x ′ = ∑ n i =1 Ψ( x ′ ) ⊤ Ψ( x i ) A ⊤ α i and α i ∈ R d is the [ i, ..., i + ( d -1)] subvector of ( κ XX + σ 2 ϕ I ) -1 vec( Φ X ) .

Proof. There are two ways to see this. First is by brute force and rearranging the terms in the posterior mean expression. The other is to leverage the vector-valued representer theorem [52] and write the posterior mean as,

after some linear algebra exercises, we can see that α i is the [ i : i + ( d -1)] sub-vector of ( κ XX + σ 2 ϕ I ) -1 vec( Φ X )

C Implementation details and further illustrations.

All illustrations are run locally on a MacbookPro 2021 with Apple M1 pro chip.

C.1 Ablation study on different notions of uncertainties captured

To demonstrate the difference between the uncertainties captured by GP-SHAP, BayesSHAP, and BayesGP-SHAP, we utilise the California housing dataset [41]. This dataset was derived from the 1990 U.S. census, each observation represent a census block group. A block group is the smallest geographical unit for which the U.S. Census Bureau publishes sample data (a block group typically has a population of 600 to 3,000 people). The dataset includes 20640 instances with 8 numerical features measuring the following:

MedInc: Median income in block group

HouseAge: Median house age in block group

AveRooms: Average number of rooms per household

AveBedrms: Average number of bedrooms per household

Population: Block group population

AveOccup: Average number of houehold members

Latitude: Block group latitude

Longitude: Block group longitude

The target variable is the median house value for California districts, expressed in hundreds of thousands of dollars. In the following, we train a GP model and extract explanations using GP-SHAP, BayesSHAP, and BayesGP-SHAP, for 4 different configurations:

trained on 25% of data, estimate the Shapley values using 50% of coalitions.

trained on 25% of data, estimate the Shapley values using 100% of coalitions.

trained on 100% of data, estimate the Shapley values using 50% of coalitions.

trained on 100% of data, estimate the Shapley values using 100% of coalitions.

To fit the GP model, we employ a sparse Variational GP approach with 200 learnable inducing point locations. The evidence lower bound is optimized using batch gradient descent with a batch size of 64 , a learning rate of 0 . 01 , and 100 iterations. The RBF kernel with learnable bandwidths initialized using the median heuristic approach is used for the sparse GP. The inducing locations are initialized using a standard clustering approach to obtain a representative set of inducing points.

After training the model, we reuse the learned inducing points and kernel bandwidths for the explanation algorithms. The explanations are obtained using the procedure described in Algorithm 1 of our work.

2

1

0

1

2

Mean stochastic Shapley values

AveBedrms

AveRooms

HouseAge

Population

Longitude

AveOccup

Latitude

MedInc

Low

High

Feature value

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Standard deviation of stochastic Shapley values

AveRooms

AveBedrms

HouseAge

Longitude

Latitude

MedInc

AveOccup

Population

Low

High

Feature value

(a) Mean of each stochastic explanations

(b) Standard deviation of each explanations

In Figure 1 of our paper, we present the stochastic Shapley values for the 11th observation, computed using the three explanation algorithms. The plot includes the 95% credible interval to visualize the uncertainties associated with the explanations.

Further illustration: In Figure 4, we plot the beeswarm plot on the mean and standard deviation of each stochastic explanations respectively. We color the point based on the relative size of the feature value compared to the rest. We see that in Figure 4a, which plotted the mean stochastic shapley values for each observation, the relationship between most features' explanation to the target variable is quite linear. For example, the higher the median income ( MedInc ), the more positive those feature contribute to predicting the respective median house value. On the other hand, Figure 4b illustrated the standard deviation of each stochastic explanations. In general, we see that the larger the feature values are, the more uncertain the explanation becomes. Nonetheless, we see that the feature contributing the most, defined as the feature having largest distance spanned by their most positive and most negative mean stochastic Shapley values, does not necessarily have the largest variation respectively.

C.2 Exploratory analysis of the stochastic explanations

For this illustration, we utilise the breast cancer dataset [42], containing 569 patients with 30 numeric features. They are computed from a digitized image of a fine needle aspirate (FNA) of a breast mass and describe characteristics of the cell nuclei present in the image:

radius (mean of distances from center to points on the perimeter)

texture (standard deviation of gray-scale values)

perimeter

area

smoothness (local variation in radius lengths)

compactness ( perimeter 2 area -1 )

concavity (severity of concave portions of the contour)

concave points (number of concave portions of the contour)

symmetry

fractal dimension ('coastline approximation' - 1)

The goal is to predict whether a tumour is malignant or benign. We first fit a GP model with RBF kernel using again the sparse Variational GP formulation with 200 learnable inducing locations. We initialise the inducing points using standard clustering techniques on the data. The evidence lower bound objective is optimised with a learning rate of 1 e -4 and 1000 iterations using batch gradient descent of batch size 64 . To obtain the explanations, we run the BayesGP-SHAP algorithm with 2 16

1.0

0.5

0.0

0.5

1.0

1.5

Mean of stochastic Shapley values

mean texture

radius error

concave points error

mean compactness

mean perimeter

worst radius

mean fractal dimension

perimeter error

worst symmetry

mean concavity

fractal dimension error

symmetry error

mean concave points

concavity error

compactness error

texture error

worst perimeter

worst texture

worst fractal dimension

worst concave points

Low

High

Feature value

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Standard deviation of stochastic Shapley values

mean texture

worst perimeter

mean concave points

worst symmetry

perimeter error

mean fractal dimension

mean area

compactness error

mean symmetry

concavity error

worst compactness

mean perimeter

mean concavity

fractal dimension error

concave points error

worst texture

worst concave points

texture error

symmetry error

worst fractal dimension

Low

High

Feature value

(a) Mean of each stochastic explanations

(b) Standard deviation of each explanations

number of coalitions. We do not compare GP-SHAP and BayesSHAP here because the BayesSHAP uncertainties have shrunk to almost zero, i.e., the mean standard deviations from the BayesSHAP uncertainties across all features and data is 0 . 0002 . This reconfirms the fact from Slack et al. [20] that as we increase the sample size the estimation error goes to zero, thus the uncertainties from BayesSHAP goes to zero as well. On the other hand, GP-SHAP uncertainties still remain valid because it represents the GP predictive uncertainties, which do not shrink to zero as we increase the number of coalitions we use to esitmate the SVs.

Further illustrations: In Figure 5, we plot two violin plots to illustrate the relationship between mean and standard deviation of the stochastic values with respect to the size of the original feature. We see that the feature 'worst fractal dimension' are the second most influential feature in terms of mean stochastic explanations and also the feature that has highest uncertainty around its explanations. In comparison with the housing prediction problem illustrated in Figure 4, the higher the feature value doesn't necessary give higher uncertainty around its explanation.

C.3 Predictive explanations

For this illustration, we utilise the Diabetes dataset [47] with 442 patient data and 10 numeric features measuring the following:

age: age in years

sex

bmi: body mass index

bp: average blood presuure

•

s1: total serum cholesterol

s2: low-density lipoproteins

s3: high-density lipoproteins

s4: total cholesterol

s5: Log of serum triglycerides level

s6: blood sugar level

The experiment is to assess the effectiveness of the Shapley prior we proposed in predicting explanations estimated using SHAP algorithms for general models, including GP-SHAP, TreeSHAP, and DeepSHAP. We use the implementation of TreeSHAP and DeepSHAP from the shap package [2].

While algorithms such FastSHAP [22] also learn a vector-valued function that returns explanations given instances, the algorithm require access to the underlying model f during training while ours required previously computed explanations. Due to this importance difference in the problem setup, we do not compare the two algorithm.

We first generate three sets of explanations to set up three regression problems:

Fit a Gaussian process model and then run GP-SHAP to obtain explanations.

Fit a random forest model and then run TreeSHAP to obtain explanations.

Fit a neural network model and then run DeepSHAP to obtain explanations.

After obtaining explanations as groundtruths for this experiment, we randomly divide 70% of them as training data and 30% of them as testing data. We then do the following,

We fit a multi-output GP using the proposed Shapley prior on the training data and predict the explanations of the unseen test data.

We fit a multi-output random forest model on the training data and predict the explanations of the unseen test data.

We fit a multi-output neural network model on the training data and predict the explanations of the unseen test data.

We repeat this experiment 10 times using different seeds and compute the RMSE between the predicted and groundtruths explanations. The results are then plotted in Figure 3.

C.4 Further ablation study: Impact of increased posterior prediction uncertainty on explanation uncertainties

In this ablation study, we aim to examine the effect of increasing the uncertainty in posterior predictions on the corresponding uncertainty in stochastic Shapley values. To demonstrate this, we utilize the diabetic dataset [47] and split the data based on recorded sex. We train our GP model on the male data and employ BayesGP-SHAP to explain the prediction results for both the male training data and the female testing data. We adopt this split because we expect the biological characteristics between males and females to be distinct enough to treat the female data as out-of-sample data, thereby naturally resulting in increased predictive uncertainty for the female data. To further amplify this uncertainty, we multiply each instance in the female testing data by distortion factors of two and three, respectively, and assess the corresponding uncertainties in the explanations.

We begin by illustrating the relationship between the out-of-sampleness of the data and the corresponding increase in predictive posterior uncertainties. This is depicted in Figure 6a, where we observe that as the data becomes more out-of-sample, the predictive uncertainties consistently rise. Even at distortion level 1 , which represents the original female data, we can already observe increased uncertainties compared to the uncertainties derived from male data prediction.

Furthermore, these increased uncertainties in the predictive posterior are reflected in the associated feature explanations. This is evident in Figure 6b, where we visualize the uncertainties associated with the feature explanations. For instance, the green bars representing the average uncertainties in explaining female data with no distortion are consistently larger than the red bars, which represent the average uncertainties of male data explanations. This observation aligns with the higher predictive uncertainties observed in Figure 6a for the female data compared to the male training data.

It is worth noting that the uncertainty for the feature 'sex' remains consistently close to zero. This is because the feature 'sex' is constant within both the female and male datasets. As a result, it acts as the null player in each dataset and obtains an almost Dirac zero as its stochastic Shapley value.

0

20

40

60

80

Posterior standard deviations of predictions

0.00

0.02

0.04

0.06

0.08

0.10

0.12

Density

Female, distortion level =3.0

Female, distortion level =2.0

Female, distortion level =1.0

Male (train)

(a) Predictive posterior standard deviation

age

sex

bmi

bp

s1

s2

s3

s4

s5

s6

0

5

10

15

20

25

30

35

Standard deviation of explanations

Female data, distortion level = 3.0

Female data, distortion level = 2.0

Female data, distortion level = 1.0

Male data (train)

(b) Mean of standard deviations of explanations

groundtruthexplanations

0.30

shapleykernelapproach(RMSE:0.05)

fastshap (RMSE: 0.09)

0.25

0.20

0.15

0.10

0.05

0.00

age

sex

bmi

bp

S1

S2

S3

S5

S6

Distribution of kendall tau score compared with groundtruth explanation

Shapleykernelapproach

FastSHAP

5

4

Density

3

2

1

0

-0.2

0.0

0.2

0.4

0.6

0.8

1.0

Kendall'sTauscore

D Comparison with FastSHAP [39]

We ran a predictive explanation experiment (see Figure 8 and 7 ) using the diabetic dataset from UCI, comparing both our method and FastSHAP. In particular, we use 70% of the data as training, i.e. we first compute the Shapley values for training data and treat them as labels, and predict the explanation for the remaining data. We evaluated performance using both RMSE and Kendall's tau distance (the closer to 1 the better). Notably, Kendall's tau distance assesses the consistency in feature importance ranking between predicted explanations and the true explanations. In the experiments, our method got RMSE 0.05, while FastSHAP got 0.09. In terms of Kendall's tau, our method got an average around 0.7 while FastSHAP got around 0.4, meaning the feature importance ranking recovered from our method is more aligned with the groundtruth SVs than FastSHAP. Our findings indicate that our method outperformed FastSHAP in this experiment.

Note however that interpretation of these comparisons should be nuanced since the two techniques have different usecases: FastSHAP uses knowledge of the model, not the previously computed SVs, whereas GPSHAP uses previously computed SVs and is agnostic to the type of model they came from.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work proposes a new kind of Shapley value (SV) based explanation method. The new methods GP-SHAP and BayesGP-SHAP are like TreeSHAP, LinearSHAP or RKHS-SHAP model-specific SV methods for Gaussian Process (GP) models. The authors further propose another approach of utilizing GP models to amortize (i.e learn) SVs for a model and data point in a predictive task (like FastSHAP). The authors name this the Shapley prior. The two main contributions (model-specific) SV method and the Shapley prior is illustrated in experiments with well-established benchmark datasets.\n\nSTRENGTHS:\nThe paper has many strengths. The work is theoretically well-formulated and motivated. It combines uncertainty research with XAI research to enrich the quality of explanations. This is very important research. The contribution is well formulated and well motivated (formally and verbally). The experimental analysis of the model-specific GP-SHAP and BayesGP-SHAP is convincing. While the work is very theoretical in nature, it is still very pleasant to read, which is hard to achieve, since bridging the gap between two streams of research is not easy. \nThe work adheres perfectly to the style guidelines, includes a well-rounded appendix with code and a small readme file for easier reproducibility. The proofs are to the most part understandable.\n\nWEAKNESSES:\nThe main weaknesses I identified are elaborated below:\n\n1. The predictive (FastSHAP-like) approach to learning SVs with GP models and the Shapley prior feels a bit tagged on and underdeveloped. In lines 256-258 the argument is made that GP models are trained only with x (instance) and y (SV for instance) pairs and the model is not required to train. This is posed as a positive side of this Shapley prior approach. This, of course, is correct that you only need the SV to train, however, doesn't this make the GP models less effective in comparison to, for instance, FastSHAP which is trained on this Shapley loss function (i.e. MSE over model output of coalition with feature i than without feature i). The main benefit I see is that you now have a GP model rather than a big black-box NN like FastSHAP proposes. The GP models, therein, are more explainable which is a desideratum in my opinion for an explanation system (this \"explaining a model with another model that needs to be explained in itself\" can be spun ad absurdum). However, the paper is missing such an experimental comparison of the Shapley prior with FastSHAP or an in-depth discussion of the benefits in my opinion. I want to reiterate that this is not a big problem, however, it is left hanging.\n\n2. The work has some accessibility issues from a ML perspective. Some concepts are hard to follow even with a good background in SV and statistics. I recon that ML readers might have a hard time understanding the contribution well because they already have problems with the problem definitions (beginning of 2). If space allows I'd suggest to certainly enrich the definition section (lines 81-92) with some explanations.\n\n3. The connection to uncertainty research could be a bit stronger. I.e. it would be nice to include a more extensive discussion of what parts of uncertainty are captured in the confidence intervals (i.e. aleatoric or epistemic)?\n\nMinor things:\n- Line 11 ... \"extensive illustrations\" sounds a bit counterintuitive consider changing this\n- Write out the main contribution again in the introduction in a seperate paragraph that is easily digestible for ML engineers\n- Line 134 \"rvs\" is introduced but (I think) not used. Even if it is used consider removing this abbreviation\n- Lines 235-240: I would like to see a more extensive discussion here.\n\nQUESTIONS:\n1. How does your approach compare to FastSHAP in terms of predictive performance? What's the tradeoff in terms of predictive performance and explainability?\n\n2. Can you say anything about what parts of uncertainty are captured in the explanations uncertainty (i.e. aleatoric or epistemic)?\n\nLIMITATIONS:\nThe work includes a limitation section discussing (a) limitations and (b) future work. However, the section is quite short and focuses solely on future work and avenues to explore (e.g. extension to Bayesian NN). A more detailed discussion of the limitations of this particular result would be beneficial.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider Shapley values computed from games $\\nu$ modelled with a Gaussian process. Since the game is stochastic, the Shapley values are now stochastic, and the stochasticity reflects epistemic uncertainty in the explanations. This is unlike previous methods like BayesSHAP, where the uncertainty is estimation error. The approach of the authors, called GP-SHAP, can be used to explain predictions of Gaussian processes. It can also be used to construct a prior over Shapley values, which in turn can be used to perform regression on Shapley values. The authors demonstrate the utility of their approach is three experiments.\n\nSTRENGTHS:\nI would like to thank the authors for their submission. I think this is interesting work.\n\n## Strengths\n\n* The paper is well written and mostly clear. The presentation is reasonably good.\n\n* Pushing a GP through the value allocation computation to explain stochastic predictions seems like a very sensible thing to do.\n\n* Doing the above to define a prior over value allocations and then using this prior for regression of Shapley values is, to the best of my knowledge, novel. I have not seen this before. I should, however, say that I'm not at all familiar with this part of the literature.\n\n* The stochasticity in explanations by GP-SHAP represents epistemic uncertainty in the explanation, whereas the stochasticity in explanations by BayesSHAP is due to estimation error. I think this is an interesting finding.\n\n* I have either skimmed through or looked in more detail at the proofs of the mathematical statements. Apart from some clarity issues (see below), I think everything generally checks out.\n\n* The experiments nicely illustrate the benefits of GP-SHAP.\n\n\n## General Remarks\n\n* In Proposition 11, I was initially very confused by that the expectation of $v_x$ appears. After reading the proof, it became clear that this is an approximation that simplifies the computation, which should actually integrate over $f$. I think this can be better explained in the statement of Propostion 11.\n\n* In Proposition 12, if I'm not mistaken, the transpose in the expression for $\\kappa$ refers to taking inner products in the RKHS of $k$. This is _definitely_ not clear at all and should be clarified!\n\n* In Figure 1, how does BayesSHAP produce its explanations? It is applied to the predictive mean of the GP prediction?\n\nWEAKNESSES:\n## Weaknesses\n\n* I think that providing a proof for Theorem 4 is unnecessarily complicating the exposition, as the Shapley's original proof for the deterministic case can be applied to immediately prove Theorem 4: Let $(\\Omega, \\mathcal{F}, \\mathbb{P})$ be the underlying probability space. For any fixed sample $\\omega \\in \\Omega$, $\\nu$ is d-game (using the authors' language). Therefore, according to Shapley's original proof, for that $\\omega \\in \\Omega$, it can be written in the form of (1). Since (1) holds for all $\\omega \\in \\Omega$, it obviously holds as an equality in terms of random variables.\n\n* Generally, I think that the exposition takes a long time to arrive at the key idea of the paper: model $\\nu$ with a GP, and push it through the value allocation computation in (1). I think that the exposition would be improved by having Section 3 starting out with immediately explaning this key idea. A paper is not supposed to be a novel: please signpost important ideas and conclusions as much as possible.\n\n* In lines 326-339, you highlight the difference between the mean of absolute SSVs and the absolute values of mean SSVs, and state that this gives a different ordering for which feature is most influential. However, you do not analyse whether this different ordering is better or worse, which means that it is not clear whether, in this case, GP-SHAP produced a better ordering or not.\n\n* In lines 340-346, you produce a local explanation graphical model. However, you do not analyse whether this graphical model is reasonable, which means that the reader is not sure whether GP-SHAP did something sensible or not.\n\n* Building a GP prior over Shapley values to do regression is an interesting idea. However, the presentation would be more convincing if you were to point out some existing applications that would actually want to do regression of Shapley values.\n\nQUESTIONS:\n## Conclusion\n\nI think this is a solid submission without any major shortcomings, which is why I am giving an accept.\n\nIf at all possible, I would like to see the following edits in a revision of the authors:\n\n* If I'm right that Theorem 4 follows immediately from the deterministic case, I would really just do that and omit the current proof.\n\n* Please start out Section 3 by explaning what you are working towards: pushing a GP through the value allocation computation\n\n* Please clarify the statement of Proposition 11 (see above).\n\n* Please clarify the transpose in Proposition 12 (see above).\n\n* In Figure 2.(b), please analyse the difference in ordering between the mean of absolute SSVs and the absolute values of SSVs and conclude which of the two orderings is more sensible.\n\n* In Figure 2.(d), please argue whether the graphical model is sensible or not.\n\nEDIT\n\nI have read the authors' rebuttal and the other reviews and remain at my current assessment.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis works introduces the stochastic Shapley values to the explainable AI community. It formulates the stochastic Shapley values for GP models with a closed-form expression for the covariance matrix. Then, it proposes BayesGP-SHAP, which incorporates both sources (of uncertainty from the GP posterior and of the Shapley value estimation). Lastly, it constructs an inference model by proposing a GP prior to the Shapley value to solve the problem of predictive explanation.\n\nSTRENGTHS:\nI enjoy reading the paper as it contains several major strengths: \n+ The motivation of predictive uncertainty in trustworthy ML models is convincing, which paves the way to the proposed GP-SHAP and BayesGP-SHAP.\n+ The literature review is sufficient to highlight the novelty of the approach.\n+ It introduces concepts of stochastic cooperative games and stochastic Shapley values to the explainable AI community.\n+ GP explanations not only satisfy favorable axioms to standard Shapley values but are also able to measure explanation uncertainties and determine statistical dependencies between explanations. \n+ The proposed BayesGP-SHAP integrates both sources of uncertainty from the GP posterior and the estimation of the Shapley value.\n+ It proposes the Shapley prior to dealing with the problem of predictive explanation, which is a nice Bayesian treatment to the predictive explanation problem.\n+ In the experiments, the paper introduces several exploratory analysis methods for practitioners to understand stochastic explanations.\n\nWEAKNESSES:\nThe paper may elaborate on other use cases of predictive explanations. Anyways, I find the Bayesian optimization application in the conclusion interesting.\n\nThere are very minor issues in the references as some words are not capitalized, e.g., Shapley.\n\nQUESTIONS:\nI do not have any questions for the authors.\n\nLIMITATIONS:\nThe authors have adequately stated the limitations in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper introduces a novel approach for explaining Gaussian processes (GPs) by utilizing the analytical covariance structure present in GPs based on extending the concept of Shapley values to stochastic cooperative games, resulting in explanations that are random variables. The GP explanations generated using this approach satisfy favorable axioms similar to standard Shapley values and possess a tractable covariance function across features and data observations. The proposed approach, GP-SHAP, and its variant BayesGP-SHAP, are shown to be effective in providing explanations for Gaussian process models.

STRENGTHS:
The paper presented a novel and principled approach for explaining GP model using stochastic Shapley values. The proposed approach is sound and thoroughly evaluated. It also fills the void that there is no model-specific explanation method for GP.

WEAKNESSES:
The paper should also discuss the limit of using Shapley value in explanation in general.

QUESTIONS:
The font in Figure 2 is too small to read.

LIMITATIONS:
Shapley-value based explanations have some limitations -- two references are provided as follows. 
* Shapley Residuals: Quantifying the limits of the Shapley value for explanations (https://proceedings.neurips.cc/paper/2021/file/dfc6aa246e88ab3e32caeaaecf433550-Paper.pdf)
* Problems with Shapley-value-based explanations as feature importance measures (https://arxiv.org/pdf/2002.11097.pdf)

These limits are inherited in this proposed method.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07