SlowGuess/NeurIPS_2023_OCR huggingface dataset:
4: Compute  $\mathbf{A} = (\mathbf{Z}^{\top}W\mathbf{Z})^{-1}\mathbf{Z}^{\top}\mathbf{W}$  ▷ Shape:  $d\times n_{Z}$
5: Compute  $\mathbf{B}(\mathbf{X},\mathcal{S}) = [(\mathbf{K}_{\tilde{\mathbf{X}}_S\tilde{\mathbf{X}}_S} + \lambda I)^{-1}k_S(\tilde{\mathbf{X}}_S,\mathbf{X}_S)\quad \mathbf{for}\ S\ in\ \mathcal{S}]$  ▷ Shape:  $n_Z\times n_I\times n$
6: Compute  $\mathbf{Q}$  where  $\mathbf{Q}_{i,l,k} = \sum_{j}\mathbf{B}(\mathbf{X},\mathcal{S})_{i,j,k}\mathbf{L}_{j,l}$ $\triangleright$  Shape:  $n_Z\times n\times n_I$
7: Compute  $\mathbf{R}$  where  $\mathbf{R}_{i,k,l} = \sum_{j}\mathbf{A}_{i,j}\mathbf{Q}_{j,k,l}$ $\triangleright$  Shape:  $d\times n\times n_I$
8: Compute  $\mathbf{V}$  where  $\mathbf{V}_{i,m,k,n} = \sum_{j,l}\mathbf{R}_{i,j,k}\mathbf{R}_{m,l,n}$  ▷Shape:dxdxn
9: Compute  $\mathbf{E}$  where  $\mathbf{E}_{i,k} = \sum_{j}\mathbf{B}(\mathbf{X},\mathcal{S})_{i,j,k}\tilde{m} (\tilde{\mathbf{X}})_j$  ▷ Shape:  $n_Z\times n$
10: Compute  $\Phi = \mathbf{A}\mathbf{E}$  The mean stochastic Shapley values of shape  $d\times n$
1: if algo = GP-SHAP then
2: return mean explanations  $\Phi$  and covariance  $\mathbf{V}$  between  $d$  features and  $n$  instances
3: else if algo = BayesGP-SHAP then
4: Compute  $s^2 = \mathrm{diag}\left((\mathbf{E} - \mathbf{Z}\Phi)^\top \mathbf{W}(\mathbf{E} - \mathbf{Z}\Phi)\right) + \mathrm{diag}(\Phi^\top \Phi)$  ▷ Shape:  $n\times 1$
5: Sample  $\sigma^2$  from Scaled-Inv-  $\chi^2\left(n_0 + n_Z,\frac{n_0\sigma_0^2 + n_Zs^2}{n_0 + n_Z}\right)$ $\triangleright$  Shape:  $n\times 1$
6: return mean explanations  $\Phi$  and covariance  $\mathbf{V} + (\mathbf{Z}^{\top}\mathbf{W}\mathbf{Z})^{-1}\sigma^{2}$
7: end if
Computational considerations. In terms of computational complexity, one of the most demanding operations in the algorithm is the computation of conditional mean embeddings in step 5. Instead of naively inverting an  $n \times n$  matrix, which would have a computational cost of  $\mathcal{O}(n^3)$ , we employ the conjugate gradient method to reduce the computation of the conditional mean embedding component to  $\mathcal{O}(n^2a)$ , where  $a \ll n$  represents the number of conjugate gradient iterations. Additionally, to further reduce runtime, we utilize the variational sparse GP model [49]. This model learns a set of inducing locations  $\tilde{\mathbf{X}}$  with a size of  $n_I \ll n$ , which can be reused for the estimation of conditional mean embeddings in the algorithm. Consequently, the computation of the conditional expectation is reduced from  $\mathcal{O}(n^2a)$  to  $\mathcal{O}(n_I^2a)$ . Another computational burden arises from the computation of the full covariance matrix across  $d$  features and  $n$  instances, which requires storage of a  $n^2d^2$  matrix. However, since the full covariance matrix can be factorized into the  $\mathbf{R}$  component from step 7 of the algorithm, we can store this low-rank component and compute covariances between specific instances when necessary. It is worth noting that this decomposition of the covariance matrix allows us to avoid redundant computations when computing the covariance component, as we no longer need to iterate over all possible coalitions twice. Finally, we can further speed up our computational by parallelising computation across the sub-sampled coalitions in step 5.
B.1 Section 2 proofs: Stochastic Shapley values
We include the full proof of the derivation of stochastic Shapley values for completeness. The proof is analogous to the original work of Shapley's [1] but extended to random variable payoffs. Ma et al. [16] has also proved the same theorem but used a different proving strategy. They started with the solution and showed it satisfies the axioms and then prove uniqueness, whereas the following proof starts from the characterisation of s-games and derive the solution from a bottom-up fashion.
To facilitate the proof, we first introduce the concept of stochastic symmetric game.
Proposition 15 (s-symmetric games). Let  $C$  be a real-valued random variables, then the symmetric game  $\nu_{C,R}(S) \coloneqq C\mathbf{1}[R \subseteq S]$  gets a stochastic shapley value as,
$$
\phi_ {i} \left(\nu_ {C, R}\right) = \frac {C}{r} \tag {16}
$$
where  $r = |R|$ .
Proof. Take any  $i, j \in R$ , pick a permutation  $\pi \in \Pi(U)$  so that  $\pi R = R$  and  $\pi i = j$ , so the induced game  $\pi \nu_{C,R} = \nu_{C,R}$ , and therefore by the s-symmetry axiom,
$$
\phi_ {j} \left(\nu_ {C, R}\right) = \phi_ {i} \left(\nu_ {C, R}\right) \tag {17}
$$
Now by the s-efficiency axiom,
$$
C = \nu_ {C, R} (R) = \sum_ {j \in R} \phi_ {j} \left(\nu_ {C, R}\right) = r \phi_ {i} \left(\nu_ {C, R}\right) \tag {18}
$$
for any  $i\in R$
Now we can characterise the form of any stochastic game as follows:
Proposition 16. All  $s$ -games with finite carrier can be written as a linear combination of  $s$ -symmetric games,
$$
\nu = \sum_ {R \subseteq N, R \neq \emptyset} \nu_ {c _ {R} (\nu), R} \tag {19}
$$
$$
C _ {R} (\nu) = \sum_ {T \subseteq R} (- 1) ^ {r - t} \nu (T) \tag {20}
$$
Proof. We start by verifying
$$
\nu (S) = \sum_ {R \subseteq N, R \neq \emptyset} \nu_ {c _ {R} (\nu), R} (S) \tag {21}
$$
holds for all  $S \subseteq U$ , and for any finite carrier  $N$  of  $\nu$ . If  $S \subseteq N$ , then we can rewrite the expression as,
$$
\begin{array}{l} \nu (S) = \sum_ {R \subseteq S} \sum_ {T \subseteq R} (- 1) ^ {r - t} \nu (T) (22) \\ = \sum_ {T \subseteq S} \sum_ {T \subseteq R \subseteq S} (- 1) ^ {r - t} \nu (T) (23) \\ = \sum_ {T \subseteq S} \nu (T) \sum_ {r = t} ^ {s} (- 1) ^ {r - t} \binom {s - t} {r - t} (24) \\ = \nu (S) (25) \\ \end{array}
$$
where in the last equation we used the fact that  $\sum_{r = t}^{s}(-1)^{r - t}\binom{s - t}{r - t}$  is a binomial expansion of  $(1 + (-1))^{s - t}$ , therefore the only non-zero expression is when  $t = s$ .
We can now prove the uniqueness of stochastic Shapley values,
Proof. First, let us denote
$$
\gamma_{i}(S):= \sum_{\substack{R\subseteq N\\ S\cup \{\overline{i}\} \subseteq R}}(-1)^{r - s}\frac{1}{r}.
$$
Applying the s-linearity axiom on  $\phi$  to the characterisation of  $\nu$  from the previous propositions leads us to the following,
$$
\begin{array}{l} \phi_ {i} (\nu) = \phi_ {i} \left(\sum_ {R \subseteq N, R \neq \emptyset} \nu_ {C _ {R} (\nu), R}\right) (26) \\ = \sum_ {R \subseteq N, R \neq \emptyset} \phi_ {i} \left(\nu_ {C _ {R} (\nu), R}\right) (27) \\ = \sum_ {R \subseteq N, i \in R} c _ {R} (\nu) \frac {1}{r} (28) \\ = \sum_ {R \subseteq N, i \in R} \frac {1}{r} \left(\sum_ {S \subseteq R} (- 1) ^ {r - s} \nu (S)\right) (29) \\ = \sum_ {S \subseteq N} \sum_ {\substack {R \subseteq N \\ S \cup \{\overline{i}\} \subseteq R}} (- 1) ^ {r - s} \nu (S) \frac {1}{r} (30) \\ = \sum_ {S \subseteq N} \gamma_ {i} (S) \nu (S) (31) \\ = \sum_ {\substack {S \subseteq N \\ i \in S}} \gamma_ {i} (S) \nu (S) + \gamma_ {i} (S - \{i \}) \nu (S - \{i \}) (32) \\ = \sum_ {\substack {S \subseteq N \\ i \in S}} \gamma_ {i} (S) (\nu (S) - \nu (S - \{i \})) (33) \\ = \sum_ {\substack {S \subseteq N \\ i \in S}} \frac {(s - 1) ! (n - s) !}{n !} (\nu (S) - \nu (S - \{i \})) (34) \\ = \sum_ {S \subseteq N \backslash \{i \}} c _ {| S |} (\nu (S \cup i) - \nu (S)) (35) \\ \end{array}
$$
where in (32) we used the following observation: given  $i \notin S' \subseteq N$ , and  $S = S' \cup \{i\}$ , then  $\gamma_i(S) = -\gamma_i(S')$ .
It satisfies uniqueness by construction.
Proof. The equivalence between mean of stochastic Shapley values and deterministic Shapley values of mean game is trivial to show leveraging the linearity of expectation. The variance of  $\mathbb{V}[\phi_i(\nu)]$  can be shown by repeatedly applying the standard identity  $\mathbb{V}[X + Y] = \mathbb{V}[X] + \mathbb{V}[Y] + 2\mathbb{C}[X,Y]$  for random variables  $X, Y$ . Now consider the deterministic Shapley values of variance game  $\mathbb{V}[\nu]$ ,
$$
\bar {\phi} _ {i} [ \mathbb {V} [ \nu (\cdot) ] ] = \sum_ {S \subseteq N \backslash \{i \}} c _ {| S |} (\mathbb {V} [ \nu (S \cup i) ] - \mathbb {V} [ \nu (S) ]) \tag {36}
$$
Comparing to the expression of  $\mathbb{V}[\phi_i(\nu)]$  from the lemma,
$$
\mathbb{V}[\phi_{i}(\nu)] = \sum_{S\subseteq N\setminus \{i\}}\sum_{S^{\prime}\subseteq N\setminus \{i\}}c_{|S|}c_{|S^{\prime}|}\bigl(\mathbb{C}[\nu_{S\cup i},\nu_{S^{\prime}\cup i}] - \mathbb{C}[\nu_{S\cup i},\nu_{S^{\prime}}] - \mathbb{C}[\nu_{S},\nu_{S^{\prime}\cup i}] + \mathbb{C}[\nu_{S},\nu_{S^{\prime}}]\bigr),
$$
even if we assume mutual independence across all payoff random variables, leading to  $\mathbb{C}[\nu(S \cup i), \nu(S)] = 0$  for all  $S$ , we still would not subtract but instead sum the variance of  $\mathbb{V}[\nu(S \cup i)]$  and  $\mathbb{V}[\nu(S)]$ . Therefore the variances of stochastic Shapley values is not the same as the deterministic Shapley values of the variance game.
B.2 Section 3.1 proofs on the stochastic Shapley values for induced stochastic game from GP
Proof. This is a direct application of Chau et al. [18, Proposition 3.2] to the distribution  $P(X \mid X_S = \mathbf{x}_S)$ .
Proof. Recall from Lundberg and Lee [2, Theorem 2], for deterministic Shapley values, given a deterministic payoff  $\bar{\mathbf{v}}_{\mathbf{x}}$  for all  $2^{d}$  coalitions, the expression of Shapley values for each  $i \in [d]$ ,
$$
\bar {\phi} _ {\mathbf {x} i} = \sum_ {S \subseteq [ d ] \backslash \{i \}} c _ {| S |} \left(\bar {\nu} _ {f} (S \cup i) - \bar {\nu} _ {f} (S)\right) \tag {37}
$$
can be written compactly as the following vector,
$$
\bar {\phi} _ {\mathbf {x}} = \mathbf {A} \bar {\mathbf {v}} _ {\mathbf {x}}. \tag {38}
$$
We can therefore similarly write down the form of the stochastic Shapley values using this linear operator  $\mathbf{A}$ , acting now on a vector of random variable output stochastic payoff vector  $\mathbf{v}_{\mathbf{x}}$
$$
\phi_ {\mathbf {x}} = \mathbf {A} \mathbf {v} _ {\mathbf {x}}. \tag {39}
$$
Nonetheless, as Proposition 8 implies that  $\mathbf{v}_{\mathbf{x}}$  is a multivariate Gaussian, therefore  $\phi_{\mathbf{x}}$  is also multivariate Gaussian with mean and covariance the following,
$$
\mathbf {v} _ {\mathbf {x}} \sim \mathcal {N} \left(A \mathbb {E} [ \mathbf {v} _ {\mathbf {x}} ], A \mathbb {V} [ \mathbf {v} _ {\mathbf {x}} ] A ^ {\top}\right). \tag {40}
$$
B.3 Section 3.2 proofs on estimation
To proceed, we first introduce the concepts of conditional mean embedding as a tool to estimate conditional expectation of functions living in their corresponding RKHSs,
Definition 17 (Conditional mean embedding [38]). Let  $X, Y$  be random variables and  $k: \mathcal{X} \to \mathcal{X} \to \mathbb{R}$  a kernel on  $X$ , then we define the following as the conditional mean embedding of  $p(X \mid Y = y)$ ,
$$
\mu_ {X \mid Y = y} := \int k (\cdot , X) d \mathbb {P} (X \mid Y = y) \tag {41}
$$
Proposition 18 (Conditional Mean estimation). For random variable  $X, Y$ , and a kernel  $k: \mathcal{X} \to \mathcal{X} \to \mathbb{R}$  on  $\mathcal{X}$  and a kernel  $l: \mathcal{Y} \to \mathcal{Y} \to \mathbb{R}$  on  $\mathcal{Y}$ . Given observations  $\mathbf{D} = \{\mathbf{X}, \mathbf{y}\}$ , the empirical conditional mean embedding can be estimated as
$$
\hat {\mu} _ {X \mid Y = y} = l (y, \mathbf {y}) (\mathbf {L} _ {\mathbf {y} \mathbf {y}} + \lambda I) ^ {- 1} k (\mathbf {X}, \cdot), \tag {42}
$$
where  $l(y,\mathbf{y}) = [l(y,y_1),\dots ,l(y,y_n)]^\top$  and  $k(\cdot ,\mathbf{X}) = [k(\cdot ,\mathbf{x}_1),\dots ,k(\cdot ,\mathbf{x}_n)]^\top$ , the parameter  $\lambda >0$  is there to stabilise the inversion. Now for  $f\in \mathcal{H}_k$ , the conditional expectation can then be estimated as,
$$
\begin{array}{l} \hat {\mathbb {E}} [ f (X) \mid Y = y ] = \left\langle \hat {\mu} _ {X \mid Y = y}, f \right\rangle (43) \\ = l (y, \mathbf {y}) \left(\mathbf {L} _ {\mathbf {y y}} + \lambda I\right) ^ {- 1} \mathbf {f}, (44) \\ \end{array}
$$
where  $\mathbf{f} = [f(\mathbf{x}_1),\dots ,f(\mathbf{x}_n)]^\top$
Proof. This is standard result from literature, please read Song et al. [50], Muandet et al. [38] for more details.
Now we can apply these propositions to estimate the mean and covariance functions of the induced stochastic game from GP,
Proof. Without loss of generality, we will demonstrate this proposition with  $\tilde{m},\tilde{k}$  obtained via standard GP regression, i.e.,
$$
\tilde {m} (\mathbf {x}) = k (\mathbf {x}, \mathbf {X}) \left(\mathbf {K} _ {\mathbf {X X}} + \sigma^ {2} I\right) ^ {- 1} \mathbf {y} \tag {45}
$$
$$
\tilde {k} (\mathbf {x}, \mathbf {x} ^ {\prime}) = k (\mathbf {x}, \mathbf {x} ^ {\prime}) - k (\mathbf {x}, \mathbf {X}) \left(\mathbf {K} _ {\mathbf {X X}} + \sigma^ {2}\right) ^ {- 1} k \left(\mathbf {X}, \mathbf {x} ^ {\prime}\right). \tag {46}
$$
Starting with the mean function,
$$
\begin{array}{l} \mathbb {E} [ \tilde {m} (X) \mid X _ {S} = \mathbf {x} _ {S} ] = \mathbb {E} _ {X} \left[ k (X, \mathbf {X}) \left(\mathbf {K} _ {\mathbf {X X}} + \sigma^ {2} I\right) ^ {- 1} \mathbf {y} \mid X _ {S} = \mathbf {x} _ {S} \right] (47) \\ = \left\langle k (\cdot , \mathbf {X}) \left(\mathbf {K} _ {\mathbf {X X}} + \sigma^ {2} I\right) ^ {- 1} \mathbf {y}, \mu_ {X \mid X _ {S} = \mathbf {x} _ {S}} \right\rangle \mathcal {H} _ {k}. (48) \\ \end{array}
$$
We can replace the population conditional mean embedding with the empirical version, and expand,
$$
\begin{array}{l} \hat {\mathbb {E}} [ \tilde {m} (X) \mid X _ {S} = \mathbf {x} _ {S} ] = \left\langle k (\cdot , \mathbf {X}) \left(\mathbf {K} _ {\mathbf {X X}} + \sigma^ {2} I\right) ^ {- 1} \mathbf {y}, \hat {\mu} _ {X \mid X _ {S} = \mathbf {x} _ {S}} \right\rangle_ {\mathcal {H} _ {k}} (49) \\ = k _ {S} \left(\mathbf {X} _ {S}, \mathbf {x} _ {S}\right) \left(\mathbf {K} _ {\mathbf {X} _ {S} \mathbf {X} _ {S}} + \lambda I\right) ^ {- 1} \mathbf {K} _ {\mathbf {X} \mathbf {X}} \left(\mathbf {K} _ {\mathbf {X} \mathbf {X}} + \sigma^ {2} I\right) ^ {- 1} \mathbf {y} (50) \\ = \mathbf {b} (\mathbf {x}, S) ^ {\top} \tilde {m} (\mathbf {X}). (51) \\ \end{array}
$$
Analogously, the conditional expectation of the posterior covariance function, i.e.,  $\mathbb{E}[\tilde{k}(X,X^{\prime})\mid X_S = \mathbf{x}_S,X_S^{\prime} = \mathbf{x}_S^{\prime}]$ , can be estimated following the steps above,
$$
\mu_ {X \mid X _ {S} = \mathbf {x} _ {S}} ^ {\top} \mu_ {X ^ {\prime} \mid X _ {S} ^ {\prime} = \mathbf {x} _ {S} ^ {\prime}} - \mu_ {X \mid X _ {S} = \mathbf {x} _ {S}} ^ {\top} k (\cdot , \mathbf {X}) \left(\mathbf {K} _ {\mathbf {X X}} + \sigma^ {2} I\right) ^ {- 1} k \left(\mathbf {X}, \cdot\right) \mu_ {X ^ {\prime} \mid X _ {S} ^ {\prime} = \mathbf {x} _ {S} ^ {\prime}}. \tag {52}
$$
After replacing the population conditional mean embedding as their empirical estimates, we can arrive at the solution.
Proof. The result follows directly from the previous proposition. Recall  $\phi (\hat{\nu}_f(\mathbf{x},\cdot)) = \mathbf{A}\hat{\mathbf{v}}_{\mathbf{x}}$  for  $\hat{\mathbf{v}}_{\mathbf{x}}$  the vector of stochastic payoffs for each coalition. To estimate the mean, we
$$
\begin{array}{l} \mathbb {E} \left[ \phi \left(\hat {\nu} _ {f} (\mathbf {x}, \cdot)\right) \right] = \mathbf {A} \mathbb {E} \left[ \hat {\mathbf {v}} _ {\mathbf {x}} \right] (53) \\ = \mathbf {A} \left[ \begin{array}{c} \hat {m} _ {\nu} (\mathbf {x}, S _ {1}) \\ \vdots \\ \hat {m} _ {\nu} (\mathbf {x}, S _ {2 ^ {d}}) \end{array} \right] (54) \\ = \mathbf {A} \left[ \begin{array}{c} \mathbf {b} (\mathbf {x}, S _ {1}) ^ {\top} \tilde {m} (\mathbf {X}) \\ \vdots \\ \mathbf {b} (\mathbf {x}, S _ {2 ^ {d}}) ^ {\top} \tilde {m} (\mathbf {X}) \end{array} \right] (55) \\ = \mathbf {A B} (\mathbf {x}, [ d ]) ^ {\top} \tilde {m} (\mathbf {X}). (56) \\ \end{array}
$$
Recall  $\mathbb{V}[\mathbf{v}_{\mathbf{x}}]_{i,j} = \hat{k}_{\nu}((\mathbf{x},S_i),(\mathbf{x},S_j)) = \mathbf{b}(\mathbf{x},S_i)^\top \tilde{\mathbf{K}}_{\mathbf{X}\mathbf{X}}\mathbf{b}(\mathbf{x},S_j)$ , the derivation for the covariance matrix then follows analogously as the derivation for the mean,
$$
\begin{array}{l} \mathbb {V} [ \phi (\hat {\nu} _ {f} (\mathbf {x}, \cdot)) ] = \mathbf {A} \mathbb {V} [ \hat {\mathbf {v}} _ {\mathbf {x}} ] \mathbf {A} ^ {\top} (57) \\ = \mathbf {A} \left[ \mathbf {b} \left(\mathbf {x}, S _ {i}\right) ^ {\top} \tilde {\mathbf {K}} _ {\mathbf {X X}} \mathbf {b} \left(\mathbf {x}, S _ {j}\right) \right] _ {i = 1, j = 1} ^ {2 ^ {d}, 2 ^ {d}} \mathbf {A} ^ {\top} (58) \\ = \mathbf {A} \mathbf {B} (\mathbf {x}, [ d ]) ^ {\top} \tilde {\mathbf {K}} _ {\mathbf {X X}} \mathbf {B} (\mathbf {x}, [ d ]) \mathbf {A} ^ {\top}. (59) \\ \end{array}
$$
Proof. See Slack et al. [20, Section 3.1].
$$
\phi \mid \sigma^ {2}, \mathbf {Z} _ {\ell}, \mathbf {x}, \mathbf {D} \sim \mathcal {N} \left(\mathbf {A} _ {\ell} \mathbf {B} (\mathbf {x}, \boldsymbol {\mathcal {S}})\right) ^ {\top} \tilde {m} (\mathbf {X}), \mathbf {A} _ {\ell} \mathbf {B} (\mathbf {x}, \boldsymbol {\mathcal {S}}) ^ {\top} \tilde {\mathbf {K}} _ {\mathbf {X X}} \mathbf {B} (\mathbf {x}, \boldsymbol {\mathcal {S}}) \mathbf {A} _ {\ell} ^ {\top} + \left(\mathbf {Z} _ {\ell} ^ {\top} \mathbf {W} _ {\ell} \mathbf {Z} _ {\ell}\right) ^ {- 1} \sigma^ {2}\left. \right)
$$
Proof. We drop the bar notation of  $\bar{\phi}$  to unify notations. Given the posterior  $\mathrm{GP}f\mid \mathbf{D}\sim \mathcal{GP}(\tilde{m},\tilde{k})$
$$
p \left(\phi \mid \sigma^ {2}, \mathbf {Z} _ {\ell}, \mathbf {x}, \mathbf {D}\right) = \int p \left(\phi \mid \sigma^ {2}, \mathbf {Z} _ {\ell}, f, \mathbf {x}, \mathbf {D}\right) p (f \mid \mathbf {D}) d f \tag {60}
$$
Using a standard Gaussian conjugacy procedure, we can derive the variance as the sum of variances from GP-SHAP and BayesSHAP. While it is possible to integrate  $p(\sigma^2 \mid \mathbf{Z}_\ell, f, \mathbf{x}, \mathbf{D})$  with respect to the posterior, this leads to a complex scaled mixture of normals that is difficult to model. Instead, we construct a scaled inverse chi-square distribution with  $s^2(\mathbb{E}[\mathbf{v}_{\mathbf{x}}])$ , which represents the error of the weighted regression with respect to the mean payoffs  $\mathbb{E}[\mathbf{v}_{\mathbf{x}}]$ . We sample  $\sigma^2$  from the following distribution:
$$
\sigma^ {2} \mid \mathbf {Z} _ {\ell}, \mathbf {x}, \mathbf {D} \sim \text {S c a l e - I n v -} \chi^ {2} \left(\ell_ {0} + \ell , \frac {\ell_ {0} \sigma_ {0} ^ {2} + \ell s ^ {2} (\mathbb {E} [ \mathbf {v} _ {\mathbf {x}} ])}{\ell_ {0} + \ell}\right). \tag {61}
$$
B.4 Proofs for section 4 on predictive explanation and Shapley prior
Proof. The proof is similar to how we proved previous propositions but applied to prior  $\mathrm{GP}f\sim \mathcal{GP}(0,k)$  instead. If we set,
$$
\nu_ {f} (\mathbf {x}, S) = \mathbb {E} [ f (X) \mid X _ {S} = \mathbf {x} _ {S} ], \tag {62}
$$
then  $\nu_{f}$  is a GP on the joint space of data and coalitions with mean 0, and covariance function,
$$
\begin{array}{l} \operatorname {c o v} \left(\nu_ {f} (\mathbf {x}, S), \nu_ {f} \left(\mathbf {x} ^ {\prime}, S ^ {\prime}\right)\right) = \mathbb {E} \left[ k \left(X, X ^ {\prime}\right) \mid X _ {S} = \mathbf {x} _ {S}, X _ {S ^ {\prime}} ^ {\prime} = \mathbf {x} _ {S ^ {\prime}} ^ {\prime} \right] (63) \\ = \mu_ {X | X _ {S} = \mathbf {x} _ {S}} ^ {\top} \mu_ {X | X _ {S ^ {\prime}} = \mathbf {x} _ {S ^ {\prime}} ^ {\prime}}. (64) \\ \end{array}
$$
Since  $\phi = \mathbf{A}\mathbf{v}_{\mathbf{x}}$  for  $\mathbf{v}_{\mathbf{x}}$  the vector of stochastic payoff from the game induced by the GP prior, the mean stays 0, and the covariance is,
$$
\begin{array}{l} \kappa (\mathbf {x}, \mathbf {x} ^ {\prime}) = \mathbf {A} \left[ \mu_ {X | X _ {S _ {i}} = \mathbf {x} _ {S _ {i}}} ^ {\top} \mu_ {X | X _ {S _ {j}} = \mathbf {x} _ {S _ {j}} ^ {\prime}} \right] _ {i = 1, j = 1} ^ {2 ^ {d}, 2 ^ {d}} \mathbf {A} ^ {\top} (65) \\ = \mathbf {A} \Psi (\mathbf {x}) ^ {\top} \Psi \left(\mathbf {x} ^ {\prime}\right) \mathbf {A} ^ {\top} (66) \\ = \mathcal {A} (\mathbf {x}) ^ {\top} \mathcal {A} \left(\mathbf {x} ^ {\prime}\right), (67) \\ \end{array}
$$
therefore we have a matrix-valued covariance kernel  $\kappa$  to build a prior over the induced Shapley values.
where  $\tilde{m}_{\phi}(\mathbf{x}^{\prime}) = b_{\kappa}(\mathbf{x}^{\prime},\mathbf{X})^{\top}\operatorname {vec}(\Phi_{\mathbf{X}}),b_{\kappa}(\mathbf{x}^{\prime},\mathbf{X}):= (\mathcal{H}_{\mathbf{XX}} + \sigma_{\phi}^{2}I)^{-1}\kappa (\mathbf{X},\mathbf{x}^{\prime}),\mathcal{H}_{\mathbf{XX}}$  is the gram matrix for kernel  $\kappa$  of size  $nd\times nd,\kappa (\mathbf{x}^{\prime},\mathbf{X}) = [\kappa (\mathbf{x}^{\prime},\mathbf{x}_{1}),\dots ,\kappa (\mathbf{x}^{\prime},\mathbf{x}_{n})]$  is of size  $d\times nd$  and  $\sigma_{\phi}^{2}$  is the noise parameter for regression.
Proof. Follows from standard vector-valued Gaussian process regression results. See Alvarez et al. [51] for a detailed discussion on regression with matrix-valued kernels.  $\square$
Proof. There are two ways to see this. First is by brute force and rearranging the terms in the posterior mean expression. The other is to leverage the vector-valued representer theorem [52] and write the posterior mean as,
$$
\begin{array}{l} \tilde {m} _ {\phi} \left(\mathbf {x} ^ {\prime}\right) = \sum_ {i = 1} ^ {n} \mathcal {A} \left(\mathbf {x} ^ {\prime}\right) ^ {\top} \mathcal {A} \left(\mathbf {x} _ {i}\right) \alpha_ {i}, \quad \alpha_ {i} \in \mathbb {R} ^ {d} (68) \\ = \sum_ {i = 1} ^ {n} \mathbf {A} \Psi \left(\mathbf {x} ^ {\prime}\right) ^ {\top} \Psi \left(\mathbf {x} _ {i}\right) \mathbf {A} ^ {\top} \alpha_ {i} (69) \\ = \mathbf {A} \left(\sum_ {i = 1} ^ {n} \Psi \left(\mathbf {x} ^ {\prime}\right) ^ {\top} \Psi \left(\mathbf {x} _ {i}\right) \mathbf {A} ^ {\top} \alpha_ {i}\right) (70) \\ = \mathbf {A} \tilde {\mathbf {v}} _ {\mathbf {x} ^ {\prime}} (71) \\ \end{array}
$$
after some linear algebra exercises, we can see that  $\alpha_{i}$  is the  $[i:i + (d - 1)]$  sub-vector of  $(\mathcal{X}_{\mathbf{X}\mathbf{X}} + \sigma_{\phi}^{2}I)^{-1}\operatorname {vec}(\Phi_{\mathbf{X}})$
C Implementation details and further illustrations.
All illustrations are run locally on a MacBookPro 2021 with Apple M1 pro chip.
C.1 Ablation study on different notions of uncertainties captured
To demonstrate the difference between the uncertainties captured by GP-SHAP, BayesSHAP, and BayesGP-SHAP, we utilise the California housing dataset [41]. This dataset was derived from the 1990 U.S. census, each observation represent a census block group. A block group is the smallest geographical unit for which the U.S. Census Bureau publishes sample data (a block group typically has a population of 600 to 3,000 people). The dataset includes 20640 instances with 8 numerical features measuring the following:
MedInc: Median income in block group
- HouseAge: Median house age in block group
- AveRooms: Average number of rooms per household
- AveBedrms: Average number of bedrooms per household
Population: Block group population
- AveOccup: Average number of household members
- Latitude: Block group latitude
- Longitude: Block group longitude
The target variable is the median house value for California districts, expressed in hundreds of thousands of dollars. In the following, we train a GP model and extract explanations using GP-SHAP, BayesSHAP, and BayesGP-SHAP, for 4 different configurations:
1. trained on  $25\%$  of data, estimate the Shapley values using  $50\%$  of coalitions.
2. trained on  $25\%$  of data, estimate the Shapley values using  $100\%$  of coalitions.
3. trained on  $100\%$  of data, estimate the Shapley values using  $50\%$  of coalitions.
4. trained on  $100\%$  of data, estimate the Shapley values using  $100\%$  of coalitions.
To fit the GP model, we employ a sparse Variational GP approach with 200 learnable inducing point locations. The evidence lower bound is optimized using batch gradient descent with a batch size of 64, a learning rate of 0.01, and 100 iterations. The RBF kernel with learnable bandwidths initialized using the median heuristic approach is used for the sparse GP. The inducing locations are initialized using a standard clustering approach to obtain a representative set of inducing points.