SlowGuess/NeurIPS_2023_OCR huggingface dataset:
After training the model, we reuse the learned inducing points and kernel bandwidths for the explanation algorithms. The explanations are obtained using the procedure described in Algorithm 1 of our work.
(a) Mean of each stochastic explanations
(b) Standard deviation of each explanations
Figure 4: We plot the beeswarm plot of the mean and standard deviations of each stochastic explanations from BayesGP-SHAP fitted on the housing dataset. The features are ranked according to the distance span by the largest and smallest mean (std) stochastic Shapley values.
In Figure 1 of our paper, we present the stochastic Shapley values for the 11th observation, computed using the three explanation algorithms. The plot includes the  $95\%$  credible interval to visualize the uncertainties associated with the explanations.
Further illustration: In Figure 4, we plot the beeswarm plot on the mean and standard deviation of each stochastic explanations respectively. We color the point based on the relative size of the feature value compared to the rest. We see that in Figure 4a, which plotted the mean stochastic shapley values for each observation, the relationship between most features' explanation to the target variable is quite linear. For example, the higher the median income (MedInc), the more positive those feature contribute to predicting the respective median house value. On the other hand, Figure 4b illustrated the standard deviation of each stochastic explanations. In general, we see that the larger the feature values are, the more uncertain the explanation becomes. Nonetheless, we see that the feature contributing the most, defined as the feature having largest distance spanned by their most positive and most negative mean stochastic Shapley values, does not necessarily have the largest variation respectively.
C.2 Exploratory analysis of the stochastic explanations
For this illustration, we utilise the breast cancer dataset [42], containing 569 patients with 30 numeric features. They are computed from a digitized image of a fine needle aspirate (FNA) of a breast mass and describe characteristics of the cell nuclei present in the image:
- radius (mean of distances from center to points on the perimeter)
- texture (standard deviation of gray-scale values)
- perimeter
area
- smoothness (local variation in radius lengths)
compactness  $\left(\frac{\text{perimeter}^2}{\text{area}-1}\right)$
- concavity (severity of concave portions of the contour)
- concave points (number of concave portions of the contour)
symmetry
- fractal dimension ("coastline approximation" - 1)
The goal is to predict whether a tumour is malignant or benign. We first fit a GP model with RBF kernel using again the sparse Variational GP formulation with 200 learnable inducing locations. We initialise the inducing points using standard clustering techniques on the data. The evidence lower bound objective is optimised with a learning rate of  $1e^{-4}$  and 1000 iterations using batch gradient descent of batch size 64. To obtain the explanations, we run the BayesGP-SHAP algorithm with  $2^{16}$
(a) Mean of each stochastic explanations
Figure 5: We plot the violin plot of the mean and standard deviations of each stochastic explanations from BayesGP-SHAP fitted on the breast cancer. The features are ranked according to the distance span by the largest and smallest mean (std) stochastic Shapley values.
(b) Standard deviation of each explanations
number of coalitions. We do not compare GP-SHAP and BayesSHAP here because the BayesSHAP uncertainties have shrunk to almost zero, i.e., the mean standard deviations from the BayesSHAP uncertainties across all features and data is 0.0002. This reconfirms the fact from Slack et al. [20] that as we increase the sample size the estimation error goes to zero, thus the uncertainties from BayesSHAP goes to zero as well. On the other hand, GP-SHAP uncertainties still remain valid because it represents the GP predictive uncertainties, which do not shrink to zero as we increase the number of coalitions we use to estimate the SVs.
Further illustrations: In Figure 5, we plot two violin plots to illustrate the relationship between mean and standard deviation of the stochastic values with respect to the size of the original feature. We see that the feature "worst fractal dimension" are the second most influential feature in terms of mean stochastic explanations and also the feature that has highest uncertainty around its explanations. In comparison with the housing prediction problem illustrated in Figure 4, the higher the feature value doesn't necessary give higher uncertainty around its explanation.
C.3 Predictive explanations
For this illustration, we utilise the Diabetes dataset [47] with 442 patient data and 10 numeric features measuring the following:
age: age in years
- sex
- bmi: body mass index
- bp: average blood pressure
s1: total serum cholesterol
s2: low-density lipoproteins
s3: high-density lipoproteins
s4: total cholesterol
s5: Log of serum triglycerides level
s6: blood sugar level
The experiment is to assess the effectiveness of the Shapley prior we proposed in predicting explanations estimated using SHAP algorithms for general models, including GP-SHAP, TreeSHAP, and DeepSHAP. We use the implementation of TreeSHAP and DeepSHAP from the shap package [2].
While algorithms such FastSHAP [22] also learn a vector-valued function that returns explanations given instances, the algorithm require access to the underlying model  $f$  during training while ours required previously computed explanations. Due to this importance difference in the problem setup, we do not compare the two algorithms.
We first generate three sets of explanations to set up three regression problems:
1. Fit a Gaussian process model and then run GP-SHAP to obtain explanations.
2. Fit a random forest model and then run TreeSHAP to obtain explanations.
3. Fit a neural network model and then run DeepSHAP to obtain explanations.
After obtaining explanations as groundtruths for this experiment, we randomly divide  $70\%$  of them as training data and  $30\%$  of them as testing data. We then do the following,
1. We fit a multi-output GP using the proposed Shapley prior on the training data and predict the explanations of the unseen test data.
2. We fit a multi-output random forest model on the training data and predict the explanations of the unseen test data.
3. We fit a multi-output neural network model on the training data and predict the explanations of the unseen test data.
We repeat this experiment 10 times using different seeds and compute the RMSE between the predicted and groundtruths explanations. The results are then plotted in Figure 3.
C.4 Further ablation study: Impact of increased posterior prediction uncertainty on explanation uncertainties
In this ablation study, we aim to examine the effect of increasing the uncertainty in posterior predictions on the corresponding uncertainty in stochastic Shapley values. To demonstrate this, we utilize the diabetic dataset [47] and split the data based on recorded sex. We train our GP model on the male data and employ BayesGP-SHAP to explain the prediction results for both the male training data and the female testing data. We adopt this split because we expect the biological characteristics between males and females to be distinct enough to treat the female data as out-of-sample data, thereby naturally resulting in increased predictive uncertainty for the female data. To further amplify this uncertainty, we multiply each instance in the female testing data by distortion factors of two and three, respectively, and assess the corresponding uncertainties in the explanations.
We begin by illustrating the relationship between the out-of-sampleness of the data and the corresponding increase in predictive posterior uncertainties. This is depicted in Figure 6a, where we observe that as the data becomes more out-of-sample, the predictive uncertainties consistently rise. Even at distortion level 1, which represents the original female data, we can already observe increased uncertainties compared to the uncertainties derived from male data prediction.
Furthermore, these increased uncertainties in the predictive posterior are reflected in the associated feature explanations. This is evident in Figure 6b, where we visualize the uncertainties associated with the feature explanations. For instance, the green bars representing the average uncertainties in explaining female data with no distortion are consistently larger than the red bars, which represent the average uncertainties of male data explanations. This observation aligns with the higher predictive uncertainties observed in Figure 6a for the female data compared to the male training data.
It is worth noting that the uncertainty for the feature "sex" remains consistently close to zero. This is because the feature "sex" is constant within both the female and male datasets. As a result, it acts as the null player in each dataset and obtains an almost Dirac zero as its stochastic Shapley value.
(a) Predictive posterior standard deviation
(b) Mean of standard deviations of explanations
Figure 6: Ablation study: (left) We begin by training a Gaussian Process (GP) model on the male data. We then make predictions using this trained model on both the male data and out-of-sample female data. To assess the impact of increasing posterior uncertainties, we multiply the female data by distortion levels of 1.0, 2.0, and 3.0. We visualize the results by plotting the density plot of the standard deviations obtained from the predictive posterior distributions. (right) Next, we focus on analyzing the average standard deviations of explanations per feature from the male and female data, considering different distortion levels. We observe that as we progressively increase the posterior uncertainties in the sample, these uncertainties are reflected in the uncertainties of the explanations provided.
Comparing FastSHAP with Shapley kernel approach over predicting test explanation
Figure 7: Comparing the predicted Shapley values with groundtruth from GP regression with Shapley kernel versus fastSHAP on the diabetic dataset.
Distribution of kendall tau score compared with groundtruth explanation
Figure 8: Comparing the Kendall's Tau score between the feature importance ranking from GP regression with Shapley kernel and FastSHAP with groundtruth feature importance ranking on the diabetic dataset.
D Comparison with FastSHAP [39]
We ran a predictive explanation experiment (see Figure 8 and 7) using the diabetic dataset from UCI, comparing both our method and FastSHAP. In particular, we use  $70\%$  of the data as training, i.e. we first compute the Shapley values for training data and treat them as labels, and predict the explanation for the remaining data. We evaluated performance using both RMSE and Kendall's tau distance (the closer to 1 the better). Notably, Kendall's tau distance assesses the consistency in feature importance ranking between predicted explanations and the true explanations. In the experiments, our method got RMSE 0.05, while FastSHAP got 0.09. In terms of Kendall's tau, our method got an average around 0.7 while FastSHAP got around 0.4, meaning the feature importance ranking recovered from our method is more aligned with the groundtruth SVs than FastSHAP. Our findings indicate that our method outperformed FastSHAP in this experiment.
Note however that interpretation of these comparisons should be nuanced since the two techniques have different usecases: FastSHAP uses knowledge of the model, not the previously computed SVs, whereas GPSHAP uses previously computed SVs and is agnostic to the type of model they came from.