SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In many applications, we must decompose total entropy into epistemic and aleatoric components—i.e., uncertainty arising from the model or the data, respectively. We achieve this via ensemble methods, using a set of  $B$  basis functions,  $\{f^1, \ldots, f^B\}$ . These may be decision trees, as in a random forest [68], or subsets of neural network nodes, as in Monte Carlo (MC) dropout [20]. Let  $f_y^b(\mathbf{x})$  be the conditional probability estimate for class  $y$  given sample  $\mathbf{x}$  for the  $b^{\text{th}}$  basis function. Then aleatoric uncertainty is given by  $h_a(\mathbf{x}) := -\frac{1}{B}\sum_{b=1}^{B}\sum_{y \in \mathcal{Y}}f_y^b(\mathbf{x})\log f_y^b(\mathbf{x})$ . Epistemic uncertainty is simply the difference [28],  $h_e(\mathbf{x}) := h_t(\mathbf{x}) - h_a(\mathbf{x})$ . Alternative methods may be appropriate for specific function classes, e.g. Gaussian processes [63] or Bayesian deep learning models [51]. We leave the choice of which uncertainty measure to explain up to practitioners. In what follows, we use the generic  $h(\mathbf{x})$  to signify whichever estimator is of relevance for a given application.
We are similarly ecumenical regarding reference distributions. This has been the subject of much debate in recent years, with authors variously arguing that  $\mathcal{D}$  should be a simple product of marginals [31]; or that the joint distribution should be modeled for proper conditioning and marginalization [1]; or else that structural information should be encoded to quantify causal effects [25]. Each approach makes sense in certain settings [8, 85], so we leave it up to practitioners to decide which is most appropriate for their use case. We stress that information theoretic games inherit all the advantages and disadvantages of these samplers from the conventional XAI setting, and acknowledge that attributions should be interpreted with caution when models are forced to extrapolate to off-manifold data [26]. Previous work has shown that no single sampling method dominates, with performance varying as a function of data type and function class [53]; see [9] for a discussion.
Finally, we adopt standard methods to efficiently sample candidate coalitions. Observe that the distribution on subsets implied by Eq. 1 induces a symmetric pmf on cardinalities  $|S| \in \{0, \dots, d - 1\}$  that places exponentially greater weight at the extrema than it does at the center. Thus while there are over 500 billion coalitions at  $d = 40$ , we can cover  $50\%$  of the total weight by sampling just over  $0.1\%$  of these subsets (i.e., those with cardinality  $\leq 9$  or  $\geq 30$ ). To reach  $90\%$  accuracy requires just over half of all coalitions. In fact, under some reasonable conditions, sampling  $\Theta(n)$  coalitions is asymptotically optimal, up to a constant factor [88]. We also employ the paired sampling approach of Covert and Lee [14] to reduce variance and speed up convergence still further.
Several authors have proposed inference procedures for Shapley values [88, 69, 93]. These methods could in principle be extended to our revised games. However, existing algorithms are typically either designed for local inference, in which case they are ill-suited to make global claims about feature relevance, or require global value functions upfront, unlike the local games we consider here. As an alternative, we describe a method for aggregating local statistics for global inference. Specifically, we test whether the random variable  $\phi(j, \mathbf{x})$  tends to concentrate around zero for a given  $j$ . We take a conformal approach [82, 41] that provides the following finite sample coverage guarantee.
Theorem 5.1 (Coverage). Partition  $n$  training samples  $\{(\mathbf{x}^{(i)},y^{(i)})\}_{i = 1}^{n}\sim \mathcal{D}$  into two equal-sized subsets  $\mathcal{I}_1,\mathcal{I}_2$  where  $\mathcal{I}_1$  is used for model fitting and  $\mathcal{I}_2$  for computing Shapley values. Fix
2The additive decomposition of total uncertainty into epistemic and aleatoric components has recently been challenged [89]. While alternative formulations are possible, we stick with this traditional view, which is widely used in deep ensembles and related methods for probabilistic ML [39, 56, 60].
a target level  $\alpha \in (0,1)$  and estimate the upper and lower bounds of the Shapley distribution from the empirical quantiles. That is, let  $\hat{q}_{lo}$  be the  $\ell$ th smallest value of  $\phi(j,\mathbf{x}^{(i)})$ ,  $i \in \mathcal{I}_2$ , for  $\ell = \lceil (n/2+1)(\alpha/2) \rceil$ , and let  $\hat{q}_{hi}$  be the  $u$ th smallest value of the same set, for  $u = \lceil (n/2+1)(1-\alpha/2) \rceil$ . Then for any test sample  $\mathbf{x}^{(n+1)} \sim \mathcal{D}$ , we have:
$$
\mathbb {P} \big (\phi (j, \mathbf {x} ^ {(n + 1)}) \in [ \hat {q} _ {l o}, \hat {q} _ {h i} ] \big) \geq 1 - \alpha .
$$
Moreover, if Shapley values have a continuous joint distribution, then the upper bound on this probability is  $1 - \alpha + 2 / (n + 2)$ .
Note that this is not a conditional coverage claim, insomuch as the bounds are fixed for a given  $X_{j}$  and do not vary with other feature values. However, Thm. 5.1 provides a PAC-style guarantee that Shapley values do not exceed a given (absolute) threshold with high probability, or that zero falls within the  $(1 - \alpha) \times 100\%$  confidence interval for a given Shapley value. These results can inform decisions about feature selection, since narrow intervals around zero are necessary (but not sufficient) evidence of uninformative predictors. This result is most relevant for tabular or text data, where features have some consistent meaning across samples; it is less applicable to image data, where individual pixels have no stable interpretation over images.
Full details of all datasets and hyperparameters can be found in Appx. C, along with supplemental experiments that did not fit in the main text. Code for all experiments and figures can be found in our dedicated GitHub repository.<sup>3</sup> We use DeepSHAP to sample out-of-coalition feature values in neural network models, and TreeSHAP for boosted ensembles. Alternative samplers are compared in a separate simulation experiment below. Since our goal is to explain predictive entropy rather than information, we use the value function  $v_{H^{*}}$  throughout, with plug-in estimators for total, epistemic, and/or aleatoric uncertainty.
6.1 Supervised Learning Examples
First, we perform a simple proof of concept experiment that illustrates the method's performance on image, text, and tabular data.
Image Data. We examine binary classifiers on subsets of the MNIST dataset. Specifically, we train deep convolutional neural nets to distinguish 1 vs. 7, 3 vs. 8, and 4 vs. 9. These digit pairs tend to look similar in many people's handwriting and are often mistaken for one another. We therefore expect relatively high uncertainty in these examples, and use a variant of DeepSHAP to visualize the pixel-wise contributions to predictive entropy, as estimated via MC dropout. We compute attributions for epistemic and aleatoric uncertainty, visually confirming that the former identifies regions of the image that most increase or reduce uncertainty (see Fig. 1A).
Applying our method, we find that epistemic uncertainty is reduced by the upper loop of the 9, as well as by the downward hook on the 7. By contrast, uncertainty is increased by the odd angle of the 8 and its small bottom loop. Aleatoric uncertainty, by contrast, is more mixed across the pixels, reflecting irreducible noise.
Text Data. We apply a transformer network to the IMDB dataset, which contains movie reviews for some 50,000 films. This is a sentiment analysis task, with the goal of identifying positive vs. negative reviews. We visualize the contribution of individual words to the uncertainty of particular predictions as calculated using the modified DeepSHAP pipeline, highlighting how some tokens tend to add or remove predictive information.
We report results for two high-entropy examples in Fig. 1B. In the first review, the model appears confused by the sentence "This is not Great Cinema but I was most entertained," which clearly conveys some ambiguity in the reviewer's sentiment. In the second example, the uncertainty comes from several sources including unexpected juxtapositions such as "laughing and crying", as well as "liar liar...you will love this movie."
<sup>3</sup>https://github.com/facebookresearch/infoshop.
Figure 1: A. MNIST examples. We highlight pixels that increase (red) and decrease (blue) predictive uncertainty in digit classification tasks (1 vs. 7, 3 vs. 8, and 4 vs. 9). B. Reviews from the IMDB dataset, with tokens colored by their relative contribution to the entropy of sentiment predictions.
Figure 2: A. Mean absolute error (MAE) as a function of sample size, with autocorrelation fixed at  $\rho = 0.5$ . B. MAE as a function of autocorrelation with sample size fixed at  $n = 2000$ . Shading represents standard errors across 50 replicates.
Tabular Data. We design a simple simulation experiment, loosely inspired by [1], in which Shapley values under the entropy game have a closed form solution (see Appx. C.1 for details). This allows us to compare various approaches for sampling out-of-coalition feature values. Variables  $\mathbf{X}$  are multivariate normally distributed with a Toeplitz covariance matrix  $\Sigma_{ij} = \rho^{|i - j|}$  and  $d = 4$  dimensions. The conditional distribution of outcome variable  $Y$  is Gaussian, with mean and variance depending on  $\mathbf{x}$ . We exhaustively enumerate all feature subsets at varying sample sizes and values of the autocorrelation parameter  $\rho$ .
For imputation schemes, we compare KernelSHAP, maximum likelihood, copula methods, empirical samplers, and ctree (see [54] for definitions of each, and a benchmark study of conditional sampling strategies for feature attributions). We note that this task is strictly more difficult than computing Shapley values under the standard  $v_{0}$ , since conditional variance must be estimated from the residuals of a preliminary model, itself estimated from the data. No single method dominates throughout, but most converge on the true Shapley value as sample size increases. Predictably, samplers that take conditional relationships into account tend to do better under autocorrelation than those that do not.
6.2 Covariate Shift and Active Learning
To illustrate the utility of our method for explaining covariate shift, we consider several semi-synthetic experiments. We start with four binary classification datasets from the UCI machine learning repository [17]—BreastCancer, Diabetes, Ionosphere, and Sonar—and make a random 80/20 train/test split on each. We use an XGBoost model [12] with 50 trees to estimate conditional probabilities and the associated uncertainty. We then perturb a random feature from the test set, adding a small amount of Gaussian noise to alter its underlying distribution. Resulting predictions have a large degree of entropy, and would therefore be ranked highly by an AL acquisition function. We compute information theoretic Shapley values for original and perturbed test sets. Results are visualized in Fig. 3.
Our method clearly identifies the source of uncertainty in these datapoints, assigning large positive or negative attributions to perturbed features in the test environment. Note that the distribution shifts
Figure 3: Information theoretic Shapley values explain the uncertainty of predictions on original and perturbed test sets. Our method correctly attributes the excess entropy to the perturbed features.
are fairly subtle in each case, rarely falling outside the support of training values for a given feature. Thus we find that information theoretic Shapley values can be used in conjunction with covariate shift detection algorithms to explain the source of the anomaly, or in conjunction with AL algorithms to explain the exploratory selection procedure.
6.3 Feature Selection
Another application of the method is as a feature selection tool when heteroskedasticity is driven by some but not all variables. For this experiment, we modify the classic Friedman benchmark [19], which was originally proposed to test the performance of nonlinear regression methods under signal sparsity. Outcomes are generated according to:
$$
Y = 1 0 \sin (\pi X _ {1} X _ {2}) + 2 0 (X _ {3} - 0. 5) ^ {2} + 1 0 X _ {4} + 5 X _ {5} + \epsilon_ {y},
$$
with input features  $\mathbf{X} \sim \mathcal{U}(0,1)^{10}$  and standard normal residuals  $\epsilon_y \sim \mathcal{N}(0,1^2)$ . To adapt this DGP to our setting, we scale  $Y$  to the unit interval and define:
$$
Z = 1 0 \sin (\pi X _ {6} X _ {7}) + 2 0 (X _ {8} - 0. 5) ^ {2} + 1 0 X _ {9} + 5 X _ {1 0} + \epsilon_ {z},
$$
with  $\epsilon_z\sim \mathcal{N}(0,\tilde{Y}^2)$ , where  $\tilde{Y}$  denotes the rescaled version of  $Y$ . Note that  $Z$ 's conditional variance depends exclusively on the first five features, while its conditional mean depends only on the second five. Thus with  $f(\mathbf{x}) = \mathbb{E}[Z\mid \mathbf{x}]$  and  $h(\mathbf{x}) = \mathbb{V}[Z\mid \mathbf{x}]$ , we should expect Shapley values for  $f$  to concentrate around zero for  $\{X_6,\ldots ,X_{10}\}$ , while Shapley values for  $h$  should do the same for  $\{X_1,\dots ,X_5\}$ .
We draw 2000 training samples and fit  $f$  using XGBoost with 100 trees. This provides estimates of both the conditional mean (via predictions) and the conditional variance (via observed residuals  $\hat{\epsilon}_y$ ). We fit a second XGBoost model  $h$  with the same hyperparameters to predict  $\log (\hat{\epsilon}_y^2)$ . Results are reported on a test set of size 1000. We compute attributions using TreeSHAP [45] and visualize results in Fig. 4A. We find that Shapley values are clustered around zero for unimportant features in each model, demonstrating the method's promise for discriminating between different modes of predictive information. In a supplemental experiment, we empirically evaluate our conformal coverage guarantee on this same task, achieving nominal coverage at  $\alpha = 0.1$  for all features (see Appx. C.3).
As an active feature-value acquisition example, we use the same modified Friedman benchmark, but this time increase the training sample size to 5000 and randomly delete some proportion of cells in the design matrix for  $\mathbf{X}$ . This simulates the effect of missing data, which may arise due to entry errors or high collection costs. XGBoost has native methods for handling missing data at training and test time, although resulting Shapley values are inevitably noisy. We refit the conditional variance estimator  $h$  and record feature rankings with variable missingness.
The goal in active feature-value acquisition is to prioritize the variables whose values will best inform future predictions subject to budgetary constraints. Fig. 4B shows receiver operating characteristic (ROC) curves for a feature importance ranking task as the frequency of missing data increases from zero to  $50\%$ . Importance is estimated via absolute Shapley values. Though performance degrades with increased missing data, as expected, we find that our method reliably ranks important features above unimportant ones in all trials. Even with fully half the data missing, we find an AUC of 0.682, substantially better than random.
Figure 4: A. Results for the modified Friedman benchmark experiment. The conditional mean depends on  $\{X_6, \ldots, X_{10}\}$ , while the conditional variance relies on  $\{X_1, \ldots, X_5\}$ . B. ROC curves for a feature ranking task with variable levels of missingness. The proposed value function gives informative results for feature-value acquisition.
Critics have long complained that Shapley values (using the conventional payoff function  $v_{0}$ ) are difficult to interpret in XAI. It is not always clear what it even means to remove features [43, 2], and large/small attributions are neither necessary nor sufficient for important/unimportant predictors, respectively [5, 29]. In an effort to ground these methods in classical statistical notions, several authors have analyzed Shapley values in the context of ANOVA decompositions [24, 6] or conditional independence tests [47, 78], with mixed results. Our information theoretic approach provides another window into this debate. With modified value functions, we show that marginal payoffs  $\Delta_v(S, j, \mathbf{x})$  have an unambiguous interpretation as a local dependence measure. Still, Shapley values muddy the waters somewhat by averaging these payoffs over coalitions.
There has been a great deal of interest in recent years on functional data analysis, where the goal is to model not just the conditional mean of the response variable  $\mathbb{E}[Y\mid \mathbf{x}]$ , but rather the entire distribution  $P(Y\mid \mathbf{x})$ , including higher moments. Distributional regression techniques have been developed for additive models [66], gradient boosting machines [79], random forests [27], and neural density estimators [58]. Few if any XAI methods have been specifically designed to explain such models, perhaps because attributions would be heavily weighted toward features with a significant impact on the conditional expectation, thereby simply reducing to classic measures. Our method provides one possible way to disentangle those attributions and focus attention on higher moments. Future work will explore more explicit connections to the domain of functional data.
One advantage of our approach is its modularity. We consider a range of different information theoretic games, each characterized by a unique value function. We are agnostic about how to estimate the relevant uncertainty measures, fix reference distributions, or sample candidate coalitions. These are all active areas of research in their own right, and practitioners should choose whichever combination of tools works best for their purpose.
However, this flexibility does not come for free. Computing Shapley values for many common function classes is #P-hard, even when features are jointly independent [81]. Modeling dependencies to impute values for out-of-coalition features is a statistical challenge that requires extensive marginalization. Some speedups can be achieved by making convenient assumptions, but these may incur substantial errors in practice. These are familiar problems in feature attribution tasks. Our method inherits the same benefits and drawbacks.
We introduced a range of methods to explain conditional entropy in ML models, bringing together existing work on uncertainty quantification and feature attributions. We studied the information theoretic properties of several games, and implemented our approach in model-specific and model-agnostic algorithms with numerous applications. Future work will continue to examine how XAI can go beyond its origins in prediction to inform decision making in areas requiring an exploration-exploitation trade-off, such as bandits and reinforcement learning.
We thank Matthew Shorvon for his feedback on an earlier draft of this paper. We are also grateful to the reviewers for their helpful comments.
[81] Guy Van den Broeck, Anton Lykov, Maximilian Schleich, and Dan Suciu. On the tractability of SHAP explanations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 2021.
[82] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World. Springer, New York, 2005.
[83] Erik Štrumbelj and Igor Kononenko. An efficient explanation of individual classifications using game theory. J. Mach. Learn. Res., 11:1-18, 2010. ISSN 1532-4435.
[84] Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. *Harvard J. Law Technol.*, 31(2):841-887, 2018.
[85] David S. Watson. Conceptual challenges for interpretable machine learning. Synthese, 200(2):65, 2022.
[86] David S Watson and Luciano Floridi. The explanation game: a formal framework for interpretable machine learning. Synthese, 198:9211-9242, 2021.
[87] David S. Watson, Limor Gultchin, Ankur Taly, and Luciano Floridi. Local explanations via necessity and sufficiency: Unifying theory and practice. *Minds Mach.*, 32(1):185-218, 2022.
[88] Brian Williamson and Jean Feng. Efficient nonparametric statistical inference on population feature importance using shapley values. In Proceedings of the 37th International Conference on Machine Learning, pages 10282-10291, 2020.
[89] Lisa Wimmer, Yusuf Sale, Paul Hofman, Bernd Bischl, and Eyke Hüllermeier. Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures? In Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, pages 2282-2292, 2023.
[90] Jinsung Yoon, James Jordan, and Mihaela van der Schaar. INVASE: Instance-wise variable selection using neural networks. In International Conference on Learning Representations, 2019.
[91] Jiji Zhang and Peter L. Spirtes. Strong faithfulness and uniform consistency in causal inference. In Proceedings of The 19th Conference on Uncertainty in Artificial Intelligence, 2003.
[92] Jiji Zhang and Peter L. Spirtes. The three faces of faithfulness. Synthese, 193(4):1011-1027, 2016.
[93] Xingyu Zhao, Wei Huang, Xiaowei Huang, Valentin Robu, and David Flynn. Baylime: Bayesian local interpretable model-agnostic explanations. In Proceedings of the 37th Conference on Uncertainty in Artificial Intelligence, pages 887–896, 2021.
[94] Indre Žliobaite. Change with delayed labeling: When is it detectable? In 2010 IEEE International Conference on Data Mining Workshops, pages 843-850. IEEE, 2010.
Proof of Prop. 4.1. Substituting  $v_{KL}$  into the definition of  $\Delta_v(S, j, \mathbf{x})$  gives:
$$
\Delta_ {K L} (S, j, \mathbf {x}) = - D _ {K L} \left(p _ {Y | \mathbf {x}} \| p _ {Y | \mathbf {x} _ {S}, x _ {j}}\right) + D _ {K L} \left(p _ {Y | \mathbf {x}} \| p _ {Y | \mathbf {x} _ {S}}\right).
$$
Rearranging and using the definition of KL-divergence, we have:
$$
\Delta_ {K L} (S, j, \mathbf {x}) = \underset {Y | \mathbf {x}} {\mathbb {E}} \left[ \log p (y \mid \mathbf {x}) - \log p (y \mid \mathbf {x} _ {S}) \right] - \underset {Y | \mathbf {x}} {\mathbb {E}} \left[ \log p (y \mid \mathbf {x}) - \log p (y \mid \mathbf {x} _ {S}, x _ {j}) \right].
$$
Cleaning up in steps:
$$
\begin{array}{l} \Delta_ {K L} (S, j, \mathbf {x}) = \underset {Y | \mathbf {x}} {\mathbb {E}} \left[ \log p (y \mid \mathbf {x}) - \log p (y \mid \mathbf {x} _ {S}) - \log p (y \mid \mathbf {x}) + \log p (y \mid \mathbf {x} _ {S}, x _ {j}) \right] \\ = \underset {Y | \mathbf {x}} {\mathbb {E}} \left[ \log p (y \mid \mathbf {x} _ {S}, x _ {j}) - \log p (y \mid \mathbf {x} _ {S}) \right] \\ = \int_ {\mathcal {Y}} p (y \mid \mathbf {x}) \log \frac {p (y \mid \mathbf {x} _ {S} , x _ {j})}{p (y \mid \mathbf {x} _ {S})} d y. \\ \end{array}
$$
Substituting  $v_{CE}$  into the definition of  $\Delta_v(S, j, \mathbf{x})$  gives:
$$
\Delta_ {C E} (S, j, \mathbf {x}) = - H \left(p _ {Y | \mathbf {x}}, p _ {Y | \mathbf {x} _ {S}, x _ {j}}\right) + H \left(p _ {Y | \mathbf {x}}, p _ {Y | \mathbf {x} _ {S}}\right).
$$
Rearranging and using the definition of cross entropy, we have:
$$
\begin{array}{l} \Delta_ {C E} (S, j, \mathbf {x}) = H \left(p _ {Y | \mathbf {x}}, p _ {Y | \mathbf {x} _ {S}}\right) - H \left(p _ {Y | \mathbf {x}}, p _ {Y | \mathbf {x} _ {S} \cup \{j \}}\right) \\ = \underset {Y | \mathbf {x}} {\mathbb {E}} \left[ - \log p (y \mid \mathbf {x} _ {S}) \right] - \underset {Y | \mathbf {x}} {\mathbb {E}} \left[ - \log p (y \mid \mathbf {x} _ {S}, x _ {j}) \right] \\ = \underset {Y | \mathbf {x}} {\mathbb {E}} \left[ \log p (y \mid \mathbf {x} _ {S}, x _ {j}) - \log p (y \mid \mathbf {x} _ {S}) \right] \\ = \int_ {\mathcal {Y}} p (y \mid \mathbf {x}) \log \frac {p (y \mid \mathbf {x} _ {S} , x _ {j})}{p (y \mid \mathbf {x} _ {S})} d y. \\ \end{array}
$$