SlowGuess/NeurIPS_2023_OCR huggingface dataset:
David S. Watson
david.watson@kcl.ac.uk
Joshua O'Hara
Niek Tax
Meta, Central Applied Science
Richard Mudd
Ido Guy
Researchers in explainable artificial intelligence have developed numerous methods for helping users understand the predictions of complex supervised learning models. By contrast, explaining the uncertainty of model outputs has received relatively little attention. We adapt the popular Shapley value framework to explain various types of predictive uncertainty, quantifying each feature's contribution to the conditional entropy of individual model outputs. We consider games with modified characteristic functions and find deep connections between the resulting Shapley values and fundamental quantities from information theory and conditional independence testing. We outline inference procedures for finite sample error rate control with provable guarantees, and implement efficient algorithms that perform well in a range of experiments on real and simulated data. Our method has applications to covariate shift detection, active learning, feature selection, and active feature-value acquisition.
Machine learning (ML) algorithms can solve many prediction tasks with greater accuracy than classical methods. However, some of the most popular and successful algorithms, such as deep neural networks, often produce models with millions of parameters and complex nonlinearities. The resulting "black box" is essentially unintelligible to humans. Researchers in explainable artificial intelligence (XAI) have developed numerous methods to help users better understand the inner workings of such models (see Sect. 2).
Despite the rapid proliferation of XAI tools, the goals of the field have thus far been somewhat narrow. The vast majority of methods in use today aim to explain model predictions, i.e. point estimates. But these are not necessarily the only model output of interest. Predictive uncertainty can also vary widely across the feature space, in ways that may have a major impact on model performance and human decision making. Such variation makes it risky to rely on the advice of a black box, especially when generalizing to new environments. Discovering the source of uncertainty can be an important first step toward reducing it.
Quantifying predictive uncertainty has many applications in ML. For instance, it is an essential subroutine in any task that involves exploration, e.g. active learning [18, 36], multi-armed bandits [73, 40], and reinforcement learning more generally [55, 76]. Other applications of predictive
uncertainty quantification include detecting covariate shift [77] and adversarial examples [70], as well as classification with reject option [23]. Our method aims to expand the scope of XAI to these varied domains by explaining predictive uncertainty via feature attributions.
Knowing the impact of individual features on local uncertainty can help drive data collection and model design. It can be used to detect the source of a suspected covariate shift, select informative features, and test for heteroskedasticity. Our attribution strategy makes use of the Shapley value framework for XAI [83, 44, 74, 49, 9], a popular approach inspired by cooperative game theory, which we adapt by altering the characteristic function and augment with inference procedures for provable error rate control. The approach is fully model-agnostic and therefore not limited to any particular function class.
Our main contributions are threefold: (1) We describe modified variants of the Shapley value algorithm that can explain higher moments of the predictive distribution, thereby extending its explanatory utility beyond mere point estimates. We provide an information theoretic interpretation of the resulting measures and study their properties. (2) We introduce a split conformal inference procedure for Shapley variables with finite sample coverage guarantees. This allows users to test the extent to which attributions for a given feature are concentrated around zero with fixed type I error control. (3) We implement model-specific and model-agnostic variants of our method and illustrate their performance in a range of simulated and real-world experiments, with applications to feature selection, covariate shift detection, and active learning.
XAI has become a major subfield of machine learning in recent years. The focus to date has overwhelmingly been on explaining predictions in supervised learning tasks, most prominently via feature attributions [64, 44, 75], rule lists [65, 38, 71], and counterfactuals [84, 50, 34]. Despite obvious differences between these methods, all arguably share the same goal of identifying minimal conditions sufficient to alter predictions in some pre-specified way [87]. Resulting explanations should be accurate, simple, and relevant for the inquiring agent [86].
Quantifying inductive uncertainty is a fundamental problem in probability theory and statistics, although machine learning poses new challenges and opportunities in this regard [30]. The classical literature on this topic comes primarily from Bayesian modeling [21] and information theory [13], which provide a range of methods for analyzing the distribution of random variables. More recent work on conformal inference [82, 41, 4] has expanded the toolkit for practitioners.
Important application areas for these methods include active learning (AL) and covariate shift detection. In AL, the goal is to selectively query labels for unlabeled instances aimed to maximize classifier improvement under a given query budget. Methods often select instances on which the model has high epistemic (as opposed to aleatoric) uncertainty [67], as for example in BatchBALD [36]. This is especially valuable when labels are sparse and costly to collect while unlabeled data is widely available. In covariate shift detection, the goal is to identify samples that are abnormal relative to the in-distribution observations that the classifier has seen during training. It is well-known that neural networks can be overconfident [22], yielding predictions with unjustifiably high levels of certainty on test samples. Addressing this issue is an active area of research, and a variety of articles take a perspective of quantifying epistemic uncertainty [30, 56]. In safety-critical applications, the degree of model uncertainty can be factored into the decision making, for example by abstaining from prediction altogether when confidence is sufficiently low [23].
Very little work has been done on explaining predictive uncertainty. A notable exception is the CLUE algorithm [3, 42], a model-specific method designed for Bayesian deep learning, which generates counterfactual samples that are maximally similar to some target observation but optimized for minimal conditional variance. This contrasts with our feature attribution approach, which is model-agnostic and thereby the first to make uncertainty explanations available to function classes beyond Bayesian deep learning models.
Predictive uncertainty is often correlated with prediction loss, and therefore explanations of model errors are close relatives of our method. LossSHAP [45] is an extension of Lundberg and Lee [44]'s SHAP algorithm designed to explain the pointwise loss of a supervised learner (e.g., squared error or cross entropy). Though this could plausibly help identify regions where the model is least certain
about predictions, it requires a large labelled test dataset, which may not be available in practice. By contrast, our method only assumes access to some unlabelled dataset of test samples, which is especially valuable when labels are slow or expensive to collect. For instance, LossSHAP is little help in learning environments where covariate shift is detectable before labels are known [94]. This is common, for example, in online advertising [37], where an impression today may lead to a conversion next week but quick detection (and explanation) of covariate shift is vital.
Previous authors have explored information theoretic interpretations of variable importance measures; see [16, Sect. 8.3] for a summary. These methods often operate at global resolutions—e.g., Sobol' indices [57] and SAGE [15]—whereas our focus is on local explanations. Alternatives such as INVASE [90] must be trained alongside the supervised learner itself and are therefore not model-agnostic. L2X [10], REAL-X [32], and SHAP-KL [33] provide post-hoc local explanations, but they require surrogate models to approximate a joint distribution over the full feature space. Chen et al. [11] propose an information theoretic variant of Shapley values for graph-structured data, which we examine more closely in Sect. 4.
Notation. We use uppercase letters to denote random variables (e.g.,  $X$ ) and lowercase for their values (e.g.,  $x$ ). Matrices and sets of random variables are denoted by uppercase boldface type (e.g.,  $\mathbf{X}$ ) and vectors by lowercase boldface (e.g.,  $\mathbf{x}$ ). We occasionally use superscripts to denote samples, e.g.  $\mathbf{x}^{(i)}$  is the  $i^{\text{th}}$  row of  $\mathbf{X}$ . Subscripts index features or subsets thereof, e.g.  $\mathbf{X}_S = \{X_j\}_{j \in S}$  and  $\mathbf{x}_S^{(i)} = \{x_j^{(i)}\}_{j \in S}$ , where  $S \subseteq [d] = \{1, \dots, d\}$ . We define the complementary subset  $\overline{S} = [d] \setminus S$ .
Information Theory. Let  $p, q$  be two probability distributions over the same  $\sigma$ -algebra of events. Further, let  $p, q$  be absolutely continuous with respect to some appropriate measure. We make use of several fundamental quantities from information theory [13], such as entropy  $H(p)$ , cross entropy  $H(p, q)$ , KL-divergence  $D_{KL}(p \parallel q)$ , and mutual information  $I(X; Y)$  (all formally defined in Appx. B.1). We use shorthand for the (conditional) probability mass/density function of the random variable  $Y$ , e.g.  $p_{Y|\mathbf{x}_S} := p(Y \mid \mathbf{X}_S = \mathbf{x}_S)$ . We speak interchangeably of the entropy of a random variable and the entropy of the associated mass/density function:  $H(Y \mid x) = H(p_{Y|x})$ . We call this the local conditional entropy to distinguish it from its global counterpart,  $H(Y \mid X)$ , which requires marginalization over the joint space  $\mathcal{X} \times \mathcal{Y}$ .
Shapley Values. Consider a supervised learning model  $f$  trained on features  $\mathbf{X} \in \mathcal{X} \subseteq \mathbb{R}^d$  to predict outcomes  $Y \in \mathcal{Y} \subseteq \mathbb{R}$ . We assume that data are distributed according to some fixed but unknown distribution  $\mathcal{D}$ . Shapley values are a feature attribution method in which model predictions are decomposed as a sum:  $f(\mathbf{x}) = \phi_0 + \sum_{j=1}^{d} \phi(j, \mathbf{x})$ , where  $\phi_0$  is the baseline expectation (i.e.,  $\phi_0 = \mathbb{E}_{\mathcal{D}}[f(\mathbf{x})]$ ) and  $\phi(j, \mathbf{x})$  denotes the Shapley value of feature  $j$  at point  $\mathbf{x}$ . To define this quantity, we require a value function  $v: 2^{[d]} \times \mathbb{R}^d \mapsto \mathbb{R}$  that quantifies the payoff associated with subsets  $S \subseteq [d]$  for a particular sample. This characterizes a cooperative game, in which each feature acts as a player. A common choice for defining payoffs in XAI is the following [83, 44, 74, 49, 9]:
$$
v _ {0} (S, \mathbf {x}) := \mathbb {E} _ {\mathcal {D}} \left[ f (\mathbf {x}) \mid \mathbf {X} _ {S} = \mathbf {x} _ {S} \right],
$$
where we marginalize over the complementary features  $\overline{S}$  in accordance with reference distribution  $\mathcal{D}$ . For any value function  $v$ , we may define the following random variable to represent  $j$ 's marginal contribution to coalition  $S$  at point  $\mathbf{x}$ :
$$
\Delta_ {v} (S, j, \mathbf {x}) := v (S \cup \{j \}, \mathbf {x}) - v (S, \mathbf {x}).
$$
Then  $j$ 's Shapley value is just the weighted mean of this variable over all subsets:
$$
\phi_ {v} (j, \mathbf {x}) := \sum_ {S \subseteq [ d ] \backslash \{j \}} \frac {| S | ! (d - | S | - 1) !}{d !} \left[ \Delta_ {v} (S, j, \mathbf {x}) \right]. \tag {1}
$$
It is well known that Eq. 1 is the unique solution to the attribution problem that satisfies certain desirable properties, including efficiency, symmetry, sensitivity, and linearity [74] (for formal statements of these axioms, see Appx. B.2.)
4 Alternative Value Functions
To see how standard Shapley values can fall short, consider a simple data generating process with  $X, Z \sim \mathcal{U}(0,1)^2$  and  $Y \sim \mathcal{N}(X,Z^2)$ . Since the true conditional expectation of  $Y$  is  $X$ , this feature will get  $100\%$  of the attributions in a game with payoffs given by  $v_0$ . However, just because  $Z$  receives zero attribution does not mean that it adds no information to our predictions—on the contrary, we can use  $Z$  to infer the predictive variance of  $Y$  and calibrate confidence intervals accordingly. This sort of higher order information is lost in the vast majority of XAI methods.
We consider information theoretic games that assign nonzero attribution to  $Z$  in the example above, and study the properties of resulting Shapley values. We start in an idealized scenario in which we have: (i) oracle knowledge of the joint distribution  $\mathcal{D}$ ; and (ii) unlimited computational budget, thereby allowing complete enumeration of all feature subsets.
INVASE [90] is a method for learning a relatively small but maximally informative subset of features  $S \subset [d]$  using the loss function  $D_{KL}(p_{Y|\mathbf{x}} \| p_{Y|\mathbf{x}S}) + \lambda |S|$ , where  $\lambda$  is a regularization penalty. Jethani et al. [33]'s SHAP-KL adapts this loss function to define a new game:
$$
v _ {K L} (S, \mathbf {x}) := - D _ {K L} \left(p _ {Y | \mathbf {x}} \| p _ {Y | \mathbf {x} _ {S}}\right),
$$
which can be interpreted as  $-1$  times the excess number of bits one would need on average to describe samples from  $Y\mid \mathbf{x}$  given code optimized for  $Y\mid \mathbf{x}_S$ .
Chen et al. [11] make a similar proposal, replacing KL-divergence with cross entropy:
$$
v _ {C E} (S, \mathbf {x}) := - H \left(p _ {Y | \mathbf {x}}, p _ {Y | \mathbf {x} _ {S}}\right).
$$
This value function is closely related to that of LossSHAP [45], which for likelihood-based loss functions can be written:
$$
v _ {L} (S, \mathbf {x}) := - \log p (Y = y \mid \mathbf {x} _ {S}),
$$
where  $y$  denotes the true value of  $Y$  at the point  $\mathbf{x}$ . As Covert et al. [16] point out, this is equivalent to the pointwise mutual information  $I(y;\mathbf{x}_S)$ , up to an additive constant. However,  $v_{L}$  requires true labels for  $Y$ , which may not be available when evaluating feature attributions on a test set. By contrast,  $v_{CE}$  averages over  $\mathcal{V}$ , thereby avoiding this issue:  $v_{CE}(S,\mathbf{x}) = -\mathbb{E}_{Y|\mathbf{x}}[v_L(S,\mathbf{x})]$ . We reiterate that in all cases we condition on some fixed value of  $\mathbf{x}$  and do not marginalize over the feature space  $\mathcal{X}$ . This contrasts with global feature attribution methods like SAGE [15], which can be characterized by averaging  $v_{L}$  over the complete joint distribution  $p(\mathbf{X},Y)$ .
It is evident from the definitions that  $v_{KL}$  and  $v_{CE}$  are equivalent up to an additive constant not depending on  $S$ , namely  $H(p_{Y|\mathbf{x}})$ . This renders the resulting Shapley values from both games identical (all proofs in Appx. A.)<sup>1</sup>
Proposition 4.1. For all features  $j \in [d]$ , coalitions  $S \subseteq [d] \backslash \{j\}$ , and samples  $\mathbf{x} \sim \mathcal{D}_X$ :
$$
\begin{array}{l} \Delta_ {K L} (S, j, \mathbf {x}) = \Delta_ {C E} (S, j, \mathbf {x}) \\ = \int_ {\mathcal {Y}} p (y \mid \mathbf {x}) \log \frac {p (y \mid \mathbf {x} _ {S} , x _ {j})}{p (y \mid \mathbf {x} _ {S})} d y. \\ \end{array}
$$
This quantity answers the question: if the target distribution were  $p_{Y|\mathbf{x}}$ , how many more bits of information would we get on average by adding  $x_{j}$  to the conditioning event  $\mathbf{x}_S$ ? Resulting Shapley values summarize each feature's contribution in bits to the distance between  $Y$ 's fully specified local posterior distribution  $p(Y\mid \mathbf{x})$  and the prior  $p(Y)$ .
Proposition 4.2. With  $v \in \{v_{KL}, v_{CE}\}$ , Shapley values satisfy  $\sum_{j=1}^{d} \phi_v(j, \mathbf{x}) = D_{KL}(p_{Y|\mathbf{x}} \| p_Y)$ .
We introduce two novel information theoretic games, characterized by negative and positive local conditional entropies:
$$
v _ {I G} (S, \mathbf {x}) := - H (Y \mid \mathbf {x} _ {S}), \quad v _ {H} (S, \mathbf {x}) := H (Y \mid \mathbf {x} _ {S}).
$$
<sup>1</sup>All propositions in this section can be adapted to the classification setting by replacing the integral with a summation over labels  $\mathcal{V}$ .
The former subscript stands for information gain; the latter for entropy. Much like  $v_{CE}$ , these value functions can be understood as weighted averages of LossSHAP payoffs over  $\mathcal{V}$ , however this time with expectation over a slightly different distribution:  $v_{IG}(S, \mathbf{x}) = -v_{H}(S, \mathbf{x}) = -\mathbb{E}_{Y|\mathbf{x}_S}\big[v_L(S, \mathbf{x})\big]$ . The marginal contribution of feature  $j$  to coalition  $S$  is measured in bits of local conditional mutual information added or lost, respectively (note that  $\Delta_{IG} = -\Delta_H$ ).
Proposition 4.3. For all features  $j \in [d]$ , coalitions  $S \subseteq [d] \backslash \{j\}$ , and samples  $\mathbf{x} \sim \mathcal{D}_X$ :
$$
\Delta_ {I G} (S, j, \mathbf {x}) = I (Y; x _ {j} \mid \mathbf {x} _ {S}).
$$
This represents the decrease in  $Y$ 's uncertainty attributable to the conditioning event  $x_{j}$  when we already know  $\mathbf{x}_S$ . This quantity is similar (but not quite equivalent) to the information gain, a common optimization objective in tree growing algorithms [61, 62]. The difference again lies in the fact that we do not marginalize over  $\mathcal{X}$ , but instead condition on a single instance. Resulting Shapley values summarize each feature's contribution in bits to the overall local information gain.
Proposition 4.4. Under  $v_{IG}$ , Shapley values satisfy  $\sum_{j=1}^{d} \phi_{IG}(j, \mathbf{x}) = I(Y; \mathbf{x})$ .
In the classic game  $v_{0}$ , out-of-coalition features are eliminated by marginalization. However, this will not generally work in our information theoretic games. Consider the modified entropy game, designed to take  $d$ -dimensional input:
$$
v _ {H ^ {*}} (S, \mathbf {x}) := \mathbb {E} _ {\mathcal {D}} \left[ H \left(p _ {Y | \mathbf {x}}\right) \mid \mathbf {X} _ {S} = \mathbf {x} _ {S} \right].
$$
This game is not equivalent to  $v_{H}$ , as shown in the following proposition.
Proposition 4.5. For all coalitions  $S\subset [d]$  and samples  $\mathbf{x}\sim \mathcal{D}_X$  ..
$$
v _ {H} (S, \mathbf {x}) - v _ {H ^ {*}} (S, \mathbf {x}) = D _ {K L} \left(p _ {Y | \mathbf {X} _ {\overline {{S}}}, \mathbf {x} _ {S}} \| p _ {Y | \mathbf {x} _ {S}}\right).
$$
The two value functions will tend to diverge when out-of-coalition features  $\mathbf{X}_{\overline{S}}$  inform our predictions about  $Y$ , given prior knowledge of  $\mathbf{x}_S$ . Resulting Shapley values represent the difference in bits between the local and global conditional entropy.
Proposition 4.6. Under  $v_{H^*}$ , Shapley values satisfy  $\sum_{j=1}^{d} \phi_{H^*}(j, \mathbf{x}) = H(Y \mid \mathbf{x}) - H(Y \mid \mathbf{X})$ .
In other words,  $\phi_{H^{*}}(j,\mathbf{x})$  is  $j$ 's contribution to conditional entropy at a given point, compared to a global baseline that averages over all points.
These games share an important and complex relationship to conditional independence structures. We distinguish here between global claims of conditional independence, e.g.  $Y \perp X \mid Z$ , and local or context-specific independence (CSI), e.g.  $Y \perp X \mid z$ . The latter occurs when  $X$  adds no information about  $Y$  under the conditioning event  $Z = z$  [7] (see Appx. B.1 for an example).
Theorem 4.7. For value functions  $v \in \{v_{KL}, v_{CE}, v_{IG}, v_{H}\}$ , we have:
(a)  $Y\perp X_{j}\mid \mathbf{X}_{S}\Leftrightarrow \sup_{\mathbf{x}\in \mathcal{X}}|\Delta_{v}(S,j,\mathbf{x})| = 0.$
(b)  $Y\perp X_{j}\mid \mathbf{x}_{S}\Rightarrow \Delta_{v}(S,j,\mathbf{x}) = 0.$
(c) The set of distributions such that  $\Delta_v(S, j, \mathbf{x}) = 0 \land Y \nsubseteq X_j \mid \mathbf{x}_S$  is Lebesgue measure zero.
Item (a) states that  $Y$  is conditionally independent of  $X_{j}$  given  $\mathbf{X}_S$  if and only if  $j$  makes no contribution to  $S$  at any point  $\mathbf{x}$ . Item (b) states that the weaker condition of CSI is sufficient for zero marginal payout. However, while the converse does not hold in general, item (c) states that the set of counterexamples is small in a precise sense—namely, it has Lebesgue measure zero. A similar result holds for so-called unfaithful distributions in causality [72, 92], in which positive and negative effects cancel out exactly, making it impossible to detect certain graphical structures. Similarly, context-specific dependencies may be obscured when positive and negative log likelihood ratios cancel out as we marginalize over  $\mathcal{V}$ . Measure zero events are not necessarily harmless, especially when working with finite samples. Near violations may in fact be quite common due to statistical noise [80]. Together, these results establish a powerful, somewhat subtle link between conditional independencies and information theoretic Shapley values. Similar results are lacking for the standard value function  $v_{0}$ —with the notable exception that conditional independence implies zero marginal payout [46]—an inevitable byproduct of the failure to account for predictive uncertainty.
5 Method
The information theoretic quantities described in the previous section are often challenging to calculate, as they require extensive conditioning and marginalization. Computing some  $\mathcal{O}(2^d)$  such quantities per Shapley value, as Eq. 1 requires, quickly becomes infeasible. (See [81] for an in-depth analysis of the time complexity of Shapley value algorithms.) Therefore, we make several simplifying assumptions that strike a balance between computational tractability and error rate control.
First, we require some uncertainty estimator  $h: \mathcal{X} \mapsto \mathbb{R}_{\geq 0}$ . Alternatively, we could train new estimators for each coalition [88]; however, this can be impractical for large datasets and/or complex function classes. In the previous section, we assumed access to the true data generating process. In practice, we must train on finite samples, often using outputs from the base model  $f$ . In the regression setting, this may be a conditional variance estimator, as in heteroskedastic error models [35]; in the classification setting, we assume that  $f$  outputs a pmf over class labels and write  $f_{y}: \mathbb{R}^{d} \mapsto [0,1]$  to denote the predicted probability of class  $y \in \mathcal{V}$ . Then predictive entropy is estimated via the plug-in formula  $h_{t}(\mathbf{x}) := -\sum_{y \in \mathcal{Y}} f_{y}(\mathbf{x}) \log f_{y}(\mathbf{x})$ , where the subscript  $t$  stands for total.