SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof of Prop. 4.2. Since the Shapley value  $\phi_v(j, \mathbf{x})$  is just the expectation of  $\Delta_v(S, j, \mathbf{x})$  under a certain distribution on coalitions  $S \subseteq [d] \setminus \{j\}$  (see Eq. 1), it follows from Prop. 4.1 that feature attributions will be identical under  $v_{KL}$  and  $v_{CE}$ . To show that resulting Shapley values sum to the KL-divergence between  $p(Y \mid \mathbf{x})$  and  $p(Y)$ , we exploit the efficiency property:
$$
\begin{array}{l} \sum_ {j = 1} ^ {d} \phi_ {K L} (j, \mathbf {x}) = v _ {K L} ([ d ], \mathbf {x}) - v _ {K L} (\emptyset , \mathbf {x}) \\ = - D _ {K L} \left(p _ {Y | \mathbf {x}} \| p _ {Y | \mathbf {x}}\right) + D _ {K L} \left(p _ {Y | \mathbf {x}} \| p _ {Y}\right) \\ = D _ {K L} \left(p _ {Y | \mathbf {x}} \| p _ {Y}\right). \\ \end{array}
$$
The last step exploits Gibbs's inequality, according to which  $D_{KL}(p\parallel q)\geq 0$ , with  $D_{KL}(p\parallel q) = 0$  iff  $p = q$ .
Proof of Prop. 4.3. Substituting  $v_{IG}$  into the definition of  $\Delta_v(S, j, \mathbf{x})$  gives:
$$
\begin{array}{l} \Delta_ {I G} (S, j, \mathbf {x}) = - H (Y \mid \mathbf {x} _ {S}, x _ {j}) + H (Y \mid \mathbf {x} _ {S}) \\ = H (Y \mid \mathbf {x} _ {S}) - H (Y \mid \mathbf {x} _ {S}, x _ {j}) \\ = I (Y; x _ {j} \mid \mathbf {x} _ {S}) \\ = \int_ {\mathcal {Y}} p \left(y, x _ {j} \mid \mathbf {x} _ {S}\right) \log \frac {p \left(y , x _ {j} \mid \mathbf {x} _ {S}\right)}{p \left(y \mid \mathbf {x} _ {S}\right) p \left(x _ {j} \mid \mathbf {x} _ {S}\right)} d y. \\ \end{array}
$$
In the penultimate line, we exploit the equality  $I(Y;X) = H(Y) - H(Y\mid X)$ , by which we define mutual information (see Appx. B.1).
Proof of Prop. 4.4. We once again rely on efficiency and the definition of mutual information in terms of marginal and conditional entropy:
$$
\begin{array}{l} \sum_ {j = 1} ^ {d} \phi_ {I G} (j, \mathbf {x}) = v _ {I G} ([ d ], \mathbf {x}) - v _ {I G} (\emptyset , \mathbf {x}) \\ = - H (Y \mid \mathbf {x}) + H (Y) \\ = H (Y) - H (Y \mid \mathbf {x}) \\ = I (Y; \mathbf {x}). \\ \end{array}
$$
Proof of Prop. 4.5. Let  $b(S, \mathbf{x})$  denote the gap between local conditional entropy formulae that take  $|S|$ - and  $d$ -dimensional input, respectively. Then we have:
$$
\begin{array}{l} b (S, \mathbf {x}) = v _ {H} (S, \mathbf {x}) - v _ {H ^ {*}} (S, \mathbf {x}) \\ = H (Y \mid \mathbf {x} _ {S}) - \underset {\mathbf {X} _ {S} \mid \mathbf {x} _ {S}} {\mathbb {E}} \left[ H (Y \mid \mathbf {x}) \mid \mathbf {X} _ {S} = \mathbf {x} _ {S} \right] \\ = - \underset {Y | \mathbf {x} _ {S}} {\mathbb {E}} \left[ \log p (y \mid \mathbf {x} _ {S}) \right] + \underset {\mathbf {x} _ {\overline {{S}}}, Y | \mathbf {x} _ {S}} {\mathbb {E}} \left[ \log p (y \mid \mathbf {x} _ {\overline {{S}}}, \mathbf {x} _ {S}) \right] \\ = \underset {\mathbf {x} _ {\overline {{S}}}, Y | \mathbf {x} _ {S}} {\mathbb {E}} \left[ \log p (y \mid \mathbf {x} _ {\overline {{S}}}, \mathbf {x} _ {S}) - \log p (y \mid \mathbf {x} _ {S}) \right] \\ = \int_ {\mathcal {X} _ {\overline {{S}}}} \int_ {\mathcal {Y}} p (\mathbf {x} _ {\overline {{S}}}, y \mid \mathbf {x} _ {S}) \log \frac {p (y \mid \mathbf {x} _ {\overline {{S}} , \mathbf {x} _ {S}})}{p (y \mid \mathbf {x} _ {S})} d \mathbf {x} _ {\overline {{S}}} d y \\ = D _ {K L} \left(p \left(Y \mid \mathbf {X} _ {\bar {S}}, \mathbf {x} _ {S}\right) \| p (Y \mid \mathbf {x} _ {S})\right). \\ \end{array}
$$
Proof of Prop. 4.6. Exploiting the efficiency property, we immediately have:
$$
\begin{array}{l} \sum_ {j = 1} ^ {d} \phi_ {H ^ {*}} (j, \mathbf {x}) = v _ {H ^ {*}} ([ d ], \mathbf {x}) - v _ {H ^ {*}} (\emptyset , \mathbf {x}) \\ = H (Y \mid \mathbf {x}) - \underset {\mathcal {D} _ {X}} {\mathbb {E}} \left[ H (Y \mid \mathbf {x}) \right] \\ = H (Y \mid \mathbf {x}) - H (Y \mid \mathbf {X}). \\ \end{array}
$$
By contrast, Shapley values for the original entropy game  $v_{H}$  sum to  $H(Y \mid \mathbf{x}) - H(Y) = -I(Y; \mathbf{x})$ . Thus whereas the baseline for an empty coalition  $S = \emptyset$  is the prior entropy  $H(Y)$  under  $v_{H}$ , the corresponding baseline for the  $d$ -dimensional version  $v_{H^{*}}$  is the global posterior entropy  $H(Y \mid \mathbf{X})$ .
Proof of Thm. 4.7. Begin with item (a). Note that the conditional independence statement  $Y \perp X_j \mid \mathbf{X}_S$  holds iff, for all points  $(\mathbf{x}, y) \sim \mathcal{D}$ , we have:
$$
p (y \mid \mathbf {x} _ {S}, x _ {j}) = p (y \mid \mathbf {x} _ {S}) \quad \text {a n d} \quad p (y, x _ {j} \mid \mathbf {x} _ {S}) = p (y \mid \mathbf {x} _ {S}) p (x _ {j} \mid \mathbf {x} _ {S}).
$$
The former guarantees that marginal payouts evaluate to zero for  $v \in \{v_{KL}, v_{CE}\}$ ; the latter does the same for  $v \in \{v_{IG}, v_{H}\}$ . This follows because the log ratio in each formula evaluates to zero when numerator and denominator are equal.
Of course, conditional independence is also sufficient for zero marginal payout with more familiar value functions such as  $v_{0}$ . But item (a) makes an additional claim—that the converse holds as well, i.e. that conditional independence is necessary for zero marginal payout across all  $\mathbf{x}$ . This follows from the definitions of the value functions themselves. Observe:
$$
\begin{array}{l} \underset {\mathbf {x} \sim \mathcal {D} _ {X}} {\mathbb {E}} \left[ \Delta_ {K L} (S, j, \mathbf {x}) \right] = \underset {(\mathbf {x}, y) \sim \mathcal {D}} {\mathbb {E}} \left[ \log \frac {p (y \mid \mathbf {x} _ {S} , x _ {j})}{p (y \mid \mathbf {x} _ {S})} \right] \\ = \underset {\mathcal {D} _ {X}} {\mathbb {E}} \left[ \underset {Y | \mathbf {x} _ {S}, x _ {j}} {\mathbb {E}} \left[ \log \frac {p (y \mid \mathbf {x} _ {S} , x _ {j})}{p (y \mid \mathbf {x} _ {S})} \right] \right] \\ = \underset {\mathcal {D} _ {X}} {\mathbb {E}} \left[ D _ {K L} \left(p _ {Y | \mathbf {x} _ {S}, x _ {j}} \parallel p _ {Y | \mathbf {x} _ {S}}\right) \right] \\ \end{array}
$$
By Gibbs's inequality, the KL-divergence between two distributions is zero iff they are equal, so setting this value to zero for all  $\mathbf{x}$  satisfies the first definition of conditional independence above. For the latter, we simply point out that:
$$
\underset {\mathbf {x} \sim \mathcal {D} _ {X}} {\mathbb {E}} \left[ \Delta_ {I G} (S, j, \mathbf {x}) \right] = I (Y; X _ {j} \mid \mathbf {X} _ {S}).
$$
Since conditional mutual information equals zero iff the relevant variables are conditionally independent, this satisfies the second definition above.
Item (b) states that CSI, which is strictly weaker than standard conditional independence, is also sufficient for zero marginal payout at a given point  $\mathbf{x}$ . This follows directly from the sufficiency argument above.
The converse relationship is more complex, however. Call a distribution **conspiratorial** if there exists some  $S, j, \mathbf{x}$  such that  $\Delta_v(S, j, \mathbf{x}) = 0 \land Y \nsubseteq X_j \mid \mathbf{x}_S$  for some  $v \in \{v_{KL}, v_{CE}, v_{IG}, v_H\}$ . Such distributions are so named because the relevant probabilities must coordinate in a very specific way to guarantee summation to zero as we marginalize over  $\mathcal{V}$ . As a concrete example, consider the following data generating process:
$$
X \sim \operatorname {B e r n} (0. 5), \quad Z \sim \operatorname {B e r n} (0. 5), \quad Y \sim \operatorname {B e r n} (0. 3 + 0. 4 X - 0. 2 Z).
$$
What is the contribution of  $X$  to coalition  $S = \emptyset$  when  $X = 1$  and  $Z = 1$ ? In this case, we have neither global nor context-specific independence, i.e.  $Y \not\perp X$ . Yet, evaluating the payoffs in a KL-divergence game, we have:
$$
\begin{array}{l} \Delta_ {K L} (S, j, \mathbf {x}) = \sum_ {y} P (y \mid X = 1, Z = 1) \log \frac {P (y \mid X = 1)}{P (y)} \\ = 0. 5 \log \frac {0 . 4}{0 . 6} + 0. 5 \log \frac {0 . 6}{0 . 4} \\ = 0. \\ \end{array}
$$
In this case, we find that negative and positive values of the log ratio cancel out exactly as we marginalize over  $\mathcal{V}$ . (Similar examples can be constructed for all our information theoretic games.) This shows that CSI is sufficient but not necessary for  $\Delta_v(S,j,\mathbf{x}) = 0$ .
However, just because conspiratorial distributions are possible does not mean that they are common. Item (c) states that the set of all such distributions has Lebesgue measure zero. Our proof strategy here follows that of Meek [48], who demonstrates a similar result in the case of unfaithful distributions, i.e. those whose (conditional) independencies are not entailed by the data's underlying graphical structure. This is an important topic in the causal discovery literature (see, e.g., [91, 92]).
For simplicity, assume a discrete state space  $\mathcal{X} \times \mathcal{Y}$ , such that the data generating process is fully parametrized by a table of probabilities  $T$  for each fully specified event. Fix some  $S, j$  such that  $Y \nsubseteq X_{j} \mid \mathbf{x}_{\mathbf{s}}$ . Let  $C$  be the number of possible outcomes,  $\mathcal{Y} = \{y_{1}, \ldots, y_{C}\}$ . Define vectors  $\mathbf{p}, \mathbf{q}, \mathbf{r}$  of length  $C$  such that, for each  $c \in [C]$ :
$$
p _ {c} = p (y _ {c} \mid \mathbf {x}), \quad q _ {c} = p (y _ {c} \mid \mathbf {x} _ {S}, x _ {j}) - p (y _ {c} \mid \mathbf {x} _ {S}), \quad r _ {c} = \log \frac {p (y _ {c} \mid \mathbf {x} _ {S} , x _ {j})}{p (y _ {c} \mid \mathbf {x} _ {S})},
$$
and stipulate that  $p(y_{c} \mid \mathbf{x}_{S}) > 0$  for all  $c \in [C]$  to avoid division by zero. Observe that these variables are all deterministic functions of the parameters encoded in the probability table  $T$ . By the assumption of local conditional dependence, we know that  $\| \mathbf{r} \|_{0} > 0$ . Yet for our conspiracy to obtain, the data must satisfy  $\mathbf{p} \cdot \mathbf{r} = 0$ . A well-known algebraic lemma of Okamoto [52] states that if a polynomial constraint is non-trivial (i.e., if there exists some settings for which it does not hold), then the subset of parameters for which it does hold has Lebesgue measure zero. The log ratio  $\mathbf{r}$  is not polynomial in  $T$ , but the difference  $\mathbf{q}$  is. The latter also has a strictly positive  $L_{0}$  norm by the assumption of local conditional independence. Crucially, the resulting dot product intersects with our previous dot product at zero. That is, though differences are in general a non-injective surjective function of log ratios, we have a bijection at the origin whereby  $\mathbf{p} \cdot \mathbf{r} = 0 \Leftrightarrow \mathbf{p} \cdot \mathbf{q} = 0$ . Thus the conspiracy requires nontrivial constraints that are linear in  $\mathbf{p}$ ,  $\mathbf{q}$  themselves polynomial in the system parameters  $T$  so we conclude that the set of conspiratorial distributions has Lebesgue measure zero.
Proof of Thm. 5.1. Our proof is an application of the split conformal method (see [41, Thm. 2.2]). Whereas that method was designed to bound the distance between predicted and observed outcomes for a regression task, we adapt the argument to measure the concentration of Shapley values for a given feature. To achieve this, we replace out-of-sample absolute residuals with out-of-sample Shapley values and drop the symmetry assumption, which will not generally apply when features are informative. The result follows immediately from the exchangeability of  $\phi(j, \mathbf{x}^{(i+1)})$  and  $\phi(j, \mathbf{x}^{(i)})$ ,  $i \in \mathcal{I}_2$ , which is itself a direct implication of the i.i.d. assumption. Since bounds are calculated so as to cover  $(1 - \alpha) \times 100\%$  of the distribution, it is unlikely that new samples will fall outside this region. Specifically, such exceptions occur with probability at most  $\alpha$ . This amounts to a sort of PAC guarantee, i.e. that Shapley values will be within a data-dependent interval with probability at least  $1 - \alpha$ . The interval can be trivially inverted to compute an associated  $p$ -value.
We identify three potential sources of error in estimating upper and lower quantiles of the Shapley distribution: (1) learning the conditional entropy model  $h$  from finite data; (2) sampling values for out-of-coalition features  $\overline{S}$ ; (3) sampling coalitions  $S$ . Convergence rates as a function of (1) and (2) are entirely dependent on the selected subroutines. With consistent methods for both, conformal prediction bands are provably close to the oracle band (see [41, Thm. 8]). As for (3), Williamson and Feng [88] show that with efficient estimators for (1) and (2), as well as an extra condition on the minimum number of subsets, sampling  $\Theta(n)$  coalitions is asymptotically optimal, up to a constant factor.
B Addenda
This section includes extra background material on information theory and Shapley values.
B.1 Information Theory
Let  $p, q$  be two probability distributions over the same  $\sigma$ -algebra of events. Further, let  $p, q$  be absolutely continuous with respect to some appropriate measure. The entropy of  $p$  is defined as  $H(p) \coloneqq \mathbb{E}_p[-\log p]$ ,
i.e. the expected number of bits required to encode the distribution. The cross entropy of  $p$  and  $q$  is defined as  $H(p, q) \coloneqq \mathbb{E}_p[-\log q]$ , i.e. the expected number of bits required to encode samples from  $p$  using code optimized for  $q$ . The KL-divergence between  $p$  and  $q$  is defined as  $D_{KL}(p \parallel q) \coloneqq \mathbb{E}_p[\log^p / q]$ , i.e. the cost in bits of modeling  $p$  with  $q$ . These three quantities are related by the formula  $D_{KL}(p \parallel q) = H(p, q) - H(p)$ . The reduction in  $Y$ 's uncertainty attributable to  $X$  is also called the mutual information,  $I(Y; X) \coloneqq H(Y) - H(Y \mid X)$ . This quantity is nonnegative, with  $I(Y; X) = 0$  if and only if the variables are independent.
However, conditioning on a specific value of  $X$  may increase uncertainty in  $Y$ , in which case the local posterior entropy exceeds the prior. Thus it is possible that  $H(Y \mid x) > H(Y)$  for some  $x \in \mathcal{X}$ . For example, consider the following data generating process:
$$
X \sim \operatorname {B e r n} (0. 8), \quad Y \sim \operatorname {B e r n} (0. 5 + 0. 2 5 X).
$$
In this case, we have  $P(Y = 1) = 0.7$ ,  $P(Y = 1 \mid X = 0) = 0.5$ , and  $P(Y = 1 \mid X = 1) = 0.75$ . It is easy to see that even though the marginal entropy  $H(Y)$  exceeds the global conditional entropy  $H(Y \mid X)$ , the local entropy at  $X = 0$  is larger than either quantity,  $H(Y \mid X = 0) > H(Y) > H(Y \mid X)$ . In other words, conditioning on the event  $X = 0$  increases our uncertainty about  $Y$ .
Similarly, there may be cases in which  $I(Y; X \mid Z) > 0$ , but  $I(Y; X \mid z) = 0$ . This is what Boutilier et al. [7] call context-specific independence (CSI). For instance, if  $X, Z \in \{0,1\}^2$  and  $Y := \max \{X,Z\}$ , then we have  $Y \not\perp X \mid Z$ , but  $Y \perp X \mid (Z = 1)$  since  $Y$ 's value is determined as soon as we know that either parent is 1.
B.2 The Shapley Axioms
For completeness, we here list the Shapley axioms.
Efficiency. Shapley values sum to the difference in payoff between complete and null coalitions:
$$
\sum_ {j = 1} ^ {d} \phi (j, \mathbf {x}) = v ([ d ], \mathbf {x}) - v (\emptyset , \mathbf {x}).
$$
Symmetry. If two players make identical contributions to all coalitions, then their Shapley values are equal:
$$
\forall S \subseteq [ d ] \backslash \{i, j \}: v (S \cup \{i \}, \mathbf {x}) = v (S \cup \{j \}, \mathbf {x}) \Rightarrow \phi (i, \mathbf {x}) = \phi (j, \mathbf {x}).
$$
Sensitivity. If a player makes zero contribution to all coalitions, then its Shapley value is zero:
$$
\forall S \subseteq [ d ] \backslash \{j \}: v (S \cup \{j \}, \mathbf {x}) = v (S, \mathbf {x}) \Rightarrow \phi (j, \mathbf {x}) = 0.
$$
Linearity. The Shapley value for a convex combination of games can be decomposed into a convex combination of Shapley values. For any  $a, b \in \mathbb{R}$  and value functions  $v_{1}, v_{2}$ , we have:
$$
\phi_ {a \cdot v _ {1} + b \cdot v _ {2}} (j, \mathbf {x}) = a \phi_ {v _ {1}} (j, \mathbf {x}) + b \phi_ {v _ {2}} (j, \mathbf {x}).
$$
C.1 Datasets.
The MNIST dataset is available online. The IMDB dataset is available on Kaggle. For the tabular data experiment in Sect. 6.1, we generate  $Y$  according to the following process:
$$
\mu (\mathbf {x}) := \boldsymbol {\beta} ^ {\top} \mathbf {x}, \quad \sigma^ {2} (\mathbf {x}) := \exp (\boldsymbol {\gamma} ^ {\top} \mathbf {x}),
$$
$$
Y \mid \mathbf {x} \sim \mathcal {N} (\mu (\mathbf {x}), \sigma^ {2} (\mathbf {x})).
$$
Coefficients  $\beta, \gamma$  are independent Rademacher distributed random vectors of length 4.
The BreastCancer, Diabetes, Ionosphere, and Sonar datasets are all distributed in the mlbench package, which is available on CRAN.
Though the term "bit" is technically reserved for units of information measured with logarithmic base 2, we use the word somewhat more loosely to refer to any unit of information.
$^{5}$ http://yann.lecun.com/exdb/mnist/.
$^{6}$ https://www.kaggle.com/datasets/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews.
<sup>7</sup>https://cran.r-project.org/web/packages/mlbench/index.html.
Table 1: Estimated quantiles and nominal coverage at  $\alpha = 0.1$  for Shapley values from the conditional mean and conditional variance models. Results are averaged over 50 replicates.
Feature	Mean	Variance
\( \hat{q}_{\text{lo}} \)	\( \hat{q}_{\text{hi}} \)	Coverage	\( \hat{q}_{\text{lo}} \)	\( \hat{q}_{\text{hi}} \)	Coverage
\( X_1 \)	-0.150	0.070	0.898	-0.512	0.541	0.900
\( X_2 \)	-0.138	0.127	0.897	-0.409	0.588	0.900
\( X_3 \)	-0.070	0.091	0.904	-0.227	0.302	0.900
\( X_4 \)	-0.094	0.096	0.900	-0.477	1.025	0.898
\( X_5 \)	-0.149	0.093	0.904	-0.266	0.324	0.901
\( X_6 \)	-4.310	2.203	0.895	-0.041	0.057	0.904
\( X_7 \)	-4.496	2.268	0.897	-0.047	0.103	0.902
\( X_8 \)	-1.279	2.293	0.898	-0.047	0.051	0.897
\( X_9 \)	-4.588	4.134	0.900	-0.052	0.129	0.900
\( X_{10} \)	-1.886	1.927	0.898	-0.051	0.065	0.902
Figure 5: Binary MNIST figure (A) and multiclass MNIST (B), featuring a "Mean" column for standard SHAP outputs. For this column, red/blue pixels indicate features that increase/decrease the probability of the true class label. These values are anticorrelated with epistemic uncertainty, as expected. The effect is more salient in the binary setting, since probability mass is spread out over 10 classes in the multiclass model.
C.2 Models.
All neural network training was conducted in PyTorch [59]. For the MNIST experiment, we train a deep neural network with the following model architecture: (1) A convolutional layer with 10 filters of size  $5 \times 5$ , followed by max pooling of size  $2 \times 2$ , ReLU activation, and a dropout layer with probability 0.3. (2) A convolutional layer with 20 filters of size  $5 \times 5$ , followed by a dropout layer, max pooling of size  $2 \times 2$ , ReLU activation, and a dropout layer with probability 0.3. (3) Fully connected (dense) layer with 320 input features and 50 output units, followed by ReLU activation and a dropout layer. (4) Fully connected layer with 50 input features and 10 output units, followed by softmax activation. We train with a batch size of 128 for 20 epochs at a learning rate of 0.01 and momentum 0.5. For Monte Carlo dropout, we do 50 forward passes to sample  $B = 50$  subnetworks. For the IMDB experiment, we use a pre-trained BERT model from the Hugging Face transformers library. $^{8}$  All hyperparameters are set to their default values. All XGBoost models are trained with the default hyperparameters, with the number of training rounds cited in the text.
C.3 Coverage
To empirically test our conformal coverage guarantee, we compute quantiles for out-of-sample Shapley values on the modified Friedman benchmark. Results for conditional expectation and conditional variance are reported in Table 1, with target level  $\alpha = 0.1$ . Note that what constitutes a "small" or "large" interval is context-dependent. The conditional variance model is fit to  $\epsilon_y^2$ , which has a tighter range than  $Z$ , leading to smaller Shapley values on average. However, nominal coverage is very close to the target  $90\%$  throughout, illustrating how the conformal method can be used for feature selection and outlier detection.
<sup>8</sup>https://huggingface.co/docs/transformers/model_doc/bert.
C.4 Extended MNIST
We supplement the original MNIST experiment with extra examples, including a mean column for the true label (Fig. 5A) and augmenting the binary problem with the classic 10-class task (Fig. 5B). We find that mean and entropy are highly associated in the binary setting, as the analytic formulae suggest, while the relationship is somewhat more opaque in the 10-class setting, since we must average over a larger set of signals.