SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuling Yao
Flatiron Institute,
New York, NY 10010.
yyao@yyao.dev
Justin Domke
University of Massachusetts,
Amherst, MA 01002.
domke@cs.umass.edu
To check the accuracy of Bayesian computations, it is common to use rank-based simulation-based calibration (SBC). However, SBC has drawbacks: The test statistic is somewhat ad-hoc, interactions are difficult to examine, multiple testing is a challenge, and the resulting p-value is not a divergence metric. We propose to replace the marginal rank test with a flexible classification approach that learns test statistics from data. This measure typically has a higher statistical power than the SBC test and returns an interpretable divergence measure of miscalibration, computed from classification accuracy. This approach can be used with different data generating processes to address simulation-based inference or traditional inference methods like Markov chain Monte Carlo or variational inference. We illustrate an automated implementation using neural networks and statistically-inspired features, and validate the method with numerical and real data experiments.
Simulation based calibration (SBC) is a default approach to diagnose Bayesian computation. SBC was originally designed to validate if computer software accurately draws samples from the exact posterior inference, such as Markov chain Monte Carlo (MCMC, [4, 38, 26]) and variational inference [41]. With recent advances in amortized and simulation-based inferences [6] and growing doubt on the sampling quality [23, 18], there has also been an increasing trend to apply SBC to likelihood-free inference such as approximate Bayesian computation [42] and normalizing-flow-based [31, 1] neural posterior sampling [22, 20], with a wide range of applications in science [15, 7, 34].
Bayesian computation tries to sample from the posterior distribution  $p(\theta | y)$  given data  $y$ . We work with the general setting where it may or may not be possible to evaluate the likelihood  $p(y | \theta)$ . Suppose we have an inference algorithm or software  $q(\theta | y)$  that attempts to approximate  $p(\theta | y)$ , and we would like to assess if this  $q$  is calibrated, meaning if  $q(\theta | y) = p(\theta | y)$  for all possible  $\theta, y$ . Simulation based calibration involves three steps: First, we draw a  $\theta$  from the prior distribution  $p(\theta)$ . Second, we simulate a synthetic observation  $y$  from the data model  $p(y | \theta)$ . Third, given  $y$  we draw a size- $M$  posterior sample  $\tilde{\theta}_1, \ldots, \tilde{\theta}_M$  from the inference engine  $q(\theta | y)$  that we need to diagnose. SBC traditionally computes the rank statistic of the prior draw  $\theta$  among the  $q$  samples, i.e.  $r = \sum_{m=1}^{M} \mathbb{1}(\theta \leq \tilde{\theta}_m)$ . If the inference  $q$  is calibrated, then given  $y$  both  $\theta$  and  $\tilde{\theta}_m$  are from the same distribution  $p(\theta | y)$ , hence with repeated simulations of  $(\theta, y)$ , we should expect such rank statistics  $r$  to appear uniform, which can be checked by a histogram visualization or a formal uniformity test.
Despite its popularity, rank-based SBC has limitations: (i) We only compute the rank of univariate parameters. In practice,  $\theta$  and  $y$  are high dimensional. We typically run SBC on each component of  $\theta$  separately, this creates many marginal histograms and does not diagnose the joint distribution or interactions. We may compare ranks of some one-dimensional test statistics, but there is no method to
Figure 1: Our discriminate calibration framework has three modules (a) generate simulation table  $(\theta, y, \tilde{\theta})$  and map it into classification examples with some label  $t$  and feature  $\phi$ , (b) train a classifier to predict labels, (c) from the learned classifier, perform hypothesis testing and estimate a divergence.
find the best summary test statistic. (ii) As long as we test multiple components of  $\theta$  or test statistics, directly computing the uniformity  $p$ -values is invalid (subject to false discovery) unless we make a multiple-testing adjustment, which drops the test power (subject to false negative) in high dimensions. (iii) Often we know inference is approximate, so the final goal of diagnostic is not to reject the null hypothesis of perfect calibration but to measure the degree of miscalibration. The  $p$ -value is not such a measure: neither can we conclude an inference with  $p = .02$  is better than  $p = .01$ , nor connect the  $p$ -value with the posterior inference error. The evidence lower bound, a quantity common in variational inference, also does not directly measure the divergence due to the unknown entropy.
Heuristic: calibration via classification. To address all these drawbacks, while maintaining versatility in various computing tasks, we propose discriminative calibration, a pragmatic and unified framework for Bayesian computing calibration and divergence estimation. The intuitive heuristic behind discriminative calibration is to use a classifier to perform similarity tests—when two or several distributions are similar, we cannot distinguish samples from them so the classification error is large. In Bayesian computation, we compare conditional distributions, the true  $p(\theta | y)$  and inferred  $q(\theta | y)$ , to which we have access via simulations, and in some occasions explicit densities. It is natural to try to classify the samples drawn from  $p$  v.s. from  $q$ , and the ability to distinguish them would suggest a miscalibration between  $p$  and  $q$ .
To formulate such heuristics into an algorithm, we design a family of "label mapping" that prepares the simulations into classification examples that contain the label and feature. In Sec. 2, we first give four concrete label mapping examples, where the rank-based SBC becomes essentially a special case. Sec. 3 states the general theory: if we train a classifier to predict the labels, then the prediction ability yields a computable divergence from  $p(\theta | y)$  to  $q(\theta | y)$ . In Sec. 4, we illustrate the practical implementation to get the divergence estimate, its confidence interval, and a valid hypothesis testing  $p$ -value. We explain why the learned classifier helps statistical power. In Sec. 4.1, we discuss the classifier design and incorporate extra known information such as ranks, likelihood, and log densities, whenever available, as features. We also show our method is applicable to MCMC without waste from thinning. We illustrate numerical and cosmology data examples in Sec. 5. We review other related posterior validation approaches and discuss the limitation and future direction in Sec. 6.
2 Generate labels, run a classifier, and obtain a divergence
As with traditional SBC, we generate a simulation table by repeatedly sampling parameters  $\theta$  and synthetic data  $y$  from the target model  $p$ , i.e. draws  $(\theta, y) \sim p(\theta, y)$ . Then, for each  $(\theta, y)$ , we run the inference routine  $q$  to obtain a set of  $M$  IID approximate posterior samples  $\tilde{\theta}_1, \dots, \tilde{\theta}_M \sim q(\theta | y)$ . We wish to assess how close, on average (over different  $y$ ), the inference procedure  $q(\theta | y)$  is to the true posterior  $p(\theta | y)$ . Here, we observe that classification example-sets can be created in several ways and these produce different divergences. We generalize the four examples and the claims on divergence in Sec. 3.
Example 1: Binary classification with full features. An intuitive way to estimate this closeness between  $q(\theta | y)$  and  $p(\theta | y)$  is to train a binary classifier.
Imagine creating a binary classification dataset of  $(t,\phi)$  pairs, where  $t$  is a binary label, and  $\phi$  are features. For each  $(\theta ,y)$  simulated from  $p$ ,  $M + 1$  pairs of examples are created. In the first,  $t = 0$  and  $\phi = (\theta ,y)$ . In the others,  $t = 1$ , and  $\phi = (\tilde{\theta}_m,y)$ ,  $1\leq m\leq M$ . Collecting all data across
one simulation run:
$(y,\theta ,\bar{\theta}_1,\dots ,\bar{\theta}_M)$
$M + 1$  examples:
label t	features φ
0	(θ, y)
1	(θ1, y)
1	...
1	(θM, y)
$1 \leq i \leq S$ , we obtain  $S(M + 1)$  pairs of  $(t, \phi)$  classification examples. A binary classifier is then trained to maximize the conditional log probability of  $t$  given  $\phi$ . If inference  $q$  were exact, no useful classification would be possible. In that case, the expected test log predictive density could be no higher than the negative binary entropy  $h(w) \coloneqq w \log w + (1 - w) \log (1 - w)$  of a Bernoulli distribution with parameter  $w \coloneqq 1 / (M + 1)$ .
Now imagine drawing a validation set in the same way, and evaluating the log predictive density of the learned classifier. We will show below (Thm. 1) that the expected log predicted density [10]  $\mathrm{ELPD} = \mathbb{E}\log \operatorname*{Pr}(t|\phi)$  of the classifier on validation data is a lower bound to a divergence  $D_{1}$  between  $p(\theta |y)$  and  $q(\theta |y)$  up to the known constant  $h(w)$ ,
$$
\mathrm {E L P D} _ {1} - h (w) \leq D _ {1} (p, q) := w \mathrm {K L} (p (\theta | y) \| r (\theta | y)) + (1 - w) \mathrm {K L} (q (\theta | y) \| r (\theta | y)), \tag {1}
$$
where  $w = 1 / (M + 1)$  and  $r(\theta |y) = wp(\theta |y) + (1 - w)q(\theta |y)$  is a mixture of posterior density. If the classifier  $c$  is optimal ( $c(t|\phi) = \operatorname*{Pr}(t|\phi)$  in the distribution) then the bound in the above equation is tight  $\max_{\mathrm{classifiers}} \mathrm{ELPD}_1 - h(w) = D_1(p,q)$ . Here  $\mathrm{KL}(p(\theta |y)\| q(\theta |y))$  denotes a standard conditional Kullback-Leibler divergence<sup>1</sup>. By optimizing the classifier,  $\max \mathrm{ELPD}_1 - h(w)$  becomes a commutable divergence, and its approaching zero is a necessary and sufficient condition for perfect calibration since  $D_1(p,q) = 0$  if and only if  $p(\theta |y) = q(\theta |y)$  almost everywhere.
Example 2: Binary classification without  $y$ . Similar to Example 1, from each simulation draw we generate  $M + 1$  pairs of  $(t,\phi)$  examples, except that the feature  $\phi$  only contains the parameters  $\theta$ , not  $y$ . A binary classifier is then trained to predict  $t$  given  $\phi$ . The ELPD of this classifier on validation data is a lower bound to a generalized divergence  $D_{2}$  between the prior  $p(\theta)$  and  $q(\theta)$ , up to a known constant  $h(w)$
t	φ
0	θ
1	θ1
1	.
1	θM
$$
\mathrm {E L P D} _ {2} - h (w) \leq D _ {2} (p, q) := w \operatorname {K L} (p (\theta) \| r (\theta)) + (1 - w) \operatorname {K L} (q (\theta) \| r (\theta)), \tag {2}
$$
where  $w = (M + 1)^{-1}$ ,  $r(\theta) = wp(\theta) + (1 - w)q(\theta)$  is the prior mixture, and the bound is tight when the classifier is optimal. A large ELPD reveals the difference between the inference  $q(\theta | y)$  and  $p(\theta | y)$ . But  $D_{2}$  is only a generalized divergence:  $D_{2} = 0$  is necessary not sufficient for  $q(\theta | y) = p(\theta | y)$ .
Example 3: Binary classification with ranks (where the classical SBC is a special case). Instead of classification using full  $(\theta, y)$ , we construct a feature: the rank statistics. From each simulation draw we generate  $M + 1$  pairs of  $(t, \phi)$  examples. The first pair is  $t = 0$  and  $\phi = \sum_{m=1}^{M} \mathbb{1}(\theta < \tilde{\theta}_m)$ , the rank statistics of the prior draw. In the others,  $t = 1$  and  $\phi = \sum_{m'=1}^{M} \mathbb{1}(\tilde{\theta}_m < \tilde{\theta}_{m'}) + \mathbb{1}(\tilde{\theta}_m < \theta)$ ,  $1 \leq m' \leq M$  are the rank statistics of the inferred samples. A binary
Output: M + 1 examples:
t	φ
0	∑m=1M Π(θ ≤ θm)
1	∑m'=1M Π(θ1 ≤ θm')
1	...
1	∑m'=1M Π(θM ≤ θm')
classifier is then trained to predict  $t$  given  $\phi$ . The ELPD of this classifier is a lower bound to a generalized divergence  $D_{3}$  between  $p(\theta |y)$  and  $q(\theta |y)$  up to a known constant  $h(w)$
$$
\mathrm {E L P D} _ {3} - h (w) \leq D _ {3} (p, q) := D _ {2} (Z (p, q) \| \text {U n i f o r m} (0, 1)), w = 1 / (M + 1), \tag {3}
$$
and again the bound is tight if the classifier is optimal. Here  $Z(p,q)$  is a random variable defined by  $Z = Q(\theta |y), (\theta ,y)\sim p(\theta ,y)$ , where  $Q$  is the cumulative distribution function of  $q(\theta |y)$ .
Training a "generative" classifier on this rank-based label-generating map is similar to testing for uniformity in rank statistics, as done in traditional SBC which estimates the distribution of  $r|t = 0$  by histograms (See Appendix A.1 for precise correspondence between SBC and the naive Bayes classifier). The success of SBC suggests the usefulness of ranks, which motivates us to include ranks or more generally feature engineering in the classifier. However,  $D_{3}$  is only a generalized divergence:  $D_{3} = 0$  is necessary but not sufficient for  $p(\theta |y) = q(\theta |y)$ . If inference always returns the prior,  $q(\theta |y) = p(\theta)$ , then  $D_{3}(p,q) = 0$ , a known counterexample of when rank-based SBC fails [32].
Example 4: Multi-class classification. We go beyond binary labeling. Given a simulation run  $(y, \theta, \tilde{\theta}_1, \dots, \tilde{\theta}_M)$ , we now create an  $(M + 1)$ -class classification dataset with  $M + 1$  pairs of  $(t, \phi)$  examples. In each one, the features are  $\phi = (y, \theta^*)$  where  $\theta^*$  is a permutation of  $(\theta, \tilde{\theta}_1, \dots, \tilde{\theta}_M)$  that moves  $\theta$  into a given location and  $t \in [0, \dots, M]$  indicates the location of  $\theta$  in the permutation
Output:  $M + 1$  examples:
t	φ
0	(θ,θ1,θ2,···,θM,y)
1	(θ1,θ,θ2···,θM,y)
.	...
M	(θ1,θ2···,θM,θ,y)
${}^{1}$  Standard notation for conditional divergences [5] is that  $\operatorname{KL}\left( {p\left( {\theta  \mid  y}\right) \parallel q\left( {\theta  \mid  y}\right) }\right)  \mathrel{\text{:=}} {\mathbb{E}}_{p\left( {y,\theta }\right) }\log \frac{p\left( {\theta  \mid  y}\right) }{q\left( {\theta  \mid  y}\right) }$  . Conditional divergence is not the divergence of conditional distributions. We interpret  $p\left( y\right)  = q\left( y\right)$  .
(See the table on the right). We train a  $(M + 1)$ -class classifier to predict  $t$  from  $\phi$ . The ELPD on a validation set is a lower bound to the following divergence  $D_{4}$  between  $p(\theta |y)$  and  $q(\theta |y)$  up to a known constant:
$$
\mathrm {E L P D} _ {4} + \log (M + 1) \leq D _ {4} (p, q) := \mathrm {K L} \left(p \left(\theta_ {0}\right) \prod_ {k = 1} ^ {M} q \left(\theta_ {k}\right), \frac {1}{M + 1} \sum_ {m = 0} ^ {M} p \left(\theta_ {m}\right) \prod_ {k \neq m} q \left(\theta_ {k}\right)\right). \tag {4}
$$
Again the bound is tight if the classifier is optimal. The divergence  $0 \leq D_4 \leq \log(M + 1)$ , and  $D_4 = 0$  if and only if  $p(\theta | y) \stackrel{\text{a.e.}}{=} q(\theta | y)$ , necessary and sufficient for calibration. In Theorem 3 we shows that as  $M \to \infty$ ,  $D_4(p, q)$  converges to  $\mathrm{KL}(p(\theta | y), q(\theta | y))$  at an  $O(1 / M)$  convergence rate.
3 Theory on calibration divergence
To generalize the previous examples, we define a "label mapping"  $\Phi : (y, \theta, \tilde{\theta}_1, \ldots, \tilde{\theta}_M) \mapsto \{(t_1, \phi_1), \ldots, (t_L, \phi_L)\}$ . that maps one simulation run  $(y, \theta, \tilde{\theta}_1, \ldots, \tilde{\theta}_M)$  into a  $K$ -class classification example-set containing  $L$  pairs of labels  $t$  and features  $\phi$ . The label  $t_l \in \{0, 1, \ldots, K - 1\}$  is deterministic and only depends on  $l$ . The features  $\phi_l$  can depend on  $l$  and  $(y, \theta, \tilde{\theta}_1, \ldots, \tilde{\theta}_M)$ . We only consider  $\Phi$  satisfying that, when  $p(\theta | y) = q(\theta | y)$ ,  $\phi$  given  $y$  is conditionally independent of  $t$  (equivalently,  $\phi|(y, t)$  has the same distribution for any  $t$ ). Let  $\mathbb{F}$  be the set of these mappings  $\Phi$ .
We train a classifier on the classification examples created by  $\Phi$  collected across repeated sampling. The classifier performance is measured by its expected log predictive density (ELPD), or equivalently the negative cross-entropy loss. Given  $p$ ,  $q$ , and the mapping  $\Phi$ , let  $c(\phi)$  be any probabilistic classifier that uses  $\phi$  to predict the label  $t$ , and  $\operatorname{Pr}(k|\phi, c)$  is the predicted  $k$ -th class probability. Taking expectations over features and labels with  $(\theta, y) \sim p$  and  $\tilde{\theta}_m \sim q(\theta|y)$  reaches the ELPD of the classifier,
$$
\operatorname {E L P D} (\Phi , c) := \mathbb {E} _ {t, \phi} \log \Pr (t = k | \phi , c), \quad (t, \phi) = \Phi (y, \theta , \tilde {\theta} _ {1}, \dots , \tilde {\theta} _ {M}) \tag {5}
$$
We then define the prediction ability  $D(p, q, \Phi, c)$  to be the ELPD plus the entropy of a categorical distribution, i.e.
$$
D (p, q, \Phi , c) = \operatorname {E L P D} (\Phi , c) - \sum_ {k = 0} ^ {K - 1} w _ {k} \log w _ {k}, \quad \text {w h e r e} w _ {k} = \frac {1}{L} \sum_ {l = 1} ^ {L} \mathbb {1} \left(t _ {l} = k\right). \tag {6}
$$
The optimal classifier is the  $c$  that achieves the highest prediction ability in the population:
$$
D ^ {\mathrm {o p t}} (p, q, \Phi) := \max  _ {c \in \mathcal {C}} D (p, q, \Phi , c), \text {w h e r e} \mathcal {C} \text {i s t h e s e t o f a l l p r o b a b i l i s t i c c l a s s i f i e r s .} \tag {7}
$$
The next theorem is the basic theory of our method: as long as we pass the simulation draws to a label mapping  $\Phi \in \mathbb{F}$ , and train a classifier on the classification examples, then  $D^{\mathrm{opt}}(p,q,\Phi)$  is a generalized divergence between  $p$  and  $q$ .
Theorem 1 (Prediction ability yields divergence). Given any  $p, q$ , and feature mapping  $\Phi \in \mathbb{F}$ , the optimal prediction ability  $D^{\mathrm{opt}}(p, q, \Phi)$  is a generalized divergence from  $p$  to  $q$  in the sense that  $D^{\mathrm{opt}}(p, q, \Phi) \geq 0$ , and  $p(\theta | y) = q(\theta | y)$  almost everywhere implies  $D^{\mathrm{opt}}(p, q, \Phi) = 0$ . This generalized divergence is reparametrization invariant and uniformly bounded. For any classifier  $c$ ,
$$
0 \leq D (p, q, \Phi , c) \leq D ^ {\mathrm {o p t}} (p, q, \Phi) \leq - \sum_ {k = 0} ^ {K - 1} w _ {k} \log w _ {k}; \tag {8}
$$
$$
p (\theta | y) \stackrel {a. e.} {=} q (\theta | y) \Rightarrow D ^ {\mathrm {o p t}} (p, q, \Phi) = 0.
$$
That is, any label mapping  $\Phi$  produces a generalized divergence  $D^{\mathrm{opt}}(p,q,\Phi)$ , the prediction ability of the optimal classifier. The prediction ability  $D(p,q,\Phi,c)$  of any classifier  $c$  estimated on validation data is a lower bound to  $D^{\mathrm{opt}}(p,q,\Phi)$ . Further, this generalized divergence  $D^{\mathrm{opt}}(p,q,\Phi)$  is always a proper Jensen-Shannon divergence in the projected feature-label space (Theorem 1 in the Appendix).
When  $p(\theta |y) \neq q(\theta |y)$ , to increase the statistical power, we wish that the generalized divergence can be as "strong" as possible such that we can detect the miscalibration. In the four examples in Section 2, we have used  $D_{1},D_{2},D_{3},D_{4}$  to denote the (generalized) divergence they yield. The next theorem shows there is a deterministic domination order among these four metrics. Moreover,  $D_{4}$  is the largest possible classification divergence from any given simulation table.
Theorem 2 (Strongest divergence). For any given  $p, q$ , and any  $\Phi \in \mathbb{F}$ , (1)  $D_4 \geq D_1 \geq D_3 \geq D_2$ . (2)  $D_4$  and  $D_1$  are proper divergences. They attain 0 if and only if  $p(\theta | y) = q(\theta | y)$  almost everywhere. They attain the corresponding upper bound in (8) if and only if  $p(\theta | y)$  are disjoint, i.e.,  $\int_A p(\theta | y) q(\theta | y) d\theta = 0$  for any measurable set  $A$  and almost surely  $y$ . (3) For any  $p, q$  and  $\Phi \in \mathbb{F}$  ( $\Phi$  can have an arbitrary example size  $L$  and class size  $K$ ),  $D_4(p, q) \geq D^{\mathrm{opt}}(p, q, \Phi)$ .
The following result shows that the divergence  $D_4$  in Eq. (4) approaches the "mode-spanning" KL divergence in the limit that the number of posterior draws  $M \to \infty$ . This is appealing because for many inference routes, increasing the number of posterior draws is cheap. Thus,  $D_4$  provides an accessible approximation to the KL divergence that is otherwise difficult to compute from samples.
Theorem 3 (Big  $M$  limit and rate). For any  $p, q$ , generate the simulation  $\{(y, \theta, \tilde{\theta}_i, \dots, \tilde{\theta}_M)\}$  and train the multiclass-classier, then  $D_4(p, q) - \mathrm{KL}(p(\theta | y) || q(\theta | y)) \to 0$ , as  $M \to \infty$ .
If further  $p(\theta | y)$  and  $q(\theta | y)$  have the same support, and if  $\mathbb{E}_{p(\theta | y)}\left[\frac{p(\theta | y)}{q(\theta | y)}\right]^2 < \infty$  for a.s.  $y$ , then
$$
D _ {4} (p, q) = \mathrm {K L} (p (\theta | y) | | q (\theta | y)) - \frac {1}{2 M} \chi^ {2} (q (\theta | y) | | p (\theta | y)) + o (M ^ {- 1}).
$$
where  $\chi^2 (\cdot |\cdot)$  is the conditional chi-squared divergence.
4 Practical implementation
Algorithm 1: Proposed method: Discriminative calibration
input : The ability to sample from  $p\left( {\theta ,y}\right)$  and  $q\left( {\theta  \mid  y}\right)$  ,and a label mapping  $\Phi$  . output: (i) estimate of a divergence between  $p\left( {\theta  \mid  y}\right)$  and  $q\left( {\theta  \mid  y}\right)$  ; (ii)  $p$  -value for testing  $p\left( {\theta  \mid  y}\right)  = q\left( {\theta  \mid  y}\right)$  . for  $\left( {i = 1 : S}\right)$  Sample  $\left( {{\theta }_{i},{y}_{i}}\right)  \sim  p\left( {\theta ,y}\right)$  ,and sample  ${\widetilde{\theta }}_{i1},{\widetilde{\theta }}_{i2}\ldots ,{\widetilde{\theta }}_{iM} \sim  q\left( {\theta  \mid  {y}_{i}}\right)$  ; > simulation table Generate a batch of  $L$  examples of  $\left( {t,\phi }\right)  = \Phi \left( {{y}_{i},{\theta }_{i},{\widetilde{\theta }}_{i1},\ldots ,{\widetilde{\theta }}_{iM}}\right) ,0 \leq  t \leq  K - 1; >$  label mapping Randomly split the LS classification examples  $\left( {t,\phi }\right)$  into training and validation sets (all  $L$  examples for a given  $i$  go to either training or validation); Train a  $K$  -class classifier to predict label  $t$  on the training examples, incorporating useful features; Compute the validation log predictive density LPDval in (9), obtain an estimate of the divergence (7) and its bootstrap confidence intervals; >divergence for  $\left( {b = 1 : B}\right)$  Randomly permute the label  $t$  in the validation set within each batch; Compute LPDb on the permutated validation set; Compute the calibration  $p$  -value  $p = 1/B\mathop{\sum }\limits_{{b = 1}}^{B}{1}\left( {{\mathrm{{LPD}}}_{b}^{\mathrm{{val}}} \geq  {\mathrm{{LPD}}}^{\mathrm{{val}}}}\right)$  . >frequentist test
Workflow for divergence estimate. We repeat the simulation of  $(y,\theta ,\tilde{\theta}_1,\dots ,\tilde{\theta}_M)$  for  $S$  times. Each time we sample  $(y,\theta)\sim p(\theta ,y)$  and  $\tilde{\theta}_{1:M}\sim q(\theta |y)$  and generate a batch of  $L$  examples through a label mapping  $\Phi \in \mathbb{F}$ . In total, we obtain  $SL$  pairs of  $(t,\phi)$  classification examples. We recommend using the binary and multiclass labeling schemes (Examples 1 and 4, where  $L = M + 1$  and  $K = 2$  or  $M + 1$ ) such that we can obtain a proper divergence estimate. We split the classification example-set  $\{(t,\phi)\}$  into the training and validation set (do not split batches) and train a  $K$ -class classifier  $c$  on the training set to minimize cross-entropy. Denote  $\mathcal{I}_{\mathrm{val}}$  to be the validation index,  $\operatorname*{Pr}(t = t_j|\phi_j,c)$  to be the learned class probability for any validation example  $(t_j,\phi_j)$ , we compute the ELPD (5) by the validation set log predictive density:
$$
\operatorname {E L P D} (\Phi , c) \approx \operatorname {L P D} ^ {\text {v a l}} (\Phi , c) := | \mathcal {I} _ {\text {v a l}} | ^ {- 1} \sum_ {j: \in \mathcal {I} _ {\text {v a l}}} \log \Pr (t = t _ {j} | \phi_ {j}, c). \tag {9}
$$